Spectroscopic information image estimation device, spectroscopic information image estimation method, and program

Abstract

To provide a spectroscopic information image estimation device which can estimate a spectroscopic information image (spectral reflectivity image, spectral image) in four or more bands of an imaged object easily with higher accuracy from an RGB image (image in three bands, image of each channel of an R channel, a G channel and a B channel).SOLUTION: A spectroscopic information image estimation device comprises: a first spectral reflectivity estimation unit which estimates a first spectroscopic information image indicating the spectral reflectivity of a subject on the basis of an RGB image being a photographed image of the subject, the imaging device spectral sensitivity being the spectral sensitivity of an imaging device that captures the photographed image and the light source spectral distribution being the spectral distribution of a light source of an environment where the photographed image is captured; and a second spectral reflectivity estimation unit which estimates a second spectroscopic information image obtained by correcting the first spectroscopic information image on the basis of the first spectroscopic information image estimated by the first spectral reflectivity estimation unit, the RGB image, the imaging device spectral sensitivity and the light source spectral distribution.SELECTED DRAWING: Figure 13

Description

[Technical Field]

[0001] The present invention relates to a spectral information image estimation device, a spectral information image estimation method, and a program for estimating a spectral information image (spectral image) from an RGB image. [Background technology]

[0002] Spectral reflectance is used when analyzing images or videos of an object. Therefore, in order to obtain the colorimetric values ​​of an object with high accuracy, color estimation is performed by estimating the spectral reflectance of the subject from the captured image. In this case, an object is imaged using a multiband camera, and a spectral image is obtained from the captured image as a multiband (four or more bands, for example, 31 bands) spectral reflectance. Furthermore, when estimating the spectral reflectance of an object from images captured by a multiband camera, the spectral sensitivity characteristics of each band of the multiband camera, as well as spectral statistical information of the subject, are used for the estimation.

[0003] However, because multiband cameras are expensive, a common technique involves estimating a spectral image containing multiband spectral information from images of three color bands (R channel, G channel, and B channel) captured by a commonly used three-band digital camera (hereinafter referred to as an RGB image). Here, a machine learning model (such as a CNN (Convolutional Neural Network) or deep learning) is used with RGB image data as input to estimate the spectral reflectance of each pixel in the RGB image, and a spectral image is output as the estimation result (see Non-Patent Documents 1 and 2). [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Jiaojiao Li, Chaoxiong Wu, Rui Song, Yunsong Li, Fei Liu, "Adaptive Weighted Attention Network with Camera Spectral Sensitivity Prior for Spectral Reconstruction from RGB Images", 2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), June 2020 [Non-Patent Document 2] Zhan Shi, Chang Chen, Zhiwei Xiong, Dong Liu, Feng Wu, "HSCNN+: Advanced CNN-Based Hyperspectral Recovery from RGB Images", 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), June 2018 [Overview of the project] [Problems that the invention aims to solve]

[0005] However, while Non-Patent Documents 1 and 2 can extract the spectral characteristics of each object with high accuracy, they estimate spectral radiance rather than spectral reflectance as the spectral characteristic. Furthermore, because the camera's spectral sensitivity is fixed during estimation, each machine learning model needs to be created for each imaging device. Furthermore, since the estimated spectral information is spectral radiance, it includes the spectral distribution of the light source in the imaging environment (hereinafter referred to as the light source spectral distribution). Therefore, it is necessary to estimate the spectral reflectance from the spectral radiance using the light source spectral distribution of the imaging environment.

[0006] Therefore, the machine learning models described in Non-Patent Document 1 and Non-Patent Document 2 cannot easily determine the spectral reflectance of an object from RGB images captured with any imaging device and in any imaging environment. In other words, in both Non-Patent Document 1 and Non-Patent Document 2, it is necessary to generate a machine learning model for each imaging environment to be estimated for each target imaging device. Furthermore, in both Non-Patent Document 1 and Non-Patent Document 2, it is necessary to perform a process to estimate the spectral reflectance of an object using the estimated spectral radiance and the spectral distribution of the light source.

[0007] The present invention has been made in view of the above circumstances and provides a spectral information image estimation device, spectral information image estimation method, and program that can easily and accurately estimate a spectral information image (spectral reflectance image, spectral image) of an object being imaged with four or more bands from an RGB image (a 3-band image, i.e., images of each channel of the R channel, G channel, and B channel). [Means for solving the problem]

[0008] To solve the above-mentioned problems, a spectral information image estimation device according to one aspect of the present invention comprises: a first spectral reflectance estimation unit that estimates a first spectral information image showing the spectral reflectance of a subject based on an RGB image which is an image of a subject; the spectral sensitivity of the imaging device that captured the image; and the spectral distribution of the light source in the environment in which the image was captured; and a second spectral reflectance estimation unit that estimates a second spectral information image obtained by correcting the first spectral information image based on the first spectral information image estimated by the first spectral reflectance estimation unit, the RGB image, the spectral sensitivity of the imaging device, and the spectral distribution of the light source. The first spectral reflectance estimation unit inputs the RGB image, the imaging device spectral sensitivity, and the light source spectral distribution to the first machine learning model to estimate the first spectral information image. The first machine learning model uses a training image, which is an RGB image with known spectral reflectance, the imaging spectral sensitivity of the imaging device that captured the training image, and the light source spectral distribution of the environment in which the training image was captured, as training data. The loss value is calculated from the error between the known spectral information of the training image and the spectral information of the first spectral information image estimated by the first machine learning model. It is characterized by the following: Furthermore, a spectral information image estimation device according to one aspect of the present invention includes: a first spectral reflectance estimation unit that estimates a first spectral information image showing the spectral reflectance of a subject based on an RGB image which is an image of the subject, the spectral sensitivity of the imaging device that captured the image, and the spectral distribution of the light source in the environment in which the image was captured; and a second spectral reflectance estimation unit that estimates a second spectral information image obtained by correcting the first spectral information image based on the first spectral information image estimated by the first spectral reflectance estimation unit, the RGB image, the spectral sensitivity of the imaging device, and the spectral distribution of the light source. The device comprises a rate estimation unit, the second spectral reflectance estimation unit inputs the first spectral information image, the RGB image, the imaging device spectral sensitivity, and the light source spectral distribution to a second machine learning model, estimates the second spectral information image, the second machine learning model generates an RGB image based on the spectral information of the input first spectral information image, the imaging device spectral sensitivity, and the light source spectral distribution, and corrects the first spectral information image by repeatedly updating the parameters of the second machine learning model so that the loss between the input RGB image and the generated RGB image is reduced.

[0009] The spectral information image estimation method according to one aspect of the present invention includes a first spectral reflectance estimation process in which a first spectral reflectance estimation unit estimates a first spectral information image indicating the spectral reflectance of a subject based on an RGB image that is an imaging image of the subject, an imaging device spectral sensitivity that is the spectral sensitivity of an imaging device that captured the imaging image, and a light source spectral distribution that is the spectral distribution of a light source in an environment where the imaging image was captured, and a second spectral reflectance estimation process in which a second spectral reflectance estimation unit estimates a second spectral information image obtained by correcting the first spectral information image based on the first spectral information image estimated by the first spectral reflectance estimation unit, the RGB image, the imaging device spectral sensitivity, and the light source spectral distribution. The first spectral reflectance estimation unit inputs the RGB image, the imaging device spectral sensitivity, and the light source spectral distribution to the first machine learning model and estimates the first spectral information image. The first machine learning model uses a training image, which is an RGB image with known spectral reflectance, the imaging spectral sensitivity of the imaging device that captured the training image, and the light source spectral distribution of the environment in which the training image was captured, as training data. The loss value is calculated from the error between the known spectral information of the training image and the spectral information of the first spectral information image estimated by the first machine learning model, and the model is a trained model that has been trained through machine learning. It is characterized by this. Furthermore, a spectral information image estimation method according to one aspect of the present invention includes a first spectral reflectance estimation process in which a first spectral reflectance estimation unit estimates a first spectral information image showing the spectral reflectance of the subject based on an RGB image which is an image of the subject, the spectral sensitivity of the imaging device which captured the image, and the spectral distribution of the light source in the environment in which the image was captured, and a second spectral reflectance estimation unit corrects the first spectral information image based on the first spectral information image estimated by the first spectral reflectance estimation unit, the RGB image, the spectral sensitivity of the imaging device, and the spectral distribution of the light source. The method includes a second spectral reflectance estimation process for estimating an image, wherein the second spectral reflectance estimation unit inputs the first spectral information image, the RGB image, the imaging device spectral sensitivity, and the light source spectral distribution to a second machine learning model to estimate the second spectral information image, the second machine learning model generates an RGB image based on the spectral information of the input first spectral information image, the imaging device spectral sensitivity, and the light source spectral distribution, and corrects the first spectral information image by repeatedly updating the parameters of the second machine learning model so that the loss between the input RGB image and the generated RGB image is reduced.

[0010] A program according to one aspect of the present invention causes a computer to function as a first spectral reflectance estimation means for estimating a first spectral information image indicating the spectral reflectance of a subject based on an RGB image that is an imaging image of the subject, an imaging device spectral sensitivity that is the spectral sensitivity of an imaging device that captured the imaging image, and a light source spectral distribution that is the spectral distribution of a light source in an environment where the imaging image was captured, and a second spectral reflectance estimation means for estimating a second spectral information image obtained by correcting the first spectral information image based on the first spectral information image estimated by the first spectral reflectance estimation means, the RGB image, the imaging device spectral sensitivity, and the light source spectral distribution. The first spectral reflectance estimation means inputs the RGB image, the spectral sensitivity of the imaging device, and the spectral distribution of the light source into a first machine learning model to estimate the first spectral information image. The first machine learning model uses a training image, which is an RGB image with known spectral reflectance, the imaging spectral sensitivity of the imaging device that captured the training image, and the spectral distribution of the light source in the environment where the training image was captured, as training data. The loss value is calculated from the error between the known spectral information of the training image and the spectral information of the first spectral information image estimated by the first machine learning model. . Furthermore, a program according to one aspect of the present invention includes a first spectral reflectance estimation means that estimates a first spectral information image showing the spectral reflectance of a subject based on an RGB image which is an image of a subject, the spectral sensitivity of the imaging device that captured the image, and the spectral distribution of the light source in the environment in which the image was captured, and a second spectral information means that estimates a second spectral information image obtained by correcting the first spectral information image based on the first spectral information image estimated by the first spectral reflectance estimation means, the RGB image, the spectral sensitivity of the imaging device, and the spectral distribution of the light source. The second spectral reflectance estimation means functions as a spectral reflectance estimation means, inputting the first spectral information image, the RGB image, the imaging device spectral sensitivity, and the light source spectral distribution into a second machine learning model to estimate the second spectral information image, and the second machine learning model generates an RGB image based on the spectral information of the input first spectral information image, the imaging device spectral sensitivity, and the light source spectral distribution, and corrects the first spectral information image by repeatedly updating the parameters of the second machine learning model so that the loss between the input RGB image and the generated RGB image is reduced.

Effects of the Invention

[0011] According to the present invention, it is possible to provide a spectral information image estimation device, a spectral information image estimation method, and a program that can easily estimate a spectral information image (spectral reflectance image, spectral image) of four or more bands of an object being imaged from an RGB image (a three-band image, that is, an image of each channel of the R channel, G channel, and B channel) with the same degree of accuracy as an image captured by a multi-band camera.

Brief Description of the Drawings

[0012] [Figure 1]This is a conceptual diagram illustrating an example of a machine learning model (such as a CNN (Convolutional Neural Network), neural network, or deep learning) that outputs a spectral image as spectral reflectance information, or a spectral information image, from an RGB image. [Figure 2] This is a block diagram showing an example configuration of a spectral information image estimation device according to the first embodiment. [Figure 3] This figure shows an example of the structure of a training data table that contains a dataset of training data used to train a machine learning model. [Figure 4] This is a conceptual diagram illustrating the synthesis of training data by the data synthesis unit 12. [Figure 5] This figure shows an example of the configuration of an image data table containing a dataset of image data used to estimate spectral information images in a machine learning model. [Figure 6] This figure shows an example of the configuration of the spectral information image data table in the spectral information image data storage unit 19. [Figure 7] This flowchart shows an example of the operation of the machine learning model generation unit 13 according to this embodiment, which performs a learning process using training data for a machine learning model. [Figure 8] This flowchart shows an example of the operation of the process that estimates a spectral information image from an RGB image using a pre-trained machine learning model by the spectral reflectance estimation unit 14 according to this embodiment. [Figure 9] This is a conceptual diagram illustrating the estimation of the coefficients of the basis functions that constitute the spectral reflectance curve using both a machine learning model and a corrected machine learning model. [Figure 10] This is a conceptual diagram illustrating the input of data for the spectral sensitivity of the imaging device and the spectral distribution of the light source to the intermediate layer of the machine learning model in the second embodiment. [Figure 11] This is a conceptual diagram illustrating an example of the configuration of a machine learning model used by the first spectral reflectance estimation unit 141 in the third embodiment. [Figure 12] This is a block diagram showing an example configuration of a spectral information image estimation device according to the fourth embodiment. [Figure 13] This is a block diagram showing an example configuration of a spectral information image estimation device according to the fifth embodiment. [Figure 14] This flowchart shows an example of the operation of the machine learning model training process using training data by the machine learning model generation unit 13B according to the fifth embodiment. [Figure 15] This flowchart shows an example of the operation of the process of estimating a spectral information image from an RGB image using a trained machine learning model by the spectral reflectance estimation unit 14B according to the fifth embodiment. [Figure 16] This flowchart shows an example of the operation of the learning process using training data for a corrected machine learning model by the machine learning model generation unit 13B according to the fifth embodiment. [Modes for carrying out the invention]

[0013] In this invention, a spectral information image (spectral image or spectral reflectance image) with wavelengths of four or more bands (31 bands as an example in the embodiments described later) is estimated and generated from an RGB image, which is an captured image consisting of the R channel of the color component R (Red), the G channel of the color component G (Green), and the B channel of the color component B (Blue), which are the wavelengths of three bands. The estimation of the spectral information image is performed, for example, using a machine learning model. The RGB values ​​(gradation of each color component) of each pixel in an RGB image are expressed by the spectral reflectance of each pixel, the spectral sensitivity of the imaging device that captured the image (hereinafter referred to as the imaging device spectral sensitivity), and the spectral distribution of the light source in the environment in which the image was captured (hereinafter referred to as the light source spectral distribution).

[0014] Figure 1 is a conceptual diagram showing an example of a machine learning model (CNN (Convolutional Neural Network), neural network, deep learning, etc.) that outputs a spectral image as spectral reflectance information, or a spectral information image, from an RGB image. Figure 1(a) shows an equation that demonstrates how the spectral radiance can be calculated by multiplying the spectral distribution of the light source, the spectral reflectance of the target object, and the spectral sensitivity of the imaging device by each wavelength (λ), and then integrating the result over the wavelength (λ) from 400 nm to 700 nm. Furthermore, spectral radiance can be calculated from the spectral distribution of the light source and the spectral reflectance of the target object. As can be seen from this formula, a machine learning model can be generated to estimate the spectral reflectance of a subject by using an RGB image, the spectral distribution of the light source, and the spectral sensitivity of the imaging device as input data sets.

[0015] Figure 1(b) is a conceptual diagram showing an example of a machine learning model that outputs a spectral image as spectral reflectance information, or a spectral information image, from an RGB image. The machine learning model 300 takes, for example, an RGB image 301, an imaging device spectral sensitivity 302, and a light source spectral distribution 303 as input data sets and outputs spectral images 305 for each of a predetermined wavelength (for example, wavelengths in 10nm increments in the wavelength range from 400nm to 700nm, 31 bands) of spectral reflectance. Here, the graph for the imaging device spectral sensitivity 302 shows wavelength on the horizontal axis and intensity on the vertical axis. Similarly, the graph for the light source spectral distribution 303 shows wavelength on the horizontal axis and intensity on the vertical axis.

[0016] <First Embodiment> Hereinafter, a first embodiment of the present invention will be described with reference to the drawings. Figure 2 is a block diagram showing an example configuration of a spectral information image estimation device according to the first embodiment. The spectral information image estimation device 10 shown in Figure 2 can be implemented using, for example, a PC (Personal Computer), a server device, or a quantum computer. In Figure 2, the spectral information image estimation device 10 comprises a data input unit 11, a data synthesis unit 12, a machine learning model generation unit 13, a spectral reflectance estimation unit 14, a spectral image output unit 15, a training image data storage unit 16, an image capture image data storage unit 17, a machine learning model storage unit 18, and a spectral information image data storage unit 19. Furthermore, the spectral reflectance estimation unit 14 comprises a first spectral reflectance estimation unit 141 and a second spectral reflectance estimation unit 142, respectively.

[0017] The data input unit 11 receives training image data supplied from an external device and writes the training image data to the training image data storage unit 16 for storage. Figure 3 shows an example of the structure of a training image data table that contains a dataset of training image data used to train a machine learning model. The training image data table includes, for each record, fields for training image identification information, RGB image index, light source spectral distribution index, imaging device spectral sensitivity index, and spectral reflectance information index.

[0018] The training image identification information is identification information that individually identifies each of the training images (RGB images in which the spectral reflectance of the subject is known). The RGB image index is an address that indicates a storage area in the training image data storage unit 16 where the image data of the captured RGB image is stored. The data for an RGB image consists of the gradation levels (e.g., 256 gradations) of each of the three color components R, G, and B (3 bands, i.e., 3 channels consisting of the R channel, G channel, and B channel) for each pixel of the RGB image (3 data points per pixel).

[0019] The light source spectral distribution index is an address that indicates a storage area in the training image data storage unit 16 where data on the light source spectral distribution, which shows the spectral distribution of the light source in the environment in which the captured image (RGB image) was taken, is stored. The light source spectral distribution shows, for example, the distribution of light intensity for each wavelength in 10 nm increments (31 bands) in the wavelength range from 400 nm to 700 nm (31 data points).

[0020] The imaging device spectral sensitivity index is an address that indicates a storage area in the training image data storage unit 16 where data indicating the spectral sensitivity of the imaging device that captured the RGB image (the captured image) is stored. The imaging device spectral sensitivity, for example, shows the distribution of light intensity for each of the color components R, G, and B at wavelengths in 10 nm increments (31 bands) in the wavelength range from 400 nm to 700 nm (93 data points (=3 × 31)).

[0021] The spectral reflectance information index is an address that indicates a storage area in the training image data storage unit 16 where spectral reflectance information (known spectral reflectance information), which is information about the spectral reflectance of the subject in the captured RGB image, is stored. Spectral reflectance information, for example, shows the distribution of light intensity for each wavelength (31 bands) in 10nm increments within the wavelength range of 400nm to 700nm for each pixel of an RGB image (31 data points per pixel).

[0022] Returning to Figure 1, the data synthesis unit 12 generates a dataset of input data to be input to the machine learning model. In other words, the data synthesis unit 12 adjusts and synthesizes the RGB image, light source spectral distribution, and imaging device spectral sensitivity data when training a machine learning model using training image data, and when estimating the image data of the spectral information image using the trained machine learning model.

[0023] As mentioned above, the input data set consists of 3 RGB image data points, 31 light source spectral distribution data points, and 31 × 3 = 93 imaging device spectral sensitivity data points, each used for a single pixel. Therefore, the number of data points in the dataset is (3 + 31 + 93) × W × H, where W is the number of pixels in the horizontal direction (rows) and H is the number of pixels in the vertical direction (columns) of the RGB image. Here, the data synthesis unit 12 performs synthesis processing on each data in the dataset, such as joining (i.e., concatenation), summation, and multiplication.

[0024] The data synthesis unit 12, in synthesizing the training data, arranges, for example, (3+31+93)×W×H data points into W×H pixels. Figure 4 is a conceptual diagram illustrating the synthesis of training data by the data synthesis unit 12. Figure 4(a) shows the arrangement of each of the W × H pixels, where the gradation of color component R is GR, the gradation of color component G is GG, and the gradation of color component B is GB for each pixel. Pixels P1 through Pn (n=W×H) are arranged sequentially in series, and the color components R, G, and B of each pixel are also arranged in series.

[0025] Then, as part of the combining process in the combining, the data synthesis unit 12 links and combines the data of the imaging device spectral sensitivity and the light source spectral distribution, as shown in Figure 4(b). Figure 4(b) illustrates the synthesis process by combining training data in the data synthesis unit 12. Here, the data synthesis unit 12, for example, inserts the data QR of each of the 31 bands of the color component R of the imaging device's spectral sensitivity in series between the gradation (pixel value) GR of the color component R and the gradation GG of the color component G in the array of each pixel.

[0026] Furthermore, the data synthesis unit 12 inserts the data QG of each of the 31 bands of the color component G of the imaging device's spectral sensitivity in series between the gradation GG of the color component G and the gradation GB of the color component B. The data synthesis unit 12 then inserts the data QB for each of the 31 bands of the spectral sensitivity of the imaging device's color component B in series after the gradation GB of the color component B.

[0027] Furthermore, the data synthesis unit 12 inserts the data S of the 31 bands of the light source spectral distribution in series after the data QB of the 31 bands of the color component B of the imaging device spectral sensitivity. Here, the series arrangement of the color components R, G, and B of the imaging device's spectral sensitivity and the light source's spectral distribution becomes the first composite value, and each of these first composite values, when inserted into the pixel arrangement shown in Figure 4(a), becomes the second composite value shown in Figure 4(b).

[0028] The data synthesis unit 12 performs the synthesis and combination process as described above, and outputs the result to the first spectral reflectance estimation unit 141. Furthermore, when performing the synthesis process, the data synthesis unit 12 reads the pixel values ​​of the RGB image, the spectral sensitivity of the imaging device, and the spectral distribution of the light source from the training image data storage unit 16 when generating a machine learning model, and reads them from the imaging image data storage unit 17 when estimating the spectral information image, and normalizes the read data.

[0029] Furthermore, the data synthesis unit 12 performs a summing process in synthesis, for example, by adding the data for each wavelength in the 31 bands of the light source spectral distribution to each data for the wavelength in the 31 bands of the imaging device spectral sensitivity (taking the sum of the data for the corresponding wavelengths). In other words, the data synthesis unit 12 adds the data of the 31 wavelengths of the light source spectral distribution to the data of the 31 wavelengths of the color component R of the imaging device spectral sensitivity, and uses each of the resulting data as the combined value of the respective wavelengths.

[0030] In this way, the data synthesis unit 12 performs a process of combining the data of the 31 bands of the light source spectral distribution by summing the data of each of the 31 bands of the color component R, color component G, and color component B of the imaging device spectral sensitivity (generation of the first synthesized value). Then, the data synthesis unit 12 synthesizes the summed data by inserting the corresponding color component of the imaging device's spectral sensitivity between the color components R, G, and B of the corresponding pixels, as shown in Figure 4(b), and concatenating them (generation of the second synthesized value).

[0031] Furthermore, the data synthesis unit 12, as part of the multiplication process in synthesis, multiplies each of the data for each wavelength in the 31 bands of the imaging device spectral sensitivity by the data for the same wavelength in the 31 bands of the light source spectral distribution (it takes the product of the data for the corresponding wavelengths). In other words, the data synthesis unit 12 multiplies the data of wavelengths 400 for 31 bands of the color component R of the imaging device's spectral sensitivity by the data of wavelengths 400 for 31 bands of the light source's spectral distribution, and uses each of the resulting data as the combined value of the respective wavelengths.

[0032] In this way, the data synthesis unit 12 performs a process of synthesizing the data of the 31 bands of the light source spectral distribution by multiplying the data of each of the 31 bands of the color component R, color component G, and color component B of the imaging device spectral sensitivity (generation of the first synthesized value). Then, the data synthesis unit 12 synthesizes the multiplied data by inserting the corresponding color component of the imaging device's spectral sensitivity between the color components R, G, and B of the corresponding pixels, as shown in Figure 4(b), and concatenating them (generation of the second synthesized value).

[0033] Furthermore, the data synthesis unit 12 may be configured to generate a second composite value by adding or multiplying each of the first composite values ​​generated by combining, summing, or multiplying the spectral sensitivity of the imaging device and the spectral distribution of the light source to the respective color components R, G, and B of the RGB image. In this case, the data synthesis unit 12 may obtain a second composite value by combining the first composite value obtained by the combining process with the pixel data of the RGB image by combining, summing, or multiplying the two. Alternatively, the data synthesis unit 12 may obtain a second composite value by combining the first composite value obtained by the summing process with the pixel data of the RGB image by combining, summing, or multiplying the two. Alternatively, the data synthesis unit 12 may obtain a second composite value by combining, summing, or multiplying the first composite value obtained by the multiplication process with the pixel data of the RGB image.

[0034] Returning to Figure 1, the machine learning model generation unit 13 outputs the training data, which is the second composite value, to the first spectral reflectance estimation unit 141. The first spectral reflectance estimation unit 141 inputs the above training data into a neural network, which is a machine learning model, and performs spectral information image estimation processing. Here, the machine learning model estimates the spectral reflectance corresponding to each pixel of the RGB image using training data generated from the RGB image, the spectral sensitivity of the imaging device, and the spectral distribution of the light source, and outputs a spectral information image of 31 bands.

[0035] Then, the machine learning model generation unit 13 reads out known spectral reflectance information of the subject in the RGB image from the training image data table in the training image data storage unit 16. Figure 5 shows an example of the configuration of an image data table containing a dataset of image data used to estimate spectral information images in a machine learning model. The image data table includes, for each record, columns for image identification information, RGB image index, light source spectral distribution index, and imaging device spectral sensitivity index.

[0036] Image identification information is identification information that identifies each captured image individually. The RGB image index is an address that indicates a storage area in the training image data storage unit 16 where the image data of the captured RGB image is stored. The data for an RGB image consists of the gradation levels (e.g., 256 gradations) for each pixel of the color component R, color component G, and color component B (3 bands) in the RGB image (3 data points per pixel).

[0037] The light source spectral distribution index is an address that indicates a storage area in the training image data storage unit 16 where data on the light source spectral distribution, which shows the spectral distribution of the light source in the environment in which the captured image (RGB image) was taken, is stored. The light source spectral distribution shows, for example, the distribution of light intensity for each wavelength in 10 nm increments (31 bands) in the wavelength range from 400 nm to 700 nm (31 data points).

[0038] The imaging device spectral sensitivity index is an address that indicates a storage area in the training image data storage unit 16 where data indicating the spectral sensitivity of the imaging device that captured the RGB image (the captured image) is stored. The imaging device spectral sensitivity, for example, shows the distribution of light intensity for each of the color components R, G, and B at wavelengths in 10 nm increments (31 bands) in the wavelength range from 400 nm to 700 nm (93 data points (=3 × 31)).

[0039] Returning to Figure 2, the machine learning model generation unit 13 calculates the difference between the pixel-by-pixel data of the 31-band spectral information image estimated by the machine learning model and the known spectral reflectance information data as the spectral error. Furthermore, the machine learning model generation unit 13 calculates the gradation of color component R, color component G, and color component B from the spectral reflectance information estimated by the machine learning model, the spectral sensitivity of the imaging device, and the spectral distribution of the light source, and uses these as estimated RGB values.

[0040] The machine learning model generation unit 13 calculates the difference between the estimated RGB values ​​(gradation of color component R, color component G, and color component B) obtained from the spectral reflectance (intensity for each wavelength) of each pixel in the spectral information image and the RGB values ​​of each pixel in the RGB image used to estimate the spectral reflectance, and uses this difference as the RGB value error. Then, the machine learning model generation unit 13 substitutes the spectral error and RGB value error into the loss function of equation (1) below to obtain the loss (loss value) L.

[0041]

number

[0042] In the loss function of equation (1) above, the loss L spectral and loss L RGB Each of these is a loss (loss value) obtained by the MSE (Mean Squared Error) loss shown in equation (2) below, or the MRAE (Mean Relative Absolute Error) loss shown in equation (3). In equation (1), τ1 is a predetermined constant. In each of the following equations (2) and (3), x est (i) The estimated value is (estimated spectral reflectance, estimated RGB value obtained from the estimated spectral reflectance) and xgt (i) is a true value (known spectral reflectance or RGB value). Also, i is the pixel number, and in the case of RGB values, for each of the color components R, color component G, and color component B, the loss L R , the loss L G , the loss L B is obtained, and these are added to obtain the loss L RGB (= L R + L G + L B ) is obtained.

[0043]

Number

[0044]

Number

[0045] Also, since the spectral reflectance of an object generally does not change steeply between adjacent bands, the loss may be obtained by the following equation (4) in which a smoothing term is added to equation (1). The loss L smooth in equation (4) is a value obtained by normalizing the difference in intensity values between adjacent bands of the estimated spectral reflectance and adding all the normalized differences between bands. Also, τ2 in equation (4) is a predetermined constant.

[0046]

Number

[0047] Then, the machine learning model generation unit 13 determines whether or not the loss L is less than or equal to a loss threshold (first threshold) that is a preset threshold value. The machine learning model generation unit 13 adjusts the coefficients of the function in the neural network, which is the machine learning model (i.e., the weight coefficients in the input or output of the function in each layer (especially the hidden layer) that makes up the neural network), and then inputs the training data back into the machine learning model to estimate the spectral information image.

[0048] Then, the machine learning model generation unit 13 adjusts the coefficients of the function in the neural network, which is the machine learning model, and trains the machine learning model until the loss L between the spectral information image data estimated by the machine learning model and the known spectral reflectance information data falls below the loss threshold. Here, the machine learning model generation unit 13 trains the model to satisfy predetermined conditions, such as when the loss L obtained by equation (1) or equation (4) is minimized (minimum value), or falls below a predetermined threshold, or when the number of iterations of the learning process for the corrected machine learning model reaches a predetermined number. Furthermore, the machine learning model generation unit 13 writes and stores the machine learning model in the machine learning model storage unit 18 as a trained machine learning model, after the loss L has been set to a loss threshold.

[0049] When the data synthesis unit 12 estimates an unknown spectral information image from an RGB image, which is image data captured by any imaging device and in any environment, it reads the spectral sensitivity of the imaging device, the spectral distribution of the light source in the captured environment, and the data of the captured image from the image data storage unit 17, generates a second composite value, and outputs it as input data to the first spectral reflectance estimation unit 141. The first spectral reflectance estimation unit 141 then receives input data supplied from the data synthesis unit 12 and reads a trained machine learning model from the machine learning model storage unit 18.

[0050] The first spectral reflectance estimation unit 141 receives input data supplied from the data synthesis unit 12 to the loaded, trained machine learning model. Here, the first spectral reflectance estimation unit 141 outputs the spectral information image estimated by the trained machine learning model using the input data as the primary spectral information image to the second spectral reflectance estimation unit 142.

[0051] The second spectral reflectance estimation unit 142 includes a neural network correction machine learning model, similar to the machine learning model of the first spectral reflectance estimation unit 141. The corrected machine learning model receives a primary spectral information image supplied from the first spectral reflectance estimation unit 141, estimates a corrected spectral information image, and outputs it. Then, the second spectral reflectance estimation unit 142 uses the captured image spectral sensitivity and light source spectral distribution corresponding to the primary spectral information image to generate a reconstructed RGB image from the corrected spectral information image.

[0052] Furthermore, the second spectral reflectance estimation unit 142 calculates the difference in the gradation levels of each color component R, color component G, and color component B at each pixel of the RGB image input to the first spectral reflectance estimation unit 141 and the corrected spectral information image as an error. The second spectral reflectance estimation unit 142 then uses a function L, which is a loss function consisting of equation (2) or (3) above. RGB Substitute the error into the following equation (5) to find the loss (loss value) L. In equation (5), τ1 is a predetermined constant.

[0053]

number

[0054] The spectral image output unit 15 writes the spectral information image data generated by the second spectral reflectance estimation unit 142 to the spectral information image data storage unit 19 for storage. The second spectral reflectance estimation unit 142 is trained to satisfy predetermined conditions, such as when the loss L obtained by equation (5) is minimized (minimum value), or falls below a predetermined threshold, or when the number of iterations of the learning process for the corrected machine learning model reaches a predetermined number. Here, the second spectral reflectance estimation unit 142 outputs the corrected spectral information image as the final spectral information image if the loss L satisfies predetermined conditions.

[0055] Furthermore, the second spectral reflectance estimation unit 142, similar to equation (4) above, uses the difference in intensity values ​​at the frequencies of adjacent bands of the corrected spectral information as an error, and calculates a loss function L consisting of equation (2) or (3) above. spectral The value obtained from equation (6) below, which includes the term, is defined as the loss L, and if the predetermined conditions are met, the corrected spectral information image may be output as the final spectral information image.

[0056]

number

[0057] Figure 6 shows an example of the configuration of the spectral information image data table in the spectral information image data storage unit 19. The spectral information image data table includes, for each record, columns for captured image identification information, RGB image index, and spectral information image index.

[0058] The captured image identification information is identification information that identifies each captured image individually, and is the same information as the captured image identification information of the RGB image used for estimation. The RGB image index is an address that indicates a storage area in the training image data storage unit 16 where the image data of the captured RGB image is stored. The data for an RGB image consists of the gradation levels (e.g., 256 gradations) for each pixel of the RGB image's color component R, color component G, and color component B (3 bands, i.e., each channel of R, G, and B) (3 data points per pixel).

[0059] The light source information image index is an address that indicates a storage area in the spectral information image data storage unit 19 where image data of the spectral information image corresponding to the captured RGB image is stored. The spectral information image uses, for example, the light intensity of each wavelength (31 bands) in 10 nm increments within the wavelength range of 400 nm to 700 nm as the grayscale of the pixels in the image. Therefore, the spectral information image consists of 31 image data points, with one image data point for each wavelength of the band.

[0060] Figure 7 is a flowchart showing an example of the operation of the machine learning model generation unit 13 according to this embodiment, which performs training on training data for a machine learning model. In the following description, the training data is a dataset consisting of an RGB image, the spectral sensitivity of the imaging device and the spectral distribution of the light source, and the known spectral reflectance of the subject in the RGB image. In the dataset, the input data for the machine learning model consists of an RGB image, the spectral sensitivity of the imaging device, and the spectral distribution of the light source, while the estimated data of the machine learning model is the spectral information image. In the following explanation, the data input unit 11 pre-writes the training image data set used for training the machine learning model into the training image data storage unit 16's training image data table.

[0061] Step S101: The data synthesis unit 12 reads out the data for the training image, the spectral sensitivity of the imaging device, and the spectral distribution of the light source from the training image data table in the training image data storage unit 16. The data synthesis unit 12 then generates a first composite value by arranging the color component R, color component G, and color component B of the imaging device's spectral sensitivity and the data of the light source's spectral distribution in series.

[0062] The data synthesis unit 12 synthesizes the generated first composite value and the pixel values ​​of the pixels in the training image as an array, for example, as shown in Figure 4(b), to generate a second composite value in which all the data is arranged in series, and outputs it to the machine learning model generation unit 13. Here, the data synthesis unit 12 generates multiple second composite values ​​for each of the multiple training images, based on the multiple training images stored in the training image data table of the training image data storage unit 16, and the imaging device spectral sensitivity and light source spectral distribution corresponding to the training images. As a result, the machine learning model generation unit 13 receives each of the training data, which are supplied from the data synthesis unit 12 as second composite values.

[0063] Step S102: The machine learning model generation unit 13 inputs the training data, which is supplied from the data synthesis unit 12 as a second composite value, into the machine learning model to be trained in parallel. The machine learning model estimates and outputs a spectral information image as a primary spectral information image (with a predetermined number of bands, e.g., 31 bands) in response to the training data. Here, the machine learning model generation unit 13 uses multiple second composite values ​​generated from multiple training images stored in the training image data table of the training image data storage unit 16, and the corresponding imaging device spectral sensitivity and light source spectral distribution, as training data to estimate each primary spectral information image.

[0064] Step S103: The machine learning model generation unit 13 calculates the spectral difference between the estimated primary spectral information image and the known spectral reflectance of the RGB image for each RGB image. Furthermore, the machine learning model generation unit 13 generates a reconstructed RGB image from the primary spectral information image, the spectral sensitivity of the imaging device, and the spectral distribution of the light source. The machine learning model generation unit 13 then calculates the RGB error as the difference between the reconstructed RGB image and the original RGB image, specifically the color component R, color component G, and color component G.

[0065] Step S104: Next, the machine learning model generation unit 13 substitutes the spectral difference into the loss function of equation (2) or (3), and the loss L spectral We seek. The machine learning model generation unit 13 calculates the loss L for all training images by taking the spectral difference between the primary spectral information image for each band of the training image and the known spectral reflectance. spectral We seek.

[0066] Furthermore, the machine learning model generation unit 13 substitutes the RGB error into the loss function of equation (2) or (3), and the loss L RGB We seek. The machine learning model generation unit 13 calculates the loss L from the RGB difference between the reconstructed RGB image and the RGB image from which the primary spectral information image was obtained for all training images. RGB We seek.

[0067] The machine learning model generation unit 13 generates a loss L spectral and loss L RGB The loss L is calculated from equation (1) or equation (4) using each of the above. Here, when the machine learning model generation unit 13 calculates the loss L using equation (4), it calculates the loss L from the first-order spectral information image. smooth We find the loss L as shown in equation (4). spectral , loss L RGB and loss L smooth The loss L is calculated by adding each of these together.

[0068] Step S105: The machine learning model generation unit 13 determines whether the number of iterations required to calculate the loss L exceeds a predetermined number. At this point, if the number of iterations required to calculate the loss L exceeds a predetermined number, the machine learning model generation unit 13 proceeds to step S106. On the other hand, if the number of iterations required to calculate the loss L is less than or equal to a predetermined number, the machine learning model generation unit 13 proceeds to step S107.

[0069] Step S106: The machine learning model generation unit 13 selects the machine learning model with the smallest loss L from among the machine learning models obtained by repeating the process a predetermined number of times, and writes this machine learning model to the machine learning model storage unit 18 for storage, marking it as trained. Then, the machine learning model generation unit 13 terminates the machine learning model training process, that is, the machine learning model generation process.

[0070] Step S107: The machine learning model generation unit 13 adjusts the coefficients of the neural network function in the machine learning model. Then, the machine learning model generation unit 13 proceeds with the processing to step S102 and performs the estimation of the primary spectral information image again.

[0071] Figure 8 is a flowchart showing an example of the operation of the process by which the spectral reflectance estimation unit 14 according to this embodiment estimates a spectral information image from an RGB image using a trained machine learning model. In the following explanation, the machine learning model memory unit 18 stores a trained machine learning model.

[0072] Step S201: The data input unit 11 reads the captured image data from an external device, which includes the spectral sensitivity of the imaging device that captured the RGB image and the spectral distribution of the light source in the environment in which the image was captured. The data input unit 11 then writes and stores the data of the read RGB image and the corresponding spectral sensitivity of the imaging device and spectral distribution of the light source into the image image data storage unit 17's image image table (see Figure 5).

[0073] Step S202: The first spectral reflectance estimation unit 141 reads out a trained machine learning model from the machine learning model storage unit 18. Furthermore, the first spectral reflectance estimation unit 141 outputs the target image identification information of the spectral information image to the data synthesis unit 12 and requests the synthesis of the input data.

[0074] Step S203: The data synthesis unit 12 reads out the RGB image, imaging device spectral sensitivity, and light source spectral distribution data from the imaging image data table in the imaging image data storage unit 17. The data synthesis unit 12 then generates a first composite value by arranging the color component R, color component G, and color component B of the imaging device's spectral sensitivity and the data of the light source's spectral distribution in series. The first spectral reflectance estimation unit 141 receives each of the training data, which are supplied from the data synthesis unit 12 as second composite values.

[0075] Step S204: The first spectral reflectance estimation unit 141 inputs the input data, which is a second composite value supplied from the data synthesis unit 12, in parallel to the trained machine learning model. The machine learning model estimates and outputs a spectral information image as a primary spectral information image (with a predetermined number of bands, e.g., 31 bands) in response to the training data. The first spectral reflectance estimation unit 141 then outputs the estimated primary spectral information image to the second spectral reflectance estimation unit 142.

[0076] Step S205: The second spectral reflectance estimation unit 142 receives a primary spectral information image supplied from the first spectral reflectance estimation unit 141. Furthermore, the second spectral reflectance estimation unit 142 reads the imaging device spectral sensitivity and light source spectral distribution corresponding to the RGB image from which the primary spectral information image was estimated, from the imaging image data storage unit 17's imaging image data table.

[0077] The second spectral reflectance estimation unit 142 then inputs the primary spectral information image to the corrected machine learning model. This allows the correction machine learning model to estimate a corrected spectral information image in response to the input primary spectral information image.

[0078] Step S206: The second spectral reflectance estimation unit 142 generates a reconstructed RGB image using the corrected spectral information image, the imaging device spectral sensitivity, and the light source spectral distribution. Then, the second spectral reflectance estimation unit 142 calculates the generation error (RGB error) from the reconstructed RGB image and the RGB image used for estimation. At this time, the second spectral reflectance estimation unit 142 loses L due to generation error according to equation (2) or (3). RGB We find the value of (5) and substitute it into equation (5) to find the loss L.

[0079] Step S207: The second spectral reflectance estimation unit 142 determines whether the number of iterations for calculating the loss L exceeds a predetermined number. At this point, if the number of iterations for calculating the loss L exceeds a predetermined number, the second spectral reflectance estimation unit 142 proceeds to step S208. On the other hand, if the number of iterations for calculating the loss L is less than or equal to a predetermined number, the second spectral reflectance estimation unit 142 proceeds to step S209.

[0080] Step S208: The second spectral reflectance estimation unit 142 selects the corrected spectral information image with the minimum loss L from the corrected spectral information images generated by the corrected machine learning model after repeating the process a predetermined number of times as the spectral image information. The second spectral reflectance estimation unit 142 then writes the spectral information image data to the spectral information image data table of the spectral information image data storage unit 19 for storage.

[0081] Step S209: The second spectral reflectance estimation unit 142 adjusts the coefficients of the neural network function in the corrected machine learning model. Then, the second spectral reflectance estimation unit 142 proceeds to step S205, and again uses the adjusted coefficient correction machine learning model to estimate the corrected spectral information image using the primary spectral information image.

[0082] As described above, in this embodiment, an RGB image, the spectral sensitivity of the imaging device that captured the RGB image, and the spectral distribution of the light source in the environment in which the RGB image was captured are each used as input data sets. A machine learning model estimates the spectral information of the subject in the RGB image as a primary spectral information image. A correction machine learning model estimates a corrected spectral information image from the primary spectral information image. The loss L is calculated from the RGB error between the reconstructed RGB image generated from the corrected spectral information image and the RGB image estimated from the primary spectral information image. The coefficients of the correction machine learning model are adjusted until the predetermined conditions described above are met. The corrected spectral information image is then estimated again by the adjusted correction machine learning model to obtain the spectral information image. In other words, the spectral information image is estimated by performing a two-stage estimation using both the machine learning model and the correction machine learning model. With this configuration, according to this embodiment, the spectral information image is a corrected spectral information image that generates a reconstructed RGB image that is closer to each of the color components R, G, and B of the RGB image used to estimate the primary spectral information image, using the primary spectral information image estimated by the machine learning model. Therefore, even if the estimation accuracy of the primary spectral information image by the first spectral reflectance estimation unit 141 is lower than when using a multiband camera due to insufficient training data or the characteristics of the machine learning model, the second spectral reflectance estimation unit 142 corrects the primary spectral information image, making it possible to easily estimate the spectral reflectance of the imaged object from the RGB image (images of each of the three bands, i.e., the R channel, G channel, and B channel) with an accuracy comparable to that of an image captured by a multiband camera.

[0083] Furthermore, in the above-described embodiment, the data synthesis unit 12 synthesizes the spectral sensitivity of the imaging device that captured the RGB image and the spectral distribution of the light source of the imaging environment to obtain a first composite value. However, the data synthesis unit 12 may be configured to use only either the imaging device spectral sensitivity or the light source spectral distribution as the first composite value, and synthesize it with the RGB image data to obtain a second composite value. In this configuration, when the machine learning model generation unit 13 trains the machine learning model, and when the data synthesis unit 12 synthesizes the data with the training image to create a second composite value, only either the imaging device spectral sensitivity or the light source spectral distribution is used as the first composite value.

[0084] Furthermore, in the embodiments described above, it was explained that each of the machine learning model and the corrected machine learning model estimates the spectral information image. However, each of the machine learning model and the corrected machine learning model may be configured to estimate the coefficients of the basis functions that constitute the spectral reflectance curve obtained by principal component analysis or the like, rather than the spectral information image, as the distribution information to be estimated.

[0085] Figure 9 is a conceptual diagram illustrating the estimation of the coefficients of the basis functions that constitute the spectral reflectance curve, using both a machine learning model and a corrected machine learning model. In other words, the coefficient rn can be estimated from the coefficient r1 multiplied by each of the basis functions bn, which is represented by synthesizing the spectral reflectance curve (a curve showing the correspondence between wavelength and intensity) extracted by principal component analysis. In this configuration, when the machine learning model generation unit 13 trains the machine learning model, it does not output spectral reflectance as the training data dataset, but rather uses known coefficients r1 to rn as the output of the estimation.

[0086] Then, the machine learning model generation unit 13 multiplies each of the estimated coefficients r1 to rn by each of the basis functions b1 to bn, synthesizes the basis functions b1 to bn to obtain a spectral reflectance curve, calculates the loss L based on the spectral error with the known spectral reflectance and the RGB error in the RGB image, and performs learning until the predetermined conditions described above are met. Furthermore, the first spectral reflectance estimation unit 141 outputs the coefficient rn from coefficient r1 as primary spectral information to the second spectral reflectance estimation unit 142 using a trained machine learning model.

[0087] As a result, the corrected machine learning model takes the coefficients r1 and rn as input data for the primary spectral information and outputs the coefficients r1 and rn as estimated values ​​for the corrected spectral information. The second spectral reflectance estimation unit 142 synthesizes basis functions b1 and bn obtained by multiplying each of the estimated coefficients r1 and rn, respectively, to obtain a spectral reflectance curve, obtains a reconstructed RGB image using the spectral reflectance, calculates the loss L based on the RGB error between the reconstructed RGB image and the RGB image used for estimation, and outputs corrected spectral information that satisfies the predetermined conditions described above as spectral information.

[0088] Furthermore, in this embodiment, if the spectral sensitivity of the imaging device that captured the RGB image is known, but the spectral distribution of the light source in the captured environment is unknown, the configuration may be such that a machine learning model for estimating the spectral distribution of the light source is generated as a machine learning model for estimating the spectral distribution of the light source. In this configuration, when capturing the RGB image to be used for estimating spectral information, a color chart with known spectral reflectances (color component R, color component G, color component B) is captured along with the subject. Similarly, an RGB image is also captured along with the subject for the training image. Furthermore, when the machine learning model generation unit 13 trains a machine learning model for estimating the spectral distribution of a light source, it uses the estimated output data as the spectral distribution of the light source and the input data as the image region of the color chart in the training image, the data of the known color components R, G, and B of the color chart, and the spectral sensitivity of the imaging device to train the machine learning model for estimating the spectral distribution of a light source.

[0089] Then, when the first spectral reflectance estimation unit 141 estimates the spectral distribution of the light source from the RGB image, it inputs the image region of the color chart, the data of the known color components R, G, and B of the color chart, and the spectral sensitivity of the imaging device into a trained light source spectral distribution machine learning model by performing a synthesis process, and then estimates the spectral distribution of the light source. As a result, the first spectral reflectance estimation unit 141 synthesizes the data from the RGB image, the imaging device spectral sensitivity, and the estimated light source spectral distribution, inputs these data into a trained machine learning model, and estimates a spectral information image. As an example, we have described a method for estimating the spectral distribution of a light source using a machine learning model, but any method other than the machine learning model described above may be used to estimate the spectral distribution of a light source.

[0090] Furthermore, in this embodiment, if the light source spectral distribution of the environment in which the RGB image was captured is known, but the spectral sensitivity of the imaging device used for the capture is unknown, the configuration may be such that a machine learning model for estimating the spectral sensitivity of the imaging device is generated as a machine learning model for estimating the spectral sensitivity of the imaging device. In this configuration, when capturing the RGB image to be used for estimating spectral information, a color chart with known spectral reflectances (color component R, color component G, color component B) is captured along with the subject. Similarly, an RGB image is also captured along with the subject for the training image. Furthermore, when the machine learning model generation unit 13 trains a machine learning model for estimating the spectral sensitivity of the imaging device, it uses the estimated output data as the spectral sensitivity of the imaging device and the input data as the image region of the color chart in the training image, the data of the known color components R, G, and B of the color chart, and the spectral distribution of the light source to train the machine learning model for estimating the spectral sensitivity of the imaging device.

[0091] Then, when the first spectral reflectance estimation unit 141 estimates the imaging device spectral sensitivity from the RGB image, it synthesizes the image region of the color chart, the data of known color components R, G, and B of the color chart, and the spectral distribution of the light source into a trained machine learning model for estimating the imaging device spectral sensitivity, and inputs this data to estimate the imaging device spectral sensitivity. As a result, the first spectral reflectance estimation unit 141 synthesizes the data from the RGB image, the light source spectral distribution, and the estimated imaging device spectral sensitivity, inputs these data into a trained machine learning model, and estimates a spectral information image. As an example, we have described a method for estimating the spectral sensitivity of an imaging device using a machine learning model, but any method other than the machine learning model described above may be used to estimate the spectral sensitivity of an imaging device.

[0092] <Second Embodiment> A second embodiment of the present invention will be described below with reference to the drawings. The spectral information image estimation device according to the second embodiment has the same configuration as the first embodiment. On the other hand, the machine learning model in the second embodiment has a different configuration from that of the first embodiment. In the following description, only the operations of the spectral information image estimation apparatus according to the second embodiment that differ from those of the first embodiment will be explained, and redundant explanations will be omitted as appropriate.

[0093] The machine learning model in the first embodiment was configured to input the spectral sensitivity of the imaging device and the spectral distribution of the light source, respectively, as input data to the input layer. However, the machine learning model in the second embodiment is configured to input the spectral sensitivity of the imaging device and the spectral distribution of the light source, respectively, as input data to the intermediate layer. Figure 10 is a conceptual diagram illustrating the input of imaging device spectral sensitivity and light source spectral distribution data to the intermediate layer of the machine learning model in this embodiment.

[0094] In Figure 10, the data synthesis unit 12 refers to the image data storage unit 17 and reads out the RGB image, imaging device spectral sensitivity, and light source spectral distribution data from the image data table. The data synthesis unit 12 then outputs the RGB image to the first spectral reflectance estimation unit 141 in its original data format. As a result, the first spectral reflectance estimation unit 141 reads the RGB image data from the image data table and inputs the RGB image data to the input layer 320 of the machine learning model 300.

[0095] Furthermore, the data synthesis unit 12 synthesizes the respective data for the imaging device spectral sensitivity and the light source spectral distribution by concatenating, summing, and multiplying them to generate a first synthesized value. The data synthesis unit 12 generates a third composite value by combining the generated first composite value with the output data of one of the multiple intermediate layers 350 in the machine learning model 300 through processes such as concatenation, summing, and multiplication, and outputs the third composite value to the first spectral reflectance estimation unit 141.

[0096] Then, the first spectral reflectance estimation unit 141 causes the next intermediate layer 350, which outputs the output data synthesized with the first composite value, to input the third composite value supplied from the data synthesis unit 12. In this embodiment, the intermediate layer that outputs the output data to be combined with the first composite value may be configured as an intermediate layer of any one stage, or it may be configured as an intermediate layer of multiple stages.

[0097] Furthermore, when combining the first composite value with the output of the hidden layer, the configuration may involve first converting the dimensionality of the first composite value (the dimensionality of the data), and then combining the output of the hidden layer with the first composite value after the dimensionality conversion. In this configuration, the data synthesis unit 12 has a dimensionality transformation neural network that increases or decreases the dimensionality of the synthesized first composite value. Then, the data synthesis unit 12 uses the dimensionality transformation neural network described above to transform the number of dimensions of the synthesized first composite value, and then outputs the first composite value to the first spectral reflectance estimation unit 141.

[0098] As described above, in this embodiment, similar to the first embodiment, an RGB image, the spectral sensitivity of the imaging device that captured the RGB image, and the spectral distribution of the light source in the environment in which the RGB image was captured are each used as input data sets. A machine learning model estimates the spectral information of the subject in the RGB image as a primary spectral information image. A correction machine learning model estimates a corrected spectral information image from the primary spectral information image. The loss L is calculated from the RGB error between the reconstructed RGB image generated from the corrected spectral information image and the RGB image estimated from the primary spectral information image. The coefficients of the correction machine learning model are adjusted until the predetermined conditions described above are met. The corrected spectral information image is then estimated again by the adjusted correction machine learning model to obtain the spectral information image. In other words, a two-stage estimation is performed by the machine learning model and the correction machine learning model, respectively, to estimate the spectral information image. With this configuration, according to this embodiment, the spectral information image is a corrected spectral information image that generates a reconstructed RGB image that is closer to each of the color components R, G, and B of the RGB image used to estimate the primary spectral information image, using the primary spectral information image estimated by the machine learning model. Therefore, even if the estimation accuracy of the primary spectral information image by the first spectral reflectance estimation unit 141 is lower than when using a multiband camera due to insufficient training data or the characteristics of the machine learning model, the second spectral reflectance estimation unit 142 corrects the primary spectral information image, making it possible to easily estimate the spectral reflectance of the imaged object from the RGB image (images of each of the three bands, i.e., the R channel, G channel, and B channel) with an accuracy comparable to that of an image captured by a multiband camera.

[0099] Furthermore, according to this embodiment, since the first composite value is combined with the output of the intermediate layer to generate a third composite value which becomes the input to the next intermediate layer, the output of the intermediate layer from which the features of the RGB image have been extracted, and the spectral sensitivity of the imaging device and the spectral distribution of the light source are combined to form the third composite value. As a result, compared to the first embodiment, the primary spectral information image is more accurately approximated to the spectral reflectance of the RGB subject, the time required for correction processing of the correction machine learning model is reduced, and the spectral information image can be generated more quickly.

[0100] Furthermore, in the above-described embodiment, the data synthesis unit 12 synthesizes the spectral sensitivity of the imaging device that captured the RGB image and the spectral distribution of the light source of the imaging environment to obtain a first composite value. However, the data synthesis unit 12 may be configured to use only either the spectral sensitivity of the imaging device or the spectral distribution of the light source as the first composite value, synthesize it with the output data of the intermediate layer of a machine learning model that has an RGB image as input to obtain a third composite value, and use it as the input for the next intermediate layer. In this configuration, when the machine learning model generation unit 13 trains the machine learning model, and when the data synthesis unit 12 synthesizes the data with the training image to create a third composite value, only either the imaging device spectral sensitivity or the light source spectral distribution is used as the first composite value.

[0101] <Third Embodiment> A third embodiment of the present invention will be described below with reference to the drawings. The spectral information image estimation device according to the third embodiment has the same configuration as the first embodiment. On the other hand, the machine learning model in the third embodiment has a different configuration from that of the first embodiment. In the following description, only the operations of the spectral information image estimation apparatus according to the third embodiment that differ from those of the first embodiment will be explained, and redundant explanations will be omitted as appropriate.

[0102] Figure 11 is a conceptual diagram illustrating an example of the configuration of the machine learning model used by the first spectral reflectance estimation unit 141 in this embodiment. In Figure 11, the machine learning model of this embodiment includes sub-machine learning model 101, sub-machine learning model 102, and sub-machine learning model 103, as well as an integrated machine learning model 110. Here, the sub-machine learning model 101 receives data for the color component R in the pixels of the RGB image (imaging device color component R data) and the color component R in the spectral sensitivity of the imaging device (imaging device color component R sensitivity data), and outputs an estimated output regarding the color component R to the data synthesis unit 12.

[0103] Similarly, the sub-machine learning model 102 receives data for the color component G in the pixels of the captured image (imaging device color component G data) and the color component G in the spectral sensitivity of the imaging device (imaging device color component G sensitivity data), and outputs an estimated output regarding the color component G to the data synthesis unit 12. The sub-machine learning model 102 receives data for color component B in the pixels of the captured image (imaging device color component B data) and color component B in the spectral sensitivity of the imaging device (imaging device color component B sensitivity data), and outputs an estimated output regarding color component B to the data synthesis unit 12.

[0104] The data synthesis unit 12 synthesizes the color component R of the pixels in the captured image (imaging device color component R data) and the color component R of the spectral sensitivity of the imaging device (imaging device color component R sensitivity data) by concatenation, sum, and multiplication to generate an input composite value SR, and outputs the input composite value SR to the first spectral reflectance estimation unit 141. The first spectral reflectance estimation unit 141 then inputs the input composite value SR supplied from the data synthesis unit 12 to the input layer of the submachine learning model 101.

[0105] Furthermore, the data synthesis unit 12 synthesizes the color component G in the pixels of the captured image (imaging device color component G data) and the color component G in the spectral sensitivity of the imaging device (imaging device color component G sensitivity data) by concatenation, sum, and multiplication to generate an input synthesis value SG, and outputs the input synthesis value SG to the first spectral reflectance estimation unit 141. The first spectral reflectance estimation unit 141 then inputs the input composite value SG supplied from the data synthesis unit 12 to the input layer of the submachine learning model 102.

[0106] Similarly, the data synthesis unit 12 synthesizes the respective data of color component B in the pixels of the captured image (imaging device color component B data) and color component B in the spectral sensitivity of the imaging device (imaging device color component B sensitivity data) by concatenation, summation, and multiplication to generate an input synthesis value SB, and outputs the input synthesis value SB to the first spectral reflectance estimation unit 141. The first spectral reflectance estimation unit 141 then inputs the composite input value SB supplied from the data synthesis unit 12 to the input layer of the submachine learning model 103.

[0107] Furthermore, the data synthesis unit 12 synthesizes the outputs of sub-machine learning models 101, 102, and 103 with the light source spectral distribution by concatenation, sum, and multiplication to generate a fourth composite value, and outputs this fourth composite value to the first spectral reflectance estimation unit 141. In this case, the data synthesis unit 12 may be configured to use a dimensionality transformation neural network to transform the dimensionality of the fourth composite value, and then output this transformed fourth composite value to the first spectral reflectance estimation unit 141. The first spectral reflectance estimation unit 141 then inputs the fourth composite value supplied from the integrated machine learning model 110 to the integrated machine learning model 110. As a result, the integrated machine learning model 110 estimates and outputs a spectral information image corresponding to the fourth composite value.

[0108] Furthermore, in this embodiment, when training the machine learning model, the machine learning model generation unit 13 adjusts the coefficients of the neural network function between the sub-machine learning models 101, 102, and 103 and the integrated machine learning model 110. Here, sub-machine learning model 101, sub-machine learning model 102, and sub-machine learning model 103 each have the same neural network configuration. Therefore, the machine learning model generation unit 13 adjusts the coefficients of the same neural network function to the same numerical values ​​in each of the sub-machine learning models 101, 102, and 103.

[0109] Furthermore, in this embodiment, instead of inputting the data of the captured image color components (R, G, B) (component data) and the sensitivity data of the imaging device color component sensitivity (R, G, B) into the respective input layers of each of the sub-machine learning models 101, 102, and 103, the first spectral reflectance estimation unit 141 may input only the captured image color components into the input layer, and the data synthesis unit 12 may synthesize the sensitivity data of the imaging device color component sensitivity with the output data of the intermediate layer, and the first spectral reflectance estimation unit 141 may input the data, which is a synthesis of the output data and the sensitivity data, into the next intermediate layer. In this case, the data synthesis unit 12 may be configured to use a dimensionality transformation neural network to transform the number of dimensions of the synthesized data (synthesized data) obtained by combining the output data and the sensitivity data, and then output this synthesized data with the transformed number of dimensions to the first spectral reflectance estimation unit 141.

[0110] Furthermore, in this embodiment, instead of inputting the captured image color component (R, G, B) data and the imaging device color component sensitivity (R, G, B) data into their respective input layers in each of the sub-machine learning models 101, 102, and 103, the first spectral reflectance estimation unit 141 may input the captured image color component into its input layer, and the data synthesis unit 12 may synthesize the component data of the captured image color component and the sensitivity data of the imaging device color component sensitivity with respect to the output data of the intermediate layer, and the first spectral reflectance estimation unit 141 may input the synthesized data of the output data, component data and sensitivity data into the next intermediate layer. In this case, the data synthesis unit 12 may be configured to use a dimensionality transformation neural network to transform the number of dimensions of the synthesized data (composite data) obtained by combining the output data, component data, and sensitivity data, and then output this synthesized data with transformed dimensions to the first spectral reflectance estimation unit 141.

[0111] Furthermore, in this embodiment, instead of inputting the captured image color component (R, G, B) data and the imaging device color component sensitivity (R, G, B) data into their respective input layers in each of the sub-machine learning models 101, 102, and 103, the first spectral reflectance estimation unit 141 may input only the captured image color component into its input layer, and the data synthesis unit 12 may synthesize the imaging device color component sensitivity data and the light source spectral distribution data with respect to the output data of the intermediate layer, and the first spectral reflectance estimation unit 141 may input the data (fifth composite value as synthesized data) obtained by synthesizing the output data, sensitivity data and distribution data into the next intermediate layer. In this case, the data synthesis unit 12 may be configured to use a dimensionality transformation neural network to transform the number of dimensions of the synthesized data (composite data) obtained by combining the output data, sensitivity data, and distribution data, and then output this synthesized data with transformed dimensions to the first spectral reflectance estimation unit 141.

[0112] Furthermore, in this embodiment, instead of inputting the captured image color component (R,G,B) data and the imaging device color component sensitivity (R,G,B) data into the input layer of each of the sub-machine learning models 101, 102, and 103, the captured image color component (R,G,B) data, the imaging device color component sensitivity (R,G,B) data, and the light source spectral distribution data may be combined (concatenated, summed, and multiplied) and input. At this time, the data synthesis unit 12 synthesizes the outputs of sub-machine learning models 101, 102, and 103 and outputs them to the first spectral reflectance estimation unit 141. The first spectral reflectance estimation unit 141 then uses the data supplied from the data synthesis unit 12 as input to an integrated machine learning model.

[0113] Furthermore, in this embodiment, a first composite value obtained by combining (concatenating, summing, and multiplying) the captured image color component (R, G, B) data and the imaging device color component sensitivity (R, G, B) data may be input to the input layer of each of the sub-machine learning models 101, 102, and 103, and the light source spectral distribution data may be input to the intermediate layers of each of the sub-machine learning models 101, 102, and 103. At this time, the first spectral reflectance estimation unit 141 inputs the light source spectral distribution data supplied from the data synthesis unit 12 to the respective intermediate layers of sub-machine learning models 101, 102, and 103.

[0114] As described above, in this embodiment, similar to the first embodiment, an RGB image, the spectral sensitivity of the imaging device that captured the RGB image, and the spectral distribution of the light source in the environment in which the RGB image was captured are each used as input data sets. A machine learning model estimates the spectral information of the subject in the RGB image as a primary spectral information image. A correction machine learning model estimates a corrected spectral information image from the primary spectral information image. The loss L is calculated from the RGB error between the reconstructed RGB image generated from the corrected spectral information image and the RGB image estimated from the primary spectral information image. The coefficients of the correction machine learning model are adjusted until the predetermined conditions described above are met. The corrected spectral information image is then estimated again by the adjusted correction machine learning model to obtain the spectral information image. In other words, a two-stage estimation is performed by the machine learning model and the correction machine learning model, respectively, to estimate the spectral information image. With this configuration, according to this embodiment, the spectral information image is a corrected spectral information image that generates a reconstructed RGB image that is closer to each of the color components R, G, and B of the RGB image used to estimate the primary spectral information image, using the primary spectral information image estimated by the machine learning model. Therefore, even if the estimation accuracy of the primary spectral information image by the first spectral reflectance estimation unit 141 is lower than when using a multiband camera due to insufficient training data or the characteristics of the machine learning model, the second spectral reflectance estimation unit 142 corrects the primary spectral information image, making it possible to easily estimate the spectral reflectance of the imaged object from the RGB image (images of each of the three bands, i.e., the R channel, G channel, and B channel) with an accuracy comparable to that of an image captured by a multiband camera.

[0115] Furthermore, according to this embodiment, a sub-machine learning model is provided for each color component of the RGB image, and it is conceivable that the feature extraction process for each of the color components R, G, and B is the same. When training the machine learning model, by adjusting the same functions of the neural networks in each of the sub-machine learning models for each color component to the same values ​​at the same time, the training of the machine learning model can be converged more quickly compared to adjusting the coefficients of each function individually without using sub-machine learning models.

[0116] Furthermore, in the above-described embodiment, the data synthesis unit 12 synthesizes the spectral sensitivity of the imaging device that captured the RGB image and the spectral distribution of the light source of the imaging environment to obtain a first composite value. However, the data synthesis unit 12 may use only either the spectral sensitivity of the imaging device or the spectral distribution of the light source as the first composite value, and synthesize it with the output data of the intermediate layer of a machine learning model for each color component, which has data for each color component of the RGB image as input, to obtain a fifth composite value, which is then used as the input for the next intermediate layer of the machine learning model for each color component. In this configuration, when the machine learning model generation unit 13 trains the machine learning model, and when the data synthesis unit 12 synthesizes it with the training image to create a fifth composite value, only either the imaging device spectral sensitivity or the light source spectral distribution is used as the first composite value.

[0117] <Fourth Embodiment> A fourth embodiment of the present invention will be described below with reference to the drawings. Figure 12 is a block diagram showing an example configuration of a spectral information image estimation device according to the fourth embodiment. The spectral information image estimation device 10A shown in Figure 12 can be implemented using, for example, a PC, a server device, or a quantum computer. In Figure 12, the spectral information image estimation device 10A comprises a data input unit 11, a spectral reflectance estimation unit 14A, a spectral image output unit 15, an image capture data storage unit 17, and a spectral information image data storage unit 19. Furthermore, the spectral reflectance estimation unit 14A comprises a first spectral reflectance estimation unit 141A and a second spectral reflectance estimation unit 142, respectively. Components similar to those in the first embodiment are denoted by the same reference numerals. In the following, explanations that overlap with the embodiments described above will be omitted as appropriate.

[0118] Unlike the first spectral reflectance estimation unit 141 in the first embodiment, the first spectral reflectance estimation unit 141A does not perform spectral reflectance estimation using a machine learning model. The first spectral reflectance estimation unit 141A estimates the spectral reflectance by synthesizing spectral reflectance basis vectors (basis functions), as shown in Figure 9, for example. Here, the spectral reflectance basis vectors are derived in advance from a set of spectral reflectance data, which is a collection of multiple spectral reflectance data, using principal component analysis, as spectral reflectance basis vectors to represent that set of spectral reflectance data. In this principal component analysis, spectral reflectance can be represented using a combination of spectral reflectance basis vectors with fewer dimensions than the actual dimensionality of spectral reflectance. In other words, by using spectral reflectance basis vectors, the dimensionality of the spectral reflectance being estimated is compressed, reducing the computational load.

[0119] In other words, the first spectral reflectance estimation unit 141A is expressed as a weighted sum of multiple spectral reflectance basis vectors B=[b1,b2,…]T, as shown in equation (7) below. To perform this weighting, the spectral reflectance spectrum s is estimated by determining the weight coefficients r=[r1,r2,…]T for each spectral reflectance basis vector. Here, the spectral reflectance basis vectors are pre-stored in the image data storage unit 17 as a set, for example. Alternatively, the image data storage unit 17 may store the spectral reflectance basis vectors in an array arranged in descending order of principal component scores in principal component analysis, and the system may be configured to read and use as needed from a predetermined order of spectral reflectance basis vectors.

[0120] As previously explained, Figure 9 shows that spectral reflectance is expressed by combining spectral reflectance basis vectors. As shown in Figure 9, the spectral reflectance spectrum s is obtained by multiplying each of the spectral reflectance basis vectors b1 to bn, obtained by principal component analysis using the spectral reflectance of each of the given objects, by a weighting coefficient, and then synthesizing the results of the multiplication. In other words, the first spectral reflectance estimation unit 141A reads out spectral reflectance basis vectors bn from the imaging image data storage unit 17 using the required number of spectral reflectance basis vectors b1. Then, the first spectral reflectance estimation unit 141A estimates the weight coefficients r1 and rn of each spectral reflectance basis vector bn from the extracted spectral reflectance basis vector b1, so as to obtain the spectral reflectance spectrum s which are the pixel values ​​of each pixel in the reference multiview image, and calculates the spectral reflectance spectrum s from the estimation results (described later).

[0121]

number

[0122] Returning to Figure 12, the first spectral reflectance estimation unit 141A estimates each weighting coefficient r by solving equation (8) below for the spectral reflectance spectrum expressed by equation (7) above. Then, the first spectral reflectance estimation unit 141A estimates the spectral reflectance spectrum s using equation (7) based on the estimated weight coefficient r. Here, the spectral reflectance spectrum si is represented as a 31-dimensional vector obtained by sampling the wavelength range from 400 nm to 700 nm in 10 nm increments.

[0123]

number

[0124] In equation (8) above, pm and k are the pixel values ​​of the k-th channel of the m-th captured image. That is, in pm and k, m is the number indicating the captured image for each filter. Also, k indicates the channel of the color component in a part of the multiview image. For example, if a part is represented by the pixel values ​​of color components R, G, and B, then channel (band) 1 is color component R (R channel), channel 2 is color component G (G channel), and channel 3 is color component B (B channel). Furthermore, the pixels whose pixel values ​​are read from the captured image of each filter are extracted from the image data of each captured image of each filter, as the pixels corresponding to each of the captured images.

[0125] Furthermore, in equation (8), the matrices Cm and k represent the spectral sensitivity of the k-th channel (wavelength band of the color component) of the captured image of the m-th filter. The elements cm and k(λ) in Cm and k represent the spectral sensitivity of wavelength λ in the k-th channel of the m-th captured image. Furthermore, the spectral distribution l of the light source is already written to the image data table of the image data storage unit 17 as measured spectral distribution data. The first spectral reflectance estimation unit 141A reads the spectral distribution of the light source in the environment in which the image of each filter was captured from the image data table of the image data storage unit 17.

[0126] Here, the first spectral reflectance estimation unit 141A may be configured to select and read a set of spectral reflectance basis vectors from the image data storage unit 17, corresponding to the number of spectral sensitivities of the imaging device in the captured image of each filter. Furthermore, when solving equation (8), the upper limit of the number of spectral reflectance basis vectors (principal components) used to represent the spectral reflectance data (hereinafter referred to as the number of spectral reflectance basis vectors) n is determined, for example, based on the number of spectral sensitivity groups from which the captured image is taken (such as the number of spectral sensitivity adjustments made by the color change filter).

[0127] When using multi-view images in q spectral sensitivity groups, the number of spectral reflectance basis vectors is determined by the following relationship: "n (number of spectral reflectance basis vectors) = q (number of types of spectral sensitivity, number of spectral sensitivity groups) × k (number of channels, hereafter referred to as the number of channels)". In other words, if the number of spectral reflectance basis vectors n representing spectral reflectance data is less than or equal to k times the number of channels of the number of spectral sensitivity groups q, then equation (8) can be solved. For example, if three color-shifting filters with different spectral sensitivities (=Nc) are used, and there are three channels (R, G, and B), the upper limit of the basis vector is 9 (=3×3). Furthermore, each image captured using each color-changing filter must be taken from the same viewpoint, as it is used in equation (8) as the m-th viewpoint image from the same viewpoint. However, any method for estimating the spectral reflectance of a subject from an RGB image is acceptable, rather than estimating the spectral reflectance using the multiple spectral sensitivities described above.

[0128] The first spectral reflectance estimation unit 141A outputs the spectral information image of the 31 bands of the obtained spectral reflectance as a primary spectral information image to the second spectral reflectance estimation unit 142. Then, as described in the first embodiment, the second spectral reflectance estimation unit 142 inputs the primary spectral information image supplied from the first spectral reflectance estimation unit 141A to the corrected machine learning model. The second spectral reflectance estimation unit 142 generates a reconstructed RGB image from the output corrected spectral information image and calculates the loss L by comparing it with the RGB image of any of the color change filters.

[0129] When generating a reconstructed RGB image from a corrected spectral information image, the second spectral reflectance estimation unit 142 generates the RGB image using the spectral sensitivity of the imaging device due to the color change filter when the RGB image to be compared was captured from the corrected spectral information image, and the spectral distribution of the light source in the captured environment. Then, the second spectral reflectance estimation unit 142 adjusts the coefficients of the neural network function of the corrected machine learning model until the loss L falls below the generation threshold. The second spectral reflectance estimation unit 142 outputs the corrected spectral information image at the time to the spectral image output unit 15 as the spectral information image when the loss L falls below the generation threshold. The spectral image output unit 15 writes the spectral information image supplied from the second spectral reflectance estimation unit 142 to the spectral information image data storage unit 19 for storage.

[0130] As described above, this embodiment estimates a primary spectral information image from an RGB image using the spectral sensitivity of the imaging device and the spectral distribution of the light source in a predetermined manner, estimates a corrected spectral information image from the primary spectral information image using a corrective machine learning model, calculates a loss L from the RGB error between the reconstructed RGB image generated from the corrected spectral information image and the RGB image estimated from the primary spectral information image, adjusts the coefficients of the corrective machine learning model until the predetermined conditions described above are met, and estimates the corrected spectral information image again using the adjusted corrective machine learning model. In other words, it performs a two-stage estimation process in which a primary spectral information image is estimated from an RGB image using a predetermined manner, and a corrected spectral information image is estimated from the primary spectral information image using a corrective machine learning model, thereby estimating the spectral information image. With this configuration, according to this embodiment, the spectral information image is a corrected spectral information image that generates a reconstructed RGB image that is closer to each of the color components R, G, and B of the RGB image used to estimate the primary spectral information image, using the primary spectral information image estimated by a predetermined estimation method. Therefore, even if the estimation accuracy of the primary spectral information image by the first spectral reflectance estimation unit 141A is lower than that when a multiband camera is used due to the characteristics of the predetermined estimation method, the second spectral reflectance estimation unit 142 corrects the primary spectral information image, making it possible to easily estimate the spectral reflectance of the imaged object from the RGB image (images of each of the three bands, i.e., the R channel, G channel, and B channel) with an accuracy comparable to that of an image captured by a multiband camera.

[0131] <Fifth Embodiment> A fifth embodiment of the present invention will be described below with reference to the drawings. Figure 13 is a block diagram showing an example configuration of a spectral information image estimation device according to the fifth embodiment. The spectral information image estimation device 10B shown in Figure 13 can be implemented using, for example, a PC, a server device, or a quantum computer. In Figure 13, the spectral information image estimation device 10B comprises a data input unit 11B, a data generation unit 20, a machine learning model generation unit 13B, a spectral reflectance estimation unit 14B, a spectral image output unit 15, a dataset data storage unit 21, a training image data storage unit 16B, an image capture image data storage unit 17, a machine learning model storage unit 18, and a spectral information image data storage unit 19. Furthermore, the spectral reflectance estimation unit 14B comprises a first spectral reflectance estimation unit 141B and a second spectral reflectance estimation unit 142B, respectively. Components similar to those in the first embodiment are denoted by the same reference numerals. In the following, explanations that overlap with the embodiments described above will be omitted as appropriate.

[0132] The data input unit 11B receives a dataset supplied from an external device and writes the dataset to the dataset data storage unit 21 for storage. The dataset includes, for example, a spectral information image dataset, a light source spectral distribution dataset, and an imaging device spectral sensitivity dataset. The spectral information image dataset is a dataset containing multiple spectral information images. The light source spectral distribution dataset is a dataset containing multiple light source spectral distributions, for example, multiple light source spectral distributions showing the spectral distribution for each illumination or the spectral distribution of sunlight. The imaging device spectral sensitivity dataset is a dataset containing multiple imaging device spectral sensitivities, for example, multiple imaging device spectral sensitivities showing the spectral sensitivity for each model of imaging device.

[0133] The data generation unit 20 generates training data to be used for training a machine learning model (first machine learning model). For example, the data generation unit 20 generates training data based on the spectral information image dataset, light source spectral distribution dataset, and imaging device spectral sensitivity dataset stored in the dataset data storage unit 21. Specifically, first, the data generation unit 20 randomly selects spectral information images, light source spectral distributions, and imaging device spectral sensitivity from each of the datasets stored in the dataset data storage unit 21. After selection, the data generation unit 20 generates an RGB image as a training image from the selected spectral information image, light source spectral distribution, and imaging device spectral sensitivity. After generation, the data generation unit 20 writes the generated RGB image and the imaging device spectral sensitivity, light source spectral distribution, and spectral reflectance of the spectral information image (i.e., the known spectral reflectance of the subject in the RGB image) used to generate the RGB image to the training image data storage unit 16B as a single training data set for storage.

[0134] The data generation unit 20 may also generate new spectral information images, light source spectral distributions, and imaging device spectral sensitivities for generating training images by applying an arbitrary magnification (scaling) to spectral information images, light source spectral distributions, and imaging device spectral sensitivities selected from each dataset stored in the dataset data storage unit 21.

[0135] Furthermore, the data generation unit 20 may generate a new light source spectral distribution or imaging device spectral sensitivity to be used as training data based on the basis vector of the training data. In this case, the data generation unit 20 multiplies the spectral components included in the light source spectral distribution dataset or imaging device spectral sensitivity dataset by a predetermined coefficient, and then adds or subtracts the multiplication result to generate a new light source spectral distribution or imaging device spectral sensitivity for generating training images. Furthermore, the light source spectral distribution dataset in the dataset data storage unit 21 has spectral components with a large proportion in the light source spectral distribution pre-written and stored. Similarly, the imaging device spectral sensitivity dataset in the dataset data storage unit 21 has spectral components with a large proportion in the imaging device spectral sensitivity pre-written and stored. The spectral components included in the light source spectral distribution dataset or the imaging device spectral sensitivity dataset are generated, for example, by principal component analysis.

[0136] The machine learning model generation unit 13B generates a machine learning model using training data, including training images, generated by the data generation unit 20. Similar to the machine learning model generation unit 13 in the first embodiment, the machine learning model generation unit 13B uses the training data to train the machine learning model, and writes the machine learning model whose loss L is below the loss threshold to the machine learning model storage unit 18 as a trained machine learning model for storage.

[0137] The machine learning model generation unit 13B may include values ​​other than those obtained from spectral error (spectral loss), RGB value error (RGB loss), and smoothing term (smoothing loss) in its losses. This can improve the learning accuracy of the machine learning model. Spectral loss is, for example, a value obtained from the error between the known spectral information (spectral reflectance) of the training image and the spectral information (spectral reflectance) of the spectral information image (first spectral information image) estimated by the machine learning model. RGB loss is, for example, the error between the RGB image (reconstructed RGB image) generated from the spectral information of the spectral information image estimated by the machine learning model, the imaging spectral sensitivity of the imaging device that captured the training image, and the light source spectral distribution of the environment in which the training image was captured, and the RGB image of the training image. Smoothing loss is, for example, the difference in spectral reflectance between adjacent wavelengths in the spectral information image estimated by the machine learning model.

[0138] For example, the machine learning model generation unit 13B may include a value obtained from the difference between the first integral value and the second integral value in its loss function. The first integral value is the integral of the product of the known spectral information of the training image and the light source spectral distribution created based on known arbitrary spectral information, spectrum by spectrum. The second integral value is the integral of the product of the spectral information of the spectral information image estimated by the machine learning model and the light source spectral distribution created based on known arbitrary spectral information, spectrum by spectrum. Furthermore, the machine learning model generation unit 13B may include in its losses a color difference (color difference loss) calculated based on the LAB color space values ​​calculated from the known spectral information of the training image and the LAB color space values ​​calculated from the spectral information of the spectral information image estimated by the machine learning model.

[0139] Furthermore, the machine learning model generation unit 13B uses losses including at least spectral loss in generating machine learning models. In the fifth embodiment, for example, the machine learning model generation unit 13B uses a weighted sum of spectral loss, RGB loss, and smoothing loss.

[0140] When the data generation unit 20 estimates an unknown spectral information image from an RGB image, which is captured image data taken with any imaging device and in any environment, it outputs the spectral sensitivity of the imaging device, the spectral distribution of the light source in the captured environment, and the captured image (RGB image) read from the image data storage unit 17 to the first spectral reflectance estimation unit 141B as input data. The first spectral reflectance estimation unit 141B then receives input data supplied from the data generation unit 20 and reads a trained machine learning model from the machine learning model storage unit 18.

[0141] The first spectral reflectance estimation unit 141B estimates a spectral information image showing the spectral reflectance of a subject based on the RGB image, which is the captured image of the subject; the spectral sensitivity of the imaging device that captured the image; and the spectral distribution of the light source in the environment in which the image was captured. Specifically, the first spectral reflectance estimation unit 141B inputs the RGB image supplied from the data generation unit 20, the spectral sensitivity of the imaging device, and the spectral distribution of the light source into a trained machine learning model read from the machine learning model storage unit 18, and estimates a spectral information image. Here, the first spectral reflectance estimation unit 141B outputs the spectral information image estimated by the trained machine learning model using the input data as the primary spectral information image to the second spectral reflectance estimation unit 142.

[0142] The second spectral reflectance estimation unit 142B includes a neural network correction machine learning model (second machine learning model), similar to the machine learning model of the first spectral reflectance estimation unit 141B. The second spectral reflectance estimation unit 142B estimates and outputs a corrected spectral information image (second spectral information image) by correcting the primary spectral information image based on the RGB image (input RGB image), imaging device spectral sensitivity, and light source spectral distribution input to the first spectral reflectance estimation unit 141B, and the primary spectral information image estimated by the first spectral reflectance estimation unit 141B. Specifically, the second spectral reflectance estimation unit 142B inputs the input RGB image, imaging device spectral sensitivity, light source spectral distribution, and primary spectral information image supplied from the first spectral reflectance estimation unit 141B into a correction machine learning model to estimate the corrected spectral information image. The correction machine learning model generates an RGB image based on the spectral information of the input primary spectral information image, the spectral sensitivity of the imaging device, and the spectral distribution of the light source. It corrects the primary spectral information image by repeatedly updating the parameters of the correction machine learning model to minimize the loss between the input RGB image and the generated RGB image (reconstructed RGB image). This loss is the value (RGB loss) obtained from the RGB value error between the input RGB image and the reconstructed RGB image. The parameters of the correction machine learning model are, for example, coefficients of a neural network function. Note that arbitrary values ​​can be set as the initial values ​​of the parameters. Furthermore, the corrective machine learning model may include the value obtained by considering the smoothing term (smoothing loss) in its loss function. This can improve the correction accuracy (estimation accuracy) of the corrective machine learning model. The smoothing loss is, for example, the difference in spectral reflectance between adjacent wavelengths in the corrected spectral information image estimated by the corrective machine learning model.

[0143] Furthermore, the second spectral reflectance estimation unit 142B uses a loss that includes at least RGB loss in the estimation of the corrected spectral information image (correction of the primary spectral information image). In the fifth embodiment, for example, the second spectral reflectance estimation unit 142B uses a loss obtained by weighting and summing the RGB loss and the smoothing loss.

[0144] Figure 14 is a flowchart showing an example of the operation of the machine learning model generation unit 13B using training data for a machine learning model according to the fifth embodiment. In the following description, the data input unit 11B pre-writes a dataset, including a spectral information image dataset, a light source spectral distribution dataset, and an imaging device spectral sensitivity dataset supplied from an external device, to the dataset data storage unit 21.

[0145] Step S301: The data generation unit 20 selects a dataset. Specifically, the data generation unit 20 randomly selects a spectral information image, a light source spectral distribution, and an imaging device spectral sensitivity dataset from each of the datasets stored in the dataset data storage unit 21.

[0146] Step S302: After selection, the data generation unit 20 generates training data. Specifically, the data generation unit 20 generates an RGB image as a training image from the selected spectral information image, light source spectral distribution, and imaging device spectral sensitivity. After generation, the data generation unit 20 writes the generated RGB image, the imaging device spectral sensitivity, light source spectral distribution, and spectral reflectance of the spectral information image (i.e., the known spectral reflectance of the subject in the RGB image) used to generate the RGB image, as a single training data to the training image data storage unit 16B for storage.

[0147] Steps S303 to S309: Steps S303 to S309 are the same as and overlap with steps S101 to S107 described in the first embodiment with reference to Figure 7, respectively, so their explanation will be omitted.

[0148] Figure 15 is a flowchart showing an example of the operation of the process of estimating a spectral information image from an RGB image using a trained machine learning model by the spectral reflectance estimation unit 14B according to the fifth embodiment. In the following explanation, the machine learning model memory unit 18 stores a trained machine learning model.

[0149] Step S401: The data input unit 11B performs data input for the captured image. Specifically, the data input unit 11B reads data from an external device that includes the input RGB image, the spectral sensitivity of the imaging device that captured the RGB image, and the spectral distribution of the light source in the environment in which the image was captured. The data input unit 11B then writes and stores the read RGB image and the data for the imaging device spectral sensitivity and light source spectral distribution corresponding to the RGB image into the imaging image data storage unit 17's imaging image table (see Figure 5).

[0150] Step S402: The first spectral reflectance estimation unit 141B reads out a trained machine learning model from the machine learning model storage unit 18.

[0151] Step S403: The first spectral reflectance estimation unit 141B reads out the RGB image, imaging device spectral sensitivity, and light source spectral distribution data from the imaging image data table in the imaging image data storage unit 17.

[0152] Step S404: The first spectral reflectance estimation unit 141B estimates a primary spectral information image. Specifically, the first spectral reflectance estimation unit 141B inputs the read input data into a trained machine learning model. As a result, the machine learning model estimates a spectral information image from the input data and outputs it as a primary spectral information image to the second spectral reflectance estimation unit 142B.

[0153] Step S405: The second spectral reflectance estimation unit 142B estimates a corrected spectral information image. Specifically, the second spectral reflectance estimation unit 142B inputs the primary spectral information image supplied from the first spectral reflectance estimation unit 141B into the corrected machine learning model. The second spectral reflectance estimation unit 142B also reads the imaging device spectral sensitivity and light source spectral distribution corresponding to the RGB image from which the primary spectral information image was estimated from the imaging image data storage unit 17 into the imaging image data table and inputs them into the corrected machine learning model. As a result, the corrected machine learning model estimates a corrected spectral information image by correcting the input primary spectral information image.

[0154] Step S406: The second spectral reflectance estimation unit 142B calculates the loss L. Specifically, the second spectral reflectance estimation unit 142B generates an RGB image (reconstructed RGB image) from the corrected spectral information image, the imaging device spectral sensitivity, and the light source spectral distribution. Then, the second spectral reflectance estimation unit 142B calculates the error (RGB error) from the reconstructed RGB image and the input RGB image used for estimation. At this time, the second spectral reflectance estimation unit 142B calculates the loss L due to error from equation (2) or equation (3). RGB We find the value of (5) and substitute it into equation (5) to find the loss L.

[0155] Step S407: The second spectral reflectance estimation unit 142B determines whether the number of iterations for calculating the loss L exceeds a predetermined number. At this point, if the number of iterations for calculating the loss L exceeds a predetermined number, the second spectral reflectance estimation unit 142B proceeds to step S408. On the other hand, if the number of iterations for calculating the loss L is less than or equal to a predetermined number, the second spectral reflectance estimation unit 142B proceeds to step S409.

[0156] Step S408: The second spectral reflectance estimation unit 142B selects the corrected spectral information image with the minimum loss L from the corrected spectral information images generated by the corrected machine learning model after repeating a predetermined number of times as the spectral image information. The second spectral reflectance estimation unit 142B then writes the spectral information image data to the spectral information image data table of the spectral information image data storage unit 19 for storage.

[0157] Step S409: The second spectral reflectance estimation unit 142B adjusts the coefficients (parameters) of the neural network function in the corrected machine learning model. Then, the second spectral reflectance estimation unit 142B proceeds to step S405, and again uses the adjusted coefficient correction machine learning model to estimate the corrected spectral information image, which is obtained by correcting the primary spectral information image.

[0158] As described above, in the fifth embodiment, the first spectral reflectance estimation unit 141B estimates the spectral information (spectral reflectance) of the subject in the RGB image as a primary spectral information image based on the RGB image of the subject, the spectral sensitivity of the imaging device that captured the RGB image, and the spectral distribution of the light source in the environment in which the RGB image was captured. The second spectral reflectance estimation unit 142B then estimates a corrected spectral information image by correcting the primary spectral information image based on the primary spectral information image, the input RGB image, the spectral sensitivity of the imaging device, and the spectral distribution of the light source. This configuration eliminates the need to generate a separate machine learning model for each imaging environment for each target imaging device, and also eliminates the need to estimate spectral reflectance before estimating spectral radiance. Furthermore, in this configuration, the fifth embodiment uses the estimated primary spectral information image to estimate a corrected spectral information image as the spectral information image, which generates a reconstructed RGB image that is closer to each of the color components R, G, and B of the RGB image used to estimate the primary spectral information image. Therefore, by correcting the primary spectral information image with the second spectral reflectance estimation unit 142B, the spectral information image estimation device 10B can easily and accurately estimate a spectral information image (spectral reflectance image, spectral image) of four or more bands of the imaged object from an RGB image (a 3-band image, i.e., images of each channel of the R channel, G channel, and B channel).

[0159] In the fifth embodiment described above, an example was given in which arbitrary values ​​are set as the initial values ​​of the parameters of the corrective machine learning model, but the invention is not limited to such an example. For example, values ​​calculated by pre-training may be set as the initial values ​​of the parameters of the corrective machine learning model. In this case, the initial values ​​of the parameters can be set with greater accuracy compared to setting arbitrary values, and the number of times the parameters of the corrective machine learning model are adjusted in the estimation of corrected spectral information images can be reduced.

[0160] Pre-training of the corrected machine learning model is performed by the machine learning model generation unit 13B or the second spectral reflectance estimation unit 142B. The training data used when pre-training the corrected machine learning model is, for example, the input data (input RGB image, imaging device spectral sensitivity, light source spectral distribution) input to the machine learning model during training, and the primary spectral information image output by the machine learning model estimation based on said input data. In the pre-training of the correction machine learning model, the parameters of the correction machine learning model are repeatedly updated to minimize the loss between the input RGB image and the generated RGB image (reconstructed RGB image), similar to the estimation of the correction spectral information image.

[0161] In the fifth embodiment described above, an example was explained in which the loss during estimation of the corrected spectral information image includes RGB loss or smoothing loss. However, the loss during pre-training of the corrected machine learning model may include losses other than RGB loss and smoothing loss. This can improve the pre-training accuracy of the corrected machine learning model. The RGB loss is the error between the RGB image (reconstructed RGB image) generated from the spectral information of the corrected spectral information image estimated by the corrected machine learning model, the imaging spectral sensitivity of the imaging device that captured the training image, and the spectral distribution of the light source in the environment in which the training image was captured, and the RGB image of the training image (input RGB image). The smoothing loss is the difference in spectral reflectance between adjacent wavelengths in the estimated corrected spectral information image.

[0162] For example, the loss may include a value (spectral loss) obtained from the error between the known spectral information of the training image and the spectral information of the corrected spectral information image estimated by the correction machine learning model. Furthermore, the value obtained from the difference between the first integral and the third integral may be included in the loss. The first integral is the integral of the product of each spectrum between the known spectral information of the training image and the light source spectral distribution created based on any known spectral information. The third integral is the integral of the product of each spectrum between the spectral information of the corrected spectral information image estimated by the corrected machine learning model and the light source spectral distribution created based on any known spectral information. Furthermore, the loss may include color difference (color difference loss) calculated based on the LAB color space values ​​calculated from the known spectral information of the training image and the LAB color space values ​​calculated from the spectral information of the corrected spectral information image estimated by the corrected machine learning model. Furthermore, in the pre-training of the corrected machine learning model, losses including at least spectral losses are used.

[0163] Figure 16 is a flowchart showing an example of the operation of the learning process using training data for a corrected machine learning model by the machine learning model generation unit 13B according to the fifth embodiment.

[0164] Step S501: The machine learning model generation unit 13B inputs the primary spectral information image to the corrected machine learning model to be trained. This primary spectral information image is, for example, the primary spectral information image estimated in step S304 of Figure 14. The machine learning model generation unit 13B also inputs the input RGB image used to estimate the primary spectral information image, the imaging device spectral sensitivity, and the light source spectral distribution to the corrected machine learning model to be trained.

[0165] Step S502: The machine learning model estimates and outputs a corrected spectral information image using the primary spectral information image, input RGB image, imaging device spectral sensitivity, and light source spectral distribution input to the machine learning model generation unit 13B as training data.

[0166] Step S503: The machine learning model generation unit 13B calculates the spectral difference between the estimated corrected spectral information image and the known spectral reflectance of the primary spectral information image for each primary spectral information image.

[0167] Step S504: Next, the machine learning model generation unit 13B calculates the loss L. Specifically, the machine learning model generation unit 13B substitutes the spectral difference into the loss function of equation (2) or (3) and calculates the loss L. spectral We calculate this and let it be the loss L.

[0168] Step S505: The machine learning model generation unit 13B determines whether the number of iterations required to calculate the loss L exceeds a predetermined number. At this point, if the number of iterations required to calculate the loss L exceeds a predetermined number, the machine learning model generation unit 13B proceeds to step S506. On the other hand, if the number of iterations required to calculate the loss L is less than or equal to a predetermined number, the machine learning model generation unit 13B proceeds to step S507.

[0169] Step S506: The machine learning model generation unit 13B selects the corrected machine learning model with the smallest loss L from among the corrected machine learning models obtained by repeating the process a predetermined number of times, and writes this corrected machine learning model to the machine learning model storage unit 18 for storage, marking it as trained. Then, the machine learning model generation unit 13B finishes the training process of the corrected machine learning model, that is, the pre-training process (generation process) of the corrected machine learning model.

[0170] Step S507: The machine learning model generation unit 13B adjusts the coefficients of the neural network function in the corrected machine learning model. Then, the machine learning model generation unit 13B proceeds with the processing to step S502 and performs estimation of the corrected spectral information image again.

[0171] Embodiments of the present invention have been described above. Each of the embodiments described above may be applied independently as an embodiment of the present invention, or all or part of an embodiment may be combined with other embodiments to be applied as an embodiment of the present invention. Furthermore, the configuration of each embodiment may be applied in place of the configuration described in other embodiments, or it may be applied in addition to the configuration described in other embodiments.

[0172] Furthermore, a program for realizing some or all of the functions of the spectral information image estimation device 10 in Figure 2, the spectral information image estimation device 10A in Figure 12, and the spectral information image estimation device 10B in Figure 13 in the present invention may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be loaded into a computer system and executed to perform the process of estimating the spectral information image of a subject from an RGB image (captured image). Herein, "computer system" includes hardware such as an OS (Operating System) and peripheral devices. Furthermore, "computer system" also includes a WWW (World Wide Web) system equipped with a homepage provisioning environment (or display environment). Furthermore, "computer-readable recording medium" refers to portable media such as flexible disks, magneto-optical disks, ROMs (Read Only Memory), CD-ROMs (Compact Disc - Read Only Memory), and storage devices such as hard disks built into a computer system. Furthermore, "computer-readable recording media" includes volatile memory (RAM (Random Access Memory)) within computer systems that act as servers or clients when programs are transmitted via networks such as the Internet or communication lines such as telephone lines, which retain programs for a certain period of time.

[0173] Furthermore, the above program may be transmitted from a computer system that stores the program in a memory device or the like to another computer system via a transmission medium or by transmission waves within the transmission medium. Here, the "transmission medium" for transmitting the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. Also, the above program may be for the purpose of realizing only a part of the functions described above. Furthermore, it may be a so-called differential file (differential program) that can realize the above functions in combination with a program already recorded in the computer system.

[0174] As described above, the embodiments of the present invention have been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and design changes and the like within the scope not departing from the gist of the present invention are also included.

[0175] In addition, the following inventions are also included in the present invention.

[0176] (Invention 1) A first spectral reflectance estimation unit that estimates a first spectral information image indicating the spectral reflectance of the subject based on an RGB image that is an imaging image of the subject, an imaging device spectral sensitivity that is the spectral sensitivity of the imaging device that captured the imaging image, and a light source spectral distribution that is the spectral distribution of the light source in the environment where the imaging image was captured. A second spectral reflectance estimation unit that estimates a second spectral information image obtained by correcting the first spectral information image based on the first spectral information image estimated by the first spectral reflectance estimation unit, the RGB image, the imaging device spectral sensitivity, and the light source spectral distribution. A spectral information image estimation device, characterized by comprising:

[0177] (Invention 2) The first spectral reflectance estimation unit inputs the RGB image, the imaging device spectral sensitivity, and the light source spectral distribution into a first machine learning model to estimate the first spectral information image. The spectral information image estimation device according to Invention 1, characterized by the above.

[0178] (Invention 3) The second spectral reflectance estimation unit inputs the first spectral information image, the RGB image, the imaging device spectral sensitivity, and the light source spectral distribution into a second machine learning model to estimate the second spectral information image. The second machine learning model generates an RGB image based on the spectral information of the input first spectral information image, the imaging device spectral sensitivity, and the light source spectral distribution, and repeatedly updates the parameters of the second machine learning model so that the loss between the input RGB image and the generated RGB image is reduced, thereby correcting the first spectral information image. A spectral information image estimation device according to Invention 1 or Invention 2, characterized by the above.

[0179] (Invention 4) The second spectral reflectance estimation unit sets the values ​​calculated by pre-training as the initial values ​​of the parameters of the second machine learning model. A spectral information image estimation device according to any one of Inventions 1 to 3, characterized by the above.

[0180] (Invention 5) The first machine learning model is a trained model that uses a training image, which is an RGB image with known spectral reflectance, the imaging spectral sensitivity of the imaging device that captured the training image, and the light source spectral distribution of the environment in which the training image was captured, as training data, and uses a value obtained from the error between the known spectral information of the training image and the spectral information of the first spectral information image estimated by the first machine learning model as its loss. A spectral information image estimation device according to any one of Inventions 1 to 4, characterized by the above.

[0181] (Invention 6) The loss includes the error between the RGB image generated from the spectral information of the first spectral information image estimated by the first machine learning model, the imaging spectral sensitivity of the imaging device that captured the training image, and the spectral distribution of the light source in the environment in which the training image was captured, and the RGB image of the training image. A spectral information image estimation device according to any one of Inventions 1 to 5, characterized by the above.

[0182] (Invention 7) The loss includes the difference in spectral reflectance between adjacent wavelengths in the estimated first spectral information image. A spectral information image estimation device according to any one of Inventions 1 to 5, characterized by the above.

[0183] (Invention 8) The loss includes a value obtained from the difference between the first integral value and the second integral value. The first integral value is the integral of the product of the known spectral information of the training image and the spectral distribution of the light source created based on known arbitrary spectral information, for each spectrum. The second integral value is the integral of the product of the spectral information of the first spectral information image estimated by the first machine learning model and the spectral distribution of the light source created based on known arbitrary spectral information, for each spectrum. A spectral information image estimation device according to any one of Inventions 1 to 5, characterized by the above.

[0184] (Invention 9) The loss includes a value in the LAB color space calculated from the known spectral information of the training image and a color difference calculated based on the value in the LAB color space calculated from the spectral information of the first spectral information image estimated by the first machine learning model. A spectral information image estimation device according to any one of Inventions 1 to 5, characterized by the above.

[0185] (Invention 10) The loss includes the difference in spectral reflectance between adjacent wavelengths in the estimated second spectral information image. A spectral information image estimation device according to any one of Inventions 1 to 5, characterized by the above.

[0186] (Invention 11) The loss includes a value obtained from the error between the known spectral information of the training image and the spectral information of the second spectral information image estimated by the second machine learning model. A spectral information image estimation device according to any one of Inventions 1 to 5, characterized by the above.

[0187] (Invention 12) The loss includes the error between the RGB image generated from the spectral information of the second spectral information image estimated by the second machine learning model, the imaging spectral sensitivity of the imaging device that captured the training image, and the spectral distribution of the light source in the environment in which the training image was captured, and the RGB image of the training image. A spectral information image estimation device according to Invention 11, characterized by the above.

[0188] (Invention 13) The loss includes the difference in spectral reflectance between adjacent wavelengths in the estimated second spectral information image. The spectral information image estimation apparatus according to Invention 11, characterized in that.

[0189] (Invention 14) The loss includes a value obtained from the difference between the first integral value and the third integral value. The first integral value is the integral value of the product for each spectrum of the known spectral information of the teacher image and the light source spectral distribution created based on any known spectral information. The third integral value is the integral value of the product for each spectrum of the spectral information of the second spectral information image estimated by the second machine learning model and the light source spectral distribution created based on any known spectral information. The spectral information image estimation apparatus according to Invention 11, characterized in that.

[0190] (Invention 15) The loss includes the color difference calculated based on the value in the LAB color space calculated from the known spectral information of the teacher image and the value in the LAB color space calculated from the spectral information of the second spectral information image estimated by the second machine learning model. The spectral information image estimation apparatus according to Invention 11, characterized in that.

[0191] (Invention 16) A data generation unit that selects a spectral information image, a light source spectral distribution, and an imaging device spectral sensitivity from each of a spectral information image dataset including a plurality of spectral information images, a light source spectral distribution dataset including a plurality of light source spectral distributions, and an imaging device spectral sensitivity dataset including a plurality of imaging device spectral sensitivities, and generates an RGB image as a teacher image from the selected spectral information image, light source spectral distribution, and imaging device spectral sensitivity, A machine learning model generation unit that generates the machine learning model using the generated teacher image. The spectral information image estimation apparatus according to any one of Inventions 1 to 5, further comprising.

[0192] (Invention 17) The data generation unit generates new spectral information images, light source spectral distributions, and imaging device spectral sensitivities for generating the training image by applying an arbitrary magnification to the selected spectral information image, light source spectral distribution, and imaging device spectral sensitivity. A spectral information image estimation device according to invention 16, characterized in that

[0193] (Invention 18) The aforementioned light source spectral distribution dataset includes spectral components that have a large proportion in the aforementioned light source spectral distribution. The imaging device spectral sensitivity dataset includes spectral components that have a large proportion in the spectral sensitivity of the imaging device. The data generation unit multiplies the spectral components by a predetermined coefficient and adds or subtracts the multiplication result to generate a new light source spectral distribution or imaging device spectral sensitivity for generating the training image. A spectral information image estimation device according to invention 16, characterized in that [Explanation of Symbols]

[0194] 10, 10A, 10B... Spectroscopic Information Image Estimation Device 11,11B...Data input section 12...Data Synthesis Unit 13,13B...Machine learning model generation unit 14,14A,14B...Spectral reflectance estimation section 15...Spectroscopic image output unit 16,16B...Teacher image data storage unit 17... Image data storage unit 18…Machine learning model memory unit 19…Spectroscopic information image data storage unit 20...Data generation unit 141,141A,141B...First spectral reflectance estimation section 142,142B...Second spectral reflectance estimation section

Claims

1. A first spectral reflectance estimation unit estimates a first spectral information image showing the spectral reflectance of the subject based on an RGB image which is an image of the subject, the spectral sensitivity of the imaging device which captured the image, and the spectral distribution of the light source in the environment in which the image was captured. A second spectral reflectance estimation unit estimates a second spectral information image obtained by correcting the first spectral information image based on the first spectral information image estimated by the first spectral reflectance estimation unit, the RGB image, the spectral sensitivity of the imaging device, and the spectral distribution of the light source. Equipped with, The first spectral reflectance estimation unit inputs the RGB image, the spectral sensitivity of the imaging device, and the spectral distribution of the light source into a first machine learning model and estimates the first spectral information image. The first machine learning model is a trained model that uses a training image, which is an RGB image with a known spectral reflectance, the imaging spectral sensitivity of the imaging device that captured the training image, and the spectral distribution of the light source in the environment in which the training image was captured, as training data, and uses a value obtained from the error between the known spectral information of the training image and the spectral information of the first spectral information image estimated by the first machine learning model as the loss. A spectroscopic information image estimation device characterized by the following features.

2. A first spectral reflectance estimation unit that estimates a first spectral information image showing the spectral reflectance of the subject based on an RGB image which is an image of the subject, the spectral sensitivity of the imaging device which is the spectral sensitivity of the imaging device which captured the image, and the spectral distribution of the light source in the environment in which the image was captured. A second spectral reflectance estimation unit estimates a second spectral information image obtained by correcting the first spectral information image based on the first spectral information image estimated by the first spectral reflectance estimation unit, the RGB image, the spectral sensitivity of the imaging device, and the spectral distribution of the light source. Equipped with, The second spectral reflectance estimation unit inputs the first spectral information image, the RGB image, the spectral sensitivity of the imaging device, and the spectral distribution of the light source into the second machine learning model and estimates the second spectral information image. The second machine learning model generates an RGB image based on the spectral information of the input first spectral information image, the spectral sensitivity of the imaging device, and the spectral distribution of the light source, and corrects the first spectral information image by repeatedly updating the parameters of the second machine learning model so that the loss between the input RGB image and the generated RGB image is reduced. A spectroscopic information image estimation device characterized by the following features.

3. The second spectral reflectance estimation unit sets the values ​​calculated by pre-training as initial values ​​for the parameters of the second machine learning model. The spectral information image estimation apparatus according to feature 2.

4. The loss includes the error between the RGB image generated from the spectral information of the first spectral information image estimated by the first machine learning model, the imaging spectral sensitivity of the imaging device that captured the training image, and the spectral distribution of the light source in the environment in which the training image was captured, and the RGB image of the training image. The spectral information image estimation apparatus according to feature 1.

5. The loss includes the difference in spectral reflectance between adjacent wavelengths in the estimated first spectral information image. The spectral information image estimation apparatus according to feature 1.

6. The loss includes a value obtained from the difference between the first integral value and the second integral value. The first integral value is the integral of the product of the known spectral information of the training image and the spectral distribution of the light source created based on known arbitrary spectral information, for each spectrum. The second integral value is the integral of the product of the spectral information of the first spectral information image estimated by the first machine learning model and the spectral distribution of the light source created based on known arbitrary spectral information, for each spectrum. The spectral information image estimation apparatus according to feature 1.

7. The loss includes a value in the LAB color space calculated from the known spectral information of the training image and a color difference calculated based on the value in the LAB color space calculated from the spectral information of the first spectral information image estimated by the first machine learning model. The spectral information image estimation apparatus according to feature 1.

8. The loss includes the difference in spectral reflectance between adjacent wavelengths in the estimated second spectral information image. The spectral information image estimation apparatus according to feature 2.

9. The loss includes a value obtained from the error between the known spectral information of the training image and the spectral information of the second spectral information image estimated by the second machine learning model. The spectral information image estimation apparatus according to feature 3.

10. The loss includes the error between the RGB image generated from the spectral information of the second spectral information image estimated by the second machine learning model, the imaging spectral sensitivity of the imaging device that captured the training image, and the spectral distribution of the light source in the environment in which the training image was captured, and the RGB image of the training image. The spectral information image estimation apparatus according to feature 9.

11. The loss includes the difference in spectral reflectance between adjacent wavelengths in the estimated second spectral information image. The spectral information image estimation apparatus according to feature 9.

12. The loss includes a value obtained from the difference between the first integral value and the third integral value. The first integral value is the integral of the product of the known spectral information of the training image and the spectral distribution of the light source created based on known arbitrary spectral information, for each spectrum. The third integral value is the integral of the product of the spectral information of the second spectral information image estimated by the second machine learning model and the light source spectral distribution created based on known arbitrary spectral information, for each spectrum. The spectral information image estimation apparatus according to feature 9.

13. The loss includes a value in the LAB color space calculated from the known spectral information of the training image and a color difference calculated based on the value in the LAB color space calculated from the spectral information of the second spectral information image estimated by the second machine learning model. The spectral information image estimation apparatus according to feature 9.

14. A data generation unit selects spectral information images, light source spectral distributions, and imaging device spectral sensitivities from each of the following datasets: a spectral information image dataset containing multiple spectral information images, a light source spectral distribution dataset containing multiple light source spectral distributions, and an imaging device spectral sensitivity dataset containing multiple imaging device spectral sensitivities. The data generation unit then generates an RGB image as a training image from the selected spectral information images, light source spectral distributions, and imaging device spectral sensitivities. A machine learning model generation unit generates a machine learning model using the generated training images, A spectral information image estimation device according to claim 1 or claim 3, further comprising:

15. The data generation unit generates new spectral information images, light source spectral distributions, and imaging device spectral sensitivities for generating the training image by applying an arbitrary magnification to the selected spectral information image, light source spectral distribution, and imaging device spectral sensitivity. The spectral information image estimation device according to feature 14.

16. The aforementioned light source spectral distribution dataset includes spectral components that have a large proportion in the aforementioned light source spectral distribution. The imaging device spectral sensitivity dataset includes spectral components that have a large proportion in the spectral sensitivity of the imaging device. The data generation unit multiplies the spectral components by a predetermined coefficient and adds or subtracts the multiplication result to generate a new light source spectral distribution or imaging device spectral sensitivity for generating the training image. The spectral information image estimation device according to feature 14.

17. The first spectral reflectance estimation unit estimates a first spectral information image showing the spectral reflectance of the subject based on an RGB image which is an image of the subject, the spectral sensitivity of the imaging device which captured the image, and the spectral distribution of the light source in the environment in which the image was captured. The second spectral reflectance estimation unit performs a second spectral reflectance estimation process which estimates a second spectral information image obtained by correcting the first spectral information image based on the first spectral information image estimated by the first spectral reflectance estimation unit, the RGB image, the spectral sensitivity of the imaging device, and the spectral distribution of the light source. Includes, The first spectral reflectance estimation unit inputs the RGB image, the spectral sensitivity of the imaging device, and the spectral distribution of the light source into a first machine learning model and estimates the first spectral information image. The first machine learning model is a trained model that uses a training image, which is an RGB image with a known spectral reflectance, the imaging spectral sensitivity of the imaging device that captured the training image, and the spectral distribution of the light source in the environment in which the training image was captured, as training data, and uses a value obtained from the error between the known spectral information of the training image and the spectral information of the first spectral information image estimated by the first machine learning model as the loss. A spectral information image estimation method characterized by the following:

18. Computers, A first spectral reflectance estimation means estimates a first spectral information image showing the spectral reflectance of the subject based on an RGB image which is an image of the subject, the spectral sensitivity of the imaging device which captured the image, and the spectral distribution of the light source in the environment in which the image was captured. A second spectral reflectance estimation means estimates a second spectral information image obtained by correcting the first spectral information image based on the first spectral reflectance estimation means, the RGB image, the spectral sensitivity of the imaging device, and the spectral distribution of the light source. To make it function as, The first spectral reflectance estimation means inputs the RGB image, the spectral sensitivity of the imaging device, and the spectral distribution of the light source into a first machine learning model to estimate the first spectral information image. The first machine learning model is a trained model that uses a training image, which is an RGB image with a known spectral reflectance, the imaging spectral sensitivity of the imaging device that captured the training image, and the spectral distribution of the light source in the environment in which the training image was captured, as training data, and uses a value obtained from the error between the known spectral information of the training image and the spectral information of the first spectral information image estimated by the first machine learning model as the loss. program.

19. A first spectral reflectance estimation process in which a first spectral reflectance estimation unit estimates a first spectral information image showing the spectral reflectance of the subject based on an RGB image which is an image of the subject, the spectral sensitivity of the imaging device which is the spectral sensitivity of the imaging device which captured the image, and the spectral distribution of the light source in the environment in which the image was captured. The second spectral reflectance estimation unit performs a second spectral reflectance estimation process which estimates a second spectral information image obtained by correcting the first spectral information image based on the first spectral information image estimated by the first spectral reflectance estimation unit, the RGB image, the spectral sensitivity of the imaging device, and the spectral distribution of the light source. Includes, The second spectral reflectance estimation unit inputs the first spectral information image, the RGB image, the spectral sensitivity of the imaging device, and the spectral distribution of the light source into the second machine learning model and estimates the second spectral information image. The second machine learning model generates an RGB image based on the spectral information of the input first spectral information image, the spectral sensitivity of the imaging device, and the spectral distribution of the light source, and corrects the first spectral information image by repeatedly updating the parameters of the second machine learning model so that the loss between the input RGB image and the generated RGB image is reduced. A spectral information image estimation method characterized by the following:

20. A computer, A first spectral reflectance estimation means estimates a first spectral information image showing the spectral reflectance of the subject based on an RGB image which is an image of the subject, the spectral sensitivity of the imaging device which captured the image, and the spectral distribution of the light source in the environment in which the image was captured. A second spectral reflectance estimation means estimates a second spectral information image obtained by correcting the first spectral information image based on the first spectral reflectance estimation means, the RGB image, the spectral sensitivity of the imaging device, and the spectral distribution of the light source. To make it function as, The second spectral reflectance estimation means inputs the first spectral information image, the RGB image, the spectral sensitivity of the imaging device, and the spectral distribution of the light source into a second machine learning model, and estimates the second spectral information image. The second machine learning model generates an RGB image based on the spectral information of the input first spectral information image, the spectral sensitivity of the imaging device, and the spectral distribution of the light source, and corrects the first spectral information image by repeatedly updating the parameters of the second machine learning model so that the loss between the input RGB image and the generated RGB image is reduced. program.

Images