Spectral information image estimation device, spectral information image estimation method, and program

Abstract

To provide a spectral information image estimation device capable of easily estimating a spectral reflectance of an object being photographed with accuracy similar to that of images captured by a multi-band camera from RGB images (three band images, or images of the R-channel, G-channel, and B-channel).SOLUTION: A spectral information image estimation device of the present invention comprises a spectral reflectance estimation unit configured to estimate spectral information images indicative of a spectral reflectance of an object from captured RGB images of the object using a machine learning model. The spectral reflectance estimation unit inputs the captured images, image capturing device spectral sensitivity representing spectral sensitivity of an image capturing device used to acquire the captured images, and a light source spectral distribution representing a spectral distribution of a light source in an environment in which the captured images were acquired into the machine learning model to estimate spectral information images.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a spectral information image estimation device, a spectral information image estimation method, and a program for estimating a spectral information image from an RGB image. [Background technology]

[0002] Spectral reflectance is used when analyzing an object from an image or video of the object. Therefore, in order to obtain the colorimetric values ​​of an object with high accuracy, color estimation is performed by estimating the spectral reflectance of the object from a captured image. In this case, an object is imaged with a multiband camera, and a spectral image is obtained from the image as multiband (four or more bands, for example, 31 bands) spectral reflectance. Furthermore, when estimating the spectral reflectance of an object from an image captured by a multiband camera, the estimation is performed using the spectral sensitivity characteristics of each band of the multiband camera and spectral statistical information of the object.

[0003] However, because multiband cameras are expensive, a spectral image containing multiband spectral information is estimated from an image of three color bands (R channel, G channel, and B channel) of color component R, color component G, and color component B (hereinafter referred to as an RGB image) captured using a commonly used three-band digital camera. Here, a machine learning model (such as CNN (Convolutional Neural Networks) or deep learning) with an RGB image as input data is used to estimate the spectral reflectance of each pixel in the RGB image, and a spectral image is output as the estimation result (see Non-Patent Documents 1 and 2). [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Jiaojiao Li, Chaoxiong Wu, Rui Song, Yunsong Li, Fei Liu, "Adaptive Weighted Attention Network with Camera Spectral Sensitivity Prior for Spectral Reconstruction from RGB Images", 2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), June 2020 [Non-patent document 2] Zhan Shi, Chang Chen, Zhiwei Xiong, Dong Liu, Feng Wu, "HSCNN+: Advanced CNN-Based Hyperspectral Recovery from RGB Images", 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), June 2018 Summary of the Invention [Problem to be solved by the invention]

[0005] However, although each of the above Non-Patent Documents 1 and 2 can extract the spectral characteristics of each object with high accuracy, they estimate the spectral radiance as the spectral characteristic, rather than the spectral reflectance. Furthermore, since the camera spectral sensitivity is estimated using a fixed value, each machine learning model needs to be created for each imaging device. Furthermore, since the estimated spectral information is spectral radiance, it includes the spectral distribution of the light source in the imaging environment (hereinafter referred to as the light source spectral distribution). Therefore, it is necessary to estimate the spectral reflectance from the spectral radiance using the light source spectral distribution in the imaging environment.

[0006] For this reason, the machine learning models in Non-Patent Document 1 and Non-Patent Document 2 cannot easily calculate the spectral reflectance of an object from each of the RGB images captured by an arbitrary imaging device in an arbitrary imaging environment. That is, in each of Non-Patent Document 1 and Non-Patent Document 2, it is necessary to generate a machine learning model for each imaging environment in which estimation is to be performed for each target imaging device. Furthermore, in each of Non-Patent Document 1 and Non-Patent Document 2, it is necessary to perform processing to estimate the spectral reflectance of the object using the estimated spectral radiance and the light source spectral distribution.

[0007] The present invention has been made in view of the above circumstances, and provides a spectral information image estimation device, a spectral information image estimation method, and a program that can easily estimate spectral information (spectral reflectance image, spectral image) of four or more bands of an object being imaged from an RGB image (a three-band image, i.e., an image of each of the R channel, G channel, and B channel) with the same degree of accuracy as an image captured by a multiband camera. [Means for solving the problem]

[0008] In order to solve the above-described problems, the spectral information image estimation device of the present invention includes a spectral reflectance estimation unit that estimates a spectral information image indicating a spectral reflectance of a subject from an RGB image that is a captured image of the subject using a machine learning model; further comprising a data synthesis unit that synthesizes input data; The above Data Synthesis a unit for acquiring the captured image, an imaging device spectral sensitivity that is a spectral sensitivity of an imaging device that captured the captured image, and a light source spectral distribution that is a spectral distribution of a light source in an environment where the captured image is captured, The input data is input and synthesized, and the spectral reflectance estimation unit estimates the data synthesized by the data synthesis unit. The spectral information image is estimated by inputting the data into the machine learning model.

[0009] The spectral information image estimation device of the present invention further includes a machine learning model generation unit that generates the machine learning model, and the machine learning model generation unit uses a teacher image, which is an image with known spectral reflectance, the imaging spectral sensitivity of the imaging device that captured the teacher image, and the light source spectral distribution of the environment in which the teacher image was captured as teacher data, and learns the machine learning model by using a value calculated from the error between the known spectral information of the teacher image and the spectral information of the spectral information image estimated by the machine learning model as a loss.

[0010] The spectral information image estimation device of the present invention is characterized in that the loss includes a reconstruction error between the teacher image and a reconstructed captured image calculated from the spectral information of the spectral information image estimated by the machine learning model, the imaging spectral sensitivity of the imaging device that captured the teacher image, and the light source spectral distribution of the environment in which the teacher image was captured.

[0011] The spectral information image estimation device of the present invention is characterized in that the loss includes a difference in spectral reflectance between adjacent wavelengths in the spectral information image to be estimated.

[0012] The spectral information image estimation device of the present invention comprises: The data synthesis unit When the captured image, the imaging device spectral sensitivity, and the light source spectral power distribution are input to the machine learning model, one of the imaging device spectral sensitivity and the light source spectral power distribution, or a result of summing, multiplying, or concatenating the imaging device spectral sensitivity and the light source spectral power distribution, is set as a first composite value, and a second composite value is obtained by summing, multiplying, or concatenating the first composite value and the captured image. To It is characterized by:

[0013] The spectral information image estimation device of the present invention comprises: The data synthesis unitWhen the captured image, the imaging device spectral sensitivity, and the light source spectral distribution are input to the machine learning model, one of the imaging device spectral sensitivity and the light source spectral distribution, or the sum, product, or concatenation of the imaging device spectral sensitivity and the light source spectral distribution, is used as a first composite value, and a third composite value is generated by the sum, product, or concatenation of the first composite value and output data of any intermediate layer of the machine learning model, and the third composite value is used as input data for an intermediate layer in a stage subsequent to the first intermediate layer. To It is characterized by:

[0014] The spectral information image estimation device of the present invention is characterized in that the data synthesis unit uses a dimension conversion neural network to convert the number of dimensions of an array of the number of data items of the third synthesis value.

[0015] The spectral information image estimation device of the present invention is characterized in that the data synthesis unit synthesizes the first synthesized value with output data of each of the intermediate layers in multiple stages of the intermediate layers that form the machine learning model, and uses each of the third synthesized values ​​as input data for the intermediate layer of the next stage.

[0016] The spectral information image estimation device of the present invention is characterized in that the machine learning model has sub-machine learning models corresponding to each of the R channel, G channel, and B channel in a captured image, and an integrated machine learning model that uses each of the output data of the sub-machine learning models as input values, the input of the sub-machine learning model being image data in the captured image corresponding to each of the channels and sensitivity data in the spectral sensitivity of the imaging device corresponding to each of the channels, a data synthesis unit that generates input data for the machine learning model generates a fourth synthesized value by summing, multiplying, or concatenating the output data of the sub-machine learning models, and the spectral reflectance estimation unit inputs the fourth synthesized value to the integrated machine learning model.

[0017] The spectral information image estimation device of the present invention is characterized in that the spectral reflectance estimation unit inputs the light source spectral distribution to the sub machine learning model or the integrated machine learning model.

[0018] The spectral information image estimation device of the present invention is characterized in that the data synthesis unit inputs, to each of the sub-machine learning models, image data of each channel in the captured image and sensitivity data in the spectral sensitivity of the imaging device corresponding to each channel, or a fifth synthesis value obtained by adding, multiplying, or concatenating the image data and the sensitivity data of each channel in the spectral sensitivity of the imaging device.

[0019] The spectral information image estimation device of the present invention is characterized in that image data for each channel of the corresponding sub-machine learning model in the captured image is input to each of the sub-machine learning models, and the data synthesis unit inputs either sensitivity data corresponding to each channel in the spectral sensitivity of the imaging device, or a fifth synthesized value obtained by adding, multiplying, or concatenating the image data and the sensitivity data corresponding to each channel in the spectral sensitivity of the imaging device, to each of the intermediate layers constituting the sub-machine learning models.

[0020] The spectral information image estimation device of the present invention is characterized in that image data for each channel of the corresponding sub-machine learning model in the captured image is input to each of the sub-machine learning models, and the data synthesis unit converts the number of dimensions of the array of data counts and inputs to each of the intermediate layers constituting the sub-machine learning models either sensitivity data corresponding to each of the channels in the spectral sensitivity of the imaging device, or a fifth synthesis value obtained by summing, multiplying, or concatenating the image data, the light source spectral distribution, and the sensitivity data corresponding to each of the channels in the spectral sensitivity of the imaging device.

[0021] The spectral information image estimation device of the present invention is characterized in that it includes a machine learning model generation unit that generates the machine learning model, each of the sub-machine learning models having the same configuration, and the machine learning model generation unit changes each of the coefficients of the functions in the neural network of each of the sub-machine learning models by the same value, thereby training the sub-machine learning model.

[0022] The spectral information image estimation device of the present invention is characterized in that the spectral reflectance estimation unit includes a first spectral reflectance estimation unit and a second spectral reflectance estimation unit, the first spectral reflectance estimation unit receives the captured image and at least one of the imaging device spectral sensitivity and the light source spectral power distribution as inputs and estimates a primary spectral information image, and the second spectral reflectance estimation unit receives the primary spectral information image estimated by the first spectral reflectance estimation unit, the captured image, the imaging device spectral sensitivity, and the light source spectral power distribution as inputs and estimates the spectral information image.

[0023] The spectral information image estimation device of the present invention is characterized in that the second spectral reflectance estimation unit determines a loss as a value calculated from a reproduction error between the captured image and a reproduced captured image obtained from the spectral information image estimated from the primary spectral information image, and corrects the spectral information image so that the loss satisfies a predetermined condition.

[0024] The spectral information image estimation device of the present invention is characterized in that the first spectral reflectance estimation unit is configured such that a machine learning model includes a neural network, and a machine learning model generation unit that generates the machine learning model calculates coefficients of a function in the neural network and marks the neural network as trained.

[0025] The spectral information image estimation method of the present invention includes a spectral reflectance estimation step in which a spectral reflectance estimation unit estimates a spectral information image indicating the spectral reflectance of a subject from an RGB image that is a captured image of the subject using a machine learning model. a data synthesis process in which a data synthesis unit synthesizes input data; and The data synthesis unit The captured image, an imaging device spectral sensitivity that is the spectral sensitivity of the imaging device that captured the captured image, and a light source spectral distribution that is the spectral distribution of a light source in an environment where the captured image was captured are acquired. The input data is input and synthesized, and the spectral reflectance estimation unit estimates the data synthesized by the data synthesis unit. The spectral information image is estimated by inputting the data into the machine learning model.

[0026] The program of the present invention includes: a spectral reflectance estimation means for estimating a spectral information image indicating the spectral reflectance of a subject from an RGB image that is a captured image of the subject using a machine learning model; data synthesis means for synthesizing input data;and Data synthesis method the captured image, the imaging device spectral sensitivity, which is the spectral sensitivity of the imaging device that captured the captured image, and the light source spectral distribution, which is the spectral distribution of the light source in the environment where the captured image was captured, The input data is input and synthesized, and the spectral reflectance estimation means , The data synthesized by the data synthesis means The program inputs the data into the machine learning model and estimates the spectral information image. [Effects of the Invention]

[0027] As described above, according to the present invention, it is possible to provide a spectral information image estimation device, a spectral information image estimation method, and a program that can easily estimate spectral information (spectral reflectance image, spectral image) of four or more bands of an object being imaged from an RGB image (a three-band image, i.e., an image of each of the R channel, G channel, and B channel) with the same degree of accuracy as an image captured by a multiband camera. [Brief explanation of the drawings]

[0028] [Figure 1] FIG. 1 is a conceptual diagram showing an example of a machine learning model (such as a convolutional neural network (CNN), neural network, or deep learning) that outputs a spectral image as spectral reflectance information from an RGB image, or a spectral information image. [Figure 2] 1 is a block diagram showing an example of the configuration of a spectral information image estimation device according to a first embodiment of the present invention. [Figure 3] FIG. 10 is a diagram illustrating an example of the configuration of a training data table having a dataset of training data used to train a machine learning model. [Figure 4] FIG. 2 is a conceptual diagram illustrating the synthesis of teacher data by a data synthesis unit 12. [Figure 5] FIG. 10 is a diagram showing an example of the configuration of a captured image data table having a dataset of captured image data used to allow a machine learning model to estimate a spectral information image. [Figure 6] 10 is a diagram showing an example of the configuration of a spectral information image data table in a spectral information image data storage unit 19. FIG. [Figure 7] 10 is a flowchart showing an example of the operation of a learning process using teacher data of a machine learning model by the machine learning model generation unit 13 according to this embodiment. [Figure 8] 10 is a flowchart showing an example of the operation of a process of estimating a spectral information image from an RGB image using a trained machine learning model by the spectral reflectance estimating unit 14 according to this embodiment. [Figure 9] FIG. 10 is a conceptual diagram illustrating the estimation of coefficients of basis functions that constitute a spectral reflectance curve using a machine learning model and a corrected machine learning model. [Figure 10] FIG. 10 is a conceptual diagram illustrating input of data on the spectral sensitivity of an imaging device and the spectral distribution of a light source to an intermediate layer of a machine learning model according to a second embodiment of the present invention. [Figure 11] FIG. 11 is a conceptual diagram illustrating an example of the configuration of a machine learning model used by a first spectral reflectance estimating unit 141 according to a third embodiment of the present invention. [Figure 12] FIG. 10 is a block diagram showing an example of the configuration of a spectral information image estimation device according to a fourth embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0029] In the present invention, a spectral information image (spectral image or spectral reflectance image) of wavelengths of four or more bands (31 bands, for example, in the embodiment described below) is estimated and generated using a machine learning model from RGB values, which are a captured image consisting of three wavelength bands: the R channel of the color component R (Red), the G channel of the color component G (Green), and the B channel of the color component B (Blue). The RGB values ​​(gradation of each color component) of each pixel in an RGB image are represented by the spectral reflectance of each pixel, the spectral sensitivity of the imaging device that captured the captured image (hereinafter referred to as imaging device spectral sensitivity), and the spectral distribution of the light source in the environment where the captured image was captured (hereinafter referred to as light source spectral distribution).

[0030] FIG. 1 is a conceptual diagram showing an example of a spectral image as spectral reflectance information from an RGB image, or a machine learning model (such as a neural network or deep learning) that outputs a spectral information image. Figure 1(a) shows the formula that shows that the product of the spectral power distribution of the light source, the spectral reflectance of the target object, and the spectral sensitivity of the imaging device for each wavelength (λ) can be calculated as the integral value for wavelengths (λ) from 400 nm to 700 nm. Furthermore, the spectral radiance can be calculated from the spectral power distribution of the light source and the spectral reflectance of the target object. As can be seen from this formula, a machine learning model for estimating the spectral reflectance of a subject can be generated using an RGB image, a light source spectral distribution, and an imaging device spectral sensitivity as input data sets.

[0031] FIG. 1(b) is a conceptual diagram showing an example of a machine learning model that outputs a spectral image as spectral reflectance information from an RGB image, or a spectral information image. The machine learning model 300 takes, for example, an RGB image 301, an image capture device spectral sensitivity 302, and a light source spectral distribution 303 as a set of input data, and outputs spectral images 305 of each of predetermined wavelengths of spectral reflectance (for example, 31 bands of wavelengths in 10 nm increments in the wavelength range from 400 nm to 700 nm). Here, the horizontal axis of the graph of the imaging device spectral response 302 represents wavelength and the vertical axis represents intensity, and the horizontal axis of the graph of the light source spectral distribution 303 represents wavelength and the vertical axis represents intensity.

[0032] First Embodiment A first embodiment of the present invention will be described below with reference to the drawings. FIG. 2 is a block diagram showing an example of the configuration of a spectral information image estimation device according to the first embodiment of the present invention. In FIG. 2, the spectral information image estimation device 10 includes a data input unit 11, a data synthesis unit 12, a machine learning model generation unit 13, a spectral reflectance estimation unit 14, a spectral image output unit 15, a teacher image data storage unit 16, a captured image data storage unit 17, a machine learning model storage unit 18, and a spectral information image data storage unit 19. The spectral reflectance estimating unit 14 includes a first spectral reflectance estimating unit 141 and a second spectral reflectance estimating unit 142 .

[0033] The data input unit 11 inputs teacher image data supplied from an external device, and writes the teacher image data into the teacher image data storage unit 16 for storage. FIG. 3 is a diagram showing an example of the configuration of a teacher image data table having a dataset of teacher image data used for training a machine learning model. The teacher image data table has columns for teacher image identification information, RGB image index, light source spectral distribution index, imaging device spectral sensitivity index, and spectral reflectance information index for each record.

[0034] The teacher image identification information is identification information that individually identifies each teacher image (an RGB image, image data in which the spectral reflectance of the subject is known). The RGB image index is an address or the like indicating a storage area in the teacher image data storage unit 16 where image data of the RGB image, which is the captured image, is stored. The data of an RGB image is the gradation (e.g., 256 gradations) of each of the color components R, G, and B (three bands, i.e., three channels consisting of the R channel, the G channel, and the B channel) for each pixel of the RGB image (number of data: 3 / pixel).

[0035] The light source spectral distribution index is an address indicating a storage area in the teacher image data storage unit 16 where data on the light source spectral distribution indicating the spectral distribution of the light source in the environment in which the RGB image, which is the captured image, is captured, is stored. The light source spectral distribution indicates, for example, the distribution of light intensity at each wavelength (31 bands) in 10 nm increments in the wavelength range from 400 nm to 700 nm (the number of data points is 31).

[0036] The imaging device spectral sensitivity index is, for example, an address indicating a storage area in the teacher image data storage unit 16 where imaging device spectral sensitivity data indicating the spectral sensitivity of the imaging device that captured the RGB image that is the captured image is stored. The spectral sensitivity of the imaging device indicates, for example, the distribution of light intensity for each of the color components R, G, and B at wavelengths (31 bands) in 10 nm increments in the wavelength range from 400 nm to 700 nm (data number 93 (= 3 × 31)).

[0037] The spectral reflectance information index is, for example, an address indicating a storage area in the teacher image data storage unit 16 in which data on spectral reflectance information (known spectral reflectance information), which is information on the spectral reflectance of the subject of the captured RGB image, is stored. The spectral reflectance information indicates, for example, the distribution of light intensity for each pixel of an RGB image at wavelengths (31 bands) in 10 nm increments in the wavelength range from 400 nm to 700 nm (31 data points per pixel).

[0038] Returning to FIG. 1, the data synthesis unit 12 generates a dataset of input data to be input to the machine learning model. That is, the data synthesis unit 12 adjusts and synthesizes the data of the RGB image, the light source spectral distribution, and the imaging device spectral sensitivity when training a machine learning model using teacher image data and when estimating image data of a spectral information image using a trained machine learning model.

[0039] As described above, the input data sets used for each pixel are three RGB image data, 31 light source spectral distribution data, and 31×3=93 imaging device spectral sensitivity data. Therefore, the number of data in the dataset is (3+31+93)×W×H, where W is the number of pixels in the horizontal direction (rows) of the RGB image and H is the number of pixels in the vertical direction (columns). Here, the data synthesis unit 12 performs synthesis processing on each piece of data in the data set, for example, synthesis by combining (that is, linking), summing, multiplying, and the like.

[0040] In synthesizing the training data, the data synthesis unit 12 arranges, for example, (3+31+93)×W×H pieces of data for every W×H pixels. FIG. 4 is a conceptual diagram illustrating the synthesis of training data by the data synthesis unit 12. In FIG. FIG. 4(a) shows an arrangement of W×H pixels, each of which is arranged as a gradient GR of color component R, a gradient GG of color component G, and a gradient GB of color component B. Pixels P1 to Pn (n=W×H) are sequentially arranged in series, and the color components R, G, and B of each pixel are also arranged in series.

[0041] Then, as a combining process in the combination process, the data combining unit 12 connects and combines the data of the imaging device spectral sensitivity and the light source spectral distribution as shown in FIG. 4(b). FIG. 4(b) explains the synthesis by combining the training data in the data synthesis unit 12. Here, the data synthesis unit 12 serially connects and inserts data QR of each of the 31 bands of color component R of the imaging device spectral sensitivity between the gradient (pixel value) GR of color component R and the gradient GG of color component G in the array of each pixel, for example.

[0042] The data synthesis unit 12 also serially connects and inserts data QG of each of the 31 bands of the color component G of the imaging device's spectral sensitivity between the gradient GG of the color component G and the gradient GB of the color component B. Then, the data synthesis unit 12 serially connects and inserts data QB of each of the 31 bands of color component B of the spectral sensitivity of the imaging device after the gradient GB of color component B.

[0043] Furthermore, the data synthesis unit 12 serially connects and inserts the data S of the 31 bands of the light source spectral distribution after the data QB of the 31 bands of the color component B of the spectral sensitivity of the imaging device. Here, the serial arrangement of the color components R, G, and B of the spectral sensitivity of the imaging device and the spectral distribution of the light source becomes the first composite value, and the data string in which each of these first composite values ​​is inserted into the pixel array shown in Figure 4(a) becomes the second composite value shown in Figure 4(b).

[0044] The data synthesis unit 12 performs the synthesis and combination through the above-described processing, and outputs the results to the first spectral reflectance estimation unit 141. When performing the synthesis process, the data synthesis unit 12 reads out data on the pixel values ​​of the RGB image, the spectral sensitivity of the imaging device, and the spectral distribution of the light source from the teacher image data storage unit 16 when generating a machine learning model, and reads out data from the captured image data storage unit 17 when estimating a spectral information image, and normalizes the read out data.

[0045] In addition, as a summation process in the synthesis, the data synthesis unit 12 adds, for example, data for each wavelength in the 31 bands of the imaging device spectral sensitivity to data for each of the same wavelengths in the 31 bands of the light source spectral distribution (taking the sum of the data for corresponding wavelengths). That is, the data synthesis unit 12 adds data on 400 wavelengths in 31 bands of the light source spectral distribution to data on 400 wavelengths in 31 bands of the color component R of the imaging device spectral sensitivity, and sets each of the resulting data as a synthesis value for the respective wavelength.

[0046] In this way, the data synthesis unit 12 performs a process of synthesizing the 31 band data of the light source spectral distribution by adding together the 31 band data of each of the color components R, G, and B of the imaging device spectral sensitivity (generation of a first synthesis value). Then, the data synthesis unit 12 synthesizes the summed data by inserting and connecting the data between the color components R, G, and B of the corresponding pixels so that the color components of the corresponding imaging device spectral sensitivities correspond to each other, as shown in Figure 4(b) (generation of a second synthesis value).

[0047] In addition, as a multiplication process in the synthesis, the data synthesis unit 12 multiplies, for example, data on each of the 31 wavelength bands of the imaging device spectral sensitivity by data on the same wavelength in each of the 31 wavelength bands of the light source spectral distribution (taking the product of the data on the corresponding wavelengths). That is, the data synthesis unit 12 multiplies the data of 400 wavelengths in 31 bands of the color component R of the spectral sensitivity of the imaging device by the data of 400 wavelengths in 31 bands of the light source spectral distribution, and sets each of the data resulting from the multiplication as a synthesis value for the respective wavelength.

[0048] In this way, the data synthesis unit 12 performs a process of synthesizing the 31 band data of the light source spectral distribution by multiplying the 31 band data of each of the color components R, G, and B of the imaging device spectral sensitivity (generation of a first synthesis value). Then, the data synthesis unit 12 synthesizes the multiplied data by inserting and connecting the data between the color components R, G, and B of the corresponding pixels so that the color components of the corresponding imaging device spectral sensitivities correspond to each other, as shown in Figure 4(b) (generation of a second synthesis value).

[0049] The data synthesis unit 12 may also be configured to generate second synthesis values ​​by adding or multiplying each of the first synthesis values ​​generated by combining, summing, or multiplying the spectral sensitivity of the imaging device and the spectral distribution of the light source to the color component R, color component G, and color component B of the RGB image, respectively. At this time, the data synthesis unit 12 may obtain a second synthesis value by combining the first synthesis value obtained by the combining process with the pixel data of the RGB image by combining, adding, or multiplying. The data synthesis unit 12 may also obtain a second synthesis value by combining the first synthesis value obtained by sum processing with pixel data of the RGB image, or by synthesizing the first synthesis value by sum or product processing. Furthermore, the data synthesis unit 12 may combine the first synthesis value obtained by the multiplication process with the pixel data of the RGB image, or may obtain a second synthesis value by performing sum or product processing.

[0050] Returning to FIG. 1, the machine learning model generation unit 13 outputs the training data, which is the second composite value, to the first spectral reflectance estimation unit 141. The first spectral reflectance estimating unit 141 inputs the training data into a neural network, which is a machine learning model, and performs estimation processing of a spectral information image. Here, the machine learning model estimates the spectral reflectance corresponding to each pixel of the RGB image using training data generated from each of the RGB image, the imaging device spectral sensitivity, and the light source spectral distribution, and estimates and outputs a 31-band spectral information image.

[0051] Then, the machine learning model generation unit 13 reads out known spectral reflectance information of the subject in the RGB image from the teacher image data table in the teacher image data storage unit 16. FIG. 5 is a diagram showing an example of the configuration of a captured image data table having a data set of captured image data used to allow a machine learning model to estimate a spectral information image. The captured image data table has columns for captured image identification information, an RGB image index, a light source spectral distribution index, and an imaging device spectral sensitivity index for each record.

[0052] The captured image identification information is identification information that individually identifies each captured image. The RGB image index is an address or the like indicating a storage area in the teacher image data storage unit 16 where image data of the RGB image, which is the captured image, is stored. The data of the RGB image is the gradation (for example, 256 gradations) of each of the color components R, G, and B (3 bands) for each pixel of the RGB image (number of data 3 / pixel).

[0053] The light source spectral distribution index is an address indicating a storage area in the teacher image data storage unit 16 where data on the light source spectral distribution indicating the spectral distribution of the light source in the environment in which the RGB image, which is the captured image, is captured, is stored. The light source spectral distribution indicates, for example, the distribution of light intensity at each wavelength (31 bands) in 10 nm increments in the wavelength range from 400 nm to 700 nm (the number of data points is 31).

[0054] The imaging device spectral sensitivity index is, for example, an address indicating a storage area in the teacher image data storage unit 16 where imaging device spectral sensitivity data indicating the spectral sensitivity of the imaging device that captured the RGB image that is the captured image is stored. The spectral sensitivity of the imaging device indicates, for example, the distribution of light intensity for each of the color components R, G, and B at wavelengths (31 bands) in 10 nm increments in the wavelength range from 400 nm to 700 nm (data number 93 (= 3 × 31)).

[0055] Returning to FIG. 2, the machine learning model generation unit 13 obtains, as a spectral error, the difference between the data for each pixel of the 31-band spectral information image estimated by the machine learning model and the data of known spectral reflectance information. In addition, the machine learning model generation unit 13 calculates the gradations of the color components R, G, and B from the spectral reflectance information estimated by the machine learning model, the imaging device spectral sensitivity, and the light source spectral distribution, and sets these as estimated RGB values.

[0056] The machine learning model generation unit 13 calculates, as an RGB value error, the difference between the estimated RGB values ​​(gradation of color components R, G, and B) calculated from the spectral reflectance (intensity for each wavelength) of each pixel of the spectral information image and the RGB values ​​of each pixel of the RGB image used to estimate the spectral reflectance. Then, the machine learning model generation unit 13 substitutes each of the spectral error and the RGB value error into the loss function of the following equation (1) to obtain the loss (loss value) L.

[0057]

number

[0058] In the loss function of the above equation (1), the loss L spectral and loss L RGB Each of these is a loss (loss value) calculated using the MSE (Mean Squared Error) loss shown in the following equation (2) or the MRAE (Mean Relative Absolute Error) loss shown in the following equation (3). In equation (1), τ1 is a predetermined constant. In each of the following equations (2) and (3), x est (i) is the estimated value (estimated spectral reflectance, estimated RGB value calculated from the estimated spectral reflectance), and xgt (i) is the true value (known spectral reflectance or RGB value). Also, i is the pixel number, and in the RGB value, the loss L R , loss L G , loss L B and add them together to find the loss L RGB (=L R +L G +L B ) is required.

[0059]

number

[0060]

number

[0061] Furthermore, since the spectral reflectance of an object generally does not change sharply between adjacent bands, the loss may be calculated using the following equation (4), which adds a smoothing term to equation (1). Loss L in equation (4) smooth is a numerical value obtained by normalizing the difference in intensity values ​​between adjacent bands of the estimated spectral reflectance and adding up all the normalized differences between the bands. In addition, τ2 in equation (4) is a predetermined constant.

[0062]

number

[0063] Then, the machine learning model generation unit 13 determines whether the loss L is equal to or less than a loss threshold (first threshold), which is a preset threshold. The machine learning model generation unit 13 adjusts the coefficients of the functions in the neural network, which is the machine learning model (i.e., the weighting coefficients at the input or output of the functions in each layer (especially the intermediate layer) that constitutes the neural network), and again inputs the training data into the machine learning model to estimate the spectral information image.

[0064] Then, the machine learning model generation unit 13 adjusts the coefficients of the functions in the neural network, which is the machine learning model, until the loss L between the data of the spectral information image estimated by the machine learning model and the data of the known spectral reflectance information becomes equal to or less than the loss threshold, thereby training the machine learning model. Here, the machine learning model generation unit 13 performs learning so as to satisfy predetermined conditions, such as the loss L calculated by equation (1) or equation (4) being minimum (minimum value), or being equal to or less than a predetermined threshold, or the number of repetitions of the learning process for the corrected machine learning model reaching a predetermined number. Furthermore, the machine learning model generation unit 13 writes and stores the machine learning model in which the loss L is equal to or less than the loss threshold in the machine learning model storage unit 18 as a trained machine learning model.

[0065] When estimating an unknown spectral information image from an RGB image, which is image data captured by an arbitrary image capturing device and in an arbitrary environment, the data synthesis unit 12 reads out data on the image capturing device spectral sensitivity of the image capturing device, the light source spectral distribution of the captured environment, and the captured image from the image capturing image data storage unit 17, generates a second synthesis value, and outputs it to the first spectral reflectance estimation unit 141 as input data. The first spectral reflectance estimation unit 141 receives the input data supplied from the data synthesis unit 12 and reads the trained machine learning model from the machine learning model storage unit 18.

[0066] The first spectral reflectance estimation unit 141 inputs the input data supplied from the data synthesis unit 12 to the read trained machine learning model. Here, the first spectral reflectance estimating unit 141 outputs the spectral information image estimated by the trained machine learning model from the input data to the second spectral reflectance estimating unit 142 as a primary spectral information image.

[0067] Similar to the machine learning model of the first spectral reflectance estimating unit 141, the second spectral reflectance estimating unit 142 includes a corrected machine learning model of a neural network. The corrected machine learning model inputs the primary spectral information image supplied from the first spectral reflectance estimating unit 141, and estimates and outputs a corrected spectral information image. Then, the second spectral reflectance estimating unit 142 reconstructs an RGB image from the corrected spectral information image using the captured image spectral sensitivity and the light source spectral distribution corresponding to the primary spectral information image.

[0068] The second spectral reflectance estimation unit 142 also calculates, as an error, the difference in gradient of each of the color components R, G, and B at each pixel between the RGB image input to the first spectral reflectance estimation unit 141 and the corrected spectral information image. The second spectral reflectance estimation unit 142 then calculates a loss function L RGB The error is substituted into the following equation (5) to find the loss (loss value) L. In equation (5), τ1 is a predetermined constant.

[0069]

number

[0070] The spectral image output unit 15 writes the image data of the spectral information image generated by the second spectral reflectance estimating unit 142 into the spectral information image data storage unit 19 for storage. The second spectral reflectance estimation unit 142 performs training so as to satisfy predetermined conditions, such as the loss L calculated by equation (5) being minimized (minimum value), or being equal to or less than a predetermined threshold, or the number of repetitions of the learning process for the corrected machine learning model reaching a predetermined number. Here, if the loss L satisfies a predetermined condition, the second spectral reflectance estimating unit 142 outputs the corrected spectral information image as the final spectral information image.

[0071] Similarly to the above equation (4), the second spectral reflectance estimation unit 142 calculates the loss function L spectral A value obtained from the following equation (6) including the term * is set as the loss L, and if a predetermined condition is satisfied, the corrected spectral information image may be output as the final spectral information image.

[0072]

number

[0073] FIG. 6 is a diagram showing an example of the structure of the spectral information image data table in the spectral information image data storage unit 19. As shown in FIG. The spectral information image data table has columns for captured image identification information, RGB image index, and spectral information image index for each record.

[0074] The captured image identification information is identification information that individually identifies each captured image, and is the same information as the captured image identification information of the RGB image used for estimation. The RGB image index is an address or the like indicating a storage area in the teacher image data storage unit 16 where image data of the RGB image, which is the captured image, is stored. The data of an RGB image is the gradation (e.g., 256 gradations) of each of the color components R, G, and B (three bands, i.e., R channel, G channel, and B channel) for each pixel of the RGB image (number of data: 3 / pixel).

[0075] The light source information image index is an address or the like indicating a storage area in the spectral information image data storage unit 19 where image data of a spectral information image corresponding to an RGB image that is a captured image is stored. In the spectral information image, for example, the light intensity of each wavelength (31 bands) in 10 nm increments in the wavelength range from 400 nm to 700 nm is used as the gradation of the pixel in the image. Therefore, the spectral information image has one image data for each wavelength band, and is made up of 31 image data items.

[0076] 7 is a flowchart showing an example of the operation of the learning process using training data of the machine learning model by the machine learning model generation unit 13 according to this embodiment. In the following explanation, the training data is a data set consisting of an RGB image, the spectral sensitivity of the image capture device, the spectral distribution of the light source, and the known spectral reflectance of the subject in the RGB image. In the data set, the input data to the machine learning model is an RGB image, the imaging device spectral sensitivity, and the light source spectral distribution, and the estimated data of the machine learning model is a spectral information image. In the following description, the data input unit 11 writes a data set of teacher image data used for learning the machine learning model in a teacher image data table of the teacher image data storage unit 16 in advance.

[0077] Step S101: The data synthesis unit 12 reads out data on the teacher image, the imaging device spectral sensitivity, and the light source spectral distribution from the teacher image data table in the teacher image data storage unit 16 . The data synthesis unit 12 then serially arranges the data on the R, G, and B color components of the spectral sensitivity of the imaging device and the light source spectral distribution to generate a first synthesis value.

[0078] The data synthesis unit 12 synthesizes the generated first synthetic value with the pixel values ​​of the pixels of the teacher image, for example, as an array as shown in Figure 4(b), to generate a second synthetic value in which all data is arranged in series, and outputs it to the machine learning model generation unit 13. Here, the data synthesis unit 12 generates a plurality of second synthesis values ​​for each of the plurality of teacher images stored in the teacher image data table of the teacher image data storage unit 16, from the plurality of teacher images and the imaging device spectral sensitivity and light source spectral distribution corresponding to the teacher images. As a result, the machine learning model generation unit 13 receives as input each piece of training data supplied from the data synthesis unit 12 as the second synthesis value.

[0079] Step S102: The machine learning model generation unit 13 inputs the training data as the second combined value supplied from the data combination unit 12 in parallel to the machine learning model to be learned. The machine learning model estimates and outputs a spectral information image as a primary spectral information image (a predetermined number of bands, for example, 31 bands) in accordance with the training data. Here, the machine learning model generation unit 13 estimates each primary spectral information image using multiple teacher images stored in the teacher image data table of the teacher image data storage unit 16 and multiple second composite values ​​generated from the imaging device spectral sensitivity and light source spectral distribution corresponding to the teacher images as teacher data.

[0080] Step S103: The machine learning model generation unit 13 calculates, for each RGB image, the spectral difference between the estimated primary spectral information image and the known spectral reflectance of the RGB image. Furthermore, the machine learning model generation unit 13 generates a reconstructed RGB image from the primary spectral information image, the imaging device spectral sensitivity, and the light source spectral distribution. Then, the machine learning model generation unit 13 calculates RGB errors as differences between the reconstructed RGB image and the RGB image in terms of the color component R, the color component G, and the color component G, respectively.

[0081] Step S104: Next, the machine learning model generation unit 13 substitutes the spectral difference into the loss function of equation (2) or (3) to obtain the loss L spectral Ask for. The machine learning model generation unit 13 calculates the loss L from the spectral difference between the primary spectral information image in each band of the teacher image and the known spectral reflectance for all the teacher images. spectral Ask for.

[0082] Furthermore, the machine learning model generation unit 13 substitutes the RGB error into the loss function of equation (2) or (3) to obtain the loss L RGB Ask for. The machine learning model generation unit 13 calculates the loss L from the RGB difference between the reconstructed RGB image and the RGB image from which the primary spectral information image is obtained for all the teacher images. RGB Ask for.

[0083] The machine learning model generation unit 13 calculates the loss L spectral and loss L RGB The loss L is calculated from equation (1) or equation (4) using each of the above. Here, when calculating the loss L by equation (4), the machine learning model generation unit 13 calculates the loss L from the primary spectral information image. smooth is calculated, and the loss L spectral , loss L RGB and loss L smooth Add each of these to find the loss L.

[0084] Step S105: The machine learning model generation unit 13 determines whether or not the number of iterations for finding the loss L exceeds a predetermined number of iterations that is set in advance. At this time, if the number of iterations for calculating the loss L exceeds a predetermined number, the machine learning model generation unit 13 advances the process to step S106. On the other hand, if the number of iterations for calculating the loss L is equal to or less than the predetermined number, the machine learning model generation unit 13 proceeds to step S107.

[0085] Step S106: The machine learning model generation unit 13 selects the machine learning model with the smallest loss L from the machine learning models obtained by repeating the process a predetermined number of times, and writes and stores the selected machine learning model in the machine learning model storage unit 18 as having been learned. Then, the machine learning model generation unit 13 ends the process of learning the machine learning model, that is, the process of generating the machine learning model.

[0086] Step S107: The machine learning model generation unit 13 adjusts the coefficients of the neural network functions in the machine learning model. Then, the machine learning model generation unit 13 advances the process to step S102, and estimates the primary spectral information image again.

[0087] FIG. 8 is a flowchart showing an example of the operation of the spectral reflectance estimating unit 14 according to this embodiment to estimate a spectral information image from an RGB image using a trained machine learning model. In the following description, the machine learning model storage unit 18 stores a trained machine learning model.

[0088] Step S201: The data input unit 11 reads data of an RGB image from an external device, the data being captured with the imaging device spectral sensitivity of the imaging device that captured the RGB image and the light source spectral distribution of the light source in the imaging environment. Then, the data input unit 11 writes and stores the read RGB image and the data on the imaging device spectral sensitivity and light source spectral distribution corresponding to the RGB image in a captured image table (see FIG. 5) of the captured image data storage unit 17.

[0089] Step S202: The first spectral reflectance estimation unit 141 reads out a trained machine learning model from the machine learning model storage unit 18. The first spectral reflectance estimation unit 141 also outputs captured image identification information of the estimation target of the spectral information image to the data synthesis unit 12, and requests synthesis of the input data.

[0090] Step S203: The data synthesis unit 12 reads out data on the RGB image, the imaging device spectral sensitivity, and the light source spectral distribution from the captured image data table in the captured image data storage unit 17 . The data synthesis unit 12 then serially arranges the data on the R, G, and B color components of the spectral sensitivity of the imaging device and the light source spectral distribution to generate a first synthesis value. The first spectral reflectance estimating unit 141 receives as input each of the training data as the second composite value supplied from the data combining unit 12.

[0091] Step S204: The first spectral reflectance estimation unit 141 inputs the input data as the second combined value supplied from the data combination unit 12 in parallel to the trained machine learning model. The machine learning model estimates and outputs a spectral information image as a primary spectral information image (a predetermined number of bands, for example, 31 bands) in accordance with the training data. Then, the first spectral reflectance estimating unit 141 outputs the estimated primary spectral information image to the second spectral reflectance estimating unit 142.

[0092] Step S205: The second spectral reflectance estimating unit 142 receives the primary spectral information image supplied from the first spectral reflectance estimating unit 141 . Furthermore, the second spectral reflectance estimating unit 142 reads from the captured image data table in the captured image data storage unit 17 each of the imaging device spectral sensitivity and light source spectral distribution corresponding to the RGB image obtained by estimating the primary spectral information image.

[0093] Then, the second spectral reflectance estimation unit 142 inputs the primary spectral information image to the corrected machine learning model. As a result, the corrected machine learning model estimates a corrected spectral information image corresponding to the input primary spectral information image.

[0094] Step S206: The second spectral reflectance estimating unit 142 reconstructs an RGB image using the corrected spectral information image, the imaging device spectral sensitivity, and the light source spectral distribution. Then, the second spectral reflectance estimating unit 142 calculates a reconstruction error (RGB error) from the reconstructed RGB image and the RGB image used for the estimation. At this time, the second spectral reflectance estimating unit 142 estimates the loss due to the reproduction error using equation (2) or (3). LRGB is calculated and substituted into equation (5) to calculate the loss L.

[0095] Step S207: The second spectral reflectance estimating unit 142 determines whether the number of repetitions for calculating the loss L exceeds a predetermined number of times. At this time, if the number of repetitions for calculating the loss L exceeds a predetermined number of times, the second spectral reflectance estimating unit 142 advances the process to step S208. On the other hand, if the number of repetitions for obtaining the loss L is equal to or less than the predetermined number of times set in advance, the second spectral reflectance estimating unit 142 proceeds to step S209.

[0096] Step S208: The second spectral reflectance estimation unit 142 determines, as the spectral image information, the corrected spectral information image in which the loss L is smallest among the corrected spectral information images generated by the corrected machine learning model after a predetermined number of repetitions. Then, the second spectral reflectance estimating unit 142 writes and stores the data of the spectral information image in the spectral information image data table of the spectral information image data storage unit 19.

[0097] Step S209: The second spectral reflectance estimation unit 142 adjusts the coefficients of the neural network function in the reproduction machine learning model. Then, the second spectral reflectance estimating unit 142 advances the process to step S205, and again estimates a corrected spectral information image from the primary spectral information image using the corrected machine learning model with adjusted coefficients.

[0098] As described above, in this embodiment, an RGB image, the imaging device spectral sensitivity of the imaging device that captured the RGB image, and the light source spectral distribution of the light source in the environment where the RGB image was captured are each used as a data set of input data, and the spectral information of the subject in the RGB image is estimated as a primary spectral information image using a machine learning model. A corrected spectral information image is estimated from the primary spectral information image using an adjusted machine learning model. A loss L is calculated from the RGB error between a reconstructed RGB image reconstructed from the corrected spectral information image and the estimated RGB image of the primary spectral information image. The coefficients of the corrected machine learning model are adjusted until the above-described predetermined condition is satisfied, and the corrected spectral information image is re-estimated using the adjusted corrected machine learning model to obtain a spectral information image. In other words, the spectral information image is estimated through two-stage estimation using each of the machine learning model and the corrected machine learning model. With this configuration, according to this embodiment, the primary spectral information image estimated by the machine learning model is used to generate a corrected spectral information image that reproduces a reproduced RGB image that is closer to each of the color components R, G, and B of the RGB image used to estimate the primary spectral information image. Therefore, even if the estimation accuracy of the primary spectral information image by the first spectral reflectance estimation unit 141 is lower than when a multiband camera is used due to a lack of learning data or the characteristics of the machine learning model, the second spectral reflectance estimation unit 142 corrects the primary spectral information image, so that the spectral reflectance of the object being imaged can be easily estimated from the RGB image (images of three bands, i.e., images of each channel of the R channel, G channel, and B channel) with the same level of accuracy as an image captured by a multiband camera.

[0099] In the above-described embodiment, the data synthesis unit 12 synthesizes the spectral sensitivity of the imaging device that captured the RGB image with the light source spectral distribution of the light source in the captured environment to obtain the first synthesis value. However, the data synthesis unit 12 may use only the imaging device spectral sensitivity or the light source spectral distribution as the first synthesis value, and synthesize it with the RGB image data to obtain the second synthesis value. In this configuration, when the machine learning model generation unit 13 trains the machine learning model, and when the data synthesis unit 12 synthesizes the model with a teacher image to create a second synthetic value, only the imaging device spectral sensitivity or the light source spectral distribution is used as the first synthetic value.

[0100] In the above-described embodiment, the machine learning model and the corrected machine learning model each estimate a spectral information image. However, each of the machine learning model and the corrected machine learning model may be configured to estimate the coefficients of basis functions that constitute the spectral reflectance curve obtained by principal component analysis or the like, rather than a spectral information image, as the distribution information to be estimated.

[0101] FIG. 9 is a conceptual diagram illustrating the estimation of coefficients of basis functions that form the curve of spectral reflectance using each of the machine learning model and the corrected machine learning model. That is, the configuration may be such that coefficients r1 to rn to be multiplied by basis functions b1 to bn, which represent a composite spectral reflectance curve (a curve showing the correspondence between wavelength and intensity) extracted by principal component analysis, are estimated. In this configuration, when the machine learning model is trained, the machine learning model generation unit 13 does not output the spectral reflectance as a data set of training data, but uses the known coefficients r1 to rn as the estimated output.

[0102] The machine learning model generation unit 13 then multiplies each of the estimated coefficients r1 to rn by each of the basis functions b1 to bn, and then synthesizes the basis functions b1 to bn to find a curve of spectral reflectance, and finds the loss L based on the spectral error from the known spectral reflectance and the RGB error in the RGB image, and continues learning until the above-mentioned predetermined conditions are met. Furthermore, the first spectral reflectance estimating unit 141 outputs the coefficients r1 to rn as primary spectral information to the second spectral reflectance estimating unit 142 using the trained machine learning model.

[0103] As a result, the corrected machine learning model uses coefficients r1 to rn as primary spectral information as input data, and outputs coefficients r1 to rn as corrected spectral information as estimated values. The second spectral reflectance estimation unit 142 obtains a spectral reflectance curve by combining basis functions b1 to bn multiplied by the estimated coefficients r1 to rn, obtains a reconstructed RGB image using the spectral reflectances, obtains a loss L based on the RGB error between the reconstructed RGB image and the RGB image used for the estimation, and outputs corrected spectral information that satisfies the above-mentioned predetermined conditions as spectral information.

[0104] Furthermore, in this embodiment, if the imaging device spectral sensitivity of the imaging device that captured the RGB image is known but the light source spectral distribution of the imaged environment is unknown, a machine learning model for estimating the light source spectral distribution may be generated as a machine learning model for estimating the light source spectral distribution. In this configuration, when capturing an RGB image for which a spectral information image is to be estimated, a color chart with known spectral reflectances (color components R, G, and B) is captured together with the subject. Similarly, an RGB image is captured together with the subject as a teacher image. Furthermore, when training the machine learning model for estimating a light source spectral distribution, the machine learning model generation unit 13 trains the machine learning model for estimating a light source spectral distribution by using the estimated output data as the light source spectral distribution and the input data as the image area of ​​the color chart in the teacher image, the data of the known color components R, G, and B of the color chart, and the spectral sensitivity of the imaging device.

[0105] When estimating the light source spectral distribution from the RGB image, the first spectral reflectance estimation unit 141 performs a synthesis process on the image area of ​​the color chart, data on the known color components R, G, and B of the color chart, and the imaging device spectral sensitivity, and inputs the data to a trained light source spectral distribution machine learning model to estimate the light source spectral distribution. As a result, the first spectral reflectance estimation unit 141 combines the data of the RGB image, the imaging device spectral sensitivity, and the estimated light source spectral distribution, inputs the combined data to the trained machine learning model, and estimates a spectral information image. As an example, a method for estimating the light source spectral power distribution using a machine learning model has been described, but any method other than using the above-mentioned machine learning model may be used to estimate the light source spectral power distribution.

[0106] Furthermore, in this embodiment, when the light source spectral distribution of the environment in which an RGB image is captured is known but the imaging device spectral sensitivity of the imaging device that captured the image is unknown, a machine learning model for estimating imaging device spectral sensitivity may be generated as a machine learning model for estimating the imaging device spectral sensitivity. In this configuration, when capturing an RGB image for which a spectral information image is to be estimated, a color chart with known spectral reflectances (color components R, G, and B) is captured together with the subject. Similarly, an RGB image is captured together with the subject as a teacher image. Furthermore, when training the machine learning model for estimating the spectral sensitivity of an imaging device, the machine learning model generation unit 13 trains the machine learning model for estimating the spectral sensitivity of an imaging device using estimated output data as the spectral sensitivity of the imaging device and input data as the image area of ​​the color chart in the teacher image, data of the known color components R, G, and B of the color chart, and the spectral distribution of the light source.

[0107] When estimating the imaging device spectral sensitivity from the RGB image, the first spectral reflectance estimation unit 141 performs a synthesis process on the image area of ​​the color chart, data on the known color components R, G, and B of the color chart, and the light source spectral distribution, and inputs the data to a trained machine learning model for estimating the imaging device spectral sensitivity, thereby estimating the imaging device spectral sensitivity. As a result, the first spectral reflectance estimation unit 141 combines the data of the RGB image, the light source spectral distribution, and the estimated image capture device spectral sensitivity, inputs the combined data to the trained machine learning model, and estimates a spectral information image. As an example, a method for estimating the spectral sensitivity of an imaging device using a machine learning model has been described, but any method other than using the above-mentioned machine learning model may be used to estimate the spectral sensitivity of an imaging device.

[0108] <Second embodiment> A second embodiment of the present invention will be described below with reference to the drawings. The spectral information image estimation device according to the second embodiment has the same configuration as that of the first embodiment. On the other hand, the machine learning model in the second embodiment has a different configuration from that in the first embodiment. Hereinafter, only the operations of the spectral information image estimation device according to the second embodiment that are different from those of the first embodiment will be described.

[0109] The machine learning model in the first embodiment is configured such that the image capture device spectral sensitivity and the light source spectral distribution are each input to the input layer as a data set of input data. However, the machine learning model in the second embodiment is configured such that the image capture device spectral sensitivity and the light source spectral distribution are input to the intermediate layer as data sets of input data. FIG. 10 is a conceptual diagram illustrating input of data on the spectral sensitivity of the imaging device and the spectral distribution of the light source to the intermediate layer of the machine learning model in this embodiment.

[0110] In FIG. 10, the data synthesis unit 12 refers to the captured image data storage unit 17 and reads out data on the RGB image, the imaging device spectral sensitivity, and the light source spectral distribution from the captured image data table. Then, the data synthesis unit 12 outputs the RGB image to the first spectral reflectance estimation unit 141 in the original data format. As a result, the first spectral reflectance estimating unit 141 reads out the RGB image data from the captured image data table, and inputs the RGB image data to the input layer 320 of the machine learning model 300.

[0111] Furthermore, the data synthesis unit 12 synthesizes the data of the imaging device spectral sensitivity and the light source spectral distribution by concatenation, addition, product, etc., to generate a first synthesis value. The data synthesis unit 12 synthesizes the generated first synthetic value with output data from any one of the multiple intermediate layers 350 in the machine learning model 300 by processing such as concatenation, addition, and multiplication to generate a third synthetic value, and outputs the third synthetic value to the first spectral reflectance estimation unit 141.

[0112] Then, the first spectral reflectance estimation unit 141 inputs the third composite value supplied from the data synthesis unit 12 to the intermediate layer 350 in the next stage after the intermediate layer 350 that output the output data synthesized with the first composite value. In this embodiment, the intermediate layer that outputs the output data to be combined with the first combined value may be an intermediate layer in one stage, or may be an intermediate layer in multiple stages.

[0113] Furthermore, when combining the first combined value with the output of the intermediate layer, the number of dimensions of the first combined value (number of dimensions of data) may be converted, and then the dimensionally converted output of the intermediate layer and the first combined value may be combined. In this configuration, the data synthesis unit 12 has a dimension conversion neural network that increases or decreases the dimension of the synthesized first synthesis value. Then, the data synthesis unit 12 converts the number of dimensions of the synthesized first synthesis value using the dimension conversion neural network, and then outputs the first synthesis value to the first spectral reflectance estimation unit 141.

[0114] As described above, in this embodiment, similarly to the first embodiment, an RGB image, the imaging device spectral sensitivity of the imaging device that captured the RGB image, and the light source spectral distribution of the light source in the environment where the RGB image was captured are each used as a data set of input data, and the spectral information of the subject in the RGB image is estimated as a primary spectral information image using a machine learning model, a corrected spectral information image is estimated from the primary spectral information image using an adjusted machine learning model, a loss L is obtained from the RGB error between a reconstructed RGB image reconstructed from the corrected spectral information image and the estimated RGB image of the primary spectral information image, the coefficients of the corrected machine learning model are adjusted until the above-mentioned predetermined condition is satisfied, and the corrected spectral information image is re-estimated using the adjusted corrected machine learning model to obtain a spectral information image; in other words, a two-stage estimation is performed using each of the machine learning model and the corrected machine learning model to estimate the spectral information image. With this configuration, according to this embodiment, the primary spectral information image estimated by the machine learning model is used to generate a corrected spectral information image that reproduces a reproduced RGB image that is closer to each of the color components R, G, and B of the RGB image used to estimate the primary spectral information image. Therefore, even if the estimation accuracy of the primary spectral information image by the first spectral reflectance estimation unit 141 is lower than when a multiband camera is used due to a lack of learning data or the characteristics of the machine learning model, the second spectral reflectance estimation unit 142 corrects the primary spectral information image, so that the spectral reflectance of the object being imaged can be easily estimated from the RGB image (images of three bands, i.e., images of each channel of the R channel, G channel, and B channel) with the same level of accuracy as an image captured by a multiband camera.

[0115] Furthermore, according to this embodiment, the output of the intermediate layer is combined with the first combined value to generate a third combined value, which is used as the input to the next-stage intermediate layer. Therefore, the output of the intermediate layer from which the features of the RGB image have been extracted is combined with each of the imaging device spectral sensitivity and the light source spectral distribution to generate the third combined value. Therefore, compared to the first embodiment, the primary spectral information image is more accurately approximated to the RGB spectral reflectance of the subject, and the time required for the correction process of the corrected machine learning model is reduced, enabling the spectral information image to be generated more quickly.

[0116] In the above-described embodiment, the data synthesis unit 12 synthesizes the spectral sensitivity of the imaging device that captured the RGB image with the light source spectral distribution of the light source in the captured environment to obtain the first synthesis value. However, the data synthesis unit 12 may be configured to use only either the imaging device spectral sensitivity or the light source spectral distribution as the first synthesis value, synthesize it with output data from an intermediate layer of a machine learning model to which an RGB image is input as input, to obtain a third synthesis value, and use this as input to the intermediate layer of the next stage. In this configuration, when the machine learning model generation unit 13 trains the machine learning model, and when the data synthesis unit 12 synthesizes the data with the teacher image to create a third synthetic value, only the imaging device spectral sensitivity or the light source spectral distribution is used as the first synthetic value.

[0117] <Third embodiment> A third embodiment of the present invention will now be described with reference to the drawings. The spectral information image estimation device according to the third embodiment has the same configuration as that of the first embodiment. On the other hand, the machine learning model in the third embodiment has a different configuration from that in the first embodiment. Hereinafter, only the operations of the spectral information image estimation device according to the third embodiment that are different from those of the first embodiment will be described.

[0118] FIG. 11 is a conceptual diagram illustrating an example of the configuration of a machine learning model used by the first spectral reflectance estimating unit 141 in this embodiment. In FIG. 11, the machine learning model of this embodiment includes sub-machine learning models 101, 102, and 103, as well as an integrated machine learning model 110. Here, the sub-machine learning model 101 inputs data on the color component R in the pixel of the RGB image (image capture device color component R data) and the color component R in the spectral sensitivity of the image capture device (image capture device color component R sensitivity data), and outputs an estimated output regarding the color component R to the data synthesis unit 12.

[0119] Similarly, the sub-machine learning model 102 inputs data on the color component G in the pixel of the captured image (image capture device color component G data) and the color component G in the spectral sensitivity of the image capture device (image capture device color component G sensitivity data), and outputs an estimated output regarding the color component G to the data synthesis unit 12. The sub-machine learning model 102 receives data on color component B in the pixel of the captured image (image capture device color component B data) and color component B in the spectral sensitivity of the image capture device (image capture device color component B sensitivity data), and outputs an estimated output regarding color component B to the data synthesis unit 12.

[0120] The data synthesis unit 12 synthesizes the data of the color component R (image capture device color component R data) in the pixel of the captured image and the data of the color component R in the spectral sensitivity of the image capture device (image capture device color component R sensitivity data) by concatenation, addition, and multiplication to generate an input synthesis value SR, and outputs the input synthesis value SR to the first spectral reflectance estimation unit 141. Then, the first spectral reflectance estimation unit 141 inputs the input composite value SR supplied from the data synthesis unit 12 to the input layer of the sub machine learning model 101.

[0121] In addition, the data synthesis unit 12 synthesizes the data of the color component G (image capture device color component G data) in the pixel of the captured image and the data of the color component G in the spectral sensitivity of the image capture device (image capture device color component G sensitivity data) by concatenation, addition, and multiplication to generate an input synthesis value SG, and outputs the input synthesis value SG to the first spectral reflectance estimation unit 141. Then, the first spectral reflectance estimation unit 141 inputs the input composite value SG supplied from the data synthesis unit 12 to the input layer of the sub machine learning model 102.

[0122] Similarly, the data synthesis unit 12 synthesizes the data of color component B (image capture device color component B data) in the pixel of the captured image and the data of color component B in the spectral sensitivity of the image capture device (image capture device color component B sensitivity data) by concatenation, addition, and multiplication to generate an input synthesis value SB, and outputs the input synthesis value SB to the first spectral reflectance estimation unit 141. Then, the first spectral reflectance estimation unit 141 inputs the input combined value SB supplied from the data combination unit 12 to the input layer of the sub machine learning model 103.

[0123] In addition, the data synthesis unit 12 synthesizes the outputs of the sub-machine learning model 101, the sub-machine learning model 102, and the sub-machine learning model 103 with the light source spectral distribution by concatenation, addition, and multiplication to generate a fourth synthetic value, and outputs the fourth synthetic value to the first spectral reflectance estimation unit 141. In this case, the data synthesis unit 12 may be configured to use a dimension conversion neural network to convert the number of dimensions of the fourth synthesis value, and then output the fourth synthesis value with the converted number of dimensions to the first spectral reflectance estimation unit 141. Then, the first spectral reflectance estimation unit 141 inputs the fourth composite value supplied from the integrated machine learning model 110 to the integrated machine learning model 110. As a result, the integrated machine learning model 110 estimates and outputs a spectral information image corresponding to the fourth composite value.

[0124] In addition, in this embodiment, when learning a machine learning model, the machine learning model generation unit 13 adjusts the coefficients of the neural network functions of the sub-machine learning model 101, the sub-machine learning model 102, and the sub-machine learning model 103 and the integrated machine learning model 110. Here, each of sub-machine learning model 101, sub-machine learning model 102, and sub-machine learning model 103 has the same neural network configuration. For this reason, machine learning model generation unit 13 adjusts the coefficients of the same neural network function in sub-machine learning model 101, sub-machine learning model 102, and sub-machine learning model 103 with the same numerical values.

[0125] Furthermore, in this embodiment, in each of the sub-machine learning models 101, 102, and 103, instead of inputting data (component data) of the captured image color components (R, G, B) and sensitivity data of the image capture device color component sensitivities (R, G, B) to the respective input layers, the first spectral reflectance estimation unit 141 may input only the captured image color components to the input layer, and the data synthesis unit 12 may synthesize sensitivity data of the image capture device color component sensitivities with the output data of the intermediate layer, and the first spectral reflectance estimation unit 141 may input the data synthesized from the output data and sensitivity data to the next intermediate layer. In this case, the data synthesis unit 12 may be configured to use a dimension conversion neural network to convert the number of dimensions of the data (synthesized data) obtained by synthesizing the output data and the sensitivity data, and then output the synthesized data with the converted number of dimensions to the first spectral reflectance estimation unit 141.

[0126] Furthermore, in this embodiment, in each of the sub-machine learning models 101, 102, and 103, instead of inputting data on the captured image color components (R, G, B) and the image capture device color component sensitivity (R, G, B) data into the respective input layers, the first spectral reflectance estimation unit 141 may input the captured image color components into the input layer, and the data synthesis unit 12 may synthesize the component data of the captured image color components and the sensitivity data of the image capture device color component sensitivity with the output data of the intermediate layer, and the first spectral reflectance estimation unit 141 may input the data synthesized from the output data, component data, and sensitivity data into the next intermediate layer. In this case, the data synthesis unit 12 may be configured to use a dimension conversion neural network to convert the number of dimensions of the data (synthesized data) obtained by synthesizing the output data, component data, and sensitivity data, and then output the dimensionally converted synthesized data to the first spectral reflectance estimation unit 141.

[0127] Furthermore, in this embodiment, in each of the sub-machine learning models 101, 102, and 103, instead of inputting data on the captured image color components (R, G, B) and the image capture device color component sensitivity (R, G, B) data into the respective input layers, the first spectral reflectance estimation unit 141 may input only the captured image color components into the input layer, and the data synthesis unit 12 may synthesize the sensitivity data of the image capture device color component sensitivity and the distribution data of the light source spectral distribution with the output data of the intermediate layer, and the first spectral reflectance estimation unit 141 may input data (a fifth synthetic value as synthetic data) obtained by synthesizing the output data, sensitivity data, and distribution data into the next intermediate layer. In this case, the data synthesis unit 12 may be configured to use a dimension conversion neural network to convert the number of dimensions of the data (synthesized data) obtained by synthesizing the output data, sensitivity data, and distribution data, and then output the dimensionally converted synthesized data to the first spectral reflectance estimation unit 141.

[0128] Furthermore, in this embodiment, instead of inputting captured image color component (R, G, B) data and imaging device color component sensitivity (R, G, B) data to the input layer for each of sub-machine learning models 101, 102, and 103, the captured image color component (R, G, B) data, imaging device color component sensitivity (R, G, B) data, and light source spectral distribution data may be combined (concatenated, added, and multiplied) and input. At this time, the data synthesis unit 12 synthesizes the outputs of the sub-machine learning model 101, the sub-machine learning model 102, and the sub-machine learning model 103, and outputs the synthesized result to the first spectral reflectance estimation unit 141. Then, the first spectral reflectance estimation unit 141 uses the data supplied from the data synthesis unit 12 as an input to the integrated machine learning model.

[0129] Furthermore, in this embodiment, for each of the sub-machine learning models 101, 102, and 103, a first composite value obtained by combining (concatenating, adding, and multiplying) the captured image color component (R, G, B) data and the imaging device color component sensitivity (R, G, B) data may be input to the input layer, and data on the light source spectral distribution may be input to the intermediate layer of each of the sub-machine learning models 101, 102, and 103. At this time, the first spectral reflectance estimation unit 141 inputs the light source spectral distribution data supplied from the data synthesis unit 12 to the intermediate layers of each of the sub-machine learning models 101, 102, and 103.

[0130] As described above, in this embodiment, similarly to the first embodiment, an RGB image, the imaging device spectral sensitivity of the imaging device that captured the RGB image, and the light source spectral distribution of the light source in the environment where the RGB image was captured are each used as a data set of input data, and the spectral information of the subject in the RGB image is estimated as a primary spectral information image using a machine learning model, a corrected spectral information image is estimated from the primary spectral information image using an adjusted machine learning model, a loss L is obtained from the RGB error between a reconstructed RGB image reconstructed from the corrected spectral information image and the estimated RGB image of the primary spectral information image, the coefficients of the corrected machine learning model are adjusted until the above-mentioned predetermined condition is satisfied, and the corrected spectral information image is re-estimated using the adjusted corrected machine learning model to obtain a spectral information image; in other words, a two-stage estimation is performed using each of the machine learning model and the corrected machine learning model to estimate the spectral information image. With this configuration, according to this embodiment, the primary spectral information image estimated by the machine learning model is used to generate a corrected spectral information image that reproduces a reproduced RGB image that is closer to each of the color components R, G, and B of the RGB image used to estimate the primary spectral information image. Therefore, even if the estimation accuracy of the primary spectral information image by the first spectral reflectance estimation unit 141 is lower than when a multiband camera is used due to a lack of learning data or the characteristics of the machine learning model, the second spectral reflectance estimation unit 142 corrects the primary spectral information image, so that the spectral reflectance of the object being imaged can be easily estimated from the RGB image (images of three bands, i.e., images of each channel of the R channel, G channel, and B channel) with the same level of accuracy as an image captured by a multiband camera.

[0131] Furthermore, according to this embodiment, a sub-machine learning model is provided for each color component of an RGB image, and it is considered that the feature extraction process for each of the color components R, G, and B is similar. When training the machine learning model, the same functions of the neural network in each of the sub-machine learning models for each color component are adjusted to the same numerical values ​​at the same time, thereby enabling the training of the machine learning model to converge more quickly compared to when adjusting the coefficients of all functions without using sub-machine learning models.

[0132] In the above-described embodiment, the data synthesis unit 12 synthesizes the spectral sensitivity of the imaging device that captured the RGB image with the light source spectral distribution of the light source in the captured environment to obtain the first synthesis value. However, the data synthesis unit 12 may be configured to use only either the imaging device spectral sensitivity or the light source spectral distribution as the first synthesis value, synthesize it with output data of an intermediate layer of a machine learning model for each color component to which data for each color component of an RGB image is input as input, obtain a fifth synthesis value, and use it as input to the next intermediate layer of the machine learning model for each color component. In this configuration, when the machine learning model generation unit 13 trains the machine learning model, and when the data synthesis unit 12 synthesizes the data with the teacher image to create the fifth synthetic value, only the imaging device spectral sensitivity or the light source spectral distribution is used as the first synthetic value.

[0133] <Fourth embodiment> A fourth embodiment of the present invention will be described below with reference to the drawings. FIG. 12 is a block diagram showing an example of the configuration of a spectral information image estimation device according to the fourth embodiment of the present invention. 12, spectral information image estimation device 10A includes data input unit 11, spectral reflectance estimation unit 14A, spectral image output unit 15, captured image data storage unit 17, and spectral information image data storage unit 19. Note that the same components as those in the first embodiment are denoted by the same reference numerals. The spectral reflectance estimating unit 14A includes a first spectral reflectance estimating unit 141A and a second spectral reflectance estimating unit 142.

[0134] Unlike the first spectral reflectance estimating unit 141 in the first embodiment, the first spectral reflectance estimating unit 141A does not estimate the spectral reflectance using a machine learning model. The first spectral reflectance estimating unit 141A estimates the spectral reflectance by combining spectral reflectance basis vectors (basis functions), for example, as shown in FIG. Here, the spectral reflectance basis vectors are derived in advance from a spectral reflectance data group, which is a collection of multiple spectral reflectance data, using principal component analysis as spectral reflectance basis vectors for expressing the spectral reflectance data group. In this principal component analysis, spectral reflectance can be expressed by a combination of spectral reflectance basis vectors with fewer dimensions than the actual number of dimensions of the spectral reflectance. In other words, by using spectral reflectance basis vectors, the number of dimensions of the spectral reflectance to be estimated is reduced, thereby reducing the calculation load.

[0135] That is, the first spectral reflectance estimating unit 141A expresses the spectral reflectance spectrum s as a weighted sum of a plurality of spectral reflectance basis vectors B=[b1, b2, ...]T as shown in the following equation (7), and to perform this weighting, it estimates the spectral reflectance spectrum s by calculating a weighting coefficient r=[r1, r2, ...]T for each of the spectral reflectance basis vectors. Here, the spectral reflectance basis vectors are stored in advance as a set in the captured image data storage unit 17. Furthermore, the captured image data storage unit 17 may store a plurality of spectral reflectance basis vectors arranged in descending order of principal component scores in principal component analysis, and may read and use as many spectral reflectance basis vectors as necessary up to a predetermined order.

[0136] As already explained, FIG. 9 shows how spectral reflectance is expressed by combining spectral reflectance basis vectors. As shown in FIG. 9, the spectral reflectance spectrum s is obtained by multiplying each of the spectral reflectance basis vectors b1 to bn obtained by principal component analysis using the spectral reflectances of each of the specified objects by a weighting coefficient and combining the multiplication results. That is, the first spectral reflectance estimating unit 141A reads out a necessary number of spectral reflectance basis vectors b1 to bn from the captured image data storage unit 17. Then, the first spectral reflectance estimation unit 141A estimates the weighting coefficients r1 to rn of the extracted spectral reflectance basis vectors b1 to bn so as to obtain the spectral reflectance spectrum s that is the pixel value of each pixel in the reference multi-viewpoint image, and calculates the spectral reflectance spectrum s from the estimation result (described later).

[0137]

number

[0138] Returning to FIG. 12, the first spectral reflectance estimating unit 141A estimates each weighting coefficient r by solving the following equation (8) for the spectral reflectance spectrum expressed by the above equation (7). Then, the first spectral reflectance estimating unit 141A estimates the spectral reflectance spectrum s using the estimated weighting coefficient r according to equation (7). Here, the spectral reflectance spectrum s is expressed as a 31-dimensional vector obtained by sampling the wavelength range from 400 nm to 700 nm in 10 nm increments, for example.

[0139]

number

[0140] In the above formula (8), pm,k is the pixel value of the k-th channel of the m-th captured image. That is, in pm,k, m is a number indicating the captured image for each filter. Also, k indicates the channel of the color component in a portion of a multi-viewpoint image. For example, if a portion is indicated by pixel values ​​of color components R, G, and B, channel (band) 1 is color component R (R channel), channel 2 is color component G (G channel), and channel 3 is color component B (B channel). Also, pixels whose pixel values ​​are read from the captured image of each filter are extracted from the image data of each captured image of each filter as pixels corresponding to the captured image.

[0141] In addition, in equation (8), the matrix Cm,k indicates the spectral sensitivity of the kth channel (wavelength band of color component) of the captured image of the mth filter. The element cm,k(λ) in this Cm,k indicates the spectral sensitivity of the wavelength λ in the kth channel of the mth captured image. Furthermore, the spectral distribution l of the light source is already written in the captured image data table of the captured image data storage unit 17 as measured spectral distribution data. The first spectral reflectance estimating unit 141A reads out from the captured image data table in the captured image data storage unit 17 the light source spectral distribution of the light source in the environment in which the captured image of each filter was captured.

[0142] Here, the first spectral reflectance estimating unit 141A may be configured to select and read out a set of spectral reflectance basis vectors from the captured image data storage unit 17 in accordance with the number of imaging device spectral sensitivities in the captured image of each filter. Furthermore, when solving equation (8), the upper limit of the number of spectral reflectance basis vectors (principal components) n used to represent the spectral reflectance data (hereinafter referred to as the number of spectral reflectance basis vectors) is determined based on, for example, the number of spectral sensitivity groups that capture the captured image (such as the number of spectral sensitivity adjustments made by color-changing filters).

[0143] When using multi-viewpoint images with q spectral sensitivity groups, the number is determined by the relational expression "n (number of spectral reflectance basis vectors) = q (number of spectral sensitivity types, number of spectral sensitivity groups) × k (number of channels, hereinafter referred to as the number of channels)." In other words, if the number n of spectral reflectance basis vectors representing the spectral reflectance data is equal to or less than k times the number of channels of the number q of spectral sensitivity groups, it is possible to solve equation (8). As an example, when three (=Nc) color change filters with different spectral sensitivities are used and there are three channels, R, G, and B, the upper limit of the basis vectors is 9 (=3×3). Furthermore, each of the captured images using each color change filter is used in equation (8) as the m-th viewpoint image at the same viewpoint, and therefore needs to be captured at the same viewpoint. However, instead of the above-described estimation of spectral reflectance using a plurality of spectral sensitivities, any method for estimating the spectral reflectance of a subject from an RGB image may be used.

[0144] The first spectral reflectance estimating unit 141A outputs the calculated spectral information image of 31 bands of spectral reflectance to the second spectral reflectance estimating unit 142 as a primary spectral information image. Then, as described in the first embodiment, the second spectral reflectance estimating unit 142 inputs the primary spectral information image supplied from the first spectral reflectance estimating unit 141A into the corrected machine learning model. The second spectral reflectance estimating unit 142 reconstructs an RGB image from the output corrected spectral information image, and finds the loss L by comparing it with the RGB image of any one of the color-changing filters.

[0145] When reproducing an RGB image from the corrected spectral information image, the second spectral reflectance estimation unit 142 reproduces the RGB image using the spectral sensitivity of the imaging device due to the color change filter when the RGB image to be compared from the corrected spectral information image was captured, and the light source spectral distribution of the imaging environment. Then, the second spectral reflectance estimating unit 142 adjusts the coefficients of the neural network function of the corrected machine learning model until the loss L becomes equal to or less than the reproduction threshold. When the loss L becomes equal to or less than the reproduction threshold, the second spectral reflectance estimating unit 142 outputs the corrected spectral information image at that time to the spectral image output unit 15 as the spectral information image. The spectral image output unit 15 writes the spectral information image supplied from the second spectral reflectance estimating unit 142 into the spectral information image data storage unit 19 for storage.

[0146] As described above, in this embodiment, a primary spectral information image is estimated from an RGB image using the spectral sensitivity of the imaging device and the spectral distribution of the light source using a predetermined method, a corrected spectral information image is estimated from the primary spectral information image using an adjusted machine learning model, a loss L is calculated from the RGB error between a reconstructed RGB image reconstructed from the corrected spectral information image and the estimated RGB image of the primary spectral information image, the coefficients of the corrected machine learning model are adjusted until the predetermined condition described above is satisfied, and a corrected spectral information image is re-estimated using the adjusted corrected machine learning model. In other words, a spectral information image is estimated by performing a two-stage estimation process in which a primary spectral information image is estimated from an RGB image using a predetermined method, and a corrected spectral information image is estimated from the primary spectral information image using the corrected machine learning model. With this configuration, according to this embodiment, a primary spectral information image estimated by a predetermined estimation method is used to generate a corrected spectral information image that reproduces a reproduced RGB image that is closer to each of the color components R, G, and B of the RGB image used to estimate the primary spectral information image. Therefore, even if the estimation accuracy of the primary spectral information image by the first spectral reflectance estimation unit 141A is lower than when a multiband camera is used due to the characteristics of the predetermined estimation method, the second spectral reflectance estimation unit 142 corrects the primary spectral information image, so that the spectral reflectance of the object being imaged can be easily estimated from the RGB image (images of three bands, i.e., images of each channel of the R channel, G channel, and B channel) with the same level of accuracy as an image captured by a multiband camera.

[0147] Note that a program for implementing the functions of the spectral information image estimation device 10 of FIG. 2 and the spectral information image estimation device 10A of FIG. 12 according to the present invention may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be loaded into a computer system and executed to estimate a spectral information image of a subject from an RGB image (captured image). Note that the term "computer system" as used herein includes hardware such as an operating system (OS) and peripheral devices. The term "computer system" also includes a World Wide Web (WWW) system equipped with a homepage provision environment (or display environment). The term "computer-readable recording medium" refers to portable media such as flexible disks, magneto-optical disks, read-only memories (ROMs), and compact disc-read-only memories (CD-ROMs), as well as storage devices such as hard disks built into computer systems. The term "computer-readable recording medium" also refers to devices that retain a program for a certain period of time, such as volatile memory (RAM (Random Access Memory)) within a computer system that acts as a server or client when the program is transmitted via a network such as the Internet or a communication line such as a telephone line.

[0148] The program may also be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by transmission waves in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may also be a program that realizes part of the above-mentioned functions. Furthermore, the program may be a so-called differential file (differential program) that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0149] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and design changes and the like are also included within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0150] 10, 10A...Spectral information image estimation device 11...Data input section 12...Data synthesis section 13...Machine learning model generation section 14,14A...Spectral reflectance estimation section 15...Spectral image output unit 16...Teacher image data storage unit 17...Captured image data storage unit 18...Machine learning model memory section 19...Spectral information image data storage unit 141,141A...First spectral reflectance estimation section 142...Second spectral reflectance estimation section

Claims

1. a spectral reflectance estimation unit that estimates a spectral information image indicating the spectral reflectance of a subject from an RGB image that is a captured image of the subject using a machine learning model; Equipped with further comprising a data synthesis unit that synthesizes input data; the data synthesis unit receives as input the captured image, an imaging device spectral sensitivity that is the spectral sensitivity of an imaging device that captured the captured image, and a light source spectral distribution that is the spectral distribution of a light source in an environment where the captured image is captured, and synthesizes the input data; The spectral reflectance estimation unit inputs the data synthesized by the data synthesis unit into the machine learning model to estimate the spectral information image. A spectral information image estimation device characterized by:

2. further comprising a machine learning model generation unit that generates the machine learning model; The machine learning model generation unit A teacher image, which is an image with known spectral reflectance, the imaging spectral sensitivity of the imaging device that captured the teacher image, and the light source spectral distribution of the environment in which the teacher image was captured are used as teacher data, and a value calculated from the error between the known spectral information of the teacher image and the spectral information of the spectral information image estimated by the machine learning model is used as a loss, and the machine learning model is trained.

2. The spectral information image estimation device according to claim 1.

3. The loss is The spectral information estimated by the machine learning model includes spectral information of the image, the imaging spectral sensitivity of the imaging device that captured the teacher image, and a reproduction error between the reproduced captured image and the teacher image, which is calculated from the light source spectral distribution of the environment in which the teacher image was captured.

3. The spectral information image estimation device according to claim 2.

4. The loss includes a difference in spectral reflectance between adjacent wavelengths in the estimated spectral information image.

4. The spectral information image estimation device according to claim 2 or 3.

5. The data synthesis unit: When the captured image, the imaging device spectral sensitivity, and the light source spectral power distribution are input to the machine learning model, one of the imaging device spectral sensitivity and the light source spectral power distribution, or a result of summing, multiplying, or concatenating the imaging device spectral sensitivity and the light source spectral power distribution, is set as a first composite value, and a second composite value is obtained by summing, multiplying, or concatenating the first composite value and the captured image.

5. The spectral information image estimation device according to claim 1, wherein the spectral information image estimation device is a spectral information image estimation device.

6. The data synthesis unit: When the captured image, the imaging device spectral sensitivity, and the light source spectral power distribution are input to the machine learning model, one of the imaging device spectral sensitivity and the light source spectral power distribution, or the sum, product, or concatenation of the imaging device spectral sensitivity and the light source spectral power distribution, is used as a first composite value, and a third composite value is generated by adding, producting, or concatenating the first composite value and output data of any intermediate layer of the machine learning model, and the third composite value is used as input data for an intermediate layer next to the intermediate layer.

5. The spectral information image estimation device according to claim 1, wherein the spectral information image estimation device is a spectral information image estimation device.

7. The data synthesis unit Using a dimension conversion neural network, the number of dimensions of the array of the number of data of the third composite value is converted.

7. The spectral information image estimation device according to claim 6.

8. The data synthesis unit The first composite value is combined with output data of each of the intermediate layers in a plurality of stages of the intermediate layers forming the machine learning model, and each of the third composite values ​​is used as input data of the intermediate layer in the next stage.

8. The spectral information image estimation device according to claim 6 or 7.

9. the machine learning model includes sub-machine learning models corresponding to the R channel, the G channel, and the B channel of a captured image, and an integrated machine learning model that uses each of the output data of the sub-machine learning models as an input value; The input of the sub-machine learning model is image data of the captured image corresponding to each of the channels and sensitivity data of the spectral sensitivity of the imaging device corresponding to each of the channels, a data synthesis unit that generates input data for the machine learning model, generating a fourth synthesis value by adding, multiplying, or concatenating output data of the sub-machine learning models; The spectral reflectance estimation unit The fourth composite value is input to the integrated machine learning model.

5. The spectral information image estimation device according to claim 1, wherein the spectral information image estimation device is a spectral information image estimation device.

10. The spectral reflectance estimation unit inputs the light source spectral distribution to the sub machine learning model or the integrated machine learning model.

10. The spectral information image estimation device according to claim 9.

11. The data synthesis unit Image data of each channel in the captured image and sensitivity data in the spectral sensitivity of the imaging device corresponding to each channel, or a fifth composite value obtained by adding, multiplying, or concatenating the image data and the sensitivity data of each channel in the spectral sensitivity of the imaging device, are input to each of the sub-machine learning models.

11. The spectral information image estimation device according to claim 9 or 10.

12. inputting image data of each channel of the corresponding sub-machine learning model in the captured image into each of the sub-machine learning models; The data synthesis unit Sensitivity data corresponding to each channel in the spectral sensitivity of the imaging device, or a fifth composite value obtained by adding, multiplying, or concatenating the image data and the sensitivity data corresponding to each channel in the spectral sensitivity of the imaging device, is input to each of the intermediate layers constituting the sub-machine learning model.

11. The spectral information image estimation device according to claim 9 or 10.

13. inputting image data of each channel of the corresponding sub-machine learning model in the captured image into each of the sub-machine learning models; The data synthesis unit The number of dimensions of an array of the number of data items is converted, and the sensitivity data corresponding to each of the channels in the spectral sensitivity of the imaging device, or a fifth composite value obtained by summing, multiplying, or concatenating the image data, the light source spectral distribution, and the sensitivity data corresponding to each of the channels in the spectral sensitivity of the imaging device, is input to each of the intermediate layers constituting the sub-machine learning model.

11. The spectral information image estimation device according to claim 9 or 10.

14. a machine learning model generation unit that generates the machine learning model, each of the sub-machine learning models has a similar configuration; The machine learning model generation unit The coefficients of the functions in the neural network of each of the sub-machine learning models are changed by the same value, and the sub-machine learning models are trained.

14. The spectral information image estimation device according to claim 9, wherein the spectral information image estimation device is a spectral information image estimation device.

15. the spectral reflectance estimating unit includes a first spectral reflectance estimating unit and a second spectral reflectance estimating unit, The first spectral reflectance estimating unit the captured image and at least one of the spectral sensitivity of the imaging device and the spectral distribution of the light source are input, and a primary spectral information image is estimated; The second spectral reflectance estimating unit The primary spectral information image estimated by the first spectral reflectance estimation unit, the captured image, the imaging device spectral sensitivity, and the light source spectral distribution are input, and the spectral information image is estimated.

3. The spectral information image estimation device according to claim 2.

16. The second spectral reflectance estimating unit A value calculated from a reproduction error between the captured image and a reproduced captured image calculated from the spectral information image estimated from the primary spectral information image is set as a loss, and the spectral information image is corrected so that the loss satisfies a predetermined condition.

16. The spectral information image estimation device according to claim 15.

17. The first spectral reflectance estimating unit The machine learning model is composed of a neural network, a machine learning model generation unit that generates the machine learning model, The coefficients of the function in the neural network are calculated, and the neural network is considered to have been trained.

17. The spectral information image estimation device according to claim 16.

18. a spectral reflectance estimation step in which a spectral reflectance estimation unit estimates a spectral information image indicating the spectral reflectance of the subject from an RGB image that is a captured image of the subject using a machine learning model; a data synthesis process in which a data synthesis unit synthesizes input data; Including, the data synthesis unit receives as input the captured image, an imaging device spectral sensitivity that is the spectral sensitivity of an imaging device that captured the captured image, and a light source spectral distribution that is the spectral distribution of a light source in an environment where the captured image is captured, and synthesizes the input data; The spectral reflectance estimation unit The data synthesized by the data synthesis unit is input to the machine learning model, and the spectral information image is estimated. A spectral information image estimation method comprising:

19. Computer, a spectral reflectance estimation means for estimating a spectral information image indicating the spectral reflectance of a subject from an RGB image that is a captured image of the subject using a machine learning model; data synthesis means for synthesizing input data; It functions as the data synthesis means receives as input the captured image, an imaging device spectral sensitivity which is the spectral sensitivity of the imaging device that captured the captured image, and a light source spectral distribution which is the spectral distribution of a light source in an environment where the captured image is captured, and synthesizes the input data; The spectral reflectance estimation means The data synthesized by the data synthesis means is input to the machine learning model, and the spectral information image is estimated. program.

Images