Estimation device, estimation system, estimation method, and program

Abstract

To estimate a spectral reflectivity of a subject by using an image captured under a light source in which a characteristic is unknown.SOLUTION: An estimation device comprises: a first image acquisition unit which acquires a first captured image in which a target object is captured in a first imaging environment irradiated by a first light source in which a characteristic is unknown; a second image acquisition unit which acquires a second captured image in which the target object is captured by a camera in a second imaging environment irradiated by a second light source in which a characteristic is known and the first light source; and a spectral reflectivity estimation unit which sets the first captured image, the second captured image, light source information, and camera information as input, and estimates a spectral reflectivity of the target object by using a machine learning model.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an estimation device, an estimation system, an estimation method, and a program. [Background technology]

[0002] There is a technology for estimating the spectral reflectance of a subject from an image. Compared to measuring the spectral reflectance using an expensive measuring device such as a two-dimensional radiance meter, this technology can estimate the spectral reflectance of a subject at low cost (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2023-040854 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, when estimating the spectral reflectance of a subject from an image, it is necessary to input the spectral distribution of the light source used in the imaging environment in which the image was captured. In other words, an image captured under a light source with a known spectral distribution must be used, and an image captured under a light source with unknown characteristics such as natural light cannot be used. For this reason, for example, a darkroom must be prepared and imaging must be performed in the darkroom under a light source with a known spectral distribution, which is a time-consuming task. In addition, objects that are too large to fit in a darkroom or objects that cannot be moved cannot be imaged in a darkroom, which limits the objects whose spectral reflectance can be estimated.

[0005] The present invention has been made in light of the above-mentioned problems, and has an object to provide an estimation device, an estimation system, an estimation method, and a program capable of estimating the spectral reflectance of a subject by using an image captured under a light source whose characteristics are unknown. [Means for solving the problem]

[0006] The estimation device of the present invention includes a first image acquisition unit that acquires a first captured image obtained by capturing an object with a camera in a first imaging environment in which the object is irradiated with a first light source having unknown characteristics; a second image acquisition unit that acquires a second captured image obtained by capturing the object with the camera in a second imaging environment in which the object is irradiated with a second light source having known characteristics and the first light source; a first training image obtained by capturing a training object with a camera in the first imaging environment, a second training image obtained by capturing the training object with the camera in the second imaging environment, and a training dataset in which light source information, which is a characteristic of the second light source, camera information regarding the camera, and the spectral reflectance of the training object are associated with each other; and a spectral reflectance estimation unit that estimates the spectral reflectance of the object using the first captured image, the second captured image, the light source information, the camera information, and the machine learning model.

[0007] The estimation system of the present invention includes the estimation device described above, and the camera that transmits the first captured image and the second captured image to the estimation device.

[0008] The estimation method of the present invention is an estimation method performed by a computer having a machine learning model that learns a correspondence between an image and a spectral reflectance of a subject using a first training image captured by a camera in a first imaging environment in which a training object having known spectral reflectance is irradiated with a first light source having unknown characteristics, a second training image captured by the camera of the training object in a second imaging environment in which the training object is irradiated with a second light source having known characteristics and the first light source, and a training dataset in which light source information that is a characteristic of the second light source, camera information about the camera, and the spectral reflectance of the training object are associated with each other, wherein a first image acquisition unit acquires a first captured image of a target object captured by the camera in the first imaging environment, a second image acquisition unit acquires a second captured image of the target object captured by the camera in the second imaging environment, and a spectral reflectance estimation unit estimates the spectral reflectance of the target object using the first captured image, the second captured image, the light source information, the camera information, and the machine learning model.

[0009] The program of the present invention is a program for causing a computer having a machine learning model that has learned a correspondence between an image and the spectral reflectance of a subject using a first training image captured by a camera in a first imaging environment in which a training object having known spectral reflectance is irradiated with a first light source having unknown characteristics, a second training image captured by the camera of the training object in a second imaging environment in which the training object is irradiated with a second light source having known characteristics and the first light source, and a training dataset in which light source information that is a characteristic of the second light source, camera information regarding the camera, and the spectral reflectance of the training object are associated with each other, to obtain a first captured image of a target object captured by the camera in the first imaging environment, and a second captured image of the target object captured by the camera in the second imaging environment, and to estimate the spectral reflectance of the target object using the first captured image, the second captured image, the light source information, the camera information, and the machine learning model. Effect of the Invention

[0010] According to the present invention, it is possible to estimate the spectral reflectance of a subject using an image captured under a light source with unknown characteristics. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of an estimation system 1 common to the embodiments. [Diagram 2] FIG. 2 is a diagram for explaining a process performed by the estimation system 1 common to the embodiments. [Diagram 3] 2 is a diagram for explaining a process performed by the estimation device 10 of the first embodiment. FIG. [Figure 4] FIG. 4 is a diagram for explaining a process performed by the estimation device 10 of the second embodiment. [Diagram 5] FIG. 11 is a diagram for explaining a process performed by the estimation device 10 of the third embodiment. [Figure 6] FIG. 13 is a diagram for explaining a process performed by the estimation device 10 of the fourth embodiment. [Figure 7] FIG. 13 is a diagram for explaining a process performed by the estimation device 10 of the fourth embodiment. [Figure 8] FIG. 11 is a diagram for explaining a process performed by the estimation device 10 according to the first modified example of the second embodiment. [Figure 9] FIG. 11 is a diagram for explaining a process performed by the estimation device 10 according to Modification 2 of the second embodiment. [Figure 10] 4 is a flowchart showing the flow of processing performed by the estimation device 10 common to the embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an estimation system 1 and an estimation device 10 according to an embodiment will be described with reference to the drawings.

[0013] <Regarding Estimation System 1 Common to the Embodiments> The estimation system 1 will be described with reference to Fig. 1 and Fig. 2. Fig. 1 is a block diagram showing an example of the configuration of the estimation system 1 common to the embodiments. Fig. 2 is a diagram for explaining the process performed by the estimation system 1 common to the embodiments.

[0014] As shown in Fig. 1, the estimation system 1 includes an estimation device 10, an imaging device 20, and a display device 30. The imaging device 20 is a camera that captures an image of a target object T. The estimation device 10 is a computer that estimates the spectral reflectance of the target object T from an image. The display device 30 displays a spectral reflectance image BG that is an image showing the spectral reflectance of the target object T based on the spectral reflectance estimated by the estimation device 10. That is, the estimation target in the estimation system 1 is the spectral reflectance image BG of the target object T.

[0015] Here, as shown in equation (1), the RGB value (gradient of each color component) of each pixel in an image is expressed as the integral of the product of the spectral reflectance, the light source spectral distribution, and the camera spectral sensitivity over a specified wavelength range (for example, a wavelength range from 400 nm to 700 nm). Spectral reflectance is information indicating the ratio of light reflected by a target object for a given wavelength, for example, for each of 31 bands of wavelengths corresponding to wavelength intervals in 10 nm increments in the wavelength range from 400 nm to 700 nm. The light source spectral distribution is information indicating the intensity of light contained in a light source L for each of 31 bands of wavelengths corresponding to wavelength intervals of 10 nm increments in a wavelength range from 400 nm to 700 nm. The camera spectral sensitivity is information indicating the intensity of light received by the camera for a given wavelength, for example, for each of 31 bands of wavelengths corresponding to wavelength divisions in 10 nm increments in the wavelength range from 400 nm to 700 nm.

[0016]

number

[0017] In the estimation system 1, the spectral reflectance of the target object T is estimated based on a captured image G of the target object T, and the estimation result is displayed as a spectral reflectance image BG. The spectral reflectance image BG is an image in which the spectral reflectance is mapped to the pixels of the captured image of the target object T.

[0018] As shown in FIG. 2, in the estimation system 1, an image capturing device 20 captures images of a target object T in two imaging environments SK1 and SK2. In the imaging environment SK1, an image of the target object T is captured in an environment illuminated by a light source L1. The light source L1 is a light source whose characteristics are unknown, such as sunlight or a light source whose spectral distribution has not been measured. The image of the target object T captured in the imaging environment SK1 is the captured image G1. In the imaging environment SK2, an image of the target object T is captured in an environment illuminated by light sources L1 and L2. The light source L2 is a light source with known characteristics, for example, a strobe light source provided in the imaging device 20. An image of the target object T captured in the imaging environment SK2 is a captured image G2. For example, when the light source L2 is a strobe light source of the imaging device 20, the imaging device 20 first captures an image G1 by capturing an image of the target object T in an environment (imaging environment SK1) where the strobe light source is not emitted. Next, the imaging device 20 captures an image G2 by capturing an image of the target object T in an environment (imaging environment SK2) where the strobe light source is emitted. Alternatively, when the light source L2 is a standard light source (for example, standard light D65 defined by the CIE (International Commission on Illumination) or JIS (Japanese Industrial Standards)), the imaging device 20 first captures an image G1 by capturing an image of the target object T in an environment (imaging environment SK1) where the standard light source is not emitted. Next, the imaging device 20 captures an image G2 by capturing an image of the target object T in an environment (imaging environment SK2) where the standard light source is emitted.

[0019] The estimation system 1 estimates the spectral reflectance of the target object T using the captured images G1 and G2 and a machine learning model NN. In the estimation system 1, the captured images G1 and G2, and, although omitted in the example of this figure, light source information that is a characteristic of the light source L2 and camera information related to the imaging device 20 are input to the machine learning model NN, and an estimation result that estimates the spectral reflectance of the target object T is output from the machine learning model NN. The estimation result here is information that makes it possible to generate a spectral reflectance image BG, and is information in which the spectral reflectance for each predetermined wavelength is indicated for the pixels of the image in which the target object T is captured.

[0020] The machine learning model NN here is a trained model that estimates the spectral reflectance of the target object T from two images (captured image G1 and captured image G2). As the machine learning model NN, for example, a neural network, deep learning, etc. can be used. The machine learning model NN is a model that has learned a learning data set in which two learning images of a learning object with known spectral reflectance are associated with the spectral reflectance of the learning object. Here, the two learning images used in the learning data set are an image of the learning object captured in the imaging environment SK1 (learning image GG1), an image of the learning object captured in the imaging environment SK2 (learning image GG2), light source information of the light source L2, and camera information of the imaging device 20. By using such a learning data set, the machine learning model NN learns the correspondence between the learning image GG1 corresponding to the imaging environment SK1, the learning image GG2 corresponding to the imaging environment SK2, the light source information of the light source L2, the camera information of the imaging device 20, and the spectral reflectance of the learning object. By learning this correspondence, the machine learning model NN becomes able to estimate the spectral reflectance of the target object T from two unlearned images (captured image G1 and captured image G2), the light source information of the light source L2, and the camera information of the imaging device 20.

[0021] <Regarding the Estimation Device 10 Common to the Embodiments> The estimation device 10 will be described with reference to Fig. 1. As shown in Fig. 1, the estimation device 10 includes a communication unit 100, a first image acquisition unit 101, a second image acquisition unit 102, a light source information acquisition unit 103, a camera information acquisition unit 104, an imaging control unit 105, a data processing unit 106, a spectral information estimation unit 107, a spectral image output unit 108, a captured image storage unit 109, a light source information storage unit 110, a camera information storage unit 111, a machine learning model storage unit 112, and a spectral information storage unit 113.

[0022] The functional units (first image acquisition unit 101, second image acquisition unit 102, light source information acquisition unit 103, camera information acquisition unit 104, imaging control unit 105, data processing unit 106, spectral information estimation unit 107, and spectral image output unit 108) that enable the estimation device 10 to realize its functions are realized by executing a program stored in a memory unit of the estimation device 10.

[0023] The configuration for storing information by the estimation device 10 (including the captured image memory unit 109, the light source information memory unit 110, the camera information memory unit 111, the machine learning model memory unit 112, and the spectral information memory unit 113) is composed of a storage medium such as a hard disk drive (HDD), flash memory, an electrically erasable programmable read only memory (EEPROM), a random access read / write memory (RAM), a read only memory (ROM), or a combination of these.

[0024] The communication unit 100 communicates with the imaging device 20 and the display device 30. The communication unit 100 acquires image information of a captured image G1 from the imaging device 20, and outputs the acquired image information to a first image acquisition unit 101. The communication unit 100 acquires image information of a captured image G2 from the imaging device 20, and outputs the acquired image information to a second image acquisition unit 102.

[0025] The communication unit 100 may employ any communication method depending on the connection with the communication partners, the imaging device 20 and the display device 30. For example, when the estimation device 10 is communicably connected to the imaging device 20 and the display device 30 via a communication network such as the Internet, the communication unit 100 communicates via the communication network. When the estimation device 10 is connected to the imaging device 20 and the display device 30 via a wired cable, the communication unit 100 communicates via the priority cable.

[0026] The first image acquisition unit 101 acquires image information of the captured image G1 captured by the imaging device 20 via the communication unit 100. The first image acquisition unit 101 may acquire image information of the captured image G1 supplied from an external device or an external memory via the communication unit 100 or an external interface, etc. The first image acquisition unit 101 writes the acquired image information of the captured image G1 in the captured image storage unit 109 to store it.

[0027] The second image acquisition unit 102, like the first image acquisition unit 101, acquires image information of the captured image G2 captured by the imaging device 20 via the communication unit 100. The second image acquisition unit 102 may acquire image information of the captured image G2 supplied from an external device or an external memory via the communication unit 100 or an external interface, etc. The second image acquisition unit 102 writes the acquired image information of the captured image G2 in the captured image storage unit 109 to store it.

[0028] The light source information acquisition unit 103 acquires light source information of the light source L2. The light source information here is, for example, light source distribution information of the light source L2. For example, when the light source L2 is a strobe light source of the imaging device 20, the light source information acquisition unit 103 acquires light source information of the strobe light source provided in the imaging device 20 via the communication unit 100. The light source information acquisition unit 103 may acquire light source information of the strobe light source posted on the website of the imaging device 20, for example, via the communication unit 100 and the Internet. Alternatively, when the light source L2 is a standard light source, the light source information acquisition unit 103 may acquire light source information of the standard light source published on a website such as CIE via the communication unit 100 and the Internet. Alternatively, if the light source L2 is a light source whose spectral distribution has been measured using a measuring instrument such as a spectrophotometer, the light source information as a measurement value supplied from the measuring instrument or an external memory may be acquired via the communication unit 100 or an external interface, etc. The light source information acquisition unit 103 writes the acquired light source information into the light source information storage unit 110 for storage.

[0029] The camera information acquisition unit 104 acquires camera information of the imaging device 20. The camera information here is information indicating the camera spectral sensitivity of the imaging device 20. For example, the camera information acquisition unit 104 acquires the camera information of the imaging device 20 posted on the website of the imaging device 20 via the communication unit 100 and the Internet. The camera information acquisition unit 104 writes the acquired camera information into the camera information storage unit 111 to store it.

[0030] The imaging control unit 105 controls imaging by the imaging device 20. The imaging control unit 105 controls the imaging device 20 so that the imaging device 20 captures an image G1 and an image G2. For example, when a strobe light source of the imaging device 20 is used as the light source L2, the imaging control unit 105 first causes the imaging device 20 to capture an image G1 by capturing an image of the target object T in an environment (imaging environment SK1) in which the strobe light source is not emitted. Next, the imaging control unit 105 causes the imaging device 20 to capture an image G2 by capturing an image of the target object T in an environment (imaging environment SK2) in which the strobe light source is emitted. Note that the estimation system 1 is not limited to using the captured images G1 and G2 captured under the control of the imaging control unit 105. It goes without saying that the estimation system 1 may use the captured images G1 and G2 captured without the control of the imaging control unit 105.

[0031] The data processing unit 106 generates information to be input to the machine learning model NN. The data processing unit 106 has a function related to preprocessing, such as processing information to be input to the machine learning model NN, so as to facilitate estimation processing by the machine learning model NN or to improve the estimation accuracy by the machine learning model NN. It should be noted that the estimation system 1 is not limited to using information processed by the data processing unit 106. It goes without saying that the estimation system 1 may use information that has not been processed by the data processing unit 106, for example, the captured images G1 and G2 themselves that have not been subjected to image processing.

[0032] For example, as in a fourth embodiment described later, in the case where a machine learning model NN4 (see FIG. 6) that inputs light source information of a light source L2 is used in the estimation system 1, if light source information is not stored in the light source information storage unit 110, the data processing unit 106 generates light source information. The data processing unit 106 generates light source information using, for example, a white image. The white image is a white image of a white object WB illuminated by the light source L2 captured by the imaging device 20 (see FIG. 7). A specific method in which the data processing unit 106 generates light source information using the white image will be described in detail in the fourth embodiment.

[0033] Furthermore, as in a second embodiment described later, when a machine learning model NN2 that inputs a differential captured image is used in the estimation system 1, the data processing unit 106 may generate a differential captured image based on the captured images G1 and G2. The differential captured image here is an image in which the pixel values ​​are values ​​corresponding to the difference between the pixel values ​​of the captured images G1 and G2, with the exposure conditions of the captured images being made equal. By generating a differential captured image, an image (differential captured image) captured of the target object T in an imaging environment where only the light source L2 is irradiated in a pseudo manner can be input to the machine learning model NN.

[0034] Furthermore, when a machine learning model NN2 that inputs a differential captured image is used in the estimation system 1 as in a second embodiment described later, the data processing unit 106 may perform image processing on the differential captured image to convert it into an image with less unevenness in brightness (referred to as a first uniform brightness image). A specific method by which the data processing unit 106 generates the first uniform brightness image will be described in detail in Modification 1 of the second embodiment, which will be described later.

[0035] Furthermore, when a machine learning model NN2 that inputs a differential captured image is used in the estimation system 1 as in a second embodiment described later, the data processing unit 106 may perform image processing on the differential captured image using a method different from that used for the first uniform brightness image, to convert the image into an image in which quantization error is suppressed and there is little unevenness in brightness (referred to as a second uniform brightness image). A specific method by which the data processing unit 106 generates the second uniform brightness image will be described in detail in Modification 2 of the second embodiment, which will be described later.

[0036] The spectral information estimation unit 107 estimates the spectral reflectance of the target object T. The spectral information estimation unit 107 inputs the captured image G1, the captured image G2, the light source information of the light source L2, and the camera information of the imaging device 20 to a machine learning model NN, and acquires information output from the machine learning model NN as an estimation result of the spectral reflectance of the target object T. The spectral information estimation unit 107 inputs, to the machine learning model NN, information corresponding to the contents of the learning data set that has been learned by the machine learning model NN. For example, in a first embodiment described later, a camera spectral sensitivity D100, a captured image G1 (first captured image D11), a captured image G2 (second captured image D12), and light source color information D130 are input to a machine learning model NN1 (see FIG. 3). In this case, the machine learning model NN1 is a trained model that has trained a training data set in which the camera spectral sensitivity, training images GG1 and GG2, light source color information, and the spectral reflectance of a training object are associated with each other. Similarly, for the other machine learning models NN (machine learning models NN2 to NN4), the spectral information estimation unit 107 inputs, to the machine learning models NN, information corresponding to the contents of the learning data set that the machine learning models NN (machine learning models NN2 to NN4) have learned. The spectral information estimation unit 107 writes and stores the estimation result of the spectral reflectance of the target object T in the spectral information storage unit 113 as spectral information.

[0037] The spectral image output unit 108 outputs image information of the spectral image (spectral reflectance image BG). The spectral image output unit 108 acquires the spectral reflectance for each predetermined wavelength at the pixel of the image of the target object T by referring to the spectral information storage unit 113. The spectral image output unit 108 outputs information in which the pixel is associated with the spectral reflectance for each predetermined wavelength as image information of the spectral image (spectral reflectance image BG). The spectral image output unit 108 outputs the image information of the spectral image (spectral reflectance image BG) to the display device 30 via the communication unit 100. The display device 30 receives image information (image information of the spectral image) from the estimation device 10, and displays the spectral image based on the received information.

[0038] The captured image storage unit 109 stores image information of the captured images G1 and G2, for example, RGB values ​​associated with each pixel in the image. The light source information storage unit 110 stores light source information indicating the characteristics of the light source L. The camera information storage unit 111 stores camera information. The machine learning model storage unit 112 stores information for constructing a machine learning model, for example, in the case of a neural network, information indicating the network configuration and the weights and bias values ​​at each edge. The spectral information storage unit 113 stores spectral information.

[0039] <Regarding the First Embodiment> The first embodiment will now be described with reference to Fig. 3. Fig. 3 is a diagram for explaining the process performed by the estimation device 10 of the first embodiment. In this embodiment, estimation is performed using a machine learning model NN1. The machine learning model NN1 is a trained model that estimates the spectral reflectance of a target object T using a camera spectral sensitivity D100, a first captured image D11, a second captured image D12, and light source color information D130 as inputs. In other words, the machine learning model NN1 is a trained model that has learned the correspondence between the image and the spectral reflectance of the subject using a training dataset in which the spectral reflectance of the training object is associated with the camera spectral sensitivity D100, training images GG1, training images GG2, and light source color information D130.

[0040] In this embodiment, camera information D10 is provided from the imaging device 20. For example, the imaging device 20 transmits the camera information D10 of the imaging device 20 to the estimation device 10. The estimation device 10 receives the camera information D10 from the imaging device 20. The estimation device 10 acquires the camera information D10 received from the imaging device 20 as camera spectral sensitivity D100. The camera spectral sensitivity D100 here is synonymous with the camera information D10. A first captured image D11 (captured image G1) is provided from the imaging device 20 and the target object T. The imaging device 20 captures an image of the target object T in an imaging environment SK1, and transmits the captured image G1 to the estimation device 10 as the first captured image D11. The estimation device 10 receives the first captured image D11 from the imaging device 20. A second captured image D12 (captured image G2) is provided from the imaging device 20, a target object T, and a light source (light source L2) whose characteristics are known. The imaging device 20 captures an image of the target object T in an imaging environment SK2, and transmits the captured image G2 to the estimation device 10 as the second captured image D12. The estimation device 10 receives the second captured image D12 from the imaging device 20. Light source information D13 is provided from a target object T and a light source (light source L2) whose characteristics are known. For example, the imaging device 20 transmits characteristics of a strobe light source of the imaging device 20 to the estimation device 10 as light source information D13. The estimation device 10 receives the light source information D13 from the imaging device 20. The estimation device 10 acquires light source color information D130 from the light source information D13 received from the imaging device 20. The light source color information D130 is information for specifying the light source spectral distribution of the light source L2, and is identification information for identifying a light source whose light source spectral distribution is specified, such as standard light D65 defined by CIE, JIS, etc. The light source color information D130 is an example of "light source information".

[0041] The estimation device 10 inputs input data to the machine learning model NN1. The input data here is a camera spectral sensitivity D100, a first captured image D11, a second captured image D12, and light source color information D130. The machine learning model NN1 estimates the spectral reflectance of the target object T based on the input data, and outputs the estimation result. The estimation device 10 outputs a spectral reflectance image BG based on the estimation result output from the machine learning model NN1.

[0042] <Regarding the Second Embodiment> The second embodiment will now be described with reference to Fig. 4. Fig. 4 is a diagram for explaining the process performed by the estimation device 10 of the second embodiment. In this embodiment, estimation is performed using a machine learning model NN2. The machine learning model NN2 is a trained model that estimates the spectral reflectance of the target object T using the camera spectral sensitivity D200, the differential captured image D240, and the light source color information D230 as inputs. That is, the machine learning model NN2 is a trained model that has learned the correspondence between the image and the spectral reflectance of the subject using a training data set in which the spectral reflectance of the training object is associated with the camera spectral sensitivity D200, the training image, and the light source color information D230. The training image here is an image generated according to formula (1) from the spectral reflectance of the training object, the camera spectral sensitivity D200, and the light source color information D230 (spectral distribution). Note that the camera spectral sensitivity D200 is equivalent to the camera spectral sensitivity D100. The light source color information D230 is equivalent to the light source color information D130.

[0043] In this embodiment, similarly to the first embodiment, the estimation device 10 acquires a camera spectral sensitivity D200, a first captured image D21 (captured image G1), a second captured image D22 (captured image G2), and light source color information D230.

[0044] The data processing unit 106 of the estimation device 10 corrects pixel values ​​of at least one of the first captured image D21 and the second captured image D22 so that the exposure conditions of the first captured image D21 and the second captured image D22 are equivalent. For example, the estimation device 10 corrects the exposure of the first captured image D21 (reference symbol D210). Alternatively, the estimation device 10 corrects the exposure of the second captured image D22 (reference symbol D220).

[0045] In general, when the imaging device 20 is provided with an exposure adjustment function, exposure adjustment is performed between imaging without emitting a strobe light source and imaging with emitting a strobe light source, and imaging is often performed under different exposure conditions such as shutter speed and aperture value. In this embodiment, in order to generate a differential captured image, the data processing unit 106 corrects the exposure conditions of both images (the first captured image D21 and the second captured image D22) so that they are equal.

[0046] For example, the data processing unit 106 acquires the set values ​​of the imaging parameters of both images (the first captured image D21 and the second captured image D22). The imaging parameters are variables related to exposure conditions such as shutter speed and aperture value. The data processing unit 106 determines which of the first captured image D21 and the second captured image D22 to correct based on the acquired set values ​​of the imaging parameters. For example, the data processing unit 106 calculates the exposure amounts (a value obtained by multiplying the amount of light received by a pixel per unit time in imaging by the light receiving time) of the first captured image D21 and the second captured image D22 based on the set values ​​of the imaging parameters, and determines to correct the image with the longer exposure time. The data processing unit 106 corrects the pixel values ​​so that the exposure amounts of both images are equal. For example, when the exposure amount of the first captured image D21 is half that of the second captured image D22, the data processing unit 106 corrects the exposure amounts of both images so that they are equal by halving the pixel values ​​of each pixel in the second captured image D22. In this way, the data processing unit 106 corrects the exposure conditions of both images (the first captured image D21 and the second captured image D22) so that they are equal to each other.

[0047] The data processing unit 106 calculates the difference in pixel values ​​between both images (the first captured image D21 and the second captured image D22) that have the same exposure conditions, and generates an image having pixel values ​​that correspond to the calculated difference as a differential captured image D240.

[0048] The estimation device 10 inputs input data to the machine learning model NN2. The input data here is the camera spectral sensitivity D200, the differential captured image D240, and the light source color information D230. The machine learning model NN2 estimates the spectral reflectance of the target object T based on the input data, and outputs the estimation result. The estimation device 10 outputs a spectral reflectance image BG based on the estimation result output from the machine learning model NN2.

[0049] <Regarding the Third Embodiment> The third embodiment will now be described with reference to Fig. 5. Fig. 5 is a diagram for explaining the process performed by the estimation device 10 of the third embodiment. In this embodiment, estimation is performed using a machine learning model NN3. The machine learning model NN3 is a trained model that estimates the spectral reflectance of the target object T using the camera spectral sensitivity D300, the first captured image D31, the second captured image D32, and the light source spectral distribution D330 as inputs. The light source spectral distribution D330 here is information indicating the intensity of light contained in the light source L2 for each of 31 bands of wavelengths corresponding to wavelength sections in 10 nm intervals in a wavelength range from 400 nm to 700 nm. The light source spectral distribution D330 is an example of "light source information." That is, the machine learning model NN3 is a trained model that has learned the correspondence between the image and the spectral reflectance of the subject using a training data set in which the spectral reflectance of the training object is associated with the camera spectral sensitivity D300, the training images GG1 and GG2, and the light source spectral distribution D330. The camera spectral sensitivity D300 is equivalent to the camera spectral sensitivity D100. The first captured image D31 is equivalent to the first captured image D11. The second captured image D32 is equivalent to the second captured image D12.

[0050] In this embodiment, similarly to the first embodiment, the estimation device 10 acquires a camera spectral sensitivity D300, a first captured image D31 (captured image G1), a second captured image D32 (captured image G2), and light source information D33. The light source information D33 is equivalent to the light source information D13.

[0051] The estimation device 10 acquires a light source spectral distribution D330 from the light source information D33 received from the imaging device 20.

[0052] The estimation device 10 inputs input data to the machine learning model NN3. The input data here is a camera spectral sensitivity D300, a first captured image D31, a second captured image D32, and a light source spectral distribution D330. The machine learning model NN3 estimates the spectral reflectance of the target object T based on the input data, and outputs the estimation result. The estimation device 10 outputs a spectral reflectance image BG based on the estimation result output from the machine learning model NN3.

[0053] <Regarding the Fourth Embodiment> The fourth embodiment will now be described with reference to Fig. 6. Fig. 6 is a diagram for explaining the process performed by the estimation device 10 of the fourth embodiment. In this embodiment, estimation is performed using a machine learning model NN4. The machine learning model NN4 is a trained model that estimates the spectral reflectance of the target object T using the camera spectral sensitivity D400, the first captured image D41, the second captured image D42, and the light source RGB information D430 as inputs. The light source RGB information D430 here is information indicating the characteristics of the light source L2 using RGB values ​​corresponding to the color of the light source. The light source RGB information D430 is an example of "light source information." That is, the machine learning model NN4 is a trained model that has learned the correspondence between the image and the spectral reflectance of the subject by using a training data set in which the spectral reflectance of the training object is associated with the camera spectral sensitivity D400, the training images GG1 and GG2, and the light source RGB information D430. Note that the camera spectral sensitivity D400 is equivalent to the camera spectral sensitivity D100. The first captured image D41 is equivalent to the first captured image D11. The second captured image D42 is equivalent to the second captured image D12.

[0054] In this embodiment, similarly to the first embodiment, the estimation device 10 acquires a camera spectral sensitivity D400, a first captured image D41 (captured image G1), a second captured image D42 (captured image G2), and light source information D43. The light source information D43 is equivalent to the light source information D13.

[0055] The data processing unit 106 of the estimation device 10 calculates light source RGB information D430 from the light source information D43 received from the imaging device 20. The data processing unit 106 of the estimation device 10 generates the light source RGB information D430 as light source information by using a white image.

[0056] Here, a method in which the data processing unit 106 generates the light source RGB information D430 as the light source information by using a white image will be described with reference to Fig. 7. Fig. 7 is a diagram for explaining the process performed by the estimation device 10 of the fourth embodiment. As shown in FIG. 7, the imaging device 20 captures a white image by taking a close-up shot of a white object WB (e.g., a white paper). The imaging device 20 first captures a first white image by taking a close-up shot of a white object WB (e.g., a white paper) in an imaging environment SK1. Next, the imaging device 20 captures a second white image by taking a close-up shot of a white object WB (e.g., a white paper) in an imaging environment SK2. Here, as a prerequisite for capturing the second white image, the second light source L2 is disposed close to the imaging device 20. Note that the imaging device 20 only needs to capture an image in which at least the color of the light irradiated by the light source L2 is reflected as the white image, and does not necessarily need to capture the image by close-up. The imaging device 20 transmits image information of the first white image and image information of the second white image to the estimation device 10. The estimation device 10 receives image information of the first white image and the second white image captured by the imaging device 20 via the communication unit 100, and generates light source RGB information D430 based on the received image information. The data processing unit 106 of the estimation device 10 processes the first white image and the second white image so that their exposure conditions are equivalent, calculates the difference between the pixel values ​​of both images (the first white image and the second white image), and generates an image having pixel values ​​corresponding to the calculated difference as a differential white image. Here, the method of processing so that the exposure conditions are equivalent can be the same as the method of processing so that the exposure conditions of the captured image G1 and the captured image G2 are equivalent when generating a differential captured image in the second embodiment. The data processing unit 106 sets the representative value of the pixel values ​​(RGB values) of the differential white image as the light source RGB information D430. The representative value here is a value calculated by statistically processing the pixel values ​​of each pixel in the differential white image, and is, for example, a simple arithmetic average value, a weighted average value, a mode value, a maximum value, a minimum value, etc.

[0057] Returning to the explanation of FIG. 6, the estimation device 10 inputs input data to the machine learning model NN4. The input data here is a camera spectral sensitivity D400, a first captured image D41, a second captured image D42, and light source RGB information D430. The machine learning model NN4 estimates the spectral reflectance of the target object T based on the input data, and outputs the estimation result. The estimation device 10 outputs a spectral reflectance image BG based on the estimation result output from the machine learning model NN4.

[0058] <Modification 1 of the second embodiment> Here, a first modified example of the second embodiment will be described. In this modified example, an image with less uneven brightness (first uniform brightness image) is generated by processing the differential captured image, and the generated image (first uniform brightness image) is input to the machine learning model NN2. Here, when making an estimation using the first uniform brightness image, an image that has not been processed into the first uniform brightness image (the difference learning image itself) may be used as the image to be trained by the machine learning model NN2 that makes the estimation, or an image that has been processed into the first uniform brightness image (the difference learning image processed into an image with less unevenness in brightness) may be used.

[0059] Here, a method for generating a first uniform brightness image by the data processing unit 106 will be described with reference to FIG. 8. FIG. 8 is a diagram for explaining a process performed by the estimation device 10 according to a modified example of the second embodiment. As shown in FIG. 8, the data processing unit 106 generates a first uniform brightness image D250 using a differential captured image D240, a light distribution LD of a second light source (light source L2), and depth information DM of a target object T. The first uniform brightness image here is an example of a "corrected image". The light distribution LD is a distribution indicating the direction and the intensity of light irradiated from a light source. The light distribution LD is, for example, a graph showing the relationship between an angle from a reference axis and light intensity.

[0060] The data processing unit 106 calculates the amount of light (attenuation) of the light from the light source L2 that attenuates before reaching the target object T, based on the light distribution of the light source L2 and the depth information DM of the target object T. The data processing unit 106 generates a first uniform brightness image D250 by correcting the brightness of each pixel in the differential captured image D240 based on the attenuation. The depth information DM here is the distance from the installation position of the light source to various points on the target object T that correspond to each pixel of the differential captured image D240. For example, when the light source L2 is a strobe light source of the imaging device 20, the depth information DM is the distance from the installation position of the imaging device 20 to various points on the target object T. In this case, the depth information DM can be calculated using a depth estimation method using, for example, the captured image G1 or G2. The depth information DM may also be calculated using the focus (focal length) of the captured image G1 or G2. Alternatively, the depth information DM may be a value obtained by measuring the distance from the installation position of the light source to various points on the target object T using a measuring device such as a tape measure.

[0061] <Modification 2 of the second embodiment> Here, a second modification of the second embodiment will be described. In this modification, an image is processed to generate an image (second uniform brightness image) in which quantization error is suppressed and brightness unevenness is small. Here, the image to be processed may be any of the captured images G1, G2, and the differential captured image D240. In addition, when the second uniform brightness image is used for estimation, an image that has not been processed into the second uniform brightness image (the learning image itself, or the differential learning image itself) may be used as an image to be trained by the machine learning model NN that performs the estimation, or an image processed into the second uniform brightness image may be used.

[0062] Here, a method for generating the second uniform lightness image by the data processing unit 106 will be described with reference to Fig. 9. Fig. 9 is a diagram for explaining the process performed by the estimation device 10 according to the modified example common to all the embodiments.

[0063] In the example of FIG. 9, a second uniform brightness image D260 is generated from a plurality of differential captured images D240 (differential captured images D240A, D240B, D240C, . . . ) having different exposure settings. As shown in FIG. 9, the data processing unit 106 acquires a plurality of image groups obtained by capturing images of the target object T with the exposure settings of the imaging device 20 being different from each other. The first image acquisition unit 101 acquires a first captured image group (captured images G1A, G1B, G1C, etc.) obtained by capturing images of the target object T with the exposure settings of the imaging device 20 being different from each other in an imaging environment SK1. The second image acquisition unit 102 acquires a second captured image group (captured images G2A, G2B, G2C, etc.) obtained by capturing images of the target object T with the exposure settings of the imaging device 20 being different from each other in an imaging environment SK2. The data processing unit 106 generates a differential captured image using a pair of captured images G1 and G2 having the same exposure condition, thereby generating a plurality of differential captured images D240 (differential captured images D240A, D240B, D240C, etc.) having different exposure amounts from each other.

[0064] The data processing unit 106 calculates the amount of light (attenuation amount) of the light source L2 attenuated before reaching the target object T and the amount of reflected light (reflected light amount) of the light source L2 received by the imaging device 20 based on the multiple differential captured images D240, the light distribution of the light source L2, the depth information DM of the target object T, and the normal information of the target object T. The data processing unit 106 generates a second uniform brightness image D260 by synthesizing the multiple differential captured images D240 based on the attenuation amount and the reflected light amount. The data processing unit 106 extracts a region corresponding to the first portion from the differential captured image D240 having a large exposure amount for the first portion having a large attenuation amount and a small reflected light amount. The data processing unit 106 extracts a region corresponding to the second portion from the differential captured image D240 having a small exposure amount for the second portion having a small attenuation amount and a large reflected light amount. The data processing unit 106 generates a synthetic image by synthesizing the extracted regions as the second uniform brightness image D260. By generating a composite image by combining multiple images with different exposure amounts according to the amount of attenuation and reflected light, an image with reduced quantization error and little uneven brightness, i.e., a second uniform brightness image D260, can be generated.

[0065] Here, a process flow common to the embodiments will be described with reference to FIG. 10. FIG. 10 is a flowchart showing a process flow performed by the estimation device 10 common to the embodiments. As shown in FIG. 10, the estimation device 10 acquires a captured image G1 (first captured image) (step S10). The estimation device 10 acquires a captured image G2 (second captured image) (step S11). The estimation device 10 acquires light source information (step S12). The estimation device 10 acquires camera information (step S13). The estimation device 10 processes input data to be input to the machine learning model NN as necessary (step S14). The estimation device 10 estimates spectral information (spectral reflectance of the target object T) (step S15). The estimation device 10 outputs a spectral reflectance image BG (step S16).

[0066] As described above, the estimation system 1 common to the embodiments includes the first image acquisition unit 101, the second image acquisition unit 102, the machine learning model NN, and the spectral information estimation unit 107. The first image acquisition unit 101 acquires a captured image G1 (first captured image). The captured image G1 is an image of a target object T captured by the imaging device 20 (camera) in an imaging environment SK1 (an imaging environment illuminated by a light source L1 (first light source) whose characteristics are unknown). The second image acquisition unit 102 acquires a captured image G2 (second captured image). The captured image G2 is an image of a target object T captured by the imaging device 20 (camera) in an imaging environment SK2 (an imaging environment illuminated by a light source L2 (second light source) whose characteristics are known and the light source L1). The machine learning model NN is a model that learns a correspondence relationship between an image and the spectral reflectance of a subject by using a learning dataset. The learning dataset is data in which a first learning image, a second learning image, light source information, camera information, and the spectral reflectance of a learning object are associated with each other. The first learning image is an image of a learning object whose spectral reflectance is known, captured by the imaging device 20 in an imaging environment SK1 (first imaging environment). The second learning image is an image of a learning object, captured by the imaging device 20 in an imaging environment SK2 (second imaging environment). The light source information is information indicating characteristics of the second light source. The camera information is information related to the camera. The spectral information estimation unit 107 estimates the spectral reflectance of the target object T using the captured image G1, the captured image G2, the light source information, the camera information, and the machine learning model NN. As a result, the estimation device 10 common to all the embodiments can use the captured image G1 to generate a spectral reflectance image BG of the target object T. In other words, the spectral reflectance of the subject can be estimated using an image captured under a light source with unknown characteristics, eliminating the need to prepare a special imaging environment such as a darkroom and eliminating any restrictions on the target object T.

[0067] In the estimation device 10 of the third embodiment, the light source information is information indicating the spectral distribution of the light source L2 (second light source). This allows the estimation device 10 of the third embodiment to estimate the spectral reflectance of the target object T using the light source spectral distribution. In the estimation device 10 of the second embodiment, the light source information is information indicating the color of the light source L2 (second light source) (for example, identification information such as standard light D65 defined by CIE, JIS, etc.). As a result, the estimation device 10 of the second embodiment can estimate the spectral reflectance of the target object T using the identification information of the light source color.

[0068] Moreover, the estimation device 10 of the fourth embodiment further includes a data processing unit 106 (preprocessing unit). The data processing unit 106 generates information to be input to the machine learning model NN. The data processing unit 106 generates, as light source information, information indicating the color of the light source L2 (for example, light source RGB information) calculated based on pixel values ​​of a white image obtained by capturing a white object illuminated by the light source L2 with the imaging device 20. This makes it possible to estimate the spectral reflectance of the target object T using the light source information generated from the white image even if the light source spectral distribution is unknown.

[0069] Furthermore, the estimation device 10 of the fourth embodiment generates a differential white image. The differential white image is an image in which pixel values ​​correspond to the difference between pixel values ​​of a first white image in which a white object WB is captured in an imaging environment SK1 and pixel values ​​of a second white image in which the white object WB is captured in an imaging environment SK2. The data processing unit 106 generates, as light source information, information indicating the color of the light source L2 calculated based on the differential white image (for example, light source RGB information). This makes it possible to estimate the spectral reflectance of the target object T using the light source information generated from the white image even if the light source spectral distribution is unknown.

[0070] In the estimation device 10 of the second embodiment, the data processing unit 106 generates a differential captured image. The spectral information estimation unit 107 estimates the spectral reflectance of the target object T using the differential captured image, light source information, camera information, and a machine learning model NN. The machine learning model in this case is the machine learning model NN2. The machine learning model NN2 is a model that learns the correspondence between the image and the spectral reflectance of the subject using a learning data set in which a differential learning image indicating the difference between the first learning image and the second learning image, light source information, camera information, and the spectral reflectance of the learning object are associated with each other. This makes it possible to estimate the spectral reflectance of the target object T using the differential captured image.

[0071] Moreover, in the estimation device 10 according to the modified example of the second embodiment, the data processing unit 106 generates a first uniform brightness image (an example of a corrected image). The first uniform brightness image is an image of the target object T captured in an imaging environment where the light source L2 is irradiated with uniform brightness. The data processing unit 106 generates the first uniform brightness image using the captured image G1, the captured image G2, the depth information DM of the target object T, and the light distribution of the light source L2. This makes it possible to accurately estimate the spectral reflectance of the target object T using the first uniform brightness image.

[0072] In the estimation device 10 according to the modified example of the second embodiment, the data processing unit 106 generates a second uniform brightness image (an example of a corrected image). The second uniform brightness image is an image of the target object T captured in an imaging environment in which quantization error is suppressed and the light source L2 is irradiated with uniform brightness. The first image acquisition unit 101 acquires a first captured image group in which the target object T is captured in the imaging environment SK1 with the exposure settings of the imaging device 20 set to different settings. The second image acquisition unit 102 acquires a second captured image group in which the target object T is captured in the imaging environment SK2 with the exposure settings of the imaging device 20 set to different settings. The data processing unit 106 generates a second uniform brightness image using the first captured image group, the second captured image group, the depth information DM of the target object T, the normal information of the target object, and the spectral distribution of the second light source. This makes it possible to accurately estimate the spectral reflectance of the target object T using the second uniform brightness image.

[0073] Furthermore, the estimation system 1 may be configured using the estimation device 10 common to the embodiments. The estimation system 1 includes the estimation device 10 and an imaging device 20. The imaging device 20 captures captured images G1 and G2, and transmits the captured images to the estimation device 10. The estimation device 10 estimates the spectral reflectance of the target object T using the images (captured images G1 and G2) received from the imaging device 20. In this way, the estimation system 1 can estimate the spectral reflectance of the subject using images captured under a light source with unknown characteristics.

[0074] The estimation system 1 and the estimation device 10 in the above-described embodiment may be realized in whole or in part by a computer. In that case, a program for realizing this function may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to realize the function. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer-readable recording medium" refers to a portable medium such as a flexible disk, an optical magnetic disk, a ROM, a CD-ROM, and a storage device such as a hard disk built into a computer system. The term "computer-readable recording medium" may also include a medium that dynamically holds a program for a short period of time, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, and a medium that holds a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in that case. The above-described program may be a program for realizing a part of the above-described function, or may be a program that can realize the above-described function in combination with a program already recorded in the computer system, or may be a program that is realized using a programmable logic device such as an FPGA. [Explanation of symbols]

[0075] 1. Estimation system 10…Estimation device 101…first image acquisition unit 102...second image acquisition unit 103...Light source information acquisition section 104…Camera information acquisition unit 105...imaging control unit 106...Data processing unit (preprocessing unit) 107...Spectral information estimation section (spectral reflectance estimation section) 108...Spectral image output unit 20...Imaging device (camera) 30…Display device

Claims

1. a first image acquisition unit that acquires a first captured image in which a target object is captured in a first imaging environment illuminated by a first light source having unknown characteristics; a second image acquisition unit that acquires a second captured image of the target object by a camera that captured the first captured image in a second imaging environment illuminated by a second light source and the first light source, the characteristics of which are known; a spectral reflectance estimation unit that estimates a spectral reflectance of the target object using a machine learning model, with the first captured image, the second captured image, light source information that is a characteristic of the second light source, and camera information related to the camera as input; An estimation device comprising:

2. The light source information is information indicating a spectral distribution of the second light source. The estimation device according to claim 1 .

3. The light source information is information indicating a color of the second light source. The estimation device according to claim 1 .

4. The spectral reflectance estimation unit further includes a preprocessing unit that generates information to be input to a machine learning model, The preprocessing unit generates, as the light source information, information indicating a color of the second light source calculated based on pixel values ​​of a white image obtained by capturing an image of a white object illuminated by the second light source with the camera. The estimation device according to claim 3 .

5. The spectral reflectance estimation unit further includes a preprocessing unit that generates information to be input to a machine learning model, the pre-processing unit generates a differential white image indicating a difference between pixel values ​​of a first white image obtained by capturing an image of a white object by the camera in the first imaging environment and pixel values ​​of a second white image obtained by capturing an image of a white object by the camera in the second imaging environment, and generates, as the light source information, information indicating a color of the second light source calculated based on the differential white image. The estimation device according to claim 3 .

6. The spectral reflectance estimating unit further includes a preprocessing unit that generates a differential captured image indicating a difference between two images, the spectral reflectance estimating unit estimates a spectral reflectance of the target object using the differential captured image generated by the pre-processing unit from the first captured image and the second captured image, the light source information, the camera information, and a machine learning model. The estimation device according to claim 1 .

7. The machine learning model is a trained model that has learned a correspondence relationship between the training captured image and the spectral reflectance of the training object by using a training dataset in which the spectral reflectance of a training object having a known spectral reflectance, the light source information of a light source having a known spectral distribution, the camera information of a training image capturing camera having a known spectral sensitivity, and a training captured image generated from the spectral reflectance of the training object, the spectral sensitivity of the training image capturing camera, and the spectral distribution of the light source are associated with each other. The estimation device according to claim 6.

8. The second light source is disposed proximate to the camera; The spectral reflectance estimation unit further includes a preprocessing unit that generates information to be input to a machine learning model, The spectral reflectance estimation unit generates a corrected image by the pre-processing unit using the first captured image, the second captured image, depth information of the target object, and the light distribution of the second light source to correct the image so as to be an image of the target object captured in an imaging environment in which the second light source is irradiated with uniform brightness, and estimates the spectral reflectance of the target object using a machine learning model with the corrected image, the light source information, and the camera information as inputs. The estimation device according to claim 1 .

9. The second light source is disposed proximate to the camera; The spectral reflectance estimation unit further includes a preprocessing unit that generates information to be input to a machine learning model, The first image acquisition unit acquires a first group of captured images of the target object in the first imaging environment with exposure settings of the camera different from each other, The second image acquisition unit acquires a second group of captured images of the target object in the second imaging environment with exposure settings of the camera different from each other, The spectral reflectance estimation unit generates, by the pre-processing unit, a corrected image, which is an image of the target object captured in an imaging environment where the target object is illuminated with the second light source at a uniform brightness, by using the first captured image group, the second captured image group, depth information of the target object, normal information of the target object, and a light distribution of the second light source, and corrects the image so as to suppress quantization error included in pixel values; and estimates the spectral reflectance of the target object using a machine learning model with the corrected image, the light source information, and the camera information as inputs. The estimation device according to claim 1 .

10. The estimation device according to claim 1 ; the camera that transmits the first captured image and the second captured image to the estimation device; An estimation system comprising:

11. A computer-implemented estimation method, comprising: a first image acquisition unit acquires a first captured image in which a target object is captured in a first imaging environment illuminated by a first light source having unknown characteristics; a second image acquisition unit acquires a second captured image of the target object by a camera that captured the first captured image in a second imaging environment illuminated with a second light source and the first light source, the characteristics of which are known; a spectral reflectance estimation unit receives the first captured image, the second captured image, light source information which is a characteristic of the second light source, and camera information related to the camera as input, and estimates a spectral reflectance of the target object using a machine learning model. Estimation method.

12. On the computer, acquiring a first captured image in which a target object is captured in a first imaging environment illuminated by a first light source having unknown characteristics; acquiring a second captured image of the target object by a camera that captured the first captured image in a second imaging environment illuminated with a second light source having known characteristics and the first light source; the first captured image, the second captured image, light source information which is a characteristic of the second light source, and camera information regarding the camera are input, and a machine learning model is used to estimate the spectral reflectance of the target object. program.

Images