Machine learning teacher data generation system, machine learning teacher data generation method and program

Abstract

To provide a machine learning teacher data generation system that adjusts present spectral information and generates teacher data of a machine learning model, as many as necessary for learning, in which spectral information such as a spectral reflectivity, a spectral distribution, and a spectral sensitivity is estimated from an image including information such as the spectral reflectivity of an RGB image, a spectral emissivity image, and the like.SOLUTION: A machine learning teacher data generation system of the present invention includes at least a basic spectral information storage unit that stores each of a spectral reflectivity of a target object constituting a picked-up image of the target object, a spectral sensitivity of an imaging apparatus of the picked-up image, and a spectral distribution of a light source in an environment where the picked-up image is imaged as spectral information, a data set specification input unit that accepts input of a specification indicating a relation between input data, to a machine learning model, that is generated from the spectral information, and output data that is estimated by the machine learning model with the input data, and a teacher data generation unit that adjusts each of the spectral information corresponding to the specification and generates teacher data including the input data and the output data.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a machine learning teacher data generation system, a machine learning teacher data generation method, and a program for generating teacher data used in training a machine learning model. [Background technology]

[0002] The main method of treatment for general outpatients is for the patient to visit a medical institution on a regular basis, undergo physical examinations using the medical and health equipment available at the institution, and have a doctor or other specialist assess the patient's condition based on the results of the examinations. In recent years, remote medical treatment in remote islands and rural areas has been increasing due to a shortage of doctors. Since patients cannot easily visit medical institutions in person, doctors at medical institutions examine patients based on images and videos taken of the patient (see, for example, Patent Documents 1 and 2).

[0003] Similarly, the same is true for patients receiving care at home, for whom the doctors are unable to visit frequently, and so the remote medical care mentioned above is being carried out. In addition, in the case of sudden illness or injury due to an accident, emergency transportation is required, but since every second counts, patients can be transported to the hospital at home, at the accident scene, or in the ambulance, and remote medical treatment similar to that described above is performed. If patients can be examined at home, at the scene of an accident, or during transport, it will be possible to anticipate tests to be performed after admission, which will help with early treatment tailored to the patient's condition.

[0004] However, when capturing an image using a normal digital camera, it is not possible to observe the accurate color of the affected area of ​​the patient due to ambient light such as lighting. For example, doctors usually examine patients in examination rooms at medical institutions, so if the environment is different, the color of the affected area may not match the color observed in the medical institution, and there is a risk that an accurate examination may not be possible.

[0005] Furthermore, in recent years, spectral reflectance has been estimated using machine learning models such as deep learning (for example, Patent Document 3). Therefore, in order to solve the above-mentioned problems, it is conceivable to use a machine learning model such as deep learning to estimate the spectral reflectance from an image of three color channels, namely, color components R, G, and B (hereinafter simply referred to as an RGB image), captured using a digital camera or the like. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent No. 3417235 [Patent Document 2] Patent No. 6001245 [Patent Document 3] JP 2020-102807 A Summary of the Invention [Problem to be solved by the invention]

[0007] However, in order to train a machine learning model to know what kind of output image should be estimated for an input image, a large amount of training data consisting of pairs of input and output images is required. Here, it is difficult to capture a sufficient number of spectral images for training a machine learning model using a spectroradiometer or the like.

[0008] It would be ideal if training data could be easily created by analyzing a printed chart, as in Patent Document 3, but depending on the object whose spectral reflectance is to be estimated, training data for estimating the spectral reflectance of, for example, a patient's affected area cannot be easily created. In other words, a large amount of training data is required to train a machine learning model, but it is not easy to prepare a large number of combinations of RGB images of the patient's affected area and the spectral reflectance or spectral images calculated from these RGB images. Therefore, if a machine learning model is trained using an insufficient amount of training data, the machine learning model will not be able to estimate a spectral image from an RGB image with high accuracy.

[0009] The present invention has been made in consideration of the above-mentioned circumstances, and provides a system, a method, and a program for generating teacher data for machine learning, which generate teacher data used in training a machine learning model that estimates spectral information such as spectral reflectance, the spectral distribution of a light source, and the spectral sensitivity of an imaging device from images containing spectral reflectance information such as RGB images and spectral emissivity images, by adjusting already-existing spectral information to the extent necessary for training the machine learning model. [Means for solving the problem]

[0010] In order to solve the above-mentioned problems, the machine learning teacher data generation system of the present invention is characterized in that it comprises: a basic spectral information storage unit in which at least the spectral reflectance of a target object that constitutes an image of the target object, the spectral sensitivity of an imaging device for the image, and the spectral distribution of the light source in the environment in which the image was taken are pre-written and stored as spectral information; a dataset specification input unit that inputs specifications indicating the relationship between input data to a machine learning model generated from the spectral information and output data estimated by the machine learning model from the input data; and a teacher data generation unit that adjusts each of the spectral information in accordance with the specifications to generate teacher data comprising the input data and the output data.

[0011] The machine learning teacher data generation system of the present invention is characterized in that the teacher data generation unit generates new spectroscopic information in the captured image that is not included in the spectroscopic information as the input data or the output data.

[0012] The system for generating teacher data for machine learning according to the present invention is characterized in that an image captured by the imaging device is generated as a teacher image as the teacher data from the spectroscopic information.

[0013] The machine learning teacher data generation system of the present invention further includes a light source spectral distribution memory unit in which adjustment spectral distributions, which are the spectral distributions of multiple types of light sources that adjust the spectral information, are written and stored in advance, and the teacher data generation unit generates the teacher captured image using the adjustment spectral distributions in addition to the spectral distributions in the spectral information.

[0014] The machine learning teacher data generation system of the present invention is characterized in that the teacher data generation unit synthesizes the spectral intensities for each wavelength between the adjustment spectral distributions at a predetermined ratio to generate a new composite spectral distribution, and uses the composite spectral distribution to generate the teacher image.

[0015] The machine learning teacher data generation system of the present invention further includes a light source spectral sensitivity memory unit in which adjustment spectral sensitivities, which are the spectral sensitivities of multiple types of imaging devices that adjust the spectral information, are pre-written and stored, and the teacher data generation unit generates the teacher captured image using the adjustment spectral sensitivities in addition to the spectral sensitivities in the spectral information.

[0016] The machine learning teacher data generation system of the present invention is characterized in that the teacher data generation unit synthesizes the spectral intensities for each wavelength between the adjustment spectral sensitivities at a predetermined ratio to generate a new composite spectral sensitivity, and uses the composite spectral sensitivity to generate the teacher image.

[0017] The machine learning teacher data generation system of the present invention further has a target object spectral characteristic memory unit in which, for each type of target object in the captured image, spectral components that have a high content ratio in the spectral reflectance of that type are pre-written and stored, and the teacher data generation unit changes the spectral reflectance of the target object using the adjusted spectral components, generates a new spectral reflectance, which is an adjustment spectral reflectance, and generates the teacher image.

[0018] The machine learning teacher data generation system of the present invention is characterized in that the teacher data generation unit generates the adjustment spectral reflectance by multiplying a spectral component by a predetermined coefficient and adding or subtracting the multiplication result from the spectral reflectance.

[0019] The machine learning teacher data generation system of the present invention is characterized in that the spectral components are a plurality of principal component spectral components obtained by principal component analysis of the spectral reflectance of each type of target object in the captured image, and the teacher data generation unit multiplies each of the principal component spectral components by a coefficient and adds each of the multiplication results of the principal component spectral components to generate the adjustment spectral reflectance.

[0020] The machine learning training data generation system of the present invention is characterized in that when the target object is skin of the human body, the adjustment spectral reflectance is generated using the spectral characteristics of at least melanin and hemoglobin as the spectral components.

[0021] The machine learning teacher data generation system of the present invention further has a missing information acquisition unit that acquires the missing information when information necessary for generating the captured image is missing from the spectroscopic information, and acquires the missing information based on other information that is different from the missing information contained in at least the spectroscopic information.

[0022] The machine learning teacher data generation system of the present invention is characterized in that the other information is complementary information used to complement the missing information in the spectral information, and identifies the spectral reflectance, the spectral sensitivity, and the spectral distribution in the spectral information.

[0023] The machine learning training data generation system of the present invention is characterized in that the spectroscopic information includes three-dimensional shape information that is information about the three-dimensional shape of the target object.

[0024] The machine learning teacher data generation system of the present invention is characterized in that the three-dimensional shape information is a multi-view image consisting of images of the same target object captured from multiple different viewpoints, and the teacher data generation unit generates the three-dimensional shape of the target object from the multi-view image.

[0025] The machine learning teacher data generation system of the present invention is characterized in that the teacher data generation unit generates a virtual image in which the three-dimensional shape is observed from a predetermined viewpoint, and an image having information on the spectral reflectance corresponding to the virtual image.

[0026] The machine learning teacher data generation system of the present invention is characterized in that, when the teacher data generation unit is supplied with information on whether or not shading is to be applied and the range to which the shading is to be applied from the dataset specification input unit, the teacher data generation unit applies shading to the range of the virtual captured image and an image having information on the spectral reflectance corresponding to the virtual captured image.

[0027] The machine learning teacher data generation system of the present invention is characterized in that when a request to generate the three-dimensional shape in which the type of voxel data is expressed by either spectral radiance, spectral reflectance, or RGB value is input, the dataset specification input unit instructs the teacher data generation unit to generate the three-dimensional shape composed of voxels of the requested type of data, and the teacher data generation unit generates the three-dimensional shape composed of voxels of the requested type of data.

[0028] The machine learning training data generation system of the present invention further includes a similar shape generation unit that generates a similar three-dimensional shape that is similar to the three-dimensional shape, and when a request to generate a similar three-dimensional shape that is similar to the three-dimensional shape of the target object is input, the dataset specification input unit instructs the similar shape generation unit to generate the similar three-dimensional shape, and the similar shape generation unit generates the similar three-dimensional shape that is similar to the three-dimensional shape by using the spectral reflectance of the three-dimensional shape of the target object as is.

[0029] The machine learning training data generation system of the present invention is characterized in that the target object is a predetermined part of the human body.

[0030] The machine learning teacher data generation system of the present invention further includes a computer graphics data generation unit that generates computer graphics data, and when a request to generate a pseudo target object using computer graphics data in addition to the target object is input, the dataset specification input unit instructs the computer graphics data generation unit to generate the pseudo target object, and the computer graphics data generation unit generates the pseudo target object using computer graphics to be synthesized into the captured image in which the target object is captured.

[0031] The machine learning teacher data generation system of the present invention is characterized in that the R, G, and B color components in the RGB image of the simulated target object are similar to the R, G, and B color components of the target object, respectively, and the spectral reflectance of the simulated target object is different from that of the target object.

[0032] The method for generating teacher data for machine learning of the present invention is characterized in that it includes a dataset specification input process for inputting specifications indicating the relationship between input data to a machine learning model, the input data being generated from spectral information in a basic spectral information storage unit in which at least the spectral reflectance of a target object constituting an image of the target object, the spectral sensitivity of an imaging device for the image, and the spectral distribution of a light source in the environment in which the image was taken are written and stored in advance as spectral information, and output data estimated by the machine learning model from the input data, and a teacher data generation process in which a teacher data generation unit adjusts each of the spectral information in accordance with the specifications to generate teacher data comprising the input data and the output data.

[0033] The program of the present invention is a program that causes a computer to function as a machine learning teacher data generation system that generates teacher data for a machine learning model from spectral information in a basic spectral information storage unit in which at least the spectral reflectance of a target object that constitutes an image of the target object, the spectral sensitivity of an imaging device for the image, and the spectral distribution of the light source in the environment in which the image was taken are written and stored in advance as spectral information, and causes the computer to function as a dataset specification input means that inputs specifications indicating the relationship between input data to the machine learning model generated from the spectral information and output data estimated by the machine learning model from the input data, and a teacher data generation means that adjusts each of the spectral information in accordance with the specifications to generate the teacher data comprising the input data and the output data. Effect of the Invention

[0034] As described above, according to the present invention, it is possible to provide a system, a method, and a program for generating teacher data for machine learning, which generate teacher data used for training a machine learning model that estimates spectral information such as spectral reflectance, the spectral distribution of a light source, and the spectral sensitivity of an imaging device from an image containing spectral reflectance information such as an RGB image or a spectral emissivity image, by adjusting already-existing spectral information in the amount required for training the machine learning model. [Brief description of the drawings]

[0035] [Figure 1] 1 is a diagram illustrating the relationship between input data of a machine learning model for which training data is generated in the present invention and output data estimated from the input data. FIG. [Diagram 2] FIG. 1 is a conceptual diagram showing an example of a machine learning model that outputs a spectral image as spectral reflectance information from an RGB image, or a spectral reflectance image. [Diagram 3] FIG. 1 is a block diagram showing an example of the configuration of a machine learning teacher data generation system according to a first embodiment of the present invention. [Figure 4]FIG. 13 is a diagram showing an example of the configuration of a spectral information table in which the spectral information stored in the basic spectral information storage unit 105 is described and which is used to generate a dataset of teacher data for a machine learning model to be learned. [Diagram 5] 1 is a flowchart showing an example of the operation of a teacher data generation process performed by the machine learning teacher data generation system 100 according to this embodiment. [Figure 6] FIG. 11 is a block diagram showing an example of the configuration for generating teacher data for machine learning according to a second embodiment of the present invention. [Figure 7] 10 is a diagram showing an example of the configuration of a light source spectral distribution table, which is a data table of light source spectral distributions written in advance in a light source spectral distribution storage unit 107. FIG. [Figure 8] 1 is a diagram showing an example of the configuration of an image capture device spectral distribution table, which is a data table of image capture device spectral distributions written in advance in an image capture device spectral sensitivity storage unit 108. FIG. [Figure 9] FIG. 11 is a conceptual diagram illustrating a process of generating teacher data by referring to a light source spectral distribution table and an imaging device spectral distribution table in the second embodiment. [Figure 10] FIG. 11 is a block diagram showing an example of the configuration of a machine learning teacher data generation system according to a third embodiment of the present invention. [Figure 11] FIG. 13 is a diagram showing an example of the configuration of a spectral component table, which is a data table of spectral components written in advance in a spectral component information storage unit 109 in the third embodiment. [Figure 12] FIG. 13 is a conceptual diagram illustrating a process of generating teacher data by referring to a spectral component table according to the third embodiment. [Figure 13] FIG. 13 is a block diagram showing an example of the configuration of a machine learning teacher data generation system according to a fourth embodiment of the present invention. [Figure 14] FIG. 13 is a diagram showing an example of the configuration of a second spectral information table written in advance in a fundamental spectral information storage unit 105 in the fifth embodiment of the present invention. [Figure 15] FIG. 13 is a block diagram showing an example of the configuration of a machine learning teacher data generation system according to a sixth embodiment of the present invention. [Figure 16] FIG. 13 is a block diagram showing an example of the configuration of a machine learning teacher data generation system according to a seventh embodiment of the present invention. [Figure 17] FIG. 13 is a conceptual diagram illustrating the process in which the teacher data generating unit 103 adds CG data to an RGB image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0036] The present invention generates training data as a data set consisting of input data and output data, which is used when training a machine learning model. A machine learning model receives input data and outputs output data estimated in response to the input data. For this reason, in order to use training data for a machine learning model, both the input data and the output data must be known.

[0037] FIG. 1 is a diagram illustrating the relationship between input data of a machine learning model for which training data is to be generated in the present invention and output data estimated from the input data. The relationship between the input data of a machine learning model for which training data is generated in the present invention and the output data estimated from the input data will be described below. That is, Fig. 1 shows the relationship between the input data and the output data in a data set of a machine learning model to be trained.

[0038] Figure 1(a) shows the formula that indicates that the RGB value (the numerical value of each of the color components R, G, and B) of each pixel in an RGB image can be calculated as the integral value of the multiplication result, for wavelengths (λ) from 400 nm to 700 nm, of the light source (spectral distribution), the spectral reflectance of the target object, and the spectral sensitivity of the imaging device. As can be seen from this formula, elements (information) that are not included in the input data cannot be estimated as output data.

[0039] Figure 1(b) is a table showing the relationship between input data and output data for each record in the machine learning model being trained. As a data set of training data, a combination of known data of this input data and output data is required.

[0040] For example, in line 201, an RGB image (RGB) is used as input data, and one or more of the following are output data: spectral reflectance (reflectance), radiance (spectral radiance), light source spectral distribution (light source), and imaging device spectral sensitivity (sensitivity). The RGB image is, for example, when the imaging device is a digital camera, and each pixel is composed of RGB values.

[0041] That is, when an RGB image is input as input data, if the machine learning model estimates, for example, the spectral reflectance of an image pickup target and outputs the result as output data, the relationship between the RGB image and the spectral reflectance must be known as training data. For this reason, the training data is a data set of the RGB image and the known spectral reflectance of the RGB image (measured in advance or calculated by a general calculation).

[0042] FIG. 2 is a conceptual diagram showing an example of a machine learning model that outputs a spectral image as spectral reflectance information from an RGB image, or a spectral reflectance image. The machine learning model 300, for example, takes an RGB image 301 as input data and outputs spectral images 302 of each of predetermined wavelengths of spectral reflectance from the RGB image 301 (for example, wavelengths in 10 nm intervals in the wavelength range from 400 nm to 700 nm). Furthermore, the machine learning model 300 uses an RGB image 301 as input data to estimate a spectral distribution of a light source (light source spectral distribution) 303 of the environment in which the RGB image 301 was captured. Furthermore, the machine learning model 300 estimates the spectral sensitivity (imaging device spectral sensitivity) 304 of the imaging device that captured the RGB image 301 using the RGB image 301 as input data.

[0043] Returning to Fig. 1, line 202 shows an example of a dataset of training data for a machine learning model in which a spectral radiance image (radiance) is input data, and either or both of a spectral reflectance (reflectance) and a light source spectral distribution (light source) are output data. For example, the imaging device of the spectral radiance image is a hyperspectral camera, and each pixel is composed of a spectral radiance value (spectral distribution x spectral reflectance). That is, when a spectral radiance image is input as input data, if the machine learning model estimates, for example, the spectral radiance of an imaged object as output data, the relationship between the spectral radiance image and the spectral reflectance must be known as training data. Therefore, the training data is a data set of the spectral radiance image and the known spectral reflectance of this spectral radiance image.

[0044] Also, in line 203, as an example of a plurality of input data, two pieces of data, an RGB image and the spectral distribution of the light source (light source) when the RGB image was captured, are used as input data, and one or more of the spectral reflectance (reflectance), radiance (spectral radiance), and imaging device spectral sensitivity (sensitivity) are used as output data. The RGB image is, for example, a case where the imaging device is a digital camera, and each pixel is composed of an RGB value. That is, when an RGB image is input as input data, if the machine learning model estimates, for example, the spectral reflectance of an image capture target and uses this as output data, the relationship between the RGB image and the spectral reflectance must be known as training data. For this reason, the training data is a data set of the RGB image and the known spectral reflectance of this RGB image.

[0045] <First embodiment> Hereinafter, a first embodiment of the present invention will be described with reference to the drawings. FIG. 3 is a block diagram showing an example of the configuration of a machine learning teacher data generation system according to the first embodiment of the present invention. In FIG. 1, the machine learning teacher data generation system 100 includes a data input unit 101, a dataset specification input unit 102, a teacher data generation unit 103, a teacher data output unit 104, a basic spectroscopic information storage unit 105, and a teacher data storage unit 106.

[0046] The data input unit 101 writes and stores in the basic spectroscopic information storage unit 105 the spectroscopic information that is supplied from an external device and that is the basis for generating training data for training a machine learning model.

[0047] Here, an example of spectral information for generating a data set of input data for a machine learning model for which training data is to be generated in the first embodiment of the present invention and output data estimated from the input data will be described. FIG. 4 is a diagram showing an example of the configuration of a spectral information table in which the spectral information stored in the basic spectral information storage unit 105 is written, and which is used to generate a dataset of teacher data for a machine learning model to be learned.

[0048] For each record, the spectral information table has columns for spectral information identification information, spectral reflectance image index (or an image including at least information on spectral reflectance and light source spectral sensitivity, such as an RGB image or a spectral radiance image), imaging device spectral sensitivity index, and light source spectral distribution index. The spectral information identification information is identification information for identifying each captured image from which each of the spectral information (spectral reflectance, imaging device spectral sensitivity, light source spectral distribution) has been extracted. Here, the imaging device spectral sensitivity is the spectral information in the case where the captured image from which the spectral information has been extracted is captured by a digital camera. In addition, it is information that is not included in the spectral radiance image captured by the hyperspectral camera.

[0049] The spectral reflectance information index is information, such as an address, indicating an area in which a spectral reflectance image previously obtained corresponding to a captured image is written. The spectral reflectance image is, for example, the already-described spectral image, or an image having information on the spectral reflectance for each pixel. In addition, the spectral reflectance information may be an image such as an RGB image or a spectral radiance image that includes at least the spectral reflectance and the spectral distribution of the light source (including the spectral sensitivity of the imaging device if the imaging device is a digital camera).

[0050] The light source spectral distribution index is information, for example an address, indicating an area in which spectral distribution information (light source spectral distribution, curve graph showing the correspondence between wavelength and spectral radiance, etc.) of the light source in the environment in which the captured image was captured is written. The imaging device spectral sensitivity index is information, for example an address, indicating an area in which spectral sensitivity information (imaging device spectral sensitivity, a curve graph showing the correspondence between wavelength and sensitivity, etc.) of the imaging device (in the case of a digital camera) that captured the captured image is written.

[0051] Returning to FIG. 3 , when specification information indicating the specifications of teacher data to be generated for training a machine learning model is input, the dataset specification input unit 102 outputs the specifications to the teacher data generation unit 103. The specification information of the training data includes at least the number of training data to be generated for training the machine learning model, i.e., the number of data sets and the type of input data, the type of output data, the size of the image to be generated (number of vertical pixels x number of horizontal pixels), and the method of image processing of the spectroscopic information.

[0052] Here, the types of input data include, for example, the types of input data shown in FIG. 1(b), such as an RGB image and a spectral radiance image. Similarly, the types of output data include, for example, a spectral reflectance image (or a spectral image), a light source spectral distribution of a light source in the environment in which the captured image was taken, the spectral sensitivity of an imaging device that took the captured image, and a spectral radiance image.

[0053] Here, the image processing method indicates each of the image processing methods used when generating images to be used as multiple teaching data based on each of the data of one spectral information. Image processing includes, for example, changing the brightness of each pixel, enlarging the image of the captured object, reducing the image of the captured object, cropping (cutting out a part of the image), rotating the image of the captured object, removing background other than the captured object, and increasing the resolution (i.e., adjusting the resolution by adjusting the number of pixels in the vertical and horizontal directions of the image).

[0054] The teacher data generating unit 103 generates teacher data for a data set consisting of input data and output data corresponding to the specification information supplied from the data set specification input unit 102 from each of the spectral information stored in the basic spectral information storage unit 105. Here, for example, the generation of teacher data by teacher data generation unit 103 will be described in the following when the specification information specifies an RGB image as input data, a spectral reflectance image as output data, 10,000 data sets, image rotation and image reduction as image processing methods, and a specified image size, and the basic spectral information storage unit 105 stores the spectral information of 10 captured images of the skin of a human hand.

[0055] The teacher data generating unit 103 generates 1000 pieces of teacher data from each of the 10 pieces of spectroscopic information in the basic spectroscopic information storage unit 105, thereby generating a total of 10000 sets of teacher data. At this time, when a spectral reflectance image is stored as spectral reflectance information in the spectral information, the teacher data generation unit 103 rotates the spectral reflectance image (the image area in which the hand is imaged) at a predetermined angle to generate 100 processed images, each having a different image angle.

[0056] Then, the teacher data generation unit 103 reduces each of the 100 processed images to 10 different sizes to generate 1000 processed images as teacher spectral reflectance images, which are spectral reflectance images to be used as teacher data. The teacher data generation unit 103 performs the process of generating a teacher spectral reflectance image for each of the 10 pieces of spectral information, so a total of 10000 images are generated. In this way, teacher spectral reflectance images are generated as output data in the data set of teacher data.

[0057] In addition, the teacher data generation unit 103 multiplies each of the teacher spectral reflectance images generated for each piece of spectral information by the light source spectral distribution and imaging device spectral sensitivity of the respective pieces of spectral information (Figure 1(a)), and generates an RGB image corresponding to each teacher spectral reflectance, with the spectral reflectance for each pixel being the RGB value. Here, the teacher data generation unit 103 converts each of the 10,000 teacher spectral reflectance images into an RGB image, and generates 10,000 RGB images as teacher RGB images to be used as teacher data corresponding to each of the teacher spectral reflectance images.

[0058] As a result, a teacher RGB image (RGB image) as input data in a data set of teacher data is newly generated as information that did not exist in the spectral information. Then, the teacher data generation unit 103 writes and stores the teacher spectral reflectance image and the teacher RGB image generated from the teacher spectral reflectance image in the teacher data storage unit 106 as a teacher data dataset (10,000 datasets).

[0059] As described above, the teacher data generation unit 103 converts the known spectroscopic information in the basic spectroscopic information storage unit 105 into spectroscopic information of different captured images by image processing, and generates teacher data of a data set consisting of each of the input data and output data shown in Figure 1(b). Furthermore, when the input data is a spectral radiance image and the output data is a spectral reflectance image, the teacher data generating unit 103 generates a teacher spectral reflectance image as output data for the number of data sets by image processing from the spectral information.

[0060] Then, the teacher data generation unit 103 multiplies the generated teacher spectral reflectance image by the light source spectral distribution to generate a teacher spectral radiance image, which is a spectral radiance image in which each pixel has a spectral radiance value, and writes and stores it in the teacher data memory unit 106.

[0061] In response to a request from an external device, the teacher data output unit 104 reads out the created teacher data written in the teacher data storage unit 106 and outputs it to the external device.

[0062] 5 is a flowchart showing an example of the operation of the training data generation process by the machine learning training data generation system 100 according to this embodiment. In the following explanation, an example will be described in which training data is created for a data set in which the input data is RGB image data and the output data is a spectral reflectance image.

[0063] Step S101: When specification information indicating the specifications of the teacher data to be generated for training a machine learning model is input from an input means not shown, the dataset specification input unit 102 outputs the specification information to the teacher data generation unit 103.

[0064] Step S102: The teacher data generating unit 103 refers to the spectral information table in the fundamental spectral information storage unit 105 and acquires the amount of spectral information stored in the fundamental spectral information storage unit 105 (checks the amount of spectral information).

[0065] Step S103: The teacher data generating unit 103 reads the spectral reflectance images in the spectral information one by one from the upper records in the spectral information table in the basic spectral information storage unit 105 .

[0066] Step S104: Then, the teacher data generating unit 103 divides the number of data sets to be generated in the specification information by the number of pieces of spectral information acquired, to obtain the number of teacher data to be generated from each piece of spectral information (calculation of the number of teacher data).

[0067] Step S105: The teacher data generation unit 103 changes the spectral reflectance image (changes the spectral information) using the image processing method (or combination of image processing methods) in the specification information, and generates a teacher spectral reflectance image as output data, the number of which corresponds to the number of teacher data generated from each piece of spectral information.

[0068] Step S106: The teacher data generating unit 103 determines whether or not changes to all of the spectral information in the spectral information table of the basic spectral information storage unit 105 have been completed. At this time, if the teacher data generating unit 103 has completed changing all of the spectral information in the spectral information table of the fundamental spectral information storage unit 105, the processing proceeds to step S107. On the other hand, if the teacher data generating unit 103 has not finished changing all of the spectral information in the spectral information table of the fundamental spectral information storage unit 105, the process proceeds to step S105.

[0069] Step S107: The teacher data generating unit 103 multiplies each of the generated teacher spectral reflectance images by the light source spectral distribution and imaging device spectral sensitivity corresponding to each teacher spectral reflectance image, to generate an RGB image as input data.

[0070] Step S108: The teacher data generating unit 103 writes teacher data consisting of the generated input data and output data into the teacher data storage unit 106 for storage.

[0071] As described above, according to the present embodiment, image processing is performed on the object captured in the spectral reflectance image using each of the spectral reflectance information (spectral reflectance image), light source spectral distribution information, and image capture device spectral sensitivity information acquired in advance by measuring an RGB image, a spectral emissivity image, or the like, to generate a plurality of different spectral reflectance images from one spectral reflectance image, and an RGB image or a spectral emissivity image is generated using this generated spectral reflectance image and a known light source spectral distribution and image capture device spectral sensitivity. This makes it possible to increase the amount of spectral information by the amount of image processing, and to generate training data used for training a machine learning model that estimates spectral information such as the spectral reflectance, the spectral distribution of the light source, and the spectral sensitivity of the image capture device from an image containing spectral reflectance information such as an RGB image or a spectral emissivity image, by adjusting already existing spectral information by image processing, in the amount necessary for training the machine learning model, and it is possible to supply training data corresponding to the estimation accuracy of the machine learning model.

[0072] <Second embodiment> A second embodiment of the present invention will now be described with reference to the drawings. FIG. 6 is a block diagram showing an example of the configuration of a machine learning teacher data generation system according to the second embodiment of the present invention. In Figure 6, the machine learning teacher data generation system 100B includes a data input unit 101, a dataset specification input unit 102, a teacher data generation unit 103, a teacher data output unit 104, a basic spectral information memory unit 105, a teacher data memory unit 106, a light source spectral distribution memory unit 107, and an imaging device spectral sensitivity memory unit 108.

[0073] Hereinafter, regarding the machine learning teacher data generation system 100B according to the second embodiment, only the configuration and operation that are different from the first embodiment will be described. The light source spectral distribution storage unit 107 stores data on the light source spectral distributions of a plurality of different light sources. FIG. 7 is a diagram showing an example of the configuration of a light source spectral distribution table, which is a data table of light source spectral distributions written in advance in the light source spectral distribution storage unit 107. As shown in FIG. In the light source spectral distribution table, spectral distribution identification information and a light source spectral distribution index are written as a pair for each record.

[0074] The spectral distribution identification information is identification information for individually identifying each piece of light source spectral distribution data. The light source spectral distribution in this light source spectral distribution table is data of a light source different from the light source spectral distributions in each piece of spectral information stored in the basic spectral information storage unit 105. The light source spectral distribution index is information, such as an address, indicating an area in which data on the spectral distribution of a light source, that is, data such as a curve graph showing the correspondence between wavelength and spectral radiance, or a function, is written.

[0075] FIG. 8 is a diagram showing an example of the configuration of an image capture device spectral distribution table, which is a data table of image capture device spectral distributions written in advance in the image capture device spectral sensitivity storage unit 108. As shown in FIG. In the imaging device spectral sensitivity table, spectral sensitivity identification information and an imaging device spectral sensitivity index are written as a pair for each record.

[0076] The spectral sensitivity identification information is identification information for individually identifying each piece of image capture device spectral sensitivity data. The image capture device spectral sensitivities in this image capture device spectral sensitivity table are data of image capture devices that are different from the image capture device spectral sensitivities in each piece of spectral information stored in the basic spectral information storage unit 105. The imaging device spectral sensitivity index is information, such as an address, indicating an area in which the spectral sensitivity data of the imaging device, that is, data such as a curve graph showing the correspondence between wavelength and optical spectrum, or a function, is written.

[0077] FIG. 9 is a conceptual diagram illustrating a process of generating teacher data by referring to a light source spectral distribution table and an imaging device spectral distribution table in the second embodiment. FIG. 9(a) shows that in the formula described in FIG. 1(a), the teacher data generation unit 103 uses the imaging device spectral sensitivity (adjustment spectral sensitivity) in the imaging device spectral sensitivity table and the light source spectral distribution (adjustment spectral distribution) in the light source spectral distribution table as the light source spectral distribution (light source) and the imaging device spectral sensitivity (camera spectral sensitivity) to be multiplied by the spectral reflectance (reflectance) when calculating the RGB value. In addition, when the imaging device is a hyperspectral camera, the teacher data generation unit 103 indicates that the light source spectral distribution in the light source spectral distribution table is used as the light source spectral distribution (light source) to be multiplied by the spectral reflectance (reflectance) when calculating the spectral radiance value.

[0078] As shown in FIG. 9B, a spectral emissivity image (spectral image) 401 and an RGB image 402 in the spectral information are used as they are, and in addition to a light source spectral distribution 403 in the spectral information, light source spectral distributions 405_1 to 405_3 in the light source spectral distribution table and imaging device spectral sensitivities 406_1 and 406_2 in the imaging device spectral sensitivity table are used.

[0079] This allows the teacher data generation unit 103 to generate multiple RGB images 407 from one spectral reflectance image 401 using multiple light source spectral distributions and imaging device spectral sensitivities that differ from the light source spectral distribution and imaging device spectral sensitivities at the time of capturing the captured image from which the spectral reflectance image was extracted. That is, by adjusting the light source spectral distribution and the imaging device spectral sensitivity described above by the teacher data generating unit 103, in addition to the image processing in the first embodiment, the amount of teacher data can be further increased.

[0080] In addition, the teacher data generation unit 103 may be configured to generate a new light source spectral distribution (synthetic spectral distribution) by multiplying each of the light source spectral distributions in the light source spectral distribution table by a predetermined ratio and adding them together, and then replace the light source spectral distribution in the above-mentioned spectral information to generate a different RGB image. Similarly, the teacher data generation unit 103 may be configured to generate a new imaging device spectral sensitivity (synthetic spectral sensitivity) by multiplying each of the imaging device spectral sensitivities in the imaging device spectral sensitivity table by a predetermined ratio and adding them together, and to replace the imaging device spectral sensitivity in the above-mentioned spectral information, thereby generating a different RGB image.

[0081] Furthermore, the teacher data generation unit 103 may be configured to generate a new light source spectral distribution by multiplying the light source spectral distribution in the spectral information and each of the light source spectral distributions in the light source spectral distribution table by a predetermined ratio and adding them together, and then replace the light source spectral distribution in the above-mentioned spectral information with the new light source spectral distribution, thereby generating a different RGB image. Similarly, the teacher data generation unit 103 may be configured to generate a new imaging device spectral sensitivity by multiplying the imaging device spectral sensitivity in the spectral information and each of the imaging device spectral sensitivities in the imaging device spectral sensitivity table by a predetermined ratio and adding them together, and then replace the imaging device spectral sensitivity in the above-mentioned spectral information with the new imaging device spectral sensitivity, thereby generating a different RGB image.

[0082] Furthermore, by dividing each RGB value of the RGB image 402 by the light source spectral distribution in the light source spectral distribution table or the imaging device spectral sensitivity in the imaging device spectral sensitivity table, it is possible to obtain a spectral reflectance different from that obtained when the light source spectral distribution and the imaging device spectral sensitivity in the spectral information are used. This makes it possible to increase the number of combinations of each of the acquired spectral reflectances with RGB images, thereby increasing the number of data sets of training data.

[0083] As described above, according to the present embodiment, similarly to the first embodiment, image processing is performed on the object captured in the spectral reflectance image using each of the spectral reflectance information (spectral reflectance image), light source spectral distribution information, and image capture device spectral sensitivity information acquired by measuring an RGB image, a spectral emissivity image, or the like in advance, to generate a plurality of different spectral reflectance images from one spectral reflectance image, and an RGB image or a spectral emissivity image is generated using the generated spectral reflectance image and a known light source spectral distribution and image capture device spectral sensitivity. This makes it possible to increase the amount of spectral information by the amount of image processing, and it becomes possible to generate training data used for training a machine learning model that estimates spectral information such as the spectral reflectance, the spectral distribution of the light source, and the spectral sensitivity of the image capture device from an image containing spectral reflectance information such as an RGB image or a spectral emissivity image, by adjusting already existing spectral information by image processing, as many times as necessary for training the machine learning model, and it is possible to supply training data corresponding to the estimation accuracy of the machine learning model.

[0084] Furthermore, according to this embodiment, by replacing each of the light source spectral distribution and the imaging device spectral sensitivity in the spectral information with the light source spectral distribution in the light source spectral distribution table, the imaging device spectral sensitivity in the imaging device spectral sensitivity table, or each of the composite spectral distribution and the composite spectral sensitivity, in addition to the combination of image processing in the first embodiment, it becomes possible to generate more training data from already existing spectral information by using combinations of spectral distributions and spectral sensitivities, and to supply training data that corresponds to the estimation accuracy of the machine learning model.

[0085] <Third embodiment> Hereinafter, a third embodiment of the present invention will be described with reference to the drawings. FIG. 10 is a block diagram showing an example of the configuration of a machine learning teacher data generation system according to the third embodiment of the present invention. In FIG. 10, the machine learning teacher data generation system 100C includes a data input unit 101, a dataset specification input unit 102, a teacher data generation unit 103, a teacher data output unit 104, a basic spectral information memory unit 105, a teacher data memory unit 106, a light source spectral distribution memory unit 107, an imaging device spectral sensitivity memory unit 108, and a spectral component information memory unit 109.

[0086] Hereinafter, regarding the machine learning teacher data generation system 100C according to the third embodiment, only the configuration and operation that are different from the second embodiment will be described. The spectral component information storage unit 109 stores data on the spectral components of the spectral reflectances of a plurality of different objects. FIG. 11 is a diagram showing an example of the configuration of a spectral component table, which is a data table of spectral components written in advance in the spectral component information storage unit 109 in the third embodiment. In the spectral component table, a pair of spectral component identification information and a spectral distribution index is written for each record.

[0087] The spectral component identification information is identification information for identifying each of the data of the spectral components individually. The spectral components in this spectral component table indicate, for example, each of a plurality of principal component curves (relation curves of the spectral intensity for each wavelength) obtained by performing a principal component analysis of the spectral reflectance of the object. The spectral component index is information, such as an address, that indicates an area in which data on spectral components, that is, data such as curve graphs showing the correspondence between wavelengths and spectral intensities, or functions, is written.

[0088] The teacher data storage unit 106 may be configured to synthesize each of the principal component curves read out from the spectral component table in the spectral component information storage unit 109 at an arbitrary ratio to generate a new spectral reflectance, replace the spectral reflectance in the spectral information, and multiply this by the light source spectral sensitivity and the imaging device spectral sensitivity to generate an RGB image having new RGB values. In addition, the teacher data storage unit 106 may be configured to combine the spectral reflectances in the spectral information and each of the principal component curves in an arbitrary ratio to generate new spectral reflectances, replace the spectral reflectances in the spectral information, and multiply this by the light source spectral sensitivity and the imaging device spectral sensitivity to generate a spectral radiance value image having a new spectral radiance value.

[0089] In addition, the teacher data storage unit 106 may be configured to combine the spectral reflectances in the spectral information and each of the principal component curves in an arbitrary ratio to generate new spectral reflectances, replace the spectral reflectances in the spectral information, and multiply this by the light source spectral sensitivity and the imaging device spectral sensitivity to generate an RGB image having new RGB values. Similarly, the teacher data storage unit 106 may be configured to combine the spectral reflectances in the spectral information and each of the principal component curves in an arbitrary ratio to generate new spectral reflectances, replace the spectral reflectances in the spectral information, and multiply this by the light source spectral sensitivity to generate a spectral radiance value image having new spectral radiance values.

[0090] FIG. 12 is a conceptual diagram illustrating a process of generating teacher data by referring to a spectral component table in the third embodiment. FIG. 12(a) shows that in the formula explained in FIG. 1(a), the teacher data generation unit 103 uses a newly generated spectral reflectance as the spectral reflectance (reflectance) when calculating the RGB value, and multiplies this spectral reflectance by each of the light source spectral distribution (light source) and the imaging device spectral sensitivity (camera spectral sensitivity). In addition, when the imaging device is a hyperspectral camera, the teacher data generation unit 103 calculates the spectral radiance value by replacing it with the spectral reflectance in the spectral information and multiplying the newly generated spectral reflectance by the light source spectral distribution.

[0091] FIG. 12(b) shows the formula for calculating the spectral reflectance (reflectance), in which each of the spectral components (nth components) is multiplied by a coefficient kn, and the multiplication results are added (synthesized) to generate a new spectral reflectance. As an example of the spectral components, as described above, data showing the relationship between the wavelengths and spectra of the principal components obtained by performing principal component analysis on the spectral reflectance of the object is used. As another example, the spectral components may be data on the spectral reflectance of each of the components that mainly determine the spectral reflectance of the object.

[0092] 12(c) shows that the teacher data generating unit 103 multiplies the spectral reflectance 501 of the first component and the spectral reflectance 502 of the second component by coefficients k1 and k2, respectively, of a part of the human body, for example, hemoglobin, which is a component that forms the skin, as the first component and melanin as the second component, and adds (combines) the multiplication results, that is, adjusts the spectral components in the object at a predetermined content ratio, to obtain a new spectral reflectance 503. The spectral reflectances of other parts of the human body can also be adjusted in a similar manner. The teacher data generating unit 103 may be configured to multiply this new spectral reflectance 503 by the light source spectral distribution and the imaging device spectral sensitivity, and generate an RGB image having RGB values ​​that are the multiplication results.

[0093] As described above, according to the present embodiment, similarly to the first embodiment, image processing is performed on the object captured in the spectral reflectance image using each of the spectral reflectance information (spectral reflectance image), light source spectral distribution information, and image capture device spectral sensitivity information acquired by measuring an RGB image, a spectral emissivity image, or the like in advance, to generate a plurality of different spectral reflectance images from one spectral reflectance image, and an RGB image or a spectral emissivity image is generated using the generated spectral reflectance image and a known light source spectral distribution and image capture device spectral sensitivity. This makes it possible to increase the amount of spectral information by the amount of image processing, and it becomes possible to generate training data used for training a machine learning model that estimates spectral information such as the spectral reflectance, the spectral distribution of the light source, and the spectral sensitivity of the image capture device from an image containing spectral reflectance information such as an RGB image or a spectral emissivity image, by adjusting already existing spectral information by image processing, as many times as necessary for training the machine learning model, and it is possible to supply training data corresponding to the estimation accuracy of the machine learning model.

[0094] Furthermore, according to this embodiment, the spectral reflectance in the spectral information is synthesized with each of the spectral components stored in the spectral component information storage unit 109 at a predetermined ratio, and the spectral reflectance of the spectral information is synthesized with each of the spectral components stored in the spectral component information storage unit 109 at a predetermined ratio, thereby making it possible to increase the number of pieces of spectral reflectance data. By combining the light source spectral distribution and imaging device spectral sensitivity in the already existing spectral information with the light source spectral distribution in the light source spectral distribution table and the imaging device spectral sensitivity in the imaging device spectral sensitivity table, it is possible to generate a large amount of training data and supply training data that corresponds to the estimation accuracy of the machine learning model.

[0095] <Fourth embodiment> Hereinafter, a fourth embodiment of the present invention will be described with reference to the drawings. FIG. 13 is a block diagram showing an example of the configuration of a machine learning teacher data generation system according to the fourth embodiment of the present invention. In Figure 13, the machine learning teacher data generation system 100D includes a data input unit 101, a dataset specification input unit 102, a teacher data generation unit 103, a teacher data output unit 104, a basic spectroscopic information storage unit 105, a teacher data storage unit 106, a missing information addition unit 110, and a complementary information storage unit 111.

[0096] Hereinafter, regarding the machine learning teacher data generation system 100D according to the fourth embodiment, only the configuration and operation that are different from the first embodiment will be described. The missing information adding unit 110 adds missing information that is missing in each of the spectral reflectance information, the light source spectral distribution, and the imaging device spectral sensitivity in the spectral information. In the complementary information storage unit 111, complementary information used to complement missing information is written and stored.

[0097] The spectral component information storage unit 109 stores data on the spectral components of the spectral reflectances of a plurality of different objects. FIG. 11 is a diagram showing an example of the configuration of a spectral component table, which is a data table of spectral components written in advance in the spectral component information storage unit 109 in the third embodiment. In the spectral component table, a pair of spectral component identification information and a spectral distribution index is written for each record.

[0098] As already mentioned, in generating training data, for an RGB image captured by a digital camera, a spectral reflectance image, imaging device spectral sensitivity, and light source spectral distribution are required as the minimum spectral information. Similarly, as described above, when generating training data, a spectral radiance image captured by a hyperspectral camera requires a spectral reflectance image and a light source spectral distribution as minimum spectral information.

[0099] For example, when there is no data on the spectral reflectance of a captured image, if there is an RGB image captured in the same environment (light source spectral distribution) and with the same imaging device (imaging device spectral sensitivity), the missing information adding unit 110 estimates the spectral reflectance from this RGB image and complements the spectral reflectance image. Furthermore, if there is information about the image capture location, and if the image capture location is a room, the missing information adding unit 110 can identify the type of light source and obtain the light source spectral distribution of the light source by searching the Internet. Furthermore, if there is information on the image capture location and image capture date and time, and the image capture location is outdoors, the missing information adding unit 110 can obtain the weather at the image capture location at the time of image capture by searching the Internet, and obtain the light source spectral distribution corresponding to that weather.

[0100] Furthermore, if the type of light source (light source type), for example, sunlight, LED (light emitting diode) light source, fluorescent light, incandescent light, etc., is known, the missing information adding unit 110 acquires representative light source spectral distributions for each by searching the Internet. Furthermore, if the type of subject is identified, missing information adding section 110 obtains the spectral reflectance of the subject by searching the Internet.

[0101] Furthermore, if the model and product number of the imaging device are known, the missing information adding section 110 acquires the imaging device spectral sensitivity of the imaging device by searching the Internet, such as from the homepage of the imaging device manufacturer. Furthermore, if the model and product number of the imaging device are known, the missing information adding section 110 acquires the imaging device spectral sensitivity of the imaging device by searching the Internet, such as from the homepage of the imaging device manufacturer. Furthermore, if the missing information adding unit 110 knows the camera settings of the imaging device, it can estimate the scale of the imaging device's spectral sensitivity (degree of brightness) at the time of capturing an image from the ISO sensitivity, shutter speed, aperture, and the like of the captured image.

[0102] Furthermore, when the spectral reflectance is unknown in the spectral information, but the RGB image, light source spectral distribution, and image capture device spectral sensitivity are known, the missing information adding unit 110 acquires the spectral reflectance by estimating the spectral reflectance that becomes the RGB values ​​of the RGB image while adjusting the coefficients by which to multiply each of the principal component curves of the spectral principal components acquired by performing principal component analysis on the spectral reflectance for each type of subject.

[0103] Furthermore, when the spectral reflectance and light source spectral distribution are unknown in the spectral information, but the RGB image of the color chart, the spectral reflectance of the color chart, and the spectral sensitivity of the imaging device are known, the missing information adding unit 110 acquires the light source spectral distribution by estimating the spectral distribution that will become the RGB values ​​of the RGB image of the color chart while adjusting the coefficients by which to multiply each of the principal component curves of the acquired spectral principal components by performing principal component analysis of the light source spectral distribution. Then, the missing information adding unit 110 acquires the spectral reflectance by estimating the spectral reflectance that becomes the RGB values ​​of the RGB image while adjusting the coefficients to be multiplied with each of the principal component curves of the spectral principal components acquired by performing principal component analysis on the spectral reflectance for each type of subject, using the known RGB image, the known imaging device spectral sensitivity, and the estimated spectral distribution.

[0104] Furthermore, when the spectral reflectance and the spectral sensitivity of the image capture device are unknown in the spectral information, but the RGB image of the color chart, the spectral reflectance of the color chart, and the light source spectral distribution are known, the missing information adding unit 110 acquires the spectral sensitivity of the image capture device by estimating the spectral sensitivity that becomes the RGB values ​​of the RGB image of the color chart while adjusting the coefficients by which to multiply each of the principal component curves of the spectral principal components acquired by performing principal component analysis of the spectral sensitivity of the image capture device. Then, the missing information adding unit 110 acquires the spectral reflectance by estimating the spectral reflectance that becomes the RGB values ​​of the RGB image while adjusting the coefficients to be multiplied with each of the principal component curves of the spectral principal components acquired by performing principal component analysis on the spectral reflectance for each type of subject using the known RGB image, the estimated imaging device spectral sensitivity, and the known light source spectral distribution.

[0105] Furthermore, when the spectral information contains data of only an RGB image and two or more pieces of data, namely, the spectral reflectance, the light source spectral distribution, and the imaging device spectral sensitivity, are missing, the missing information adding unit 110 may be configured to complement the missing data by estimating coefficients by which the spectral reflectance, the light source spectral distribution, and the imaging device spectral sensitivity are multiplied by each of the principal component curves while adjusting them using the least squares method, or by estimating them using a machine learning model. In the complementary information storage unit 111, information such as the above-mentioned chart image, image capture location, image capture time, light source type, subject type, camera model and part number, camera settings, etc. is written and stored in advance for each spectral information in the basic spectral information storage unit 105.

[0106] As described above, according to the present embodiment, similarly to the first embodiment, image processing is performed on the object captured in the spectral reflectance image using each of the spectral reflectance information (spectral reflectance image), light source spectral distribution information, and image capture device spectral sensitivity information acquired by measuring an RGB image, a spectral emissivity image, or the like in advance, to generate a plurality of different spectral reflectance images from one spectral reflectance image, and an RGB image or a spectral emissivity image is generated using the generated spectral reflectance image and a known light source spectral distribution and image capture device spectral sensitivity. This makes it possible to increase the amount of spectral information by the amount of image processing, and it becomes possible to generate training data used for training a machine learning model that estimates spectral information such as the spectral reflectance, the spectral distribution of the light source, and the spectral sensitivity of the image capture device from an image containing spectral reflectance information such as an RGB image or a spectral emissivity image, by adjusting already existing spectral information by image processing, as many times as necessary for training the machine learning model, and it is possible to supply training data corresponding to the estimation accuracy of the machine learning model.

[0107] Furthermore, according to this embodiment, when the spectral information is missing the spectral reflectance, the light source spectral distribution, and the imaging device spectral sensitivity, and teacher data cannot be generated, this missing information can be obtained using the complementary information stored in the complementary information storage unit 111, making it possible to generate a large amount of teacher data and supply teacher data that corresponds to the estimation accuracy of the machine learning model.

[0108] <Fifth embodiment> Hereinafter, a fifth embodiment of the present invention will be described with reference to the drawings. The configuration of the fifth embodiment of the present invention is the same as that of the first embodiment in Fig. 3. Below, a description will be given of the configuration and operation of the machine learning teacher data generation system 100E of the fifth embodiment that differs from the machine learning teacher data generation system 100 of the first embodiment. The machine learning teacher data generation system 100E according to the fifth embodiment expands a two-dimensional captured image of a target object in the first embodiment into a three-dimensional shape model of the target object.

[0109] In other words, the machine learning models for which training data is provided in this embodiment include not only a machine learning model that takes an RGB image as input data and estimates a spectral reflectance image as output data, but also a machine learning model that takes a 3D RGB shape model, in which the RGB values ​​for each voxel of the shape constituent units of a 3D shape model (three-dimensional shape information) are indicated as input data, estimates the spectral reflectance of each voxel of the 3D shape model, and outputs output data as a 3D spectral reflectance shape model. In addition, by capturing images of a three-dimensional shape model using virtual imaging devices placed at different viewpoints in a three-dimensional virtual space and generating new virtual captured images, the data set based on captured images (two-dimensional) can be expanded.

[0110] 14 is a diagram showing an example of the configuration of a second spectral information table pre-written in the fundamental spectral information storage unit 105 in the fifth embodiment of the present invention. The spectral information table described in the first embodiment is also pre-written in the fundamental spectral information storage unit 105 together with the second spectral information table in this embodiment. In FIG. 14, the second spectral information table has columns for each record: three-dimensional shape spectral information identification information, a spectral reflectance image index (or at least an imaging device spectral sensitivity index such as an RGB image or a spectral radiance image), a light source spectral distribution index (spectral reflectance), and a three-dimensional shape model (multiple viewpoint image) information index.

[0111] The three-dimensional spectral information identification information is identification information for identifying each three-dimensional shape model (or multi-viewpoint image) from which each of the spectral information (spectral reflectance, imaging device spectral sensitivity, light source spectral distribution) is extracted. Here, the imaging device spectral sensitivity is the spectral information in the case where the captured image from which the spectral information is extracted is captured by a digital camera. In addition, it is information that is not included in the spectral radiance image captured by the hyperspectral camera.

[0112] The light source spectral distribution index is information, e.g., an address, indicating an area in which the spectral distribution information (light source spectral distribution, a curve graph showing the correspondence between wavelength and spectral radiance, etc.) of the light source in the environment in which the captured image used to generate the 3D shape model was captured is written. The imaging device spectral sensitivity index is information, for example an address, that indicates an area in which spectral sensitivity information (imaging device spectral sensitivity, a curve graph showing the correspondence between wavelength and sensitivity, etc.) of the imaging device (in the case of a digital camera) that captured the image used when generating the three-dimensional shape model is written.

[0113] The three-dimensional shape model information index is information, such as an address, indicating an area in which the coordinates and spectral reflectance of each voxel constituting a three-dimensional shape model previously obtained corresponding to each captured image (multiple viewpoint image) are written. The multi-viewpoint image is, for example, the already explained spectral image, or an image having information on the spectral reflectance of each pixel. In addition, the multi-viewpoint image may be an image such as an RGB image or a spectral radiance image that contains at least the spectral reflectance of each voxel and information on the spectral distribution of the light source (including the spectral sensitivity of the imaging device if the imaging device is a digital camera).

[0114] Furthermore, in addition to the information of the first embodiment, the specification information input from the dataset specification input unit 102 includes at least a three-dimensional RGB shape model, a three-dimensional spectral radiance shape model, and a three-dimensional spectral reflectance shape model as input data for the machine learning model, and at least a three-dimensional spectral reflectance shape model as output data. The image processing method in the specification information includes at least the number of different viewpoint images to be generated, mask settings, the presence or absence of shading, the shading pattern, and the like.

[0115] The number of different viewpoint images generated (the number of viewpoints) indicates the number of virtual captured images of a 3D shape model of the same object (e.g., a part of a human body such as a head or a hand) captured by a virtual imaging device arranged at different viewpoints in a 3D virtual space, and indicates the number of viewpoints for capturing the virtual captured images. The number of different viewpoint images generated is a numerical value included in the number of data, which is the number of teacher data to be generated in the specification information. As a result, the teacher data generating unit 103 can easily increase the number of data sets that can be generated from one piece of spectral information by increasing the number of viewpoints captured in the 3D virtual space and generating the number of data sets of teacher data, for example, two-dimensional RGB images and spectral reflectance images observed from a plurality of different viewpoints, from spectral information corresponding to one three-dimensional shape.

[0116] The mask setting is data showing an image of any three-dimensional shape seen from a specific direction as a template when a virtual captured image of a three-dimensional shape captured from a specific viewpoint position in a three-dimensional virtual space, i.e., a specific direction, is to be obtained. For example, when a virtual captured image of a human body part such as a face or the back of a hand is required, a general image of a face seen from the front or an image of the back of a hand is input as a mask, and the teacher data generating unit 103 adjusts the viewpoint of the three-dimensional virtual space for capturing each of the three-dimensional shape models in the basic spectral information storage unit 105 so that the image corresponds to the mask, and generates a virtual captured image by capturing the image from the viewpoint.

[0117] The presence or absence of shading (shadows generated by illuminating an object from a light source) is data that indicates whether or not shading is superimposed on the virtual captured image. If there is no shading, shading removal is performed. This shading removal refers to a process of removing shading in a virtual captured image generated from a three-dimensional spectral reflectance shape model, for example. In a three-dimensional virtual space, the area in which shading is applied to the three-dimensional shape model differs depending on the position of the light source that irradiates light, so the position of shading in the captured image of the three-dimensional shape model changes. If no shading is specified in the specification information, the teacher data generation unit 103 removes the shading that was originally present in the above-mentioned three-dimensional spectral reflectance image.

[0118] On the other hand, when shading is present, for example, shading is superimposed on the RGB image that is used as input data in the training data, and the machine learning model is made to learn in such a way that the influence of shading is eliminated from the spectral reflectance image of the estimated output data. The pattern is data for setting an area in the virtual captured image where shading data is to be superimposed on the mask when shading is present. For this reason, the teacher data generating unit 103 performs a process of superimposing a change in shading corresponding to the pattern on an RGB image, which is a virtual captured image, by using this pattern setting, for example.

[0119] When a request to generate a 3D shape model in which the type of voxel data is expressed as either spectral radiance, spectral reflectance, or RGB values ​​is input, the dataset specification input unit instructs the teacher data generation unit 103 to generate a 3D shape composed of voxels of the requested type of data. As a result, when generating a dataset of teacher data in accordance with the specification information of the dataset specifications, if 3D shape model data such as a 3D RGB shape model, a 3D spectral radiance shape model, and a 3D spectral reflectance shape model, as well as the number of different viewpoint images to be generated, mask settings, shading, etc. are required, the teacher data generation unit 103 restores the 3D shape from the multi-viewpoint images and generates a 3D RGB shape model, a 3D spectral radiance shape model, and a 3D spectral reflectance shape model, etc.

[0120] Then, the teacher data generation unit 103 captures each of the 3D RGB shape model, the 3D spectral radiance shape model, and the 3D spectral reflectance shape model from any viewpoint in the 3D virtual space or from an input viewpoint, and generates an RGB image, a spectral radiance image, and a spectral reflectance image, respectively, as virtual captured images. In addition, the process of generating teacher data in the teacher data generation unit 103 for the RGB image, the spectral radiance image, and the spectral reflectance image, and for each of the RGB values, the spectral reflectance, and the spectral radiance, is the same as in the first to fourth embodiments. Furthermore, the teacher data generating unit 103 generates teacher data in response to specification information such as the number of different viewpoint images to be generated, mask settings, the presence or absence of shading, and the shading pattern.

[0121] As described above, according to the present embodiment, similarly to the first embodiment, image processing is performed on the object captured in the spectral reflectance image using each of the spectral reflectance information (spectral reflectance image), light source spectral distribution information, and image capture device spectral sensitivity information acquired by measuring an RGB image, a spectral emissivity image, or the like in advance, to generate a plurality of different spectral reflectance images from one spectral reflectance image, and an RGB image or a spectral emissivity image is generated using the generated spectral reflectance image and a known light source spectral distribution and image capture device spectral sensitivity. This makes it possible to increase the amount of spectral information by the amount of image processing, and it becomes possible to generate training data used for training a machine learning model that estimates spectral information such as the spectral reflectance, the spectral distribution of the light source, and the spectral sensitivity of the image capture device from an image containing spectral reflectance information such as an RGB image or a spectral emissivity image, by adjusting already existing spectral information by image processing, as many times as necessary for training the machine learning model, and it is possible to supply training data corresponding to the estimation accuracy of the machine learning model.

[0122] Furthermore, according to this embodiment, in a three-dimensional virtual space, three-dimensional shape models (three-dimensional RGB shape model, three-dimensional spectral radiance shape model, and three-dimensional spectral reflectance shape model) of an object are captured from multiple viewpoints to generate virtual captured images (RGB image, spectral radiance image, and spectral reflectance image), and it is further possible to arbitrarily set whether or not to apply shading and the area to which the shading data is applied, making it possible to generate a large amount of training data and supply training data that corresponds to the estimation accuracy of the machine learning model.

[0123] Sixth embodiment Hereinafter, a sixth embodiment of the present invention will be described with reference to the drawings. FIG. 15 is a block diagram showing an example of the configuration of a machine learning teacher data generation system according to the sixth embodiment of the present invention. In Figure 15, the machine learning teacher data generation system 100F includes a data input unit 101, a dataset specification input unit 102, a teacher data generation unit 103, a teacher data output unit 104, a basic spectroscopic information memory unit 105, a teacher data memory unit 106, a similar 3D shape generation unit 112, and a similar 3D shape memory unit 113.

[0124] Hereinafter, the system 100F for generating teacher data for machine learning according to this embodiment will be described, focusing only on the differences from the system 100E for generating teacher data for machine learning according to the fifth embodiment. When a request to generate a similar three-dimensional shape similar to a three-dimensional shape model of a target object is input as specification information, the data set specification input unit 102 instructs the similar three-dimensional shape generation unit 112 to generate a similar three-dimensional shape. As a result, similar three-dimensional shape generator 112 generates a similar three-dimensional shape similar to the three-dimensional shape model by directly using the spectral reflectance of the three-dimensional shape model in the spectral information of the target object. Here, similar three-dimensional shape generating unit 112 generates a three-dimensional shape model of an object of the same type as the three-dimensional shape model of the second spectral information table in basic spectral information storage unit 105 (for example, each part of the human body).

[0125] That is, for example, if the 3D shape model in the second spectral information table is of a head in a human body, the similar 3D shape generation unit 112 uses the spectral reflectance (or RGB value, spectral radiance) in the 3D shape model of this head for a 3D shape model of another human head, and generates a 3D shape model of the object (3D RGB shape model, 3D spectral radiance shape model and 3D spectral reflectance shape model) to be used in generating training data.

[0126] A 3D shape model (or a multi-viewpoint image) of the same type as the object is written in advance and stored for each object in the similar 3D shape storage unit 113. This 3D shape model may be information of only the coordinate values ​​of voxels, or may include RGB values, spectral reflectance, and spectral radiance values ​​for each voxel. When the specification information for generating a data set contains an instruction to generate similar data of a three-dimensional shape, the similar three-dimensional shape generation unit 112 reads out a three-dimensional shape model of the same type as the object for which training data is to be generated, by referring to the similar three-dimensional shape memory unit 113.

[0127] Then, the similar three-dimensional shape generating unit 112 sets the read three-dimensional shape model as a similar three-dimensional shape model, and reflects the spectral reflectance (or RGB value, spectral radiance) of the three-dimensional shape model in the second spectral information table to the similar three-dimensional shape model. At this time, for example, when the target object is a head, which is a part of the human body, the similar 3D shape generation unit 112 performs processing to convert the spectral reflectances (or RGB values, spectral radiance) of the eyes, mouth, nose, ears, etc., which are components of the head, which is the similar 3D shape model, into the corresponding spectral reflectances (or RGB values, spectral radiance) of the eyes, mouth, nose, ears, etc., which are components of the head, which is the 3D shape model in the second spectral information table.

[0128] In addition, the similar 3D shape generation unit 112 may be configured to generate a similar 3D shape model as another 3D shape model of an object of the same type (e.g., each part of the human body) by arbitrarily deforming the 3D shape model of the second spectral information table in the basic spectral information storage unit 105. The above-mentioned parts to be deformed are set as specification information of the data set specification. For example, if the target object is the head, which is a part of the human body, the eyes, mouth, nose, ears, etc., which are components of the head, are input as the specification information for deformation of the part as the components to be deformed. Here, the deformation also includes adjustment of the size and shape of the components, adjustment of the distance between each of the components, etc.

[0129] Then, for example, if the 3D shape model in the second spectral information table is for a head of a human body, the similar 3D shape generation unit 112 uses the spectral reflectance (or RGB value, spectral radiance) in this 3D shape model of the head in the same way for the deformed 3D shape model of the head, and generates a 3D shape model of the object (3D RGB shape model, 3D spectral radiance shape model and 3D spectral reflectance shape model) to be used in generating training data.

[0130] As described above, according to the present embodiment, similarly to the first embodiment, image processing is performed on the object captured in the spectral reflectance image using each of the spectral reflectance information (spectral reflectance image), light source spectral distribution information, and image capture device spectral sensitivity information acquired by measuring an RGB image, a spectral emissivity image, or the like in advance, to generate a plurality of different spectral reflectance images from one spectral reflectance image, and an RGB image or a spectral emissivity image is generated using the generated spectral reflectance image and a known light source spectral distribution and image capture device spectral sensitivity. This makes it possible to increase the amount of spectral information by the amount of image processing, and it becomes possible to generate training data used for training a machine learning model that estimates spectral information such as the spectral reflectance, the spectral distribution of the light source, and the spectral sensitivity of the image capture device from an image containing spectral reflectance information such as an RGB image or a spectral emissivity image, by adjusting already existing spectral information by image processing, as many times as necessary for training the machine learning model, and it is possible to supply training data corresponding to the estimation accuracy of the machine learning model.

[0131] Furthermore, according to this embodiment, in a 3D virtual space, based on a 3D shape model of an object (a 3D RGB shape model, a 3D spectral radiance shape model, and a 3D spectral reflectance shape model), a similar 3D shape model is generated that is of the same type as the 3D shape model but has a different 3D shape, and by increasing the number of 3D shapes used when generating training data, it becomes possible to generate a large amount of training data and supply training data that corresponds to the estimation accuracy of the machine learning model.

[0132] Seventh embodiment Hereinafter, a seventh embodiment of the present invention will be described with reference to the drawings. FIG. 16 is a block diagram showing an example of the configuration of a machine learning teacher data generation system according to the seventh embodiment of the present invention. In Figure 16, the machine learning teacher data generation system 100G includes a data input unit 101, a dataset specification input unit 102, a teacher data generation unit 103, a teacher data output unit 104, a basic spectroscopic information memory unit 105, a teacher data memory unit 106, a computer graphic data generation unit 114, and a computer graphic data memory unit 115.

[0133] Hereinafter, the system 100F for generating teacher data for machine learning according to this embodiment will be described, focusing only on the differences from the system 100 for generating teacher data for machine learning according to the first embodiment. The computer graphics data generating unit 114 generates CG (computer graphics) data of different shapes for the specified object, the number of times being indicated in the specification information of the data set. Furthermore, the computer graphic data generating unit 114 adds color information having the same RGB values ​​as the object indicated in the specification information but different spectral reflectance to the generated CG data.

[0134] FIG. 17 is a conceptual diagram illustrating the process in which the teacher data generating unit 103 adds CG data to an RGB image. The teacher data generation unit 103 places CG data as an object (pseudo target object) 552 adjacent to the back of the hand 551, which is generated as teacher data, in an RGB image 451 in which the back of the hand 551 is captured as a target object, and generates an RGB image 452. Here, the computer graphic data generation unit 114 generates the color components R, G, and B of the pixel values ​​of the back of the hand 551, which is the subject in the RGB image, as similar numerical values ​​to the color components R', G', and B' of the pixel values ​​of the object 552.

[0135] That is, the computer graphic data generating unit 114 sets the color components R', G', and B' as values ​​at which the back of the hand 551 and the object 552 are visually recognized by a human being as having the same skin color. Here, the computer graphic data generation unit 114 sets the numerical values ​​of the color components R', G', and B' so that, for example, the difference between color components R and R', the difference between color components G and G', and the difference between color components B and B' are each values ​​that fall within a predetermined difference range.

[0136] Then, the computer graphic data generating unit 114 writes and stores the generated CG data in the computer graphic data storage unit 115 in correspondence with each target object for which the data is to be used. When the generation of CG data is specified in the specification information of the data set, the teacher data generation unit 103 reads out CG data corresponding to the target object from the computer graphic data storage unit 115, and places each piece of the CG data in the same RGB image or in different RGB images to generate a new RGB image.

[0137] Furthermore, the computer graphic data generating unit 114 sets the spectral reflectance 561 of the back of the hand 551 and the spectral reflectance 562 of the object 552 as different spectral reflectances, that is, as spectral characteristics having different wavelength-spectral intensity curves. In the spectral reflectance in Fig. 17, the horizontal axis indicates the wavelength and the vertical axis indicates the spectral intensity. 17, RGB image 402 including object 552 is used as input data, and a spectral reflectance image (or a spectral image) of only the back of hand 551 not including object 552 is used as output data to generate a data set of teacher data, and a machine learning model is trained. This makes it possible to obtain a machine learning model that can estimate a spectral reflectance image of only the target object to be obtained from an RGB image that includes images of similar colors but different shapes as noise.

[0138] As described above, according to the present embodiment, similarly to the first embodiment, image processing is performed on the object captured in the spectral reflectance image using each of the spectral reflectance information (spectral reflectance image), light source spectral distribution information, and image capture device spectral sensitivity information acquired by measuring an RGB image, a spectral emissivity image, or the like in advance, to generate a plurality of different spectral reflectance images from one spectral reflectance image, and an RGB image or a spectral emissivity image is generated using the generated spectral reflectance image and a known light source spectral distribution and image capture device spectral sensitivity. This makes it possible to increase the amount of spectral information by the amount of image processing, and it becomes possible to generate training data used for training a machine learning model that estimates spectral information such as the spectral reflectance, the spectral distribution of the light source, and the spectral sensitivity of the image capture device from an image containing spectral reflectance information such as an RGB image or a spectral emissivity image, by adjusting already existing spectral information by image processing, as many times as necessary for training the machine learning model, and it is possible to supply training data corresponding to the estimation accuracy of the machine learning model.

[0139] Furthermore, according to this embodiment, by placing CG data of a color having the same RGB color components as the object but different spectral reflectance as noise together with the object captured in the RGB image used as input data, it is possible to supply a dataset of training data that can obtain a machine learning model with high estimation accuracy, which makes it possible to estimate a spectral reflectance image (spectral image) of only the object with noise removed in the estimation process of actual output data. Although the present embodiment has been described in two dimensions, the fifth and sixth embodiments may be configured to generate and use three-dimensional CG data.

[0140] Moreover, the above-described first to seventh embodiments may be combined with each other.

[0141] In addition, a program for realizing the functions of the machine learning teacher data generation system 100 in FIG. 3, the machine learning teacher data generation system 100B in FIG. 6, the machine learning teacher data generation system 100C in FIG. 10, the machine learning teacher data generation system 100D in FIG. 13, the machine learning teacher data generation system 100F in FIG. 15, and the machine learning teacher data generation system 100G in FIG. 16 in the present invention may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read and executed by a computer system to generate a data set of teacher data used in training a machine learning model. In addition, the "computer system" as used herein includes hardware such as an OS (Operating System) and peripheral devices. In addition, the "computer system" also includes a WWW (World Wide Web) system equipped with a homepage providing environment (or display environment). In addition, the "computer-readable recording medium" refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM (Read Only Memory), a CD-ROM (Compact Disc - Read Only Memory), and a storage device such as a hard disk built into a computer system. Furthermore, the term "computer-readable recording medium" includes devices that retain a program for a certain period of time, such as volatile memory (RAM (Random Access Memory)) within a computer system that serves as a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line.

[0142] The above program may also be transmitted from a computer system in which the program is stored in a storage device or the like to another computer system via a transmission medium, or by transmission waves in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The above program may also be for realizing part of the above-mentioned functions. Furthermore, it may be a so-called difference file (difference program) that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0143] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and design changes and the like that do not deviate from the gist of the present invention are also included. [Explanation of symbols]

[0144] 100, 100B, 100C, 100D, 100F, 100G... Machine learning training data generation system 101...Data entry section 102...Data set specification input section 103...Teacher data generation unit 104...Teacher data output unit 105…Basic spectral information storage unit 106…Teacher data storage unit 107…Light source spectral distribution storage unit 108...imaging device spectral sensitivity memory unit 109...Spectral component information storage unit 110…Missing information acquisition unit 111…Supplementary information storage unit 112…Similar 3D shape generation unit 113…Similar 3D shape memory unit 114…Computer graphic data generation unit 115...Computer graphic data storage unit

Claims

1. a basic spectral information storage unit in which at least a spectral reflectance of a target object constituting a captured image of the target object, a spectral sensitivity of an image capturing device for the captured image, and a spectral distribution of a light source in an environment in which the captured image is captured are written and stored in advance as spectral information; a data set specification input unit that inputs a specification indicating a relationship between input data to a machine learning model generated from the spectroscopic information and output data estimated by the machine learning model based on the input data; a teacher data generating unit that generates teacher data including the input data and the output data by adjusting each of the spectral information in accordance with the specifications; Equipped A system for generating teacher data for machine learning.

2. The teacher data generation unit, New spectroscopic information of the captured image that is not included in the spectroscopic information is generated as the input data or the output data.

2. The system for generating training data for machine learning according to claim 1 .

3. A captured image captured by the imaging device is generated as a teacher captured image as the teacher data from the spectral information.

3. The system for generating training data for machine learning according to claim 1 or 2.

4. A light source spectral distribution storage unit in which adjustment spectral distributions, which are the spectral distributions of a plurality of types of light sources for adjusting the spectral information, are written and stored in advance. and The teacher data generation unit, The teacher-use captured image is generated using the adjustment spectral distribution in addition to the spectral distribution in the spectral information.

4. The system for generating training data for machine learning according to claim 3.

5. The teacher data generating unit synthesizes the spectral intensities for each wavelength between the adjustment spectral distributions at a predetermined ratio to generate a new synthetic spectral distribution, and generates the teacher captured image using the synthetic spectral distribution.

5. The system for generating training data for machine learning according to claim 4.

6. an image capture device spectral sensitivity storage unit in which adjustment spectral sensitivities, which are the spectral sensitivities of a plurality of types of image capture devices for adjusting the spectral information, are written and stored in advance; and The teacher data generation unit, The teacher image is generated by using the adjustment spectral sensitivity in addition to the spectral sensitivity in the spectral information. The system for generating teacher data for machine learning according to any one of claims 3 to 5.

7. The teacher data generation unit, The spectral intensities for each wavelength between the adjustment spectral sensitivities are synthesized at a predetermined ratio to generate a new synthetic spectral sensitivity, and the teacher captured image is generated using the synthetic spectral sensitivity.

7. The system for generating training data for machine learning according to claim 6.

8. a target object spectral characteristic storage unit in which, for each type of target object in the captured image, a spectral component having a high content ratio in the spectral reflectance of the corresponding type is written and stored in advance; and The teacher data generation unit, The spectral reflectance of the target object is changed by the adjusted spectral components to generate an adjustment spectral reflectance, which is a new spectral reflectance, and the teacher's captured image is generated. The system for generating training data for machine learning according to any one of claims 3 to 7.

9. The teacher data generation unit, The spectral components are multiplied by a predetermined coefficient, and the multiplication result is added to or subtracted from the spectral reflectance to generate the adjustment spectral reflectance.

9. The system for generating training data for machine learning according to claim 8.

10. the spectral components are a plurality of principal spectral components obtained by principal component analysis of the spectral reflectance of each type of target object in the captured image, The teacher data generation unit, Multiplying each of the principal spectral components by a coefficient, and adding the multiplication results of the principal spectral components together to generate the adjustment spectral reflectance.

9. The system for generating training data for machine learning according to claim 8.

11. When the target object is skin of a human body, the spectral components are at least the spectral characteristics of melanin and hemoglobin, The teacher data generation unit, The spectral characteristics of the melanin and the hemoglobin are used to generate the adjustment spectral reflectance.

9. The system for generating training data for machine learning according to claim 8.

12. a missing information acquiring unit that acquires missing information when information necessary for generating the captured image is missing from the spectral information and The missing information is acquired based on other information different from the missing information included in at least the spectral information. The system for generating training data for machine learning according to any one of claims 1 to 11.

13. The other information is complementary information for specifying the spectral reflectance, the spectral sensitivity, and the spectral distribution in the spectral information, the complementary information being used to complement the missing information in the spectral information. The system for generating training data for machine learning according to claim 12 .

14. The spectral information includes three-dimensional shape information that is information about a three-dimensional shape of the target object. The system for generating training data for machine learning according to any one of claims 1 to 13.

15. the three-dimensional shape information is a multi-viewpoint image consisting of captured images of the same target object captured from a plurality of different viewpoints, The teacher data generation unit, Generate the three-dimensional shape of the target object from the multi-view images. The system for generating training data for machine learning according to claim 14 .

16. The teacher data generation unit, A virtual captured image in which the three-dimensional shape is observed from a predetermined viewpoint and an image having information on the spectral reflectance corresponding to the virtual captured image are generated. The system for generating training data for machine learning according to claim 14 or 15.

17. When the teacher data generation unit is supplied with information on the presence or absence of shading and the range to which the shading is to be applied from the data set specification input unit, the teacher data generation unit applies shading to the range of the virtual captured image and an image having information on the spectral reflectance corresponding to the virtual captured image. The system for generating training data for machine learning according to claim 16.

18. The data set specification input unit: when a request for generating the three-dimensional shape in which the type of voxel data is expressed by any one of spectral radiance, spectral reflectance, and RGB values ​​is input, instructing the teacher data generating unit to generate the three-dimensional shape composed of voxels of the requested type of data; The teacher data generating unit generates the three-dimensional shape consisting of voxels of the requested type of data. The system for generating training data for machine learning according to any one of claims 14 to 17.

19. A similar shape generating unit that generates a similar three-dimensional shape similar to the three-dimensional shape, when a request for generating an analogous three-dimensional shape similar to the three-dimensional shape of the target object is input, the data set specification input unit instructs the analogous shape generating unit to generate the analogous three-dimensional shape; The similar shape generating unit generates the similar three-dimensional shape similar to the three-dimensional shape of the target object by directly using the spectral reflectance of the three-dimensional shape of the target object. The system for generating training data for machine learning according to any one of claims 14 to 18.

20. The target object is a predetermined part of a human body.

20. The system for generating training data for machine learning according to claim 1 .

21. A computer graphic data generating unit that generates computer graphic data, when a request to generate a pseudo target object using computer graphic data in addition to the target object is input, the data set specification input unit instructs the computer graphic data generation unit to generate the pseudo target object; The computer graphic data generating unit generates, by computer graphics, the pseudo target object to be synthesized into the captured image in which the target object is captured.

20. The system for generating training data for machine learning according to claim 1 .

22. The color components R, G, and B in the RGB image of the simulated target object are similar to the color components R, G, and B of the target object, respectively, and the spectral reflectance of the simulated target object is different from that of the target object. The system for generating training data for machine learning according to claim 21.

23. a dataset specification input process for inputting a specification indicating a relationship between input data to a machine learning model, the input data being generated from spectral information in a basic spectral information storage unit in which at least the spectral reflectance of a target object constituting a captured image of the target object, the spectral sensitivity of an imaging device for the captured image, and the spectral distribution of a light source in an environment in which the captured image was captured are written and stored in advance as spectral information, and output data estimated by the machine learning model based on the input data; a teacher data generating step in which a teacher data generating unit adjusts each of the spectral information in accordance with the specification to generate teacher data including the input data and the output data; Includes A method for generating training data for machine learning.

24. a program for causing a computer to function as a machine learning teacher data generation system for generating teacher data for a machine learning model from spectral information in a basic spectral information storage unit in which at least a spectral reflectance of a target object constituting a captured image of the target object, a spectral sensitivity of an image capturing device for the captured image, and a spectral distribution of a light source in an environment in which the captured image is captured are written and stored in advance as spectral information, the program comprising: The computer, a dataset specification input means for inputting a specification indicating a relationship between input data to the machine learning model, the input data being generated from the spectroscopic information, and output data estimated by the machine learning model based on the input data; A training data generating means for generating the training data including the input data and the output data by adjusting each of the spectral information in accordance with the specifications. A program to function as a

Images