Image processing system, image processing method, and program

Abstract

To provide an image processing system, an image processing method and a program capable of improving processing accuracy without performing relearning by data extension in processing using an existing machine learning model.SOLUTION: An image processing system includes an image generation unit for generating a plurality of RGB images on the basis of a spectral image obtained by photographing an object and a plurality of pieces of photographing condition information in which combination contents obtained by combining a light source spectral distribution and camera spectral sensitivity are different, a machine learning processing unit for inputting the generated plurality of RGB images to a machine learning model for performing processing based on an RGB image to acquire a processing result to be outputted from the machine learning model for each inputted RGB image, and an output integration unit for integrating a plurality of processing results acquired on the basis of the plurality of RGB images to generate a final processing result.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing system, an image processing method, and a program. [Background technology]

[0002] In recent years, machine learning has been used in various processes. For example, machine learning may be used in processes such as classification, identification, estimation, and conversion of images. The accuracy of processes using machine learning tends to depend on the variability, variety, and amount of data in the data set at the time of learning. For example, in the case of image processing, even if the input image is the same, the accuracy of the processing is affected by changes in brightness, color, etc. Furthermore, if an image with brightness or color not included in the data set at the time of learning is input, the accuracy of the processing decreases. For this reason, it is desirable for the data set at the time of learning to contain various types of data without excess or deficiency, but collecting the data involves considerable costs.

[0003] Therefore, various techniques have been proposed that can reduce the cost of collecting data by performing data expansion on existing data sets. For example, Non-Patent Document 1 below discloses a technique for improving accuracy and generalization performance by performing data expansion by image conversion on images used as training data in training. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks.”, Advances in Neural Information Processing Systems 25 (NIPS 2012). Summary of the Invention [Problem to be solved by the invention]

[0005] However, even if data can be expanded by the technology described in Non-Patent Document 1, if a machine learning model is already in use, it is necessary to re-learn using images obtained by data expansion. Therefore, even if the cost of collecting data can be reduced by data expansion, it will cost money to re-learn the machine learning model.

[0006] In view of the above-mentioned problems, an object of the present invention is to provide an image processing system, an image processing method, and a program capable of improving processing accuracy in processing using an existing machine learning model without performing re-learning through data expansion. [Means for solving the problem]

[0007] In order to solve the above-described problems, an image processing system according to one aspect of the present invention is an image processing system including: an image generation unit that generates a plurality of RGB images based on a spectral image of an object and a plurality of pieces of shooting condition information having different combinations of a light source spectral distribution and a camera spectral sensitivity; a machine learning processing unit that inputs the generated plurality of RGB images to a machine learning model that performs processing based on the RGB images, and obtains a processing result output from the machine learning model for each of the input RGB images; and an output integrating unit that integrates a plurality of processing results obtained based on the plurality of RGB images to generate a final processing result.

[0008] An image processing method according to one aspect of the present invention is an image processing method executed by a computer, the image processing method including: an image generation step of generating a plurality of RGB images based on a spectral image of an object and a plurality of pieces of shooting condition information each having different combinations of a light source spectral distribution and a camera spectral sensitivity; a machine learning processing step of inputting the generated plurality of RGB images to a machine learning model that performs processing based on the RGB images, and obtaining a processing result output from the machine learning model for each of the input RGB images; and an output integration step of integrating a plurality of processing results obtained based on the plurality of RGB images to generate a final processing result.

[0009] A program according to one aspect of the present invention is a program for causing a computer to function as: an image generation means for generating a plurality of RGB images based on a spectral image of an object photographed and a plurality of pieces of shooting condition information having different combinations of a light source spectral distribution and a camera spectral sensitivity; a machine learning processing means for inputting the generated plurality of RGB images to a machine learning model that performs processing based on the RGB images, and obtaining a processing result output from the machine learning model for each of the input RGB images; and an output integration means for integrating a plurality of processing results obtained based on the plurality of RGB images, and generating a final processing result. Effect of the Invention

[0010] According to the present invention, it is possible to improve the processing accuracy in processing using an existing machine learning model without performing re-learning through data expansion. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram showing an example of a configuration of an image processing system according to a first embodiment. [Diagram 2] 5 is a flowchart showing an example of a processing flow in the image processing system according to the first embodiment. [Diagram 3] 5 is a flowchart showing an example of the flow of image generation processing according to the first embodiment. [Figure 4]10 is a flowchart showing an example of the flow of an output integration process according to the first embodiment. [Diagram 5] FIG. 2 is a diagram showing a first specific example of the machine learning process according to the first embodiment. [Figure 6] FIG. 11 is a diagram showing a second specific example of the machine learning process according to the first embodiment. [Figure 7] FIG. 11 is a diagram showing a third specific example of the machine learning process according to the first embodiment. [Figure 8] FIG. 11 is a block diagram showing an example of a configuration of an image processing system according to a second embodiment. [Figure 9] 10 is a flowchart showing an example of a processing flow in an image processing system according to a second embodiment. [Figure 10] 10 is a flowchart showing an example of the flow of an image generation process according to a second embodiment. [Figure 11] 13 is a flowchart showing an example of the flow of an output integration process according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0013] <<1. First embodiment>> The first embodiment will be described with reference to FIGS.

[0014] <1-1. Image processing system configuration> The configuration of an image processing system according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the configuration of an image processing system according to the first embodiment. The image processing system 1 shown in FIG. 1 is a system that performs processing based on an RGB image using a machine learning model (hereinafter also referred to as an "RGB image processing model") that performs processing based on an RGB image. The processing based on an RGB image is, for example, an image classification process, an estimation process, a prediction process, an image generation process, an image conversion process, and the like. In the image classification process, for example, classification of objects appearing in an RGB image is performed. In the estimation process, for example, estimation regarding objects appearing in an RGB image is performed. In the prediction process, for example, prediction regarding objects appearing in an RGB image is performed. In the image generation process, for example, an image based on an RGB image is generated. In the image conversion process, for example, an RGB image is converted into another image.

[0015] The image processing system 1 is realized by devices such as a PC (Personal Computer), a smartphone, a tablet terminal, a server device, a quantum computer, etc. The functions of the image processing system 1 are realized by causing a CPU (Central Processing Unit) provided as hardware in each device that realizes the image processing system 1 to execute a program. Note that the image processing system 1 may be realized by one device, by multiple devices, or by a combination of multiple devices of different types.

[0016] As shown in FIG. 1, the image processing system 1 includes a data input unit 11, a spectroscopic image estimation unit 12, a data storage unit 13, an image generation unit 14, a machine learning processing unit 15, and an output integration unit 16.

[0017] (1) Data Entry Section 11 The data input unit 11 has a function of performing data input processing. For example, the data input unit 11 acquires image data to be input to the image processing system 1 as an input image, and outputs it to the spectroscopic image estimation unit 12. The input image is image data to be processed, for example, an RGB image of an object. The input image acquired by the data input unit 11 is an image that is the source of data expansion (hereinafter, also referred to as an "original image"). The data input unit 11 may write the acquired input image in the data storage unit 13 for storage.

[0018] (2) Spectral image estimation unit 12 The spectral image estimation unit 12 has a function of performing a spectral image estimation process. The spectral image estimation unit 12 estimates a spectral image having spectral information based on an input image acquired by the data input unit 11. The spectral information is, for example, a spectral reflectance. That is, it can be said that the spectral image estimation unit 12 estimates a spectral reflectance based on the input image. It should be noted that the spectral image is estimated based on the input image, which is an RGB image obtained by photographing an object, and is therefore an image obtained by photographing an object, similar to the input image.

[0019] The RGB values ​​in an RGB image, the spectral reflectance of an object appearing in the RGB image, the light source spectral distribution of the light source used when the RGB image was captured, and the camera spectral sensitivity of the camera that captured the RGB image are related by the following equation (1).

[0020]

number

[0021] Equation (1) shows that the RGB value can be calculated as the integral value of the wavelength (λ) from 400 nm to 700 nm obtained by multiplying each element of the spectral reflectance, the light source spectral distribution, and the camera spectral sensitivity for each wavelength (λ). As can be seen from equation (1), a machine learning model that estimates the spectral reflectance of an object (hereinafter also referred to as a "spectral image estimation model") can be generated using each of the RGB image, the light source spectral distribution, and the camera spectral sensitivity as a set of input data.

[0022] The spectral image estimation unit 12 estimates a spectral image (spectral reflectance) using a spectral image estimation model generated based on, for example, formula (1). The spectral image estimation model is a model that receives an RGB image, a light source spectral distribution, and a camera spectral sensitivity as input, and outputs a spectral image. The spectral image estimation unit 12 inputs the RGB image acquired by the data input unit 11, and the light source spectral distribution and camera spectral sensitivity corresponding to the RGB image to the spectral image estimation model, and acquires a spectral image output from the spectral image estimation model as an estimation result. The spectroscopic image estimation unit 12 outputs the estimated spectroscopic image to the image generation unit 14.

[0023] (3) Data storage unit 13 The data storage unit 13 has a function of storing various information. The data storage unit 13 is configured by a storage medium provided as hardware in the image processing system 1, such as a hard disk drive (HDD), a solid state drive (SSD), a flash memory, an electrically erasable programmable read only memory (EEPROM), a random access read / write memory (RAM), a read only memory (ROM), or any combination of these storage media.

[0024] The data storage unit 13 stores, for example, teacher data, data distribution information of the teacher data, and the like. The teacher data is data used for learning a machine learning model (RGB image processing model). The data distribution information is information indicating the distribution of data related to the teacher data. For example, an image (RGB image) used as teacher data is given a label such as the type of light source or the type (model) of camera when the image was captured. In this case, the data distribution information is information indicating the proportion of each type of light source and the proportion of each type of camera included in the teacher data stored in the data storage unit 13 for each type. The data storage unit 13 may store an input image acquired by the data input unit 11 and a spectroscopic image generated by the spectroscopic image estimation unit 12. The data storage unit 13 may also store data generated or output by the image generation unit 14, the machine learning processing unit 15, and the output integration unit 16, which will be described later.

[0025] (4) Image generation unit 14 The image generating unit 14 has a function of performing image generation processing. The image generating unit 14 generates a plurality of RGB images based on, for example, a spectral image and a plurality of pieces of shooting condition information. The plurality of RGB images generated by the image generating unit 14 are also virtual images virtually generated from the spectral images. The image generating unit 14 generates an RGB image for each of a plurality of shooting conditions from the spectral images estimated by the spectral image estimating unit 12. The plurality of RGB images generated by the image generating unit 14 become input images to be input to the RGB image processing model. That is, the image generating unit 14 performs data expansion of the input image (original image) acquired by the data input unit 11 into a plurality of input images.

[0026] The photographing condition information is information that combines a light source spectral distribution and a camera spectral sensitivity. A plurality of pieces of photographing condition information are prepared so that the combinations of arbitrary light source spectral distributions and arbitrary camera spectral sensitivities are different. The combinations of arbitrary light source spectral distributions and arbitrary camera spectral sensitivities may be selected according to uniformly distributed random numbers, or may be selected according to the distribution ratio of the types of data sets (teaching data) used to train the RGB image processing model, or may be determined according to the ratio of the light source (e.g., lighting) or the penetration rate of cameras in the market.

[0027] In order to generate a plurality of RGB images, the image generating unit 14 first generates an occurrence probability distribution of the light source and the camera based on the data distribution information stored in the data storage unit 13. The occurrence probability distribution is information indicating the distribution of data related to the teacher data, similar to the information indicated by the data distribution information. One method of generating an occurrence probability distribution from data distribution information is to use the data distribution information as the occurrence probability distribution. For example, assume that the data distribution information indicates that 20% of the images are taken with an LED light source, 30% are taken with a fluorescent light, and 50% are taken with sunlight in the teacher data stored in the data storage unit 13. In this case, the image generation unit 14 generates information indicating that the LED light source is 0.2, the fluorescent light is 0.3, and the sunlight is 0.5 as the occurrence probability distribution. Another method for generating an occurrence probability distribution from data distribution information is to use a scatter diagram in a 31-dimensional space. For example, the image generating unit 14 can obtain a scatter diagram of the light source and the camera in the 31-dimensional space by expressing the spectral information as a 31-dimensional vector in 10 nm increments from 400 nm to 700 nm and plotting the light source and the camera in the teacher data in the 31-dimensional space. The image generating unit 14 can obtain the occurrence probability distribution of the light source and the camera in the 31-dimensional space by applying a Gaussian mixture model to this scatter diagram.

[0028] Next, the image generation unit 14 generates a plurality of pieces of shooting condition information based on the generated occurrence probability distribution. The image generation unit 14 selects the spectral information of the light source and the camera to be generated based on random numbers generated with occurrence probabilities according to the occurrence probability distribution of the light source and the camera. For example, when an occurrence probability distribution is defined for labels indicating the type of light source and the type of camera, the image generating unit 14 randomly selects which type of light source and which type of camera to set as the light source and the camera when expanding the data according to the occurrence probability distribution. The image generating unit 14 generates a combination of the light source spectral distribution and the camera spectral sensitivity corresponding to the selected type of light source and the selected type of camera as the shooting condition information. On the other hand, when the occurrence probability distribution of the light source and the camera in the 31-dimensional space is defined, the image generation unit 14 generates shooting condition information by using the random numbers generated according to the occurrence probability distribution as the spectral information of the light source and the camera.

[0029] It should be noted that the image generating unit 14 does not necessarily need to select the light source and the camera at the time of data expansion according to the occurrence probability distribution. The image generating unit 14 can also select the light source and the camera at the time of data expansion according to a uniformly distributed random number. In this case, the spectral information of the light source and the camera at the time of data expansion does not follow the occurrence probability distribution. Even in this case, the output integration unit 16 described later performs output integration processing according to the occurrence probability distribution, so that it is possible to improve the accuracy of the machine learning processing.

[0030] Then, the image generating unit 14 reconstructs (generates) a plurality of RGB images with different combinations using an arbitrary light source spectral distribution and an arbitrary camera spectral sensitivity (photographing condition information) for the spectral image estimated by the spectral image estimating unit 12. The image generation unit 14 outputs the generated multiple RGB images to the machine learning processing unit 15.

[0031] (5) Machine learning processing unit 15 The machine learning processing unit 15 has a function of performing machine learning processing. For example, the machine learning processing unit 15 performs processing based on an RGB image using an RGB image processing model. The RGB image processing model is a model that performs processing based on an RGB image, and is a model that machine-learns the relationship between the RGB image and the processing result. The machine learning processing unit 15 inputs a plurality of RGB images generated by the image generation unit 14 to the RGB image processing model, and obtains a processing result output from the RGB image processing model for each input RGB image. The machine learning processing unit 15 outputs the acquired multiple processing results to the output integration unit 16.

[0032] The RGB image processing model may be any model, such as a uniquely constructed model, an existing publicly available model, or a model constructed by a deep learning method such as a Convolutional Neural Network (CNN), etc. The RGB image processing model according to this embodiment can be realized, for example, by using ResNET.

[0033] (6) Output Integration Unit 16 The output integrating unit 16 has a function of performing output integration processing. For example, the output integrating unit 16 integrates a plurality of processing results obtained by the machine learning processing unit 15 based on a plurality of RGB images, and generates a final processing result.

[0034] In order to integrate multiple processing results, the output integrating unit 16 first generates an appearance probability distribution of the light source and the camera based on the data distribution information stored in the data storage unit 13. The method of generating the appearance probability distribution by the output integrating unit 16 is similar to the method of generating the appearance probability distribution by the image generating unit 14. Next, the output integration unit 16 integrates the multiple processing results based on the generated occurrence probability distribution. For example, the output integration unit 16 calculates a weighted average of the multiple processing results using a weighting function according to the occurrence probability distribution, and sets the calculated weighted average as the final processing result. In this example, calculating the weighted average of the multiple processing results is the integration of the multiple processing results. The output integration unit 16 outputs the generated final processing result.

[0035] In the output integration process, the output integration unit 16 may use the occurrence probability distribution generated by the image generation unit 14 in the image generation process. This allows the output integration unit 16 to omit the process of generating the occurrence probability distribution.

[0036] <1-2. Processing flow> The configuration of the image processing system 1 according to the first embodiment has been described above. Next, the flow of processing according to the first embodiment will be described with reference to Fig. 2 to Fig. 4. Fig. 2 is a flowchart showing an example of the flow of processing in the image processing system 1 according to the first embodiment.

[0037] 2, first, the data input unit 11 performs a data input process (step S11). Specifically, the data input unit 11 acquires an input image to be input to the image processing system 1, and inputs the acquired input image to the spectroscopic image estimation unit 12. Next, the spectral image estimation unit 12 performs a spectral image estimation process (step S12). Specifically, the spectral image estimation unit 12 estimates a spectral image (spectral reflectance) based on the input image acquired by the data input unit 11, and outputs the estimated spectral image to the image generation unit 14. Next, the image generating unit 14 performs an image generating process (step S13). The image generating unit 14 generates a plurality of RGB images based on the spectral images estimated by the spectral image estimating unit 12 and a plurality of pieces of shooting condition information.

[0038] Here, the image generation processing according to the first embodiment will be specifically described with reference to Fig. 3. Fig. 3 is a flowchart showing an example of the flow of the image generation processing according to the first embodiment. As shown in FIG. 3, first, the image generating unit 14 acquires the spectroscopic image estimated by the spectroscopic image estimating unit 12 (step S131). Next, the image generating unit 14 acquires the data distribution information stored in the data storage unit 13 (step S132). Next, the image generating unit 14 generates an appearance probability distribution of the light source and the camera based on the acquired data distribution information (step S133). Next, the image generating unit 14 generates a plurality of pieces of shooting condition information based on the generated occurrence probability distribution (step S134). Then, the image generating unit 14 generates a plurality of RGB images based on the acquired spectral images and the plurality of pieces of generated shooting condition information (step S135).

[0039] 2, the machine learning processing unit 15 performs machine learning processing (step S14). Specifically, the machine learning processing unit 15 inputs the multiple RGB images generated by the image generating unit 14 to an RGB image processing model, obtains multiple processing results output from the RGB image processing model, and outputs the obtained multiple processing results to the output integrating unit 16. Next, the output integrating unit 16 performs an output integration process (step S15). The output integrating unit 16 integrates a plurality of processing results acquired by the machine learning processing unit 15 to generate a final processing result.

[0040] The output integration process according to the first embodiment will now be described in detail with reference to Fig. 4. Fig. 4 is a flowchart showing an example of the flow of the output integration process according to the first embodiment. As shown in FIG. 4, first, the output integration unit 16 acquires a plurality of processing results acquired by the machine learning processing unit 15 (step S151). Next, the output integration unit 16 acquires the data distribution information stored in the data storage unit 13 (step S152). Next, the output integration unit 16 generates an appearance probability distribution of the light source and the camera based on the acquired data distribution information (step S153). Next, the output integration unit 16 generates a weighting function for each of the multiple processing results based on the generated occurrence probability distribution (step S154). Then, the output integration unit 16 calculates a weighted average of the multiple processing results using the generated weighting function, and sets the calculated weighted average as the final processing result (step S155).

[0041] Returning to FIG. 2, the image processing system 1 ends the process.

[0042] <1-3. Specific examples> The above describes the process flow according to the first embodiment. Next, a specific example of the machine learning process according to the first embodiment will be described with reference to Figs.

[0043] (1) First Specific Example A first specific example will be described with reference to Fig. 5. Fig. 5 is a diagram showing a first specific example of the machine learning process according to the first embodiment. In the first specific example, an example will be described in which the machine learning process is an image classification process. The image classification process is performed using an RGB image processing model that can learn the relationship between objects appearing in an input image and their types and classify the types of objects appearing in the input image.

[0044] As shown in Fig. 5, first, the data input unit 11 acquires an input image A1 (original image) in a data input process. In the example shown in Fig. 5, the input image A1 is an image obtained by photographing a mannequin, which is an object. Next, the spectral image estimation unit 12 estimates a spectral image A2 based on the input image A1 in a spectral image estimation process.

[0045] Next, the image generating unit 14 generates a plurality of input images A3 to A5 from the spectral image A2 in an image generation process (i.e., a data expansion process). At this time, the image generating unit 14 generates a plurality of pieces of shooting condition information based on the occurrence probability distribution generated from the data distribution information stored in the data storage unit 13, and uses the information in the image generation process. In the example shown in FIG. 5, shooting condition information B1, which is a combination of the light source a and the camera a, shooting condition information B2, which is a combination of the light source b and the camera b, and shooting condition information B3, which is a combination of the light source b and the camera c, are generated. The image generating unit 14 generates an input image A3 from the spectral image A2 and the shooting condition information B1, generates an input image A4 from the spectral image A2 and the shooting condition information B2, and generates an input image A5 from the spectral image A2 and the shooting condition information B3.

[0046] Next, the machine learning processing unit 15 inputs the input images A3 to A5 to the RGB image processing model in the image classification process, and obtains a plurality of classification results C1 to C3 (processing results) output from the RGB image processing model. The classification result C1 is a classification result corresponding to the input image A3, and indicates that the probability that the object is a mannequin is 0.62, the probability that the object is a person is 0.42, the probability that the object is a cat is 0.02, and the probability that the object is a dog is 0.04. The classification result C2 is a classification result corresponding to the input image A4, and indicates that the probability that the object is a mannequin is 0.52, the probability that the object is a person is 0.32, the probability that the object is a cat is 0.14, and the probability that the object is a dog is 0.02. The classification result C3 is a classification result corresponding to the input image A5, and indicates that the probability that the object is a mannequin is 0.34, the probability that the object is a person is 0.42, the probability that the object is a cat is 0.11, and the probability that the object is a dog is 0.13.

[0047] Next, the output integrating unit 16 integrates the classification results C1 to C3 in an output integration process to generate a final classification result C4. At this time, the output integrating unit 16 generates a plurality of weighting functions based on the occurrence probability distribution generated from the data distribution information stored in the data storage unit 13, and uses them in the output integration process. In the example shown in FIG. 5, a weighting function D1 that sets a weight of 0.6 for the classification result C1, a weighting function D2 that sets a weight of 0.3 for the classification result C2, and a weighting function D3 that sets a weight of 0.1 for the classification result C3 are generated. The output integrating unit 16 calculates a weighted average based on the classification results C1 to C3 and the weighting functions D1 to D3 to generate a final classification result C4.

[0048] (2) Second Specific Example A second specific example will be described with reference to Fig. 6. Fig. 6 is a diagram showing a second specific example of the machine learning process according to the first embodiment. In the second specific example, an example will be described in which the machine learning process is an estimation process. The estimation process is performed using an RGB image processing model that can learn the relationship between an object appearing in an input image and its price, and estimate the price of the object appearing in the input image.

[0049] In the example shown in FIG. 6, similarly to the example shown in FIG. 5, a data input process, a spectral image estimation process, and an image generation process are performed. After the image generation process, the machine learning processing unit 15 inputs the input images A3 to A5 to the RGB image processing model in the estimation process, and obtains a plurality of estimation results C5 to C7 (processing results) output from the RGB image processing model. The estimation result C5 is an estimation result corresponding to the input image A3, and indicates that the price of the object shown in the input image A3 is estimated to be 1,500 yen. The estimation result C6 is an estimation result corresponding to the input image A4, and indicates that the price of the object shown in the input image A4 is estimated to be 1,700 yen. The estimation result C6 is an estimation result corresponding to the input image A5, and indicates that the price of the object shown in the input image A5 is estimated to be 1,000 yen.

[0050] Next, the output integrating unit 16 integrates the estimation results C5 to C7 in an output integration process to generate a final estimation result C8. At this time, the output integrating unit 16 generates a plurality of weighting functions to be used in the output integration process, similar to the example shown in Fig. 5. The output integrating unit 16 calculates a weighted average based on the estimation results C5 to C7 and the weighting functions D1 to D3 to generate the final estimation result C8.

[0051] (3) Third Specific Example A third specific example will be described with reference to Fig. 7. Fig. 7 is a diagram showing a third specific example of the machine learning process according to the first embodiment. In the third specific example, an example will be described in which the machine learning process is an image generation process. The image generation process is performed using an RGB image processing model that is capable of learning the relationship between an input image and an image to be generated, and generating the image to be generated from the input image.

[0052] In the example shown in FIG. 7, similarly to the example shown in FIG. 5, a data input process, a spectroscopic image estimation process, and an image generation process are performed. After the image generation process, the machine learning processing unit 15 inputs the input images A3 to A5 to the RGB image processing model in the image generation process, and obtains a plurality of generation results C9 to C11 (processing results) output from the RGB image processing model. The generation result C9 is a generation result corresponding to the input image A3, and shows the image to be generated from the input image A3. The generation result C10 is a generation result corresponding to the input image A4, and shows the image to be generated from the input image A4. The generation result C11 is a generation result corresponding to the input image A5, and shows the image to be generated from the input image A5.

[0053] Next, the output integrating unit 16 integrates the generation results C9 to C11 in an output integration process to generate a final generation result C12. At this time, the output integrating unit 16 generates a plurality of weighting functions to be used in the output integration process, similar to the example shown in Fig. 5. The output integrating unit 16 calculates a weighted average based on the generation results C9 to C11 and the weighting functions D1 to D3, and generates the final generation result C12.

[0054] (4) Fourth Specific Example A fourth specific example will be described. In the fourth specific example, an example will be described in which the machine learning process is an image conversion process. The image conversion process is performed using an RGB image processing model that can learn the relationship between an input image and an image to be converted and generate an image to be converted from the input image. Details of the image conversion process are the same as those of the image generation process described as the third specific example, and therefore will not be described here. In the image conversion process by the image processing system 1, for example, a process such as an img2img task in a Generative Adversarial Network (GAN) (for example, converting an image of a horse into an image of a zebra) can be performed in a similar manner.

[0055] As described above, the image processing system 1 according to the first embodiment includes an image generation unit 14 that generates a plurality of RGB images based on a spectral image of an object and a plurality of pieces of shooting condition information having different combinations of light source spectral distributions and camera spectral sensitivities, a machine learning processing unit 15 that inputs the generated plurality of RGB images to an RGB image processing model that performs processing based on the RGB images and obtains a processing result output from the RGB image processing model for each input RGB image, and an output integration unit 16 that integrates a plurality of processing results obtained based on the plurality of RGB images to generate a final processing result.

[0056] With this configuration, one output result can be obtained from multiple inputs, so multiple images obtained by data augmentation of the original image can be used as inputs as is. This makes re-learning using multiple images obtained by data augmentation unnecessary. In the case of a configuration in which one output result is obtained from one input as in conventional methods, re-learning using multiple images obtained by data augmentation was necessary. In addition, it is possible to obtain a final processing result that takes into account multiple processing results, resulting in a more accurate processing result compared to the conventional method of obtaining one processing result from one image. Therefore, the image processing system 1 according to the first embodiment makes it possible to improve the processing accuracy in processing using an existing machine learning model without performing re-learning through data expansion.

[0057] <<2. Second embodiment>> Having described the first embodiment, the second embodiment will now be described with reference to Figs. In the first embodiment described above, an example has been described in which an occurrence probability distribution is generated based on data distribution information stored in the data storage unit 13 in the image generation process by the image generation unit 14 and the output integration process by the output integration unit 16, but the present invention is not limited to such an example. In the second embodiment, an example will be described in which an occurrence probability distribution is estimated from an input image when data distribution information is unknown (for example, not stored in the data storage unit 13). The image processing system according to the second embodiment has some functions different from those of the image processing system 1 according to the first embodiment, and will be described with a different reference numeral 1a. In the following description, the description of the contents that overlap with those described in the first embodiment will be omitted as appropriate.

[0058] <2-1. Image processing system configuration> The configuration of an image processing system 1a according to the second embodiment will be described with reference to Fig. 8. Fig. 8 is a block diagram showing an example of the configuration of an image processing system 1a according to the second embodiment. As shown in FIG. 8, the image processing system 1a includes a data input unit 11, a spectroscopic image estimation unit 12, a data storage unit 13, an image generation unit 14, a machine learning processing unit 15, an output integration unit 16, and an occurrence probability estimation unit 17.

[0059] (1) Data Entry Section 11 The data input unit 11 according to the second embodiment acquires an input image in the same manner as the data input unit 11 according to the first embodiment. The data input unit 11 according to the second embodiment outputs the acquired input image not only to the spectroscopic image estimation unit 12 but also to the occurrence probability estimation unit 17.

[0060] (2) Spectral image estimation unit 12 The spectral image estimation unit 12 according to the second embodiment estimates a spectral image in the same manner as the spectral image estimation unit 12 according to the first embodiment, and outputs the estimated spectral image to the image generation unit .

[0061] (3) Data storage unit 13 The data storage unit 13 according to the second embodiment does not store data distribution information. On the other hand, the data storage unit 13 according to the second embodiment stores information other than the data distribution information in the same manner as the data storage unit 13 according to the first embodiment.

[0062] (4) Image generation unit 14 The image generating unit 14 according to the second embodiment generates a plurality of RGB images in the same manner as the image generating unit 14 according to the first embodiment. However, the image generating unit 14 according to the second embodiment does not generate an occurrence probability distribution based on data distribution information because it cannot acquire data distribution information from the data storage unit 13. Instead, the image generating unit 14 according to the second embodiment acquires an occurrence probability distribution estimated by an occurrence probability estimation unit 17 and uses it to generate shooting condition information.

[0063] (5) Machine learning processing unit 15 The machine learning processing unit 15 according to the second embodiment acquires a plurality of processing results and outputs them to the output integration unit 16 in the same manner as the machine learning processing unit 15 according to the first embodiment.

[0064] (6) Output Integration Unit 16 The output integrating unit 16 according to the second embodiment integrates a plurality of processing results to generate a final processing result, in the same manner as the output integrating unit 16 according to the first embodiment. However, the output integrating unit 16 according to the second embodiment cannot acquire data distribution information from the data storage unit 13, and therefore does not generate an occurrence probability distribution based on the data distribution information. Instead, the output integrating unit 16 according to the second embodiment acquires an occurrence probability distribution estimated by an occurrence probability estimation unit 17, and uses it to generate a weighting function.

[0065] (7) Occurrence Probability Estimation Unit 17 The occurrence probability estimation unit 17 has a function of performing occurrence probability distribution estimation processing. The occurrence probability estimation unit 17 estimates the occurrence probability distribution based on the original image. For example, the occurrence probability estimation unit 17 estimates the occurrence probability distribution related to the type of light source and camera using the RGB image (original image) acquired by the data input unit 11 as input. The occurrence probability estimation unit 17 estimates the occurrence probability distribution from the RGB image by using a neural network that has machine-learned the relationship between the RGB image and the occurrence probability distribution of the light source and the camera. When estimating a spectral image from an RGB image acquired by the data input unit 11, the RGB image input to the spectral image estimation unit 12 and the RGB image input to the occurrence probability estimation unit 17 may be the same image. Note that the spectral image may be a spectral image captured by a hyperspectral camera, instead of a spectral image estimated from an RGB image by the spectral image estimation unit 12. In this case, the RGB image input to the occurrence probability estimation unit 17 is an RGB image of an object captured by a smartphone camera or the like under the same circumstances as when the spectral image was captured by the hyperspectral camera.

[0066] In the output integration process in the output integration unit 16, using the occurrence probability distribution estimated by the occurrence probability estimation unit 17 is particularly effective for light sources. For example, in the case of a machine learning method in which images taken outdoors are input, it is expected that many images taken outdoors will also be included in the learning data. Therefore, the occurrence probability estimation unit 17 estimates the occurrence probability so that the occurrence probability of sunlight is high, so that the weight of the processing result from the virtual image generated using sunlight as the light source in the image generation process (data augmentation process) in the image generation unit 14 is high.

[0067] <2-2. Processing flow> The configuration of the image processing system 1a according to the second embodiment has been described above. Next, the flow of processing according to the second embodiment will be described with reference to Fig. 9 to Fig. 11. Fig. 9 is a flowchart showing an example of the flow of processing in the image processing system 1a according to the second embodiment.

[0068] 9, first, the data input unit 11 performs a data input process (step S21). Specifically, the data input unit 11 acquires an input image to be input to the image processing system 1a, and inputs the acquired input image to the spectral image estimation unit 12 and the occurrence probability estimation unit 17. Next, the spectral image estimation unit 12 performs a spectral image estimation process (step S22). Specifically, the spectral image estimation unit 12 estimates a spectral image (spectral reflectance) based on the input image acquired by the data input unit 11, and outputs the estimated spectral image to the image generation unit 14. Next, the occurrence probability estimation unit 17 performs an occurrence probability distribution estimation process (step S23). Specifically, the occurrence probability estimation unit 17 estimates the occurrence probability distribution of the light source and the camera based on the input image acquired by the data input unit 11, and outputs the estimated occurrence probability distribution to the image generation unit 14 and the output integration unit 16. Next, the image generating unit 14 performs an image generating process (step S24). The image generating unit 14 generates a plurality of RGB images based on the spectral images estimated by the spectral image estimating unit 12 and a plurality of pieces of shooting condition information.

[0069] Here, the image generation processing according to the second embodiment will be specifically described with reference to Fig. 10. Fig. 10 is a flowchart showing an example of the flow of the image generation processing according to the second embodiment. As shown in FIG. 10, first, the image generating unit 14 acquires the spectroscopic image estimated by the spectroscopic image estimating unit 12 (step S241). Next, the image generating unit 14 acquires the appearance probability distribution of the light source and the camera estimated by the appearance probability estimating unit 17 (step S242). Next, the image generating unit 14 generates a plurality of pieces of shooting condition information based on the acquired occurrence probability distribution (step S243). Then, the image generating unit 14 generates a plurality of RGB images based on the acquired spectral images and the plurality of pieces of generated shooting condition information (step S244).

[0070] 9, the machine learning processing unit 15 performs machine learning processing (step S25). Specifically, the machine learning processing unit 15 inputs a plurality of RGB images generated by the image generating unit 14 to an RGB image processing model, obtains a plurality of processing results output from the RGB image processing model, and outputs the obtained plurality of processing results to the output integrating unit 16. Next, the output integrating unit 16 performs an output integration process (step S26). The output integrating unit 16 integrates a plurality of processing results acquired by the machine learning processing unit 15, and generates a final processing result.

[0071] The output integration process according to the second embodiment will now be described in detail with reference to Fig. 11. Fig. 11 is a flowchart showing an example of the flow of the output integration process according to the second embodiment. As shown in FIG. 11, first, the output integration unit 16 acquires a plurality of processing results acquired by the machine learning processing unit 15 (step S261). Next, the output integration unit 16 acquires the appearance probability distribution of the light source and the camera estimated by the appearance probability estimation unit 17 (step S262). Next, the output integration unit 16 generates a weighting function for each of the multiple processing results based on the obtained occurrence probability distribution (step S263). Then, the output integration unit 16 calculates a weighted average of the multiple processing results using the generated weighting function, and sets the calculated weighted average as the final processing result (step S264).

[0072] Returning to FIG. 9, the image processing system 1a ends the process.

[0073] <2-3. Specific Examples> The above describes the flow of the process according to the second embodiment. Next, a specific example of the machine learning process according to the second embodiment will be described. A specific example of the machine learning process according to the second embodiment is the same as the specific example of the machine learning process according to the first embodiment. Since the content of the specific example is similar, the description thereof will be omitted.

[0074] <<3. Modifications>> The embodiments of the present invention have been described above. Next, modified examples of the above-mentioned embodiments will be described. Each modified example described below may be applied to each embodiment alone, or may be applied to each embodiment in combination. Furthermore, each modified example may be applied in place of the configuration described in each embodiment, or may be applied in addition to the configuration described in each embodiment.

[0075] In the above-described first embodiment, an example has been described in which the image generating unit 14 generates shooting condition information based on the light source type and the camera type indicated by the labels added to the teacher data, but the present invention is not limited to such an example. The light source type and the camera type targeted by the shooting condition information may include light source types and camera types that are not included in the teacher data. When an RGB image generated by the image generation unit 14 based on a light source type and a camera type not included in the teacher data is input to the RGB image processing model, there is a possibility that the accuracy of the processing result output from the RGB image processing model will decrease. However, when the output integration unit 16 generates a weighting function in the output integration process, a low weighting function is generated for the output result based on a light source type and a camera type not included in the teacher data, so that it is possible to prevent a significant decrease in the accuracy of the final processing result after integration.

[0076] In addition, in the above-described first embodiment, an occurrence probability distribution generated based on data distribution information stored in the data storage unit 13 is used, and in the above-described second embodiment, an occurrence probability distribution estimated by the occurrence probability estimation unit 17 is used. However, the present invention is not limited to such an example. For example, the embodiment may use both the occurrence probability distribution generated based on the data distribution information stored in the data storage unit 13 and the occurrence probability distribution estimated by the occurrence probability estimation unit 17, or may select and use one of the two occurrence probability distributions. As an example, in the second embodiment, it is assumed that the data distribution information is stored in the data storage unit 13, and the image generation unit 14 and the output integration unit 16 also have a function of generating an occurrence probability distribution based on the data distribution information. In this case, it is possible to realize an embodiment in which the image generation unit 14 and the output integration unit 16 use both the occurrence probability distribution generated by themselves and the occurrence probability distribution estimated by the occurrence probability estimation unit 17, or an embodiment in which one of the two occurrence probability distributions is selected and used.

[0077] The embodiment of the present invention has been described above. In addition, a part or the whole of the image processing system 1, 1a in the above-mentioned embodiment may be realized by a computer. In that case, a program for realizing this function may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to realize the function. In addition, the "computer system" here includes hardware such as an OS and peripheral devices. In addition, the "computer-readable recording medium" refers to a portable medium such as a flexible disk, an optical magnetic disk, a ROM, a CD-ROM, and a storage device such as a hard disk built into a computer system. Furthermore, the "computer-readable recording medium" may include a device that dynamically holds a program for a short period of time, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, and a device that holds a program for a certain period of time, such as a volatile memory inside a computer system that is a server or a client in that case. In addition, the above-mentioned program may be a program for realizing a part of the above-mentioned function, or may be a program that can realize the above-mentioned function in combination with a program already recorded in the computer system, or may be a program that is realized using a programmable logic device such as an FPGA (Field Programmable Gate Array).

[0078] Although the embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design changes, etc. are possible within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0079] 1, 1a... image processing system, 11... data input unit, 12... spectroscopic image estimation unit, 13... data storage unit, 14... image generation unit, 15... machine learning processing unit, 16... output integration unit, 17... occurrence probability estimation unit

Claims

1. an image generating unit that generates a plurality of RGB images based on a spectral image of an object and a plurality of pieces of photographing condition information having different combinations of a light source spectral distribution and a camera spectral sensitivity; a machine learning processing unit that inputs the generated RGB images to a machine learning model that performs processing based on RGB images, and obtains a processing result output from the machine learning model for each of the input RGB images; an output integration unit that integrates a plurality of processing results obtained based on the plurality of RGB images to generate a final processing result; An image processing system comprising:

2. The output integration unit integrates the plurality of processing results based on an appearance probability distribution of a light source and a camera. The image processing system according to claim 1 .

3. A data storage unit that stores data distribution information of data used for learning the machine learning model; Further equipped with The output integration unit generates the occurrence probability distribution based on the data distribution information. The image processing system according to claim 2 .

4. the output integration unit calculates a weighted average of the plurality of processing results using a weighting function according to the occurrence probability distribution, and sets the calculated weighted average as the final processing result.

4. The image processing system according to claim 2.

5. The image generation unit generates a plurality of pieces of shooting condition information based on an appearance probability distribution of a light source and a camera.

3. The image processing system according to claim 1.

6. A data storage unit that stores data distribution information of data used for learning the machine learning model; Further equipped with The image generation unit generates the occurrence probability distribution based on the data distribution information. The image processing system according to claim 5 .

7. an occurrence probability estimation unit that estimates the occurrence probability distribution based on an original image that is an RGB image showing the object; The image processing system of claim 2 further comprising:

8. an image generation process for generating a plurality of RGB images based on a spectral image of an object and a plurality of pieces of photographing condition information having different combinations of a light source spectral distribution and a camera spectral sensitivity; a machine learning processing step of inputting the generated RGB images into a machine learning model that performs processing based on RGB images, and obtaining a processing result output from the machine learning model for each of the input RGB images; an output integration step of integrating a plurality of processing results obtained based on the plurality of RGB images to generate a final processing result; 23. A computer-implemented image processing method comprising:

9. Computer, an image generating means for generating a plurality of RGB images based on a spectral image of an object and a plurality of pieces of photographing condition information having different combinations of a light source spectral distribution and a camera spectral sensitivity; a machine learning processing means for inputting the generated RGB images into a machine learning model that performs processing based on RGB images, and obtaining a processing result output from the machine learning model for each of the input RGB images; an output integration unit that integrates a plurality of processing results obtained based on the plurality of RGB images to generate a final processing result; A program to function as a

Images