Learned model generation device, information processing device, learned model generation method, information processing method, learned model generation program, and information processing program

Abstract

To perform color correction similar to a color correction method using a color chart on a subject without imaging the color chart together with the subject.SOLUTION: An information processing device 10 acquires a learning image including a subject for learning and a color chart. The information processing device 10 calculates a correction coefficient for correcting a color included in the color chart within the learning image, to a reference color as a reference. The information processing device 10 generates learning data obtained by associating an image of a subject portion for learning within the learning image with the correction coefficient. The information processing device 10 generates a learned model of outputting a correction coefficient for correcting the color of an image when the image including the subject is input.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The disclosed technology relates to a trained model generation device, an information processing device, a trained model generation method, an information processing method, a trained model generation program, and an information processing program. [Background technology]

[0002] Patent Document 1 discloses a tongue color correction method based on a deep neural network. Specifically, Patent Document 1 discloses a technology for using a deep neural network to resolve color shifts and color distortions of tongue images generated by tongue images captured on a mobile device. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Chinese Patent Application Publication No. 115482160 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology disclosed in Patent Document 1 is a technology for performing tongue color correction using a deep neural network. As also disclosed in Patent Document 1, the color of an image obtained by photographing a subject is significantly affected by the lighting environment at the time of photographing. This poses a problem in that the color of the subject captured in the image differs from its original color, making it difficult to perform analysis using the subject's original color information. For this reason, a method is generally adopted in which a color chart is used as reference information during photography, and color correction is performed on the subject based on the reference, thereby restoring the color of the subject captured in the image to its original color.

[0005] Conventional methods using a color chart require capturing an image of the subject along with the color chart. Specifically, an image of the subject and the color chart is acquired, and the color of the color chart in the image is converted to a reference color, thereby correcting the color of the subject in the image.

[0006] However, when using the above-mentioned conventional method, it is necessary to simultaneously photograph the color target every time a subject is photographed, which is a cumbersome process.

[0007] The disclosed technology has been made in consideration of the above circumstances, and provides a trained model generation device, an information processing device, a trained model generation method, an information processing method, a trained model generation program, and an information processing program that can perform color correction on a subject similar to the color correction method using a color chart, without having to photograph a color chart together with the subject. [Means for solving the problem]

[0008] In order to achieve the above object, a first aspect of the present disclosure is a trained model generation device including a training acquisition unit that acquires training images that include a training subject and a color chart; a calculation unit that calculates a correction coefficient that corrects the color of the color chart in the training image to a standard reference color; a training data generation unit that generates training data that associates an image of the training subject portion in the training image with the correction coefficient; and a trained model generation unit that generates a trained model that, when an image of a subject is input, outputs a correction coefficient that corrects the color of the image, based on the training data.

[0009] A second aspect of the present disclosure is a trained model generation method in which a computer executes the following processes: acquiring a training image containing a training subject and a color chart; calculating a correction coefficient for correcting the color of the color chart in the training image to a standard reference color; generating training data that associates an image of the training subject portion in the training image with the correction coefficient; and generating a trained model based on the training data that, when an image containing a subject is input, outputs a correction coefficient for correcting the color of the image.

[0010] A third aspect of the present disclosure is a trained model generation program for causing a computer to execute a process of acquiring a training image containing a training subject and a color chart, calculating a correction coefficient for correcting the color of the color chart in the training image to a standard reference color, generating training data that associates an image of the training subject portion in the training image with the correction coefficient, and generating a trained model that, when an image containing a subject is input, outputs a correction coefficient for correcting the color of the image, based on the training data.

[0011] A fourth aspect of the present disclosure is an information processing device that includes: an acquisition unit that acquires an image containing a target subject; and a correction unit that inputs the image acquired by the acquisition unit into a pre-generated trained model to acquire a correction coefficient output from the trained model and corrects the color of the image using the correction coefficient, wherein the trained model is a trained model that outputs a correction coefficient for correcting the color of an image containing a subject when the image is input, and is a trained model that has been trained in advance based on training data in which a training subject and the training correction coefficient are associated, and the training correction coefficient is a correction coefficient that corrects the color of the color chart in a training image containing the training subject and a color chart to a standard reference color.

[0012] A fifth aspect of the present disclosure is an information processing method in which a computer performs processing to acquire an image containing a target subject, input the acquired image into a pre-generated trained model to acquire a correction coefficient output from the trained model, and correct the color of the image using the correction coefficient, wherein the trained model is a trained model that outputs a correction coefficient for correcting the color of an image containing a subject when the image is input, and is a trained model that has been trained in advance based on training data in which the training subject and the training correction coefficient are associated, and the training correction coefficient is a correction coefficient that corrects the color of the color chart in a training image containing the training subject and a color chart to a standard reference color.

[0013] A sixth aspect of the present disclosure is an information processing program for causing a computer to execute a process of acquiring an image containing a target subject, inputting the acquired image into a pre-generated trained model to acquire a correction coefficient output from the trained model, and correcting the color of the image using the correction coefficient, wherein the trained model is a trained model that outputs a correction coefficient for correcting the color of an image containing a subject when the image is input, and is a trained model that has been trained in advance based on training data in which the training subject and the training correction coefficient are associated, and the training correction coefficient is a correction coefficient that corrects the color of the color chart in a training image containing the training subject and a color chart to a standard reference color. [Effects of the Invention]

[0014] According to the disclosed technology, it is possible to obtain an effect that color correction similar to the color correction method using a color chart can be performed on a subject without photographing a color chart together with the subject. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 1 illustrates an example of a schematic configuration of an information processing apparatus according to an embodiment. [Figure 2]FIG. 10 is a diagram for explaining color correction using a color chart. [Figure 3] FIG. 10 is a diagram for explaining an increase in learning data. [Figure 4] FIG. 1 is a diagram for explaining a trained model according to an embodiment. [Figure 5] FIG. 1 illustrates an example of a computer that constitutes an information processing apparatus. [Figure 6] FIG. 10 is a diagram illustrating an example of a trained model generation process executed by the information processing apparatus according to the embodiment. [Figure 7] FIG. 2 illustrates an example of information processing executed by the information processing apparatus according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0016] Hereinafter, embodiments of the disclosed technology will be described in detail with reference to the drawings.

[0017] <Information processing device of embodiment>

[0018] Fig. 1 shows an information processing device 10 according to an embodiment. As shown in Fig. 1, the information processing device 10 functionally includes a data storage unit 20, a learning data acquisition unit 22, a calculation unit 24, a learning data generation unit 26, a learning data storage unit 28, a trained model generation unit 30, a trained model storage unit 32, an acquisition unit 34, a correction unit 36, and an output unit 38. The information processing device 10 is realized by a computer as described below.

[0019] It is assumed that the lighting environment when a subject is photographed will vary. Therefore, in order to grasp the true color of the subject, it is necessary to correct the color of the image when the subject is photographed. A known method for correcting the color of an image is a color correction method using a color chart.

[0020] FIG. 2 is a diagram illustrating color correction using a color chart. In this embodiment, consider the case of correcting the color of an image that shows a tongue, which is an example of a subject. When correcting the color of an image using a color chart, as shown in FIG. 2, a color chart C1 is also photographed when an image IM1 of the tongue is photographed. Then, the color of image IM1 is corrected using a correction coefficient that converts the color of color chart C1 in image IM1 into a reference color that serves as a standard. Specifically, IM2 shown in FIG. 2 is the image after color correction, and the color of color chart C2 in image IM2 is the reference color.

[0021] However, as mentioned above, there is a problem in that the method of simultaneously photographing a subject and a color target is cumbersome. Therefore, the information processing device 10 of this embodiment uses a machine learning model to acquire correction coefficients for color correction of the image. This will be specifically described below.

[0022] A plurality of learning images, each of which shows a tongue as a learning subject and a color chart, are stored in the data storage unit 20. Note that, hereinafter, these learning images will be referred to as first learning images.

[0023] The learning acquisition unit 22 reads out a plurality of first learning images stored in the data storage unit 20, thereby acquiring a plurality of first learning images.

[0024] The calculation unit 24 uses a known method to calculate a correction coefficient for correcting the color appearing on the color chart in the first learning image to the reference color that serves as the standard.

[0025] Specifically, first, the calculation unit 24 extracts a color chart area from the first training image using a known image processing method. For example, the calculation unit 24 extracts a color chart area from the first training image using a trained model for color chart extraction that outputs 1 for color chart areas and 0 for areas other than the color chart areas for the input image. Note that this trained model for color chart extraction can be constructed using known machine learning technology.

[0026] Next, the calculation unit 24 separates each panel (the multiple square regions shown in FIG. 2) in the color chart using a known image processing method, and then calculates a correction coefficient using a known method to convert the color of each panel in the color chart into a standard reference color.

[0027] For example, consider the case where a color c=(R,G,B) in an image is corrected to a reference color c'=(R',G',B'). Note that each of R, G, and B corresponds to a pixel value of 0 to 255, for example. In this case, color c can be corrected to reference color c' using the following linear conversion formula (1):

[0028] c'=Wc+b (1)

[0029] In the above formula (1), W is a matrix and b is a vector. In this embodiment, a correction coefficient vector w, which is a combination of each component of the matrix W and each component of b, is defined as follows: Note that the following correction coefficient vector w corresponds to one point in the correction coefficient vector space. As shown below, the correction coefficient vector is a 12-dimensional vector, and therefore the correction coefficient vector space is a 12-dimensional space.

[0030] JPEG2025181449000002.jpg11134

[0031] Note that when the color c of each pixel in an image is corrected to a reference color c' using a correction coefficient vector w with a large number of components (12 components) as described above, the large number of variables may result in the correction to a color different from the actual reference color. Therefore, in order to improve the accuracy of color correction, in this embodiment, the dimensions of the correction coefficient vector w are reduced to generate an encoding correction coefficient vector w' shown below. Note that the encoding correction coefficient vector w' corresponds to one point in the encoding correction coefficient vector space. The encoding correction coefficient vector space is a five-dimensional space.

[0032] JPEG2025181449000003.jpg1067

[0033] When converting the correction coefficient vector w to the encoding correction coefficient vector w', the correction coefficient vector w obtained when converting the encoding correction coefficient vector w' back to the correction coefficient vector w is C and the original correction coefficient vector w' is used. Examples of the conversion function include a principal component analysis or autoencoder type neural network, which will be described later. In the following, a case where the conversion function is a principal component analysis or autoencoder type neural network will be described as an example, but the present invention is not limited to this. Any conversion function may be used as long as it can appropriately convert the correction coefficient vector w into the encoded correction coefficient vector w'.

[0034] For example, consider a case where principal component analysis is used as the above-mentioned transformation function. In this case, principal component analysis is performed on the distribution of multiple correction coefficient vectors w corresponding to a group of images with color charts collected in advance. Then, based on a transformation matrix U in which the eigenvectors of the main principal component axes obtained by the principal component analysis are arranged, an operation of w' = Uw is performed to calculate an encoded correction coefficient vector w' corresponding to the original correction coefficient vector w.

[0035] Alternatively, an existing autoencoder neural network may be used as the above-mentioned transformation function. An autoencoder neural network consists of two layers: an encode layer that converts the input into a low-dimensional vector and a decode layer that decodes it back to the original dimension. In this case, when a correction coefficient vector w is input to a trained autoencoder neural network that maximizes the decoding accuracy for the entire training image set, an encoding correction coefficient vector w' is obtained from the encode layer of the autoencoder neural network.

[0036] The above five-dimensional encoding correction coefficient vector w' is just an example, and the number of dimensions may be adjusted depending on the color distribution of the target image group.

[0037] Furthermore, the learning data generation unit 26 increases the learning data. Fig. 3 is a diagram for explaining the increase in learning data. The coordinate space in Fig. 3 represents the color space.

[0038] As shown in Fig. 3, the learning data generation unit 26 corrects the color of the image IM1, which is the first learning image, using the encoding correction coefficient vector w' described above, to generate a corrected image IM2. As shown in Fig. 3, by assigning each component of the encoding correction coefficient vector w' to each component of the matrix W and each component of the vector b in the above equation (1) and then calculating the above equation (1), the color c of the first learning image IM1 is corrected to the reference color c', and a corrected image IM2 is generated.

[0039] More specifically, first, according to the above formula (1), a correction coefficient vector w is calculated that minimizes the error between the color c' and the reference color c for all panels in the color chart appearing in the image IM1, which is the first training image. Next, as described above, an encoding correction coefficient vector w' corresponding to the correction coefficient vector w is calculated using a transformation function such as principal component analysis or an autoencoder-type neural network. Then, as shown in FIG. 3, the encoding correction coefficient vector w' is used to perform the transformation process shown in the above formula (1), thereby correcting the color c of the first training image IM1 to the reference color c', and a corrected image IM2 is generated.

[0040] Next, the learning data generation unit 26 generates a plurality of second learning images IM3, IM4, and IM5 by changing the color of the corrected image IM2. At this time, the learning data generation unit 26 generates the plurality of second learning images so that the pixel values ​​of the plurality of first learning images fall within the distribution ranges of the pixel values ​​of the plurality of first learning images.

[0041] Specifically, by changing the encoding correction coefficient vector w' shown in FIG. 3 using random numbers, multiple teacher transformation vectors w t i ' is generated. i is the index for identifying the teacher transformation vector. Next, the teacher transformation vector w t i Each component of ' is assigned to each component of matrix W and each component of b in equation (1) above. Then, color conversion is performed on the corrected image IM2 according to the following equation (2). The following equation (2) converts the reference color c', which is the color of the corrected image IM2, into color c. This color c corresponds to the color of the image obtained in an actual lighting environment and is also a pseudo color.

[0042] c=W -1 (c'-b) (2)

[0043] Multiple different teacher transformation vectors w t i By performing the above transformation process on each of the training vectors w ′, the second training images IM3, IM4, and IM5 shown in FIG. 3 are generated. t i The probability distribution of the generation of ' is generated so as to match the probability distribution within the distribution range D of the sample group of the first learning images. For this reason, as shown in FIG. 3, the learning data generation unit 26 generates a plurality of second learning images IM3, IM4, IM5 within the distribution range D in the color space of the sample group of the plurality of first learning images. Specifically, as shown in FIG. 3, the teacher transformation vector w t 1 ' to change the color of the corrected image IM2, and a second training image IM3 is generated. t 2A second learning image IM5 is generated by changing the color of the corrected image IM2 using '. This generates a plurality of second learning images IM3, IM4, IM5 that are similar to the color variations of actual images taken under various light sources. Each of these plurality of second learning images IM3, IM4, IM5 is a pseudo image, and as will be described later, can be used as learning data.

[0044] In addition, the teacher transformation vector w t 1 ' is also an encoding correction coefficient vector for converting the color of the second learning image IM3 into the color of the corrected image IM2. Similarly, the teacher conversion vector w t 2 ' is also an encoding correction coefficient vector for converting the color of the second learning image IM5 into the color of the corrected image IM2. Therefore, each of the multiple second learning images and the multiple teacher conversion vectors w t i Pairs of each of the plurality of first learning images and a plurality of encoding correction coefficient vectors w′ are used as second learning data, as will be described later. Furthermore, pairs of each of the plurality of first learning images and a plurality of encoding correction coefficient vectors w′ are used as first learning data, as will be described later.

[0045] For this reason, for simplicity of explanation, hereinafter, the encoding correction coefficient vector w′ used to convert the color of the first learning image into the color of the corrected image will also be simply referred to as the first correction coefficient, and the teacher conversion vector w′ used to convert the color of the second learning image into the color of the corrected image will also be simply referred to as the first correction coefficient. t ' is also simply referred to as the second correction coefficient.

[0046] The learning data generation unit 26 generates first learning data for each of a plurality of first learning images, associating the first learning image with a first correction coefficient. The learning data generation unit 26 also generates second learning data for each of a plurality of second learning images, associating the second learning image with a second correction coefficient. The first correction coefficient and the second correction coefficient are pre-calculated values ​​and also serve as training data.

[0047] The learning data storage unit 28 stores the first learning data and the second learning data generated by the learning data generation unit 26.

[0048] The trained model generation unit 30 uses a known machine learning algorithm based on the first training data and the second training data stored in the training data storage unit 28 to generate a trained model that outputs a correction coefficient for correcting the color of an image when an image showing a tongue is input. The trained model is, for example, a known neural network model. When generating the trained model, training is performed to minimize the error between the correction coefficient output from the model and the training data for the correction coefficient.

[0049] 4 is a diagram for explaining the trained model of this embodiment. As shown in Fig. 4, when an image showing a tongue is input to the trained model of this embodiment, a correction coefficient for correcting the color of the image is output.

[0050] The trained model storage unit 32 stores the trained model generated by the trained model generation unit 30.

[0051] The acquisition unit 34 acquires an image of a target tongue. This image is different from the first learning image and the second learning image, and is an image to be subjected to color correction.

[0052] The correction unit 36 ​​inputs the image acquired by the acquisition unit 34 into the trained model stored in the trained model storage unit 32, thereby acquiring correction coefficients output from the trained model. Then, the correction unit 36 ​​corrects the color of the image using the correction coefficients.

[0053] The output unit 38 outputs the image whose color has been corrected by the correction unit 36 ​​as a result.

[0054] The user operating the information processing device 10 checks the output result and confirms the color of the tongue shown in the image.

[0055] The information processing device 10 can be realized, for example, by a computer 50 shown in Fig. 5. The computer 50 includes a CPU 51, a memory 52 as a temporary storage area, and a non-volatile storage unit 53. The computer 50 also includes an input / output interface (I / F) 54 to which external devices, output devices, etc. are connected, and a read / write (R / W) unit 55 that controls reading and writing of data from and to a recording medium. The computer 50 also includes a network I / F 56 that is connected to a network such as the Internet. The CPU 51, memory 52, storage unit 53, input / output I / F 54, R / W unit 55, and network I / F 56 are connected to one another via a bus 57.

[0056] The storage unit 53 can be realized by a hard disk drive (HDD), a solid state drive (SSD), a flash memory, etc. The storage unit 53 as a storage medium stores a program for causing the computer 50 to function. The CPU 51 reads the program from the storage unit 53, loads it into the memory 52, and sequentially executes the processes contained in the program.

[0057] [Operation of the information processing device of the embodiment]

[0058] Next, a description will be given of a specific operation of the information processing device 10 according to the embodiment. The information processing device 10 executes a trained model generation process shown in FIG.

[0059] First, in step S100, the learning acquisition unit 22 reads out a plurality of first learning images stored in the data storage unit 20, thereby acquiring a plurality of first learning images.

[0060] In step S101, the calculation unit 24 sets one first learning image from among the plurality of first learning images acquired in step S100.

[0061] Next, in step S102, the calculation unit 24 uses a known method to calculate a first correction coefficient that corrects the color appearing on the color chart in the first learning image set in step S101 to a standard reference color.

[0062] In step S104, the learning data generation unit 26 corrects the color of the first learning image using the first correction coefficient calculated in step S102, thereby generating a corrected image.

[0063] In step S106, the learning data generation unit 26 generates a plurality of second learning images by changing the color of each pixel of the corrected image generated in step S104.

[0064] In step S108, the learning data generation unit 26 calculates, for each of the multiple second learning images generated in step S106, a second correction coefficient used to correct the color of the second learning image to the color of the corrected image.

[0065] In step S110, the training data generation unit 26 generates first training data that associates the image of the tongue portion in the first training image set in step S101 with the first correction coefficient calculated in step S102. Also in step S110, the training data generation unit 26 generates second training data that associates the image of the tongue portion in the second training image with the second correction coefficient calculated in step S108 for each of the plurality of second training images generated in step S106. Then, the training data generation unit 26 stores the first training data and the second training data in the training data storage unit 28.

[0066] In step S112, the learning data generation unit 26 determines whether or not the processes of steps S101 to S110 have been performed on all first learning images stored in the data storage unit 20. If the processes of steps S101 to S110 have been performed on all first learning images stored in the data storage unit 20, the process proceeds to step S112. On the other hand, if there are first learning images for which the processes of steps S101 to S110 have not been performed, the process returns to step S101.

[0067] In step S112, the trained model generation unit 30 uses a known machine learning algorithm to generate a trained model that outputs a correction coefficient for correcting the color of an image when an image showing a tongue is input, based on the plurality of first training data and the plurality of second training data stored in the training data storage unit 28. The trained model generation unit 30 then stores the generated trained model in the trained model storage unit 32.

[0068] By executing the trained model generation process of Figure 6, a trained model is generated that outputs a correction coefficient to correct the color of an image when an image containing a tongue is input, and the correction coefficient output from the trained model can be used to correct the color of the tongue image.

[0069] Next, upon receiving a predetermined instruction signal, the information processing device 10 executes the information processing shown in FIG.

[0070] In step S200, the acquisition unit 34 acquires an image showing the target tongue.

[0071] In step S202, the correction unit 36 ​​reads out the trained model from the trained model storage unit 32.

[0072] In step S204, the correction unit 36 ​​inputs the image acquired in step S200 into the trained model read out in step S202, thereby acquiring the correction coefficients output from the trained model.

[0073] In step S206, the correction unit 36 ​​corrects the color of the image acquired in step S200 using the correction coefficient acquired in step S204.

[0074] In step S208, the output unit 38 outputs the image whose colors have been corrected in step S206 as a result.

[0075] As described above, the information processing device 10 of the embodiment acquires a training image that includes a training tongue and a color chart. The information processing device 10 then calculates a correction coefficient for correcting the color of the color chart in the training image to a reference color. The information processing device 10 generates training data that associates a training image of the tongue in the training image with the correction coefficient. Based on the training data, the information processing device 10 generates a trained model that outputs a correction coefficient for correcting the color of an image that includes a tongue when the image is input. This makes it possible to acquire a correction coefficient for performing color correction similar to the color correction method using a color chart, even without photographing the color chart together with the tongue. Furthermore, the correction coefficient can be used to correct the color of an image that includes a tongue, allowing color correction similar to the color correction method using a color chart, even without photographing the color chart together with the tongue.

[0076] Furthermore, the information processing device 10 generates second training data and generates a trained model based on the second training data, thereby obtaining correction coefficients for more accurately correcting the colors of an image. Specifically, an increase in the number of training data allows the trained model to be properly trained. Furthermore, by generating multiple second training images so that each pixel value falls within a distribution range of the multiple first training images, multiple second training images are generated that exhibit color variations similar to those of actual images captured under various light sources.

[0077] The technology of the present disclosure is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the spirit of the present invention.

[0078] For example, although the present specification has described an embodiment in which a program is pre-installed, the program may be provided by being stored on a computer-readable recording medium.

[0079] In the above embodiments, the processes executed by the CPU after reading the software (program) may be executed by various processors other than the CPU. Examples of such processors include programmable logic devices (PLDs) whose circuit configuration can be changed after fabrication, such as field-programmable gate arrays (FPGAs), and dedicated electrical circuits, such as application-specific integrated circuits (ASICs), which are processors with circuit configurations specifically designed to execute specific processes. Alternatively, a general-purpose graphics processing unit (GPGPU) may be used as the processor. Each process may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of a CPU and an FPGA). The hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor devices.

[0080] In addition, in each of the above embodiments, the program is described as being pre-stored (installed) in storage, but this is not limiting. The program may be provided in a form stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network.

[0081] Furthermore, each process of this embodiment may be implemented by a computer or server equipped with a general-purpose processor and a storage device, and each process may be executed by a program. This program is stored in a storage device, and may be recorded on a recording medium such as a magnetic disk, optical disk, or semiconductor memory, or may be provided via a network. Of course, any other components do not have to be implemented by a single computer or server, and may be distributed across multiple computers connected via a network.

[0082] In the above embodiment, the subject is a human tongue, but the present invention is not limited to this. The subject may be anything. For example, the present embodiment is applicable to subjects such as a human face, biological tissue, and landscapes.

[0083] For example, the following modifications are possible with respect to the subject.

[0084] [Variation 1] For example, each exposed part of the human body (e.g., skin, eyes, lips, hair, etc. exposed to the outside of the human body) may be set as the subject. For example, if the subject is skin, it is possible to correct the skin color, and it is possible to reproduce how the skin looks under a reference lighting environment. More specifically, for example, if a photo of skin with makeup applied under a certain lighting environment is taken, it is possible to reproduce how the skin looks under a reference lighting environment. Furthermore, once the skin color under the reference lighting environment is identified, it is possible to reproduce how the skin color looks under various lighting environments. Therefore, the technology of this embodiment may be useful, for example, when evaluating cosmetics applied to the skin. Similarly, when the subject is an eye, it is possible to reproduce the eye color under a reference lighting environment when wearing color contact lenses, for example. Similarly, once the eye color under a reference lighting environment has been identified, it is possible to reproduce the color that the eye will appear to be under various lighting environments. Therefore, the technology of this embodiment can be useful, for example, when evaluating color contact lenses. Similarly, when the subject is lips, it is possible to reproduce the eye color under a reference lighting environment when lipstick is applied to the lips. Similarly, once the lip color under a reference lighting environment is identified, it is possible to reproduce the color of the eyes under various lighting environments. Therefore, the technology of this embodiment can be useful, for example, when evaluating lipstick. Similarly, when the subject is hair, it is possible to reproduce the color of hair under a reference lighting environment when the hair is dyed. Similarly, when the hair color under a reference lighting environment is identified, it is possible to reproduce what color the hair will appear to be under various lighting environments. Therefore, the technology of this embodiment can be useful, for example, when evaluating hair dyeing techniques. As described above, when evaluating beauty products or services, the color of a subject in a specific lighting environment (e.g., the lighting environment in a beauty facility) often differs from the color of the subject in a different lighting environment (e.g., outdoors, in an office, hotel, restaurant, etc.). In such cases, even if the subject looks good in a specific lighting environment (e.g., a beauty facility), it may not look good in an actual lighting environment (e.g., outdoors). For this reason, for example, using the technology of this embodiment, the color of the subject obtained in a specific lighting environment (e.g., a beauty facility) can be first corrected to a color in a reference lighting environment and then converted to a color in various lighting environments, thereby enabling the evaluation of beauty products or services. Note that the conversion process from a color in a reference lighting environment to a color in various lighting environments can be performed using known technology.

[0085] [Variation 2] Alternatively, for example, an unexposed part of the human body (for example, various biological tissues of the human body that are not exposed to the outside) may be used as the subject. For example, teeth, gums, uvula, organs, blood vessel walls, airway walls, intestinal walls, ear cavity walls, nasal cavity walls, vaginal inner walls, or uterine inner walls may be used as the subject. In this case, by correcting the color of the subject photographed by various photographing devices, the color of the subject under a reference lighting environment can be identified, which becomes useful information when performing various medical procedures. Therefore, the technology of this embodiment can also be a useful technology when performing medical procedures.

[0086] [Variation 3] Alternatively, various products may be used as the subject of the image. For example, the color of a craft product in a light source environment, such as a factory or prototype room, where the craft product is manufactured often differs from the color of the craft product in the lighting environment where the craft product will actually be used. In such cases, a craft product that looks good in a specific lighting environment (e.g., a factory) may not look good in the actual lighting environment. For this reason, for example, by using the technology of this embodiment, the color of various products, such as craft products, obtained in a specific lighting environment (e.g., a factory) can be first corrected to a color under a reference lighting environment and then converted to a color under various lighting environments, thereby making it possible to evaluate the color of various products, such as craft products.

[0087] [Variation 4] Alternatively, for example, various foods (e.g., meat, fish, vegetables, or fruits) may be used as the subject. For example, the color of various foods under a specific light source environment is often different from the color of the foods under the lighting environment in which they are actually displayed. For this reason, for example, by using the technology of this embodiment, the color of various foods obtained under a specific lighting environment is first corrected to the color under a reference lighting environment, and then converted to the color under various lighting environments, making it possible to evaluate the color (or evaluate the freshness) of various foods.

[0088] [Variation 5] Alternatively, for example, a group may be generated for each combination of the lighting environment in which the subject is placed and the imaging device that captures the subject (more specifically, various settings of the imaging device), and a trained model may be generated for each group. Generally, the combination of the lighting environment and the imaging device determines how the subject's true color changes. The lighting environment may be defined by the type of light source. The imaging device may be defined by the characteristics of the image sensor in the imaging device and the color temperature conversion settings of the terminal software. Therefore, a group may be generated for each combination of the lighting environment in which the subject is placed and the imaging device that captures the subject, and the training data may be amplified to reproduce the variation in color change for each group, generating a trained model that outputs a correction coefficient for each group. This allows for the generation of a trained model that outputs a correction coefficient according to the combination of the lighting environment in which the subject is placed and the imaging device that captures the subject. This makes it possible to individually generate trained models, such as a trained model for obtaining correction coefficients for outdoor imaging, a trained model for obtaining correction coefficients for indoor imaging, and a trained model that outputs correction coefficients for imaging internal tissues. For example, when targeting endoscopic images, a dedicated light source and dedicated imaging device are assumed. Therefore, by amplifying a group of image samples of endoscopic images and generating a trained model based on the amplified image samples, a trained model specialized for outputting correction coefficients for correcting the color of endoscopic images can be obtained, making it possible to more appropriately correct the color of endoscopic images. Note that when amplifying a group of image samples of endoscopic images, as described above, the images are generated so that they have a probability distribution that matches the spatial distribution of correction coefficients for the entire group of image samples. In addition, by using groupings that depend on the characteristics of the light source, such as a group of image samples specialized for indoor portraits or a group of image samples specialized for outdoor scenes under natural light, it becomes possible to generate trained models specialized for each, making it possible to more appropriately correct the colors of each image.

[0089] Furthermore, in the above embodiment, an example has been described in which the information processing device 10 executes the trained model generation process of FIG. 6 and the information processing of FIG. 7, but this is not limiting. For example, the trained model generation device may execute the trained model generation process of FIG. 6, and the information processing device may execute the information processing of FIG. 7. In this case, the trained model generation device includes at least a training data acquisition unit 22, a calculation unit 24, a training data generation unit 26, and a trained model generation unit 30. In this case, the information processing device includes at least an acquisition unit 34 and a correction unit 36.

[0090] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[0091] (Addendum) The following additional notes are provided regarding aspects of the present disclosure.

[0092] (Appendix 1) a learning acquisition unit that acquires learning images including a learning subject and a color chart; a calculation unit that calculates a correction coefficient for correcting the color of the learning image that appears on the color chart to a reference color; a learning data generation unit that generates learning data that associates an image of the learning subject portion in the learning image with the correction coefficient; a trained model generation unit that generates a trained model based on the training data, and outputs a correction coefficient for correcting the color of an image when an image including a subject is input; A trained model generation device including: (Appendix 2) the training image is a first training image, the correction coefficient is a first correction coefficient, the learning data is first learning data, The learning data generation unit generating a corrected image by correcting a color of an image of the learning subject portion in the first learning image using the first correction coefficient; generating a plurality of second learning images by changing the color of the corrected image; calculating a second correction coefficient for correcting the color of each of the second learning images to the color of the corrected image, and generating second learning data that associates the second learning images with the second correction coefficient; the trained model generation unit generates the trained model based on the first training data and the second training data. 2. The trained model generation device according to claim 1. (Appendix 3) the learning data generation unit generates a plurality of second learning images so that each pixel value falls within a distribution range of the plurality of first learning images. 10. The trained model generation device according to claim 1 or 2. (Appendix 4) an acquisition unit that acquires an image showing a target subject; a correction unit that inputs the image acquired by the acquisition unit into a trained model generated in advance, thereby acquiring correction coefficients output from the trained model, and corrects the color of the image using the correction coefficients; Including, The trained model is a trained model that outputs a correction coefficient for correcting a color of an image when an image containing a subject is input, and is trained in advance based on training data in which a training subject and the training correction coefficient are associated with each other, The correction coefficient for learning is a correction coefficient that corrects the color of the color chart in the learning image, in which the learning subject and the color chart are captured, to a reference color that serves as a standard. Information processing device. (Appendix 5) Acquire a learning image containing a learning subject and a color target; Calculating a correction coefficient for correcting the color of the color chart in the learning image to a reference color; generating learning data that associates an image of the learning subject portion in the learning image with the correction coefficient; generating a trained model that outputs a correction coefficient for correcting the color of an image when an image including a subject is input based on the training data; A method for generating trained models in which processing is performed by a computer. (Appendix 6) Acquire an image of the target subject, The acquired image is input into a trained model generated in advance, thereby obtaining correction coefficients output from the trained model, and correcting the color of the image using the correction coefficients; The trained model is a trained model that outputs a correction coefficient for correcting a color of an image when an image containing a subject is input, and is trained in advance based on training data in which a training subject and the training correction coefficient are associated with each other, The correction coefficient for learning is a correction coefficient that corrects the color of the color chart in the learning image, in which the learning subject and the color chart are captured, to a reference color that serves as a standard. An information processing method in which processing is performed by a computer. (Appendix 7) Acquire a learning image containing a learning subject and a color target; Calculating a correction coefficient for correcting the color of the color chart in the learning image to a reference color; generating learning data that associates an image of the learning subject portion in the learning image with the correction coefficient; generating a trained model that outputs a correction coefficient for correcting the color of an image when an image including a subject is input based on the training data; A trained model generation program that allows a computer to execute processing. (Appendix 8) Acquire an image of the target subject, The acquired image is input to the trained model generated by the trained model generation program described in Supplementary Note 7, thereby obtaining the correction coefficient output from the trained model, and correcting the color of the image using the correction coefficient; The trained model is a trained model that outputs a correction coefficient for correcting a color of an image when an image containing a subject is input, and is trained in advance based on training data in which a training subject and the training correction coefficient are associated with each other, The correction coefficient for learning is a correction coefficient that corrects the color of the color chart in the learning image, in which the learning subject and the color chart are captured, to a reference color that serves as a standard. An information processing program that causes a computer to execute a process. [Explanation of symbols]

[0093] 10. Information processing equipment 20 Data storage unit 22 Learning acquisition section 24 Calculation section 26 Learning data generation unit 28 Learning data storage unit 30 Trained model generation unit 32 Trained model memory 34 Acquisition Department 36 Correction unit 38 Output section

Claims

1. a learning acquisition unit that acquires learning images including a learning subject and a color chart; a calculation unit that calculates a correction coefficient for correcting the color of the learning image that appears on the color chart to a reference color; a learning data generation unit that generates learning data that associates an image of the learning subject portion in the learning image with the correction coefficient; a trained model generation unit that generates a trained model based on the training data, and outputs a correction coefficient for correcting the color of an image when an image including a subject is input; A trained model generation device including:

2. the learning image is a first learning image, the correction coefficient is a first correction coefficient, the learning data is first learning data, The learning data generation unit generating a corrected image by correcting a color of an image of the learning subject portion in the first learning image using the first correction coefficient; generating a plurality of second learning images by changing the color of the corrected image; calculating a second correction coefficient for correcting the color of each of the second learning images to the color of the corrected image, and generating second learning data that associates the second learning images with the second correction coefficient; The trained model generation unit generates the trained model based on the first training data and the second training data. The trained model generation device according to claim 1 .

3. the learning data generation unit generates a plurality of second learning images so that each pixel value falls within a distribution range of the plurality of first learning images; The trained model generation device according to claim 1 or 2.

4. an acquisition unit that acquires an image showing a target subject; a correction unit that inputs the image acquired by the acquisition unit into a trained model generated in advance, thereby acquiring correction coefficients output from the trained model, and corrects the color of the image using the correction coefficients; Including, The trained model is a trained model that outputs a correction coefficient for correcting a color of an image when an image containing a subject is input, and is trained in advance based on training data in which a training subject and the training correction coefficient are associated with each other, The correction coefficient for learning is a correction coefficient that corrects the color of the color chart in the learning image, in which the learning subject and the color chart are captured, to a reference color that serves as a standard. Information processing device.

5. Acquire a learning image containing a learning subject and a color target; Calculating a correction coefficient for correcting the color of the color chart in the learning image to a reference color; generating learning data that associates an image of the learning subject portion in the learning image with the correction coefficient; generating a trained model that outputs a correction coefficient for correcting the color of an image when an image including a subject is input based on the training data; A method for generating trained models in which processing is performed by a computer.

6. Acquire an image of the target subject, The acquired image is input into a trained model generated in advance, thereby obtaining correction coefficients output from the trained model, and correcting the color of the image using the correction coefficients; The trained model is a trained model that outputs a correction coefficient for correcting a color of an image when an image containing a subject is input, and is trained in advance based on training data in which a training subject and the training correction coefficient are associated with each other, The correction coefficient for learning is a correction coefficient that corrects the color of the color chart in the learning image, in which the learning subject and the color chart are captured, to a reference color that serves as a standard. An information processing method in which processing is performed by a computer.

7. Acquire a learning image containing a learning subject and a color target; Calculating a correction coefficient for correcting the color of the color chart in the learning image to a reference color; generating learning data that associates an image of the learning subject portion in the learning image with the correction coefficient; generating a trained model that outputs a correction coefficient for correcting the color of an image when an image including a subject is input based on the training data; A trained model generation program that allows a computer to execute processing.

8. Acquire an image of the target subject, The acquired image is input into a trained model generated in advance, thereby obtaining correction coefficients output from the trained model, and correcting the color of the image using the correction coefficients; The trained model is a trained model that outputs a correction coefficient for correcting a color of an image when an image containing a subject is input, and is trained in advance based on training data in which a training subject and the training correction coefficient are associated with each other, The correction coefficient for learning is a correction coefficient that corrects the color of the color chart in the learning image, in which the learning subject and the color chart are captured, to a reference color that serves as a standard. An information processing program that causes a computer to execute a process.

Images