Teacher data, image processing device, imaging device, learning device, creation method of teacher data, generation method of learned model, image processing method, inference method, and program

Abstract

To provide teacher data capable of contributing to generation of a learned model for outputting an image in which a reproduction degree of colors of an imaged subject is higher than an inputted photographed image, to provide an image processing device, to provide an imaging device, to provide a learning device, to provide a creation method of teacher data, and to provide a generation method of learned model.SOLUTION: Teacher data includes a question image which is determined assuming a photographed image acquired by a subject being photographed, and a correct answer image. The question image is an image expressed by a plurality of first signal values showing three primary colors of light. The correct answer mage is an image expressed by a plurality of second signal values showing three primary colors. The plurality of first signal values is saturated according as the luminance of the subject becomes higher, and at least two out of the plurality of second signal values are not saturated in a low luminance region and in an intermediate luminance region out of the low luminance region of the subject, the intermediate luminance region of the subject and a high luminance region of the subject, and increase according as the luminance of the subject becomes higher.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present disclosure relates to teacher data, an image processing device, an imaging device, a learning device, a method for creating teacher data, a method for generating a trained model, an image processing method, an inference method, and a program. [Background technology]

[0002] Patent Document 1 discloses an embodiment in which a captured image is input into a machine learning model, blur components of the captured image are corrected to generate a first image, and a second image is generated based on the captured image, the first image, and a weight map, and the weight map is generated based on information regarding the brightness of the captured image or information regarding the scene of the captured image, and information based on saturated areas in the captured image.

[0003] Patent document 2 discloses an image processing method including the steps of acquiring a training image, an out-of-dynamic range map of the training image based on a signal value of the training image and a threshold value of the signal value, and ground truth data, and training a neural network to perform a recognition or regression task using input data including the training image and the out-of-dynamic range map, and the ground truth data.

[0004] Patent Document 3 discloses a method for manufacturing a trained model by machine learning, which uses a radiographic image including a subject as an input and outputs a luminance adjustment parameter for the radiographic image. The trained model manufacturing method described in Patent Document 3 includes the steps of: using an input image included in training data as an input, and outputting a luminance adjustment parameter for the input image by a training model; acquiring a loss function value related to the luminance adjustment parameter using the luminance adjustment parameter by the training model; and optimizing the training model so that the loss function value is small. The loss function is configured to bias learning in the direction of decreasing contrast by outputting a relatively large value when the contrast of a predetermined region of the luminance-adjusted image is larger than that of a predetermined region of the luminance-adjusted image in which the luminance is adjusted based on the luminance adjustment parameter, with respect to the training data. In addition, the loss function is configured to output a relatively large value when whiteout or blackout occurs due to saturation of pixel values ​​in a predetermined region of the luminance-adjusted image, compared to when saturation of pixel values ​​does not occur in the predetermined region. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2023-000274 A [Patent Document 2] JP 2020-166628 A [Patent Document 3] International Publication No. 2020 / 070834 Summary of the Invention

[0006] One embodiment of the present disclosure provides teacher data, an image processing device, an imaging device, a learning device, a method for creating teacher data, a method for generating a trained model, an image processing method, an inference method, and a program that can generate an image that has a higher degree of color reproduction of the subject depicted in the image than an input captured image for a trained model. [Means for solving the problem]

[0007] A first aspect of the present disclosure is training data used in machine learning of a model, the training data comprising an example image determined based on an image obtained by imaging a subject, and a correct image, wherein the example image is an image represented by a plurality of first signal values ​​indicating the three primary colors of light, the correct image is an image represented by a plurality of second signal values ​​indicating the three primary colors, the plurality of first signal values ​​saturate in sequence as the luminance of the subject increases, and at least two of the plurality of second signal values ​​do not saturate in the low luminance region of the subject, the medium luminance region of the subject, and the low luminance region and the medium luminance region of the subject's high luminance region, and increase as the luminance of the subject increases.

[0008] A second aspect according to the present disclosure is training data according to the first aspect, in which the plurality of first signal values ​​are saturated in an order according to the color of the object as the luminance of the object increases.

[0009] A third aspect of the present disclosure is training data relating to the first or second aspect, in which the example image and the correct answer image are images generated based on a reference image showing a subject, the reference image is an image represented by a plurality of third signal values ​​indicating the three primary colors, and the plurality of third signal values ​​increase at a constant rate as the luminance of the subject increases.

[0010] A fourth aspect of the present disclosure is training data related to the third aspect, in which the ratio at which the multiple first signal values ​​increase as the luminance of the subject increases is higher than the ratio at which the multiple third signal values ​​increase as the luminance of the subject increases.

[0011] A fifth aspect of the present disclosure is training data according to the third or fourth aspect, in which the example image is an image obtained by increasing the gain of the reference image.

[0012] A sixth aspect of the present disclosure is training data relating to any one of the third to fifth aspects, in which the correct image is an image obtained by increasing the gain of a reference image, maintaining the magnitude relationship of multiple third signal values ​​across low, medium, and high luminance regions, and saturating at least two of the multiple third signal values ​​in the high luminance region without saturating them in the low and medium luminance regions, and the amount of gain increase varies depending on the luminance of the subject.

[0013] A seventh aspect of the present disclosure is a trained model in which the model is optimized by performing machine learning on the model using training data relating to any one of the first to sixth aspects.

[0014] An eighth aspect of the present disclosure is an image processing device including a first processor, the first processor acquiring a first image output from a trained model according to the seventh aspect by inputting an image for inference to the trained model, and an captured image, and generating a second image by mixing the first image and the captured image.

[0015] A ninth aspect of the present disclosure is the image processing device according to the eighth aspect, in which the first processor generates the second image by mixing the first image and the captured image in reference region units.

[0016] A tenth aspect of the present disclosure is an image processing device according to the ninth aspect, in which a first processor generates a second image by mixing the first image and the captured image according to a mixing ratio determined on a reference area basis, the mixing ratio being a value based on at least one of a first mixing ratio, a second mixing ratio, and a third mixing ratio, the first mixing ratio being determined according to a classification result obtained by performing an object classification process using AI on the captured image or the first image on a reference area basis, the second mixing ratio being determined according to the highest signal value among a plurality of fourth signal values ​​indicating the three primary colors on a reference area basis for the second image, and the third mixing ratio being determined according to the lowest signal value among the plurality of fourth signal values.

[0017] An eleventh aspect of the present disclosure is an imaging device comprising a second processor and an image sensor, wherein the second processor inputs an image for inference to a trained model relating to the seventh aspect, obtains an inference result output from the trained model in response to the input of the image for inference, and obtains an captured image by capturing an image of a subject by the image sensor.

[0018] A twelfth aspect of the present disclosure is a learning device including a third processor, the third processor optimizing a model by performing machine learning on the model using training data relating to any one of the first to sixth aspects.

[0019] A thirteenth aspect of the present disclosure is a method for creating teacher data used in machine learning of a model, the teacher data comprising a correct image and an example image determined based on an image obtained by imaging a subject, the method including creating the correct image and creating the example image, the example image being an image represented by a plurality of first signal values ​​indicating the three primary colors of light, the correct image being an image represented by a plurality of second signal values ​​indicating the three primary colors, the plurality of first signal values ​​saturating in sequence as the luminance of the subject increases, and at least two of the plurality of second signal values ​​do not saturate in the low luminance region, the medium luminance region, and the low luminance region and the medium luminance region of the subject's high luminance region, but increase as the luminance of the subject increases.

[0020] A fourteenth aspect of the present disclosure is a method for generating a trained model by performing machine learning using training data on a model to generate a trained model, the training data comprising a correct image and an example image determined based on an image obtained by imaging a subject, the example image being an image represented by a plurality of first signal values ​​indicating the three primary colors of light, the correct image being an image represented by a plurality of second signal values ​​indicating the three primary colors, the plurality of first signal values ​​saturating in sequence as the luminance of the subject increases, and at least two of the plurality of second signal values ​​do not saturate in the low-luminance region, the medium-luminance region, and the high-luminance region of the subject, and increase as the luminance of the subject increases, the method including inputting the example image into the model, the model outputting an image to be evaluated in response to the input of the example image, and optimizing the model based on a comparison result between the image to be evaluated and the correct image.

[0021] A fifteenth aspect of the present disclosure is an image processing method that includes obtaining a first image output from a trained model according to the seventh aspect by inputting an image for inference to the trained model, and a captured image, and generating a second image by mixing the first image and the captured image.

[0022] A sixteenth aspect of the present disclosure is an inference method including inputting an image for inference into a trained model relating to the seventh aspect, and obtaining an inference result output from the trained model in response to the input of the image for inference.

[0023] A seventeenth aspect of the present disclosure is a program for causing a computer to execute processing including obtaining a first image output from a trained model according to the seventh aspect by inputting an image for inference to the trained model, and a captured image, and generating a second image by mixing the first image and the captured image.

[0024] An eighteenth aspect of the present disclosure is a program for causing a computer to execute processing including inputting an image for inference into a trained model relating to the seventh aspect, and obtaining an inference result output from the trained model in response to the input of the image for inference. [Brief description of the drawings]

[0025] [Figure 1] FIG. 1 is a conceptual diagram showing an example of an embodiment in which an imaging device is used. [Diagram 2] 2 is a block diagram showing an example of an electrical hardware configuration of an imaging device, an example of main functions, and an example of color signal values ​​representing a part of an image region included in a captured image. FIG. [Diagram 3] FIG. 2 is a conceptual diagram illustrating an example of the configuration of a learning device. [Figure 4] FIG. 13 is a conceptual diagram showing an example of a method for creating an example image. [Diagram 5] FIG. 13 is a conceptual diagram showing an example of a method for creating a correct image. [Figure 6] FIG. 1 is a conceptual diagram showing an example of a method for generating a high-quality image. [Figure 7] 13 is a flowchart showing an example of the flow of a learning process. [Figure 8] 10 is a flowchart showing an example of the flow of image quality improvement processing. [Figure 9] FIG. 13 is a conceptual diagram showing a first modified example of color signal values ​​representing a correct image. [Figure 10] FIG. 13 is a conceptual diagram showing a second modified example of color signal values ​​representing a correct image. [Figure 11] 13 is a conceptual diagram showing a modified example of a method for generating a high-quality image. [Figure 12] 4 is a conceptual diagram showing an example of the contents of a first mixing ratio table, a second mixing ratio table, and a third mixing ratio table. FIG. [Figure 13] FIG. 13 is a conceptual diagram showing an example of a method for calculating a mixing ratio used to mix a captured image and an AI image. [Figure 14]11 is a conceptual diagram showing an example of a form in which image quality improvement processing is performed by an external device in response to a request from an imaging device, and the processing result is received by the imaging device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] Below, examples of embodiments of the teacher data, image processing device, imaging device, learning device, teacher data creation method, trained model generation method, image processing method, inference method, and program related to the present disclosure will be described with reference to the attached drawings.

[0027] First, the terms used in the following description will be explained.

[0028] CPU is an abbreviation for "Central Processing Unit". GPU is an abbreviation for "Graphics Processing Unit". GPGPU is an abbreviation for "General-purpose computing on graphics processing units". APU is an abbreviation for "Accelerated Processing Unit". TPU is an abbreviation for "Tensor processing unit". NVM is an abbreviation for "Non-volatile memory". RAM is an abbreviation for "Random Access Memory". IC is an abbreviation for "Integrated Circuit". ASIC is an abbreviation for "Application Specific Integrated Circuit". PLD is an abbreviation for "Programmable Logic Device". FPGA is an abbreviation for "Field-Programmable Gate Array". SoC is an abbreviation for "System-on-a-chip". SSD is an abbreviation for "Solid State Drive". USB is an abbreviation for "Universal Serial Bus". EEPROM is an abbreviation for "Electrically Erasable and Programmable Read Only Memory." I / F is an abbreviation for "Interface." UI is an abbreviation for "User Interface." CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor." CCD is an abbreviation for "Charge Coupled Device." AI is an abbreviation for "Artificial Intelligence."

[0029] In the following description, a coded processor (hereinafter simply referred to as a "processor") may be one physical or virtual arithmetic device, or a combination of multiple physical or virtual arithmetic devices. Also, a processor may be one type of arithmetic device, or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU, a GPU, a GPGPU, an APU, and a TPU.

[0030] In the following description, a signed memory is a memory, such as a RAM, in which information is temporarily stored and which is used by a processor as a working memory.

[0031] In the following description, a storage with a symbol is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory, magnetic disks, and magnetic tapes. Another example of storage is cloud storage.

[0032] In the following embodiments, the external I / F with a symbol controls the transmission and reception of various information between multiple devices connected to each other. An example of the external I / F is a USB interface. A communication I / F including a communication processor and an antenna may be applied to the external I / F. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G, Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0033] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0034] As an example, as shown in Fig. 1, an imaging device 10 captures an image of an imaging target area 12 designated as a subject. The imaging target area 12 is determined by an angle of view designated by an imager 13 who is a user of the imaging device 10. In the example shown in Fig. 1, the imaging target area 12 includes a blue sky 12A as well as clouds, mountains, trees, roads, and the like. In this embodiment, the imaging device 10 is an example of an "imaging device" according to the present disclosure.

[0035] The imaging device 10 captures an image of the imaging target area 12 according to instructions given by an imager 13, thereby generating a captured image 14 indicative of the imaging target area 12. In this embodiment, the imaging target area 12 is an example of a "subject" according to the present disclosure, and the captured image 14 is an example of a "captured image" according to the present disclosure.

[0036] The imaging device 10 is a consumer digital camera. Examples of consumer digital cameras include interchangeable lens digital cameras and fixed lens digital cameras. The consumer digital camera is merely an example, and the imaging device 10 may be an industrial digital camera. The present disclosure may also be applied to imaging devices mounted on various electronic devices such as drive recorders, smart devices, wearable terminals, endoscope devices, cell observation devices, ophthalmic observation devices, and surgical microscopes.

[0037] 2, the imaging device 10 includes a computer 30, an image sensor 32, a UI device 34, and an external I / F 36. In this embodiment, the image sensor 32 is an example of the "image sensor" according to the present disclosure, and the computer 30 is an example of the "image processing device" and "computer" according to the present disclosure.

[0038] The computer 30 includes a processor 42, a storage 44, and a memory 46. The processor 42, the storage 44, and the memory 46 are connected to a bus 48. In this embodiment, the processor 42 is an example of the "first processor" and the "second processor" according to the present disclosure.

[0039] The storage 44 is a non-volatile storage device (i.e., a computer-readable non-transitory storage medium) that stores various programs, various parameters, and the like. An example of the storage 44 is a flash memory (e.g., an EEPROM). The memory 46 is a storage area in which information is temporarily stored, and is used as a working memory by the processor 42. An example of the memory 46 is a RAM.

[0040] The image sensor 32 has a plurality of photosensitive pixels (not shown) arranged in a matrix. Each photosensitive pixel is a physical pixel having a photodiode (not shown), which photoelectrically converts received light and outputs an electrical signal according to the amount of received light. In addition, color filters (not shown) of the three primary colors of light, that is, red (hereinafter also referred to as "R"), green (hereinafter also referred to as "G"), or blue (hereinafter also referred to as "B"), are arranged in a predetermined pattern arrangement in the plurality of photosensitive pixels. In this embodiment, a Bayer arrangement is used as an example of the predetermined pattern arrangement. However, the Bayer arrangement is merely an example, and the present disclosure is valid even if the predetermined pattern arrangement is another type of pattern arrangement such as a G-stripe R / G complete checkerboard, an X-Trans (registered trademark) arrangement, or a honeycomb arrangement.

[0041] For convenience of explanation, the photosensitive pixel having a microlens and an R color filter is referred to as an "R photosensitive pixel", the photosensitive pixel having a microlens and a G color filter is referred to as a "G photosensitive pixel", and the photosensitive pixel having a microlens and a B color filter is referred to as a "B photosensitive pixel". For convenience of explanation, the electrical signal output from the R photosensitive pixel is referred to as an "R signal" or an "R color signal", the electrical signal output from the G photosensitive pixel is referred to as a "G signal" or a "G color signal", and the electrical signal output from the B photosensitive pixel is referred to as a "B signal" or a "B color signal". For convenience of explanation, the R color signal, the G color signal, and the B color signal are referred to as "color signals" when there is no need to distinguish between them. For convenience of explanation, the signal value of the R signal is referred to as an "R signal value", the signal value of the G signal is referred to as a "G signal value", and the signal value of the B signal is referred to as a "B signal value". Furthermore, when there is no need to distinguish between R signal values, G signal values, and B signal values, they will be referred to as "color signal values."

[0042] The image sensor 32 is connected to the bus 48, and under the control of the processor 42, the image sensor 32 captures an image of the imaging target area 12 (see FIG. 1) to generate a captured image 14. The captured image 14 is an image in RGB format, and is formed by R pixels (hereinafter referred to as "R pixels"), G pixels (hereinafter referred to as "G pixels"), and B pixels (hereinafter referred to as "B pixels").

[0043] A CMOS image sensor is an example of the image sensor 32. Although a CMOS image sensor is given here as an example of the image sensor 32, this is merely an example, and the image sensor 32 may be other types of image sensors such as a CCD image sensor.

[0044] The UI device 34 is connected to the bus 48. The UI device 34 receives instructions from the image capturer 13 (see FIG. 1), and outputs a signal indicating the received instruction to the processor 42. The UI device 34 also presents various information to the image capturer 13 under the control of the processor 42. Presentation of the various information is realized, for example, by displaying the various information on a display or outputting the various information as sound from a speaker.

[0045] The external I / F 36 controls the exchange of various information with devices (hereinafter also referred to as "external devices") that exist outside the imaging device 10. An example of the external I / F 36 is a USB interface. To the USB interface, external devices (not shown), such as a smart device, a personal computer, a server, a USB memory, a memory card, and / or a printer, are directly or indirectly connected.

[0046] Meanwhile, the captured image 14 has an image region 16 showing a blue sky 12A (see FIG. 1). The luminance region (in other words, the luminance range) of the image region 16 is divided into a high luminance region H, a medium luminance region M, and a low luminance region L. In the example shown in FIG. 2, the luminance region of the image region 16 is divided into three regions, namely, the high luminance region H, the medium luminance region M, and the low luminance region L, at equal intervals. The high luminance region H is a high luminance region (in other words, a high luminance range) of the image region 16, the medium luminance region M is a medium luminance region (in other words, a medium luminance range) of the image region 16, and the low luminance region L is a low luminance region (in other words, a low luminance range) of the image region 16. For ease of explanation, the luminance range of image area 16 is used as an example here, but luminance ranges are similarly defined for image areas other than image area 16 (i.e., image areas showing subjects other than blue sky 12A), and the colors of image areas other than image area 16 are also represented by R signal values, G signal values, and B signal values.

[0047] The image region 16 is an image region represented by an R signal value 54, a G signal value 56, and a B signal value 58. That is, the color of the image region 16 is represented by the R signal value 54, the G signal value 56, and the B signal value 58. As the luminance of the image region 16 increases, the R signal value 54, the G signal value 56, and the B signal value 58 become saturated in an order according to the color of the image region 16 (i.e., the color of the blue sky 12A). In this embodiment, the R signal value 54, the G signal value 56, and the B signal value 58 representing the captured image 14 are an example of the "plurality of fourth signal values" according to the present disclosure.

[0048] In the low luminance region L, the magnitude relationship between the R signal value 54, the G signal value 56, and the B signal value 58 is "B signal value 58>G signal value 56>R signal value 54." In the low luminance region L, the R signal value 54, the G signal value 56, and the B signal value 58 increase at a constant rate as the luminance of the image region 16 increases. The B signal value 58 saturates at the boundary position between the low luminance region L and the medium luminance region M.

[0049] In the mid-luminance region M as well, the magnitude relationship between the R signal value 54, the G signal value 56, and the B signal value 58 is "B signal value 58>G signal value 56>R signal value 54." In the mid-luminance region M, the B signal value 58 remains saturated. The G signal value 56 and the B signal value 58 increase at the same rate as in the low-luminance region L. The G signal value 56 saturates at the boundary between the mid-luminance region M and the high-luminance region H.

[0050] In the high luminance region H, the B signal value 58 and the G signal value 56 are maintained in a saturated state. The R signal value 54 continues to increase while maintaining the rate of increase in the low luminance region L and the medium luminance region M, and saturates at the upper limit luminance of the high luminance region H. In the example shown in FIG. 2, the saturated state of the B signal value 58 is maintained from the boundary position between the low luminance region L and the medium luminance region M to the upper limit luminance of the high luminance region H, and the saturated state of the G signal value 56 is maintained from the boundary position between the medium luminance region M and the high luminance region H to the upper limit luminance of the high luminance region H.

[0051] In this way, when the B signal value 58 and the G signal value 56 are saturated at a lower luminance than the R signal value 54, a color saturated region 16A occurs in the image region 16. The color saturated region 16A is a region (for example, a region where white is more dominant than other colors) that is expressed in a color other than the original color of the blue sky 12A (for example, a color perceived by the naked eye of a person with normal visual ability).

[0052] For example, in the color saturation region 16A, blue would normally be displayed, but cyan is displayed instead because the B signal value 58 is saturated on the lower luminance side than the other color signal values ​​from the boundary position between the low luminance region L and the medium luminance region M to the high luminance region H. Also, in the color saturation region 16A, blue, cyan, or an intermediate color between blue and cyan is normally displayed, but the G signal value 56 is saturated on the lower luminance side than the R signal value 54 from the boundary position between the medium luminance region M and the high luminance region H to the high luminance region H, resulting in blown out highlights.

[0053] Here, the reproducibility of the color of the blue sky 12A is taken as an example for explanation, but the same can be said for subjects other than the blue sky 12A. For example, an image obtained by capturing an orange balustrade may have an image area where the color signal values ​​are saturated due to the same principle as the color saturation area 16A. The magnitude relationship of the color signal values ​​expressing the image area showing the orange balustrade is "R signal value 54>G signal value 56>B signal value 58", and the R signal value 54 is saturated on the lower luminance side than the G signal value 56 and the B signal value 58, and the G signal value 56 is saturated on the lower luminance side than the B signal value 58. Therefore, blown-out highlights occur due to the same principle as the color saturation area 16A.

[0054] Therefore, in this embodiment, in order to make the color reproduction level of the subject captured in the captured image 14 the same as the color of the subject in real space (i.e., to express the original color of the subject in the image), an image quality improvement process is performed by the processor 42. To achieve the image quality improvement process, an image quality improvement program 50 and a trained model 52 are stored in the storage 44. The image quality improvement program 50 is an example of a "program" according to the present disclosure.

[0055] The processor 42 reads the image quality improvement program 50 from the storage 44, and executes the read image quality improvement program 50 on the memory 46. The image quality improvement process is realized by the processor 42 operating as the acquisition unit 42A and the generation unit 42B in accordance with the image quality improvement program 50 executed on the memory 46. As will be described in detail later, the trained model 52 is used by the acquisition unit 42A.

[0056] Next, an example of a method for generating the trained model 52 will be described with reference to FIGS.

[0057] As an example, as shown in Fig. 3, the learning device 79 includes a processor 80, a storage 72, and a memory 84. The hardware configuration of the learning device 79 (e.g., the processor 80, the storage 72, and the memory 84) is basically the same as the hardware configuration of the computer 30 shown in Fig. 2, so a description of the hardware configuration of the learning device 79 will be omitted here. Note that in this embodiment, the processor 80 is an example of a "third processor" according to the present disclosure, and the learning device 79 is an example of a "learning device" according to the present disclosure.

[0058] A learning program 90 is stored in the storage 82. The processor 80 reads out the learning program 90 from the storage 82 and executes the read learning program 90 on the memory 84. The processor 80 performs a learning process according to the learning program 90 executed on the memory 84. The learning process is a process of generating a trained model 52 from a model 98. The trained model 52 is generated by the processor 80 executing machine learning on the model 98. That is, the trained model 52 is generated by optimizing the model 98 by machine learning. For example, the model 98 is a neural network having hundreds of millions to trillions of intermediate layers. An example of the model 98 is a model for a generation AI that generates and outputs an image with higher image quality than an input image (for example, an image in which at least the color saturation region 16A (see FIG. 2) is suppressed).

[0059] The storage 82 stores a plurality of (e.g., tens of thousands to hundreds of billions) pieces of teacher data 92. The teacher data 92 is used for machine learning of a model 98. That is, in the learning device 79, the processor 80 acquires the plurality of teacher data 92 from the storage 82, and optimizes the model 98 by performing machine learning on the model 98 using the acquired plurality of teacher data 92, thereby generating a trained model 52.

[0060] The teacher data 92 is labeled data. The labeled data is, for example, data in which an example image 94 (in other words, example data) and a correct answer image 96 (in other words, correct answer data) are associated with each other. Here, the teacher data 92 is an example of "teacher data" according to the present disclosure, the example image 94 is an example of "example image" according to the present disclosure, and the correct answer image 96 is an example of "correct answer image" according to the present disclosure.

[0061] Example image 94 is an image in RGB format that was determined based on captured image 14. In the example shown in Fig. 3, example image 94 has image area 95, which is an image area showing a blue sky and is based on image area 16 shown in Fig. 2. Image area 95 includes color saturation area 95A that is based on color saturation area 16A shown in Fig. 2.

[0062] In this embodiment, the example image 94 is an image generated based on an image obtained by actually capturing an image of a sample subject (e.g., the subject captured in the answer image 96 shown in FIG. 3) using an imaging device (e.g., an imaging device of the same type and specifications as the imaging device 10). However, this is merely one example, and the image may be virtually generated as an image envisioning the captured image 14 obtained by capturing an image using the imaging device 10. One example of a virtually generated image is an image generated by a generation AI. The generation AI may be an AI specialized in generating images, or may be an AI such as GPT-4 (Internet search engine optimization).<https: / / openai.com / gpt-4> The AI ​​may be a generation AI that generates and outputs example images 94 according to input instruction data (so-called prompts), such as ChatGPT, which uses the above-mentioned AI.

[0063] The correct image 96 is an image in RGB format, similar to the example image 94. Moreover, the correct image 96 is an image of higher image quality than the example image 94. An example of an image of higher image quality is an image in which the color saturation region 95A of the example image 94 is suppressed (for example, an image that includes an image region 95 in which the color saturation region 95A is eliminated, and is the same as the example image 94 except for the image region 95).

[0064] The processor 80 acquires the teacher data 92 one by one from the storage 82. The processor 80 inputs the example image 94 from the teacher data 92 acquired from the storage 82 to the model 98. When the example image 94 is input to the model 98, the model 98 generates a comparison image 100 which is an image with higher image quality than the example image 94 (here, as an example, an image in which the color saturation area 95A is suppressed). The comparison image 100 is an image used for comparison with the correct answer image 96 associated with the example image 94 input to the model 98. Here, the comparison image 100 is an example of an "evaluation target image" according to the present disclosure.

[0065] The processor 80 calculates an error 102 between a ground truth image 96 that is associated with an example image 94 input to the model 98 and a comparison image 100. Here, the error 102 is an example of a "comparison result" according to the present disclosure.

[0066] Processor 80 calculates a plurality of adjustment values ​​104 that minimizes error 102. Processor 80 then adjusts a plurality of optimization variables in model 98 using the plurality of adjustment values ​​104. Here, the plurality of optimization variables refers to, for example, a plurality of connection weights and a plurality of offset values ​​included in model 98.

[0067] The processor 80 repeatedly performs a series of processes, including inputting the example image 94 to the model 98, calculating the error 102, calculating the multiple adjustment values ​​104, and adjusting the multiple optimization variables in the model 98, using multiple teacher data 92 stored in the storage 82. That is, the processor 80 optimizes the model 98 by adjusting the multiple optimization variables in the model 98 using the multiple adjustment values ​​104 calculated so as to minimize the error 102 for each of the multiple example images 94 included in the multiple teacher data 92 stored in the storage 82. The processor 80 optimizes the model 98 to generate a trained model 52. When the example image 94 is input to the trained model 52 thus generated, the trained model 52 generates and outputs an image of the same image quality as the correct image 96 (i.e., an image in which the color saturation region 95A is suppressed at the same level as the correct image 96) as an image corresponding to the input example image 94.

[0068] FIG. 4 is a conceptual diagram showing an example of a method for creating the sample image 94. As shown in FIG. 4, the sample image 94 is an image generated based on the reference image 110. The reference image 110 is an image obtained by capturing an image of a subject 108 including a person 108A and a blue sky 108B, etc., by a reference imaging device 500 (for example, an imaging device of the same type and specifications as the imaging device 10) underexposed so that color signal values ​​are not saturated. The reference image 110 is an image in RGB format and is expressed by color signal values. The R signal value 54, the G signal value 56, and the B signal value 58 expressing the reference image 110 increase at a constant rate as the luminance of the subject 108 increases. The R signal value 54, the G signal value 56, and the B signal value 58 expressing the blue sky 108B captured in the reference image 110 increase monotonically from the low luminance region L to the high luminance region H without saturation. The color signal values ​​of the image region representing the blue sky 108B have a magnitude relationship of "B signal value 58>G signal value 56>R signal value 54." In the example shown in Fig. 4, the R signal value 54, the G signal value 56, and the B signal value 58 representing the reference image 110 are an example of the "multiple third signal values ​​indicating the three primary colors" according to the present disclosure.

[0069] The example image 94 is an image represented by the R signal value 54, the G signal value 56, and the B signal value 58. That is, the color of the example image 94 is represented by the R signal value 54, the G signal value 56, and the B signal value 58. The example image 94 is obtained by increasing the gains (i.e., digital gains) of the R signal value 54, the G signal value 56, and the B signal value 58 by a uniform increase rate (for example, an increase rate determined according to the subject (here, as an example, the blue sky 108B) appearing in the reference image 110) by an amount of underexposure by the reference imaging device 500 in the imaging for obtaining the reference image 110. If the color signal value exceeds a saturation value (for example, 255) due to the gain being multiplied, the color signal value is clipped at the saturation value. The R signal value 54, the G signal value 56, and the B signal value 58 representing the example image 94 are saturated in order as the luminance of the subject (here, as an example, the luminance of the blue sky 108B) increases. In the example shown in FIG. 4, the color signal values ​​are saturated in the order of B signal value 58, G signal value 56, and R signal value 54.

[0070] In this embodiment, the blue sky 108B is intentionally captured underexposed by the reference imaging device 500. The reason for this is to intentionally prevent the color signal values ​​from becoming saturated. Therefore, in this embodiment, in the process of creating the example image 94 based on the reference image 110, the gains of the R signal value 54, the G signal value 56, and the B signal value 58 are increased at a uniform rate by an amount corresponding to the underexposure, so that the ratio between the R signal value 54, the G signal value 56, and the B signal value 58 representing the image region 95 included in the example image 94 (for example, the ratio at which the R signal value 54, the G signal value 56, and the B signal value 58 in the low-brightness region L increase in accordance with the brightness of the subject) is made higher than the ratio between the R signal value 54, the G signal value 56, and the B signal value 58 representing the blue sky 108B captured in the reference image 110 (i.e., the ratio at which the R signal value 54, the G signal value 56, and the B signal value 58 representing the blue sky 108B captured in the reference image 110 increase in accordance with the brightness of the subject).

[0071] As a result, the ratio between the R signal value 54, the G signal value 56, and the B signal value 58 expressing the image region 95 (for example, the ratio in the low luminance region L) is realized to be the same as the ratio between the R signal value 54, the G signal value 56, and the B signal value 58 expressing the captured image 14 (see FIGS. 1 and 2) obtained by capturing an image of the blue sky 12A (see FIG. 1) by the imaging device 10 under actual imaging conditions (for example, the ratio in the low luminance region L). In the example shown in FIG. 4, the R signal value 54, the G signal value 56, and the B signal value 58 expressing the image region 95 included in the example image 94 exhibit the same behavior as the R signal value 54, the G signal value 56, and the B signal value 58 expressing the image region 16 shown in FIG. 2 with respect to the luminance of the subject captured in the example image 94 (here, the luminance of the blue sky as an example).

[0072] That is, the R signal value 54, the G signal value 56, and the B signal value 58 expressing the image region 95 are saturated in an order according to the color of the image region 95 (here, as an example, the color of blue sky 108B) as the luminance of the image region 95 increases. In the case of the image region 95, as the luminance of the image region 95 increases, the B signal value 58, the G signal value 56, and the R signal value 54 are saturated in that order. In the example shown in FIG. 4, the R signal value 54, the G signal value 56, and the B signal value 58 expressing the image region 95 included in the example image 94 are an example of the "plurality of first signal values ​​indicating the three primary colors of light" according to the present disclosure.

[0073] Note that, although the case of blue sky 108B is illustrated here, this is merely one example, and in the case of a subject other than blue sky 108B (i.e., an image area other than image area 95), the R signal value 54, G signal value 56, and B signal value 58 representing the other image area, which is an image area other than image area 95, also saturate in an order according to the color of the other image area.

[0074] In addition, here, an example of the form in which the reference image 110 is obtained by imaging using the reference imaging device 500 has been given, but this is merely one example. For example, the reference image 110 may be a virtually generated image that is an image that is assumed to be obtained by imaging the subject 108 underexposed so that the color signal value is not saturated by the reference imaging device 500. An example of a virtually generated image is an image generated by a generation AI. The generation AI may be an AI specialized in image generation, or may be a generation AI that generates and outputs the reference image 110 according to input instruction data (so-called prompts), such as ChatGPT using GPT-4 or the like.

[0075] FIG. 5 is a conceptual diagram showing an example of a method for creating a correct image 96. As shown in FIG. 5, the correct image 96 is an image generated based on a reference image 110. Like the example image 94, the correct image 96 is also an image expressed by an R signal value 54, a G signal value 56, and a B signal value 58. That is, the color of the correct image 96 is expressed by an R signal value 54, a G signal value 56, and a B signal value 58. The R signal value 54, the G signal value 56, and the B signal value 58 expressing the correct image 96 are not saturated in the low brightness region L and the medium brightness region M of the subject 108, but increase as the brightness of the subject 108 increases. 5 shows an example in which the R signal value 54, the G signal value 56, and the B signal value 58 expressing the image region 95 showing the blue sky 108B within the entire image region of the correct answer image 96 do not saturate in the low luminance region L and the medium luminance region M, but increase as the luminance of the blue sky 108B in the correct answer image 96 increases. However, this is merely an example, and the R signal value 54, the G signal value 56, and the B signal value 58 also increase as the luminance of the subject increases for subjects other than the blue sky 108B, without saturating in the low luminance region L and the medium luminance region M. In this embodiment, the R signal value 54, the G signal value 56, and the B signal value 58 expressing the correct answer image 96 are an example of the "plurality of second signal values" according to the present disclosure.

[0076] The correct image 96 is obtained by increasing the gains of the R signal value 54, the G signal value 56, and the B signal value 58 by the amount of underexposure caused by the reference imaging device 500 in the imaging for obtaining the reference image 110, and saturating the color signal values ​​in the high brightness region H without saturating them in the low brightness region L and the medium brightness region M. The amount of increase in gain varies depending on the brightness of the subject. Here, an increase amount derived in advance according to the brightness of the blue sky 108B by an actual machine test and / or a computer simulation is used as the amount of increase in gain. Also, similar to the reference image 110, the magnitude relationship of the R signal value 54, the G signal value 56, and the B signal value 58 is maintained from the low brightness region L to the high brightness region H in the correct image 96. That is, the magnitude relationship of "B signal value 58>G signal value 56>R signal value 54" is maintained among the R signal value 54, the G signal value 56, and the B signal value 58.

[0077] Note that, here, an example is given in which the magnitude relationship of the color signal values ​​representing the blue sky 108B is "B signal value 58>G signal value 56>R signal value 54", but this is merely one example, and in the case of a subject other than the blue sky 108B, the magnitude relationship of the color signal values ​​is determined according to the color of the subject.

[0078] The R signal value 54, the G signal value 56, and the B signal value 58 expressing the correct image 96 are saturated at the maximum luminance of the subject (here, as an example, the maximum luminance of the blue sky 108B). In the example shown in FIG. 5, the maximum luminance of the subject refers to the upper limit luminance of the high luminance region H. The R signal value 54, the G signal value 56, and the B signal value 58 expressing the blue sky 108B in the correct image 96 (i.e., the R signal value 54, the G signal value 56, and the B signal value 58 expressing the image region 95) are controlled to be blown out at the upper limit luminance of the high luminance region H. The color signal values ​​are classified into a minimum signal value, which is the lowest signal value among the color signal values, a maximum signal value, which is the highest signal value among the color signal values, and an intermediate signal value, which is a signal value between the minimum signal value and the maximum signal value among the color signal values. In the example shown in FIG. 5, the lowest signal value representing the correct image 96 is the R signal value 54, the highest signal value representing the correct image 96 is the B signal value 58, and the intermediate signal value representing the correct image 96 is the G signal value 56. The rate at which the R signal value 54 representing the correct image 96 increases with respect to luminance (i.e., the increase rate of the R signal value 54) is constant. The rate at which the G signal value 56 representing the correct image 96 increases with respect to luminance (i.e., the increase rate of the G signal value 56) decreases in the order of the low luminance region L, the medium luminance region M, and the high luminance region H. Similar to the increase rate of the G signal value 56, the rate at which the B signal value 58 representing the correct image 96 increases with respect to luminance (i.e., the increase rate of the B signal value 58) also decreases in the order of the low luminance region L, the medium luminance region M, and the high luminance region H.

[0079] Furthermore, in the correct image 96, the gains of the R signal value 54, the G signal value 56, and the B signal value 58 in dark areas (for example, low luminance area L) increase at a uniform increase rate (i.e., an increase rate determined according to the subject captured in the reference image 110). Here, as an example, the gains of the R signal value 54, the G signal value 56, and the B signal value 58 in dark areas increase uniformly at an increase rate determined according to the blue sky 108B captured in the reference image 110.

[0080] Furthermore, in the correct image 96, in the medium luminance region M, saturation of the highest signal value (in the example shown in FIG. 5, the B signal value 58) is suppressed, and the G signal value 56 and the B signal value 58 are suppressed so as to maintain the ratio among the R signal value 54, the G signal value 56, and the B signal value 58 as much as possible.

[0081] Here, an example is given in which the correct image 96 is generated based on the reference image 110, but the correct image 96 may be an image virtually generated to satisfy the various conditions described above. An example of a virtually generated image is an image generated by a generation AI. The generation AI may be an AI specialized in generating images, or may be a generation AI that generates and outputs the correct image 96 according to input instruction data (so-called prompts), such as ChatGPT using GPT-4 or the like.

[0082] FIG. 6 is a conceptual diagram showing an example of an operation stage (i.e., a stage in which the trained model 52 makes an inference) of the trained model 52 generated by performing the learning process of the example shown in FIG. 3. As shown in FIG. 6, in the imaging device 10, the acquisition unit 42A acquires the captured image 14 from the image sensor 32. The acquisition unit 42A inputs the captured image 14 to the trained model 52 stored in the storage 44, causes the trained model 52 to generate and output an AI image 112, and acquires the AI ​​image 112 output from the trained model 52. The AI ​​image 112 is an image in which the color saturation region 16A of the captured image 14 is suppressed (in the example shown in FIG. 6, an image in which the color saturation region 16A is erased from the captured image 14). Here, the trained model 52 is an example of a "trained model" according to the present disclosure, the captured image 14 is an example of an "image for inference" according to the present disclosure, and the AI ​​image 112 is an example of a "first image" and an "inference result" according to the present disclosure.

[0083] The generating unit 42B generates a high-quality image 114 by mixing the captured image 14 and the AI ​​image 112. The high-quality image 114 is an image with higher quality than the captured image 14. An image with higher quality than the captured image 14 refers to an image with higher quality than the captured image 14 in that at least the color saturation region 16A of the captured image 14 is suppressed.

[0084] The high-quality image 114 is an image generated by mixing the captured image 14 and the AI ​​image 112 in units of pixels whose positions correspond to each other. Here, an example is given in which the captured image 14 and the AI ​​image 112 are mixed in units of pixels whose positions correspond to each other, but this is merely one example, and the captured image 14 and the AI ​​image 112 may be mixed in units of blocks (i.e., units each consisting of a group of multiple pixels), which are units larger than pixels.

[0085] Mixing the captured image 14 and the AI ​​image 112 refers to, for example, averaging the captured image 14 and the AI ​​image 112, or mixing the captured image 14 and the AI ​​image 112 at a default mixing ratio. Here, the mixing ratio refers to, for example, the ratio at which the AI ​​image 112 and the captured image 14 are added. For example, if the mixing ratio is 1:1, half of the color signal value of the AI ​​image 112 and half of the color signal value of the captured image 14 are added in pixel units, if the mixing ratio is 1:0, the color signal value of the AI ​​image 112 is adopted in pixel units, and the color signal value of the captured image 14 is not adopted, and if the mixing ratio is 0:1, the color signal value of the captured image 14 is adopted in pixel units, and the color signal value of the AI ​​image 112 is adopted in pixel units. In FIG. 6, the high-quality image 114 is an example of a "second image" according to the present disclosure, and the pixel unit is an example of a "reference area unit" according to the present disclosure.

[0086] The generating unit 42B performs image processing, including development, on the high-quality image 114 to convert the high-quality image 114 into a file in a default file format, and outputs the file to a default output destination. A first example of the default output destination is the storage 44. A second example of the default output destination is the UI-based device 34. A third example of the default output destination is a device connected to the external I / F 36 (e.g., a USB memory, a smart device, a personal computer, and / or a server, etc.).

[0087] Next, the operation of the part of the learning device 79 according to the present disclosure will be described with reference to Fig. 7. Fig. 7 shows an example of the flow of the learning process executed by the processor 80. The flow of the learning process shown in Fig. 7 is an example of the "method of generating a trained model" according to the present disclosure.

[0088] In the learning process shown in Figure 7, first, in step ST10, the processor 80 executes the processing of step ST10 to acquire unprocessed teacher data 92 (i.e., teacher data 92 that has not been used in the learning process shown in Figure 7) from the storage 82, and then the learning process proceeds to step ST12.

[0089] In step ST12, the processor 80 inputs the example image 94 included in the training data 92 acquired in step ST10 to the model 98. After the processing of step ST12 is executed, the learning processing proceeds to step ST14. As a result of the processing of step ST12 being executed, a comparison image 100 is output from the model 98.

[0090] In step ST14, the processor 80 acquires the comparison image 100 output from the model 98. After the process of step ST14 is executed, the learning process proceeds to step ST16.

[0091] In step ST16, the processor 80 compares the comparison image 100 acquired in step ST14 with the correct answer image 96 included in the training data 92 acquired in step ST10. After the processing of step ST16 is executed, the learning processing proceeds to step ST18.

[0092] In step ST18, the processor 80 adjusts the model 98 using a plurality of adjustment values ​​104 obtained by comparing the comparison image 100 and the correct answer image 96 in step ST16. The process of step ST18 is repeatedly executed based on all the training data 92 stored in the storage 82, thereby optimizing the model 98. After the process of step ST18 is executed, the learning process proceeds to step ST20.

[0093] In step ST20, the processor 80 determines whether or not unprocessed teacher data 92 is stored in the storage 82. In step ST20, if unprocessed teacher data 92 is stored in the storage 82, the determination is affirmative, and the learning process proceeds to step ST10. In step ST20, if unprocessed teacher data 92 is not stored in the storage 82, the determination is negative, and the learning process ends.

[0094] Next, the operation of the portion of the imaging device 10 according to the present disclosure will be described with reference to Fig. 8. The flow of the image quality improvement process shown in Fig. 8 is an example of an "inference method" according to the present disclosure. For convenience of explanation, the description will be given on the assumption that the trained model 52 has already been stored in the storage 44.

[0095] 8, first, in step ST50, the acquisition unit 42A determines whether or not one frame's worth of image has been captured by the image sensor 32. In step ST50, if one frame's worth of image has not been captured by the image sensor 32, the determination is negative, and the image quality improvement process proceeds to step ST58. If one frame's worth of image has been captured by the image sensor 32, the determination is positive, and the image quality improvement process proceeds to step ST52.

[0096] In step ST52, the acquisition unit 42A acquires the captured image 14 from the image sensor 32. After the process of step ST52 is executed, the image quality improvement process proceeds to step ST54.

[0097] In step ST54, the acquisition unit 42A inputs the captured image 14 acquired in step ST52 to the trained model 52. After the processing of step ST54 is executed, the image quality improvement processing proceeds to step ST56. By executing the processing of step ST54, the trained model 52 generates and outputs the AI ​​image 112.

[0098] In step ST56, the acquisition unit 42A acquires the AI ​​image 112 output from the trained model 52. After the processing of step ST56 is executed, the image quality improvement processing proceeds to step ST58.

[0099] In step ST58, the generation unit 42B generates a high-quality image 114 by mixing the captured image 14 acquired by the acquisition unit 42A in step ST52 and the AI ​​image 112 acquired by the acquisition unit 42A in step ST54. After the process of step ST58 is executed, the image quality improvement process proceeds to step ST60.

[0100] In step ST60, the generation unit 42B determines whether or not a condition for terminating the image quality improvement process (hereinafter referred to as the "termination condition") has been satisfied. An example of the termination condition is that an instruction to terminate the image quality improvement process has been accepted by the UI device 34. If the termination condition has not been satisfied in step ST60, the determination is negative, and the image quality improvement process proceeds to step ST50. If the termination condition has been satisfied in step ST60, the determination is positive, and the image quality improvement process terminates.

[0101] As described above, the teacher data 92 according to this embodiment is used for machine learning of the model 98. The trained model 52 is generated by machine learning the model 98. The teacher data 92 includes the example image 94 and the correct image 96, and the example image 94 and the correct image 96 are associated with each other. The example image 94 is an image determined by assuming the captured image 14, and the color of each pixel of the example image 94 is expressed by a color signal value. Similarly to the example image 94, the color of each pixel of the correct image 96 is also expressed by a color signal value. The color signal values ​​of the image region 16 (see FIG. 2) obtained by capturing the blue sky 12A (see FIG. 1) are saturated in the order of the B signal value 58, the G signal value 56, and the R signal value 54 as the luminance of the image region 16 increases. Here, if the B signal value 58 and the G signal value 56 are saturated in the medium luminance region M, a color saturation region 16A occurs. The color saturation region 16A is an image region that is whitish compared to the original color of the blue sky 12A (see FIG. 1) shown by the image region 16. Therefore, the color signal value expressing the correct image 96 included in the teacher data 92 is configured not to saturate in the low brightness region L and the medium brightness region M, but to saturate as the subject (here, as an example, the brightness of the blue sky 108B) increases. Then, machine learning using the teacher data 92 configured in this way is performed on the model 98, thereby generating a trained model 52 in which the model 98 is optimized. The trained model 52 generates and outputs an AI image 112 in response to an input of the captured image 14. The AI ​​image 112 is an image in which the color reproduction of the subject is higher than that of the captured image 14 (i.e., an image in which the color saturation region 16A is suppressed). In this way, when a captured image 14 is input to a trained model 52 generated by optimizing the model 98 by using the teacher data 92 of this embodiment for machine learning on the model 98, the trained model 52 can generate an AI image 112 (i.e., an image in which the color saturation area 16A is suppressed) that has a higher degree of color reproduction of the subject depicted in the captured image 14 than the input captured image 14.

[0102] Furthermore, in this embodiment, the image capturing device 10 uses the trained model 52 generated by performing machine learning on the model 98 using the teacher data 92. Therefore, by inputting the captured image 14 to the trained model 52, the image capturing device 10 can generate and output the AI ​​image 112 in which the color saturation region 16A is suppressed more than in the captured image 14.

[0103] In addition, in this embodiment, the color signal values ​​expressing the example image 94 are saturated in the order according to the color of the subject 108 as the brightness of the subject 108 increases. For example, the color signal values ​​expressing the example image 94 are saturated in the order of the B signal value 58, the G signal value 56, and the R signal value 54 as the brightness of the blue sky 108B increases. Therefore, when the captured image 14 is input to the trained model 52 generated by optimizing the model 98 by using the teacher data 92 including the example image 94 configured in this way for machine learning for the model 98, the trained model 52 can generate an AI image 112 (i.e., an image in which the color saturation area 16A is suppressed) having a higher reproduction degree of the color of the subject shown in the image than the input captured image 14.

[0104] In this embodiment, the color signal values ​​expressing the reference image 110 increase at a constant rate as the luminance of the subject 108 increases. The example image 94 and the correct image 96 are images generated based on the reference image 110 expressed with the same color signal values ​​as the example image 94 and the correct image 96. In this way, the example image 94 and the correct image 96 are generated based on a common image, i.e., the reference image 110, so the example image 94 and the correct image 96 can be generated more efficiently than when the example image 94 and the correct image 96 are generated based on different images. Also, the example image 94 and the correct image 96 can be easily generated with the same angle of view as the reference image 110.

[0105] Furthermore, in this embodiment, the constant rate at which the color signal values ​​representing the reference image 110 increase is lower than the rate at which the color signal values ​​representing the example image 94 increase. Therefore, it is easier to generate an ideal example image 94 based on the reference image 110, compared to a case where the constant rate at which the color signal values ​​representing the reference image 110 increase is higher than the rate at which the color signal values ​​representing the example image 94 increase.

[0106] In this embodiment, the example image 94 is generated by increasing the gain of the color signal values ​​expressing the reference image 110. Therefore, the example image 94 which is a projection of the captured image 14 can be easily obtained.

[0107] In this embodiment, the correct image 96 is generated by increasing the gain of the color signal values ​​expressing the reference image 110, maintaining the magnitude relationship between the R signal value 54, the G signal value 56, and the B signal value 58 from the low luminance region L to the high luminance region H, and saturating the color signal values ​​in the high luminance region H without saturating them in the low luminance region L and the medium luminance region M. The amount of increase in the gain differs depending on the luminance of the subject 108. For example, the amount of increase in the gain used for the blue sky 108B is determined in advance depending on the luminance of the blue sky 108B. Therefore, the correct image 96 in which the color saturation region 95A is suppressed can be easily obtained.

[0108] In this embodiment, the captured image 14 and the AI ​​image 112 are mixed to generate a high-quality image 114. The high-quality image 114 is an image in which the color saturation region 16A is suppressed more than in the captured image 14. The AI ​​image 112 is an image in which the color saturation region 16A is suppressed more than in the high-quality image 114. However, if the components of the color saturation region 16A are too few, it becomes difficult to reproduce the color of the original imaging target region 12 (here, the blue sky 12A). Therefore, in this embodiment, the captured image 14 and the AI ​​image 112 are mixed to leave a small amount of the components of the color saturation region 16A. This makes it possible to obtain a high-quality image 114 that is close to the color of the actual imaging target region 12. In addition, since the captured image 14 and the AI ​​image 112 are mixed in units of pixels whose positions correspond to each other, a high-quality image 114 that is close to the color of the actual imaging target region 12 can be obtained compared to the case where they are not mixed in units of pixels.

[0109] In the above embodiment, the R signal value 54, the G signal value 56, and the B signal value 58 expressing the correct image 96 are not saturated in the low brightness region L and the medium brightness region M, but are saturated at the upper limit brightness of the high brightness region H. However, this is merely an example. The maximum signal value of the R signal value 54, the G signal value 56, and the B signal value 58 expressing the correct image 96 may be saturated on the lower brightness side than the intermediate signal value and the minimum signal value. For example, as shown in FIG. 9, the B signal value 58 of the R signal value 54, the G signal value 56, and the B signal value 58 expressing the image region 95 may be saturated on the lower brightness side than the G signal value 56 and the R signal value 54. In the example shown in FIG. 9, the B signal value 58 expressing the image region 95 is saturated at the boundary position between the low brightness region L and the medium brightness region M, and the saturated state is maintained from the boundary position between the low brightness region L and the medium brightness region M to the upper limit brightness of the high brightness region H. Even when a trained model 52 is generated by performing machine learning on the model 98 using teacher data 92 including the correct image 96 configured in this manner, an image in which the color saturation area 16A is suppressed more than in the captured image 14 can be generated and output as an AI image 112 by the trained model 52.

[0110] 10, the B signal value 58 of the R signal value 54, G signal value 56, and B signal value 58 expressing an image region 95 may be saturated on the lower luminance side than the G signal value 56 and the R signal value 54, and the G signal value 56 may be saturated on the lower luminance side than the R signal value 54. In the example shown in FIG. 10, the B signal value 58 expressing the image region 95 is saturated at the boundary position between the low luminance region L and the medium luminance region M, and the saturated state is maintained from the boundary position between the low luminance region L and the medium luminance region M to the upper limit luminance of the high luminance region H. Also, the G signal value 56 expressing the image region 95 is saturated halfway through the high luminance region H, and the saturated state is maintained from the halfway through the high luminance region H to the upper limit luminance of the high luminance region H. Even when a trained model 52 is generated by performing machine learning on the model 98 using teacher data 92 including the correct image 96 configured in this manner, an image in which the color saturation area 16A is suppressed more than in the captured image 14 can be generated and output as an AI image 112 by the trained model 52.

[0111] In the above embodiment, the captured image 14 and the AI ​​image 112 are averaged or mixed at a preset mixing ratio to generate the high-quality image 114, but the present disclosure is not limited to this. For example, the high-quality image 114 may be generated by mixing the captured image 14 and the AI ​​image 112 according to a mixing ratio determined for each reference area based on the color signal value expressing the captured image 14 and the classification result obtained by performing an object classification process using AI on the captured image 14 or the AI ​​image 112 for each reference area (for example, pixel unit).

[0112] In this case, for example, as shown in FIG. 11, the generation unit 42B calculates a mixing ratio 118 determined on a pixel-by-pixel basis by referring to the color signal values ​​representing the captured image 14 and the segmentation image 116, and generates a high-quality image 114 by mixing the captured image 14 and the AI ​​image 112 according to the calculated mixing ratio 118.

[0113] To achieve this, in the imaging device 10, the storage 44 stores a classification model 120. The classification model 120 is a trained model in which machine learning using multiple teacher data including example images in which one or more objects are captured and answer images to which labels indicating the types of one or more objects captured in the example images are added is performed on an encoder-decoder structure model (e.g., SegNet, U-Net, HRNet, etc.) to optimize the model. The captured image 14 is input to the classification model 120. The classification model 120 is a model that classifies multiple objects captured in the input captured image 14 on a pixel-by-pixel basis. For example, the classification model 120 classifies multiple objects captured in the input captured image 14 by a segmentation method (e.g., a method using semantic segmentation, a method using instance segmentation, or a method using panoptic segmentation). In the example shown in FIG. 11, the classification model 120 is an example of the "AI" according to the present disclosure.

[0114] The processor 42 differs from the processor 42 described in the above embodiment in that it further operates as a classification unit 42C. The classification unit 42C performs object classification processing using AI on a pixel-by-pixel basis for the captured image 14 (i.e., processing for classifying a plurality of objects appearing in the captured image 14 on a pixel-by-pixel basis). That is, the classification unit 42C inputs the captured image 14 to a classification model 120 stored in the storage 44, thereby causing the classification model 120 to classify a plurality of objects appearing in the captured image 14 on a pixel-by-pixel basis, and outputs a segmentation image 116 indicating the classification result.

[0115] In order to calculate the mixing ratio 118 by referring to the color signal values ​​representing the captured image 14 and the segmentation image 116, the generation unit 42B uses a first mixing ratio table 122, a second mixing ratio table 124, and a third mixing ratio table 126, as shown in FIG. 12 as an example.

[0116] 12, the storage 44 stores a first mixing ratio table 122, a second mixing ratio table 124, and a third mixing ratio table 126. The first mixing ratio table 122 is a table in which the classification result of the segmentation image 116 in units of pixels (i.e., the result of classifying each object in the captured image 14 in units of pixels) is associated with the first mixing ratio. The first mixing ratio is the ratio at which the AI ​​image 112 is used when the captured image 14 and the AI ​​image 112 are mixed by the generation unit 42B. The first mixing ratio is set for each classification result (i.e., each object in the captured image 14) within a range of 0 to 1.

[0117] The second mixing ratio table 124 is a table in which the maximum signal value and the second mixing ratio are associated with each other. As with the first mixing ratio, the second mixing ratio is also a ratio at which the AI ​​image 112 is used when the captured image 14 and the AI ​​image 112 are mixed by the generation unit 42B. The second mixing ratio is set in a range of 0 to 1. The second mixing ratio is "0" from the maximum signal value to the specific signal value, and increases monotonically from the specific signal value to the saturation signal value (i.e., a saturated signal value). Here, the specific signal value refers to a signal value derived in advance by a test using an actual device and / or a computer simulation, etc., as a lower limit value of the maximum signal value at which the color saturation region 16A is likely to occur. By setting the second mixing ratio to a value exceeding "0" from the specific signal value to the saturation signal value, the color saturation region 16A becomes more susceptible to the influence of the AI ​​image 112 than image regions other than the color saturation region 16A. As a result, the AI ​​image 112 is mixed more predominantly than the captured image 14 in image regions where saturation of color signal values ​​is likely to occur, making it easier to obtain a high-quality image 114 in which the color saturation region 16A is suppressed.

[0118] The third mixing ratio table 126 is a table in which the minimum signal value is associated with the third mixing ratio. As with the first and second mixing ratios, the third mixing ratio is also a ratio at which the AI ​​image 112 is used when the captured image 14 and the AI ​​image 112 are mixed by the generating unit 42B. However, if the color saturation region 16A is colored, the image as a whole tends to become unnatural. Therefore, the third mixing ratio is such that the minimum signal value monotonically decreases from "0" to the saturation signal value so that the blown-out parts in the captured image 14 mixed with the AI ​​image 112 are less likely to be affected by the AI ​​image 112.

[0119] 13 as an example, the generation unit 42B calculates a mixing ratio 118 based on a first mixing ratio table 122, a second mixing ratio table 124, and a third mixing ratio table 126 stored in the storage 44. The mixing ratio 118 is determined on a pixel-by-pixel basis, and is determined based on the first mixing ratio, the second mixing ratio, and the third mixing ratio.

[0120] The generating unit 42B derives a first mixing ratio corresponding to the classification result in the classification model 120 from a first mixing ratio table 122 for each pixel in the segmentation image 116. The generating unit 42B also derives a second mixing ratio corresponding to the highest signal value (B signal value 58 in the example shown in FIG. 2) of the captured image 14 acquired from the image sensor 32 by the acquiring unit 42A from a second mixing ratio table 124 on a pixel-by-pixel basis. The generating unit 42B also derives a third mixing ratio corresponding to the lowest signal value (R signal value 54 in the example shown in FIG. 2) of the captured image 14 acquired from the image sensor 32 by the acquiring unit 42A from a third mixing ratio table 126 on a pixel-by-pixel basis.

[0121] The generation unit 42B calculates the mixing ratio 118 by multiplying the first mixing ratio, the second mixing ratio, and the third mixing ratio on a pixel-by-pixel basis. Then, the generation unit 42B generates the high-quality image 114 by mixing the captured image 14 and the AI ​​image 112 according to the mixing ratio 118 on a pixel-by-pixel basis (see FIG. 11). In the example shown in FIG. 13, the mixing ratio 118 is an example of the "mixing ratio" according to the present disclosure.

[0122] In this way, the mixing ratio 118 is calculated by multiplying the first mixing ratio, the second mixing ratio, and the third mixing ratio on a pixel-by-pixel basis, and the captured image 14 and the AI ​​image 112 are mixed on a pixel-by-pixel basis in accordance with the calculated mixing ratio 118, thereby preventing the high-quality image 114 from being excessively influenced by the captured image 14 or the AI ​​image 112, or preventing the high-quality image 114 from being excessively influenced by the captured image 14 or the AI ​​image 112. As a result, the influence of the AI ​​image 112 can be appropriately reflected in the color saturation region 16A, and excessive coloring in the color saturation region 16A can be prevented.

[0123] 11 and 13, an example is given in which the captured image 14 is input to the classification model 120 and the segmentation image 116 for the captured image 14 is output from the classification model 120, but this is merely one example, and the AI ​​image 112 may be input to the classification model 120 and the segmentation image 116 for the AI ​​image 112 may be output from the classification model 120. In this case as well, the same effect can be expected.

[0124] 11 and 13, the classification unit 42C classifies a plurality of objects captured in the captured image 14 by the segmentation method, but this is merely one example, and the classification unit 42C may classify a plurality of objects captured in the captured image 14 by the bounding box method. In this case, the mixture ratio 118 is calculated for each bounding box, and the captured image 14 and the AI ​​image 112 are mixed in accordance with the mixture ratio 118 for each bounding box to generate the high-quality image 114.

[0125] In this way, the generation unit 42B calculates the mixing ratio 118 in reference area units (for example, pixel units, or sets of multiple pixels such as bounding boxes), and generates a high-quality image 114 by mixing the captured image 14 and the AI ​​image 112 in reference area units according to the calculated mixing ratio 118.

[0126] In the example shown in Figs. 11 to 13, the mixing ratio 118 is determined based on the first mixing ratio, the second mixing ratio, and the third mixing ratio. However, this is merely an example, and the mixing ratio 118 may be determined based on one or two of the first mixing ratio, the second mixing ratio, and the third mixing ratio. One or two of the first mixing ratio, the second mixing ratio, and the third mixing ratio used to create the mixing ratio 118 may be determined according to an instruction received by a receiving device connected to the UI device 34 or the external I / F 36. The high-quality image 114 generated according to the mixing ratio 118 thus determined may be displayed on a display device connected to the UI device 34 or the external I / F 36. The multiple high-quality images 114 generated according to the multiple mixing ratios 118 may be displayed on a display device connected to the UI device 34 or the external I / F 36 in a comparative state. In addition, with multiple high-quality images 114 being displayed in a comparative manner on a display device connected to the UI device 34 or the external I / F 36, the high-quality image 114 to be ultimately adopted may be determined from the multiple high-quality images 114 in accordance with instructions accepted by a reception device connected to the UI device 34 or the external I / F 36.

[0127] 13 shows an example in which the first mixture ratio is derived from the first mixture ratio table 122, but this is merely one example. For example, the first mixture ratio may be output on a pixel-by-pixel basis by the classification model 120. In this case, for example, the first mixture ratio may be included in a label (in other words, an annotation given to each pixel) included in teacher data used in machine learning for generating the classification model 120, together with information indicating the type of object.

[0128] In the above embodiment, an example in which the image quality improvement process is performed by the processor 42 of the computer 30 included in the imaging device 10 has been described, but the present disclosure is not limited to this, and the device performing the image quality improvement process may be provided outside the imaging device 10. In this case, as an example, an imaging system 136 may be used as shown in FIG. 14. The imaging system 136 includes the imaging device 10 and an external device 138. The external device 138 is, for example, a server. The server is realized by, for example, cloud computing. Here, cloud computing is exemplified, but this is merely an example, and for example, the server may be realized by a mainframe, or may be realized by network computing such as fog computing, edge computing, or grid computing. Here, a server is exemplified as an example of the external device 138, but this is merely an example, and at least one personal computer or the like may be used as the external device 138 instead of the server.

[0129] The external device 138 includes a processor 140, a storage 142, a memory 144, and a communication I / F 146, and the processor 140, the storage 142, the memory 144, and the communication I / F 146 are connected to each other via a bus 148. The communication I / F 146 is connected to the imaging device 10 via a network 150. The network 150 is, for example, the Internet. Note that the network 150 is not limited to the Internet, and may be a WAN and / or a LAN such as an intranet.

[0130] The storage 142 stores the image quality improvement program 50 and the trained model 52. The processor 140 executes the image quality improvement program 50 in the memory 144. The processor 140 performs the image quality improvement process described above according to the image quality improvement program 50 executed on the memory 144. When performing the image quality improvement process, the processor 140 processes the captured image 14 using the trained model 52 in the manner described in the above embodiment. The captured image 14 is transmitted from the imaging device 10 to the external device 138 via the network 150. The communication I / F 146 of the external device 138 receives the captured image 14. The processor 140 performs a process on the captured image 14 received by the communication I / F 146 using the trained model 52 described in the above embodiment, and a process of generating a high-quality image 114 by mixing the captured image 14 and the AI ​​image 112. The processor 140 transmits the generated high-quality image 114 to the imaging device 10 via the communication I / F 146. The imaging device 10 receives the high-quality image 114 transmitted from the external device 138 via the external I / F 36 (see FIG. 2).

[0131] 14 shows an example in which the image capture device 10 causes the external device 138 to execute image quality improvement processing, but this is merely one example. For example, the image capture device 10 and the external device 138 may execute the image quality improvement processing in a distributed manner, or a plurality of devices including the image capture device 10 and the external device 138 may execute the image quality improvement processing in a distributed manner.

[0132] In the above embodiment, an example in which the image quality improvement program 50 is stored in the storage 44 has been described, but the present disclosure is not limited to this. For example, the image quality improvement program 50 may be stored in a portable computer-readable non-transitory storage medium such as an SSD or a USB memory. The image quality improvement program 50 stored in the non-transitory storage medium is installed in the computer 30 of the imaging device 10. The processor 42 executes image quality improvement processing in accordance with the image quality improvement program 50.

[0133] In addition, the image quality improvement program 50 may be stored in a storage device such as another computer or server device connected to the imaging device 10 via a network, and the image quality improvement program 50 may be downloaded and installed on the computer 30 in response to a request from the imaging device 10.

[0134] It should be noted that it is not necessary to store the entire image quality improvement program 50 in a storage device such as another computer or server device connected to the imaging device 10, or in the storage 44, and it is acceptable to store only a part of the image quality improvement program 50. Note that although the image quality improvement program 50 is mentioned here, the same can be said about the learning program 90.

[0135] 2 includes a built-in computer 30, the present disclosure is not limited to this. For example, the computer 30 may be provided outside the imaging device 10.

[0136] In the above embodiment, the computer 30 is exemplified, but the present disclosure is not limited thereto, and a device including an ASIC, an FPGA, and / or a PLD may be applied instead of the computer 30. Also, instead of the computer 30, a combination of a hardware configuration and a software configuration may be used.

[0137] As the hardware resource for executing the image quality improvement processing and / or learning processing described in the above embodiment, various processors as shown below can be used. An example of the processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing the image quality improvement processing and / or learning processing by executing software, i.e., a program. Another example of the processor is a dedicated electric circuit, which is a processor having a circuit configuration designed exclusively for executing a specific process, such as an FPGA, a PLD, or an ASIC. Each processor has a built-in or connected memory, and each processor uses the memory to execute the image quality improvement processing and / or learning processing.

[0138] The hardware resource for executing the image quality improvement process and / or the learning process may be one of these various processors, or may be a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource for executing the image quality improvement process and / or the learning process may be a single processor.

[0139] As an example of a configuration using one processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes image quality improvement processing and / or learning processing. Second, there is a configuration in which a processor is used that realizes the functions of the entire system including multiple hardware resources that execute image quality improvement processing and / or learning processing in a single IC chip, as typified by SoC. In this way, the image quality improvement processing and / or learning processing is realized using one or more of the above various processors as hardware resources.

[0140] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements. Also, the image quality improvement process and / or learning process described above are merely examples. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the order of processing may be changed, without departing from the spirit of the invention.

[0141] The above description and illustrations are detailed descriptions of the parts related to the present disclosure and are merely an example of the present disclosure. For example, the above description of the configuration, function, action, and effect is an example of the configuration, function, action, and effect of the parts related to the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above description and illustrations within the scope of the gist of the present disclosure. In addition, in order to avoid confusion and to facilitate understanding of the parts related to the present disclosure, the above description and illustrations omit explanations of technical common sense that do not require explanation in order to enable the implementation of the present disclosure.

[0142] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, and standard was specifically and individually indicated to be incorporated by reference.

[0143] The following supplementary notes are further disclosed regarding the above embodiment. (Appendix 1) Training data used in machine learning of a model, A sample image is determined assuming a captured image obtained by capturing an image of a subject; A correct answer image; The above example image is an image represented by a plurality of first signal values ​​indicating the three primary colors of light, the correct image is an image represented by a plurality of second signal values ​​indicating the three primary colors; the plurality of first signal values ​​are saturated in order as the luminance of the object increases, At least two of the plurality of second signal values ​​are not saturated in the low brightness region of the object, the medium brightness region of the object, and the high brightness region of the object, and increase as the brightness of the object increases. Teacher data.

[0144] (Appendix 2) A trained model in which the model is optimized by performing the machine learning on the model using the training data described in Appendix 1.

[0145] (Appendix 3) Acquiring a first image output from the trained model described in Supplementary Note 2 by inputting an image for inference to the trained model, and the captured image; and A computer-readable non-transitory storage medium storing a program for causing a computer to execute a process including generating a second image by blending the first image and the captured image.

[0146] (Appendix 4) Inputting the inference image into the trained model described in Appendix 2; and A computer-readable non-transitory storage medium storing a program for executing a process including obtaining an inference result output from the trained model in response to input of the inference image. [Explanation of symbols]

[0147] 10. Imaging device 12 Imaging target area 12A Blue Sky 13 Photographer 14 Captured images 16 Image Area 16A Color saturation area 30 Computers 32 Image Sensor 34 UI devices 36 External I / F 42,80,140 processors 42A Acquisition Department 42B Generation part 42C Classification section 44,82,142 Storage 46,84,144 Memory 48 Bus 50 High-definition program 52 Pre-trained models 54 R signal value 56 G signal value 58 B signal value 79 Learning Device 90 Study Programs 92 Teacher Data 94 Example images 95 Image Area 95A Color saturation area 96 Correct Images 98 Models 100 Comparison Images 102 error 104 Adjustment value 108 Subject 108A People 108B Blue Sky 110 Reference Image 112 AI images 114 High Quality Images 116 Segmentation Images 118 Mixing ratio 120 Classification Models 122 First Mixture Table 124 Second Mixture Table 126 Third Mixture Table 136 Imaging System 138 External device 146 Communication I / F 500 Reference Imaging Device H High brightness area L Low brightness area M Mid-luminance area

Claims

1. Training data used in machine learning of a model, A sample image is determined assuming a captured image obtained by capturing an image of a subject; A correct answer image; The example image is an image represented by a plurality of first signal values ​​indicating the three primary colors of light, the correct image is an image represented by a plurality of second signal values ​​indicating the three primary colors, the plurality of first signal values ​​are saturated in order as the luminance of the object increases, At least two of the plurality of second signal values ​​are not saturated in the low brightness region, the medium brightness region, and the high brightness region of the subject, and increase as the brightness of the subject increases. Teacher data.

2. The plurality of first signal values ​​are saturated in an order according to the color of the object as the luminance of the object increases. The training data according to claim 1 .

3. the example image and the correct image are images generated based on a reference image showing the subject, the reference image is an image represented by a plurality of third signal values ​​indicating the three primary colors, The third signal values ​​increase at a constant rate as the luminance of the object increases. The training data according to claim 1 .

4. The rate at which the first signal values ​​increase as the luminance of the object increases is higher than the rate at which the third signal values ​​increase as the luminance of the object increases. The training data according to claim 3.

5. The example image is an image obtained by increasing the gain of the reference image. The training data according to claim 3.

6. the correct image is an image obtained by increasing a gain of the reference image, maintaining a magnitude relationship between the plurality of third signal values ​​across the low luminance region, the medium luminance region, and the high luminance region, and saturating at least two of the plurality of third signal values ​​in the high luminance region without saturating them in the low luminance region and the medium luminance region, The amount of increase in the gain varies depending on the brightness of the subject. The training data according to claim 3.

7. A trained model in which the model is optimized by performing the machine learning on the model using the training data according to any one of claims 1 to 6.

8. A first processor is provided, The first processor, A method for acquiring a first image output from the trained model according to claim 7 by inputting an image for inference to the trained model, and the captured image; A second image is generated by blending the first image and the captured image. Image processing device.

9. The first processor generates the second image by blending the first image and the captured image in units of reference regions. The image processing device according to claim 8.

10. The first processor generates the second image by mixing the first image and the captured image in accordance with a mixing ratio defined for each reference area; the mixing ratio is a value based on at least one of a first mixing ratio, a second mixing ratio, and a third mixing ratio; the first mixture ratio is determined according to a classification result obtained by performing an object classification process using AI on the captured image or the first image in units of the reference region, the second mixing ratio is determined in accordance with a maximum signal value among a plurality of fourth signal values ​​that indicate the three primary colors in the reference area unit for the second image, The third mixing ratio is determined according to the lowest signal value among the plurality of fourth signal values. The image processing device according to claim 9 .

11. A second processor; and An image sensor, The second processor, Inputting an inference image into the trained model according to claim 7, Obtaining an inference result output from the trained model in response to input of the inference image; The captured image is obtained by capturing an image of the subject with the image sensor. Imaging device.

12. A third processor is provided, The third processor optimizes the model by performing the machine learning on the model using the teacher data according to any one of claims 1 to 6. Learning device.

13. A method for creating training data used in machine learning of a model, comprising the steps of: The teacher data includes a correct image and an example image that is determined based on an image obtained by capturing an image of a subject, Creating the ground truth image; and creating said example image; The example image is an image represented by a plurality of first signal values ​​indicating the three primary colors of light, the correct image is an image represented by a plurality of second signal values ​​indicating the three primary colors, the plurality of first signal values ​​are saturated in order as the luminance of the object increases, At least two of the plurality of second signal values ​​are not saturated in the low brightness region, the medium brightness region, and the high brightness region of the subject, and increase as the brightness of the subject increases. How to create training data.

14. A method for generating a trained model, which generates a trained model by performing machine learning using teacher data on a model, The teacher data includes a correct image and an example image that is determined based on an image obtained by capturing an image of a subject, The example image is an image represented by a plurality of first signal values ​​indicating the three primary colors of light, the correct image is an image represented by a plurality of second signal values ​​indicating the three primary colors, the plurality of first signal values ​​are saturated in order as the luminance of the object increases, at least two of the plurality of second signal values ​​are not saturated in the low luminance region and the medium luminance region of the object, and are increased as the luminance of the object increases, among the low luminance region, the medium luminance region, and the high luminance region of the object; inputting said example images into said model; The model outputs an evaluation target image in response to the input of the example image; and and optimizing the model based on a comparison result between the evaluation target image and the correct answer image. How to generate a trained model.

15. A method for acquiring a first image output from the trained model according to claim 7 by inputting an inference image to the trained model, and the captured image; and and generating a second image by blending the first image and the captured image. Image processing methods.

16. Inputting an inference image into the trained model according to claim 7; and and acquiring an inference result output from the trained model in response to input of the inference image. Reasoning method.

17. On the computer, A method for acquiring a first image output from the trained model according to claim 7 by inputting an inference image to the trained model, and the captured image; and A program for executing a process including generating a second image by blending the first image and the captured image.

18. On the computer, Inputting an inference image into the trained model according to claim 7; and A program for executing a process including obtaining an inference result output from the trained model in response to input of the inference image.

Images