Training data and method for creating same, image processing device, imaging device, learning device, method for generating learned model, image processing method, reasoning method, and computer program product

Abstract

The invention provides training data, an image processing device, an imaging device, a learning device, a method for creating training data, a method for generating a learned model, and an image processing method, wherein the training data can contribute to generating a learned model which outputs an image in which the reproduction degree of the color of an imaged object is higher than that of an input imaged image; the image processing device, the imaging device, the learning device, the training data creating method, the learned model generating method, and the image processing method. An inference method and a computer program product. The training data includes: an example image assuming to be determined by a captured image obtained by capturing a subject; and a correct answer image. The example image is an image expressed by a plurality of first signal values representing three primary colors of light. The correct answer image is an image expressed by a plurality of second signal values representing three primary colors. The plurality of first signal values are sequentially saturated as the brightness of the subject increases, and at least two of the plurality of second signal values are not saturated in a low-brightness region and a medium-brightness region of the subject, among a low-brightness region of the subject, a medium-brightness region of the subject, and a high-brightness region of the subject, and increase as the brightness of the subject increases.

Description

Technical Field

[0001] The present invention relates to training data, an image processing device, a camera device, a learning device, a method for creating training data, a method for generating a learned model, an image processing method, an inference method, and a computer program product. Background Art

[0002] Patent document 1 discloses the following method: a camera image is input into a machine learning model, a blur component of the camera image is corrected to generate a first image, and a second image is generated based on the camera image, the first image, and a weight map, wherein the weight map is generated based on information related to the brightness of the camera image or information related to the scene of the camera image and information based on a saturated area in the camera image.

[0003] Patent document 2 discloses an image processing method, which includes the following steps: obtaining a training image, a dynamic range out-map of the training image based on a signal value of the training image and a threshold of the signal value, and correct answer data; and using input data including the training image and the dynamic range out-map and the correct answer data to learn a neural network for performing a recognition or regression task.

[0004] Patent document 3 discloses a method for manufacturing a learned model based on machine learning, which uses a radiographic image of a subject as input and outputs a brightness adjustment parameter of the radiographic image. The method for manufacturing a learned model described in Patent document 3 includes the following steps: using an input image included in training data as input, outputting a brightness adjustment parameter of the input image through a learning model; using the brightness adjustment parameter based on the learning model to obtain a value of a loss function related to the brightness adjustment parameter; and optimizing the learning model to reduce the value of the loss function. The loss function is configured to output a relatively large value when the contrast of a specified area of ​​the brightness-adjusted image after brightness adjustment according to the brightness adjustment parameter becomes smaller relative to the training data, and the learning is biased toward reducing the contrast by outputting a relatively large value. Furthermore, the loss function is configured to output a relatively large value when the specified area of ​​the brightness-adjusted image becomes white or black due to saturation of pixel values, compared to when saturation of pixel values ​​does not occur in the specified area.

[0005] Patent Document 1: Japanese Patent Application Publication No. 2023-000274

[0006] Patent Document 2: Japanese Patent Application Publication No. 2020-166628

[0007] Patent Document 3: International Publication No. 2020 / 070834 Summary of the invention

[0008] One embodiment of the present invention provides training data, an image processing device, a camera device, a learning device, a method for creating training data, a method for generating a learned model, an image processing method, an inference method, and a computer program product that enable a learned model to generate an image in which the color of the projected object is higher than that of an input camera image.

[0009] The first method involved in the present invention is training data, which is used for machine learning of a model and comprises: an example image, which is assumed to be a camera image obtained by photographing a subject; and a correct answer image, wherein the example image is an image expressed by multiple first signal values ​​representing three primary colors of light, and the correct answer image is an image expressed by multiple second signal values ​​representing the three primary colors, wherein the multiple first signal values ​​are successively saturated as the brightness of the subject increases, and at least two of the multiple second signal values ​​are unsaturated in the low brightness area and the medium brightness area of ​​the subject among the low brightness area, the medium brightness area and the high brightness area of ​​the subject, and increase as the brightness of the subject increases.

[0010] According to a second aspect of the present invention, in the training data according to the first aspect, the plurality of first signal values ​​are saturated in an order corresponding to the color of the subject as the brightness of the subject increases.

[0011] The third method involved in the present invention is that in the training data involved in the first method or the second method, the example image and the correct answer image are images generated based on a reference image representing the subject, the reference image is an image expressed by multiple third signal values ​​representing three primary colors, and the multiple third signal values ​​increase at a constant rate as the brightness of the subject increases.

[0012] According to a fourth aspect of the present invention, in the training data according to the third aspect, a ratio at which the plurality of first signal values ​​increase as the brightness of the subject increases is higher than a ratio at which the plurality of third signal values ​​increase as the brightness of the subject increases.

[0013] According to a fifth aspect of the present invention, in the training data according to the third aspect or the fourth aspect, the example image is an image obtained by increasing a gain of a reference image.

[0014] The 6th mode involved in the present invention is that in the training data involved in any one of the 3rd mode to the 5th mode, the correct answer image is an image obtained as follows: the gain of the reference image is increased, the size relationship of multiple third signal values ​​is maintained throughout the low brightness area, the medium brightness area and the high brightness area, and at least two of the multiple third signal values ​​are not saturated in the low brightness area and the medium brightness area but are saturated in the high brightness area, and the increase in gain varies according to the brightness of the subject.

[0015] The seventh aspect of the present invention is a learned model, which is obtained by performing machine learning on the model using the training data involved in any one of the first to sixth aspects to optimize the model.

[0016] The eighth embodiment of the present invention is an image processing device comprising a first processor, which performs the following processing: acquiring a first image and a camera image, wherein the first image is output from the learned model by inputting an inference image into the learned model involved in the seventh embodiment; and generating a second image by mixing the first image and the camera image.

[0017] A ninth aspect according to the present invention is the image processing device according to the eighth aspect, wherein the first processor generates the second image by mixing the first image and the captured image in units of reference regions.

[0018] In the tenth mode involved in the present invention, in the image processing device involved in the ninth mode, the first processor generates a second image by mixing the first image and the camera image according to a mixing ratio determined in reference area units, the mixing ratio being a value based on at least one of the first mixing ratio, the second mixing ratio and the third mixing ratio, the first mixing ratio being determined based on a classification result obtained by performing object classification processing using AI on the camera image or the first image in reference area units, the second mixing ratio being determined based on the highest signal value among multiple fourth signal values ​​representing three primary colors of the second image in reference area units, and the third mixing ratio being determined based on the lowest signal value among the multiple fourth signal values.

[0019] The 11th method involved in the present invention is a camera device, which includes: a second processor; and an image sensor, the second processor performs the following processing: inputting an image for reasoning into the learned model involved in the 7th method; and obtaining an inference result output from the learned model based on the input of the image for reasoning, and the camera image is obtained by photographing the subject with the image sensor.

[0020] A twelfth aspect of the present invention is a learning device including a third processor configured to optimize a model by performing machine learning on a model using training data according to any one of the first to sixth aspects.

[0021] The 13th method involved in the present invention is a method for creating training data, which is used for machine learning of a model. The training data includes a correct answer image and an example image that is assumed to be obtained by photographing a subject. The method for creating training data includes the following steps: creating a correct answer image; and creating an example image, the example image being an image expressed by multiple first signal values ​​representing three primary colors of light, the correct answer image being an image expressed by multiple second signal values ​​representing the three primary colors, the multiple first signal values ​​being saturated in sequence as the brightness of the subject increases, and at least two of the multiple second signal values ​​being unsaturated in a low brightness area and a medium brightness area among a low brightness area of ​​the subject, a medium brightness area of ​​the subject, and a high brightness area of ​​the subject, and increasing as the brightness of the subject increases.

[0022] The 14th method involved in the present invention is a method for generating a learned model, which generates the learned model by performing machine learning on the model using training data, the training data comprising a correct answer image and an example image that is assumed to be obtained by photographing a subject, the example image being an image expressed by multiple first signal values ​​representing three primary colors of light, the correct answer image being an image expressed by multiple second signal values ​​representing the three primary colors, the multiple first signal values ​​being saturated in sequence as the brightness of the subject increases, at least two of the multiple second signal values ​​being unsaturated in a low brightness area of ​​the subject, a medium brightness area of ​​the subject, and a low brightness area and a medium brightness area among a high brightness area of ​​the subject, and increasing as the brightness of the subject increases, the method for generating the learned model comprising the following steps: inputting an example image into the model; outputting an evaluation object image from the model based on the input of the example image; and optimizing the model based on the comparison result between the evaluation object image and the correct answer image.

[0023] The 15th method involved in the present invention is an image processing method, which includes the following steps: acquiring a first image and a camera image, wherein the first image is output from the learned model by inputting an inference image into the learned model involved in the 7th method; and generating a second image by mixing the first image and the camera image.

[0024] The 16th method involved in the present invention is an inference method, which includes the following steps: inputting an inference image into the learned model involved in the 7th method; and obtaining an inference result output from the learned model based on the input of the inference image.

[0025] The 17th method involved in the present invention is a program, which is used to enable a computer to perform processing including the following steps: acquiring a first image and a camera image, wherein the first image is output from the learned model by inputting an inference image into the learned model involved in the 7th method; and generating a second image by mixing the first image and the camera image.

[0026] The 18th method involved in the present invention is a program, which is used to enable a computer to execute a process including the following steps: inputting an image for inference into the learned model involved in the 7th method; and obtaining an inference result output from the learned model based on the input of the image for inference. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a conceptual diagram showing an example of a method of using the imaging device.

[0028] Figure 2 This is a block diagram showing an example of the hardware configuration of the electrical system of the imaging device, an example of the functions of the main parts, and an example of expressing the color signal value of a partial image area included in the captured image.

[0029] Figure 3 This is a conceptual diagram showing an example of the structure of a learning device.

[0030] Figure 4 This is a conceptual diagram showing an example of a method of creating an example image.

[0031] Figure 5 This is a conceptual diagram showing an example of a method for creating a correct answer image.

[0032] Figure 6 This is a conceptual diagram showing an example of a method for generating a high-quality image.

[0033] Figure 7 This is a flowchart showing an example of the flow of the learning process.

[0034] Figure 8 This is a flowchart showing an example of the flow of high-quality image processing.

[0035] Fig. 9 This is a conceptual diagram showing a first variation of color signal values ​​expressing a correct answer image.

[0036] Fig.10 This is a conceptual diagram showing a second variation of the color signal value expressing the correct answer image.

[0037] Fig.11 This is a conceptual diagram showing a modified example of the method for generating a high-quality image.

[0038] Fig.12 This is a conceptual diagram showing an example of the contents of the first mixing ratio table, the second mixing ratio table, and the third mixing ratio table.

[0039] Fig.13 This is a conceptual diagram showing an example of a method for calculating a mixing ratio for mixing a captured image and an AI image.

[0040] Fig.14This is a conceptual diagram showing an example of a format in which an external device performs high-quality image processing in response to a request from an imaging device, and the imaging device receives the processing result. DETAILED DESCRIPTION

[0041] Below, an example of implementation of the training data, image processing device, camera device, learning device, training data creation method, learned model generation method, image processing method, inference method and program involved in the present invention is described based on the accompanying drawings.

[0042] First, the words and phrases used in the following description are explained.

[0043] CPU is the abbreviation of “Central Processing Unit”. GPU is the abbreviation of “Graphics Processing Unit”. GPGPU is the abbreviation of “General-purpose computing on graphics processing units”. APU is the abbreviation of “Accelerated Processing Unit”. TPU is the abbreviation of “Tensor processing unit”. NVM is the abbreviation of “Non-volatile memory”. RAM is the abbreviation of “Random Access Memory”. IC is the abbreviation of “Integrated Circuit”. ASIC is the abbreviation of “Application Specific Integ rated Circuit”. PLD is the abbreviation of “Programmable Logic Device”. FPGA is the abbreviation of “Field-Programmable Gate Array”. SoC is the abbreviation of “System-on-a-chip”. SSD is the abbreviation of “Solid State Drive”. USB is the abbreviation of “Universal Serial Bus”. EEPROM is the abbreviation of “Electrically Erasable and Programmable Read Only Memory”. I / F is the abbreviation of “Interface”. UI is the abbreviation of “User Interface”. CMOS is the abbreviation of “Complementary Metal Oxide Semiconductor”. CCD is the abbreviation of “Charge Coupled Device”. AI is the abbreviation of “Artificial Intelligence”.

[0044] In the following description, a processor with a symbol (hereinafter referred to as a "processor") can be a physical or virtual computing device, or a combination of multiple physical or virtual computing devices. In addition, a processor can be a computing device, or a combination of multiple computing devices. As an example of a computing device, a CPU, a GPU, a GPGPU, an APU, or a TPU can be cited.

[0045] In the following description, a memory with a symbol is a memory such as a RAM that temporarily stores information and is used as a working memory by a processor.

[0046] In the following description, the storage device with a symbol is one or more non-volatile storage devices that store various programs and various parameters, etc. As an example of a non-volatile storage device, a flash memory, a magnetic disk, or a magnetic tape, etc., can be cited. And as another example of a storage device, a cloud storage device can be cited.

[0047] In the following embodiments, the external I / F with a symbol controls the exchange of various information between multiple devices connected to each other. As an example of an external I / F, a USB interface can be cited. The external I / F can also be applied to a communication I / F including a communication processor and an antenna. The communication I / F controls the communication between multiple computers. As an example of a communication standard applicable to the communication I / F, wireless communication standards including 5G, Wi-Fi (registered trademark) or Bluetooth (registered trademark) can be cited.

[0048] In the following embodiments, "A and / or B" has the same meaning as "at least one of A and B". That is, "A and / or B" means that it can be only A, only B, or a combination of A and B. In addition, in this specification, when three or more items are expressed in association with "and / or", the same concept as "A and / or B" also applies.

[0049] As an example, Figure 1 As shown in FIG. 1 , the imaging device 10 captures an imaging target area 12 designated as a subject. The imaging target area 12 is determined according to the angle of view designated by a photographer 13 who is a user of the imaging device 10. Figure 1 In the example shown, the imaging target area 12 includes not only the blue sky 12A but also clouds, mountains, trees, roads, etc. In the present embodiment, the imaging device 10 is an example of the “imaging device” according to the present invention.

[0050] The imaging device 10 generates a captured image 14 showing the imaging target area 12 by capturing the imaging target area 12 according to an instruction from the photographer 13. In the present embodiment, the imaging target area 12 is an example of a "subject" involved in the present invention, and the captured image 14 is an example of a "captured image" involved in the present invention.

[0051] The camera device 10 is a civilian digital camera. As an example of a civilian digital camera, a lens-interchangeable digital camera or a lens-fixed digital camera can be cited. A civilian digital camera is only an example, and even if the camera device 10 is an industrial digital camera, the present invention is also applicable. Even if the camera device 10 is an image-taking device mounted on various electronic devices such as a driving recorder, a smart device, a wearable terminal, an endoscope device, a cell observation device, an ophthalmic observation device, or a surgical microscope, the present invention is also applicable.

[0052] As an example, Figure 2 As shown, the imaging device 10 includes a computer 30, an image sensor 32, a UI device 34, and an external I / F 36. In this embodiment, the image sensor 32 is an example of the "image sensor" involved in the present invention, and the computer 30 is an example of the "image processing device" and "computer" involved in the present invention.

[0053] The computer 30 includes a processor 42, a storage device 44, and a memory 46. The processor 42, the storage device 44, and the memory 46 are connected to a bus 48. In the present embodiment, the processor 42 is an example of the "first processor" and the "second processor" according to the present invention.

[0054] The storage device 44 is a non-volatile storage device (i.e., a non-temporary storage medium readable by a computer) that stores various programs and various parameters. As the storage device 44, for example, a flash memory (e.g., EEPROM) can be cited. The memory 46 is a storage area for temporarily storing information, and is used as a working memory by the processor 42. As an example of the memory 46, a RAM can be cited.

[0055] The image sensor 32 has a plurality of photosensitive pixels (not shown) arranged in a matrix. Each photosensitive pixel is a physical pixel having a photodiode (not shown), which performs photoelectric conversion on the received light and outputs an electrical signal corresponding to the amount of light received. In addition, color filters (not shown) of the three primary colors of light (i.e., red (hereinafter, also referred to as "R"), green (hereinafter, also referred to as "G"), or blue (hereinafter, also referred to as "B")) are arranged in a predetermined pattern on the plurality of photosensitive pixels. In the present embodiment, the Bayer arrangement is used as an example of a predetermined pattern arrangement. However, the Bayer arrangement is only an example, and the present invention is also applicable even if the predetermined pattern arrangement is other types of pattern arrangements such as a G stripe R / G complete grid, an X-Trans (registered trademark) arrangement, or a honeycomb arrangement.

[0056] In addition, hereinafter, for the convenience of explanation, a photosensitive pixel having a microlens and an R color filter is referred to as an "R photosensitive pixel", a photosensitive pixel having a microlens and a G color filter is referred to as a "G photosensitive pixel", and a photosensitive pixel having a microlens and a B color filter is referred to as a "B photosensitive pixel". Also, hereinafter, for the convenience of explanation, an electrical signal output from an R photosensitive pixel is referred to as an "R signal" or an "R color signal", an electrical signal output from a G photosensitive pixel is referred to as a "G signal" or a "G color signal", and an electrical signal output from a B photosensitive pixel is referred to as a "B signal" or a "B color signal". Also, hereinafter, for the convenience of explanation, when there is no need to distinguish between the R color signal, the G color signal, and the B color signal, they are referred to as "color signals". Also, hereinafter, for the convenience of explanation, a signal value of an R signal is referred to as an "R signal value", a signal value of a G signal is referred to as a "G signal value", and a signal value of a B signal is referred to as a "B signal value". Also, when there is no need to distinguish between the R signal value, the G signal value, and the B signal value, they are referred to as "color signal values".

[0057] The image sensor 32 is connected to the bus 48. The image sensor 32 captures the image capturing area 12 (see FIG. Figure 1 ) to generate the captured image 14. The captured image 14 is an image in RGB format, and is formed by R pixels (hereinafter referred to as "R pixels"), G pixels (hereinafter referred to as "G pixels"), and B pixels (hereinafter referred to as "B pixels").

[0058] A CMOS image sensor can be cited as an example of the image sensor 32. Here, a CMOS image sensor is cited as an example of the image sensor 32, but this is only an example, and the image sensor 32 may be another type of image sensor such as a CCD image sensor.

[0059] The UI system device 34 is also connected to the bus 48. The UI system device 34 receives the information from the photographer 13 (refer to Figure 1 ) and outputs a signal indicating the received instruction to the processor 42. In addition, the UI system device 34 presents various information to the photographer 13 under the control of the processor 42. The presentation of various information is achieved by, for example, displaying the various information on a display or outputting the various information in the form of sound from a speaker.

[0060] The external I / F 36 controls the exchange of various information with a device (hereinafter, also referred to as an "external device") existing outside the camera 10. As an example of the external I / F 36, a USB interface can be cited. External devices such as smart devices, personal computers, servers, USB memories, memory cards, and / or printers (not shown) are directly or indirectly connected to the USB interface.

[0061] The captured image 14 has a blue sky 12A (reference Figure 1 ) of the image area 16. The brightness area (in other words, the brightness range) of the image area 16 is divided into a high brightness area H, a medium brightness area M, and a low brightness area L. Figure 2 In the example shown, the brightness region of the image region 16 is divided into three regions at equal intervals: a high brightness region H, a medium brightness region M, and a low brightness region L. The high brightness region H is a region of high brightness in the image region 16 (in other words, a range of high brightness), the medium brightness region M is a region of medium brightness in the image region 16 (in other words, a range of medium brightness), and the low brightness region L is a region of low brightness in the image region 16 (in other words, a range of low brightness). In addition, for the sake of convenience, the brightness range of the image region 16 is used as an example for explanation, but the brightness region is also defined similarly for the image region other than the image region 16 (that is, the image region representing the subject other than the blue sky 12A), and the color of the image region other than the image region 16 is also expressed by the R signal value, the G signal value, and the B signal value.

[0062] The image region 16 is an image region expressed by the R signal value 54, the G signal value 56, and the B signal value 58. That is, the color of the image region 16 is expressed by the R signal value 54, the G signal value 56, and the B signal value 58. The R signal value 54, the G signal value 56, and the B signal value 58 are saturated in an order corresponding to the color of the image region 16 (that is, the color of the blue sky 12A) as the brightness of the image region 16 increases. In the present embodiment, the R signal value 54, the G signal value 56, and the B signal value 58 expressing the captured image 14 are an example of the "plurality of fourth signal values" involved in the present invention.

[0063] In the low brightness region L, the magnitude relationship of the R signal value 54, the G signal value 56, and the B signal value 58 is "B signal value 58>G signal value 56>R signal value 54". In addition, in the low brightness region L, the R signal value 54, the G signal value 56, and the B signal value 58 increase at a constant ratio as the brightness of the image region 16 increases. In addition, the B signal value 58 is saturated at the boundary position between the low brightness region L and the medium brightness region M.

[0064] In the medium brightness region M, the magnitude relationship among the R signal value 54, the G signal value 56, and the B signal value 58 is also "B signal value 58>G signal value 56>R signal value 54". In addition, in the medium brightness region M, the B signal value 58 maintains a saturated state. The G signal value 56 and the B signal value 58 increase at the same rate as in the low brightness region L. In addition, the G signal value 56 is saturated at the boundary position between the medium brightness region M and the high brightness region H.

[0065] In the high brightness region H, the B signal value 58 and the G signal value 56 are maintained in a saturated state. The R signal value 54 continues to increase while maintaining the increase rate in the low brightness region L and the medium brightness region M, and is saturated at the upper limit brightness of the high brightness region H. Figure 2 In the example shown, the saturation state of the B signal value 58 is maintained from the boundary position of the low brightness area L and the medium brightness area M to the upper limit brightness of the high brightness area H, and the saturation state of the G signal value 56 is maintained from the boundary position of the medium brightness area M and the high brightness area H to the upper limit brightness of the high brightness area H.

[0066] Thus, if the B signal value 58 and the G signal value 56 are saturated at a lower brightness than the R signal value 54, a color-saturated region 16A is generated in the image region 16. The color-saturated region 16A is a region (e.g., a region where white dominates over other colors) that is not the original color of the blue sky 12A (e.g., a color perceived by a person with normal visual ability through the naked eye).

[0067] For example, in the color saturation area 16A, blue is originally to be expressed, but cyan is expressed because the B signal value 58 is saturated from the boundary position between the low brightness area L and the medium brightness area M to the high brightness area H on the lower brightness side than the other color signal values. Also, in the color saturation area 16A, blue, cyan, or a color intermediate between blue and cyan is originally to be expressed, but whitening occurs because the G signal value 56 is saturated from the boundary position between the medium brightness area M and the high brightness area H to the high brightness area H on the lower brightness side than the R signal value 54.

[0068] Here, the color reproducibility of the blue sky 12A is used as an example for explanation, but of course, the same explanation can be given for subjects other than the blue sky 12A. For example, in an image obtained by photographing an orange railing, an image area with saturated color signal values ​​may also be generated based on the same principle as the color saturation area 16A. The magnitude relationship of the color signal values ​​of the image area representing the orange railing is expressed as "R signal value 54>G signal value 56>B signal value 58", and the R signal value 54 is saturated on the lower brightness side than the G signal value 56 and the B signal value 58, and the G signal value 56 is saturated on the lower brightness side than the B signal value 58. Therefore, whitening occurs based on the same principle as the color saturation area 16A.

[0069] Therefore, in order to make the color reproduction of the subject shown in the captured image 14 reach the same level as the color of the subject in the real space (that is, to express the original color of the subject on the image), in this embodiment, the processor 42 performs high image quality processing. In order to realize the high image quality processing, the storage device 44 stores a high image quality program 50 and a learned model 52. The high image quality program 50 is an example of the "program" involved in the present invention.

[0070] The processor 42 reads the high-quality image processing program 50 from the storage device 44 and executes the read high-quality image processing program 50 on the memory 46. The high-quality image processing is realized by the processor 42 operating as the acquisition unit 42A and the generation unit 42B according to the high-quality image processing program 50 executed on the memory 46. The learned model 52 is used by the acquisition unit 42A, and the details will be described later.

[0071] Next, refer to Figure 3 to Figure 5 , an example of a method for generating the learned model 52 is described.

[0072] As an example, Figure 3 As shown in FIG. 8 , the learning device 79 includes a processor 80, a storage device 82, and a memory 84. The hardware structure of the learning device 79 (for example, the processor 80, the storage device 82, and the memory 84) is similar to Figure 2 The hardware structure of the computer 30 shown is basically the same, so the description related to the hardware structure of the learning device 79 is omitted here. In addition, in this embodiment, the processor 80 is an example of the "third processor" involved in the present invention, and the learning device 79 is an example of the "learning device" involved in the present invention.

[0073] The storage device 82 stores a learning program 90. The processor 80 reads the learning program 90 from the storage device 82 and executes the read learning program 90 on the memory 84. The processor 80 performs a learning process according to the learning program 90 executed on the memory 84. The learning process is a process of generating a learned model 52 from a model 98. The learned model 52 is generated by the processor 80 performing machine learning on the model 98. That is, the learned model 52 is generated by optimizing the model 98 through machine learning. For example, the model 98 is a neural network having hundreds of millions to trillions of intermediate layers. As an example of the model 98, there is a method of generating and outputting an image with higher image quality than the input image (for example, at least the color saturation area 16A (reference Figure 2 ) obtains the suppressed image) using a model for generating AI.

[0074] The storage device 82 stores a plurality of (e.g., tens of thousands to hundreds of billions) training data 92. The training data 92 is used for machine learning of the model 98. That is, in the learning device 79, the processor 80 acquires the plurality of training data 92 from the storage device 82, and optimizes the model 98 by using the acquired plurality of training data 92 to perform machine learning on the model 98, thereby generating a learned model 52.

[0075] The training data 92 is labeled data. Labeled data is data in which a corresponding association is established between an example question image 94 (in other words, example question data) and a correct answer image 96 (in other words, correct answer data). Here, the training data 92 is an example of "training data" involved in the present invention, the example question image 94 is an example of "example question image" involved in the present invention, and the correct answer image 96 is an example of "correct answer image" involved in the present invention.

[0076] The example image 94 is an image in RGB format that is determined based on the assumption that the captured image 14 is captured. Figure 3 In the example shown, the example image 94 has the assumption that Figure 2 The image area 16 is shown as an image area 95 representing the image area of ​​the blue sky. The image area 95 includes the assumed Figure 2 Color saturation region 95A is shown as color saturation region 16A.

[0077] In the present embodiment, as the example image 94, a sample subject (for example, a subject shown on a screen) is actually photographed by an imaging device (for example, an imaging device of the same type and specification as the imaging device 10). Figure 3 However, this is only an example, and it is also possible to have an image virtually generated as an image of a camera image 14 assumed to be obtained by taking a picture by the camera device 10. As an example of a virtually generated image, there can be cited an image generated by a generative AI. The generative AI can be an AI specifically used to generate images, or it can be a generative AI that generates and outputs the example image 94 based on the input instruction data (the so-called prompt), such as ChatGPT using GPT-4 (Internet search <https: / / openai.com / gpt-4>).

[0078] Correct answer image 96 is an image in RGB format similar to example question image 94. Correct answer image 96 is a high-definition image of ratio question image 94. An example of a high-definition image is an image in which color saturation region 95A of example question image 94 is suppressed (for example, an image including image region 95 in which color saturation region 95A is eliminated and which is the same as example question image 94 except for image region 95).

[0079] The processor 80 acquires the training data 92 one by one from the storage device 82. The processor 80 inputs the example image 94 from the training data 92 acquired from the storage device 82 to the model 98. When the example image 94 is input, the model 98 generates a comparison image 100 which is a high-definition image of the ratio question image 94 (here, as an example, an image in which the color saturation area 95A is suppressed). The comparison image 100 is an image for comparison with the correct answer image 96 associated with the example image 94 input to the model 98. Here, the comparison image 100 is an example of the "evaluation target image" involved in the present invention.

[0080] Processor 80 calculates error 102 between correct answer image 96 associated with example question image 94 input to model 98 and comparison image 100. Error 102 is an example of "comparison result" according to the present invention.

[0081] The processor 80 calculates a plurality of adjustment values ​​104 that minimize the error 102. Then, the processor 80 uses the plurality of adjustment values ​​104 to adjust a plurality of optimization variables in the model 98. Here, the plurality of optimization variables refers to, for example, a plurality of connection weights and a plurality of offset values ​​included in the model 98.

[0082] The processor 80 repeatedly performs a series of processes, including input of the example image 94 to the model 98, calculation of the error 102, calculation of the plurality of adjustment values ​​104, and adjustment of the plurality of optimization variables in the model 98, using the plurality of training data 92 stored in the storage device 82. That is, the processor 80 adjusts the plurality of optimization variables in the model 98 using the plurality of adjustment values ​​104 calculated so as to minimize the error 102 for each of the plurality of example images 94 included in the plurality of training data 92 stored in the storage device 82, thereby optimizing the model 98. The processor 80 generates the learned model 52 by optimizing the model 98. When the example image 94 is input to the learned model 52 generated in this way, the learned model 52 generates and outputs an image having the same image quality as the correct answer image 96 (i.e., an image in which the color saturation region 95A is suppressed at the same level as the correct answer image 96) as an image corresponding to the input example image 94.

[0083] Figure 4 94 is a conceptual diagram showing an example of a method for creating an example image 94. Figure 4As shown, example image 94 is an image generated based on reference image 110. Reference image 110 is an image obtained by photographing subjects 108 including person 108A and blue sky 108B in an underexposed manner by reference camera 500 (for example, a camera of the same type and specification as camera 10) so that color signal values ​​are desaturated. Reference image 110 is an image in RGB format, expressed by color signal values. R signal value 54, G signal value 56, and B signal value 58 expressing reference image 110 increase at a constant rate as the brightness of subject 108 increases. R signal value 54, G signal value 56, and B signal value 58 expressing blue sky 108B reflected in reference image 110 increase monotonically from low brightness area L that is unsaturated to high brightness area H. With respect to the color signal value of the image area representing blue sky 108B, the magnitude relationship of "B signal value 58>G signal value 56>R signal value 54" holds. In Figure 4 In the illustrated example, the R signal value 54 , the G signal value 56 , and the B signal value 58 expressing the reference image 110 are examples of “a plurality of third signal values ​​representing the three primary colors” according to the present invention.

[0084] Example image 94 is an image expressed by R signal value 54, G signal value 56, and B signal value 58. That is, the color of example image 94 is expressed by R signal value 54, G signal value 56, and B signal value 58. Example image 94 is obtained by increasing the gain (i.e., digital gain) of R signal value 54, G signal value 56, and B signal value 58 at a uniform increase rate (e.g., an increase rate determined according to the subject (here, as an example, the blue sky 108B) reflected in the reference image 110) by the amount of underexposure by the reference camera device 500 in the shooting used to obtain the reference image 110. In the case where the color signal value exceeds the saturation value (e.g., 255) due to the application of gain, the color signal value is clipped to the saturation value. R signal value 54, G signal value 56, and B signal value 58 expressing example image 94 are saturated in sequence as the brightness of the subject (here, as an example, the brightness of the blue sky 108B) increases. Figure 4 In the example shown, the color signal values ​​are saturated in the order of B signal value 58 , G signal value 56 , and R signal value 54 .

[0085] In the present embodiment, the blue sky 108B is intentionally photographed in an underexposed manner by the reference imaging device 500. The reason for this is to intentionally prevent the color signal value from being saturated. Therefore, in the present embodiment, in the process of creating the example image 94 based on the reference image 110, by increasing the gains of the R signal value 54, the G signal value 56 and the B signal value 58 by a uniform increase rate, the ratio between the R signal value 54, the G signal value 56 and the B signal value 58 expressing the image area 95 included in the example image 94 (for example, the ratio at which the R signal value 54, the G signal value 56 and the B signal value 58 in the low brightness area L increase according to the brightness of the subject) becomes higher than the ratio between the R signal value 54, the G signal value 56 and the B signal value 58 expressing the blue sky 108B reflected in the reference image 110 (that is, the ratio at which the R signal value 54, the G signal value 56 and the B signal value 58 expressing the blue sky 108B reflected in the reference image 110 increase according to the brightness of the subject).

[0086] Thus, as an expression of the ratio among the R signal value 54, the G signal value 56, and the B signal value 58 of the image region 95 (for example, the ratio in the low brightness region L), the same as that achieved by expressing the blue sky 12A captured by the camera device 10 under the actual imaging conditions (reference Figure 1 ) obtained by taking a camera image 14 (reference Figure 1 and Figure 2 ) is the same ratio as the ratio among the R signal value 54, the G signal value 56, and the B signal value 58 (for example, the ratio in the low brightness area L). Figure 4 In the example shown, the R signal value 54, the G signal value 56, and the B signal value 58 of the image area 95 included in the example image 94 are expressed with respect to the brightness of the subject (here, as an example, the brightness of the blue sky) shown in the example image 94. Figure 2 The R signal value 54 , the G signal value 56 , and the B signal value 58 of the image region 16 shown behave in the same manner.

[0087] That is, R signal value 54, G signal value 56, and B signal value 58 expressing image region 95 are saturated in the order corresponding to the color of image region 95 (here, as an example, the color of blue sky 108B) as the brightness of image region 95 increases. In the case of image region 95, as the brightness of image region 95 increases, B signal value 58, G signal value 56, and R signal value 54 are saturated in this order. Figure 4 In the example shown, R signal value 54, G signal value 56, and B signal value 58 of image region 95 included in expression example image 94 are examples of “plurality of first signal values ​​representing three primary colors of light” according to the present invention.

[0088] In addition, the case of the blue sky 108B is illustrated here, but this is only an example, and in the case of a subject other than the blue sky 108B (i.e., an image area other than the image area 95), the R signal value 54, the G signal value 56, and the B signal value 58 expressing the other image areas as the image areas other than the image area 95 are also saturated in an order corresponding to the colors of the other image areas.

[0089] Furthermore, here, an example of a form in which the reference image 110 is obtained by shooting based on the reference camera 500 is listed, but this is only an example. For example, the reference image 110 may also be an image virtually generated as an image that is assumed to be obtained by shooting the subject 108 by the reference camera 500 in an underexposed manner so that the color signal value is not saturated. As an example of a virtually generated image, an image generated by a generation AI can be cited. The generation AI may be an AI specifically used to generate images, or it may be a generation AI that generates and outputs the reference image 110 based on input instruction data (so-called prompts), such as ChatGPT using GPT-4 or the like.

[0090] Figure 5 96 is a conceptual diagram showing an example of a method for creating a correct answer image 96. Figure 5 As shown, the correct answer image 96 is an image generated based on the reference image 110. Like the example image 94, the correct answer image 96 is also an image expressed by the R signal value 54, the G signal value 56, and the B signal value 58. That is, the color of the correct answer image 96 is expressed by the R signal value 54, the G signal value 56, and the B signal value 58. The R signal value 54, the G signal value 56, and the B signal value 58 expressing the correct answer image 96 are not saturated in the low brightness area L and the medium brightness area M of the subject 108, but increase as the brightness of the subject 108 increases. Figure 5 In the example shown, the following example is shown: the R signal value 54, G signal value 56, and B signal value 58 of the image area 95 expressing the blue sky 108B in the entire image area representing the correct answer image 96 are not saturated in the low brightness area L and the medium brightness area M, and increase as the brightness of the blue sky 108B reflected in the correct answer image 96 increases. However, this is only an example, and with respect to objects other than the blue sky 108B, the R signal value 54, G signal value 56, and B signal value 58 are also not saturated in the low brightness area L and the medium brightness area M, and increase as the brightness of the object increases. In the present embodiment, the R signal value 54, G signal value 56, and B signal value 58 expressing the correct answer image 96 are an example of the "plurality of second signal values" involved in the present invention.

[0091] The correct answer image 96 is obtained by increasing the gain of the R signal value 54, the G signal value 56, and the B signal value 58 by the amount of underexposure by the reference camera 500 in the shooting for obtaining the reference image 110, and making the color signal value unsaturated in the low brightness area L and the medium brightness area M, and saturated in the high brightness area H. The amount of gain increase varies depending on the brightness of the subject. Here, the amount of gain increase is used as the amount of gain increase, which is derived in advance from the brightness of the blue sky 108B through experiments based on actual machines and / or computer simulations. In addition, similar to the reference image 110, in the correct answer image 96, the magnitude relationship of the R signal value 54, the G signal value 56, and the B signal value 58 is maintained from the low brightness area L to the high brightness area H. That is, the magnitude relationship of "B signal value 58>G signal value 56>R signal value 54" is maintained between the R signal value 54, the G signal value 56, and the B signal value 58.

[0092] In addition, here, as an example of expressing the size relationship of the color signal values ​​of the blue sky 108B, an example of a form in which the size relationship of "B signal value 58>G signal value 56>R signal value 54" is listed, but this is only an example, and in the case of subjects other than the blue sky 108B, the size relationship of the color signal values ​​depends on the color of the subject.

[0093] The R signal value 54, G signal value 56, and B signal value 58 expressing the correct answer image 96 are saturated at the highest brightness of the subject (here, as an example, the highest brightness of the blue sky 108B). Figure 5 In the example shown, the maximum brightness of the subject refers to the upper limit brightness of the high brightness area H. The R signal value 54, G signal value 56, and B signal value 58 expressing the blue sky 108B reflected in the correct answer image 96 (i.e., the R signal value 54, G signal value 56, and B signal value 58 expressing the image area 95) are controlled to be whitish at the upper limit brightness of the high brightness area H. The color signal value is classified into a minimum signal value which is the lowest signal value among the color signal values, a maximum signal value which is the highest signal value among the color signal values, and an intermediate signal value which is a signal value between the minimum signal value and the maximum signal value among the color signal values. Figure 5In the example shown, the lowest signal value expressing the correct answer image 96 is the R signal value 54, the highest signal value expressing the correct answer image 96 is the B signal value 58, and the middle signal value expressing the correct answer image 96 is the G signal value 56. The ratio of the R signal value 54 expressing the correct answer image 96 to the increase in brightness (i.e., the increase rate of the R signal value 54) is constant. The ratio of the G signal value 56 expressing the correct answer image 96 to the increase in brightness (i.e., the increase rate of the G signal value 56) decreases in the order of the low brightness region L, the medium brightness region M, and the high brightness region H. Similar to the increase rate of the G signal value 56, the ratio of the B signal value 58 expressing the correct answer image 96 to the increase in brightness (i.e., the increase rate of the B signal value 58) also decreases in the order of the low brightness region L, the medium brightness region M, and the high brightness region H.

[0094] Furthermore, in the correct answer image 96, in the dark portion (e.g., low brightness region L), the gains of the R signal value 54, the G signal value 56, and the B signal value 58 increase at a uniform increase rate (i.e., an increase rate determined according to the subject shown in the reference image 110). Here, as an example, the gains of the R signal value 54, the G signal value 56, and the B signal value 58 in the dark portion uniformly increase at an increase rate determined according to the blue sky 108B shown in the reference image 110.

[0095] Furthermore, in the correct answer image 96, in the medium brightness region M, the highest signal value (in Figure 5 In the example shown, the B signal value 58 is saturated, and the G signal value 56 and the B signal value 58 are suppressed so that the ratio among the R signal value 54, the G signal value 56 and the B signal value 58 is maintained as much as possible.

[0096] In addition, here, an example of a form in which the correct answer image 96 is generated based on the reference image 110 is listed, but the correct answer image 96 may also be an image virtually generated in a manner that satisfies the above-mentioned various conditions. As an example of a virtually generated image, an image generated by a generation AI may be cited. The generation AI may be an AI specifically used to generate images, or may be a generation AI that generates and outputs the correct answer image 96 based on input instruction data (so-called prompts), such as ChatGPT using GPT-4 or the like.

[0097] Figure 6 It means that by Figure 3 1 is a conceptual diagram of an example of the application stage of the learned model 52 generated by the learning process of the example shown in FIG. Figure 6As shown, in the camera device 10, the acquisition unit 42A acquires the camera image 14 from the image sensor 32. The acquisition unit 42A inputs the camera image 14 to the learned model 52 stored in the storage device 44, so that the learned model 52 generates and outputs the AI ​​image 112, and acquires the AI ​​image 112 output from the learned model 52. The AI ​​image 112 is an image in which the color saturation area 16A of the camera image 14 is suppressed (in Figure 6 In the example shown, the image is an image after the color saturated area 16A is removed from the camera image 14). Here, the learned model 52 is an example of the "learned model" involved in the present invention, the camera image 14 is an example of the "image for inference" involved in the present invention, and the AI ​​image 112 is an example of the "first image" and "inference result" involved in the present invention.

[0098] The generating unit 42B generates a high-quality image 114 by mixing the captured image 14 and the AI ​​image 112. The high-quality image 114 is an image with a higher quality than the captured image 14. An image with a higher quality than the captured image 14 is an image with a higher quality than the captured image 14 in that at least the color saturation region 16A of the captured image 14 is suppressed.

[0099] The high-quality image 114 is an image generated by mixing the captured image 14 and the AI ​​image 112 in units of pixels corresponding to each other. Here, an example of mixing the captured image 14 and the AI ​​image 112 in units of pixels corresponding to each other is given, but this is only an example, and the captured image 14 and the AI ​​image 112 may also be mixed in units of blocks (i.e., units composed of a plurality of pixels) which are larger than pixels.

[0100] Mixing the camera image 14 and the AI ​​image 112, for example, refers to averaging the camera image 14 and the AI ​​image 112 or mixing the camera image 14 and the AI ​​image 112 at a predetermined mixing ratio. Here, the mixing ratio refers to, for example, the ratio of adding the AI ​​image 112 and the camera image 14. For example, if the mixing ratio is 1:1, half of the color signal value of the AI ​​image 112 and half of the color signal value of the camera image 14 are added in units of pixels; if the mixing ratio is 1:0, the color signal value of the AI ​​image 112 is used in units of pixels, and the color signal value of the camera image 14 is not used; if the mixing ratio is 0:1, the color signal value of the camera image 14 is used in units of pixels, and the color signal value of the AI ​​image 112 is not used. Figure 6 In the figure, the high-quality image 114 is an example of the “second image” involved in the present invention, and the pixel unit is an example of the “reference area unit” involved in the present invention.

[0101] The generator 42B files the high-quality image 114 in a predetermined file format by performing image processing including development on the high-quality image 114, and outputs the high-quality image 114 to a predetermined output destination. As a first example of the predetermined output destination, the storage device 44 can be cited. As a second example of the predetermined output destination, the UI system device 34 can be cited. As a third example of the predetermined output destination, a device connected to the external I / F 36 (for example, a USB memory, a smart device, a personal computer and / or a server, etc.) can be cited.

[0102] Next, refer to Figure 7 The operation of the portion of the learning device 79 that is related to the present invention will be described. Figure 7 An example of the flow of the learning process executed by the processor 80 is shown in FIG. Figure 7 The illustrated flow of the learning process is an example of the “method for generating a learned model” according to the present invention.

[0103] exist Figure 7 In the learning process shown, first, in step ST10, the processor 80 executes to obtain the unprocessed training data 92 (ie, Figure 7 After the processing of step ST10 of the training data 92 not used in the learning process shown in the figure, the learning process proceeds to step ST12.

[0104] In step ST12, processor 80 inputs example image 94 included in training data 92 acquired in step ST10 to model 98. After executing the process of step ST12, the learning process proceeds to step ST14. By executing the process of step ST12, comparison image 100 is output from model 98.

[0105] In step ST14, processor 80 acquires comparison image 100 output from model 98. After executing the process of step ST14, the learning process proceeds to step ST16.

[0106] In step ST16, processor 80 compares comparison image 100 acquired in step ST14 with correct answer image 96 included in training data 92 acquired in step ST10. After executing the process of step ST16, the learning process proceeds to step ST18.

[0107] In step ST18, processor 80 adjusts model 98 using a plurality of adjustment values ​​104 obtained by comparing comparison image 100 with correct answer image 96 in step ST16. Model 98 is optimized by repeatedly executing the process of step ST18 based on all training data 92 stored in storage device 82. After executing the process of step ST18, the learning process proceeds to step ST20.

[0108] In step ST20, processor 80 determines whether unprocessed training data 92 is stored in storage device 82. In step ST20, if unprocessed training data 92 is stored in storage device 82, the determination is affirmative, and the learning process proceeds to step ST10. In step ST20, if unprocessed training data 92 is not stored in storage device 82, the determination is negative, and the learning process ends.

[0109] Next, refer to Figure 8 The operation of the portion of the imaging device 10 according to the present invention will be described. Figure 8 The illustrated flow of the image quality enhancement process is an example of the “inference method” according to the present invention. Here, for the sake of convenience, the description is given on the premise that the learned model 52 has already been stored in the storage device 44 .

[0110] exist Figure 8 In the high-quality image processing shown, first, in step ST50, the acquisition unit 42A determines whether one frame of image is captured by the image sensor 32. In step ST50, if one frame of image is not captured by the image sensor 32, the determination is negative, and the high-quality image processing proceeds to step ST58. If one frame of image is captured by the image sensor 32, the determination is positive, and the high-quality image processing proceeds to step ST52.

[0111] In step ST52, the acquisition unit 42A acquires the captured image 14 from the image sensor 32. After the process of step ST52 is executed, the image quality improvement process proceeds to step ST54.

[0112] In step ST54, the acquisition unit 42A inputs the captured image 14 acquired in step ST52 to the learned model 52. After the process of step ST54 is executed, the image quality improvement process proceeds to step ST56. By executing the process of step ST54, the learned model 52 generates and outputs the AI ​​image 112.

[0113] In step ST56, the acquisition unit 42A acquires the AI ​​image 112 output from the learned model 52. After the process of step ST56 is executed, the image quality improvement process proceeds to step ST58.

[0114] In step ST58, the generator 42B generates the high-quality image 114 by mixing the captured image 14 acquired by the acquirer 42A in step ST52 and the AI ​​image 112 acquired by the acquirer 42A in step ST54. After executing the process of step ST58, the high-quality image processing proceeds to step ST60.

[0115] In step ST60, the generator 42B determines whether a condition for ending the high-quality image processing (hereinafter referred to as "ending condition") is satisfied. As the ending condition, the UI system device 34 receives an instruction to end the high-quality image processing. In step ST60, if the ending condition is not satisfied, the determination is negative, and the high-quality image processing proceeds to step ST50. In step ST60, if the ending condition is satisfied, the determination is positive, and the high-quality image processing ends.

[0116] As described above, the training data 92 involved in the present embodiment is used for machine learning of the model 98. The learned model 52 is generated by machine learning of the model 98. The training data 92 includes an example image 94 and a correct answer image 96, and a corresponding association is established between the example image 94 and the correct answer image 96. The example image 94 is an image determined by assuming the camera image 14, and the color of each pixel of the example image 94 is expressed by a color signal value. And, similar to the example image 94, with respect to the correct answer image 96, the color of each pixel of the correct answer image 96 is also expressed by a color signal value. By photographing the blue sky 12A (reference Figure 1 ) obtained by image area 16 (reference Figure 2 ) saturates in the order of B signal value 58, G signal value 56, and R signal value 54 as the brightness of the image area 16 increases. Here, if the B signal value 58 and the G signal value 56 are saturated in the medium brightness area M, a color saturation area 16A is generated. The color saturation area 16A is larger than the blue sky 12A (reference Figure 1 ) of the image area whose original color is white. Therefore, the color signal value expressing the correct answer image 96 included in the training data 92 is constituted to be unsaturated in the low brightness area L and the medium brightness area M and to be saturated as the brightness of the subject (here, as an example, the brightness of the blue sky 108B) increases. Furthermore, a learned model 52 in which the model 98 is optimized is generated by performing machine learning on the model 98 using the training data 92 constituted in this way. The learned model 52 generates and outputs an AI image 112 based on the input of the camera image 14. The AI ​​image 112 is an image in which the degree of reproduction of the color of the projected subject is higher than that of the camera image 14 (i.e., an image in which the color saturation area 16A is suppressed). In this way, when a video image 14 is input into a learned model 52 generated by optimizing the model 98 by using the training data 92 involved in this embodiment for machine learning of the model 98, the learned model 52 can generate an AI image 112 in which the color of the projected subject is reproduced more accurately than that of the input video image 14 (i.e., an image in which the color saturation area 16A is suppressed).

[0117] Furthermore, in the present embodiment, the imaging device 10 uses the learned model 52 generated by performing machine learning on the model 98 using the training data 92. Therefore, the imaging device 10 can generate and output the AI ​​image 112 in which the color saturation area 16A is suppressed compared to the captured image 14 by inputting the captured image 14 into the learned model 52.

[0118] Furthermore, in the present embodiment, the color signal value expressing the example image 94 saturates in the order corresponding to the color of the subject 108 as the brightness of the subject 108 increases. For example, the color signal value expressing the example image 94 saturates in the order of the B signal value 58, the G signal value 56, and the R signal value 54 as the brightness of the blue sky 108B increases. Therefore, when the camera image 14 is input to the learned model 52 generated by using the training data 92 including the example image 94 thus constructed for machine learning of the model 98 to optimize the model 98, the learned model 52 can generate an AI image 112 (i.e., an image in which the color saturation region 16A is suppressed) in which the color of the projected subject is reproduced more highly than the input camera image 14.

[0119] Furthermore, in the present embodiment, the color signal value expressing the reference image 110 increases at a constant rate as the brightness of the subject 108 increases. The example image 94 and the correct answer image 96 are images generated from the reference image 110 expressed by the same color signal value as the example image 94 and the correct answer image 96. In this way, the example image 94 and the correct answer image 96 are generated from a common image (i.e., the reference image 110), so that the example image 94 and the correct answer image 96 can be generated more efficiently than when the example image 94 and the correct answer image 96 are generated from different images. Furthermore, the example image 94 and the correct answer image 96 having the same viewing angle as the reference image 110 can be easily generated.

[0120] Furthermore, in the present embodiment, the constant rate of increase of the color signal value of the expression reference image 110 is lower than the rate of increase of the color signal value of the expression example image 94. Therefore, compared with the case where the constant rate of increase of the color signal value of the expression reference image 110 is higher than the rate of increase of the color signal value of the expression example image 94, it is easier to generate an ideal example image 94 based on the reference image 110.

[0121] Furthermore, in the present embodiment, the example image 94 is generated by increasing the gain of the color signal value expressing the reference image 110. Therefore, the example image 94 assuming the captured image 14 can be easily obtained.

[0122] Furthermore, in the present embodiment, the gain of the color signal value expressing the reference image 110 is increased, the magnitude relationship of the R signal value 54, the G signal value 56, and the B signal value 58 is maintained from the low brightness region L to the high brightness region H, and the color signal value is made unsaturated in the low brightness region L and the medium brightness region M but saturated in the high brightness region H, thereby generating the correct answer image 96. Furthermore, the amount of gain increase varies depending on the brightness of the subject 108. For example, the amount of gain increase used for the blue sky 108B is determined in advance according to the brightness of the blue sky 108B. Therefore, the correct answer image 96 in which the color saturation region 95A is suppressed can be easily obtained.

[0123] Furthermore, in the present embodiment, the high-quality image 114 is generated by mixing the captured image 14 and the AI ​​image 112. The high-quality image 114 is an image in which the color saturation region 16A is suppressed compared to the captured image 14. The AI ​​image 112 is an image in which the color saturation region 16A is suppressed compared to the high-quality image 114. However, if the component of the color saturation region 16A is too small, it is difficult to reproduce the color of the original imaging target region 12 (here, the blue sky 12A). Therefore, in the present embodiment, the captured image 14 and the AI ​​image 112 are mixed to leave a very small amount of the component of the color saturation region 16A. As a result, a high-quality image 114 close to the color of the actual imaging target region 12 can be obtained. Furthermore, since the captured image 14 and the AI ​​image 112 are mixed in pixel units corresponding to each other in position, a high-quality image 114 close to the color of the actual imaging target region 12 can be obtained compared to a case where they are not mixed in pixel units.

[0124] In the above embodiment, an example is given in which the R signal value 54, G signal value 56, and B signal value 58 expressing the correct answer image 96 are not saturated in the low brightness region L and the medium brightness region M, but are saturated at the upper limit brightness of the high brightness region H. However, this is only an example. The highest signal value among the R signal value 54, G signal value 56, and B signal value 58 expressing the correct answer image 96 may be saturated at a lower brightness side than the middle signal value and the lowest signal value. For example, Fig. 9 As shown, the B signal value 58 of the R signal value 54, the G signal value 56, and the B signal value 58 expressing the image area 95 can be saturated at a lower brightness side than the G signal value 56 and the R signal value 54. Fig. 9In the example shown, the B signal value 58 of the expression image area 95 is saturated at the boundary position between the low brightness area L and the medium brightness area M, and the saturated state is maintained from the boundary position between the low brightness area L and the medium brightness area M to the upper limit brightness of the high brightness area H. Even in the case where the learned model 52 is generated by machine learning of the model 98 using the training data 92 including the correct answer image 96 thus constructed, the learned model 52 can generate and output an image as the AI ​​image 112 in which the color saturation area 16A is suppressed compared to the captured image 14.

[0125] And, as an example, Fig.10 As shown, the B signal value 58 of the R signal value 54, the G signal value 56, and the B signal value 58 expressing the image area 95 may be saturated at a lower brightness side than the G signal value 56 and the R signal value 54, and the G signal value 56 may be saturated at a lower brightness side than the R signal value 54. Fig.10 In the example shown, the B signal value 58 of the expression image region 95 is saturated at the boundary position between the low brightness region L and the medium brightness region M, and the saturated state is maintained from the boundary position between the low brightness region L and the medium brightness region M to the upper limit brightness of the high brightness region H. Furthermore, the G signal value 56 of the expression image region 95 is saturated in the middle of the high brightness region H, and the saturated state is maintained from the middle of the high brightness region H to the upper limit brightness of the high brightness region H. Even in the case where the learned model 52 is generated by machine learning of the model 98 using the training data 92 including the correct answer image 96 thus constructed, the learned model 52 can be caused to generate and output an image as the AI ​​image 112 in which the color saturation region 16A is suppressed compared to the captured image 14.

[0126] In the above embodiment, an example of generating a high-quality image 114 by adding and averaging the camera image 14 and the AI ​​image 112 or mixing the camera image 14 and the AI ​​image 112 at a predetermined mixing ratio is cited, but the present invention is not limited thereto. For example, the high-quality image 114 may be generated by mixing the camera image 14 and the AI ​​image 112 at a mixing ratio determined in reference area units based on the color signal value expressing the camera image 14 and the classification result obtained by performing an object classification process using AI on the camera image 14 or the AI ​​image 112 in reference area units (e.g., pixel units).

[0127] In this case, for example, Fig.11 As shown, the generating unit 42B calculates a mixing ratio 118 determined in pixel units by referring to the color signal value expressing the captured image 14 and the segmented image 116 , and mixes the captured image 14 and the AI ​​image 112 according to the calculated mixing ratio 118 , thereby generating a high-quality image 114 .

[0128] To achieve this purpose, in the camera device 10, a classification model 120 is stored in the storage device 44. The classification model 120 is a learned model that optimizes the model by performing machine learning on a model of an encoder / decoder structure (for example, SegNet, U-Net, or HRNet, etc.) using a plurality of training data including example images showing one or more objects and correct answer images to which labels representing the types of one or more objects shown in the example images are assigned. The camera image 14 is input to the classification model 120. The classification model 120 is a model that classifies a plurality of objects shown in the input camera image 14 in units of pixels. For example, the classification model 120 classifies a plurality of objects shown in the input camera image 14 in a segmented manner (for example, using a semantic segmentation manner, using an instance segmentation manner, or using a panoramic segmentation manner). Fig.11 In the example shown, the classification model 120 is an example of the “AI” involved in the present invention.

[0129] The processor 42 is different from the processor 42 described in the above embodiment in that it also operates as a classification unit 42C. The classification unit 42C performs object classification processing using AI on the camera image 14 in units of pixels (i.e., a process of classifying a plurality of objects reflected in the camera image 14 in units of pixels). That is, the classification unit 42C inputs the camera image 14 to the classification model 120 stored in the storage device 44, causes the classification model 120 to classify a plurality of objects reflected in the camera image 14 in units of pixels, and outputs a segmented image 116 indicating the classification result.

[0130] In order to calculate the mixing ratio 118 by referring to the color signal value expressing the camera image 14 and the segmented image 116, as an example, Fig.12 As shown, the generation unit 42B uses the first mixing ratio table 122 , the second mixing ratio table 124 , and the third mixing ratio table 126 .

[0131] like Fig.12 As shown, the storage device 44 stores a first mixing ratio table 122, a second mixing ratio table 124, and a third mixing ratio table 126. The first mixing ratio table 122 is a table in which a classification result classified in pixel units of the segmented image 116 (i.e., a result of classifying each object shown in the camera image 14 in pixel units) is associated with a first mixing ratio. The first mixing ratio is a ratio of using the AI ​​image 112 when the camera image 14 and the AI ​​image 112 are mixed by the generating unit 42B. The first mixing ratio is determined for each classification result (i.e., an object shown in the camera image 14) within a range of not less than 0 and not more than 1.

[0132] The second mixing ratio table 124 is a table in which a correspondence is established between the maximum signal value and the second mixing ratio. Similar to the first mixing ratio, the second mixing ratio is also the ratio of the AI ​​image 112 used when the camera image 14 and the AI ​​image 112 are mixed by the generating unit 42B. The second mixing ratio is set in a range of greater than 0 and less than 1. The second mixing ratio is from the maximum signal value "0" to the specific signal value "0", and increases monotonically from the specific signal value to the saturated signal value (i.e., the saturated signal value). Here, the specific signal value refers to, for example, a signal value derived in advance through experiments based on actual machines and / or computer simulations as the lower limit value of the maximum signal value with a high probability of generating the color saturated area 16A. By setting the second mixing ratio from the specific signal value to a value exceeding "0" to the saturated signal value, the color saturated area 16A is more susceptible to the influence of the AI ​​image 112 than the image area outside the color saturated area 16A. Thus, the AI ​​image 112 is mixed so as to dominate the captured image 14 in an image region where color signal value saturation is likely to occur, so that an image in which the color saturation region 16A is suppressed can be easily obtained as a high-quality image 114 .

[0133] The third mixing ratio table 126 is a table in which the minimum signal value and the third mixing ratio are correspondingly associated. Similar to the first mixing ratio and the second mixing ratio, the third mixing ratio is also the ratio of using the AI ​​image 112 when the camera image 14 and the AI ​​image 112 are mixed by the generating unit 42B. If the color saturated area 16A is colored, the entire image tends to become unnatural. Therefore, the third mixing ratio is monotonically reduced from the minimum signal value of "0" to the saturated signal value so that the whitish part in the camera image 14 mixed with the AI ​​image 112 is not easily affected by the AI ​​image 112.

[0134] As an example, Fig.13 As shown, the generator 42B calculates the mixing ratio 118 based on the first mixing ratio table 122, the second mixing ratio table 124, and the third mixing ratio table 126 stored in the storage device 44. The mixing ratio 118 is determined in pixel units and is determined based on the first mixing ratio, the second mixing ratio, and the third mixing ratio.

[0135] The generation unit 42B derives the first mixing ratio corresponding to the classification result in the classification model 120 from the first mixing ratio table 122 for each pixel in the segmented image 116. Furthermore, the generation unit 42B derives, from the second mixing ratio table 124, the maximum signal value (in the pixel unit) of the captured image 14 acquired by the acquisition unit 42A from the image sensor 32. Figure 2 In the example shown, it is the second mixing ratio corresponding to the B signal value 58). Furthermore, the generating unit 42B derives the lowest signal value (in pixels) of the captured image 14 acquired by the acquiring unit 42A from the image sensor 32 from the third mixing ratio table 126. Figure 2In the example shown, it is the third mixing ratio corresponding to the R signal value 54).

[0136] The generating unit 42B calculates the mixing ratio 118 by multiplying the first mixing ratio, the second mixing ratio, and the third mixing ratio in units of pixels. The generating unit 42B generates the high-quality image 114 by mixing the captured image 14 and the AI ​​image 112 in units of pixels at the mixing ratio 118 (see FIG. 114 ). Fig.11 ).exist Fig.13 In the example shown, the mixing ratio 118 is an example of the “mixing ratio” involved in the present invention.

[0137] In this way, by multiplying the first mixing ratio, the second mixing ratio, and the third mixing ratio in pixel units to calculate the mixing ratio 118, and mixing the captured image 14 and the AI ​​image 112 in pixel units according to the calculated mixing ratio 118, it is possible to suppress the occurrence of a situation where the high-quality image 114 is excessively affected by the captured image 14 or the AI ​​image 112, or where the high-quality image 114 is excessively unaffected by the captured image 14 or the AI ​​image 112. As a result, the influence of the AI ​​image 112 can be appropriately reflected in the color-saturated region 16A, and the color-saturated region 16A can be suppressed from being excessively colored.

[0138] In addition, Fig.11 and Fig.13 In the example shown, the captured image 14 is input to the classification model 120, and the segmented image 116 for the captured image 14 is output from the classification model 120. However, this is only an example, and the AI ​​image 112 may be input to the classification model 120, and the segmented image 116 for the AI ​​image 112 may be output from the classification model 120. In this case, the same effect can be expected.

[0139] And, in Fig.11 and Fig.13 In the example shown, an example is given in which the classification unit 42C classifies the multiple objects shown in the camera image 14 in a segmented manner, but this is only an example, and the classification unit 42C may classify the multiple objects shown in the camera image 14 in a bounding box manner. In this case, the mixing ratio 118 is calculated in units of bounding boxes, and the camera image 14 and the AI ​​image 112 are mixed in units of bounding boxes according to the mixing ratio 118, thereby generating a high-quality image 114.

[0140] In this way, the generating unit 42B only needs to calculate the mixing ratio 118 by reference area unit (for example, pixel unit or a collection unit of multiple pixels such as a boundary box), and mix the camera image 14 and the AI ​​image 112 by reference area unit according to the calculated mixing ratio 118, thereby generating a high-quality image 114.

[0141] And, in Figure 11 to Figure 13 In the example shown, an example of a form in which the mixing ratio 118 is determined according to the first mixing ratio, the second mixing ratio, and the third mixing ratio is listed, but this is only an example, and the mixing ratio 118 may also be determined according to one or two of the first mixing ratio, the second mixing ratio, and the third mixing ratio. Furthermore, one or two of the first mixing ratio, the second mixing ratio, and the third mixing ratio used to produce the mixing ratio 118 may be determined according to an instruction received by the UI system device 34 or a receiving device connected to the external I / F 36. Furthermore, the high-quality image 114 generated according to the mixing ratio 118 determined in this way may be displayed on the UI system device 34 or a display device connected to the external I / F 36. Furthermore, a plurality of high-quality images 114 generated according to a plurality of mixing ratios 118 may be displayed on the UI system device 34 or a display device connected to the external I / F 36 in a state in which they can be compared. Furthermore, it is also possible to determine the high-quality image 114 to be finally adopted from among the multiple high-quality images 114 based on an instruction received by the UI system device 34 or the receiving device connected to the external I / F 36 when the multiple high-quality images 114 are displayed on the UI system device 34 or the display device connected to the external I / F 36 in a state in which the multiple high-quality images 114 can be compared.

[0142] exist Fig.13 In the example shown, an example of the form in which the first mixing ratio is derived from the first mixing ratio table 122 is listed, but this is only an example. For example, the first mixing ratio may be outputted in pixel units by the classification model 120. In this case, for example, as long as the label included in the training data used in the machine learning for generating the classification model 120 (in other words, the annotation assigned to each pixel) and the information indicating the type of the object also include the first mixing ratio.

[0143] In the above embodiment, an example is given in which the processor 42 of the computer 30 included in the camera device 10 performs high-quality image processing, but the present invention is not limited to this, and a device for performing high-quality image processing may be provided outside the camera device 10. In this case, as an example, Fig.14 As shown, a camera system 136 can be used. The camera system 136 includes a camera device 10 and an external device 138. The external device 138 is, for example, a server. The server is implemented, for example, by cloud computing. Here, cloud computing is illustrated, but this is only an example. For example, the server can also be implemented by a large computer, or by network computing such as fog computing, edge computing, or grid computing. Here, as an example of the external device 138, a server is listed, but this is only an example. At least one personal computer or the like can be used as the external device 138 instead of the server.

[0144] The external device 138 includes a processor 140, a storage device 142, a memory 144, and a communication I / F 146, and the processor 140, the storage device 142, the memory 144, and the communication I / F 146 are connected by a bus 148. The communication I / F 146 is connected to the camera 10 via a network 150. The network 150 is, for example, the Internet. In addition, the network 150 is not limited to the Internet, and may also be a LAN such as a WAN and / or an intranet.

[0145] The storage device 142 stores the image quality enhancement program 50 and the learned model 52. The processor 140 executes the image quality enhancement program 50 on the memory 144. The processor 140 performs the above-mentioned image quality enhancement processing according to the image quality enhancement program 50 executed on the memory 144. When performing the image quality enhancement processing, the processor 140 processes the captured image 14 using the learned model 52 in the manner described in the above-mentioned embodiment. The captured image 14 is transmitted from the camera 10 to the external device 138 via the network 150. The communication I / F 146 of the external device 138 receives the captured image 14. The processor 140 performs the processing using the learned model 52 described in the above-mentioned embodiment and the processing of generating the high-quality image 114 by mixing the captured image 14 and the AI ​​image 112 on the captured image 14 received by the communication I / F 146. The processor 140 transmits the generated high-quality image 114 to the camera 10 via the communication I / F 146. The camera device 10 is connected to the external I / F 36 (reference Figure 2 ) receives the high-definition image 114 sent from the external device 138.

[0146] Fig.14 , an example of a form in which the camera device 10 causes the external device 138 to perform high-quality image processing is shown, but this is only an example. For example, the camera device 10 and the external device 138 may perform high-quality image processing in a distributed manner, or a plurality of devices including the camera device 10 and the external device 138 may perform high-quality image processing in a distributed manner.

[0147] In the above embodiment, an example in which the high-quality image processing program 50 is stored in the storage device 44 is given for explanation, but the present invention is not limited to this. For example, the high-quality image processing program 50 may also be stored in a portable computer-readable non-temporary storage medium such as an SSD or a USB memory. The high-quality image processing program 50 stored in the non-temporary storage medium is installed in the computer 30 of the camera device 10. The processor 42 performs high-quality image processing according to the high-quality image processing program 50.

[0148] Furthermore, the image quality enhancement program 50 may be stored in a storage device such as another computer or server device connected to the imaging device 10 via a network, and downloaded in response to a request from the imaging device 10 to be installed in the computer 30 .

[0149] In addition, the entire image quality enhancement program 50 does not need to be stored in a storage device or the storage device 44 of another computer or server device connected to the imaging device 10, and a portion of the image quality enhancement program 50 may be stored. In addition, here, the image quality enhancement program 50 is described, but the learning program 90 can also be described in the same manner.

[0150] and, Figure 2 Although the imaging device 10 shown in the figure has the computer 30 built therein, the present invention is not limited thereto, and for example, the computer 30 may be provided outside the imaging device 10 .

[0151] In the above embodiment, the computer 30 is exemplified, but the present invention is not limited thereto, and a device including ASIC, FPGA and / or PLD may be used instead of the computer 30. Furthermore, a combination of hardware structure and software structure may be used instead of the computer 30.

[0152] As hardware resources for performing the high-quality image processing and / or learning processing described in the above embodiments, various processors shown below can be used. As a processor, for example, a CPU can be cited, which is a general-purpose processor that performs the function of hardware resources for performing high-quality image processing and / or learning processing by executing software (i.e., a program). In addition, as a processor, for example, a dedicated circuit can be cited, which is a processor with a circuit structure specially designed for performing specific processing, such as FPGA, PLD or ASIC. Regardless of the type of processor, a memory is built in or connected, and regardless of the type of processor, the high-quality image processing and / or learning processing is performed by using the memory.

[0153] The hardware resource for performing high-definition processing and / or learning processing may be composed of one of these various processors, or may be composed of a combination of two or more processors of the same type or different types (for example, a combination of multiple FPGAs or a combination of a CPU and an FPGA). Furthermore, the hardware resource for performing high-definition processing and / or learning processing may also be a single processor.

[0154] As an example of a system composed of one processor, first, there is a form in which a processor is composed of a combination of one or more CPUs and software, and the processor performs the function of a hardware resource for performing high-definition processing and / or learning processing. Secondly, there is a form represented by SoC, etc., in which a processor is used to realize the function of the entire system including multiple hardware resources for performing high-definition processing and / or learning processing by one IC chip. In this way, high-definition processing and / or learning processing are realized by using one or more of the above-mentioned various processors as hardware resources.

[0155] Furthermore, as the hardware structure of these various processors, more specifically, a circuit formed by combining circuit elements such as semiconductor elements can be used. In addition, the above-mentioned high-definition processing and / or learning processing is only an example. Therefore, it is of course possible to delete unnecessary steps, add new steps, or change the processing order within the scope of the main purpose.

[0156] The above-mentioned records and illustrated contents are detailed descriptions of the parts involved in the present invention, and are only examples of the present invention. For example, the description of the above-mentioned structure, function, action and effect is a description of an example of the structure, function, action and effect of the parts involved in the present invention. Therefore, it is of course possible to delete unnecessary parts, add new elements or replace the above-mentioned records and illustrated contents without departing from the gist of the present invention. In addition, in order to avoid trouble and facilitate understanding of the parts involved in the present invention, the description of technical common sense that does not need to be particularly explained when implementing the present invention is omitted in the above-mentioned records and illustrated contents.

[0157] All documents, patent applications, and technical standards described in this specification are incorporated herein by reference to the same extent as if each document, patent application, and technical standard was specifically and individually indicated to be incorporated by reference.

[0158] Regarding the above embodiment, the following additional notes are also disclosed.

[0159] (Note 1)

[0160] A training data used for machine learning of a model, which has:

[0161] The example image is assumed to be a camera image obtained by photographing a subject; and

[0162] Correct answer image,

[0163] The above example image is an image expressed by a plurality of first signal values ​​representing the three primary colors of light.

[0164] The correct answer image is an image expressed by a plurality of second signal values ​​representing the three primary colors.

[0165] The plurality of first signal values ​​are saturated in sequence as the brightness of the object increases.

[0166] At least two of the plurality of second signal values ​​are not saturated in the low brightness region and the medium brightness region among the low brightness region, the medium brightness region and the high brightness region of the object and increase as the brightness of the object increases.

[0167] (Note 2)

[0168] A learned model, which optimizes the above-mentioned model by performing the above-mentioned machine learning on the above-mentioned model using the training data described in Note 1.

[0169] (Note 3)

[0170] A computer-readable non-transitory storage medium storing a program for causing a computer to execute a process comprising the following steps:

[0171] acquiring a first image and the camera image, the first image being output from the learned model by inputting an inference image into the learned model described in Supplementary Note 2; and

[0172] The second image is generated by mixing the first image and the captured image.

[0173] (Note 4)

[0174] A computer-readable non-transitory storage medium storing a program for causing a computer to execute a process comprising the following steps:

[0175] Inputting an image for inference into the learned model described in Supplementary Note 2; and

[0176] The inference result output from the learned model based on the input of the inference image is obtained.

[0177] Explanation of symbols

[0178] 10-camera, 12-image target area, 12A-blue sky, 13-photographer, 14-camera image, 16-image area, 16A-color saturation area, 30-computer, 32-image sensor, 34-UI system device, 36-external I / F, 42, 80, 140-processor, 42A-acquisition unit, 42B-generation unit, 42C-classification unit, 44, 82, 142-storage device, 46, 84, 144-memory, 48-bus, 50-high image quality program, 52-learned model, 54-R signal value, 56-G signal value, 58-B signal value, 79-learning device, 90-learning program, 92-training data, 94-example Question image, 95-image area, 95A-color saturation area, 96-correct answer image, 98-model, 100-comparison image, 102-error, 104-adjustment value, 108-subject, 108A-person, 108B-blue sky, 110-reference image, 112-AI image, 114-high quality image, 116-segmented image, 118-mixing ratio, 120-classification model, 122-1st mixing ratio table, 124-2nd mixing ratio table, 126-3rd mixing ratio table, 136-camera system, 138-external device, 146-communication I / F, 500-reference camera device, H-high brightness area, L-low brightness area, M-medium brightness area.

Claims

1. A training data used for machine learning of a model, having: The example image is assumed to be a camera image obtained by photographing a subject; and Correct answer image, The example image is an image expressed by a plurality of first signal values ​​representing the three primary colors of light. The correct answer image is an image expressed by a plurality of second signal values ​​representing the three primary colors. The plurality of first signal values ​​are saturated in sequence as the brightness of the object increases, At least two of the plurality of second signal values ​​are not saturated in the low brightness region and the medium brightness region among the low brightness region, the medium brightness region, and the high brightness region of the object, and increase as the brightness of the object increases.

2. The training data according to claim 1, wherein: The plurality of first signal values ​​are saturated in an order corresponding to a color of the object as the brightness of the object increases.

3. The training data according to claim 1, wherein: The example question image and the correct answer image are images generated based on a reference image representing the subject. The reference image is an image expressed by a plurality of third signal values ​​representing the three primary colors. The plurality of third signal values ​​increase at a constant rate as the brightness of the object increases.

4. The training data according to claim 3, wherein: A ratio at which the plurality of first signal values ​​increase as the brightness of the object increases is higher than a ratio at which the plurality of third signal values ​​increase as the brightness of the object increases.

5. The training data according to claim 3, wherein: The example image is an image obtained by increasing the gain of the reference image.

6. The training data according to claim 3, wherein: The correct answer image is an image obtained by increasing the gain of the reference image, maintaining the magnitude relationship of the plurality of third signal values ​​throughout the low brightness region, the medium brightness region, and the high brightness region, and saturating at least two of the plurality of third signal values ​​in the high brightness region instead of saturating in the low brightness region and the medium brightness region, The amount of increase of the gain varies depending on the brightness of the object.

7. A learned model, obtained by optimizing the model by performing the machine learning on the model using the training data according to any one of claims 1 to 6.

8. An image processing device comprising a first processor, The first processor performs the following processing: acquiring the first image output from the learned model and the captured image by inputting the inference image into the learned model according to claim 7; and A second image is generated by mixing the first image and the captured image.

9. The image processing apparatus according to claim 8, wherein: The first processor generates the second image by mixing the first image and the captured image in reference area units.

10. The image processing device according to claim 9, wherein: The first processor generates the second image by mixing the first image and the captured image at a mixing ratio determined in units of the reference area. The mixing ratio is a value based on at least one of a first mixing ratio, a second mixing ratio, and a third mixing ratio. The first mixing ratio is determined based on a classification result obtained by performing an object classification process using AI on the captured image or the first image in units of the reference area. The second mixing rate is determined based on the highest signal value among a plurality of fourth signal values ​​representing the three primary colors for the second image in units of the reference area. The third mixing rate is determined according to the lowest signal value among the plurality of fourth signal values.

11. A camera device comprising: a second processor; and Image sensor, The second processor performs the following processing: Inputting an image for inference into the learned model according to claim 7; and acquiring an inference result output from the learned model based on the input of the inference image, The captured image is obtained by capturing the subject with the image sensor.

12. A learning device comprising a third processor, The third processor optimizes the model by causing the model to perform the machine learning using the training data according to any one of claims 1 to 6.

13. A method for creating training data for machine learning of a model, The training data includes a correct answer image and an example image that is assumed to be obtained by photographing a subject. The method for creating the training data includes the following steps: Creating the correct answer image; and Create the example image, The example image is an image expressed by a plurality of first signal values ​​representing the three primary colors of light. The correct answer image is an image expressed by a plurality of second signal values ​​representing the three primary colors. The plurality of first signal values ​​are saturated in sequence as the brightness of the object increases, At least two of the plurality of second signal values ​​are not saturated in the low brightness region and the medium brightness region among the low brightness region, the medium brightness region, and the high brightness region of the object, and increase as the brightness of the object increases.

14. A method for generating a learned model by performing machine learning on the model using training data to generate the learned model, The training data includes a correct answer image and an example image that is determined by assuming that the image is obtained by photographing a subject. The example image is an image expressed by a plurality of first signal values ​​representing the three primary colors of light. The correct answer image is an image expressed by a plurality of second signal values ​​representing the three primary colors. The plurality of first signal values ​​are saturated in sequence as the brightness of the object increases, At least two of the plurality of second signal values ​​are unsaturated in the low brightness region and the medium brightness region among the low brightness region of the object, the medium brightness region of the object, and the high brightness region of the object, and increase as the brightness of the object increases. The method for generating the learned model comprises the following steps: Inputting the example image into the model; Outputting an evaluation object image by the model according to the input of the example image; and The model is optimized according to the comparison result between the evaluation object image and the correct answer image.

15. An image processing method comprising the following steps: acquiring the first image output from the learned model and the captured image by inputting the inference image into the learned model according to claim 7; and A second image is generated by mixing the first image and the captured image.

16. A reasoning method comprising the following steps: Inputting an image for inference into the learned model according to claim 7; and An inference result output from the learned model based on the input of the inference image is acquired.

17. A computer program product for causing a computer to execute a process comprising the steps of: acquiring the first image output from the learned model and the captured image by inputting the inference image into the learned model according to claim 7; and A second image is generated by mixing the first image and the captured image.

18. A computer program product for causing a computer to execute a process comprising the steps of: Inputting an image for inference into the learned model according to claim 7; and An inference result output from the learned model based on the input of the inference image is acquired.

Images