Teacher data, trained model, imaging device, learning device, method for creating teacher data, and method for generating trained model

Abstract

To provide teacher data, a trained model, an imaging device, a learning device, a method for creating teacher data, and a method for generating a trained model that can generate an image with a deeper bit depth than the input image for a trained model.SOLUTION: Training data is used for machine learning of a model and include a ground truth image and an example image having a shallower bit depth than the ground truth image.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present disclosure relates to teacher data, a trained model, an imaging device, a learning device, a method for creating teacher data, and a method for generating a trained model. [Background technology]

[0002] Patent Document 1 discloses an image transmission / reception system. At least one of the transmission devices provided in the image transmission / reception system disclosed in Patent Document 1 includes a machine learning unit that generates model data by machine learning to generate an improved image closer to the original image from a low-bit-rate encoded image obtained by encoding the original image to a low bit-rate. Also, at least one of the transmission devices provided in the image transmission / reception system disclosed in Patent Document 1 includes a transmission unit that transmits the low-bit-rate encoded image and the model data to the outside of the device. In the image transmission / reception system disclosed in Patent Document 1, a receiving device includes an improved image generation unit that generates an improved image of the low-bit-rate encoded image from the received low-bit-rate encoded image and model data.

[0003] Patent Document 2 discloses an image coding device that codes an image. The image coding device described in Patent Document 2 includes a classification means for classifying pixels constituting an image into predetermined classes according to their properties, a mapping coefficient storage means for storing predetermined mapping coefficients for each class, a calculation means for calculating correction data obtained by correcting a pixel of interest in an image by performing a predetermined calculation using the pixel of interest and the mapping coefficient corresponding to the class of the pixel of interest, and a restriction means for restricting the level of the correction data to generate coded data obtained by coding the image.

[0004] In the image coding device described in Patent Document 2, the mapping coefficient has the following characteristics. That is, the mapping coefficient is generated by performing learning using image data for learning. Alternatively, the mapping coefficient is obtained by performing learning so that the prediction error of the prediction result of predicting an original image from the encoded data with respect to the original image is minimized. Alternatively, the mapping coefficient is obtained by performing learning so that the prediction error of the prediction result of predicting an original image from the encoded data with respect to the original image is equal to or less than a predetermined value. Alternatively, the mapping coefficient is a predetermined coefficient of an optimal value obtained by classifying pixels constituting the learning image into one of classes according to their properties, performing a predetermined calculation using a target learning pixel of interest in the learning image and a predetermined coefficient corresponding to the class of the target learning pixel, calculating learning correction data that corrects the target learning pixel, obtaining learning restriction data that restricts the level of the learning correction data, predicting a predicted value of the learning image based on the learning restriction data, calculating a prediction error of the predicted value of the learning image with respect to the learning image, and changing the predetermined coefficient based on the prediction error, and repeating the above steps until the predetermined coefficient becomes an optimal value. Alternatively, the mapping coefficient is obtained by compressing a learning image by reducing the number of pixels thereof, limiting the level of learning compressed data obtained by compressing the learning image, correcting the level-limited learning compressed data, outputting learning correction data, predicting a learning image based on the learning correction data and outputting the predicted value, calculating a prediction error of the predicted value of the learning image for the learning image, and determining the appropriateness of the learning correction data based on the prediction error, all of which are repeated until the learning correction data becomes appropriate, and using the appropriate learning correction data and the learning image. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2020-109897 A [Patent Document 2] JP 2006-109499 A Summary of the Invention

[0006] One embodiment of the present disclosure provides teacher data, a trained model, an imaging device, a learning device, a method for creating teacher data, and a method for generating a trained model that can generate an image having a deeper bit depth than an input image for a trained model. [Means for solving the problem]

[0007] A first aspect of the present disclosure is training data used in machine learning of a model, the training data including a master image and an example image having a shallower bit depth than the master image.

[0008] A second aspect of the present disclosure is training data according to the first aspect, in which the correct image is an image obtained by synthesizing a plurality of single images.

[0009] A third aspect of the present disclosure is training data according to the second aspect, in which the example image is a representative image of a plurality of single images.

[0010] A fourth aspect of the present disclosure is training data according to the second aspect, in which the example image is an image having a shallower bit depth than the multiple single images.

[0011] A fifth aspect according to the present disclosure is training data according to any one of the first to fourth aspects, in which the single image is a bit-shifted image.

[0012] A sixth aspect of the present disclosure is training data relating to the first aspect, in which the correct image is an image obtained by combining a processing target area and a corresponding area in the bit-shifted image, the corresponding area having an arrangement pattern corresponding to the processing target area in the image, or an image obtained by combining a processing target pixel and an adjacent pixel in the bit-shifted image, the adjacent pixel being regularly adjacent to the processing target pixel in the bit-shifted image.

[0013] A seventh aspect of the present disclosure is training data according to the sixth aspect, in which the example image is an image before bit shifting or an image generated based on the image before bit shifting.

[0014] An eighth aspect of the present disclosure is training data relating to any one of the first to seventh aspects, wherein the correct image and the example image are images obtained by imaging using a first imaging device.

[0015] A ninth aspect of the present disclosure is training data according to any one of the first to eighth aspects, in which the correct image and the example image are images in RAW format.

[0016] A tenth aspect of the present disclosure is training data relating to any one of the first to eighth aspects, in which a model inputs and outputs images in RAW format, the correct image is in Log format, and the RAW format image output from the model is converted into an image in Log format and compared with the correct image to optimize the model.

[0017] An eleventh aspect of the present disclosure is training data relating to any one of the first to eighth aspects, in which the model outputs an image in a second file format other than RAW format in response to an input of an image in a first file format other than RAW format, the format of the example image is the first file format, and the format of the correct image is the second file format.

[0018] A twelfth aspect of the present disclosure is training data relating to any one of the first to eleventh aspects, in which the example image is divided into a plurality of first image regions, and the correct image is divided into a plurality of second image regions corresponding to the plurality of first image regions.

[0019] A thirteenth aspect of the present disclosure is training data according to the twelfth aspect, in which a label capable of identifying image quality is assigned to the second image regions.

[0020] A fourteenth aspect of the present disclosure is training data relating to the twelfth or thirteenth aspects, in which the example image is an image that is assumed to be obtained by capturing an image by a second imaging device, and the image quality is determined by the characteristics of the second imaging device.

[0021] A fifteenth aspect according to the present disclosure is the training data according to the fourteenth aspect, in which the second imaging device has an optical system, and the characteristics include an image height related to the optical system.

[0022] A sixteenth aspect of the present disclosure is a trained model in which the model is optimized by performing machine learning on the model using training data relating to any one of the first to fifteenth aspects.

[0023] A 17th aspect of the present disclosure is an imaging device comprising a first processor and an image sensor, wherein the first processor inputs an image obtained by capturing an image with the image sensor into a trained model according to the 16th aspect, and obtains an inference result output from the trained model in response to the input of the captured image.

[0024] An 18th aspect of the present disclosure is a learning device including a second processor, the second processor optimizing a model by performing machine learning on the model using training data relating to any one of the first to fifteenth aspects.

[0025] A 19th aspect of the present disclosure is a method for creating training data for use in machine learning of a model, the training data including a master image and an example image, and including creating a master image and creating an example image having a shallower bit depth than the master image.

[0026] A twentieth aspect of the present disclosure is a method for generating a trained model, which generates a trained model by performing machine learning on a model using training data including a master image and an example image, wherein the example image has a shallower bit depth than the master image, the method including inputting the example image into the model, the model outputting an image to be evaluated in response to the input of the example image, and optimizing the model based on a comparison result between the image to be evaluated and the master image. [Brief description of the drawings]

[0027] [Figure 1] 1 is a schematic diagram showing an example of the overall configuration of an imaging device; [Diagram 2] FIG. 2 is a schematic diagram showing an example of a hardware configuration of an optical system and an electrical system of the imaging apparatus. [Diagram 3] FIG. 2 is a schematic diagram showing an example of the contents of a RAW image obtained by capturing an image using an imaging device. [Figure 4] FIG. 2 is a schematic diagram showing an example of the configuration of a learning device. [Diagram 5] FIG. 13 is a conceptual diagram showing an example of a method for creating teacher data. [Figure 6] FIG. 2 is a block diagram showing an example of the functions of an image processing engine. [Figure 7] 11 is a flowchart showing an example of the flow of a learning process performed by a learning device. [Figure 8] 10 is a flowchart showing an example of the flow of image quality improvement processing performed by the imaging device. [Figure 9] FIG. 13 is a conceptual diagram showing a first modified example of a method for creating teacher data. [Figure 10] FIG. 13 is a conceptual diagram showing a second modified example of the method for creating teacher data. [Figure 11] FIG. 13 is a conceptual diagram showing a third modified example of the method for creating teacher data. [Figure 12] FIG. 13 is a conceptual diagram showing a fourth modified example of the method for creating teacher data. [Figure 13] FIG. 13 is a conceptual diagram showing a fifth modified example of the method for creating teacher data. [Figure 14] FIG. 13 is a conceptual diagram showing a first modified example of teacher data. [Figure 15] FIG. 11 is a conceptual diagram showing a first modified example of image quality improvement processing. [Figure 16] FIG. 13 is a conceptual diagram showing a second modified example of the teacher data. [Figure 17] FIG. 13 is a conceptual diagram showing a third modified example of the teacher data. [Figure 18] FIG. 11 is a conceptual diagram showing a second modified example of the image quality improvement process. [Figure 19] FIG. 13 is a conceptual diagram showing a fourth modified example of the teacher data. [Figure 20] FIG. 13 is a conceptual diagram showing a third modified example of the image quality improvement process. [Figure 21] FIG. 13 is a conceptual diagram showing a fifth modified example of teacher data. [Figure 22] FIG. 13 is a conceptual diagram showing a fourth modified example of the image quality improvement process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] Below, examples of embodiments of the teacher data, trained model, imaging device, learning device, teacher data creation method, and trained model generation method related to the present disclosure are described with reference to the attached drawings.

[0029] First, the terms used in the following description will be explained.

[0030] CPU is an abbreviation for "Central Processing Unit". GPU is an abbreviation for "Graphics Processing Unit". GPGPU is an abbreviation for "General-purpose computing on graphics processing units". APU is an abbreviation for "Accelerated Processing Unit". TPU is an abbreviation for "Tensor processing unit". NVM is an abbreviation for "Non-volatile memory". RAM is an abbreviation for "Random Access Memory". IC is an abbreviation for "Integrated Circuit". ASIC is an abbreviation for "Application Specific Integrated Circuit". PLD is an abbreviation for "Programmable Logic Device". FPGA is an abbreviation for "Field-Programmable Gate Array". SoC is an abbreviation for "System-on-a-chip". SSD is an abbreviation for "Solid State Drive". USB is an abbreviation for "Universal Serial Bus". HDD is an abbreviation for "Hard Disk Drive". EEPROM is an abbreviation for "Electrically Erasable and Programmable Read Only Memory". EL is an abbreviation for "Electro-Luminescence". I / F is an abbreviation for "Interface". UI is an abbreviation for "User Interface". fps is an abbreviation for "frame per second". MF is an abbreviation for "Manual Focus". AF is an abbreviation for "Auto Focus". CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor". CCD is an abbreviation for "Charge Coupled Device". AI is an abbreviation for "Artificial Intelligence". A / D is an abbreviation for "Analog / Digital".FIR is an abbreviation for "Finite Impulse Response". IIR is an abbreviation for "Infinite Impulse Response". JPEG is an abbreviation for "Joint Photographic Experts Group". TIFF is an abbreviation for "Tagged Image File Format". JPEG XR is an abbreviation for "Joint Photographic Experts Group Extended Range". MPEG is an abbreviation for "Moving Picture Expert Group". AVI is an abbreviation for "Audio Video Interleaved".

[0031] In the following description, a coded processor (hereinafter simply referred to as a "processor") may be one physical or virtual arithmetic device, or a combination of multiple physical or virtual arithmetic devices. Also, a processor may be one type of arithmetic device, or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU, a GPU, a GPGPU, an APU, and a TPU.

[0032] In the following description, a signed memory is a memory, such as a RAM, in which information is temporarily stored and which is used as a working memory by a processor.

[0033] In the following description, a storage with a symbol is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory, magnetic disks, and magnetic tapes. Another example of storage is cloud storage.

[0034] In the following embodiments, the external I / F with a symbol controls the transmission and reception of various information between multiple devices connected to each other. An example of the external I / F is a USB interface. A communication I / F including a communication processor and an antenna may be applied to the external I / F. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G, Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0035] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0036] Fig. 1 is a schematic diagram showing an example of the overall configuration of an imaging device. As shown in Fig. 1, imaging device 10 is a device that captures an image of a subject, and includes an image processing engine 12, an imaging device body 16, and an interchangeable lens 18. Here, imaging device 10 is an example of the "imaging device" and "second imaging device" according to the present disclosure.

[0037] The image processing engine 12 is built into the imaging device body 16, and controls the entire imaging device 10. The interchangeable lens 18 is replaceably attached to the imaging device body 16. The interchangeable lens 18 is provided with a focus ring 18A. The focus ring 18A is operated by a user of the imaging device 10 (hereinafter simply referred to as a "user") when the user manually adjusts the focus of the imaging device 10 on a subject.

[0038] 1, a digital camera with interchangeable lenses is shown as an example of the imaging device 10. However, this is merely one example, and the imaging device 10 may be a digital camera with fixed lenses, or may be a digital camera built into various electronic devices such as a smart device, a wearable terminal, an endoscope device, a cell observation device, an ophthalmic observation device, or a surgical microscope.

[0039] The imaging device body 16 is provided with an image sensor 20. The image sensor 20 is an example of an "image sensor" according to the present disclosure. The image sensor 20 is a CMOS image sensor. The image sensor 20 captures an image of an imaging range including at least one subject. When the interchangeable lens 18 is attached to the imaging device body 16, subject light indicative of the subject passes through the interchangeable lens 18 and is focused on the image sensor 20, and image data indicative of the image of the subject is generated by the image sensor 20.

[0040] In this embodiment, a CMOS image sensor is exemplified as the image sensor 20, but the present disclosure is not limited to this, and the present disclosure also applies even if the image sensor 20 is another type of image sensor, such as a CCD image sensor.

[0041] A release button 22 and a dial 24 are provided on the top surface of the imaging device body 16. The dial 24 is operated when setting an operation mode of the imaging system and an operation mode of the playback system, and by operating the dial 24, an imaging mode, a playback mode, and a setting mode are selectively set as an operation mode in the imaging device 10. The imaging mode is an operation mode that causes the imaging device 10 to capture an image. The playback mode is an operation mode that plays back an image (e.g., a still image and / or a moving image) obtained by capturing an image for recording in the imaging mode. The setting mode is an operation mode that is set for the imaging device 10 when, for example, setting various setting values ​​used in control related to imaging.

[0042] The release button 22 functions as an imaging preparation instruction section and an imaging instruction section, and is capable of detecting two-stage pressing operations: an imaging preparation instruction state and an imaging instruction state. The imaging preparation instruction state refers to a state in which the button is pressed, for example, from a standby position to an intermediate position (half-pressed position), and the imaging instruction state refers to a state in which the button is pressed beyond the intermediate position to a final pressed position (fully pressed position).

[0043] In the following, the “state in which the button is pressed from the standby position to the half-pressed position” will be referred to as the “half-pressed state,” and the “state in which the button is pressed from the standby position to the fully-pressed position” will be referred to as the “fully-pressed state.” Depending on the configuration of imaging device 10, the imaging preparation instruction state may be a state in which the user's finger is in contact with release button 22, and the imaging instruction state may be a state in which the operating user's finger has moved from a state in which it is in contact with release button 22 to a state in which it is released.

[0044] The imaging device body 16 is provided on the rear surface with instruction keys 26 and a touch panel display 32. The touch panel display 32 includes a display 28 and a touch panel 30 (see also FIG. 2). An example of the display 28 is an EL display (e.g., an organic EL display or an inorganic EL display). The display 28 may be another type of display, such as a liquid crystal display, instead of an EL display.

[0045] The display 28 displays images and / or text information, etc. When the imaging device 10 is in imaging mode, the display 28 is used for capturing images for live view images, i.e., for displaying live view images obtained by performing continuous imaging. Here, the term "live view image" refers to a moving image for display based on image data obtained by capturing images using the image sensor 20. The imaging performed to obtain the live view image (hereinafter also referred to as "capturing images for live view images") is performed according to a frame rate of, for example, 60 fps. 60 fps is merely an example, and the frame rate may be less than 60 fps or may be more than 60 fps.

[0046] The display 28 is also used to display a still image obtained by capturing an image for a still image when an instruction to capture an image for a still image is given to the imaging device 10 via the release button 22. The display 28 is also used to display a playback image, etc. when the imaging device 10 is in a playback mode. Furthermore, the display 28 is also used to display a menu screen on which various menus can be selected, and a setting screen for setting various setting values, etc. used in control related to imaging, when the imaging device 10 is in a setting mode.

[0047] The touch panel 30 is a transmissive touch panel, and is overlaid on the surface of the display area of ​​the display 28. The touch panel 30 receives instructions from the user by detecting contact with a pointing object such as a finger or a stylus pen. For ease of explanation, the above-mentioned "full press state" hereinafter also includes a state in which the user turns on a soft key for starting imaging via the touch panel 30.

[0048] In this embodiment, an out-cell type touch panel display in which the touch panel 30 is overlaid on the surface of the display area of ​​the display 28 is given as an example of the touch panel display 32, but this is merely one example. For example, an on-cell type or in-cell type touch panel display can also be applied as the touch panel display 32.

[0049] The instruction keys 26 accept various instructions. Here, "various instructions" refers to, for example, an instruction to display a menu screen, an instruction to select one or more menus, an instruction to confirm the selected content, an instruction to erase the selected content, various instructions such as zooming in, zooming out, and frame-by-frame advance. These instructions may also be given by the touch panel 30.

[0050] FIG. 2 is a schematic diagram showing an example of a hardware configuration of an optical system and an electrical system of an imaging device. As shown in FIG. 2, the image sensor 20 includes a photoelectric conversion element 72. The photoelectric conversion element 72 has a light receiving surface 72A. The photoelectric conversion element 72 is disposed in the imaging device body 16 so that the center of the light receiving surface 72A coincides with the optical axis OA (see also FIG. 1). The photoelectric conversion element 72 has a plurality of photosensitive pixels arranged in a matrix, and the light receiving surface 72A is formed by the plurality of photosensitive pixels. Each photosensitive pixel has a microlens (not shown). Each photosensitive pixel is a physical pixel having a photodiode (not shown), which photoelectrically converts received light and outputs an electrical signal according to the amount of received light.

[0051] In addition, color filters (not shown) of the three primary colors of light, i.e., red (hereinafter also referred to as "R"), green (hereinafter also referred to as "G"), or blue (hereinafter also referred to as "B"), are arranged in a predetermined pattern array in the multiple photosensitive pixels. In this embodiment, a Bayer array is used as an example of the predetermined pattern array. However, the Bayer array is merely an example, and the present disclosure is valid even if the predetermined pattern array is another type of pattern array such as a G-stripe R / G complete checkerboard, an X-Trans (registered trademark) array, or a honeycomb array.

[0052] For convenience of explanation, the photosensitive pixel having a microlens and an R color filter is referred to as an R pixel, the photosensitive pixel having a microlens and a G color filter is referred to as a G pixel, and the photosensitive pixel having a microlens and a B color filter is referred to as a B pixel. For convenience of explanation, the electric signal output from the R pixel of the photosensitive pixel is referred to as an "R signal", the electric signal output from the G pixel of the photosensitive pixel is referred to as a "G signal", and the electric signal output from the B pixel of the photosensitive pixel is referred to as a "B signal". For convenience of explanation, the R signal, the G signal, and the B signal are also referred to as "RGB color signals". For convenience of explanation, the R pixel, the G pixel, and the B pixel constituting the RAW image 75A generated based on the RGB color signals are also referred to as "R pixel", "G pixel", and "B pixel". Here, R pixels, G pixels, and B pixels are shown as examples, but this is merely one example. If colors other than R, G, and B (i.e., colors other than the primary colors) are regularly arranged along with R, G, and B in the color filter, the RAW image will also include pixels of colors other than R, G, and B pixels, and this disclosure will still apply in such cases.

[0053] The interchangeable lens 18 includes an imaging lens 40. The imaging lens 40 has an objective lens 40A, a focus lens 40B, a zoom lens 40C, and an aperture 40D. The objective lens 40A, the focus lens 40B, the zoom lens 40C, and the aperture 40D are arranged in this order along the optical axis OA from the subject side (object side) to the imaging device body 16 side (image side).

[0054] The interchangeable lens 18 also includes a control device 36, a first actuator 37, a second actuator 38, and a third actuator 39. The control device 36 controls the entire interchangeable lens 18 in accordance with instructions from the imaging device body 16. The control device 36 is, for example, a device having a computer including a processor, storage, memory, and the like. The storage of the control device 36 is, for example, an EEPROM. Various programs and various parameters are stored in the storage of the control device 36. The memory of the control device 36 is, for example, a RAM, which temporarily stores various information and is used as a work memory. In the control device 36, the processor reads out necessary programs from the storage and executes the various programs read out on the memory to control the entire imaging lens 40.

[0055] Although a device having a computer is given here as an example of the control device 36, this is merely an example and devices including an ASIC, an FPGA, and / or a PLD may also be applied. Also, a device realized by combining a hardware configuration and a software configuration, for example, may be used as the control device 36.

[0056] The first actuator 37 includes a focusing slide mechanism (not shown) and a focusing motor (not shown). The focusing slide mechanism has a focus lens 40B attached thereto so as to be slidable along the optical axis OA. The focusing motor is also connected to the focusing slide mechanism, and the focusing slide mechanism is actuated by receiving power from the focusing motor to move the focus lens 40B along the optical axis OA.

[0057] The second actuator 38 includes a zoom slide mechanism (not shown) and a zoom motor (not shown). The zoom lens 40C is attached to the zoom slide mechanism so as to be slidable along the optical axis OA. The zoom motor is also connected to the zoom slide mechanism, and the zoom slide mechanism operates by receiving power from the zoom motor to move the zoom lens 40C along the optical axis OA.

[0058] The third actuator 39 includes a power transmission mechanism (not shown) and a diaphragm motor (not shown). The diaphragm 40D has an opening 40D1, and the size of the opening 40D1 is variable. The opening 40D1 is formed by, for example, a plurality of diaphragm blades 40D2. The plurality of diaphragm blades 40D2 are connected to the power transmission mechanism. In addition, a diaphragm motor is connected to the power transmission mechanism, and the power transmission mechanism transmits the power of the diaphragm motor to the plurality of diaphragm blades 40D2. The plurality of diaphragm blades 40D2 change the size of the opening 40D1 by operating upon receiving the power transmitted from the power transmission mechanism. The diaphragm 40D adjusts the exposure by changing the size of the opening 40D1.

[0059] The focus motor, zoom motor, and aperture motor are connected to a control device 36, and the control device 36 controls the driving of the focus motor, zoom motor, and aperture motor. In this embodiment, a stepping motor is used as an example of the focus motor, zoom motor, and aperture motor. Therefore, the focus motor, zoom motor, and aperture motor operate in synchronization with a pulse signal in response to a command from the control device 36. Here, an example is shown in which the focus motor, zoom motor, and aperture motor are provided in the interchangeable lens 18, but this is merely an example, and at least one of the focus motor, zoom motor, and aperture motor may be provided in the imaging device body 16. The components and / or operation method of the interchangeable lens 18 can be changed as necessary.

[0060] In the imaging device 10, in the imaging mode, the MF mode and the AF mode are selectively set according to an instruction given to the imaging device body 16. The MF mode is an operation mode in which the focus is adjusted manually. In the MF mode, for example, when the focus ring 18A or the like is operated by the user, the focus lens 40B moves along the optical axis OA by an amount of movement corresponding to the amount of operation of the focus ring 18A or the like, thereby adjusting the focus.

[0061] In the AF mode, the imaging device body 16 calculates the in-focus position according to the subject distance, and adjusts the focus by moving the focus lens 40B toward the calculated in-focus position. Here, the in-focus position refers to the position on the optical axis OA of the focus lens 40B when the subject is in focus.

[0062] The imaging device main body 16 includes an image sensor 20, an image processing engine 12, a system controller 44, an image memory 46, a UI device 48, an external I / F 50, a communication I / F 52, a photoelectric conversion element driver 54, and an input / output interface 70. The image sensor 20 also includes a photoelectric conversion element 72 and an A / D converter 74.

[0063] The input / output interface 70 is connected to the image processing engine 12, the image memory 46, the UI device 48, the external I / F 50, the photoelectric conversion element driver 54, the mechanical shutter driver 56, and an A / D converter 74. In addition, the input / output interface 70 is also connected to the control device 36 of the interchangeable lens 18.

[0064] The system controller 44 includes a processor (not shown), a storage (not shown), and a memory (not shown). In the system controller 44, the storage is a computer-readable non-transient storage medium, and various parameters and various programs are stored in the storage. The storage of the system controller 44 is, for example, an EEPROM. However, this is merely an example, and instead of the EEPROM, or together with the EEPROM, a HDD and / or an SSD may be applied as the storage of the system controller 44. In addition, the memory of the system controller 44 temporarily stores various information and is used as a work memory. In the system controller 44, the processor reads out necessary programs from the storage and executes the various programs read out on the memory to control the entire imaging device 10. That is, in the example shown in FIG. 2, the image processing engine 12, the image memory 46, the UI device 48, the external I / F 50, the communication I / F 52, the photoelectric conversion element driver 54, and the control device 36 are controlled by the system controller 44.

[0065] The image processing engine 12 operates under the control of the system controller 44. The image processing engine 12 includes a processor 62, a storage 64, and a memory 66. Here, the processor 62 is an example of a "first processor" according to the present disclosure.

[0066] The processor 62, the storage 64, and the memory 66 are connected via a bus 68, which is connected to an input / output interface 70. In the example shown in Fig. 2, for convenience of illustration, one bus is shown as the bus 68, but multiple buses may be used. The bus 68 may be a serial bus, or may be a parallel bus including a data bus, an address bus, a control bus, and the like.

[0067] The storage 64 is a computer-readable non-transitory storage medium, and stores various parameters and programs different from the various parameters and programs stored in the storage of the system controller 44. The storage 64 is, for example, an EEPROM. However, this is merely one example, and instead of the EEPROM, or together with the EEPROM, an HDD and / or an SSD may be used as the storage 64. The memory 66 is, for example, a RAM, which temporarily stores various information and is used as a work memory.

[0068] The processor 62 reads out a necessary program from the storage 64, and executes the read program in the memory 66. The processor 62 performs image processing in accordance with the program executed on the memory 66.

[0069] The photoelectric conversion element 72 is connected to the photoelectric conversion element driver 54. The photoelectric conversion element driver 54 supplies an imaging timing signal that specifies the timing of imaging performed by the photoelectric conversion element 72 to the photoelectric conversion element 72 in accordance with an instruction from the processor 62. The photoelectric conversion element 72 performs resetting, exposure, and output of an electrical signal in accordance with the imaging timing signal supplied from the photoelectric conversion element driver 54. Examples of the imaging timing signal include a vertical synchronization signal and a horizontal synchronization signal.

[0070] When the interchangeable lens 18 is attached to the imaging device body 16, subject light incident on the imaging lens 40 is imaged on the light receiving surface 72A by the imaging lens 40. Under the control of the photoelectric conversion element driver 54, the photoelectric conversion element 72 photoelectrically converts the subject light received by the light receiving surface 72A, and outputs an electrical signal according to the amount of subject light to the A / D converter 74 as analog image data indicating the subject light. Specifically, the A / D converter 74 reads out the analog image data from the photoelectric conversion element 72 in units of frames and for each horizontal line using an exposure sequential readout method.

[0071] The A / D converter 74 digitizes the analog image data to generate a RAW image 75A. The RAW image 75A is an image in which R pixels, G pixels, and B pixels are arranged in a mosaic pattern. The RAW image 75A is an example of a "captured image" according to the present disclosure.

[0072] In this embodiment, as an example, the bit depth of the R, B, and G pixels included in the RAW image 75A, i.e., the number of bits (in other words, the bit length, the number of color bits, or the color depth) which is a value expressing the gradation of each of the R, B, and G pixels included in the RAW image 75A, is 14 bits. Note that 14 bits is merely an example, and the bit depth of the R, B, and G pixels included in the RAW image 75A may be more than 14 bits or less than 14 bits.

[0073] In this embodiment, as an example, the processor 62 of the image processing engine 12 acquires a RAW image 75A from the A / D converter 74, and performs image processing, including development, on the acquired RAW image 75A to generate an image file 75B. Here, development refers to a process of compressing a luminance and color difference signal according to a default compression method. Examples of the default compression method (i.e., the format of the image file) include JPEG, TIFF, JPEG XR, MPEG, and AVI. The image processing also includes image quality adjustment of the RAW image 75A. The image quality adjustment of the RAW image 75A is realized by focal length adjustment, aperture value adjustment, lens characteristic adjustment, thinning characteristic adjustment between pixels P, a gradation correction function (for example, a process of correcting the gradation of an RGB image according to a gamma value), a gain correction function, and a noise reduction function.

[0074] Other examples of image quality adjustments to the RAW image 75A include color space conversion processing (i.e., processing for converting the color space of an RGB image that has been gamma corrected from the RGB color space to the YCbCr color space), luminance filtering processing (i.e., processing for filtering the luminance signal (the so-called Y signal) using a luminance filter (not shown)), color difference processing (i.e., processing for filtering the Cb signal and the Cr signal to reduce high-frequency noise), and / or resizing processing (i.e., processing for adjusting the luminance and color difference signals so that the size of the image indicated by the luminance and color difference signals matches the size specified by a user, etc.).

[0075] An example of the image file 75B is an MPEG file. However, the MPEG file is merely an example, and the image file 75B may be other types of image files such as a JPEG file, a JPEG XR file, a TIFF file, or an AVI file. The image file 75B is stored in the image memory 46 by the processor 62.

[0076] The UI-based device 48 includes a display 28, and the processor 62 causes the display 28 to display various pieces of information. The UI-based device 48 also includes a reception device 76. The reception device 76 includes a touch panel 30 and a hard key unit 78. The hard key unit 78 is a plurality of hard keys including the instruction keys 26 (see FIG. 1 ). The processor 62 operates in accordance with various instructions received by the touch panel 30. Note that, although the hard key unit 78 is included in the UI-based device 48 here, the present disclosure is not limited to this, and for example, the hard key unit 78 may be connected to the external I / F 50.

[0077] The external I / F 50 controls the exchange of various information with devices (hereinafter also referred to as "external devices") that exist outside the imaging device 10. An example of the external I / F 50 is a USB interface. To the USB interface, external devices (not shown), such as a smart device, a personal computer, a server, a USB memory, a memory card, and / or a printer, are directly or indirectly connected.

[0078] The communication I / F 52 is connected to a network (not shown). The communication I / F 52 controls the exchange of information between a communication device (not shown) such as a server on the network and the system controller 44. For example, the communication I / F 52 transmits information in response to a request from the system controller 44 to the communication device via the network. The communication I / F 52 also receives information transmitted from the communication device and outputs the received information to the system controller 44 via the input / output interface 70.

[0079] Incidentally, in the editing of moving images used in movies or television, the dynamic range is widely known as one of the very important specifications. Examples of a method for recording moving images with a wide dynamic range include the RAW method (e.g., a method for recording 14-bit RGB images) and the Log method (e.g., a method for recording 10-bit YC images). In the RAW method, a gamma correction process is performed later by software to raise the shadow parts where the gamma rise (i.e., the side where the input value is small) is steep, and in the Log method, a process for converting from RGB gradation to Log gradation is performed during recording. In either case, if sufficient bit depth is not ensured, a degradation in image quality will occur, such as gradation jumps called banding, or the disappearance of fine structures in the subject captured in the RAW image 75A (in other words, the disappearance of details), as shown in FIG. 3 as an example. In addition, there is currently a demand to increase the frame rate so as to prevent image retention and / or rolling distortion due to rolling shutter, and since there is a trade-off between increasing the frame rate and increasing the bit depth, there is a demand for high-performance image processing technology that achieves both an increase in frame rate and an increase in bit depth.

[0080] Therefore, in this embodiment, the image processing engine 12 applies AI to the RAW image 75A to perform processing to improve the image quality of the RAW image 75A (for example, processing including processing to deepen the bit depth for each pixel). Below, a method of generating AI to be applied to the RAW image 75A will be described with reference to Figs. 4 to 6.

[0081] Fig. 4 is a conceptual diagram showing an example of the configuration of a learning device 79. As shown in Fig. 4, the learning device 79 includes a processor 80, a storage 82, and a memory 84. The hardware configuration of the processor 80, storage 82, memory 84, etc. in the learning device 79 is basically the same as the hardware configuration of the above-mentioned processor 62, storage 64, memory 66, etc., so a description thereof will be omitted here. In the example shown in Fig. 4, the learning device 79 is an example of a "learning device" according to the present disclosure.

[0082] The storage 82 stores a learning program 90. The processor 80 is an example of a "second processor" according to the present disclosure. The processor 80 reads out the learning program 90 from the storage 82 and executes the read learning program 90 on the memory 84. The processor 80 performs a learning process according to the learning program 90 executed on the memory 84. The learning process is a process of generating a trained model 106 from a model 98. The trained model 106 is generated by the processor 80 executing machine learning on the model 98. That is, the trained model 106 is generated by optimizing the model 98 by machine learning. For example, the model 98 is a neural network having hundreds of millions to trillions of intermediate layers. An example of the model 98 is a model for a generation AI that generates and outputs an image with higher image quality than an input image (for example, at least an image with a deeper bit depth than the input image).

[0083] The storage 82 stores a plurality of (e.g., tens of thousands to hundreds of billions) pieces of teacher data 92. The teacher data 92 is used for machine learning of the model 98. That is, in the learning device 79, the processor 80 acquires the plurality of teacher data 92 from the storage 82, and performs machine learning on the model 98 using the acquired plurality of teacher data 92.

[0084] The teacher data 92 is labeled data. The labeled data is, for example, data in which an example image 94 (in other words, example data) and a correct answer image 96 (in other words, correct answer data) are associated with each other. Here, the teacher data 92 is an example of "teacher data" according to the present disclosure, the example image 94 is an example of "example image" according to the present disclosure, and the correct answer image 96 is an example of "correct answer image" according to the present disclosure.

[0085] The example image 94 is an image in RAW format, and has a shallower bit depth than the correct image 96. The example image 94 is an image that is modeled after the RAW image 75A. An image that is modeled after the RAW image 75A refers to an image that has the same image quality as the RAW image 75A, including the bit depth, for example. Here, the bit depth refers to a value that expresses gradation (i.e., a value that indicates the fineness of the gradation expression of color). Terms that are used in the same sense as bit depth include bit length, color bit count, bit number, or color depth.

[0086] In this embodiment, an image obtained by actually capturing an image of a sample subject (for example, a subject captured in the correct answer image 96 shown in FIG. 4) is used as the example image 94. However, this is merely an example, and an image that is virtually generated as an image that assumes the RAW image 75A may also be used. An example of a virtually generated image is an image generated by a generation AI or the like.

[0087] The correct image 96 is a RAW format image, like the example image 94. The correct image 96 is an image with higher image quality than the example image 94. An example of an image with higher image quality is an image with a deeper bit depth than the example image 94. In this embodiment, the correct image 96 has a bit depth of 14 bits, and the example image 94 has a bit depth of 12 bits. However, these bit depths are merely examples, and the present disclosure applies regardless of the bit depth as long as the correct image 96 has a deeper bit depth than the example image 94.

[0088] The processor 80 acquires the teacher data 92 one by one from the storage 82. The processor 80 inputs the example image 94 from the teacher data 92 acquired from the storage 82 to the model 98. When the example image 94 is input, the model 98 generates a comparison image 100 which is an image with a deeper bit depth than the example image 94 (here, as an example, an image with a 14-bit depth). The comparison image 100 is an image used for comparison with the correct answer image 96 which is associated with the example image 94 input to the model 98. In this embodiment, the comparison image 100 is an example of an "evaluation target image" according to the present disclosure.

[0089] The processor 80 calculates an error 102 between a ground truth image 96 associated with an example image 94 input to the model 98 and a comparison image 100. The error 102 is an example of a "comparison result" according to the present disclosure. The processor 80 calculates a number of adjustment values ​​104 that minimize the error 102. The processor 80 then adjusts a number of optimization variables in the model 98 using the multiple adjustment values ​​104. Here, the multiple optimization variables refer to, for example, a number of connection weights and a number of offset values ​​included in the model 98.

[0090] The processor 80 repeatedly performs a series of processes, including inputting the example image 94 to the model 98, calculating the error 102, calculating the multiple adjustment values ​​104, and adjusting the multiple optimization variables in the model 98, using the multiple teacher data 92 stored in the storage 82. That is, the processor 80 optimizes the model 98 by adjusting the multiple optimization variables in the model 98 using the multiple adjustment values ​​104 calculated so as to minimize the error 102 for each of the multiple example images 94 included in the multiple teacher data 92 stored in the storage 82. The processor 80 optimizes the model 98 to generate a trained model 106. When the example image 94 is input to the trained model 106 thus generated, the trained model 106 generates and outputs an image having the same bit depth as the correct image 96 as an image corresponding to the input example image 94.

[0091] Fig. 5 is a conceptual diagram showing an example of a method for creating the example image 94 and the correct image 96. As shown in Fig. 5, in this embodiment, an imaging device 500 is used to create the example image 94 and the correct image 96. The imaging device 500 is an example of a "first imaging device" according to the present disclosure.

[0092] The image sensor 20 mounted on the imaging device 500 is provided with a photoelectric conversion element 72, similar to the imaging device 10. Also, similar to the imaging device 10, pixels P of different colors (i.e., R pixels, G pixels, and B pixels) are regularly arranged in the photoelectric conversion element 72. The arrangement pattern of the pixels P of different colors is the same as that of the imaging device 10. In the example shown in FIG. 5, in order to facilitate understanding of the present disclosure, the number of pixels P is illustrated as being smaller than the actual number of pixels P (4×4 pixels in the example shown in FIG. 5).

[0093] The imaging device 500 has the same configuration and specifications as the imaging device 10, except that the imaging device 500 can generate RAW format images with a deeper bit depth than the imaging device 10. Therefore, in the following, the components of the imaging device 500 are denoted by the same reference numerals as the components of the imaging device 10, and descriptions thereof will be omitted.

[0094] The imaging device 500 captures an image of a sample subject to generate a first single image 108A for one frame as a RAW image with a bit depth of 14 bits. The first single image 108A obtained by capturing an image of the sample subject by the image sensor 20 of the imaging device 500 is used as a correct image 96.

[0095] In the first single image 108A, pixels P1 of different colors are regularly arranged. Here, the pixel P1 refers to a pixel (in other words, an image pixel) constituting a RAW image obtained by imaging using the image sensor 20 of the imaging device 500. The position of the pixel P1 in the RAW image obtained by imaging using the image sensor 20 of the imaging device 500 coincides with the position of the pixel P in the photoelectric conversion element 72 of the image sensor 20 mounted on the imaging device 500.

[0096] In the example shown in FIG. 5, the second single image 108B is created based on the first single image 108A. The second single image 108B is a RAW format image with a bit depth of 12 bits. The second single image 108B is obtained by bit-shifting the first single image 108A. For example, the bit depth of each of all pixels P1 of the first single image 108A is shifted right by 2 bits, "0" is inserted into the two most significant bits that are left, and the two least significant bits that are left by the right shift are discarded to create the second single image 108B. The second single image 108B thus obtained is used as the example image 94.

[0097] In the example shown in FIG. 5, a first single image 108A obtained by capturing an image of a sample subject with the imaging device 500 and a second single image 108B obtained by bit-shifting the first single image 108A are illustrated, but this is merely an example. For example, the first single image 108A and / or the second single image 108B may be virtual images generated by a generation AI. The generation AI may be an AI specialized in generating images, or may be an AI such as GPT-4 (Internet search engine<https: / / openai.com / gpt-4> The AI ​​may be a generation AI that generates and outputs the first single image 108A and / or the second single image 108B according to input instruction data (so-called prompts), such as ChatGPT, which uses the above-mentioned command data.

[0098] FIG. 6 is a conceptual diagram showing an example of an operation stage (i.e., a stage in which the trained model 106 makes an inference) of the trained model 106 generated by performing the learning process of the example shown in FIG. 4. As shown in FIG. 6, in the imaging device 10, the trained model 106 is stored in the storage 64. Also, the storage 64 stores an image quality improvement program 107. In the imaging device 10, the processor 62 reads out the image quality improvement program 107 from the storage 64 and executes the read image quality improvement program 107 on the memory 66. The processor 62 performs image quality improvement processing according to the image quality improvement program 107 executed on the memory 66. The image quality improvement processing is a process of inputting a RAW image 75A (see also FIG. 2) to the trained model 106 stored in the storage 64, causing the trained model 106 to generate and output a high-image-quality image 75A1 that is an image with higher image quality than the RAW image 75A, and causing the processor 62 to acquire the high-image-quality image 75A1. Here, an image with higher image quality than the RAW image 75A refers to, for example, an image with a deeper bit depth than the RAW image 75A. An example of the RAW image 75A shown in Fig. 6 is a 12-bit image, and an example of an image with a deeper bit depth than the RAW image 75A (i.e., the high-image-quality image 75A1) is a 14-bit image. In this embodiment, the high-image-quality image 75A1 is an example of an "inference result" according to the present disclosure.

[0099] The processor 62 performs image processing, including development, on the high-quality image 75A1 to generate an image file 75B, and stores the generated image file 75B in the image memory .

[0100] Next, the operation of the part of the learning device 79 according to the present disclosure will be described with reference to Fig. 7. Fig. 7 shows an example of the flow of the learning process executed by the processor 80. The flow of the learning process shown in Fig. 7 is an example of the "method of generating a trained model" according to the present disclosure.

[0101] In the learning process shown in Figure 7, first, in step ST10, the processor 80 executes the processing of step ST10 to acquire unprocessed teacher data 92 (i.e., teacher data 92 that has not been used in the learning process shown in Figure 7) from the storage 82, and then the learning process proceeds to step ST12.

[0102] In step ST12, the processor 80 inputs the example image 94 included in the training data 92 acquired in step ST10 to the model 98. After the processing of step ST12 is executed, the learning processing proceeds to step ST14. As a result of the processing of step ST12 being executed, a comparison image 100 is output from the model 98.

[0103] In step ST14, the processor 80 acquires the comparison image 100 output from the model 98. After the process of step ST14 is executed, the learning process proceeds to step ST16.

[0104] In step ST16, the processor 80 compares the comparison image 100 acquired in step ST14 with the correct answer image 96 included in the training data 92 acquired in step ST10. After the processing of step ST16 is executed, the learning processing proceeds to step ST18.

[0105] In step ST18, the processor 80 adjusts the model 98 using a plurality of adjustment values ​​104 obtained by comparing the comparison image 100 and the correct answer image 96 in step ST16. The process of step ST18 is repeatedly executed based on all the training data 92 stored in the storage 82, thereby optimizing the model 98. After the process of step ST18 is executed, the learning process proceeds to step ST20.

[0106] In step ST20, the processor 80 determines whether or not unprocessed teacher data 92 is stored in the storage 82. In step ST20, if unprocessed teacher data 92 is stored in the storage 82, the determination is affirmative, and the learning process proceeds to step ST10. In step ST20, if unprocessed teacher data 92 is not stored in the storage 82, the determination is negative, and the learning process ends.

[0107] Next, the operation of the portion of the imaging device 10 according to the present disclosure will be described with reference to Fig. 8. For convenience of explanation, the description will be given on the assumption that the trained model 106 has already been stored in the storage 64.

[0108] 8, first, in step ST50, the processor 62 determines whether or not one frame's worth of image has been captured by the image sensor 20. In step ST50, if one frame's worth of image has not been captured by the image sensor 20, the determination is negative, and the image quality improvement process proceeds to step ST58. If one frame's worth of image has been captured by the image sensor 20, the determination is positive, and the image quality improvement process proceeds to step ST52.

[0109] In step ST52, the processor 62 acquires the RAW image 75A from the image sensor 20. After the process of step ST52 is executed, the image quality improvement process proceeds to step ST54.

[0110] In step ST54, the processor 62 inputs the RAW image 75A acquired in step ST52 to the trained model 106. After the processing of step ST54 is executed, the image quality improvement processing proceeds to step ST56. By executing the processing of step ST54, the trained model 106 generates and outputs an image in which the image quality of the RAW image 75A has been improved (in other words, an image with a deeper bit depth than the RAW image 75A), i.e., a high-image-quality image 75A1.

[0111] In step ST56, the processor 62 acquires the high-quality image 75A1. Then, the processor 62 generates an image file 75B based on the high-quality image 75A1 and stores it in the image memory 46. After the process of step ST56 is executed, the image quality improvement process proceeds to step ST58.

[0112] In step ST58, the processor 62 determines whether or not a condition for terminating the image quality improvement process (hereinafter referred to as the "termination condition") has been satisfied. An example of the termination condition is that an instruction to terminate the image quality improvement process has been accepted by the acceptance device 76. If the termination condition has not been satisfied in step ST58, the determination is negative, and the image quality improvement process proceeds to step ST50. If the termination condition has been satisfied in step ST58, the determination is positive, and the image quality improvement process terminates.

[0113] As described above, the teacher data 92 according to this embodiment is used for machine learning of the model 98. The trained model 106 is generated by machine learning the model 98. The teacher data 92 includes an example image 94 and a correct answer image 96, and the example image 94 and the correct answer image 96 are associated with each other. The example image 94 is an image with a smaller bit depth than the correct answer image 96. Therefore, by inputting the RAW image 75A to the trained model 106 thus configured, it is possible to generate a high-quality image 75A1 as an image with a deeper bit depth than the RAW image 75A.

[0114] That is, it is possible to obtain a high-quality image 75A1 having a deeper bit depth than the RAW image 75A obtained by imaging using the imaging device 10 (in other words, an image with a higher bit precision than the original performance of the imaging device 10 (i.e., the performance before the trained model 106 is installed)), and as a result, it is possible to suppress the loss of the fine structure of the subject on the image and the occurrence of banding. Also, even if the imaging device 10 generates a RAW image 75A having a bit depth below a certain level at a frame rate above a certain level, it is possible to generate a high-quality image 75A1 having a deeper bit depth than the RAW image 75A by the trained model 106 installed in the imaging device 10 without lowering the frame rate.

[0115] In this embodiment, both the example image 94 and the correct answer image 96 are images in RAW format. Therefore, by inputting the RAW image 75A to the trained model 106, a high-quality image 75A1 in RAW format can be generated as an image with a deeper bit depth than the RAW image 75A.

[0116] In addition, in this embodiment, the master image 96 is bit-shifted to create the example image 94 having a shallower bit depth than the master image 96. Therefore, the example image 94 can be created more easily than when the example image 94 is created independently of the master image 96. This contributes to reducing the effort required to create the teacher data 92.

[0117] In this embodiment, the correct image 96 is generated by capturing an image of a sample subject by the imaging device 500. The example image 94 is created from the correct image 96 generated in this manner. Therefore, the characteristics of the imaging device 500 can be reflected in the example image 94 and the correct image 96. If the imaging device 500 is of the same type and has the same specifications as the imaging device 10, the characteristics of the imaging device 10 can also be reflected in the example image 94 and the correct image 96. This allows the trained model 106 to generate and output a high-quality image 75A1 in which degradation in image quality caused by the characteristics of the imaging device 10 is suppressed.

[0118] In the above embodiment, the first single image 108A of one frame obtained by imaging the sample object by the imaging device 500 is used as the correct image 96, and the sample image 94 is created by bit-shifting the correct image 96. However, this is merely an example, and the sample image 94 and the correct image 96 may be created based on a plurality of images. To achieve this, for example, as shown in FIG. 9, the imaging device 500 first images the sample object to generate third to sixth single images 109A to 109D as images in a RAW format with a bit depth of 12 bits. Then, the sample image 94 is created by synthesizing the third to sixth single images 109A to 109D. The synthesis of the third to sixth single images 109A to 109D means, for example, that the third to sixth single images 109A to 109D are averaged in units of pixels P1 whose positions correspond to each other. Example image 94 shown in FIG. 9 is an example of a "representative image" according to the present disclosure.

[0119] In the example shown in FIG. 9, the correct image 96 is created based on the seventh to tenth single images 110A to 110D. The seventh to tenth single images 110A to 110D correspond to the third to sixth single images 109A to 109D, and are images in RAW format with a deeper bit depth than the third to sixth single images 109A to 109D. The seventh to tenth single images 110A to 110D have a bit depth of 14 bits. The seventh single image 110A is obtained by bit-shifting the third single image 109A. For example, the seventh single image 110A is created by shifting the bit depth of each of all pixels P1 of the third single image 109A to the left by two bits, and putting "0" in the two least significant bits vacated by the left shift. In a similar manner, the eighth single image 110B is obtained by bit-shifting the fourth single image 109B, the ninth single image 110C is obtained by bit-shifting the fifth single image 109C, and the tenth single image 110D is obtained by bit-shifting the sixth single image 109D.

[0120] The correct image 96 is created by combining the seventh to tenth single images 110A to 110D. Combining the seventh to tenth single images 110A to 110D means, for example, averaging the seventh to tenth single images 110A to 110D in units of pixels P1 that correspond to each other in position.

[0121] In this way, in the example shown in FIG. 9, the sample image 94 is obtained by synthesizing the third to sixth single images 109A to 109D (here, as an example, the third to sixth single images 109A to 109D are averaged in units of pixels P1 whose positions correspond to each other). Also, the correct image 96 is obtained by synthesizing the seventh to tenth single images 110A to 110D whose bit depths are deeper than the third to sixth single images 109A to 109D (here, as an example, the seventh to tenth single images 110A to 110D are averaged in units of pixels P1 whose positions correspond to each other). Therefore, the correct image 96 whose bit depth is deeper than the sample image 94 can be easily obtained.

[0122] 9, the example image 94 is obtained by synthesizing the third to sixth single images 109A to 109D that are the basis of the seventh to tenth single images 110A to 110D used to obtain the correct image 96. That is, the example image 94 is obtained by using an image that represents the third to sixth single images 109A to 109D used to obtain the correct image 96. Therefore, compared to the case where the example image 94 is obtained independently of the correct image 96, the example image 94 that has a high affinity with the correct image 96 can be obtained more efficiently.

[0123] 9, images obtained by bit-shifting the third to sixth single images 109A to 109D are used as the seventh to tenth single images 110A to 110D. Therefore, it is possible to easily obtain a correct answer image 96 (i.e., a correct answer image 96 having the same bit depth as the high-quality image 75A1) that is assumed to be the high-quality image 75A1 generated and output by the trained model 106 at the stage when the trained model 106 is operated in the imaging device 10.

[0124] 9 shows an example in which the sample image 94 is created by combining the third to sixth single images 109A to 109D, but this is merely one example, and for example, as shown in Fig. 10, any one of the third to sixth single images 109A to 109D before bit shifting may be used as the sample image 94. In this case, for example, if the third to sixth single images 109A to 109D are images obtained by continuous shooting, an image obtained between the third single image 109A and the sixth single image 109D, that is, the fourth single image 109B or the fifth single image 109C may be used as the sample image 94.

[0125] Also, an image obtained by combining the fourth single image 109B and the fifth single image 109C (for example, by averaging the fourth single image 109B and the fifth single image 109C in units of pixels P1 whose positions correspond to each other) may be adopted as the example image 94. Also, an image generated based on three or less images among the third to sixth single images 109A to 109D before bit shifting (i.e., an image obtained by combining three or less images among the third to sixth single images 109A to 109D before bit shifting) may be adopted as the example image 94.

[0126] It should be noted that any one of the third to sixth single images 109A to 109D used as example image 94, or an image obtained by combining three or less images of the third to sixth single images 109A to 109D, is an example of a "representative image" according to the present disclosure.

[0127] The third to sixth single images 109A to 109D before bit shifting are all images used to obtain the correct image 96. Therefore, as shown in Fig. 10, any one of the third to sixth single images 109A to 109D is adopted as the example image 94, or an image obtained by synthesizing three or less images among the third to sixth single images 109A to 109D is adopted as the example image 94, whereby the example image 94 having a high affinity with the correct image 96 can be obtained more efficiently than when the example image 94 is obtained independently of the correct image 96.

[0128] 10 shows an example in which any one of the third to sixth single images 109A to 109D is used as the example image 94, but this is merely an example, and an image generated based on at least one of the third to sixth single images 109A to 109D may be used as the example image 94. An image generated based on at least one of the third to sixth single images 109A to 109D refers to, for example, an image in which at least one of the third to sixth single images 109A to 109D is processed. Examples of images in which at least one of the third to sixth single images 109A to 109D has been processed include an image in which one of the third to sixth single images 109A to 109D has been converted from RAW format to Log format, an image obtained by applying some kind of image quality improvement processing (e.g., noise reduction and / or edge extraction processing, etc.) to one of the third to sixth single images 109A to 109D, or an image in which multiple of the third to sixth single images 109A to 109D have been converted from RAW format to Log format and then averaged.

[0129] In the example shown in Fig. 9, the seventh to tenth single images 110A to 110D are combined to create the correct image 96, but this is merely an example, and for example, as shown in Fig. 11, the third to sixth single images 109A to 109D may be added in units of pixels P1 whose positions correspond to each other to create the correct image 96 with a bit depth of 14 bits. In this way, it is not necessary to create the seventh to tenth single images 110A to 110D from the third to sixth single images 109A to 109D (i.e., it is not necessary to bit-shift the third to sixth single images 109A to 109D), so that the correct image 96 with a bit depth of 14 bits can be created more easily than in the example shown in Fig. 9.

[0130] In the above embodiment, the example image 94 is illustrated, but this is merely an example, and the present disclosure is valid even if an example image 94A is used instead of the example image 94, as shown in FIG. 12. The example image 94A is different from the example image 94 in that it has a lower bit depth. That is, the bit depth of the example image 94 is 12 bits, whereas the bit depth of the example image 94A is 10 bits. In the example shown in FIG. 12, the example image 94A is created based on the eleventh to fourteenth single images 112A to 112D, which have a lower bit depth than the third to sixth single images 109A to 109D.

[0131] In the example shown in FIG. 12, the eleventh to fourteenth single images 112A to 112D (i.e., the eleventh to fourteenth single images 112A to 112D with a bit depth of 10 bits) with a bit depth of two bits less than the third to sixth single images 109A to 109D are created by bit-shifting the third to sixth single images 109A to 109D in the same manner as the second single image 108B with a bit depth of two bits less than the third to sixth single images 109A to 109D is created by bit-shifting the third to sixth single images 109A to 109D. Then, the example image 94A is created by synthesizing the eleventh to fourteenth single images 112A to 112D. Synthesizing the eleventh to fourteenth single images 112A to 112D means, for example, averaging the eleventh to fourteenth single images 112A to 112D in units of pixels P1 whose positions correspond to each other.

[0132] The difference between the bit depth of the example image 94A and the bit depth of the correct image 96 thus obtained is larger than the difference between the bit depth of the example image 94 and the correct image 96. Therefore, the trained model 106 generated by optimizing the model 98 by using the example image 94A and the correct image 96 for machine learning for the model 98 can have a larger difference in the bit depth of the input and output images than the trained model 106 generated by optimizing the model 98 by using the example image 94 and the correct image 96 for machine learning for the model 98. For example, when a RAW image 75A with a bit depth of 10 bits is input to the trained model 106 generated by optimizing the model 98 by using the example image 94A and the correct image 96 for machine learning for the model 98, the trained model 106 can generate and output a high-quality image 75A1 (i.e., a 14-bit high-quality image 75A1) with a bit depth 4 bits deeper than the input RAW image 75A.

[0133] In the example shown in FIG. 12, an example is given in which the correct image 96 is created by taking the arithmetic average of the seventh to tenth single images 110A to 110D. However, this is merely one example, and the correct image 96 having a bit depth of 12 bits may also be created by taking the arithmetic average of the third to sixth single images 109A to 109D.

[0134] 10 shows an example in which the seventh to tenth single images 110A to 110D are combined to create the correct image 96, but this is merely one example, and the present disclosure can be applied even if the correct image 96A is used instead of the correct image 96, as shown in Fig. 13. The correct image 96A has the same bit depth as the correct image 96, but is different in that the image size is 1 / 4 of the correct image 96.

[0135] The correct image 96A is created based on a plurality of image regions obtained by dividing the seventh single image 110A. Here, an example in which the seventh single image 110A is divided is given, but this is merely an example, and any of the eighth to tenth single images 110B to 110D (see Figs. 9 to 12) may be divided.

[0136] The seventh single image 110A is divided into a processing target area 110A1, a first adjacent area 110A2 adjacent to the right side of the processing target area 110A1, a second adjacent area 110A3 adjacent to the lower side of the processing target area 110A1, and a third adjacent area 110A4 adjacent to the lower right side of the processing target area 110A1. Here, the processing target area 110A1 is an example of a "processing target area" according to the present disclosure, and the first adjacent area 110A2, the second adjacent area 110A3, and the third adjacent area 110A4 are examples of a "corresponding area" according to the present disclosure.

[0137] The processing target region 110A1, the first adjacent region 110A2, the second adjacent region 110A3, and the third adjacent region 110A4 are all 2×2 sets of pixels P1 (in other words, blocks), and are defined by R pixels, G pixels, G pixels, and B pixels in corresponding arrangement patterns in the seventh single image 110A. That is, the positions of the R pixels, G pixels, G pixels, and B pixels are aligned between the processing target region 110A1, the first adjacent region 110A2, the second adjacent region 110A3, and the third adjacent region 110A4.

[0138] 13, the first adjacent region 110A2, the second adjacent region 110A3, and the third adjacent region 110A4 are synthesized with respect to the processing target region 110A1, and an image having the same bit depth as the seventh single image 110A is created as the correct answer image 96A. Here, the synthesis of the first adjacent region 110A2, the second adjacent region 110A3, and the third adjacent region 110A4 means, for example, that the processing target region 110A1, the first adjacent region 110A2, the second adjacent region 110A3, and the third adjacent region 110A4 are averaged in units of pixels P1 whose positions correspond to each other.

[0139] In this way, the correct answer image 96A created by synthesizing the first adjacent region 110A2, the second adjacent region 110A3, and the third adjacent region 110A4 obtained by dividing the seventh single image 110A has the same bit depth as the seventh single image 110A, and has an image size that is ¼ of the seventh single image 110A. That is, the correct answer image 96A has the same bit depth as the correct answer image 96, and has an image size that is ¼ of the correct answer image 96. Therefore, when the RAW image 75A is input to the trained model 106 in which the model 98 is optimized by using the example image 94 and the correct image 96A for machine learning of the model 98, the trained model 106 can generate and output a high-quality image 75A1 (i.e., a high-quality image 75A1 with a bit depth of 14 bits and an image size of 1 / 4) having a deeper bit depth and a smaller image size than the input RAW image 75A. Here, the Bayer array is illustrated, but even if it is another type of array pattern such as a quad Bayer array, multiple partition areas (e.g., in the case of a quad Bayer array, multiple partition areas each consisting of multiple pixels of the same color that are adjacent to each other) whose array patterns in the image (i.e., the image corresponding to the seventh single image 110A) correspond to each other may be synthesized. In addition, for example, when adjacent pixels of the same color that are regularly adjacent to the pixel to be processed (i.e., one pixel of interest) are present in an image (i.e., a bit-shifted image), an image corresponding to the correct image 96A may be created by combining (e.g., averaging) the pixel to be processed and at least one of the adjacent pixels, such as in a Quad Bayer array.

[0140] In the example shown in Figure 13, an example is given in which the first adjacent region 110A2, the second adjacent region 110A3, and the third adjacent region 110A4 are all synthesized with the processing target region 110A1, but this is merely one example, and it is also possible to synthesize two or less of the first adjacent region 110A2, the second adjacent region 110A3, and the third adjacent region 110A4 with the processing target region 110A1, and discard the rest.

[0141] 13 shows an example in which the image size is reduced to 1 / 4, but this is merely one example, and it is possible to reduce the image size to a size other than 1 / 4. In this case, the number of divisions into which the seventh single image 110A is divided may be changed according to the image size, and multiple image regions obtained by dividing the seventh single image 110A may be synthesized.

[0142] In the above embodiment, as an example of improving the image quality of the RAW image 75A, an example of deepening the bit depth of the entire RAW image 75A has been given. However, the RAW image 75A may be divided into a plurality of image regions, and the image quality of each image region may be improved in a category different from the bit depth, thereby realizing a further improvement in the image quality of the RAW image 75A. In this case, for example, as shown in FIG. 14, the example image 94 is divided into a plurality of first image regions 111. In the example shown in FIG. 14, a 5×5 first image region 111 is shown as an example of the plurality of first image regions 111. Each of the plurality of first image regions 111 is assigned an image height 114 related to the optical system of the imaging device 500. An example of the image height related to the optical system of the imaging device 500 is the image height of the imaging lens 40 (see FIG. 2). Here, the imaging lens 40 is an example of the "optical system" according to the present disclosure, and the image height 114 is an example of the "characteristics of the second imaging device" and the "image height" according to the present disclosure.

[0143] In the example shown in Fig. 14, the correct image 96 is divided into a plurality of second image regions 116 corresponding to a plurality of first image regions 111. In the example shown in Fig. 14, a 5 × 5 second image region 116 is shown as an example of the plurality of second image regions 116.

[0144] The image quality of each of the example image 94 and the correct image 96 includes image quality of a category different from the bit depth. An example of image quality of a category different from the bit depth is image quality determined by the image height 114, which is one of the characteristics of the imaging device 500. An example of image quality determined by the image height 114 is shading of the imaging lens 40 and / or shading on the light receiving surface 72A. The shading of the imaging lens 40 refers to, for example, a phenomenon in which the imaging lens 40 gradually darkens in the radial direction from the optical axis OA (in other words, a phenomenon in which the amount of light gradually decreases from the center to the periphery of the imaging lens 40), and the shading on the light receiving surface 72A refers to a phenomenon in which the signal-to-noise ratio changes from the center to the periphery of the light receiving surface 72A (in other words, a phenomenon in which the signal-to-noise ratio is non-uniform from the center to the periphery of the light receiving surface 72A). In addition, generally, the peripheral part of the light receiving surface 72A is easily affected by aberrations in the peripheral part of the imaging lens 40, and therefore artifacts are likely to occur.

[0145] Therefore, when the example image 94 is divided into a plurality of first image regions 111 and the correct image 96 is divided into a plurality of second image regions 116, an image height 114 is assigned to each of the plurality of first image regions 111 and each of the plurality of second image regions 116. Then, a label 118 capable of identifying an image quality of a category different from the bit depth of the correct image 96 is assigned to each of the plurality of second image regions 116.

[0146] The image height 114 is an indicator of how far the image is from the center of the imaging lens 40. The larger the image height 114, the darker the image quality. Therefore, a label 118 is associated with the image height 114 for each second image region 116 as an indicator of the degree of darkness.

[0147] Note that, here, image quality determined by image height 114 is exemplified as an image quality in a category different from bit depth, but this is merely one example. For example, since it is important that the dynamic range of the shadow part is recognized by the trained model 106 by separating noise from normal image signals, noise determined by the characteristics of an imaging device 500 of the same type and specifications as the imaging device 10 (for example, noise caused by the characteristics of the image sensor 20) may be superimposed on the example image 94 and the correct image 96, and the label 118 may include information indicating that noise is included and enabling identification of the type of noise.

[0148] In addition, even if there is a defective pixel, aberration, flare, ghost, and / or blur caused by the characteristics of the imaging device 500, the label 118 can be made to include information capable of identifying the defective pixel, aberration, flare, ghost, and / or blur in a similar manner.

[0149] As an example, as shown in Fig. 15, a RAW image 75A is divided into a plurality of third image regions 120 in a manner similar to the way in which the example image 94 is divided into a plurality of first image regions 111. In the example shown in Fig. 15, 5 x 5 third image regions 120 are shown. An image height 122 of the imaging lens 40 mounted on the imaging device 10 is assigned to each of the plurality of third image regions 120.

[0150] As shown in FIG. 15, the trained model 106, which is generated by optimizing the model 98 by using the example image 94 and the correct image 96 shown in FIG. 14 for machine learning of the model 98, is used in the image quality improvement process by the processor 62. When the RAW image 75A is input to the trained model 106, the trained model 106 generates and outputs a high-image-quality image 75A1. The high-image-quality image 75A1 is divided into a plurality of fourth image regions 124 in the same manner as the RAW image 75A is divided into a plurality of third image regions 120. In the example shown in FIG. 15, 5×5 fourth image regions 124 are shown. Each fourth image region 124 is assigned the same image height 122 as the third image region 120 whose position corresponds to the fourth image region 120. In addition, each fourth image region 124 is assigned image quality information 126. The image quality information 126 is information that specifies the image quality of the fourth image region 124 (here, as an example, information that specifies the image quality determined by the image height 122).

[0151] The high image quality process includes an image quality adjustment process. The image quality adjustment process is a process of adjusting the image quality of the fourth image area 124 according to the image quality information 126 (for example, a process of improving the image quality). In the image quality adjustment process, image quality adjustment (for example, adjustment of shading, defective pixels, aberration, flare, ghost, and / or blur, etc.) is performed for each of all the fourth image areas 124 according to the image quality information 126 assigned to each of the fourth image areas 124. In this manner, the image quality of each of all the fourth image areas 124 is adjusted, thereby adjusting the overall image quality of the high image quality image 75A1. The high image quality image 75A1 whose image quality has been adjusted is filed in the same manner as in the above embodiment, and the image file 75B obtained by filing is stored in the image memory 46.

[0152] In this manner, the example image 94 is divided into a plurality of first image regions 111, and the correct image 96 is divided into a plurality of second image regions 116 corresponding to the plurality of first image regions 111. By configuring the example image 94 and the correct image 96 in this manner, machine learning can be performed on the model 98 for each first image region 111.

[0153] Moreover, a label 118 is given to each of the second image regions 116. The label 118 is information capable of identifying the image quality of the first image region 111 whose position corresponds to the second image region 116 to which the label 118 is given. Therefore, when the RAW image 75A is input to the trained model 106 optimized by performing machine learning on the model 98 using the correct answer image 96 and the example image 94 configured in this way, it is possible to identify the image quality for each fourth image region 124 of the high image quality image 75A1 generated by the trained model 106. If the image quality for each fourth image region 124 can be identified in this way, it is possible to adjust the image quality for each fourth image region 124 by image quality adjustment processing. Moreover, the image quality identified from the label 118 given to the second image region 116 is determined by the characteristics of the imaging device 500. Therefore, it is possible to identify the image quality determined by the characteristics (e.g., image height 114) of the imaging device 500 for each fourth image region 124 of the high image quality image 75A1 generated by the trained model 106.

[0154] In the example shown in Fig. 14, a label 118 is assigned to each second image region 116, but the present disclosure is not limited thereto, and a label 118 may not be assigned to each second image region 116. For example, as shown in Fig. 16, an image in which the brightness difference occurring according to the image height 114 in the example image 94 is eliminated for each second image region 116 may be used as the correct answer image 96. In this case, an image in which the brightness difference occurring according to the image height 122 is eliminated is generated and output as the high-quality image 75A1 by the trained model 106, and therefore the image quality adjustment process shown in Fig. 15 is not required.

[0155] In the examples shown in Figs. 14 and 16, an example in which the image height 114 is assigned to each first image region 111 of the example image 94 is given, but this is merely one example. For example, as shown in Fig. 17, the example image 94 may not be divided into a plurality of first image regions 111, and the image height 114 may not be assigned. Here, for example, as shown in Fig. 18, when an image in which shading occurs is input as the RAW image 75A to the trained model 106 generated by optimizing the model 98 by performing machine learning on the model 98 using the example image 94 and the correct answer image 96 shown in Fig. 17, an image divided into a plurality of fourth image regions 124 and in which the image height 122 is assigned to each fourth image region 124 is generated by the trained model 106 and output as the high-quality image 75A1. In this case, the processor 62 performs image quality adjustment processing on the high-image-quality image 75A1, whereby image quality adjustment according to the image height 122 assigned to each fourth image region 124 is performed on each fourth image region 124. Image quality adjustment according to the image height 122 refers to, for example, image processing (so-called shading correction) that eliminates shading of the imaging lens 40 and / or shading on the light receiving surface 72A. By performing the examples shown in Figures 17 and 18, it is possible to suppress degradation of image quality in the high-image-quality image 75A1 that occurs depending on the image height 122 (i.e., shading of the imaging lens 40 and / or shading on the light receiving surface 72A).

[0156] In the above embodiment, an image in RAW format is exemplified as the correct image 96, but the present disclosure is not limited thereto. For example, as shown in FIG. 19, an image in Log (logarithmic) format may be used as the correct image 96. In this case, for example, an image in RAW format is input as the example image 94 to the model 98, and an image in RAW format is output as the comparison image 100 from the model 98. The learning process includes a first conversion process. The first conversion process is a process of converting the comparison image 100 from an image in RAW format to an image in Log format. The comparison image 100 obtained as an image in Log format by the processor 80 performing the first conversion process is compared with the correct image 96, and the model 98 is optimized based on the comparison result.

[0157] With this configuration, since the Log format image has a smaller amount of data than the RAW format image, the load required for comparing the correct answer image 96 with the comparison image 100 can be lighter than comparing the correct answer image 96 in the RAW format with the comparison image 100 in the RAW format. Also, it is the shadow area where gradation skips are likely to occur due to insufficient bit precision. Therefore, as shown in FIG. 19, a RAW format image is used as the example image 94, and in the evaluation stage (i.e., the stage of comparing the image output from the model 86 with the correct answer image 96), a Log format image with high precision in the shadow area is used, thereby generating a trained model 106 that generates and outputs an image in which the gradation in the shadow area is highly accurately corrected from the input RAW format image.

[0158] As an example, as shown in FIG. 20, the high image quality processing includes a second conversion processing. The second conversion processing is a processing for converting an image in RAW format into an image in Log format, similar to the first conversion processing. When a RAW image 75A is input to the trained model 106 optimized in the manner shown in FIG. 19, the trained model 106 generates and outputs a high image quality image 75A1 in RAW format. The high image quality image 75A1 in RAW format is converted into a high image quality image 75A1 in Log format by the processor 62 executing the second conversion processing. The high image quality image 75A1 in Log format is filed in the same manner as in the above embodiment, and the image file 75B obtained by the file is stored in the image memory 46.

[0159] In the example shown in FIG. 19, an image in RAW format is illustrated as the example image 94, but the present disclosure is not limited thereto. For example, as shown in FIG. 21, the example image 94 may be an image in Log format. In this case, the correct image 96 is also an image in Log format. The Log format is an example of the "first file format" and the "second file format" according to the present disclosure. An image in Log format is input as the example image 94 to the model 98, and an image in Log format is output as the comparison image 100 from the model 98. Then, in the same manner as in the above embodiment, the comparison image 100 and the correct image 96 are compared, and the model 98 is optimized based on the comparison result. In this way, since the image in Log format has a smaller amount of data than an image in RAW format, the load required for comparing the correct image 96 with the comparison image 100 can be reduced compared to comparing the correct image 96 in RAW format with the comparison image 100 in RAW format.

[0160] As an example, as shown in Fig. 22, when a Log image 75C obtained by converting a RAW image 75A into a Log format is input to the trained model 106 generated in the manner shown in Fig. 21, the trained model 106 generates and outputs a high-quality image 75A1 in the Log format. The high-quality image 75A1 in the Log format is converted into a file in the same manner as in the above embodiment, and the image file 75B obtained by the conversion is stored in the image memory 46.

[0161] 21 is used in the high image quality processing, the same high image quality as in the above embodiment can be achieved by using an image in Log format, which has a smaller amount of data than an image in RAW format, even if the image capture device 10 is a model that does not support images in RAW format (for example, a model that does not support images in RAW format due to a limitation on the transfer speed because the amount of data is large). Also, if a transfer speed limitation is imposed on the image capture device 10 due to the amount of image data, an image in YC format after Log conversion or gamma conversion may be used, and in this case, the image in YC format may be used in the learning process instead of the image in Log format.

[0162] In the above embodiment, various images (e.g., example image 94 and correct image 96) in which R pixels, G pixels, and B pixels are regularly arranged are exemplified, but the various images may also be monochrome images (e.g., an R image consisting of only multiple R images, a G image consisting of only multiple G images, or a B image consisting of only multiple B images).

[0163] In the above embodiment, an example in which the image quality improvement process is performed by the processor 62 of the image processing engine 12 included in the imaging device 10 has been described, but the present disclosure is not limited thereto, and the device performing the image quality improvement process may be, for example, a device provided outside the imaging device 10, such as a server. For example, the server may be realized by cloud computing. Also, the server may be realized by network computing such as fog computing, edge computing, or grid computing. Here, a server is illustrated, but this is merely an example, and at least one personal computer or the like may be used instead of the server.

[0164] In the above embodiment, an example in which the image quality improvement program 107 is stored in the storage 64 has been described, but the present disclosure is not limited to this. For example, the image quality improvement program 107 may be stored in a portable computer-readable non-transitory storage medium such as an SSD or a USB memory. The image quality improvement program 107 stored in the non-transitory storage medium is installed in the image processing engine 12 of the imaging device 10. The processor 62 executes image quality improvement processing in accordance with the image quality improvement program 107.

[0165] In addition, the image quality improvement program 107 may be stored in a storage device such as another computer or server device connected to the imaging device 10 via a network, and the image quality improvement program 107 may be downloaded in response to a request from the imaging device 10 and installed in the image processing engine 12.

[0166] It should be noted that it is not necessary to store the entire image quality improvement program 107 in a storage device such as another computer or server device connected to the imaging device 10, or in the storage 64, and only a part of the image quality improvement program 107 may be stored therein. Note that although the image quality improvement program 107 is mentioned here, the same can be said about the learning program 90.

[0167] In addition, although the image processing engine 12 is built into the imaging device 10 shown in FIGS. 1 and 2, the present disclosure is not limited to this. For example, the image processing engine 12 may be provided outside the imaging device 10.

[0168] In the above embodiment, the image processing engine 12 is exemplified, but the present disclosure is not limited thereto, and a device including an ASIC, an FPGA, and / or a PLD may be applied instead of the image processing engine 12. Also, instead of the image processing engine 12, a combination of a hardware configuration and a software configuration may be used.

[0169] As the hardware resource for executing the image quality improvement processing and / or learning processing described in the above embodiment, various processors as shown below can be used. An example of the processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing the image quality improvement processing and / or learning processing by executing software, i.e., a program. Another example of the processor is a dedicated electric circuit, which is a processor having a circuit configuration designed exclusively for executing a specific process, such as an FPGA, a PLD, or an ASIC. Each processor has a built-in or connected memory, and each processor uses the memory to execute the image quality improvement processing and / or learning processing.

[0170] The hardware resource for executing the image quality improvement process and / or the learning process may be one of these various processors, or may be a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource for executing the image quality improvement process and / or the learning process may be a single processor.

[0171] As an example of a configuration using one processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes image quality improvement processing and / or learning processing. Second, there is a configuration in which a processor is used that realizes the functions of the entire system including multiple hardware resources that execute image quality improvement processing and / or learning processing in a single IC chip, as typified by SoC. In this way, the image quality improvement processing and / or learning processing is realized using one or more of the above various processors as hardware resources.

[0172] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements. Also, the image quality improvement process and / or learning process described above are merely examples. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the order of processing may be changed, without departing from the spirit of the invention.

[0173] The above description and illustrations are detailed descriptions of the parts related to the present disclosure and are merely an example of the present disclosure. For example, the above description of the configuration, function, action, and effect is an example of the configuration, function, action, and effect of the parts related to the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above description and illustrations within the scope of the gist of the present disclosure. In addition, in order to avoid confusion and to facilitate understanding of the parts related to the present disclosure, the above description and illustrations omit explanations of technical common sense that do not require explanation in order to enable the implementation of the present disclosure.

[0174] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or standard was specifically and individually indicated to be incorporated by reference.

[0175] The following supplementary notes are further disclosed regarding the above embodiment.

[0176] (Appendix 1) Training data used in machine learning of a model, The correct image and An example image having a bit depth smaller than that of the correct image; Teacher data.

[0177] (Appendix 2) The above example image is an image in which the bit depth is made shallower than that of the above correct image by bit-shifting the above correct image. Teacher data described in Appendix 1.

[0178] (Appendix 3) The ground truth image is an image obtained by adding together a number of single images. Teacher data described in Appendix 1.

[0179] (Appendix 4) The ground truth image is an image obtained by averaging a number of single images. Teacher data described in Appendix 1.

[0180] (Appendix 5) The above example image is a representative image of multiple single images, The representative image is at least one image of the plurality of single images, or an image generated based on at least one of the plurality of single images. Teacher data described in Appendix 1.

[0181] (Appendix 6) The correct image is an image obtained by averaging the processing target area in the bit-shifted image and at least one area adjacent to the processing target area. Teacher data described in Appendix 1.

[0182] (Appendix 7) The correct image is divided into a plurality of image regions. Teacher data described in Appendix 1.

[0183] (Appendix 8) The correct image is an image obtained by capturing an image using an imaging device having an optical system, Each of the plurality of image regions is assigned an image height related to the optical system. Teacher data described in Appendix 7.

[0184] (Appendix 9) A trained model in which the model is optimized by performing the machine learning on the model using the training data according to any one of Supplementary Note 1 to Supplementary Note 8.

[0185] (Appendix 10) Inputting an image captured by an image sensor into the trained model described in Supplementary Note 9; and A program (e.g., image quality improvement program 107) for causing a computer (e.g., image processing engine 12) to execute image quality improvement processing including obtaining an inference result output from the trained model in response to input of the captured image.

[0186] (Appendix 11) A program (e.g., a learning program 90) that causes a computer (e.g., a learning device 79) to execute a learning process that generates a trained model by performing machine learning on a model using teacher data including a correct image and an example image, The example image has a smaller bit depth than the correct image. The learning process is as follows: Inputting the example image into the model; The model outputs an image to be evaluated (e.g., comparison image 100) in response to the input of the example image; and and optimizing the model based on a comparison result between the evaluation target image and the ground truth image. program.

[0187] (Appendix 12) Inputting an image captured by an image sensor into the trained model described in Supplementary Note 9; and A computer-readable non-transitory storage medium storing a program (e.g., image quality improvement program 107) for causing a computer (e.g., image processing engine 12) to execute image quality improvement processing including obtaining an inference result output from the trained model in response to input of the captured image.

[0188] (Appendix 13) A computer-readable non-transitory storage medium storing a program (e.g., a learning program 90) that causes a computer (e.g., a learning device 79) to execute a learning process that generates a trained model by performing machine learning on a model using teacher data including a correct image and an example image, The example image has a smaller bit depth than the correct image. The learning process is as follows: Inputting the example image into the model; The model outputs an image to be evaluated (e.g., comparison image 100) in response to the input of the example image; and and optimizing the model based on a comparison result between the evaluation target image and the ground truth image. Non-transitory storage media. [Explanation of symbols]

[0189] 10,500 Imaging Device 12 Image Processing Engine 16 Imaging device body 18 Interchangeable Lenses 18A Focus Ring 20 Image Sensor 22 Release button 24 Dial 26 Instruction keys 28 Display 30 Touch Panel 32 Touch panel display 36 Control device 37 First Actuator 38 Second Actuator 39 Third Actuator 40 Imaging Lens 40A Objective Lens 40B Focus Lens 40C Zoom Lens 40D aperture 40D1 aperture 40D2 aperture blades 44 System Controller 46 Image Memory 48 UI Devices 50 External I / F 52 Communication I / F 54 Photoelectric conversion element driver 62,80 Processor 64,82 Storage 66,84 Memory 68 Bus 70 Input / Output Interface 72 Photoelectric conversion element 72A Photosensitive surface 74 A / D Converter 75A RAW image 75A1 High quality image 75B Image file 76 Reception Device 78 Hard Key Section 79 Learning Device 90 Study Programs 92 Teacher Data 94,94A Example image 96,96A Correct image 98 Models 100 Comparison Images 102 error 104 Adjustment value 106 trained models 107 High-definition program 108A 1st single image 108B Second single image 109A 3rd single image 109B 4th single image 109C 5th single image 109D 6th single image 110A 7th single image 110A1 Processing area 110A2 First adjacent region 110A3 Second adjacent region 110A4 Third adjacent area 110B 8th single image 110C 9th single image 110D 10th single image 111 First image area 112A 11th single image 112B 12th single image 112C 13th single image 112D 14th single image 114,122 Image height 116 Second Image Area 118 Label 120 Third Image Area 124 4th Image Area 126 Image quality information OA optical axis P, P1 pixels

Claims

1. Training data used in machine learning of a model, The correct image and An example image having a bit depth smaller than that of the target image. Teacher data.

2. The correct image is an image obtained by synthesizing a plurality of single images. The training data according to claim 1.

3. The example image is a representative image of the plurality of single images. The training data according to claim 2.

4. The example image is an image having a smaller bit depth than the plurality of single images. The training data according to claim 2.

5. The single image is a bit-shifted image. The training data according to claim 2.

6. The correct image is An image obtained by combining a processing target area in a bit-shifted image and a corresponding area in the image, the corresponding area having a corresponding arrangement pattern in the image, when the corresponding area exists in the image, or The training data according to claim 1, wherein the training data is an image obtained by combining a pixel to be processed and an adjacent pixel of the same color that is regularly adjacent to the pixel to be processed in the bit-shifted image when the adjacent pixel is present in the image.

7. The example image is an image before bit shifting or an image generated based on an image before bit shifting. The training data according to claim 6.

8. The correct image and the example image are images obtained by imaging using a first imaging device. The training data according to claim 1.

9. The correct image and the example image are images in RAW format. The training data according to claim 1.

10. The model inputs and outputs images in RAW format, The format of the correct image is Log format, The RAW format image output from the model is converted into a Log format image and compared with the correct answer image to optimize the model. The training data according to claim 1.

11. The model outputs an image in a second file format other than the RAW format in response to an input of an image in a first file format other than the RAW format; the example image is in the first file format; The format of the correct image is the second file format. The training data according to claim 1.

12. The example image is partitioned into a plurality of first image regions; The correct image is divided into a plurality of second image regions corresponding to the plurality of first image regions. The training data according to claim 1.

13. A label capable of identifying image quality is assigned to the second image regions. The training data according to claim 12.

14. The example image is an image that is assumed to be obtained by capturing an image by a second imaging device, The image quality is determined by the characteristics of the second imaging device. The training data according to claim 13.

15. the second imaging device has an optical system; The characteristics include an image height for the optical system. The training data according to claim 14.

16. A trained model in which the model is optimized by performing the machine learning on the model using the training data according to any one of claims 1 to 15.

17. A first processor; An image sensor, The first processor, A captured image obtained by capturing an image with the image sensor is input to the trained model according to claim 16, An inference result output from the trained model is obtained in response to the input of the captured image. Imaging device.

18. A second processor is provided, The second processor optimizes the model by performing the machine learning on the model using the teacher data according to any one of claims 1 to 15. Learning device.

19. A method for creating training data used in machine learning of a model, comprising the steps of: The training data includes a correct answer image and a sample image, Creating a ground truth image; and and creating example images having a smaller bit depth than the ground truth image. How to create training data.

20. A method for generating a trained model, which generates a trained model by performing machine learning on a model using teacher data including a correct image and an example image, The example image has a smaller bit depth than the target image; inputting said example images into said model; The model outputs an evaluation target image in response to the input of the example image; and and optimizing the model based on a comparison result between the evaluation target image and the correct answer image. How to generate a trained model.

Images