Teacher data, learned model, imaging device, learning device, creation method of teacher data, and generation method of learned model

Abstract

To provide teacher data capable of generating an image of higher picture quality than an inputted image with respect to a learned model, to provide a learned model, to provide an imaging device, to provide a learning device, to provide a creation method of teacher data, and to provide a generation method of learned model.SOLUTION: Teacher data is used in machine learning of a model. The teacher data includes a correct answer image in which a plurality of single images is synthesized, and a question image representing the plurality of single images.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to teacher data, a trained model, an imaging device, a learning device, a method for creating teacher data, and a method for generating a trained model. [Background technology]

[0002] Patent Document 1 discloses an image processing device including an image receiving unit that receives a first image and a second image of a different type from the first image for the same object, a first image quality improvement processing unit that improves the image quality of the first image received by the receiving unit using an image quality improvement function that has been trained to improve the image quality of the first image, and a second image quality improvement processing unit that improves the image quality of the second image using the first high image quality image and the second image generated by the first image quality improvement processing unit. The image processing device described in Patent Document 1 further includes a learning unit that learns the image quality improvement function.

[0003] Patent Document 2 discloses a computer-implemented method for increasing the image resolution of a digital image. The computer-implemented method described in Patent Document 2 includes the steps of performing bicubic upsampling of the digital image to generate a base high-resolution (HR) image, converting the digital image from a red-green-blue (RGB) color space to a luma (Y), chroma blue difference (Cb), and chroma red difference (Cr) (YCbCr) color space to generate a low-resolution (LR) residual image, converting the LR residual image into multiple HR residual sub-images corresponding to the digital image using multiple convolution layers of a neural network model, and generating an HR image corresponding to the digital image using the base HR image and the multiple HR residual sub-images.

[0004] In addition, the computer-implemented method described in Patent Document 2 further includes a step of training a neural network model with a plurality of training image pairs, each of the plurality of training image pairs including an LR image corresponding to a training image, the LR image having degraded image quality, configured as an input to the neural network model, and a plurality of HR residual sub-images corresponding to the training images and configured as a target output of the neural network model.

[0005] Patent Document 3 discloses a super-resolution processing device that performs super-resolution processing on an image of a second resolution to create an image of a third resolution higher than the second resolution. The super-resolution processing device described in Patent Document 3 includes a first-layer filter acquisition unit, a reduced image creation unit, a candidate image creation unit, and a super-resolution processing unit. The first-layer filter acquisition unit acquires a first filter for performing super-resolution processing on an image of the second resolution to an image of the third resolution, and a second filter for performing super-resolution processing on an image of the first resolution, which is a lower resolution than the second resolution, to an image of the second resolution, which are machine-learned using the learning subset for each of a plurality of learning subsets in the first layer that are subsets of a learning set including an image of the third resolution. The reduced image creation unit creates a reduced image of the first resolution from an input image of the second resolution. The candidate image creation unit uses each second filter acquired by the first-layer filter acquisition unit to perform super-resolution processing on the reduced image of the first resolution to create a candidate image of the second resolution. The super-resolution processing unit creates a super-resolution image of a third resolution by super-resolution processing the input image of the second resolution using a first filter corresponding to a second filter that minimizes the difference between the candidate image of the second resolution created by the candidate image creation unit and the input image of the second resolution. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2022-096484 A [Patent Document 2] Special Publication No. 2022-536807 [Patent Document 3] JP 2018-151747 A Summary of the Invention

[0007] One embodiment of the present disclosure provides teacher data, a trained model, an imaging device, a learning device, a method for creating teacher data, and a method for generating a trained model that can generate an image of higher image quality than an input image for a trained model. [Means for solving the problem]

[0008] A first aspect of the present disclosure is training data used in machine learning of a model, the training data including a master image synthesized from a plurality of single images and an example image representative of the plurality of single images.

[0009] A second aspect of the present disclosure is training data according to the first aspect, in which the correct image is an image whose resolution has been increased by synthesizing a plurality of single images.

[0010] A third aspect of the present disclosure is training data according to the first or second aspect, in which the correct image is an image having a higher resolution than the example image.

[0011] A fourth aspect of the present disclosure is training data according to the first or second aspect, in which the correct image is an image having a larger number of pixels than the example image.

[0012] A fifth aspect of the present disclosure is training data according to the first or second aspect, in which the correct image is an image having a higher visual resolution than the example image.

[0013] A sixth aspect of the present disclosure is training data according to the second aspect, in which each of the plurality of single images is a pixel-shifted image.

[0014] A seventh aspect of the present disclosure is training data according to the sixth aspect, in which each of the plurality of single images is an image shifted by 1 / 2 pixel.

[0015] An eighth aspect of the present disclosure is training data related to the first aspect, in which each of the multiple single images is a pixel-shifted image, and the correct image is an image of high image quality obtained by synthesizing the multiple single images.

[0016] A ninth aspect of the present disclosure is training data according to the eighth aspect, in which pixels of different colors are regularly arranged in the multiple single images, and each of the multiple single images is an image that has been pixel-shifted to a position where the pixels of the different colors overlap.

[0017] A tenth aspect of the present disclosure is training data relating to the fifth aspect, in which a plurality of single images are obtained by capturing images using a first image sensor including phase difference pixels and non-phase difference pixels, and each of the plurality of single images is an image pixel-shifted to a position where pixels corresponding to the phase difference pixels and pixels corresponding to the non-phase difference images overlap.

[0018] An eleventh aspect of the present disclosure is training data relating to any one of the first to tenth aspects, in which the correct image is an image having improved image quality compared to a single image due to a factor that affects image quality, and the example image is an image having lower image quality compared to the correct image due to a factor.

[0019] A twelfth aspect of the present disclosure is training data according to the first aspect, in which the factors are focal length, aperture value, lens characteristics, pixel thinning characteristics, tone correction function, gain correction function, and / or noise reduction function.

[0020] A thirteenth aspect of the present disclosure is training data relating to any one of the first to twelfth aspects, in which the correct image and the example image include in-focus and out-of-focus areas, and the correct image is an image in which the degree of image quality improvement differs between the in-focus and out-of-focus areas.

[0021] A fourteenth aspect according to the present disclosure is training data according to the thirteenth aspect, in which the correct image is an image in which the out-of-focus region is imaged to a smaller degree of image quality improvement than the in-focus region.

[0022] A fifteenth aspect of the present disclosure is training data according to any one of the first to fourteenth aspects, in which the correct image and the example image are RAW images.

[0023] A sixteenth aspect of the present disclosure is training data according to any one of the first to fourteenth aspects, in which the correct image and the example image are images based on a RAW image.

[0024] A seventeenth aspect of the present disclosure is training data according to any one of the first to sixteenth aspects, in which the correct image and the example image are images in RGB format or images in YCbCr format.

[0025] An 18th aspect of the present disclosure is training data relating to any one of the 1st to 17th aspects, wherein the example image is a first image based on a number of single images that is fewer than the number of single images among the plurality of single images.

[0026] A 19th aspect of the present disclosure is training data relating to any one of the 1st to 18th aspects, wherein the example image is a second image obtained by thinning pixels out of the plurality of single images, the number of which is equal to or less than the number of the plurality of single images.

[0027] A twentieth aspect of the present disclosure is training data relating to any one of the first to nineteenth aspects, wherein each of the multiple single images is obtained by capturing an image from a different imaging position by a second image sensor and is an image shifted by 1 / 2 pixel, and the example image is a single image among the multiple single images that has a center closest to the center of gravity when the multiple single images are superimposed.

[0028] A 21st aspect of the present disclosure is training data relating to any one of the 1st to 20th aspects, wherein the example image is a single image having false color and / or false resolution among a plurality of single images, or an image generated based on a single image having false color and / or false resolution among a plurality of single images.

[0029] A 22nd aspect of the present disclosure is a trained model generated by optimizing a model by performing machine learning on the model using training data relating to any one of the first to 21st aspects.

[0030] A 23rd aspect of the present disclosure is an imaging device comprising a first processor and a third image sensor, wherein the first processor inputs an image obtained by capturing an image using the third image sensor into a trained model according to the 22nd aspect, and obtains an inference result output from the trained model in response to the input of the captured image.

[0031] A 24th aspect of the present disclosure is a learning device including a second processor, the second processor optimizing a model by performing machine learning on the model using training data relating to any one of the first to 21st aspects.

[0032] A 25th aspect of the present disclosure is a method for creating training data for use in machine learning of a model, the training data including a master image and an example image, and including creating a master image by synthesizing a plurality of single images, and creating an image that represents the plurality of single images as an example image.

[0033] A 26th aspect of the present disclosure is a method for generating a trained model by performing machine learning on a model using training data including a correct image and an example image, wherein the correct image is an image synthesized from a plurality of single images and the example image is an image representative of the plurality of single images, the method including inputting the example image into the model, the model outputting an image to be evaluated in response to the input of the example image, and optimizing the model based on a comparison result between the image to be evaluated and the correct image. [Brief description of the drawings]

[0034] [Figure 1] 1 is a schematic diagram showing an example of the overall configuration of an imaging device; [Diagram 2] FIG. 2 is a schematic diagram showing an example of a hardware configuration of an optical system and an electrical system of the imaging apparatus. [Diagram 3] FIG. 2 is a schematic diagram showing an example of the configuration of a learning device. [Figure 4] FIG. 13 is a conceptual diagram showing an example of a method for creating a correct image. [Diagram 5] FIG. 13 is a conceptual diagram showing an example of a method for creating an example image. [Figure 6] FIG. 2 is a block diagram showing an example of the functions of an image processing engine. [Figure 7] 11 is a flowchart showing an example of the flow of a learning process performed by a learning device. [Figure 8] 10 is a flowchart showing an example of the flow of image quality improvement processing performed by the imaging device. [Figure 9] FIG. 13 is a conceptual diagram showing a first modified example of a method for creating teacher data. [Figure 10] FIG. 13 is a conceptual diagram showing a second modified example of the method for creating teacher data. [Figure 11] FIG. 13 is a conceptual diagram showing a third modified example of the method for creating teacher data. [Figure 12] FIG. 13 is a conceptual diagram showing a fourth modified example of the method for creating teacher data. [Figure 13]10 is a conceptual diagram showing an example of a configuration of training data used to improve image quality in an in-focus area more than in an out-of-focus area. FIG. [Figure 14] 1 is a conceptual diagram showing an example of a configuration of training data including an example image whose image quality has been reduced by a factor and which has false color and spurious resolution, and a correct image whose image quality is higher than that of the example image; [Figure 15] FIG. 13 is a conceptual diagram showing an example of an embodiment in which both the correct image and the example image are JPEG files having images in RGB format. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0035] Below, examples of embodiments of the teacher data, trained model, imaging device, learning device, teacher data creation method, and trained model generation method related to the present disclosure are described with reference to the attached drawings.

[0036] First, the terms used in the following description will be explained.

[0037] CPU is an abbreviation for "Central Processing Unit". GPU is an abbreviation for "Graphics Processing Unit". GPGPU is an abbreviation for "General-purpose computing on graphics processing units". APU is an abbreviation for "Accelerated Processing Unit". TPU is an abbreviation for "Tensor processing unit". NVM is an abbreviation for "Non-volatile memory". RAM is an abbreviation for "Random Access Memory". IC is an abbreviation for "Integrated Circuit". ASIC is an abbreviation for "Application Specific Integrated Circuit". PLD is an abbreviation for "Programmable Logic Device". FPGA is an abbreviation for "Field-Programmable Gate Array". SoC is an abbreviation for "System-on-a-chip". SSD is an abbreviation for "Solid State Drive". USB is an abbreviation for "Universal Serial Bus". HDD is an abbreviation for "Hard Disk Drive". EEPROM is an abbreviation for "Electrically Erasable and Programmable Read Only Memory". EL is an abbreviation for "Electro-Luminescence". I / F is an abbreviation for "Interface". UI is an abbreviation for "User Interface". fps is an abbreviation for "frame per second". MF is an abbreviation for "Manual Focus". AF is an abbreviation for "Auto Focus". CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor". CCD is an abbreviation for "Charge Coupled Device". AI is an abbreviation for "Artificial Intelligence". A / D is an abbreviation for "Analog / Digital".FIR is an abbreviation for "Finite Impulse Response". IIR is an abbreviation for "Infinite Impulse Response". JPEG is an abbreviation for "Joint Photographic Experts Group". TIFF is an abbreviation for "Tagged Image File Format". JPEG XR is an abbreviation for "Joint Photographic Experts Group Extended Range". MPEG is an abbreviation for "Moving Picture Expert Group". AVI is an abbreviation for "Audio Video Interleaved". MTF is an abbreviation for "Modulation Transfer Function".

[0038] In the following description, a coded processor (hereinafter simply referred to as a "processor") may be one physical or virtual arithmetic device, or a combination of multiple physical or virtual arithmetic devices. Also, a processor may be one type of arithmetic device, or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU, a GPU, a GPGPU, an APU, and a TPU.

[0039] In the following description, a signed memory is a memory, such as a RAM, in which information is temporarily stored and which is used as a working memory by a processor.

[0040] In the following description, a storage with a symbol is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory, magnetic disks, and magnetic tapes. Another example of storage is cloud storage.

[0041] In the following embodiments, the external I / F with a symbol controls the transmission and reception of various information between multiple devices connected to each other. An example of the external I / F is a USB interface. A communication I / F including a communication processor and an antenna may be applied to the external I / F. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G, Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0042] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0043] Fig. 1 is a schematic diagram showing an example of the overall configuration of an imaging device. As shown in Fig. 1, imaging device 10 is a device that captures an image of a subject, and includes an image processing engine 12, an imaging device body 16, and an interchangeable lens 18. Here, imaging device 10 is an example of the "imaging device" according to the present disclosure.

[0044] The image processing engine 12 is built into the imaging device body 16, and controls the entire imaging device 10. The interchangeable lens 18 is interchangeably attached to the imaging device body 16. The interchangeable lens 18 is provided with a focus ring 18A. The focus ring 18A is operated by a user of the imaging device 10 (hereinafter simply referred to as a "user") when the user manually adjusts the focus of the imaging device 10 on a subject.

[0045] 1, a digital camera with interchangeable lenses is shown as an example of the imaging device 10. However, this is merely one example, and the imaging device 10 may be a digital camera with fixed lenses, or may be a digital camera built into various electronic devices such as a smart device, a wearable terminal, an endoscope device, a cell observation device, an ophthalmic observation device, or a surgical microscope.

[0046] The imaging device body 16 is provided with an image sensor 20. The image sensor 20 is an example of a "third image sensor" according to the present disclosure. The image sensor 20 is a CMOS image sensor. The image sensor 20 captures an image of an imaging range including at least one subject. When the interchangeable lens 18 is attached to the imaging device body 16, subject light indicative of the subject passes through the interchangeable lens 18 and is focused on the image sensor 20, and image data indicative of the image of the subject is generated by the image sensor 20.

[0047] In this embodiment, a CMOS image sensor is exemplified as the image sensor 20, but the present disclosure is not limited to this, and the present disclosure also applies even if the image sensor 20 is another type of image sensor, such as a CCD image sensor.

[0048] A release button 22 and a dial 24 are provided on the top surface of the imaging device body 16. The dial 24 is operated when setting an operation mode of the imaging system and an operation mode of the playback system, and by operating the dial 24, an imaging mode, a playback mode, and a setting mode are selectively set as an operation mode in the imaging device 10. The imaging mode is an operation mode that causes the imaging device 10 to capture an image. The playback mode is an operation mode that plays back an image (e.g., a still image and / or a moving image) obtained by capturing an image for recording in the imaging mode. The setting mode is an operation mode that is set for the imaging device 10 when, for example, setting various setting values ​​used in control related to imaging.

[0049] The release button 22 functions as an imaging preparation instruction section and an imaging instruction section, and is capable of detecting two-stage pressing operations: an imaging preparation instruction state and an imaging instruction state. The imaging preparation instruction state refers to a state in which the button is pressed, for example, from a standby position to an intermediate position (half-pressed position), and the imaging instruction state refers to a state in which the button is pressed beyond the intermediate position to a final pressed position (fully pressed position).

[0050] In the following, the “state in which the button is pressed from the standby position to the half-pressed position” will be referred to as the “half-pressed state,” and the “state in which the button is pressed from the standby position to the fully-pressed position” will be referred to as the “fully-pressed state.” Depending on the configuration of imaging device 10, the imaging preparation instruction state may be a state in which the user's finger is in contact with release button 22, and the imaging instruction state may be a state in which the operating user's finger has moved from a state in which it is in contact with release button 22 to a state in which it is released.

[0051] The imaging device body 16 is provided on the rear surface with instruction keys 26 and a touch panel display 32. The touch panel display 32 includes a display 28 and a touch panel 30 (see also FIG. 2). An example of the display 28 is an EL display (e.g., an organic EL display or an inorganic EL display). The display 28 may be another type of display, such as a liquid crystal display, instead of an EL display.

[0052] The display 28 displays images and / or text information, etc. When the imaging device 10 is in imaging mode, the display 28 is used for capturing images for live view images, i.e., for displaying live view images obtained by performing continuous imaging. Here, the term "live view image" refers to a moving image for display based on image data obtained by capturing images using the image sensor 20. The imaging performed to obtain the live view image (hereinafter also referred to as "capturing images for live view images") is performed according to a frame rate of, for example, 60 fps. 60 fps is merely an example, and the frame rate may be less than 60 fps or may be more than 60 fps.

[0053] The display 28 is also used to display a still image obtained by capturing an image for a still image when an instruction to capture an image for a still image is given to the imaging device 10 via the release button 22. The display 28 is also used to display a playback image, etc. when the imaging device 10 is in a playback mode. Furthermore, the display 28 is also used to display a menu screen on which various menus can be selected, and a setting screen for setting various setting values, etc. used in control related to imaging, when the imaging device 10 is in a setting mode.

[0054] The touch panel 30 is a transmissive touch panel, and is overlaid on the surface of the display area of ​​the display 28. The touch panel 30 receives instructions from the user by detecting contact with a pointing object such as a finger or a stylus pen. For ease of explanation, the above-mentioned "full press state" hereinafter also includes a state in which the user turns on a soft key for starting imaging via the touch panel 30.

[0055] In this embodiment, an out-cell type touch panel display in which the touch panel 30 is overlaid on the surface of the display area of ​​the display 28 is given as an example of the touch panel display 32, but this is merely one example. For example, an on-cell type or in-cell type touch panel display can also be applied as the touch panel display 32.

[0056] The instruction keys 26 accept various instructions. Here, "various instructions" refers to, for example, an instruction to display a menu screen, an instruction to select one or more menus, an instruction to confirm the selected content, an instruction to erase the selected content, various instructions such as zooming in, zooming out, and frame-by-frame advance. These instructions may also be given by the touch panel 30.

[0057] FIG. 2 is a schematic diagram showing an example of a hardware configuration of an optical system and an electrical system of an imaging device. As shown in FIG. 2, the image sensor 20 includes a photoelectric conversion element 72. The photoelectric conversion element 72 has a light receiving surface 72A. The photoelectric conversion element 72 is disposed in the imaging device body 16 so that the center of the light receiving surface 72A coincides with the optical axis OA (see also FIG. 1). The photoelectric conversion element 72 has a plurality of photosensitive pixels arranged in a matrix, and the light receiving surface 72A is formed by the plurality of photosensitive pixels. Each photosensitive pixel has a microlens (not shown). Each photosensitive pixel is a physical pixel having a photodiode (not shown), which photoelectrically converts received light and outputs an electrical signal according to the amount of received light.

[0058] In addition, color filters (not shown) of the three primary colors of light, i.e., red (hereinafter also referred to as "R"), green (hereinafter also referred to as "G"), or blue (hereinafter also referred to as "B"), are arranged in a predetermined pattern array in the multiple photosensitive pixels. In this embodiment, a Bayer array is used as an example of the predetermined pattern array. However, the Bayer array is merely an example, and the present disclosure is valid even if the predetermined pattern array is another type of pattern array such as a G-stripe R / G complete checkerboard, an X-Trans (registered trademark) array, or a honeycomb array.

[0059] For convenience of explanation, the photosensitive pixel having a microlens and an R color filter is referred to as an R pixel, the photosensitive pixel having a microlens and a G color filter is referred to as a G pixel, and the photosensitive pixel having a microlens and a B color filter is referred to as a B pixel. For convenience of explanation, the electric signal output from the R pixel of the photosensitive pixel is referred to as an "R signal", the electric signal output from the G pixel of the photosensitive pixel is referred to as a "G signal", and the electric signal output from the B pixel of the photosensitive pixel is referred to as a "B signal". For convenience of explanation, the R signal, the G signal, and the B signal are also referred to as "RGB color signals". For convenience of explanation, the R pixel, the G pixel, and the B pixel constituting the RAW image 75A generated based on the RGB color signals are also referred to as "R pixel", "G pixel", and "B pixel". Here, R pixels, G pixels, and B pixels are shown as examples, but this is merely one example. If colors other than R, G, and B (i.e., colors other than the primary colors) are regularly arranged along with R, G, and B in the color filter, RAW image 75A will also include pixels of colors other than R, G, and B pixels, and this disclosure will still apply in such cases.

[0060] The interchangeable lens 18 includes an imaging lens 40. The imaging lens 40 has an objective lens 40A, a focus lens 40B, a zoom lens 40C, and an aperture 40D. The objective lens 40A, the focus lens 40B, the zoom lens 40C, and the aperture 40D are arranged in this order along the optical axis OA from the subject side (object side) to the imaging device body 16 side (image side).

[0061] The interchangeable lens 18 also includes a control device 36, a first actuator 37, a second actuator 38, and a third actuator 39. The control device 36 controls the entire interchangeable lens 18 in accordance with instructions from the imaging device body 16. The control device 36 is, for example, a device having a computer including a processor, storage, memory, and the like. The storage of the control device 36 is, for example, an EEPROM. Various programs and various parameters are stored in the storage of the control device 36. The memory of the control device 36 is, for example, a RAM, which temporarily stores various information and is used as a work memory. In the control device 36, the processor reads out necessary programs from the storage and executes the various programs read out on the memory to control the entire imaging lens 40.

[0062] Although a device having a computer is given here as an example of the control device 36, this is merely an example and devices including an ASIC, an FPGA, and / or a PLD may also be applied. Also, a device realized by combining a hardware configuration and a software configuration, for example, may be used as the control device 36.

[0063] The first actuator 37 includes a focusing slide mechanism (not shown) and a focusing motor (not shown). The focusing slide mechanism has a focus lens 40B attached thereto so as to be slidable along the optical axis OA. The focusing motor is also connected to the focusing slide mechanism, and the focusing slide mechanism is actuated by receiving power from the focusing motor to move the focus lens 40B along the optical axis OA.

[0064] The second actuator 38 includes a zoom slide mechanism (not shown) and a zoom motor (not shown). The zoom lens 40C is attached to the zoom slide mechanism so as to be slidable along the optical axis OA. The zoom motor is also connected to the zoom slide mechanism, and the zoom slide mechanism operates by receiving power from the zoom motor to move the zoom lens 40C along the optical axis OA.

[0065] The third actuator 39 includes a power transmission mechanism (not shown) and a diaphragm motor (not shown). The diaphragm 40D has an opening 40D1, and the size of the opening 40D1 is variable. The opening 40D1 is formed by, for example, a plurality of diaphragm blades 40D2. The plurality of diaphragm blades 40D2 are connected to the power transmission mechanism. In addition, a diaphragm motor is connected to the power transmission mechanism, and the power transmission mechanism transmits the power of the diaphragm motor to the plurality of diaphragm blades 40D2. The plurality of diaphragm blades 40D2 change the size of the opening 40D1 by operating upon receiving the power transmitted from the power transmission mechanism. The diaphragm 40D adjusts the exposure by changing the size of the opening 40D1.

[0066] The focus motor, zoom motor, and aperture motor are connected to the control device 36, and the control device 36 controls the driving of the focus motor, zoom motor, and aperture motor. In this embodiment, a stepping motor is used as an example of the focus motor, zoom motor, and aperture motor. Therefore, the focus motor, zoom motor, and aperture motor operate in synchronization with a pulse signal according to a command from the control device 36. Here, an example is shown in which the focus motor, zoom motor, and aperture motor are provided in the interchangeable lens 18, but this is merely an example, and at least one of the focus motor, zoom motor, and aperture motor may be provided in the imaging device body 16. The components and / or operation method of the interchangeable lens 18 can be changed as necessary.

[0067] In the imaging device 10, in the imaging mode, the MF mode and the AF mode are selectively set according to an instruction given to the imaging device body 16. The MF mode is an operation mode in which the focus is adjusted manually. In the MF mode, for example, when the focus ring 18A or the like is operated by the user, the focus lens 40B moves along the optical axis OA by an amount of movement corresponding to the amount of operation of the focus ring 18A or the like, thereby adjusting the focus.

[0068] In the AF mode, the imaging device body 16 calculates the in-focus position according to the subject distance, and adjusts the focus by moving the focus lens 40B toward the calculated in-focus position. Here, the in-focus position refers to the position on the optical axis OA of the focus lens 40B when the subject is in focus.

[0069] The imaging device main body 16 includes an image sensor 20, an image processing engine 12, a system controller 44, an image memory 46, a UI device 48, an external I / F 50, a communication I / F 52, a photoelectric conversion element driver 54, and an input / output interface 70. The image sensor 20 also includes a photoelectric conversion element 72 and an A / D converter 74.

[0070] The input / output interface 70 is connected to the image processing engine 12, the image memory 46, the UI device 48, the external I / F 50, the photoelectric conversion element driver 54, the mechanical shutter driver 56, and an A / D converter 74. In addition, the input / output interface 70 is also connected to the control device 36 of the interchangeable lens 18.

[0071] The system controller 44 includes a processor (not shown), a storage (not shown), and a memory (not shown). In the system controller 44, the storage is a computer-readable non-transient storage medium, and various parameters and various programs are stored in the storage. The storage of the system controller 44 is, for example, an EEPROM. However, this is merely an example, and instead of the EEPROM, or together with the EEPROM, a HDD and / or an SSD may be applied as the storage of the system controller 44. In addition, the memory of the system controller 44 temporarily stores various information and is used as a work memory. In the system controller 44, the processor reads out necessary programs from the storage and executes the various programs read out on the memory to control the entire imaging device 10. That is, in the example shown in FIG. 2, the image processing engine 12, the image memory 46, the UI device 48, the external I / F 50, the communication I / F 52, the photoelectric conversion element driver 54, and the control device 36 are controlled by the system controller 44.

[0072] The image processing engine 12 operates under the control of the system controller 44. The image processing engine 12 includes a processor 62, a storage 64, and a memory 66. Here, the processor 62 is an example of a "first processor" according to the present disclosure.

[0073] The processor 62, the storage 64, and the memory 66 are connected via a bus 68, which is connected to an input / output interface 70. In the example shown in Fig. 2, for convenience of illustration, one bus is shown as the bus 68, but multiple buses may be used. The bus 68 may be a serial bus, or may be a parallel bus including a data bus, an address bus, a control bus, and the like.

[0074] The storage 64 is a computer-readable non-transitory storage medium, and stores various parameters and programs different from the various parameters and programs stored in the storage of the system controller 44. The storage 64 is, for example, an EEPROM. However, this is merely one example, and instead of the EEPROM, or together with the EEPROM, an HDD and / or an SSD may be used as the storage 64. The memory 66 is, for example, a RAM, which temporarily stores various information and is used as a work memory.

[0075] The processor 62 reads out a necessary program from the storage 64, and executes the read program in the memory 66. The processor 62 performs image processing in accordance with the program executed on the memory 66.

[0076] The photoelectric conversion element 72 is connected to the photoelectric conversion element driver 54. The photoelectric conversion element driver 54 supplies an imaging timing signal that specifies the timing of imaging performed by the photoelectric conversion element 72 to the photoelectric conversion element 72 in accordance with an instruction from the processor 62. The photoelectric conversion element 72 performs resetting, exposure, and output of an electrical signal in accordance with the imaging timing signal supplied from the photoelectric conversion element driver 54. Examples of the imaging timing signal include a vertical synchronization signal and a horizontal synchronization signal.

[0077] When the interchangeable lens 18 is attached to the imaging device body 16, subject light incident on the imaging lens 40 is imaged on the light receiving surface 72A by the imaging lens 40. Under the control of the photoelectric conversion element driver 54, the photoelectric conversion element 72 photoelectrically converts the subject light received by the light receiving surface 72A, and outputs an electrical signal according to the amount of subject light to the A / D converter 74 as analog image data indicating the subject light. Specifically, the A / D converter 74 reads out the analog image data from the photoelectric conversion element 72 in units of frames and for each horizontal line using an exposure sequential readout method.

[0078] The A / D converter 74 digitizes the analog image data to generate a RAW image 75A. The RAW image 75A is an image in which R pixels, G pixels, and B pixels are arranged in a mosaic pattern. The RAW image 75A is an example of a "captured image" according to the present disclosure. In the present embodiment, as an example, the number of bits (in other words, the bit length, color bit number, or color depth) that is a value expressing the gradation of each of the R pixels, B pixels, and G pixels included in the RAW image 75A is 14 bits. Note that 14 bits is merely an example, and the number of bits may be more than 14 bits or less than 14 bits.

[0079] In this embodiment, as an example, the processor 62 of the image processing engine 12 acquires a RAW image 75A from the A / D converter 74, and performs image processing, including development, on the acquired RAW image 75A to generate an image file 75B. Here, development refers to a process of compressing a luminance and color difference signal according to a default compression method. Examples of the default compression method (i.e., the format of the image file) include JPEG, TIFF, JPEG XR, MPEG, and AVI. The image processing also includes image quality adjustment of the RAW image 75A. The image quality adjustment of the RAW image 75A is realized by focal length adjustment, aperture value adjustment, lens characteristic adjustment, thinning characteristic adjustment between pixels P, a gradation correction function (for example, a process of correcting the gradation of an RGB image according to a gamma value), a gain correction function, and a noise reduction function. Other examples of image quality adjustments to the RAW image 75A include color space conversion processing (i.e., processing for converting the color space of an RGB image that has been gamma corrected from the RGB color space to the YCbCr color space), luminance filtering processing (i.e., processing for filtering the luminance signal (the so-called Y signal) using a luminance filter (not shown)), color difference processing (i.e., processing for filtering the Cb signal and the Cr signal to reduce high-frequency noise), and / or resizing processing (i.e., processing for adjusting the luminance and color difference signals so that the size of the image indicated by the luminance and color difference signals matches the size specified by a user, etc.).

[0080] An example of the image file 75B is a JPEG file. Note that the JPEG file is merely an example, and the image file 75B may be other types of image files, such as a JPEG XR file, a TIFF file, an MPEG file, an AVI file, etc. The image file 75B is stored in the image memory 46 by the processor 62.

[0081] The UI-based device 48 includes a display 28, and the processor 62 causes the display 28 to display various pieces of information. The UI-based device 48 also includes a reception device 76. The reception device 76 includes a touch panel 30 and a hard key unit 78. The hard key unit 78 is a plurality of hard keys including the instruction keys 26 (see FIG. 1 ). The processor 62 operates in accordance with various instructions received by the touch panel 30. Note that, although the hard key unit 78 is included in the UI-based device 48 here, the present disclosure is not limited to this, and for example, the hard key unit 78 may be connected to the external I / F 50.

[0082] The external I / F 50 controls the exchange of various information with devices (hereinafter also referred to as "external devices") that exist outside the imaging device 10. An example of the external I / F 50 is a USB interface. To the USB interface, external devices (not shown), such as a smart device, a personal computer, a server, a USB memory, a memory card, and / or a printer, are directly or indirectly connected.

[0083] The communication I / F 52 is connected to a network (not shown). The communication I / F 52 controls the exchange of information between a communication device (not shown) such as a server on the network and the system controller 44. For example, the communication I / F 52 transmits information in response to a request from the system controller 44 to the communication device via the network. The communication I / F 52 also receives information transmitted from the communication device and outputs the received information to the system controller 44 via the input / output interface 70.

[0084] Incidentally, the image processing engine 12 performs processing to improve the image quality of the RAW image 75A by applying AI to the RAW image 75A. Hereinafter, a method of generating AI to be applied to the RAW image 75A will be described with reference to FIGS.

[0085] Fig. 3 is a conceptual diagram showing an example of the configuration of a learning device 79. As shown in Fig. 3, the learning device 79 includes a processor 80, a storage 82, and a memory 84. The hardware configuration of the processor 80, storage 82, memory 84, etc. in the learning device 79 is basically the same as the hardware configuration of the above-mentioned processor 62, storage 64, memory 66, etc., so a description thereof will be omitted here. In the example shown in Fig. 3, the learning device 79 is an example of a "learning device" according to the present disclosure.

[0086] The storage 82 stores a learning program 90. The processor 80 is an example of a "second processor" according to the present disclosure. The processor 80 reads out the learning program 90 from the storage 82 and executes the read learning program 90 on the memory 84. The processor 80 performs a learning process according to the learning program 90 executed on the memory 84. The learning process is a process of generating a trained model 106 from a model 98. The trained model 106 is generated by the processor 80 executing machine learning on the model 98. That is, the trained model 106 is generated by optimizing the model 98 by machine learning. For example, the model 98 is a neural network having hundreds of millions to trillions of intermediate layers. An example of the model 98 is a model for a generation AI that generates and outputs an image with higher image quality than an input image (for example, an image having at least a resolution several times higher than the input image).

[0087] The storage 82 stores a plurality of (e.g., tens of thousands to hundreds of billions) pieces of teacher data 92. The teacher data 92 is used for machine learning of the model 98. That is, in the learning device 79, the processor 80 acquires the plurality of teacher data 92 from the storage 82, and performs machine learning on the model 98 using the acquired plurality of teacher data 92.

[0088] The teacher data 92 is labeled data. The labeled data is, for example, data in which an example image 94 (in other words, example data) and a correct answer image 96 (in other words, correct answer data) are associated with each other. Here, the teacher data 92 is an example of "teacher data" according to the present disclosure, the example image 94 is an example of "example image" according to the present disclosure, and the correct answer image 96 is an example of "correct answer image" according to the present disclosure.

[0089] The example image 94 is an image that assumes the RAW image 75A. An image that assumes the RAW image 75A refers to an image of the same image quality as the RAW image 75A, including the resolution, for example. In this embodiment, an image obtained by actually capturing an image of a sample subject (for example, the subject captured in the correct answer image 96 shown in FIG. 3) is used as the example image 94. However, this is merely an example, and the image that assumes the RAW image 75A may be an image that is virtually generated. An example of a virtually generated image is an image generated by a generation AI or the like.

[0090] The correct image 96 is an image in which the image quality of the example image 94 has been increased. An example of an image in which the image quality has been increased is an image in which the resolution is higher than that of the example image 94. An image in which the resolution is higher than that of the example image 94 refers to an image having a greater number of pixels than the example image 94, for example. In other words, an image in which the resolution is higher than that of the example image 94 can be said to have a higher visual resolution than the example image 94.

[0091] The processor 80 acquires the teacher data 92 one by one from the storage 82. The processor 80 inputs the example image 94 from the teacher data 92 acquired from the storage 82 to the model 98. When the example image 94 is input to the model 98, the model 98 generates a comparison image 100 which is an image with a higher resolution than the example image 94 (i.e., an image with a larger number of pixels than the example image 94). The comparison image 100 is an image used for comparison with the correct answer image 96 associated with the example image 94 input to the model 98. In this embodiment, the comparison image 100 is an example of an "evaluation target image" according to the present disclosure.

[0092] The processor 80 calculates an error 102 between a ground truth image 96 associated with an example image 94 input to the model 98 and a comparison image 100. The error 102 is an example of a "comparison result" according to the present disclosure. The processor 80 calculates a number of adjustment values ​​104 that minimize the error 102. The processor 80 then adjusts a number of optimization variables in the model 98 using the multiple adjustment values ​​104. Here, the multiple optimization variables refer to, for example, a number of connection weights and a number of offset values ​​included in the model 98.

[0093] The processor 80 repeatedly performs a series of processes, including inputting the example image 94 to the model 98, calculating the error 102, calculating the multiple adjustment values ​​104, and adjusting the multiple optimization variables in the model 98, using the multiple teacher data 92 stored in the storage 82. That is, the processor 80 optimizes the model 98 by adjusting the multiple optimization variables in the model 98 using the multiple adjustment values ​​104 calculated so as to minimize the error 102 for each of the multiple example images 94 included in the multiple teacher data 92 stored in the storage 82. The processor 80 optimizes the model 98 to generate a trained model 106. When the example image 94 is input to the trained model 106 thus generated, the trained model 106 generates and outputs an image having the same resolution as the correct image 96 as an image corresponding to the input example image 94.

[0094] FIG. 4 is a conceptual diagram showing an example of a method for creating a correct image 96. As shown in FIG. 4, in this embodiment, an imaging device 500 is used to create the correct image 96. The imaging device 500 is different from the imaging device 10 in that it has a pixel shift device 502. The image sensor 20 mounted on the imaging device 500 is provided with a photoelectric conversion element 72, similar to the imaging device 10. Also, similar to the imaging device 10, pixels P of different colors (i.e., R pixels, G pixels, and B pixels) are regularly arranged in the photoelectric conversion element 72. The arrangement pattern of the pixels P of different colors is the same as that of the imaging device 10. In the example shown in FIG. 4, in order to facilitate understanding of the present disclosure, the number of pixels P is illustrated as being smaller than the actual number of pixels P.

[0095] The pixel shifting device 502 is mechanically coupled to the photoelectric conversion element 72. The pixel shifting device 502 includes a power source (e.g., a voice coil motor, etc.), and selectively shifts the photoelectric conversion element 72 up, down, left, and right by one pixel pitch or 1 / 2 (0.5) pixel pitch by transmitting power generated by the power source to the photoelectric conversion element 72. The pixel shifting device 502 pixel-shifts the photoelectric conversion element 72 under the control of the system controller 44 (see FIG. 2). In this embodiment, the photoelectric conversion element 72 selectively shifts one pixel or 1 / 2 pixel up, down, left, and right (i.e., the photoelectric conversion element 72 selectively moves up, down, left, and right by one pixel pitch or 1 / 2 (0.5) pixel pitch). In the example shown in FIG. 4, the image sensor 20 mounted on the imaging device 500 is an example of the "first image sensor" and the "second image sensor" according to the present disclosure.

[0096] In the imaging device 500, the first to fourth single images 108A to 108D are generated as RAW images in the first to fourth steps. Furthermore, the fifth to eighth single images 110A to 110D are generated as RAW images in the fifth to eighth steps. Furthermore, the ninth to twelfth single images 112A to 112D are generated as RAW images in the ninth to twelfth steps. Furthermore, the thirteenth to sixteenth single images 114A to 114D are generated as RAW images in the thirteenth to sixteenth steps. Furthermore, the first to fourth finely divided images 116 to 122 are generated in the seventeenth to twentieth steps. Furthermore, the correct image 96 is generated in the twenty-first step.

[0097] In the following, for ease of explanation, when there is no need to distinguish between the first to fourth single images 108A to 108D, the fifth to eighth single images 110A to 110D, the ninth to twelfth single images 112A to 112D, and the thirteenth to sixteenth single images 114A to 114D, they will be referred to as "single images (or unit images)" without being given reference numerals.

[0098] In the first step, the sample object is imaged by the image sensor 20 with the photoelectric conversion element 72 positioned at the home position, thereby generating a first single image 108A. In the second step, the sample object is imaged by the image sensor 20 with the photoelectric conversion element 72 shifted leftward by 1 / 2 pixel from the imaging position of the photoelectric conversion element 72 in the first step, thereby generating a second single image 108B. In the third step, the sample object is imaged by the image sensor 20 with the photoelectric conversion element 72 shifted downward by 1 / 2 pixel from the imaging position of the photoelectric conversion element 72 in the second step, thereby generating a third single image 108C. In the fourth step, the sample object is imaged by the image sensor 20 with the photoelectric conversion element 72 shifted rightward by 1 / 2 pixel from the imaging position of the photoelectric conversion element 72 in the third step, thereby generating a fourth single image 108D.

[0099] Each of the first to fourth single images 108A to 108D thus generated has pixels P1 of different colors regularly arranged and is an image that has been pixel-shifted to positions where the pixels P1 of different colors overlap (i.e., an image that has been pixel-shifted so that the pixels P1 of different colors overlap). Here, the pixel P1 refers to a pixel that constitutes a RAW image obtained by imaging using the image sensor 20 of the imaging device 500 (in other words, an image pixel). The position of the pixel P1 in the RAW image obtained by imaging using the image sensor 20 of the imaging device 500 coincides with the position of the pixel P in the photoelectric conversion element 72 of the image sensor 20 mounted on the imaging device 500.

[0100] In the fifth step, the sample object is imaged by the image sensor 20 in a state shifted one pixel to the left from the imaging position of the photoelectric conversion element 72 in the first step, thereby generating a fifth single image 110A. In the sixth step, the sample object is imaged by the image sensor 20 in a state shifted 1 / 2 pixel to the left from the imaging position of the photoelectric conversion element 72 in the fifth step, thereby generating a sixth single image 110B. In the seventh step, the sample object is imaged by the image sensor 20 in a state shifted 1 / 2 pixel downward from the imaging position of the photoelectric conversion element 72 in the sixth step, thereby generating a seventh single image 110C. In the eighth step, the sample object is imaged by the image sensor 20 in a state shifted 1 / 2 pixel to the right from the imaging position of the photoelectric conversion element 72 in the seventh step, thereby generating an eighth single image 110D.

[0101] Each of the fifth to eighth single images 110A to 110D generated in this manner is an image in which pixels P1 of different colors are regularly arranged and pixel-shifted to positions where the pixels P1 of different colors overlap (i.e., an image pixel-shifted so that the pixels P1 of different colors overlap).

[0102] In the ninth step, the sample object is imaged by the image sensor 20 in a state shifted downward by one pixel from the imaging position of the photoelectric conversion element 72 in the fifth step, thereby generating a ninth single image 112A. In the tenth step, the sample object is imaged by the image sensor 20 in a state shifted 1 / 2 pixel to the left from the imaging position of the photoelectric conversion element 72 in the ninth step, thereby generating a tenth single image 112B. In the eleventh step, the sample object is imaged by the image sensor 20 in a state shifted 1 / 2 pixel downward from the imaging position of the photoelectric conversion element 72 in the tenth step, thereby generating an eleventh single image 112C. In the twelfth step, the sample object is imaged by the image sensor 20 in a state shifted 1 / 2 pixel to the right from the imaging position of the photoelectric conversion element 72 in the eleventh step, thereby generating a twelfth single image 112D.

[0103] Each of the ninth to twelfth single images 112A to 112D generated in this manner is an image in which pixels P1 of different colors are regularly arranged and pixel-shifted to positions where the pixels P1 of different colors overlap (i.e., an image pixel-shifted so that the pixels P1 of different colors overlap).

[0104] In the thirteenth step, the sample object is imaged by the image sensor 20 in a state shifted one pixel to the right from the imaging position of the photoelectric conversion element 72 in the ninth step, thereby generating a thirteenth single image 114A. In the fourteenth step, the sample object is imaged by the image sensor 20 in a state shifted 1 / 2 pixel to the left from the imaging position of the photoelectric conversion element 72 in the thirteenth step, thereby generating a fourteenth single image 114B. In the fifteenth step, the sample object is imaged by the image sensor 20 in a state shifted 1 / 2 pixel downward from the imaging position of the photoelectric conversion element 72 in the fourteenth step, thereby generating a fifteenth single image 114C. In the sixteenth step, the sample object is imaged by the image sensor 20 in a state shifted 1 / 2 pixel to the right from the imaging position of the photoelectric conversion element 72 in the fifteenth step, thereby generating a sixteenth single image 114D.

[0105] Each of the 13th to 16th single images 114A to 114D generated in this manner is an image in which pixels P1 of different colors are regularly arranged and pixel-shifted to positions where the pixels P1 of different colors overlap (i.e., an image pixel-shifted so that the pixels P1 of different colors overlap).

[0106] In the 17th step, the first finely divided image 116 is created by combining the first to fourth single images 108A to 108D. For example, the first finely divided image 116 is an image created by overlapping the first to fourth single images 108A to 108D. Here, each of the first to fourth single images 108A to 108D is an image shifted by 1 / 2 pixel. In other words, each of the first to fourth single images 108A to 108D can be said to be an image having a relationship of being shifted by 1 / 2 pixel from each other. Therefore, in the first finely divided image 116 created by overlapping the first to fourth single images 108A to 108D, the pixel P1 is divided into four parts, R, G, G, and B, and therefore the pixel P1 is finely divided and the resolution is increased compared to each of the first to fourth single images 108A to 108D.

[0107] In the eighteenth step, the fifth to eighth single images 110A to 110D are combined to generate the second finely divided image 118. For example, the second finely divided image 118 is an image generated by overlapping the fifth to eighth single images 110A to 110D. Here, each of the fifth to eighth single images 110A to 110D is an image shifted by 1 / 2 pixel. In other words, each of the fifth to eighth single images 110A to 110D can be said to be an image having a relationship of being shifted by 1 / 2 pixel from each other. Therefore, in the second finely divided image 118 generated by overlapping the fifth to eighth single images 110A to 110D, the pixel P1 is divided into four parts, R, G, G, and B, and therefore the pixel P1 is finer than each of the fifth to eighth single images 110A to 110D.

[0108] In the nineteenth step, the third finely divided image 120 is created by combining the ninth to twelfth single images 112A to 112D. For example, the third finely divided image 120 is an image created by overlapping the ninth to twelfth single images 112A to 112D. Here, each of the ninth to twelfth single images 112A to 112D is an image shifted by 1 / 2 pixel. In other words, each of the ninth to twelfth single images 112A to 112D can be said to be an image having a relationship of being shifted by 1 / 2 pixel from each other. Therefore, in the third finely divided image 120 created by overlapping the ninth to twelfth single images 112A to 112D, the pixel P1 is divided into four parts by R, G, G, and B, and therefore the pixel P1 is finely divided and the resolution is increased compared to each of the ninth to twelfth single images 112A to 112D.

[0109] In the twentieth step, the fourth finely divided image 122 is generated by combining the thirteenth to sixteenth single images 114A to 114D. For example, the fourth finely divided image 122 is an image generated by combining the thirteenth to sixteenth single images 114A to 114D (here, as an example, an image generated by overlapping the thirteenth to sixteenth single images 114A to 114D). Here, each of the thirteenth to sixteenth single images 114A to 114D is an image shifted by 1 / 2 pixel. In other words, each of the thirteenth to sixteenth single images 114A to 114D can be said to be images having a relationship of being shifted by 1 / 2 pixel from each other. Therefore, in the fourth finely divided image 122 created by combining the thirteenth to sixteenth single images 114A to 114D, pixel P1 is divided into four parts: R, G, G, and B, so that pixel P1 is finer than that in each of the fifth to eighth single images 114A to 114D, and the resolution is higher.

[0110] In the 21st step, the first to fourth finely divided images 116-122 created from the first to fourth single images 108A-108D, the fifth to eighth single images 110A-110D, the ninth to twelfth single images 112A-112D, and the thirteenth to sixteenth single images 114A-114D are synthesized (here, as an example, superimposed in units of pixels P1 whose positions correspond to each other) to create a correct image 96. The correct image 96 created in this way is an image with higher image quality than each of the first to fourth single images 108A-108D, the fifth to eighth single images 110A-110D, the ninth to twelfth single images 112A-112D, and the thirteenth to sixteenth single images 114A-114D. In the example shown in FIG. 4, the image with higher image quality refers to an image with higher resolution. In other words, the correct image 96 thus created is an image having a larger number of pixels than each of the first to fourth single images 108A to 108D, the fifth to eighth single images 110A to 110D, the ninth to twelfth single images 112A to 112D, and the thirteenth to sixteenth single images 114A to 114D. In other words, the correct image 96 thus created is an image having a higher visual resolution than each of the first to fourth single images 108A to 108D, the fifth to eighth single images 110A to 110D, the ninth to twelfth single images 112A to 112D, and the thirteenth to sixteenth single images 114A to 114D. In addition, since the correct image 96 is an image in which the first to fourth detailed images 116-122 are combined, it can be said that the correct image 96 has a higher degree of color reproduction than each of the first to fourth detailed images 116-122.

[0111] Also, the first to fourth single images 108A to 108D, the fifth to eighth single images 110A to 110D, the ninth to twelfth single images 112A to 112D, and the thirteenth to sixteenth single images 114A to 114D are images generated as RAW images. Therefore, the first to fourth finely divided images 116 to 122 generated from the first to fourth single images 108A to 108D, the fifth to eighth single images 110A to 110D, the ninth to twelfth single images 112A to 112D, and the thirteenth to sixteenth single images 114A to 114D are also RAW images. Therefore, the first to fourth finely divided images 116 to 122 are superimposed on each other in units of corresponding pixels P1, so that the correct image 96 is also generated as a RAW image.

[0112] In the example shown in FIG. 4, the correct image 96 is generated based on the first to fourth single images 108A to 108D, the fifth to eighth single images 110A to 110D, the ninth to twelfth single images 112A to 112D, and the thirteenth to sixteenth single images 114A to 114D obtained by capturing images of a sample subject by the imaging device 500, but this is merely an example. For example, at least one of the first to fourth single images 108A to 108D, the fifth to eighth single images 110A to 110D, the ninth to twelfth single images 112A to 112D, and the thirteenth to sixteenth single images 114A to 114D used to generate the correct image 96 may be a virtual image generated by a generating AI. The generating AI may be an AI specialized in generating images, or may be a GPT-4 (Internet search engine).<https: / / openai.com / gpt-4> The AI ​​may be a generation AI that generates and outputs at least one of the first to fourth single images 108A to 108D, the fifth to eighth single images 110A to 110D, the ninth to twelfth single images 112A to 112D, and the thirteenth to sixteenth single images 114A to 114D in accordance with input instruction data (so-called prompts), such as ChatGPT, which uses

[0113] FIG. 5 is a conceptual diagram showing an example of a method for creating the sample image 94. FIG. 5 shows a schematic example of the first to fourth single images 108A to 108D, the fifth to eighth single images 110A to 110D, the ninth to twelfth single images 112A to 112D, and the thirteenth to sixteenth single images 114A to 114D viewed from the front. As shown in FIG. 5, the first to fourth single images 108A to 108D have centers 108A1 to 108D1. The first to fourth single images 108A to 108D have centers G1 when the first to fourth single images 108A to 108D are superimposed. Similarly, the fifth to eighth single images 110A to 110D have centers 110A1 to 110D1, respectively. The fifth to eighth single images 110A to 110D have a center of gravity G2 when the fifth to eighth single images 110A to 110D are superimposed. Similarly, the ninth to twelfth single images 112A to 112D have centers 112A1 to 112D1, respectively. The ninth to twelfth single images 112A to 112D have a center of gravity G3 when the ninth to twelfth single images 112A to 112D are superimposed. Similarly, the thirteenth to sixteenth single images 114A to 114D have centers 114A1 to 114D1, respectively. The thirteenth to sixteenth single images 114A to 114D have a center of gravity G4 when the thirteenth to sixteenth single images 114A to 114D are superimposed.

[0114] Here, the example image 94 is an image created by synthesizing the first single image 108A, the fifth single image 110A, the ninth single image 112A, and the thirteenth single image 114A (here, as an example, superimposing them in units of pixels P1 whose positions correspond to each other). The first single image 108A is an image representative of the first to fourth single images 108A to 108D. The fifth single image 110A is an image representative of the fifth to eighth single images 110A to 110D. The ninth single image 112A is an image representative of the ninth to twelfth single images 112A to 112D. The thirteenth single image 114A is an image representative of the thirteenth to sixteenth single images 114A to 114D.

[0115] The first single image 108A used to create the example image 94 is the image having the center closest to the center of gravity G1 among the first to fourth single images 108A to 108D. That is, among the centers 108A1 to 108D1, the center closest to the center of gravity G1 is the center 108A1, and since the center 108A1 is the center of the first single image 108A, the first single image 108A is used to create the example image 94.

[0116] The fifth single image 110A used to create the example image 94 is the image having the center closest to the center of gravity G2 among the fifth to eighth single images 110A to 110D. That is, among the centers 110A1 to 110D1, the center closest to the center of gravity G2 is the center 110A1, and since the center 110A1 is the center of the fifth single image 110A, the fifth single image 110A is used to create the example image 94.

[0117] The ninth single image 112A used to create the example image 94 is the image having the center closest to the center of gravity G3 among the ninth to twelfth single images 112A to 112D. That is, among the centers 112A1 to 112D1, the center closest to the center of gravity G3 is the center 112A1, and since the center 112A1 is the center of the ninth single image 112A, the ninth single image 112A is used to create the example image 94.

[0118] The thirteenth single image 114A used to create the example image 94 is the image having the center closest to the center of gravity G4 among the thirteenth to sixteenth single images 114A to 114D. That is, among the centers 114A1 to 114D1, the center closest to the center of gravity G4 is the center 114A1, and since the center 114A1 is the center of the thirteenth single image 114A, the thirteenth single image 114A is used to create the example image 94.

[0119] FIG. 6 is a conceptual diagram showing an example of an operation stage (i.e., a stage in which the trained model 106 makes inference) of the trained model 106 generated by performing the learning process of the example shown in FIG. 3. As shown in FIG. 6, in the imaging device 10, the trained model 106 is stored in the storage 64. Also, the storage 64 stores a high image quality program 124. In the imaging device 10, the processor 62 reads the high image quality program 124 from the storage 64 and executes the read high image quality program 124 on the memory 66. The processor 62 performs the high image quality process according to the high image quality program 124 executed on the memory 66. The high image quality process is a process in which a RAW image 75A (see also FIG. 2) is input to the trained model 106 stored in the storage 64, the trained model 106 generates and outputs a high resolution image 75A1, which is an image in which the RAW image 75A has been made high resolution, and the processor 62 acquires the high resolution image 75A1. Here, the high resolution image 75A1 is an example of an "inference result" according to the present disclosure.

[0120] RAW image 75A is an image with the same resolution as example image 94, and high-resolution image 75A1 is an image with the same resolution as answer image 96 (see FIGS. 3 and 4). Processor 62 performs image processing, including development, on high-resolution image 75A1 to generate image file 75B, and stores generated image file 75B in image memory 46.

[0121] Next, the operation of the part of the learning device 79 according to the present disclosure will be described with reference to Fig. 7. Fig. 7 shows an example of the flow of the learning process executed by the processor 80. The flow of the learning process shown in Fig. 7 is an example of the "method of generating a trained model" according to the present disclosure.

[0122] In the learning process shown in Figure 7, first, in step ST10, the processor 80 executes the processing of step ST10 to acquire unprocessed teacher data 92 (i.e., teacher data 92 that has not been used in the learning process shown in Figure 7) from the storage 82, and then the learning process proceeds to step ST12.

[0123] In step ST12, the processor 80 inputs the example image 94 included in the training data 92 acquired in step ST10 to the model 98. After the processing of step ST12 is executed, the learning processing proceeds to step ST14. As a result of the processing of step ST12 being executed, a comparison image 100 is output from the model 98.

[0124] In step ST14, the processor 80 acquires the comparison image 100 output from the model 98. After the process of step ST14 is executed, the learning process proceeds to step ST16.

[0125] In step ST16, the processor 80 compares the comparison image 100 acquired in step ST14 with the correct answer image 96 included in the training data 92 acquired in step ST10. After the processing of step ST16 is executed, the learning processing proceeds to step ST18.

[0126] In step ST18, the processor 80 adjusts the model 98 using the multiple adjustment values ​​104 obtained by comparing the comparison image 100 and the correct image 96 in step ST16. The process of step ST18 is repeatedly executed based on all the training data 92 stored in the storage 82, thereby optimizing the model 98. After the process of step ST18 is executed, the learning process proceeds to step ST20.

[0127] In step ST20, the processor 80 determines whether or not unprocessed teacher data 92 is stored in the storage 82. In step ST20, if unprocessed teacher data 92 is stored in the storage 82, the determination is affirmative, and the learning process proceeds to step ST10. In step ST20, if unprocessed teacher data 92 is not stored in the storage 82, the determination is negative, and the learning process ends.

[0128] Next, the operation of the portion of the imaging device 10 according to the present disclosure will be described with reference to Fig. 8. For convenience of explanation, the description will be given on the assumption that the trained model 106 has already been stored in the storage 64.

[0129] 8, first, in step ST50, the processor 62 determines whether or not one frame's worth of image has been captured by the image sensor 20. In step ST50, if one frame's worth of image has not been captured by the image sensor 20, the determination is negative, and the image quality improvement process proceeds to step ST58. If one frame's worth of image has been captured by the image sensor 20, the determination is positive, and the image quality improvement process proceeds to step ST52.

[0130] In step ST52, the processor 62 acquires the RAW image 75A from the image sensor 20. After the process of step ST52 is executed, the image quality improvement process proceeds to step ST54.

[0131] In step ST54, the processor 62 inputs the RAW image 75A acquired in step ST52 to the trained model 106. After the process of step ST54 is executed, the image quality improvement process proceeds to step ST56. By executing the process of step ST54, the trained model 106 generates and outputs an image in which the image quality of the RAW image 75A is improved, that is, a high-resolution image 75A1.

[0132] In step ST56, the processor 62 acquires the high resolution image 75A1. Then, the processor 62 generates an image file 75B based on the high resolution image 75A1 and stores it in the image memory 46. After the process of step ST56 is executed, the image quality improvement process proceeds to step ST58.

[0133] In step ST58, the processor 62 determines whether or not a condition for terminating the image quality improvement process (hereinafter referred to as the "termination condition") has been satisfied. An example of the termination condition is that an instruction to terminate the image quality improvement process has been accepted by the acceptance device 76. If the termination condition has not been satisfied in step ST58, the determination is negative, and the image quality improvement process proceeds to step ST50. If the termination condition has been satisfied in step ST58, the determination is positive, and the image quality improvement process terminates.

[0134] As described above, the teacher data 92 according to this embodiment is used for machine learning of the model 98. The model 98 is machine-learned to generate the trained model 106. The teacher data 92 includes an example image 94 and a correct answer image 96, and the example image 94 and the correct answer image 96 are associated with each other. The correct answer image 96 is an image obtained by synthesizing the first to fourth single images 108A to 108D, the fifth to eighth single images 110A to 110D, the ninth to twelfth single images 112A to 112D, and the thirteenth to sixteenth single images 114A to 114D. The example image 94 is an image (here, the 1st single image 108A as an example) representing the 1st to 4th single images 108A to 108D, an image (here, the 5th single image 110A as an example) representing the 5th to 8th single images 110A to 110D, an image (here, the 13th single image 114A as an example) representing the 9th to 12th single images 112A to 112D, and an image (here, the 13th single image 114A as an example) synthesized from an image (here, the 14th single image 114A as an example) representing the 13th to 16th single images 114A to 114D. That is, the correct image 96 is an image of higher image quality than the example image 94. Therefore, by inputting the RAW image 75A to the trained model 106 configured in this way, an image of higher image quality than the RAW image 75A can be generated.

[0135] In addition, in this embodiment, the example image 94 and the correct image 96 are created from a plurality of single images (a plurality of unit images). Therefore, the example image 94 and the correct image 96 can be created more easily than when the example image 94 and the correct image 96 are created without using a plurality of single images. This contributes to reducing the effort required to create the teacher data 92.

[0136] In the teacher data 92 according to the present embodiment, the correct answer image 96 is an image whose resolution has been increased by synthesizing an image representing the first to fourth single images 108A to 108D, an image representing the fifth to eighth single images 110A to 110D, an image representing the ninth to twelfth single images 112A to 112D, and an image representing the thirteenth to sixteenth single images 114A to 114D. The image whose resolution has been increased refers to an image whose resolution has been increased higher than the images ... In other words, the image with increased resolution has more pixels than the image representing the first to fourth single images 108A to 108D, the image representing the fifth to eighth single images 110A to 110D, the image representing the ninth to twelfth single images 112A to 112D, and the image representing the thirteenth to sixteenth single images 114A to 114D. In other words, the image with increased resolution has a higher visual resolution than the image representing the first to fourth single images 108A to 108D, the image representing the fifth to eighth single images 110A to 110D, the image representing the ninth to twelfth single images 112A to 112D, and the image representing the thirteenth to sixteenth single images 114A to 114D. Therefore, by inputting the RAW image 75A to the trained model 106 obtained by performing machine learning using the teacher data 92 configured in this way, an image with a higher resolution than the RAW image 75A can be output.

[0137] In addition, in the teacher data 92 according to the present embodiment, the first to fourth single images 108A to 108D, the fifth to eighth single images 110A to 110D, the ninth to twelfth single images 112A to 112D, and the thirteenth to sixteenth single images 114A to 114D used to create the example image 94 and the correct answer image 96 are images obtained by performing a pixel shift of one pixel and a pixel shift of one-half pixel. Therefore, even without increasing the number of pixels of the photoelectric conversion element 72 of the imaging device 10, by inputting the RAW image 75A to the trained model 106 obtained by machine learning using the example image 94 and the correct answer image 96, an image with a higher resolution than the RAW image 75A can be obtained. In other words, even without increasing the development cost and manufacturing cost of the photoelectric conversion element 72 of the imaging device 10, an image with a higher resolution than the RAW image 75A can be obtained.

[0138] In addition, in the teacher data 92 according to this embodiment, the first to fourth single images 108A to 108D, the fifth to eighth single images 110A to 110D, the ninth to twelfth single images 112A to 112D, and the thirteenth to sixteenth single images 114A to 114D used to create the example image 94 and the correct answer image 96 have pixels P of different colors (here, as an example, pixels P of R, G, and B) regularly arranged (for example, arranged in a Bayer array). In addition, the first to fourth single images 108A to 108D, the fifth to eighth single images 110A to 110D, the ninth to twelfth single images 112A to 112D, and the thirteenth to sixteenth single images 114A to 114D are images pixel-shifted so that pixels P of different colors overlap. For example, they are images pixel-shifted so that R pixels, G pixels, G pixels, and B pixels overlap. Therefore, when a RAW image in which different colors (here, as an example, R, G, and B) are regularly arranged is input to the trained model 106, the trained model 106 can generate and output an image that does not require demosaic processing. As a result, an image with a high degree of reproduction of the colors of the captured subject can be provided to the user of the imaging device 10, as compared to a case in which demosaic processing is performed.

[0139] In addition, in the teacher data 92 according to this embodiment, the example image 94 and the correct image 96 are RAW images. Therefore, by inputting the RAW image 75A to the trained model 106 generated as described above, it is possible to cause the trained model 106 to generate and output an image with higher image quality (high resolution in the above example) than the RAW image 75A.

[0140] In addition, in the teacher data 92 of this embodiment, the first to fourth single images 108A to 108D, the fifth to eighth single images 110A to 110D, the ninth to twelfth single images 112A to 112D, and the thirteenth to sixteenth single images 114A to 114D are each obtained by capturing images from different imaging positions by the image sensor 20 of the imaging device 500, and are images shifted by 1 / 2 pixel. In addition, the example image 94 is an image obtained by combining a first single image 108A having a center 108A1 closest to the center of gravity G1 when the first to fourth single images 108A to 108D are superimposed, a fifth single image 110A having a center 110A1 closest to the center of gravity G2 when the fifth to eighth single images 110A to 110D are superimposed, a ninth single image 112A having a center 112A1 closest to the center of gravity G3 when the ninth to twelfth single images 112A to 112D are superimposed, and a thirteenth single image 114A having a center 114A1 closest to the center of gravity G4 when the thirteenth to sixteenth single images 114A to 114D are superimposed. Therefore, by inputting RAW image 75A to trained model 106, the degree of reproduction of the subject viewed through the image generated and output by trained model 106 can be improved compared to when an image obtained by synthesizing the image having the center farthest from center of gravity G1 among the first to fourth single images 108A to 108D, the image having the center farthest from center of gravity G2 among the fifth to eighth single images 110A to 110D, the image having the center farthest from center of gravity G3 among the ninth to twelfth single images 112A to 112D, and the image having the center farthest from center of gravity G4 among the thirteenth to sixteenth single images 114A to 114D is used as example image 94.

[0141] In the above embodiment, the teacher data 92 is exemplified, but this is merely an example. As an example, as shown in FIG. 9, teacher data 92A may be used instead of the teacher data 92. The teacher data 92A is different from the teacher data 92 in that a correct answer image 96A is applied instead of the correct answer image 96, and an example image 94A is applied instead of the example image 94. In the example shown in FIG. 9, the example image 94A is any one of the first single image 108A, the fifth single image 110A, the ninth single image 112A, and the thirteenth single image 114A. In the example shown in FIG. 9, the correct answer image 96A is an image created by synthesizing (here, as an example, superimposing) the first single image 108A, the fifth single image 110A, the ninth single image 112A, and the thirteenth single image 114A. In the example shown in FIG. 9, the example image 94A is the first single image 108A, the fifth single image 110A, the ninth single image 112A, or the thirteenth single image 114A. The correct image 96A has a higher color reproducibility than the example image 94A. Therefore, by inputting the RAW image 75A to the trained model 106 generated by optimizing the model 98 by machine learning using the teacher data 92A, it is possible to generate and output a RAW format image having a higher color reproducibility than the RAW image 75A for the trained model 106. Note that, although the correct image 96A is illustrated here, the correct image 96 may also be applied. In this case, a high-resolution image can be generated and output for the trained model 106 in the same manner as in the example shown in FIG. 6. The same can be said about the following modified examples.

[0142] In the example shown in Fig. 9, the first single image 108A, the fifth single image 110A, the ninth single image 112A, and the thirteenth single image 114A are shown, but this is merely an example. For example, instead of the first single image 108A, the fifth single image 110A, the ninth single image 112A, and the thirteenth single image 114A, the second single image 108B, the sixth single image 110B, the tenth single image 112B, and the fourteenth single image 114B may be used, or the third single image 108C, the seventh single image 110C, the eleventh single image 112C, and the fifteenth single image 114C may be used, or the fourth single image 108D, the eighth single image 110D, the twelfth single image 112D, and the sixteenth single image 114D may be used. The same can be said about this in the following modified examples.

[0143] In the example shown in FIG. 9, the teacher data 92A is shown, but this is merely an example, and as an example, as shown in FIG. 10, teacher data 92B may be used instead of the teacher data 92A. The teacher data 92B is different from the teacher data 92A in that the example image 94B is used instead of the example image 94A. The example image 94B is an example of the "first image" according to the present disclosure. It is an image based on three or less images among the first single image 108A, the fifth single image 110A, the ninth single image 112A, and the thirteenth single image 114A. In the example shown in FIG. 10, the images based on the first single image 108A, the fifth single image 110A, and the ninth single image 112A are shown as example images 94B. An example of an image based on the first single image 108A, the fifth single image 110A, and the ninth single image 112A is an image created by combining the first single image 108A, the fifth single image 110A, and the ninth single image 112A (here, as an example, by overlaying them in units of pixels P1 whose positions correspond to each other).

[0144] In this way, when a RAW image 75A of the same image quality as the example image 94B is input to the trained model 106, which has been generated by optimizing the model 98 through machine learning based on the teacher data 92B in which images based on three or fewer images out of the first single image 108A, the fifth single image 110A, the ninth single image 112A, and the thirteenth single image 114A are set as the example image 94B, the trained model 106 can be made to generate and output an image of higher image quality than the input RAW image 75A.

[0145] In the example shown in FIG. 9, the teacher data 92A is shown, but this is merely an example, and as an example, as shown in FIG. 11, teacher data 92C may be used instead of the teacher data 92A. The teacher data 92C differs from the teacher data 92A in that a correct answer image 96A (see also FIG. 10) and an example image 94C are applied. The example image 94C is an example of a "second image" according to the present disclosure. The example image 94C is an image obtained by the first thinned image 108Aa, the second thinned image 110Aa, the third thinned image 112Aa, and the fourth thinned image 114Aa. When N is an even number, the first thinned image 108Aa is an image obtained by thinning out the Nth row of the first single image 108A, the second thinned image 110Aa is an image obtained by thinning out the Nth row of the fifth single image 110A, the third thinned image 112Aa is an image obtained by thinning out the Nth row of the ninth single image 112A, and the fourth thinned image 114Aa is an image obtained by thinning out the Nth row of the thirteenth single image 114A.

[0146] In the example shown in FIG. 11, example image 94C is an image created by combining a first thinned image 108Aa, a second thinned image 110Aa, a third thinned image 112Aa, and a fourth thinned image 114Aa (here, as an example, by overlaying them in units of pixels P1 whose positions correspond to each other).

[0147] Here, the thinning pattern of the Nth row is illustrated, but other thinning patterns (for example, the thinning pattern of the Nth column, or the thinning pattern of the Nth row and Nth column, etc.) may be used. In addition, the first thinned image 108Aa, the second thinned image 110Aa, the third thinned image 112Aa, and the fourth thinned image 114Aa are illustrated, but this is merely an example, and the example image 94C may be an image obtained by synthesizing (for example, superimposing in units of pixels P1 whose positions correspond to each other) three or less images among the first thinned image 108Aa, the second thinned image 110Aa, the third thinned image 112Aa, and the fourth thinned image 114Aa.

[0148] In this way, when a RAW image 75A of the same image quality as the example image 94C (for example, a RAW image 75A in which pixel P1 has been thinned out using a thinning pattern similar to the example shown in FIG. 11) is input to the trained model 106, which has been generated by optimizing the model 98 through machine learning based on the teacher data 92C in which an image created by synthesizing the first thinned image 108Aa, the second thinned image 110Aa, the third thinned image 112Aa, and the fourth thinned image 114Aa is used as the example image 94C, the trained model 106 can generate and output an image of higher image quality than the input RAW image 75A.

[0149] In the example shown in Fig. 9, teacher data 92A is shown, but this is merely an example, and teacher data 92D may be used instead of teacher data 92A, as shown in Fig. 12. Teacher data 92D differs from teacher data 92A in that a correct answer image 96B and a sample image 94D are used.

[0150] In the above embodiment, the photoelectric conversion element 72 is exemplified as a photoelectric conversion element having no function of detecting a phase difference, but this is merely an example. For example, as shown in FIG. 12, the photoelectric conversion element 72 may be a photoelectric conversion element having a function of detecting a phase difference, that is, a photoelectric conversion element including a phase difference pixel P2 and a non-phase difference pixel P3. The phase difference pixel P2 is a pixel that detects a phase difference. In addition, the phase difference pixel P2 is a G pixel. The non-phase difference pixel P3 is the same pixel as the pixel P described above (i.e., a pixel that does not detect a phase difference). In the photoelectric conversion element 72, the phase difference pixel P2 and the non-phase difference pixel P3 are regularly arranged while maintaining the arrangement pattern of the R pixel, G pixel, and B pixel described in the above embodiment.

[0151] The first single image 108A, the fifth single image 110A, the ninth single image 112A, and the thirteenth single image 114A obtained by performing imaging by the image sensor 20 having the photoelectric conversion element 72 configured in this manner include pixels P2a and P3a. The pixel P2a is a pixel corresponding to the pixel P2, and the pixel P3a is a pixel corresponding to the non-phase difference pixel P3. The first single image 108A, the fifth single image 110A, the ninth single image 112A, and the thirteenth single image 114A are images pixel-shifted to a position where the pixel P3a overlaps with the pixel P2a when the first single image 108A, the fifth single image 110A, the ninth single image 112A, and the thirteenth single image 114A are synthesized (for example, when the first single image 108A, the fifth single image 110A, the ninth single image 112A, and the thirteenth single image 114A are superimposed).

[0152] The correct image 96B is an image created by synthesizing the first single image 108A, the fifth single image 110A, the ninth single image 112A, and the thirteenth single image 114A (for example, by superimposing the first single image 108A, the fifth single image 110A, the ninth single image 112A, and the thirteenth single image 114A). In this case, the pixel P2a and the pixel P3a are superimposed between the first single image 108A, the fifth single image 110A, the ninth single image 112A, and the thirteenth single image 114A. Moreover, the R pixel, the G pixel, and the B pixel are superimposed.

[0153] The example image 94D is any one of the first single image 108A, the fifth single image 110A, the ninth single image 112A, and the thirteenth single image 114A. The example image 94D may be an image obtained by synthesizing (for example, superimposing in units of pixels P1 whose positions correspond to each other) three or less images among the first single image 108A, the fifth single image 110A, the ninth single image 112A, and the thirteenth single image 114A.

[0154] By performing machine learning based on the teacher data 92D configured as shown in Figure 12, the trained model 106 is optimized, so that even if the photoelectric conversion element 72 of the image sensor 20 mounted on the imaging device 10 is a photoelectric conversion element in which the phase difference pixel P2 and the non-phase difference pixel P3 are arranged, when a RAW image 75A of the same image quality as the example image 94D (for example, a RAW image 75A in which the pixel P2a and the pixel P3a are mixed) is input to the trained model 106, the trained model 106 can generate and output an image of higher image quality than the input RAW image 75A.

[0155] As an example, as shown in FIG. 13, the sample image 94 may include a non-focused region 126 (i.e., an out-of-focus region) and a focused region 128 (i.e., an in-focus region). When using such a sample image 94, the correct answer image 96 also includes a position corresponding to the non-focused region 126 in the sample image 94, and includes a focused region 132 at a position corresponding to the focused region 128 in the sample image 94. The non-focused region 130 and the focused region 132 are made to have different degrees of image quality improvement. In the example shown in FIG. 13, the non-focused region 130 is made to have a smaller degree of image quality improvement than the focused region 132.

[0156] By performing machine learning based on teacher data 92 including example image 94 and correct image 96 configured in this manner, the trained model 106 is optimized, so that when a RAW image 75A including out-of-focus and in-focus areas is input to the trained model 106, the trained model 106 can generate and output an image in which the image quality of the in-focus areas is better than the image quality of the out-of-focus areas.

[0157] Note that, here, an example is given in which the degree of image quality improvement in the non-focused region 130 is less than that in the focused region 132, but this is merely one example, and the degree of image quality improvement in the focused region 132 may be less than that in the non-focused region 130.

[0158] As an example, as shown in Fig. 14, an image whose image quality has been improved compared to a single image due to factors affecting image quality (hereinafter also simply referred to as "factors") may be used as a correct answer image 96, and an image whose image quality has been lowered compared to the correct answer image 96 due to the factors may be used as a sample image 94. In the example shown in Fig. 14, the image quality of a focused area 128 of the sample image 94 is lower than the image quality of a focused area 132 of the correct answer image 96 due to the factors. Moreover, the sample image 94 shown in Fig. 14 is an image whose image quality has been lowered compared to the sample image 94 whose image quality has been improved due to the factors being effective, because the factors were not effective.

[0159] Examples of factors include focal length, aperture value, lens characteristics (e.g., inherent characteristics that vary with lens settings (MTF, aberration, and / or shading)), thinning characteristics between pixels P, gradation correction function, gain correction function, and noise reduction function. In other words, the correct image 96 is an image created from a plurality of single images (a plurality of unit images) obtained by the focal length, aperture value, lens characteristics, thinning characteristics between pixels P, gradation correction function, gain correction function, and noise reduction function working to achieve image quality at a certain level or higher. The correct image 96 is created from a plurality of single images (a plurality of unit images) whose image quality has been improved by the focal length, aperture value, lens characteristics, thinning characteristics between pixels P, gradation correction function, gain correction function, and noise reduction function.

[0160] Example image 94 shown in Figure 14 is an image created from multiple single images that have been affected by degraded image quality due to focal length, degraded image quality due to aperture value, degraded image quality due to lens characteristics, degraded image quality due to thinning characteristics between pixels P, degraded image quality due to not using a gradation correction function or lowering the level of use of the gradation correction function below a certain level, degraded image quality due to not using a gain correction function or lowering the level of use of the gradation correction function below a certain level, and degraded image quality due to not using a noise reduction function or lowering the level of use of the gradation correction function below a certain level. In other words, the example image 94 shown in Figure 14 can be said to be an image of lower image quality than the example image 94 created from multiple single images that have been affected by the high image quality due to the focal length, the high image quality due to the aperture value, the high image quality due to the lens characteristics, the high image quality due to the thinning characteristics between pixels P, the high image quality due to the use of the gradation correction function or raising the usage level of the gradation correction function above a certain level, the high image quality due to the use of the gain correction function or raising the usage level of the gradation correction function above a certain level, and the high image quality due to the use of the noise reduction function or raising the usage level of the gradation correction function above a certain level.

[0161] Moreover, the example image 94 is an image of image quality assumed as the image quality of the RAW image 75A generated by the imaging device 10 (or an image obtained by performing image processing on the RAW image 75A by the image processing engine 12). For example, the minimum level setting values ​​of the focal length, aperture value, lens characteristics, thinning characteristics between pixels P, gradation correction function, gain correction function, and noise reduction function listed as factors (i.e., setting values ​​for making the image quality of the example image 94 the minimum level) are the same as the focal length, aperture value, lens characteristics, thinning characteristics between pixels P, gradation correction function, gain correction function, and noise reduction function used for the imaging device 10 (e.g., setting values ​​for making the image quality of the RAW image 75A the minimum level). Here, the focal length, the aperture value, the lens characteristics, the thinning characteristics between pixels P, the gradation correction function, the gain correction function, and the noise reduction function are exemplified, but if the image quality adjustment is performed on the RAW image 75A by the imaging device 10 through color space conversion processing, luminance filter processing, color difference processing, and / or resizing processing, etc., the image quality adjustment performed on the RAW image 75A by the imaging device 10, i.e., color space conversion processing, luminance filter processing, color difference processing, and / or resizing processing, etc., may also be performed on the example image 94. The image quality adjustment performed on the RAW image 75A by the imaging device 10, i.e., color space conversion processing, luminance filter processing, color difference processing, and / or resizing processing, etc., is included in the concept of factors.

[0162] In the example shown in FIG. 14, the sample image 94 is a single image having false color and false resolution previously assumed as the false color and false resolution of the RAW image 75A generated by the imaging device 10 (for example, an image having the greatest amount of false color and false resolution identified by previously quantifying the amount of false color and false resolution by visual inspection and / or computer simulation, etc.), or an image generated based on a single image having false color and false resolution previously assumed as the false color and false resolution of the RAW image 75A generated by the imaging device 10. For example, an image generated based on a single image having false color and false resolution refers to an image obtained by processing a single image having false color and false resolution. Examples of an image obtained by processing a single image having false color and false resolution include an image obtained by thinning out a single image having false color and false resolution, or an image file obtained by developing a single image having false color and false resolution. In the example shown in FIG. 14, the in-focus area 128 includes an image area of ​​false color and an image area of ​​false resolution. However, this is merely one example, and image regions of false color and / or false resolution may be included in image regions other than the in-focus region 128. Note that false color and false resolution are included in the concept of factors.

[0163] 14 is configured based on the training data 92 including the example image 94 and the correct image 96, the trained model 106 is optimized by performing machine learning, and when a RAW image 75A whose image quality has been degraded by a factor is input to the trained model 106, the trained model 106 can generate and output an image with higher image quality than the input RAW image 75A. Furthermore, compared to the case where a single image with the least amount of false color and false resolution is used as the example image 94, when a RAW image 75A having a focus area including an image area of ​​false color and an image area of ​​false resolution is input to the trained model 106, the trained model 106 can generate and output an image in which false color and false resolution are suppressed in an area corresponding to the focus area of ​​the RAW image 75A compared to the input RAW image 75A.

[0164] In addition, in FIG. 14, as an example of factors, focal length, aperture value, lens characteristics, thinning characteristics between pixels P, gradation correction function, gain correction function, noise reduction function, etc. are illustrated, but this is merely an example. The factor may be one or more of focal length, aperture value, lens characteristics, thinning characteristics between pixels P, gradation correction function, gain correction function, and noise reduction function. In addition, the factor may be one or more of focal length, aperture value, lens characteristics, thinning characteristics between pixels P, gradation correction function, gain correction function, noise reduction function, color space conversion processing, luminance filter processing, color difference processing, resizing processing, false color, and / or spurious resolution. In addition, in FIG. 14, a single image having false color and spurious resolution is illustrated, but this is merely an example, and a single image having false color or spurious resolution may be used. In addition, the example shown in FIG. 14 gives an example of a form in which the image quality of the in-focus region 128 is degraded by a factor, but this is merely one example, and the image area other than the in-focus region 128 may also be degraded by a factor.

[0165] Furthermore, multiple trained models 106 generated by optimizing a model 98 by performing machine learning on the model 98 based on teacher data 92 including example images 94 and correct images 96 reflecting one type of factor different from each other and / or multiple types of factors different from each other may be installed in the imaging device 10. In this case, the RAW image 75A may be input to one or more trained models 106 selected from the multiple trained models 106 in response to instructions and / or various conditions given by a user or the like, and the trained model 106 may generate and output a RAW image 75A with improved image quality.

[0166] In this way, machine learning based on training data 92 including example images 94 and correct images 96 reflecting one different type of factor and / or multiple different types of factors is performed on model 98, and one or more trained models 106 generated by optimizing model 98 are installed in imaging device 10. Even if imaging device 10 does not have a function for performing the above-mentioned image quality adjustment, by inputting RAW image 75A to trained model 106, it is possible to obtain an image with higher image quality than RAW image 75A input to trained model 106.

[0167] In the above embodiment, an example of a form in which the trained model 106 mounted on the imaging device 10 generates and outputs an image in RAW format is given, but this is merely an example. For example, the trained model 106 mounted on the imaging device 10 may generate and output an image based on the RAW image 75A (for example, an image file in a default file format). To achieve this, in the example shown in FIG. 15, training data 92E is used instead of the training data 92. The training data 92E includes an example image 94E and a correct answer image 96C. The example image 94E is different from the example image 94 shown in FIG. 14 in that it is a JPEG file including an image in RGB format. The correct answer image 96C is different from the correct answer image 96 shown in FIG. 14 in that it is a JPEG file including an image in RGB format. Here, an image in RGB format is exemplified, but this is merely an example, and an image in YCbCr format may be used.

[0168] 15, the trained model 106 is optimized by performing machine learning based on the teacher data 92E configured as shown in Fig. 15, so that when an image in RGB format is input, the trained model 106 can generate and output an image in RGB format with higher image quality than the input image in RGB format. Also, when a JPEG file is input to the trained model 106, the trained model 106 can generate and output a JPEG file with higher image quality than the input JPEG file.

[0169] In addition, in the example shown in FIG. 15, a JPEG file is illustrated, but this is merely one example, and the image file format may be TIFF, JPEG XR, MPEG, AVI, or the like.

[0170] In the above embodiment, an example was given in which the first single image 108A is used to create the example image 94 because the center 108A1 is the closest to the center of gravity G1 among the centers 108A1 to 108D1 (see FIG. 5). However, this is merely an example, and three or fewer images among the first to fourth single images 108A to 108D may be used to create the example image 94.

[0171] In the above embodiment, an example was given in which the fifth single image 110A is used to create the example image 94 because the center 110A1 is the closest to the center of gravity G2 among the centers 110A1 to 110D1 (see Figure 5), but this is merely one example, and three or fewer images among the fifth to eighth single images 110A to 110D may be used to create the example image 94.

[0172] In the above embodiment, an example was given in which the ninth single image 112A is used to create the example image 94 because the center 112A1 is the closest to the center of gravity G3 among the centers 112A1 to 112D1 (see Figure 5), but this is merely one example, and it is sufficient that three or fewer images among the ninth to twelfth single images 112A to 112D are used to create the example image 94.

[0173] In the above embodiment, an example was given in which the 13th single image 114A is used to create the example image 94 because the center 114A1 is the closest to the center of gravity G4 among the centers 114A1 to 114D1 (see Figure 5), but this is merely one example, and it is sufficient that three or fewer images among the 13th to 16th single images 114A to 114D are used to create the example image 94.

[0174] In the above embodiment, a single image in which R pixels, G pixels, and B pixels are regularly arranged is exemplified, but the single image may also be a monochromatic image (for example, an R image consisting of only multiple R images, a G image consisting of only multiple G images, or a B image consisting of only multiple B images).

[0175] In the above embodiment, an example in which the image quality improvement process is performed by the processor 62 of the image processing engine 12 included in the imaging device 10 has been described, but the present disclosure is not limited thereto, and the device performing the image quality improvement process may be, for example, a device provided outside the imaging device 10, such as a server. For example, the server may be realized by cloud computing. Also, the server may be realized by network computing such as fog computing, edge computing, or grid computing. Here, a server is illustrated, but this is merely an example, and at least one personal computer or the like may be used instead of the server.

[0176] In the above embodiment, an example in which the image quality improvement program 124 is stored in the storage 64 has been described, but the present disclosure is not limited to this. For example, the image quality improvement program 124 may be stored in a portable computer-readable non-transitory storage medium such as an SSD or a USB memory. The image quality improvement program 124 stored in the non-transitory storage medium is installed in the image processing engine 12 of the imaging device 10. The processor 62 executes image quality improvement processing in accordance with the image quality improvement program 124.

[0177] In addition, the image quality improvement program 124 may be stored in a storage device such as another computer or server device connected to the imaging device 10 via a network, and the image quality improvement program 124 may be downloaded in response to a request from the imaging device 10 and installed in the image processing engine 12.

[0178] It should be noted that it is not necessary to store the entire image quality improvement program 124 in a storage device such as another computer or server device connected to the imaging device 10, or in the storage 64, and it is acceptable to store only a part of the image quality improvement program 124. Note that although the image quality improvement program 124 is mentioned here, the same can be said about the learning program 90.

[0179] In addition, although the image processing engine 12 is built into the imaging device 10 shown in FIGS. 1 and 2, the present disclosure is not limited to this. For example, the image processing engine 12 may be provided outside the imaging device 10.

[0180] In the above embodiment, the image processing engine 12 is exemplified, but the present disclosure is not limited thereto, and a device including an ASIC, an FPGA, and / or a PLD may be applied instead of the image processing engine 12. Also, instead of the image processing engine 12, a combination of a hardware configuration and a software configuration may be used.

[0181] As the hardware resource for executing the image quality improvement processing and / or learning processing described in the above embodiment, various processors as shown below can be used. An example of the processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing the image quality improvement processing and / or learning processing by executing software, i.e., a program. Another example of the processor is a dedicated electric circuit, which is a processor having a circuit configuration designed exclusively for executing a specific process, such as an FPGA, a PLD, or an ASIC. Each processor has a built-in or connected memory, and each processor uses the memory to execute the image quality improvement processing and / or learning processing.

[0182] The hardware resource for executing the image quality improvement process and / or the learning process may be one of these various processors, or may be a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource for executing the image quality improvement process and / or the learning process may be a single processor.

[0183] As an example of a configuration using one processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes image quality improvement processing and / or learning processing. Second, there is a configuration in which a processor is used that realizes the functions of the entire system including multiple hardware resources that execute image quality improvement processing and / or learning processing in a single IC chip, as typified by SoC. In this way, the image quality improvement processing and / or learning processing is realized using one or more of the above various processors as hardware resources.

[0184] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements. Also, the image quality improvement process and / or learning process described above are merely examples. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the order of processing may be changed, without departing from the spirit of the invention.

[0185] The above description and illustrations are detailed descriptions of the parts related to the present disclosure and are merely an example of the present disclosure. For example, the above description of the configuration, function, action, and effect is an example of the configuration, function, action, and effect of the parts related to the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above description and illustrations within the scope of the gist of the present disclosure. In addition, in order to avoid confusion and to facilitate understanding of the parts related to the present disclosure, the above description and illustrations omit explanations of technical common sense that do not require explanation in order to enable the implementation of the present disclosure.

[0186] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, and standard was specifically and individually indicated to be incorporated by reference.

[0187] The following supplementary notes are further disclosed regarding the above embodiment. (Appendix 1) Training data used in machine learning of a model, A ground truth image obtained by synthesizing a plurality of single images; and an example image representative of the plurality of single images. Teacher data.

[0188] (Appendix 2) A trained model generated by optimizing the model by performing the machine learning on the model using the training data described in Appendix 1.

[0189] (Appendix 3) Inputting an image captured by the image sensor into the trained model described in Appendix 2; and A program (e.g., image quality improvement program 124) for causing a computer (e.g., image processing engine 12) to execute image quality improvement processing including obtaining an inference result output from the trained model in response to input of the captured image.

[0190] (Appendix 4) A program (e.g., a learning program 90) that causes a computer (e.g., a learning device 79) to execute a learning process that generates a trained model by performing machine learning on a model using teacher data including a correct image and an example image, The correct image is an image obtained by synthesizing a plurality of single images, the example image is a representative image of the plurality of single images; The learning process is as follows: Inputting the example image into the model; The model outputs an evaluation target image in response to the input of the example image; and and optimizing the model based on a comparison result between the evaluation target image and the ground truth image. program.

[0191] (Appendix 5) Inputting an image captured by the image sensor into the trained model described in Appendix 2; and A computer-readable non-transitory storage medium storing a program (e.g., image quality improvement program 124) for causing a computer (e.g., image processing engine 12) to execute image quality improvement processing including obtaining an inference result output from the trained model in response to input of the captured image.

[0192] (Appendix 6) A computer-readable non-transitory storage medium storing a program (e.g., a learning program 90) that causes a computer (e.g., a learning device 79) to execute a learning process that generates a trained model by performing machine learning on a model using teacher data including a correct image and an example image, The correct image is an image obtained by synthesizing a plurality of single images, the example image is a representative image of the plurality of single images; The learning process is as follows: Inputting the example image into the model; The model outputs an evaluation target image in response to the input of the example image; and and optimizing the model based on a comparison result between the evaluation target image and the ground truth image. Non-transitory storage media. [Explanation of symbols]

[0193] 10,500 Imaging Device 12 Image Processing Engine 16 Imaging device body 18 Interchangeable Lenses 18A Focus Ring 20 Image Sensor 22 Release button 24 Dial 26 Instruction keys 28 Display 30 Touch Panel 32 Touch panel display 36 Control device 37 First Actuator 38 Second Actuator 39 Third Actuator 40 Imaging Lens 40A Objective Lens 40B Focus Lens 40C Zoom Lens 40D aperture 40D1 aperture 40D2 aperture blades 44 System Controller 46 Image Memory 48 UI Devices 50 External I / F 52 Communication I / F 54 Photoelectric conversion element driver 62,80 Processor 64,82 Storage 66,84 Memory 68 Bus 70 Input / Output Interface 72 Photoelectric conversion element 72A Photosensitive surface 74 A / D Converter 75A RAW image 75A1 High Resolution Image 75B Image File 76 Reception Device 78 Hard Key Section 79 Learning Device 90 Study Programs 92, 92A, 92B, 92C, 92D, 92E Teacher data 94, 94A, 94B, 94C, 94D, 94E Example images 96, 96A, 96B, 96C Correct image 98 Models 100 Comparison Images 102 error 104 Adjustment value 106 trained models 108A 1st single image 108A1,110A1,112A1,114A1,108B1,110B1,112B1,114B1,108C1,110C1,112C1,114C1,108D1,110D1,112D1,114D1 Center 108B Second single image 108C 3rd single image 108D 4th single image 110A 5th single image 110B 6th single image 110C 7th single image 110D 8th single image 112A 9th single image 112B 10th single image 112C 11th single image 112D 12th single image 114A 13th Single Image 114B 14th single image 114C 15th single image 114D 16th single image 116 First fine-grained image 118 Second fine-grained image 120 3rd refined image 122 4th fine-grained image 124 High-definition program 126,130 out of focus area 128,132 Focus area 502 Pixel shifter G1,G2,G3,G4 Center of gravity OA optical axis P, P1, P2a pixels P2 Phase detection pixel P3 Non-phase detection pixel

Claims

1. Training data used in machine learning of a model, A ground truth image obtained by synthesizing a plurality of single images; and an example image representative of the plurality of single images. Teacher data.

2. The correct image is an image having a high resolution obtained by synthesizing the plurality of single images. The training data according to claim 1 .

3. The correct image is an image with a higher resolution than the example image. The training data according to claim 1 .

4. The correct image has a larger number of pixels than the example image. The training data according to claim 1 .

5. The correct image is an image having a higher visual resolution than the example image. The training data according to claim 1 .

6. Each of the plurality of single images is a pixel-shifted image. The training data according to claim 2.

7. Each of the plurality of single images is a half-pixel shifted image. The training data according to claim 6.

8. each of the plurality of single images is a pixel-shifted image; The correct image is an image with high image quality obtained by synthesizing the plurality of single images. The training data according to claim 1 .

9. The plurality of single images have pixels of different colors regularly arranged therein; Each of the plurality of single images is a pixel-shifted image where the pixels of the different colors overlap. The training data according to claim 8.

10. The plurality of single images are obtained by capturing images using a first image sensor including a phase difference pixel and a non-phase difference pixel; Each of the plurality of single images is an image pixel-shifted to a position where a pixel corresponding to the non-phase difference pixel overlaps with a pixel corresponding to the phase difference pixel. The training data according to claim 5.

11. the correct image is an image whose image quality is improved from that of the single image by a factor affecting image quality, The example image is an image whose image quality has been degraded by the factor compared to the correct image. The training data according to claim 1 .

12. The factors are a focal length, an aperture value, a lens characteristic, a pixel thinning characteristic, a gradation correction function, a gain correction function, and / or a noise reduction function. The training data according to claim 11.

13. The target image and the example image include a focused region and a non-focused region, The correct image is an image in which the degree of image quality improvement differs between the focused area and the unfocused area. The training data according to claim 1 .

14. The correct image is an image in which the degree of image quality improvement in the out-of-focus area is smaller than that in the in-focus area. The training data according to claim 13.

15. The correct image and the example image are RAW images. The training data according to any one of claims 1 to 14.

16. The correct image and the example image are images based on RAW images. The training data according to claim 1 .

17. The correct image and the example image are images in RGB format or images in YCbCr format. The training data according to claim 1 .

18. The example image is a first image based on a number of the single images that is less than the number of the single images in the plurality of single images. The training data according to claim 1 .

19. The example image is a second image obtained by thinning out pixels of the single images, the number of which is equal to or less than the number of the single images among the plurality of single images. The training data according to claim 1 .

20. Each of the plurality of single images is obtained by capturing an image from a different imaging position by a second image sensor, and is an image shifted by 1 / 2 pixel; The example image is a single image among the plurality of single images that has a center closest to the center of gravity when the plurality of single images are superimposed. The training data according to claim 1 .

21. The example image is a single image having false color and / or false resolution among the plurality of single images, or an image generated based on a single image having false color and / or false resolution among the plurality of single images. The training data according to claim 1 .

22. A trained model generated by optimizing the model by performing the machine learning on the model using the training data according to any one of claims 1 to 21.

23. A first processor; a third image sensor; The first processor, A captured image obtained by capturing an image by the third image sensor is input to the trained model according to claim 22; An inference result output from the trained model is obtained in response to the input of the captured image. Imaging device.

24. A second processor is provided, The second processor optimizes the model by performing the machine learning on the model using the teacher data according to any one of claims 1 to 21. Learning device.

25. A method for creating training data used in machine learning of a model, comprising the steps of: The training data includes a correct answer image and a sample image, Creating the ground truth image by combining a plurality of single images; and creating an image representative of said plurality of single images as said example image. How to create training data.

26. A method for generating a trained model, which generates a trained model by performing machine learning on a model using teacher data including a correct image and an example image, The correct image is an image obtained by synthesizing a plurality of single images, the example image is a representative image of the plurality of single images; inputting said example images into said model; The model outputs an evaluation target image in response to the input of the example image; and and optimizing the model based on a comparison result between the evaluation target image and the correct answer image. How to generate a trained model.

Images