Training data, learned model, imaging device, learning device, method for creating training data, and method for generating learned model

Abstract

The invention provides training data, a learned model, an imaging device, a learning device, a method for creating the training data, and a method for generating the learned model, wherein the training data enables the learned model to generate an image with higher image quality than an input image. The training data is used for machine learning of the model. The training data includes: a correct answer image formed by combining a plurality of single images; and example images representing a plurality of single images.

Description

Technical Field

[0001] The present invention relates to training data, a learned model, a camera device, a learning device, a method for creating training data, and a method for generating a learned model. Background Art

[0002] Patent document 1 discloses an image processing device, which includes: an image receiving unit that receives a first image of the same object and a second image of a different type from the first image; a first image quality improvement unit that improves the image quality of the first image received by the receiving unit using an image quality improvement function learned in a manner to improve the image quality of the first image; and a second image quality improvement unit that improves the image quality of the second image using the first image quality improvement image and the second image generated by the first image quality improvement unit. In addition, the image processing device described in Patent document 1 further includes a learning unit that learns the image quality improvement function.

[0003] Patent document 2 discloses a computer-implemented method for improving the image resolution of a digital image. The computer-implemented method described in Patent document 2 includes: a step of performing bicubic upsampling of a digital image to generate a base high-resolution (HR) image; a step of converting the digital image from a red-green-blue (RGB) color space to a luminance (Y), chroma blue difference (Cb), and chroma red difference (Cr) (YCbCr) color space to generate a low-resolution (LR) residual image; a step of converting the LR residual image into a plurality of HR residual sub-images corresponding to the digital image using a plurality of convolutional layers of a neural network model; and a step of generating an HR image corresponding to the digital image using the base HR image and the plurality of HR residual sub-images.

[0004] Furthermore, the computer-implemented method described in Patent Document 2 also includes the step of training a neural network model using a plurality of training image pairs, in which each of the plurality of training image pairs includes an LR image and a plurality of HR residual sub-images, wherein the LR image corresponds to the training image and has degraded image quality and constitutes an input to the neural network model, and the plurality of HR residual sub-images correspond to the training image and constitute a target output of the neural network model.

[0005] Patent document 3 discloses a super-resolution processing device that performs super-resolution processing on an image of a second resolution to create an image of a third resolution that is higher than the second resolution. The super-resolution processing device described in Patent document 3 includes a first-layer filter acquisition unit, a reduced image creation unit, a candidate image creation unit, and a super-resolution processing unit. The first-layer filter acquisition unit acquires, for a plurality of learning subsets of the first layer that are subsets of a learning set that includes an image of a third resolution, a first filter for super-resolution processing of an image of a second resolution into an image of a third resolution and a second filter for super-resolution processing of an image of a first resolution that is lower in resolution than the second resolution into an image of a second resolution, which are machine-learned using the learning subsets. The reduced image creation unit creates a reduced image of the first resolution based on an input image of the second resolution. The candidate image creation unit performs super-resolution processing on the reduced image of the first resolution using each of the second filters acquired by the first-layer filter acquisition unit to create a candidate image of the second resolution. The super-resolution processing unit creates a super-resolution image of a third resolution by performing super-resolution processing on the input image of the second resolution using a first filter corresponding to a second filter that minimizes the difference between the candidate image of the second resolution created by the candidate image creation unit and the input image of the second resolution.

[0006] Patent Document 1: Japanese Patent Application Publication No. 2022-096484

[0007] Patent Document 2: Japanese Patent Application No. 2022-536807

[0008] Patent Document 3: Japanese Patent Application Publication No. 2018-151747 Summary of the invention

[0009] One embodiment of the present invention provides training data, a learned model, a camera device, a learning device, a method for creating training data, and a method for generating a learned model that enable a learned model to generate an image with higher quality than an input image.

[0010] The first method involved in the present invention is training data, which is used for machine learning of a model and includes: a correct answer image, which is synthesized from multiple single images; and an example question image, which represents multiple single images.

[0011] According to a second aspect of the present invention, in the training data according to the first aspect, the correct answer image is an image obtained by synthesizing a plurality of single images to obtain a higher resolution.

[0012] According to a third aspect of the present invention, in the training data according to the first aspect or the second aspect, the correct answer image is a high-resolution image of the ratio question image.

[0013] According to a fourth aspect of the present invention, in the training data according to the first aspect or the second aspect, the correct answer image is an image having a larger number of pixels than the question image.

[0014] According to a fifth aspect of the present invention, in the training data according to the first aspect or the second aspect, the correct answer image is an image having a higher visual resolution than the question image.

[0015] According to a sixth aspect of the present invention, in the training data according to the second aspect, each of the plurality of single images is a pixel-shifted image.

[0016] According to a seventh aspect of the present invention, in the training data according to the sixth aspect, each of the plurality of single images is an image after 1 / 2 pixel shift.

[0017] According to an eighth aspect of the present invention, in the training data according to the first aspect, each of the plurality of single images is an image after pixel shifting, and the correct answer image is an image with high image quality obtained by synthesizing the plurality of single images.

[0018] According to a ninth aspect of the present invention, in the training data according to the eighth aspect, the plurality of single images are regularly arranged with pixels of different colors, and the plurality of single images are images in which pixels are shifted to positions where pixels of different colors overlap.

[0019] In the tenth method involved in the present invention, in the training data involved in the fifth method, a plurality of single images are obtained by photographing by a first image sensor including phase difference pixels and non-phase difference pixels, and the plurality of single images are images in which pixels are shifted to positions where pixels equivalent to non-phase difference pixels overlap pixels equivalent to phase difference pixels.

[0020] In the 11th mode involved in the present invention, in the training data involved in any one of the 1st to 10th modes, the correct answer image is an image whose image quality is improved compared with a single image due to factors affecting image quality, and the example image is an image whose image quality is reduced compared with the correct answer image due to factors.

[0021] In the twelfth mode of the present invention, in the training data involved in the first mode, the factors are focal length, aperture value, lens characteristics, pixel spacing elimination characteristics, grayscale correction function, gain correction function and / or noise reduction function.

[0022] The 13th method involved in the present invention is that in the training data involved in any one of the 1st to 12th methods, the correct answer image and the example image include a focus area and a non-focus area, and the correct answer image is an image with different degrees of high image quality in the focus area and the non-focus area.

[0023] According to a fourteenth aspect of the present invention, in the training data according to the thirteenth aspect, the correct answer image is an image in which the degree of image quality improvement in the out-of-focus area is smaller than that in the in-focus area.

[0024] According to a fifteenth aspect of the present invention, in the training data according to any one of the first to fourteenth aspects, the correct answer image and the example question image are RAW images.

[0025] According to a sixteenth aspect of the present invention, in the training data according to any one of the first to fourteenth aspects, the correct answer image and the example question image are images based on RAW images.

[0026] According to a seventeenth aspect of the present invention, in the training data according to any one of the first to sixteenth aspects, the correct answer image and the example question image are images in RGB format or images in YCbCr format.

[0027] According to an eighteenth aspect of the present invention, in the training data according to any one of the first to seventeenth aspects, the example image is a first image based on a single image whose number is less than the number of the multiple single images.

[0028] According to the 19th mode of the present invention, in the training data according to any one of the 1st to 18th modes, the example image is a second image obtained by thinning out pixels of a number of single images less than the number of single images among a plurality of single images.

[0029] The 20th method involved in the present invention is that in the training data involved in any one of the first to nineteenth methods, the multiple single images are respectively obtained by photographing by the second image sensor from different camera positions and shifted by 1 / 2 pixels, and the example image is a single image having a center closest to the center of gravity when the multiple single images are overlapped.

[0030] In the 21st mode involved in the present invention, in the training data involved in any one of the first to 20th modes, the example image is a single image with false color and / or false resolution among multiple single images, or an image generated based on a single image with false color and / or false resolution among multiple single images.

[0031] The 22nd aspect of the present invention is a learned model generated by optimizing the model by performing machine learning on the model using the training data involved in any one of the 1st to 21st aspects.

[0032] The 23rd method involved in the present invention is a camera device, which comprises: a first processor; and a third image sensor, the first processor performs the following processing: inputting a camera image obtained by taking a picture by the third image sensor into the learned model involved in the 22nd method; and obtaining an inference result output from the learned model based on the input of the camera image.

[0033] A 24th aspect of the present invention is a learning device comprising a second processor configured to optimize a model by performing machine learning on the model using training data according to any one of the first to 21st aspects.

[0034] The 25th method involved in the present invention is a method for creating training data, which is used for machine learning of a model. The training data includes a correct answer image and an example image. The method for creating training data includes the following steps: creating a correct answer image by synthesizing multiple single images; and creating an image representing multiple single images as an example image.

[0035] The 26th method involved in the present invention is a method for generating a learned model, which generates the learned model by performing machine learning on the model using training data including a correct answer image and an example question image, wherein the correct answer image is an image synthesized from multiple single images, and the example question image is an image representing multiple single images, and the method for generating the learned model includes the following steps: inputting an example question image into the model; outputting an evaluation object image from the model based on the input of the example question image; and optimizing the model based on the comparison result between the evaluation object image and the correct answer image. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a schematic structural diagram showing an example of the overall structure of the imaging device.

[0037] Figure 2 This is a schematic configuration diagram showing an example of the hardware configuration of the optical system and the electrical system of the imaging device.

[0038] Figure 3 This is a schematic configuration diagram showing an example of the configuration of a learning device.

[0039] Figure 4 This is a conceptual diagram showing an example of a method for creating a correct answer image.

[0040] Figure 5 This is a conceptual diagram showing an example of a method of creating an example image.

[0041] Figure 6 This is a block diagram showing an example of the function of the image processing engine.

[0042] Figure 7This is a flowchart showing an example of the flow of the learning process performed by the learning device.

[0043] Figure 8 This is a flowchart showing an example of the flow of image quality improvement processing performed by the imaging device.

[0044] Fig. 9 This is a conceptual diagram showing a first variation of the method for creating training data.

[0045] Fig.10 This is a conceptual diagram showing a second variation of the method for creating training data.

[0046] Fig.11 This is a conceptual diagram showing a third variation of the method for creating training data.

[0047] Fig.12 This is a conceptual diagram showing a fourth variation of the method for creating training data.

[0048] Fig.13 This is a conceptual diagram showing an example of the structure of training data for improving the image quality of a focus area compared to a non-focus area.

[0049] Fig.14 This is a conceptual diagram showing an example of the structure of training data including example question images whose image quality is reduced by factors and have false colors and false resolutions, and correct answer images whose image quality is increased by ratio question images.

[0050] Fig.15 This is a conceptual diagram showing an example of a format in which both the correct answer image and the example question image are JPEG files having an RGB format. DETAILED DESCRIPTION

[0051] Hereinafter, an example of an implementation of the training data, the learned model, the camera device, the learning device, the training data creation method, and the learned model generation method involved in the present invention is described with reference to the accompanying drawings.

[0052] First, the words and phrases used in the following description are explained.

[0053] CPU is the abbreviation of “Central Processing Unit”. GPU is the abbreviation of “Graphics Processing Unit”. GPGPU is the abbreviation of “General-purpose computing on graphics processing units”. APU is the abbreviation of “Accelerated Processing Unit”. TPU is the abbreviation of “Tensor processing unit”. NVM is the abbreviation of “Non-volatile memory”. RAM is the abbreviation of “Random Access Memory”. IC is the abbreviation of “Integrated Circuit”. ASIC is the abbreviation of “Application Specific Integrated Circuit”. PLD is the abbreviation of “Programmable Logic Device”. FPGA is the abbreviation of “Field-Programmable Gate Array”. SoC is the abbreviation of “System-on-a-chip”. SSD is the abbreviation of “Solid State Drive”. USB is the abbreviation of “Universal Serial Bus”. HDD is the abbreviation of “Hard Disk Drive”. EEPROM is the abbreviation of “Electrically Erasable and Programmable ReadOnly Memory”. EL is the abbreviation of “Electro-Luminescence”. I / F is the abbreviation of “Interface”. UI is the abbreviation of “User Interface”. FPs is the abbreviation of “frame per second”. MF is the abbreviation of “Manual Focus”. AF is the abbreviation of “Auto Focus”. CMOS is the abbreviation of “Complementary Metal Oxide Semiconductor”.CCD is the abbreviation of “Charge Coupled Device”. AI is the abbreviation of “Artificial Intelligence”. A / D is the abbreviation of “A nalog / Digital”. FIR is the abbreviation of “Finite Impulse Response”. IIR is the abbreviation of “Infinite Impulse Response”. JPEG is the abbreviation of “Joint Photographic Experts Group”. TIFF is the abbreviation of “Tagged Image File Format”. JPEG XR is the abbreviation of “Joint Photographic Experts Group Extended Range”. MPEG is the abbreviation of “Moving Picture Expert Group”. AVI is the abbreviation of “Audio Video Interleave”. MTF is the abbreviation of “Modulation Transfer Function”.

[0054] In the following description, a processor with a symbol (hereinafter referred to as a "processor") can be a physical or virtual computing device, or a combination of multiple physical or virtual computing devices. In addition, a processor can be a computing device, or a combination of multiple computing devices. As an example of a computing device, a CPU, a GPU, a GPGPU, an APU, or a TPU can be cited.

[0055] In the following description, a memory with a symbol is a memory such as a RAM that temporarily stores information and is used as a working memory by a processor.

[0056] In the following description, the storage device with a symbol is one or more non-volatile storage devices that store various programs and various parameters, etc. As an example of a non-volatile storage device, a flash memory, a magnetic disk, or a magnetic tape, etc., can be cited. And as another example of a storage device, a cloud storage device can be cited.

[0057] In the following embodiments, the external I / F with a symbol controls the exchange of various information between multiple devices connected to each other. As an example of an external I / F, a USB interface can be cited. The external I / F can also be applied to a communication I / F including a communication processor and an antenna. The communication I / F controls the communication between multiple computers. As an example of a communication standard applicable to the communication I / F, wireless communication standards including 5G, Wi-Fi (registered trademark) or Bluetooth (registered trademark) can be cited.

[0058] In the following embodiments, "A and / or B" has the same meaning as "at least one of A and B". That is, "A and / or B" means that it can be only A, only B, or a combination of A and B. In addition, in this specification, when three or more items are expressed in association with "and / or", the same concept as "A and / or B" also applies.

[0059] Figure 1 FIG. 1 is a schematic structural diagram showing an example of the overall structure of a camera device. Figure 1 As shown, the imaging device 10 is a device for photographing a subject, and includes an image processing engine 12, an imaging device body 16, and an interchangeable lens 18. Here, the imaging device 10 is an example of the "imaging device" according to the present invention.

[0060] The image processing engine 12 is built into the camera body 16, and controls the entire camera 10. The interchangeable lens 18 is replaceably mounted on the camera body 16. The interchangeable lens 18 is provided with a focus ring 18A. The focus ring 18A is operated by a user of the camera 10 (hereinafter, simply referred to as a "user") or the like when the user manually adjusts the focus of a subject through the camera 10.

[0061] exist Figure 1 In the example shown, a lens-interchangeable digital camera is shown as an example of the imaging device 10. However, this is only an example, and a lens-fixed digital camera may be used, or a digital camera built into various electronic devices such as a smart device, a wearable terminal, an endoscope device, a cell observation device, an ophthalmic observation device, or a surgical microscope may be used.

[0062] The camera body 16 is provided with an image sensor 20. The image sensor 20 is an example of the "third image sensor" involved in the present invention. The image sensor 20 is a CMOS image sensor. The image sensor 20 captures a camera range including at least one subject. When the interchangeable lens 18 is mounted on the camera body 16, subject light representing the subject is transmitted through the interchangeable lens 18 and formed on the image sensor 20, and image data representing the image of the subject is generated by the image sensor 20.

[0063] In the present embodiment, a CMOS image sensor is exemplified as the image sensor 20 , but the present invention is not limited thereto. For example, even if the image sensor 20 is another type of image sensor such as a CCD image sensor, the present invention is also applicable.

[0064] A release button 22 and a dial 24 are provided on the upper surface of the camera body 16. The dial 24 is operated when setting the operation mode of the camera system and the operation mode of the playback system, and by operating the dial 24, the camera mode, the playback mode, and the setting mode are selectively set as the operation mode in the camera 10. The camera mode is an operation mode in which the camera 10 performs shooting. The playback mode is an operation mode in which images (for example, still images and / or moving images) obtained by shooting for recording in the camera mode are played. The setting mode is an operation mode set for the camera 10 when setting various setting values ​​used in control related to shooting.

[0065] The release button 22 functions as a shooting preparation indication unit and a shooting indication unit, and can detect pressing operations in two stages: a shooting preparation indication state and a shooting indication state. The shooting preparation indication state refers to, for example, a state in which the button is pressed from the standby position to an intermediate position (half-pressed position), and the shooting indication state refers to a state in which the button is pressed to a final pressed position (full-pressed position) beyond the intermediate position.

[0066] In addition, hereinafter, the "state of being pressed from the standby position to the half-pressed position" is referred to as the "half-pressed state", and the "state of being pressed from the standby position to the fully-pressed position" is referred to as the "fully-pressed state". Depending on the structure of the imaging device 10, the imaging preparation instruction state may be a state in which the user's finger is in contact with the release button 22, and the imaging instruction state may be a state in which the user's finger is in contact with the release button 22 and then released.

[0067] The back of the camera body 16 is provided with a pointer key 26 and a touch panel display 32. The touch panel display 32 includes a display 28 and a touch panel 30 (see also Figure 2 As an example of the display 28, an EL display (for example, an organic EL display or an inorganic EL display) can be cited. The display 28 may also be another type of display such as a liquid crystal display instead of an EL display.

[0068] The display 28 displays images and / or character information, etc. When the camera device 10 is in the camera mode, the display 28 is used to display a live preview image, which is obtained by shooting for live preview images (i.e., continuous shooting). Here, the "live preview image" refers to a dynamic image for display based on image data obtained by shooting by the image sensor 20. The shooting performed to obtain the live preview image (hereinafter also referred to as "shooting for live preview images") is performed, for example, at a frame rate of 60 fps. 60 fps is only an example, and a frame rate of less than 60 fps may be used, or a frame rate of more than 60 fps may be used.

[0069] When the camera 10 is instructed to shoot a still image via the release button 22, the display 28 is used to display a still image obtained by shooting a still image. In addition, the display 28 is used to display a playback image when the camera 10 is in a playback mode. Furthermore, when the camera 10 is in a setting mode, the display 28 is used to display a menu screen for selecting various menus and a setting screen for setting various setting values ​​used in control related to shooting.

[0070] The touch panel 30 is a transmissive touch panel, which is superimposed on the surface of the display area of ​​the display 28. The touch panel 30 receives instructions from the user by detecting contact with an indicator such as a finger or a stylus. In addition, for the sake of convenience, the above-mentioned "full press state" also includes a state in which the user presses a soft key for starting shooting via the touch panel 30.

[0071] In this embodiment, as an example of the touch screen display 32, an external touch screen display in which the touch panel 30 is superimposed on the surface of the display area of ​​the display 28 is cited, but this is only an example. For example, as the touch screen display 32, an external or internal touch screen display may also be applied.

[0072] The instruction key 26 accepts various instructions. Here, the “various instructions” include, for example, instructions for displaying a menu screen, instructions for selecting one or more menus, instructions for confirming selected content, instructions for deleting selected content, various instructions such as zooming in, zooming out, and frame forwarding, etc. Moreover, these instructions can also be performed through the touch panel 30.

[0073] Figure 2 FIG. 1 is a schematic diagram showing an example of the hardware configuration of the optical system and the electrical system of the imaging device. Figure 2 As shown in FIG. 1 , the image sensor 20 includes a photoelectric conversion element 72. The photoelectric conversion element 72 has a light receiving surface 72A. The photoelectric conversion element 72 is arranged in the imaging device body 16 in such a manner that the center of the light receiving surface 72A coincides with the optical axis OA (see also FIG. 1 ). Figure 1). The photoelectric conversion element 72 has a plurality of photosensitive pixels arranged in a matrix, and the light receiving surface 72A is formed by a plurality of photosensitive pixels. Each photosensitive pixel has a microlens (not shown). Each photosensitive pixel is a physical pixel having a photodiode (not shown), which performs photoelectric conversion on the received light and outputs an electrical signal corresponding to the amount of received light.

[0074] Furthermore, color filters (not shown) of the three primary colors of light (i.e., red (hereinafter, also referred to as "R"), green (hereinafter, also referred to as "G"), or blue (hereinafter, also referred to as "B")) are arranged in a predetermined pattern on the plurality of photosensitive pixels. In the present embodiment, the Bayer arrangement is used as an example of a predetermined pattern arrangement. However, the Bayer arrangement is only an example, and the present invention is also applicable even if the predetermined pattern arrangement is another type of pattern arrangement such as a G stripe R / G complete grid, an X-Trans (registered trademark) arrangement, or a honeycomb arrangement.

[0075] In addition, hereinafter, for the convenience of explanation, the photosensitive pixel having a microlens and an R color filter is referred to as an R pixel, the photosensitive pixel having a microlens and a G color filter is referred to as a G pixel, and the photosensitive pixel having a microlens and a B color filter is referred to as a B pixel. Also, hereinafter, for the convenience of explanation, the electrical signal output from the R pixel of the photosensitive pixel is referred to as an "R signal", the electrical signal output from the G pixel of the photosensitive pixel is referred to as a "G signal", and the electrical signal output from the B pixel of the photosensitive pixel is referred to as a "B signal". Also, hereinafter, for the convenience of explanation, the R signal, the G signal, and the B signal are also referred to as "RGB color signals". Also, hereinafter, for the convenience of explanation, the R pixels, the G pixels, and the B pixels constituting the RAW image 75A generated based on the RGB color signals are also referred to as "R pixels", "G pixels", and "B pixels". Here, R pixels, G pixels, and B pixels are illustrated, but this is merely an example, and when colors other than R, G, and B (i.e., colors other than the primary colors) are also regularly arranged in the color filter together with R, G, and B, the RAW image 75A also includes pixels of colors other than R pixels, G pixels, and B pixels. Even in this case, the present invention is also applicable.

[0076] The interchangeable lens 18 includes an imaging lens 40. The imaging lens 40 includes an objective lens 40A, a focusing lens 40B, a zoom lens 40C, and an aperture 40D. The objective lens 40A, the focusing lens 40B, the zoom lens 40C, and the aperture 40D are arranged in order along the optical axis OA from the subject side (object side) to the imaging device body 16 side (image side).

[0077] Furthermore, the interchangeable lens 18 includes a control device 36, a first actuator 37, a second actuator 38, and a third actuator 39. The control device 36 controls the entire interchangeable lens 18 according to instructions from the camera body 16. The control device 36 is, for example, a device having a computer including a processor, a storage device, and a memory. The storage device of the control device 36 is, for example, an EEPROM. Various programs and various parameters are stored in the storage device of the control device 36. The memory of the control device 36 is, for example, a RAM, which temporarily stores various information and is used as a working memory. In the control device 36, the processor reads out necessary programs from the storage device and executes the various programs read out on the memory, thereby controlling the entire imaging lens 40.

[0078] In addition, here, as an example of the control device 36, a device having a computer is listed, but this is only an example, and a device including an ASIC, FPGA and / or PLD may also be applied. In addition, as the control device 36, for example, a device implemented by a combination of a hardware structure and a software structure may also be used.

[0079] The first actuator 37 includes a focusing slide mechanism (not shown) and a focusing motor (not shown). The focusing slide mechanism is provided with a focusing lens 40B so as to be slidable along the optical axis OA. The focusing motor is connected to the focusing slide mechanism, and the focusing slide mechanism is operated by receiving power from the focusing motor, thereby moving the focusing lens 40B along the optical axis OA.

[0080] The second actuator 38 includes a zoom sliding mechanism (not shown) and a zoom motor (not shown). The zoom sliding mechanism has the zoom lens 40C mounted thereon so as to be slidable along the optical axis OA. The zoom motor is connected to the zoom sliding mechanism, and the zoom sliding mechanism receives power from the zoom motor to operate, thereby moving the zoom lens 40C along the optical axis OA.

[0081] The third actuator 39 includes a power transmission mechanism (not shown) and an aperture motor (not shown). The aperture 40D has an opening 40D1, and is an aperture that can change the size of the opening 40D1. The opening 40D1 is formed by, for example, a plurality of aperture blades 40D2. The plurality of aperture blades 40D2 are connected to the power transmission mechanism. In addition, the aperture motor is connected to the power transmission mechanism, and the power transmission mechanism transmits the power of the aperture motor to the plurality of aperture blades 40D2. The plurality of aperture blades 40D2 receive the power transmitted from the power transmission mechanism and operate, thereby changing the size of the opening 40D1. The aperture 40D adjusts the exposure by changing the size of the opening 40D1.

[0082] The focusing motor, the zoom motor, and the diaphragm motor are connected to the control device 36, and the driving of the focusing motor, the zoom motor, and the diaphragm motor is controlled by the control device 36. In addition, in the present embodiment, a stepping motor is used as an example of the focusing motor, the zoom motor, and the diaphragm motor. Therefore, the focusing motor, the zoom motor, and the diaphragm motor operate in synchronization with the pulse signal according to the command from the control device 36. In addition, here, an example in which the focusing motor, the zoom motor, and the diaphragm motor are provided in the interchangeable lens 18 is shown, but this is only an example, and at least one of the focusing motor, the zoom motor, and the diaphragm motor may be provided in the camera body 16. In addition, the structure and / or operation method of the interchangeable lens 18 may be changed as needed.

[0083] In the imaging device 10, when in imaging mode, the MF mode and the AF mode can be selectively set according to an instruction given to the imaging device body 16. The MF mode is an operation mode for manual focus. In the MF mode, for example, when the user operates the focus ring 18A or the like, the focus lens 40B moves along the optical axis OA by an amount corresponding to the amount of operation of the focus ring 18A or the like, thereby adjusting the focus.

[0084] In the AF mode, the imaging device body 16 calculates a focus position corresponding to the subject distance and moves the focus lens 40B toward the calculated focus position to adjust the focus. Here, the focus position refers to the position of the focus lens 40B on the optical axis OA in the focused state.

[0085] The imaging device body 16 includes an image sensor 20, an image processing engine 12, a system controller 44, an image memory 46, a UI device 48, an external I / F 50, a communication I / F 52, a photoelectric conversion element driver 54, and an input / output interface 70. The image sensor 20 includes a photoelectric conversion element 72 and an A / D converter 74.

[0086] The image processing engine 12, the image memory 46, the UI system device 48, the external I / F 50, the photoelectric conversion element driver 54, the mechanical shutter driver (not shown), and the A / D converter 74 are connected to the input / output interface 70. In addition, the control device 36 of the interchangeable lens 18 is also connected to the input / output interface 70.

[0087] The system controller 44 includes a processor (not shown in the figure), a storage device (not shown in the figure), and a memory (not shown in the figure). In the system controller 44, the storage device is a non-temporary storage medium that is readable by a computer and stores various parameters and various programs. The storage device of the system controller 44 is, for example, an EEPROM. However, this is only an example, and a HDD and / or SSD etc. may be used as the storage device of the system controller 44 instead of or together with the EEPROM. Furthermore, the memory of the system controller 44 temporarily stores various information and is used as a working memory. In the system controller 44, the processor reads out necessary programs from the storage device and executes the various programs read out on the memory, thereby controlling the entire imaging device 10. That is, in Figure 2 In the illustrated example, the image processing engine 12 , the image memory 46 , the UI system device 48 , the external I / F 50 , the communication I / F 52 , the photoelectric conversion element driver 54 , and the control device 36 are controlled by the system controller 44 .

[0088] The image processing engine 12 operates under the control of the system controller 44. The image processing engine 12 includes a processor 62, a storage device 64, and a memory 66. Here, the processor 62 is an example of the "first processor" according to the present invention.

[0089] The processor 62, the storage device 64, and the memory 66 are connected via a bus 68, and the bus 68 is connected to an input / output interface 70. Figure 2 In the example shown, for the sake of convenience, one bus is shown as the bus 68, but a plurality of buses may be provided. The bus 68 may be a serial bus or a parallel bus including a data bus, an address bus, and a control bus.

[0090] The storage device 64 is a non-temporary storage medium readable by a computer, and stores various parameters and various programs different from the various parameters and various programs stored in the storage device of the system controller 44. The storage device 64 is, for example, an EEPROM. However, this is only an example, and a HDD and / or an SSD etc. may be used as the storage device 64 instead of or together with the EEPROM. In addition, the memory 66 is, for example, a RAM, which temporarily stores various information and is used as a working memory.

[0091] The processor 62 reads out a necessary program from the storage device 64, and executes the read program on the memory 66. The processor 62 performs image processing according to the program executed on the memory 66.

[0092] The photoelectric conversion element 72 is connected to the photoelectric conversion element driver 54. The photoelectric conversion element driver 54 supplies an imaging timing signal to the photoelectric conversion element 72 according to an instruction from the processor 62. The imaging timing signal specifies the timing of imaging by the photoelectric conversion element 72. The photoelectric conversion element 72 performs reset, exposure, and output of an electrical signal according to the imaging timing signal supplied from the photoelectric conversion element driver 54. Examples of the imaging timing signal include a vertical synchronization signal and a horizontal synchronization signal.

[0093] When the interchangeable lens 18 is mounted on the imaging device body 16, the subject light incident on the imaging lens 40 is imaged on the light receiving surface 72A by the imaging lens 40. The photoelectric conversion element 72 performs photoelectric conversion on the subject light received by the light receiving surface 72A under the control of the photoelectric conversion element driver 54, and outputs an electrical signal corresponding to the light amount of the subject light as analog image data representing the subject light to the A / D converter 74. Specifically, the A / D converter 74 reads out the analog image data for each horizontal line in units of one frame from the photoelectric conversion element 72 in an exposure sequential readout manner.

[0094] The A / D converter 74 generates a RAW image 75A by digitizing the analog image data. The RAW image 75A is an image in which R pixels, G pixels, and B pixels are arranged in a mosaic shape. The RAW image 75A is an example of a "camera image" involved in the present invention. In addition, in the present embodiment, as an example, the number of bits (in other words, the bit length, the number of color bits, or the color depth) used to express the grayscale value of each pixel of the R pixels, B pixels, and G pixels included in the RAW image 75A is 14 bits. In addition, 14 bits is only an example, and the number of bits may be more than 14 bits, or the number of bits may be less than 14 bits.

[0095] In the present embodiment, as an example, the processor 62 of the image processing engine 12 acquires a RAW image 75A from the A / D converter 74, and performs image processing including development on the acquired RAW image 75A, thereby generating an image file 75B. Here, development refers to the process of compressing the brightness and color difference signals according to a predetermined compression method. As the predetermined compression method (i.e., the format of the image file), for example, JPEG, TIFF, JPEG XR, MPEG, or AVI can be cited. In addition, the image processing includes image quality adjustment for the RAW image 75A. The image quality adjustment for the RAW image 75A is achieved by adjusting the focal length, adjusting the aperture value, adjusting the lens characteristics, adjusting the interval elimination characteristics between pixels P, a grayscale correction function (for example, a process of correcting the grayscale of an RGB image according to a gamma value), a gain correction function, and a noise reduction function. As another example of image quality adjustment for the RAW image 75A, there can be cited color space conversion processing (i.e., processing for converting the color space of the RGB image that has been gamma-corrected from RGB color space to YCbCr color space), brightness filtering processing (i.e., processing for filtering the brightness signal (so-called Y signal) using a brightness filter (not shown)), color difference processing (i.e., processing for filtering the Cb signal and the Cr signal to reduce high-frequency noise) and / or resizing processing (i.e., processing for adjusting the brightness and color difference signals so that the size of the image represented by the brightness and color difference signals corresponds to the size indicated by the user, etc.), etc.

[0096] An example of the image file 75B is a JPEG file. The JPEG file is only an example, and the image file 75B may also be a JPEG XR file, a TIFF file, an MPEG file, or another type of image file such as MPEG or AV I. The image file 75B is stored in the image memory 46 by the processor 62.

[0097] The UI system device 48 includes a display 28, and the processor 62 displays various information on the display 28. In addition, the UI system device 48 includes a receiving device 76. The receiving device 76 includes a touch panel 30 and a hard key unit 78. The hard key unit 78 includes the indication key 26 (see Figure 1 The processor 62 operates according to various instructions received through the touch panel 30. In addition, here, the hard key unit 78 is included in the UI system device 48, but the present invention is not limited to this. For example, the hard key unit 78 can also be connected to the external I / F 50.

[0098] The external I / F 50 controls the exchange of various information with a device (hereinafter, also referred to as an "external device") existing outside the camera 10. As an example of the external I / F 50, a USB interface can be cited. External devices such as smart devices, personal computers, servers, USB memories, memory cards, and / or printers (not shown) are directly or indirectly connected to the USB interface.

[0099] The communication I / F 52 is connected to a network (not shown). The communication I / F 52 controls the exchange of information between a communication device (not shown) such as a server on the network and the system controller 44. For example, the communication I / F 52 sends information corresponding to a request from the system controller 44 to the communication device via the network. In addition, the communication I / F 52 receives information sent from the communication device and outputs the received information to the system controller 44 via the input / output interface 70.

[0100] The image processing engine 12 applies AI to the RAW image 75A to perform processing to improve the image quality of the RAW image 75A. Figure 3 to Figure 5 , a method for generating AI applicable to RAW image 75A is described.

[0101] Figure 3 79 is a conceptual diagram showing an example of the structure of the learning device 79. Figure 3 As shown, the learning device 79 includes a processor 80, a storage device 82, and a memory 84. The hardware structure of the processor 80, the storage device 82, and the memory 84 in the learning device 79 is basically the same as the hardware structure of the processor 62, the storage device 64, and the memory 66 described above, so the description is omitted here. Figure 3 In the example shown, the learning device 79 is an example of the “learning device” involved in the present invention.

[0102] A learning program 90 is stored in the storage device 82. The processor 80 is an example of the "second processor" involved in the present invention. The processor 80 reads the learning program 90 from the storage device 82 and executes the read learning program 90 on the memory 84. The processor 80 performs a learning process according to the learning program 90 executed on the memory 84. The learning process is a process of generating a learned model 106 from a model 98. The learned model 106 is generated by performing machine learning on the model 98 by the processor 80. That is, the learned model 106 is generated by optimizing the model 98 through machine learning. For example, the model 98 is a neural network having hundreds of millions to trillions of intermediate layers. As an example of the model 98, a generative AI model that generates and outputs an image with higher image quality than the input image (for example, an image with a resolution at least several times that of the input image) can be cited.

[0103] The storage device 82 stores a plurality of (e.g., tens of thousands to hundreds of billions) training data 92. The training data 92 is used for machine learning of the model 98. That is, in the learning device 79, the processor 80 acquires the plurality of training data 92 from the storage device 82, and uses the acquired plurality of training data 92 to cause the model 98 to perform machine learning.

[0104] The training data 92 is labeled data. Labeled data is data in which a corresponding association is established between an example question image 94 (in other words, example question data) and a correct answer image 96 (in other words, correct answer data). Here, the training data 92 is an example of "training data" involved in the present invention, the example question image 94 is an example of "example question image" involved in the present invention, and the correct answer image 96 is an example of "correct answer image" involved in the present invention.

[0105] The example image 94 is an image that is assumed to be the RAW image 75A. The image that is assumed to be the RAW image 75A is, for example, an image of the same quality as the RAW image 75A including resolution. In the present embodiment, an image obtained by actually photographing a sample subject (for example, a Figure 3 However, this is only an example, and the image assumed to be the RAW image 75A may also be a virtually generated image. An example of a virtually generated image may be an image generated by a generation AI or the like.

[0106] Correct answer image 96 is an image obtained by converting example question image 94 into a higher quality image. An example of a higher quality image is an image obtained by converting ratio question image 94 into a higher quality image. An image obtained by converting ratio question image 94 into a higher quality image is, for example, an image having a larger number of pixels than question image 94. In other words, an image obtained by converting ratio question image 94 into a higher quality image can be said to have a higher visual resolution than question image 94.

[0107] The processor 80 acquires the training data 92 one by one from the storage device 82. The processor 80 inputs the example image 94 from the training data 92 acquired from the storage device 82 to the model 98. When the example image 94 is input, the model 98 generates a comparison image 100 which is an image with a higher resolution than the question image 94 (i.e., an image with a larger number of pixels than the question image 94). The comparison image 100 is an image for comparison with the correct answer image 96 associated with the example image 94 input to the model 98. In the present embodiment, the comparison image 100 is an example of the "evaluation target image" involved in the present invention.

[0108] The processor 80 calculates an error 102 between the correct answer image 96 and the comparison image 100, which are associated with the example image 94 input to the model 98. The error 102 is an example of the "comparison result" involved in the present invention. The processor 80 calculates a plurality of adjustment values ​​104 that minimize the error 102. Then, the processor 80 uses the plurality of adjustment values ​​104 to adjust the plurality of optimization variables in the model 98. Here, the plurality of optimization variables refers to, for example, a plurality of connection weights and a plurality of offset values ​​included in the model 98.

[0109] The processor 80 repeatedly performs a series of processes, including input of the example image 94 to the model 98, calculation of the error 102, calculation of the plurality of adjustment values ​​104, and adjustment of the plurality of optimization variables in the model 98, using the plurality of training data 92 stored in the storage device 82. That is, the processor 80 adjusts the plurality of optimization variables in the model 98 using the plurality of adjustment values ​​104 calculated so as to minimize the error 102 for each of the plurality of example images 94 included in the plurality of training data 92 stored in the storage device 82, thereby optimizing the model 98. The processor 80 generates a learned model 106 by optimizing the model 98. When the example image 94 is input to the learned model 106 generated in this way, the learned model 106 generates and outputs an image having the same resolution as the correct answer image 96 as an image corresponding to the input example image 94.

[0110] Figure 4 96 is a conceptual diagram showing an example of a method for creating a correct answer image 96. Figure 4 As shown, in the present embodiment, in order to create the correct answer image 96, the camera device 500 is used. The camera device 500 is different from the camera device 10 in that it has a pixel shift device 502. Similar to the camera device 10, the image sensor 20 mounted on the camera device 500 is provided with a photoelectric conversion element 72. Also, similar to the camera device 10, pixels P of different colors (i.e., R pixels, G pixels, and B pixels) are regularly arranged on the photoelectric conversion element 72. The arrangement pattern of the pixels P of different colors is the same as that of the camera device 10. In Figure 4 In the illustrated example, the number of pixels P is smaller than the actual number of pixels P in order to facilitate understanding of the present invention.

[0111] The pixel shift device 502 is mechanically connected to the photoelectric conversion element 72. The pixel shift device 502 includes a power source (e.g., a voice coil motor, etc.), and transmits power generated by the power source to the photoelectric conversion element 72, thereby selectively shifting the photoelectric conversion element 72 upward, downward, leftward, and rightward at a 1-pixel pitch or a 1 / 2 (0.5)-pixel pitch. The pixel shift device 502 is controlled by the system controller 44 (see Figure 2) to shift the photoelectric conversion element 72 pixels. In this embodiment, the photoelectric conversion element 72 is selectively shifted by 1 pixel or 1 / 2 pixel in the upward, downward, left, and right directions (i.e., the photoelectric conversion element 72 is selectively shifted by 1 pixel pitch or by 1 / 2 (0.5) pixel pitch in the upward, downward, left, and right directions). Figure 4 In the illustrated example, the image sensor 20 mounted on the imaging device 500 is an example of the “first image sensor” and the “second image sensor” according to the present invention.

[0112] In the imaging device 500, in the first to fourth steps, the first to fourth single images 108A to 108D are generated as RAW images, respectively. Furthermore, in the fifth to eighth steps, the fifth to eighth single images 110A to 110D are generated as RAW images, respectively. Furthermore, in the ninth to twelfth steps, the ninth to twelfth single images 112A to 112D are generated as RAW images, respectively. Furthermore, in the thirteenth to sixteenth steps, the thirteenth to sixteenth single images 114A to 114D are generated as RAW images, respectively. Furthermore, in the seventeenth to twentieth steps, the first to fourth refined images 116 to 122 are generated. Furthermore, in the twenty-first step, the correct answer image 96 is generated.

[0113] In addition, for the sake of convenience in explanation, when there is no need to distinguish between the 1st to 4th single images 108A to 108D, the 5th to 8th single images 110A to 110D, the 9th to 12th single images 112A to 112D, and the 13th to 16th single images 114A to 114D, they are referred to as "single images (or unit images)" without being annotated.

[0114] In the first step, the first single image 108A is generated by capturing the sample object by the image sensor 20 in a state where the photoelectric conversion element 72 is located at the home position. In the second step, the second single image 108B is generated by capturing the sample object by the image sensor 20 in a state where the photoelectric conversion element 72 is shifted to the left by 1 / 2 pixel from the imaging position of the photoelectric conversion element 72 in the first step. In the third step, the third single image 108C is generated by capturing the sample object by the image sensor 20 in a state where the photoelectric conversion element 72 is shifted downward by 1 / 2 pixel from the imaging position of the photoelectric conversion element 72 in the second step. In the fourth step, the fourth single image 108D is generated by capturing the sample object by the image sensor 20 in a state where the photoelectric conversion element 72 is shifted to the right by 1 / 2 pixel from the imaging position of the photoelectric conversion element 72 in the third step.

[0115] The first to fourth single images 108A to 108D generated in this way are images in which pixels P1 of different colors are regularly arranged and the pixels are shifted to positions where the pixels P1 of different colors overlap (i.e., images in which the pixels P1 of different colors are shifted so that the pixels P1 of different colors overlap). Here, the pixel P1 refers to a pixel constituting a RAW image obtained by photographing by the image sensor 20 of the camera device 500 (in other words, an image pixel). The position of the pixel P1 in the RAW image obtained by photographing by the image sensor 20 of the camera device 500 coincides with the position of the pixel P in the photoelectric conversion element 72 of the image sensor 20 mounted on the camera device 500.

[0116] In the fifth step, the fifth single image 110A is generated by capturing the sample object by the image sensor 20 in a state where the image position of the photoelectric conversion element 72 in the first step is shifted by 1 pixel to the left. In the sixth step, the sixth single image 110B is generated by capturing the sample object by the image sensor 20 in a state where the image position of the photoelectric conversion element 72 in the fifth step is shifted by 1 / 2 pixel to the left. In the seventh step, the seventh single image 110C is generated by capturing the sample object by the image sensor 20 in a state where the image position of the photoelectric conversion element 72 in the sixth step is shifted by 1 / 2 pixel downward. In the eighth step, the eighth single image 110D is generated by capturing the sample object by the image sensor 20 in a state where the image position of the photoelectric conversion element 72 in the seventh step is shifted by 1 / 2 pixel to the right.

[0117] The fifth to eighth single images 110A to 110D generated in this way are images in which pixels P1 of different colors are regularly arranged and pixels are shifted to positions where pixels P1 of different colors overlap (ie, images in which pixels P1 of different colors are shifted so that pixels P1 of different colors overlap).

[0118] In the ninth step, the ninth single image 112A is generated by capturing the sample object by the image sensor 20 in a state where the image capturing position of the photoelectric conversion element 72 in the fifth step is shifted downward by 1 pixel. In the tenth step, the tenth single image 112B is generated by capturing the sample object by the image sensor 20 in a state where the image capturing position of the photoelectric conversion element 72 in the ninth step is shifted to the left by 1 / 2 pixel. In the eleventh step, the eleventh single image 112C is generated by capturing the sample object by the image sensor 20 in a state where the image capturing position of the photoelectric conversion element 72 in the tenth step is shifted downward by 1 / 2 pixel. In the twelfth step, the twelfth single image 112D is generated by capturing the sample object by the image sensor 20 in a state where the image capturing position of the photoelectric conversion element 72 in the eleventh step is shifted to the right by 1 / 2 pixel.

[0119] The ninth to twelfth single images 112A to 112D generated in this way are images in which pixels P1 of different colors are regularly arranged and pixels are shifted to positions where pixels P1 of different colors overlap (ie, images in which pixels P1 of different colors are shifted so that pixels P1 of different colors overlap).

[0120] In step 13, a 13th single image 114A is generated by capturing the sample object by the image sensor 20 in a state where the image capturing position of the photoelectric conversion element 72 in step 9 is shifted by 1 pixel to the right. In step 14, a 14th single image 114B is generated by capturing the sample object by the image sensor 20 in a state where the image capturing position of the photoelectric conversion element 72 in step 13 is shifted by 1 / 2 pixel to the left. In step 15, a 15th single image 114C is generated by capturing the sample object by the image sensor 20 in a state where the image capturing position of the photoelectric conversion element 72 in step 14 is shifted by 1 / 2 pixel downward. In step 16, a 16th single image 114D is generated by capturing the sample object by the image sensor 20 in a state where the image capturing position of the photoelectric conversion element 72 in step 15 is shifted by 1 / 2 pixel to the right.

[0121] The 13th to 16th single images 114A to 114D generated in this way are images in which pixels P1 of different colors are regularly arranged and pixels are shifted to positions where pixels P1 of different colors overlap (that is, images in which pixels are shifted so that pixels P1 of different colors overlap).

[0122] In step 17, the first refined image 116 is created by synthesizing the first to fourth single images 108A to 108D. For example, the first refined image 116 is an image created by overlapping the first to fourth single images 108A to 108D. Here, the first to fourth single images 108A to 108D are images after 1 / 2 pixel shifting. In other words, it can be said that the first to fourth single images 108A to 108D are images that have a correlation of being shifted by 1 / 2 pixel. Therefore, the pixel P1 of the first refined image 116 created by overlapping the first to fourth single images 108A to 108D is divided into four parts by R, G, G, and B, so the pixel P1 is finer than each of the first to fourth single images 108A to 108D, thereby improving the resolution.

[0123] In step 18, the second refined image 118 is created by synthesizing the fifth to eighth single images 110A to 110D. For example, the second refined image 118 is an image created by overlapping the fifth to eighth single images 110A to 110D. Here, the fifth to eighth single images 110A to 110D are images after 1 / 2 pixel shifting. In other words, it can be said that the fifth to eighth single images 110A to 110D are images that have a correlation of being shifted by 1 / 2 pixel. Therefore, the pixel P1 of the second refined image 118 created by overlapping the fifth to eighth single images 110A to 110D is divided into four parts by R, G, G, and B, so the pixel P1 is refined compared to each of the fifth to eighth single images 110A to 110D.

[0124] In step 19, the third refined image 120 is created by synthesizing the 9th to 12th single images 112A to 112D. For example, the third refined image 120 is an image created by overlapping the 9th to 12th single images 112A to 112D. Here, the 9th to 12th single images 112A to 112D are images after 1 / 2 pixel shift. In other words, it can be said that the 9th to 12th single images 112A to 112D are images that have a correlation of being shifted by 1 / 2 pixel. Therefore, the pixel P1 of the third refined image 120 created by overlapping the 9th to 12th single images 112A to 112D is divided into four parts by R, G, G, and B, so the pixel P1 is finer than each of the 9th to 12th single images 112A to 112D, thereby improving the resolution.

[0125] In step 20, the fourth refined image 122 is created by synthesizing the 13th to 16th single images 114A to 114D. For example, the fourth refined image 122 is an image created by synthesizing the 13th to 16th single images 114A to 114D (here, as an example, an image created by overlapping the 13th to 16th single images 114A to 114D). Here, the 13th to 16th single images 114A to 114D are images after 1 / 2 pixel shift. In other words, it can be said that the 13th to 16th single images 114A to 114D are images that are related to each other by shifting 1 / 2 pixel. Therefore, the pixel P1 of the fourth refined image 122 created by synthesizing the 13th to 16th single images 114A to 114D is divided into four parts by R, G, G, and B, so the pixel P1 is refined compared with each of the 13th to 16th single images 114A to 114D, thereby improving the resolution.

[0126] In step 21, correct answer image 96 is created by synthesizing (here, as an example, overlapping in units of pixels P1 whose positions correspond to each other) 1st to 4th refined images 116 to 122 created from 1st to 4th single images 108A to 108D, 5th to 8th single images 110A to 110D, 9th to 12th single images 112A to 112D, and 13th to 16th single images 114A to 114D. Correct answer image 96 created in this way is an image with higher image quality than each of the 1st to 4th single images 108A to 108D, 5th to 8th single images 110A to 110D, 9th to 12th single images 112A to 112D, and 13th to 16th single images 114A to 114D. Figure 4 In the example shown, the high-quality image refers to a high-resolution image. In other words, the correct answer image 96 created in this way can be said to be an image with a larger number of pixels than the first to fourth single images 108A to 108D, the fifth to eighth single images 110A to 110D, the ninth to twelfth single images 112A to 112D, and the thirteenth to sixteenth single images 114A to 114D. Furthermore, in other words, the correct answer image 96 created in this way can also be said to be an image with a higher visual resolution than the first to fourth single images 108A to 108D, the fifth to eighth single images 110A to 110D, the ninth to twelfth single images 112A to 112D, and the thirteenth to sixteenth single images 114A to 114D. Furthermore, since the correct answer image 96 is an image obtained by synthesizing the first to fourth refined images 116 to 122 , it can be said that the correct answer image 96 is an image having a higher degree of color reproduction than each of the first to fourth refined images 116 to 122 .

[0127] Furthermore, the first to fourth single images 108A to 108D, the fifth to eighth single images 110A to 110D, the ninth to twelfth single images 112A to 112D, and the thirteenth to sixteenth single images 114A to 114D are images generated as RAW images. Therefore, the first to fourth refined images 116 to 122 created from the first to fourth single images 108A to 108D, the fifth to eighth single images 110A to 110D, the ninth to twelfth single images 112A to 112D, and the thirteenth to sixteenth single images 114A to 114D are also RAW images. Therefore, by superimposing the first to fourth refined images 116 to 122 in units of corresponding pixels P1, the correct answer image 96 is also created as a RAW image.

[0128] In addition, Figure 4In the example shown, the correct answer image 96 is created based on the first to fourth single images 108A to 108D, the fifth to eighth single images 110A to 110D, the ninth to twelfth single images 112A to 112D, and the thirteenth to sixteenth single images 114A to 114D obtained by photographing the sample subject by the camera device 500. However, this is only an example. For example, at least one of the first to fourth single images 108A to 108D, the fifth to eighth single images 110A to 110D, the ninth to twelfth single images 112A to 112D, and the thirteenth to sixteenth single images 114A to 114D used to create the correct answer image 96 may be a virtual image generated by the generation AI. The generative AI may be an AI specifically used to generate images, or it may be a generative AI that generates and outputs at least one of the 1st to 4th single images 108A to 108D, the 5th to 8th single images 110A to 110D, the 9th to 12th single images 112A to 112D, and the 13th to 16th single images 114A to 114D according to input instruction data (so-called prompts), as in ChatGPT using GPT-4 (Internet search <https: / / openai.com / gpt-4>).

[0129] Figure 5 This is a conceptual diagram showing an example of a method of creating the example image 94. Figure 5 2 shows an example of a schematic method of frontally observing the first to fourth single images 108A to 108D, the fifth to eighth single images 110A to 110D, the ninth to twelfth single images 112A to 112D, and the thirteenth to sixteenth single images 114A to 114D. Figure 5 As shown, the first to fourth single images 108A to 108D have centers 108A1 to 108D1. Furthermore, the first to fourth single images 108A to 108D have a center of gravity G1 when the first to fourth single images 108A to 108D are overlapped. Furthermore, similarly, the fifth to eighth single images 110A to 110D have centers 110A1 to 110D1, respectively. Furthermore, the fifth to eighth single images 110A to 110D have a center of gravity G2 when the fifth to eighth single images 110A to 110D are overlapped. Furthermore, similarly, the ninth to twelfth single images 112A to 112D have centers 112A1 to 112D1, respectively. Furthermore, the ninth to twelfth single images 112A to 112D have a center of gravity G3 when the ninth to twelfth single images 112A to 112D are overlapped. Similarly, the 13th to 16th single images 114A to 114D have centers 114A1 to 114D1, respectively. Also, the 13th to 16th single images 114A to 114D have a center of gravity G4 when the 13th to 16th single images 114A to 114D are superimposed.

[0130] Here, the example image 94 is an image created by synthesizing (here, as an example, overlapping in units of pixels P1 whose positions correspond to each other) the first single image 108A, the fifth single image 110A, the ninth single image 112A, and the thirteenth single image 114A. The first single image 108A is an image representing the first to fourth single images 108A to 108D. The fifth single image 110A is an image representing the fifth to eighth single images 110A to 110D. The ninth single image 112A is an image representing the ninth to twelfth single images 112A to 112D. The thirteenth single image 114A is an image representing the thirteenth to sixteenth single images 114A to 114D.

[0131] The first single image 108A used to create the example image 94 is the image having the center closest to the center of gravity G1 among the first to fourth single images 108A to 108D. That is, the center closest to the center of gravity G1 among the centers 108A1 to 108D1 is the center 108A1, and the center 108A1 is the center of the first single image 108A, so the first single image 108A is used when creating the example image 94.

[0132] The fifth single image 110A used to create the example image 94 is the image having the center closest to the center of gravity G2 among the fifth to eighth single images 110A to 110D. That is, the center closest to the center of gravity G2 among the centers 110A1 to 110D1 is the center 110A1, and the center 110A1 is the center of the fifth single image 110A, so the fifth single image 110A is used when creating the example image 94.

[0133] The ninth single image 112A used to create the example image 94 is the image having the center closest to the center of gravity G3 among the ninth to twelfth single images 112A to 112D. That is, the center closest to the center of gravity G3 among the centers 112A1 to 112D1 is the center 112A1, and the center 112A1 is the center of the ninth single image 112A, so the ninth single image 112A is used when creating the example image 94.

[0134] The 13th single image 114A used to create the example image 94 is the image having the center closest to the center of gravity G4 among the 13th to 16th single images 114A to 114D. That is, the center closest to the center of gravity G4 among the centers 114A1 to 114D1 is the center 114A1, and the center 114A1 is the center of the 13th single image 114A, so the 13th single image 114A is used when creating the example image 94.

[0135] Figure 6 It means that by Figure 31 is a conceptual diagram of an example of the application stage of the learned model 106 generated by the learning process of the example shown in FIG. Figure 6 As shown, in the imaging device 10, the storage device 64 stores the learned model 106. In addition, the storage device 64 stores the high-quality image processing program 124. In the imaging device 10, the processor 62 reads the high-quality image processing program 124 from the storage device 64 and executes the read high-quality image processing program 124 on the memory 66. The processor 62 performs high-quality image processing according to the high-quality image processing program 124 executed on the memory 66. The high-quality image processing is as follows: the RAW image 75A is input to the learned model 106 stored in the storage device 64 (while referring to the high-quality image processing program 124). Figure 2 ), the learned model 106 generates and outputs a high-resolution image 75A1 as an image obtained by converting the RAW image 75A into a high-resolution image, and the processor 62 acquires the high-resolution image 75A1. Here, the high-resolution image 75A1 is an example of the "inference result" involved in the present invention.

[0136] RAW image 75A is an image with the same resolution as example image 94, and high-resolution image 75A1 is an image with the same resolution as correct answer image 96 (refer to Figure 3 and Figure 4 The processor 62 generates an image file 75B by performing image processing including development on the high-resolution image 75A1, and stores the generated image file 75B in the image memory 46.

[0137] Next, refer to Figure 7 The operation of the portion of the learning device 79 that is related to the present invention will be described. Figure 7 An example of the flow of the learning process executed by the processor 80 is shown in FIG. Figure 7 The illustrated flow of the learning process is an example of the “method for generating a learned model” according to the present invention.

[0138] exist Figure 7 In the learning process shown, first, in step ST10, the processor 80 executes to obtain the unprocessed training data 92 (ie, Figure 7 After the processing of step ST10 of the training data 92 not used in the learning process shown in the figure, the learning process proceeds to step ST12.

[0139] In step ST12, processor 80 inputs example image 94 included in training data 92 acquired in step ST10 to model 98. After executing the process of step ST12, the learning process proceeds to step ST14. By executing the process of step ST12, comparison image 100 is output from model 98.

[0140] In step ST14, processor 80 acquires comparison image 100 output from model 98. After executing the process of step ST14, the learning process proceeds to step ST16.

[0141] In step ST16, processor 80 compares comparison image 100 acquired in step ST14 with correct answer image 96 included in training data 92 acquired in step ST10. After executing the process of step ST16, the learning process proceeds to step ST18.

[0142] In step ST18, processor 80 adjusts model 98 using a plurality of adjustment values ​​104 obtained by comparing comparison image 100 with correct answer image 96 in step ST16. Model 98 is optimized by repeatedly executing the process of step ST18 based on all training data 92 stored in storage device 82. After executing the process of step ST18, the learning process proceeds to step ST20.

[0143] In step ST20, processor 80 determines whether unprocessed training data 92 is stored in storage device 82. In step ST20, if unprocessed training data 92 is stored in storage device 82, the determination is affirmative, and the learning process proceeds to step ST10. In step ST20, if unprocessed training data 92 is not stored in storage device 82, the determination is negative, and the learning process ends.

[0144] Next, refer to Figure 8 The operation of the portion of the imaging device 10 according to the present invention will be described. Here, for the sake of convenience, the description will be given on the premise that the learned model 106 has already been stored in the storage device 64 .

[0145] exist Figure 8 In the high-quality image processing shown, first, in step ST50, the processor 62 determines whether one frame of image is captured by the image sensor 20. In step ST50, if one frame of image is not captured by the image sensor 20, the determination is negative, and the high-quality image processing proceeds to step ST58. If one frame of image is captured by the image sensor 20, the determination is positive, and the high-quality image processing proceeds to step ST52.

[0146] In step ST52, the processor 62 acquires the RAW image 75A from the image sensor 20. After the processing of step ST52 is executed, the image quality improvement processing proceeds to step ST54.

[0147] In step ST54, the processor 62 inputs the RAW image 75A acquired in step ST52 to the learned model 106. After executing the process of step ST54, the high-quality image processing proceeds to step ST56. By executing the process of step ST54, the learned model 106 generates and outputs an image obtained by converting the RAW image 75A into a high-quality image, that is, a high-resolution image 75A1.

[0148] In step ST56, the processor 62 acquires the high-resolution image 75A1. Then, the processor 62 generates the image file 75B based on the high-resolution image 75A1 and stores it in the image memory 46. After executing the process of step ST56, the high-quality image processing proceeds to step ST58.

[0149] In step ST58, the processor 62 determines whether a condition for ending the high-quality image processing (hereinafter referred to as "ending condition") is satisfied. As the ending condition, the condition that the receiving device 76 receives an instruction to end the high-quality image processing can be cited. In step ST58, if the ending condition is not satisfied, the determination is negative, and the high-quality image processing proceeds to step ST50. In step ST58, if the ending condition is satisfied, the determination is positive, and the high-quality image processing ends.

[0150] As described above, the training data 92 according to the present embodiment is used for machine learning of the model 98. The learned model 106 is generated by machine learning of the model 98. The training data 92 includes the example image 94 and the correct answer image 96, and the example image 94 and the correct answer image 96 are associated with each other. Furthermore, the correct answer image 96 is an image obtained by synthesizing the first to fourth single images 108A to 108D, the fifth to eighth single images 110A to 110D, the ninth to twelfth single images 112A to 112D, and the thirteenth to sixteenth single images 114A to 114D. Furthermore, the example image 94 is an image synthesized by combining images representing the first to fourth single images 108A to 108D (here, as an example, the first single image 108A), images representing the fifth to eighth single images 110A to 110D (here, as an example, the fifth single image 110A), images representing the ninth to twelfth single images 112A to 112D (here, as an example, the ninth single image 112A), and images representing the thirteenth to sixteenth single images 114A to 114D (here, as an example, the thirteenth single image 114A). That is, the correct answer image 96 is an image of higher quality than the ratio question image 94. Therefore, by inputting the RAW image 75A to the learned model 106 configured in this way, an image of higher quality than the RAW image 75A can be generated.

[0151] Furthermore, in the present embodiment, example question image 94 and correct answer image 96 are created from a plurality of single images (a plurality of unit images). Therefore, example question image 94 and correct answer image 96 can be created more easily than when example question image 94 and correct answer image 96 are created without using a plurality of single images. This can help reduce the number of man-hours required for creating training data 92.

[0152] Furthermore, in the training data 92 involved in the present embodiment, the correct answer image 96 is an image with a higher resolution by synthesizing the images representing the first to fourth single images 108A to 108D, the images representing the fifth to eighth single images 110A to 110D, the images representing the ninth to twelfth single images 112A to 112D, and the images representing the thirteenth to sixteenth single images 114A to 114D. The high-resolution image is an image with a higher resolution than the images representing the first to fourth single images 108A to 108D, the images representing the fifth to eighth single images 110A to 110D, the images representing the ninth to twelfth single images 112A to 112D, and the images representing the thirteenth to sixteenth single images 114A to 114D. In other words, the high-resolution image can be said to be an image having a larger number of pixels than the images representing the first to fourth single images 108A to 108D, the images representing the fifth to eighth single images 110A to 110D, the images representing the ninth to twelfth single images 112A to 112D, and the images representing the thirteenth to sixteenth single images 114A to 114D. Furthermore, in other words, the high-resolution image can also be said to be an image having a higher visual resolution than the images representing the first to fourth single images 108A to 108D, the images representing the fifth to eighth single images 110A to 110D, the images representing the ninth to twelfth single images 112A to 112D, and the images representing the thirteenth to sixteenth single images 114A to 114D. Therefore, by inputting the RAW image 75A to the learned model 106 obtained by performing machine learning using the training data 92 configured in this way, an image having a higher resolution than the RAW image 75A can be output.

[0153] Furthermore, in the training data 92 according to the present embodiment, the first to fourth single images 108A to 108D, the fifth to eighth single images 110A to 110D, the ninth to twelfth single images 112A to 112D, and the thirteenth to sixteenth single images 114A to 114D used to create the example image 94 and the correct answer image 96 are images obtained by performing a pixel shift of 1 pixel and a pixel shift of 1 / 2 pixel, respectively. Therefore, even without increasing the number of pixels of the photoelectric conversion element 72 of the imaging device 10, by inputting the RAW image 75A to the learned model 106 obtained by machine learning using the example image 94 and the correct answer image 96, an image with a higher resolution than the RAW image 75A can be obtained. That is, even without increasing the development cost and manufacturing cost of the photoelectric conversion element 72 of the imaging device 10, an image with a higher resolution than the RAW image 75A can be obtained.

[0154] Furthermore, in the training data 92 involved in the present embodiment, the first to fourth single images 108A to 108D, the fifth to eighth single images 110A to 110D, the ninth to twelfth single images 112A to 112D, and the thirteenth to sixteenth single images 114A to 114D used to create the example image 94 and the correct answer image 96 are respectively arranged with pixels P of different colors (here, as an example, pixels P of R, G, and B) arranged regularly (for example, arranged in a Bayer arrangement). Furthermore, the first to fourth single images 108A to 108D, the fifth to eighth single images 110A to 110D, the ninth to twelfth single images 112A to 112D, and the thirteenth to sixteenth single images 114A to 114D are images in which pixels P of different colors are shifted so that pixels P of different colors overlap. For example, the pixels are shifted so that R pixels, G pixels, G pixels, and B pixels overlap. Therefore, when a RAW image in which different colors (here, as an example, R, G, and B) are regularly arranged is input to the learned model 106, the learned model 106 can generate and output an image that does not require demosaicing. As a result, an image with a high degree of reproduction of the color of the captured subject can be provided to the user of the imaging device 10, etc., compared with the case of performing demosaicing.

[0155] Furthermore, in the training data 92 involved in the present embodiment, the example image 94 and the correct answer image 96 are RAW images. Therefore, by inputting the RAW image 75A to the learned model 106 generated as described above, the learned model 106 can generate and output an image with higher image quality (in the above example, high resolution) than the RAW image 75A.

[0156] Furthermore, in the training data 92 involved in the present embodiment, the 1st to 4th single images 108A to 108D, the 5th to 8th single images 110A to 110D, the 9th to 12th single images 112A to 112D and the 13th to 16th single images 114A to 114D are respectively images captured by the image sensor 20 of the camera device 500 from different camera positions and shifted by 1 / 2 pixels. Furthermore, the example image 94 is an image obtained by synthesizing the first single image 108A having a center 108A1 closest to the center of gravity G1 when the first to fourth single images 108A to 108D are overlapped, the fifth single image 110A having a center 110A1 closest to the center of gravity G2 when the fifth to eighth single images 110A to 110D are overlapped, the ninth single image 112A having a center 112A1 closest to the center of gravity G3 when the ninth to twelfth single images 112A to 112D are overlapped, and the thirteenth single image 114A having a center 114A1 closest to the center of gravity G4 when the thirteenth to sixteenth single images 114A to 114D are overlapped. Therefore, compared to the case where an image obtained by synthesizing an image having a center farthest from the center of gravity G1 among the 1st to 4th single images 108A to 108D, an image having a center farthest from the center of gravity G2 among the 5th to 8th single images 110A to 110D, an image having a center farthest from the center of gravity G3 among the 9th to 12th single images 112A to 112D, and an image having a center farthest from the center of gravity G4 among the 13th to 16th single images 114A to 114D is used as the example image 94, the reproduction of the subject visually recognized using the image generated and output by the learned model 106 by inputting the RAW image 75A into the learned model 106 can be improved.

[0157] In addition, in the above embodiment, the training data 92 is illustrated, but this is only an example. Fig. 9 As shown, training data 92A may be used to replace training data 92. The difference between training data 92A and training data 92 is that correct answer image 96A is used to replace correct answer image 96, and example image 94A is used to replace example image 94. Fig. 9 In the example shown, the example image 94A is any one of the first single image 108A, the fifth single image 110A, the ninth single image 112A, and the thirteenth single image 114A. Fig. 9 In the example shown, the correct answer image 96A is an image created by synthesizing (here, as an example, superimposing) the first single image 108A, the fifth single image 110A, the ninth single image 112A, and the thirteenth single image 114A. Fig. 9In the example shown, the example image 94A is the first single image 108A, the fifth single image 110A, the ninth single image 112A, or the thirteenth single image 114A. The color reproduction degree of the correct answer image 96A is higher than that of the question image 94A. Therefore, by inputting the RAW image 75A into the learned model 106 generated by optimizing the model 98 through machine learning using the training data 92A, the learned model 106 can generate and output an image in RAW format with a higher color reproduction degree than the RAW image 75A. In addition, here, the correct answer image 96A is illustrated, but the correct answer image 96 can also be applied. In this case, it can be compared with Figure 6 In the example shown, the learned model 106 is similarly caused to generate and output a high-resolution image. This can also be described in the following modified examples in the same manner.

[0158] exist Fig. 9 In the example shown, the first single image 108A, the fifth single image 110A, the ninth single image 112A, and the thirteenth single image 114A are shown, but this is only an example. For example, instead of the first single image 108A, the fifth single image 110A, the ninth single image 112A, and the thirteenth single image 114A, the second single image 108B, the sixth single image 110B, the tenth single image 112B, and the fourteenth single image 114B may be used, the third single image 108C, the seventh single image 110C, the eleventh single image 112C, and the fifteenth single image 114C may be used, or the fourth single image 108D, the eighth single image 110D, the twelfth single image 112D, and the sixteenth single image 114D may be used. In addition, this can also be described in the following modified examples.

[0159] exist Fig. 9 In the example shown, training data 92A is shown, but this is only an example. Fig.10 As shown, training data 92B may be used instead of training data 92A. The difference between training data 92B and training data 92A is that example image 94B is used instead of example image 94A. Example image 94B is an example of the "first image" involved in the present invention. It is an image based on no more than three images among the first single image 108A, the fifth single image 110A, the ninth single image 112A, and the thirteenth single image 114A. Fig.10In the example shown, an image based on the first single image 108A, the fifth single image 110A, and the ninth single image 112A is shown as an example image 94B. As an example of an image based on the first single image 108A, the fifth single image 110A, and the ninth single image 112A, there can be cited an image created by synthesizing (here, as an example, overlapping in units of pixels P1 corresponding to each other) the first single image 108A, the fifth single image 110A, and the ninth single image 112A.

[0160] In this way, when a RAW image 75A with the same image quality as the example image 94B is input into the learned model 106 generated by optimizing the model 98 through machine learning based on the training data 92B (taking images based on no more than three of the first single image 108A, the fifth single image 110A, the ninth single image 112A and the thirteenth single image 114A as the example image 94B), the learned model 106 can generate and output an image with higher image quality than the input RAW image 75A.

[0161] exist Fig. 9 In the example shown, training data 92A is shown, but this is only an example. Fig.11 As shown, training data 92C may be used instead of training data 92A. The difference between training data 92C and training data 92A is that correct answer image 96A is used (see also Fig.10 ) and example image 94C. Example image 94C is an example of the "second image" involved in the present invention. Example image 94C is an image obtained by the first thinned-out image 108Aa, the second thinned-out image 110Aa, the third thinned-out image 112Aa, and the fourth thinned-out image 114Aa. When N is set to an even number, the first thinned-out image 108Aa is an image in which the Nth line of the first single image 108A is thinned out, the second thinned-out image 110Aa is an image in which the Nth line of the fifth single image 110A is thinned out, the third thinned-out image 112Aa is an image in which the Nth line of the ninth single image 112A is thinned out, and the fourth thinned-out image 114Aa is an image in which the Nth line of the 13th single image 114A is thinned out.

[0162] exist Fig.11 In the example shown, as example image 94C, an image created by synthesizing (here, as an example, overlapping in units of pixels P1 whose positions correspond to each other) the first thinning image 108Aa, the second thinning image 110Aa, the third thinning image 112Aa, and the fourth thinning image 114Aa is shown.

[0163] Here, the Nth row thinning pattern is illustrated, but other thinning patterns may be used (for example, the Nth column thinning pattern or the Nth row and Nth column thinning pattern, etc.). Also, here, the first thinning image 108Aa, the second thinning image 110Aa, the third thinning image 112Aa, and the fourth thinning image 114Aa are illustrated, but this is only an example, and the example image 94C may be an image obtained by synthesizing (for example, overlapping in units of pixels P1 corresponding to each other) three or less of the first thinning image 108Aa, the second thinning image 110Aa, the third thinning image 112Aa, and the fourth thinning image 114Aa.

[0164] In this way, when the RAW image 75A of the same image quality as the example image 94C (for example, the same as the example image 94C) is input to the learned model 106 generated by optimizing the model 98 by performing machine learning based on the use of the image created by synthesizing the first thinning image 108Aa, the second thinning image 110Aa, the third thinning image 112Aa, and the fourth thinning image 114Aa as the training data 92C of the example image 94C, the RAW image 75A of the same image quality as the example image 94C is input. Fig.11 Similar to the example shown, in the case of a RAW image 75A in which the pixels P1 are thinned out using a thinning pattern, the learned model 106 can generate and output an image having higher quality than the input RAW image 75A.

[0165] exist Fig. 9 In the example shown, training data 92A is shown, but this is only an example. Fig.12 As shown, training data 92D may be used instead of training data 92A. Training data 92D is different from training data 92A in that correct answer image 96B and example question image 94D are used.

[0166] In the above embodiment, as the photoelectric conversion element 72, a photoelectric conversion element that does not have the function of detecting a phase difference is exemplified, but this is only an example. Fig.12 As shown, the photoelectric conversion element 72 can also be a photoelectric conversion element having a function of detecting a phase difference, that is, a photoelectric conversion element including a phase difference pixel P2 and a non-phase difference pixel P3. The phase difference pixel P2 is a pixel for detecting a phase difference. And, the phase difference pixel P2 is a G pixel. The non-phase difference pixel P3 is a pixel identical to the above-mentioned pixel P (that is, a pixel that does not detect a phase difference). In the photoelectric conversion element 72, the phase difference pixels P2 and the non-phase difference pixels P3 are regularly arranged in a state of maintaining the arrangement pattern of the R pixels, G pixels, and B pixels described in the above embodiment.

[0167] The first single image 108A, the fifth single image 110A, the ninth single image 112A, and the thirteenth single image 114A obtained by capturing images with the image sensor 20 having the photoelectric conversion element 72 configured in this manner include pixels P2a and P3a. The pixel P2a is a pixel corresponding to the pixel P2, and the pixel P3a is a pixel corresponding to the non-phase difference pixel P3. The first single image 108A, the fifth single image 110A, the ninth single image 112A, and the thirteenth single image 114A are images in which the pixel P3a is shifted to a position where the pixel P3a overlaps the pixel P2a when the first single image 108A, the fifth single image 110A, the ninth single image 112A, and the thirteenth single image 114A are synthesized (for example, when the first single image 108A, the fifth single image 110A, the ninth single image 112A, and the thirteenth single image 114A are superimposed).

[0168] Correct answer image 96B is an image created by synthesizing first single image 108A, fifth single image 110A, ninth single image 112A, and thirteenth single image 114A (e.g., overlapping first single image 108A, fifth single image 110A, ninth single image 112A, and thirteenth single image 114A). In this case, pixel P2a overlaps with pixel P3a between first single image 108A, fifth single image 110A, ninth single image 112A, and thirteenth single image 114A. Furthermore, R pixels, G pixels, and B pixels overlap.

[0169] Example image 94D is any one of first single image 108A, fifth single image 110A, ninth single image 112A, and thirteenth single image 114A. Alternatively, example image 94D may be an image obtained by synthesizing (for example, overlapping in units of pixels P1 corresponding to each other) three or less of first single image 108A, fifth single image 110A, ninth single image 112A, and thirteenth single image 114A.

[0170] By conducting Fig.12 The learned model 106 is optimized by machine learning of the training data 92D constructed as shown. As a result, even if the photoelectric conversion element 72 of the image sensor 20 mounted on the camera device 10 is a photoelectric conversion element configured with phase difference pixels P2 and non-phase difference pixels P3, when a RAW image 75A with the same image quality as the example image 94D (for example, a RAW image 75A in which pixels P2a and pixels P3a are mixed) is input to the learned model 106, the learned model 106 can generate and output an image with higher image quality than the input RAW image 75A.

[0171] As an example, Fig.13As shown, the example image 94 may also include a non-focus area 126 (i.e., an area that is not focused) and a focus area 128 (i.e., an area that is focused). When using such an example image 94, the correct answer image 96 also includes a position corresponding to the non-focus area 126 in the example image 94, and includes a focus area 132 at a position corresponding to the focus area 128 in the example image 94. Furthermore, the degree of high image quality is made different in the non-focus area 130 and the focus area 132. Fig.13 In the example shown, the degree of image quality improvement in the non-focus area 130 is smaller than that in the focus area 132 .

[0172] The learned model 106 is optimized by performing machine learning based on training data 92 including the example image 94 and the correct answer image 96 constructed in this way. As a result, when a RAW image 75A including an out-of-focus area and an in-focus area is input into the learned model 106, the learned model 106 can generate and output an image in which the image quality of the in-focus area is better than that of the out-of-focus area.

[0173] Here, an example is given in which the degree of image quality improvement of the non-focus area 130 is smaller than that of the focus area 132 . However, this is only an example, and the degree of image quality improvement of the focus area 132 may be smaller than that of the non-focus area 130 .

[0174] As an example, Fig.14 As shown, as the correct answer image 96, an image whose image quality is improved compared to a single image by a factor affecting image quality (hereinafter also referred to as a "factor") may be used, and as the example question image 94, an image whose image quality is reduced compared to the correct answer image 96 by the factor may be used. Fig.14 In the example shown, the image quality of the focused area 128 of the example question image 94 is reduced by the factor to be lower than the image quality of the focused area 132 of the correct answer image 96. Fig.14 The example image 94 shown is an image whose image quality is lower than that of the example image 94 whose image quality is improved due to the fact that the factor does not work.

[0175] As an example of the factors, focal length, aperture value, lens characteristics (for example, inherent characteristics that vary in lens settings (MTF, aberration and / or shading)), spacing characteristics between pixels P, grayscale correction function, gain correction function and noise reduction function can be cited. That is, the correct answer image 96 is an image created by multiple single images (multiple unit images), and the multiple single images are obtained by the focal length, aperture value, lens characteristics, spacing characteristics between pixels P, grayscale correction function, gain correction function and noise reduction function to achieve image quality above a certain level. The correct answer image 96 is created by multiple single images (multiple unit images) with high image quality through the focal length, aperture value, lens characteristics, spacing characteristics between pixels P, grayscale correction function, gain correction function and noise reduction function.

[0176] Fig.14 The example image 94 shown is an image created from multiple single images affected by the following: low image quality caused by focal length, low image quality caused by aperture value, low image quality caused by lens characteristics, low image quality caused by spacing characteristics between pixels P, low image quality caused by not using the grayscale correction function or reducing the grayscale correction function to a certain level, low image quality caused by not using the gain correction function or reducing the gain correction function to a certain level, and low image quality caused by not using the noise reduction function or reducing the noise reduction function to a certain level. In other words, Fig.14 The example image 94 shown can also be said to be an image of lower quality than the example image 94 created from multiple single images affected by the following: high image quality caused by focal length, high image quality caused by aperture value, high image quality caused by lens characteristics, high image quality caused by interval elimination characteristics between pixels P, high image quality caused by using grayscale correction function or increasing the use level of grayscale correction function to above a certain level, high image quality caused by using gain correction function or increasing the use level of gain correction function to above a certain level, and high image quality caused by using noise reduction function or increasing the use level of noise reduction function to above a certain level.

[0177] Furthermore, the example image 94 is an image of the quality of the RAW image 75A (or an image obtained by performing image processing on the RAW image 75A by the image processing engine 12) that is considered to be generated by the camera device 10. For example, the focal length, aperture value, lens characteristics, the spacing characteristics between pixels P, the grayscale correction function, the gain correction function, and the noise reduction function listed as factors have the same minimum setting values ​​(i.e., the setting values ​​for setting the image quality of the example image 94 to the minimum level) as the focal length, aperture value, lens characteristics, the spacing characteristics between pixels P, the grayscale correction function, the gain correction function, and the noise reduction function used for the camera device 10 (e.g., the setting values ​​for setting the image quality of the RAW image 75A to the minimum level). Here, focal length, aperture value, lens characteristics, spacing characteristics between pixels P, grayscale correction function, gain correction function and noise reduction function are exemplified, but as long as the image quality of the RAW image 75A is adjusted by color space conversion processing, brightness filtering processing, color difference processing and / or resizing processing, etc., performed by the camera 10, the image quality adjustment (i.e., color space conversion processing, brightness filtering processing, color difference processing and / or resizing processing, etc.) performed by the camera 10 on the RAW image 75A can also be performed on the example image 94. The image quality adjustment (i.e., color space conversion processing, brightness filtering processing, color difference processing and / or resizing processing, etc.) performed by the camera 10 on the RAW image 75A is included in the concept of factors.

[0178] And, in Fig.14 In the example shown, the example image 94 is a single image having false colors and false resolution that are previously believed to be possessed by the RAW image 75A generated by the imaging device 10 (for example, an image having the largest amount of false colors and false resolution determined by quantifying the amount of false colors and false resolution in advance by visual observation and / or computer simulation) or an image generated from a single image having false colors and false resolution that are previously believed to be possessed by the RAW image 75A generated by the imaging device 10. For example, an image generated from a single image having false colors and false resolution refers to an image obtained by processing the single image having false colors and false resolution. As an example of an image obtained by processing a single image having false colors and false resolution, there can be cited an image obtained by thinning out a single image having false colors and false resolution or an image file obtained by developing a single image having false colors and false resolution. Fig.14 In the example shown, the focus area 128 includes a false color image area and a false resolution image area. However, this is only an example, and the image area other than the focus area 128 may include a false color image area and / or a false resolution image area. In addition, false color and false resolution are included in the concept of factors.

[0179] By conducting Fig.14 The learned model 106 is optimized by machine learning of the training data 92 of the example image 94 and the correct answer image 96 of the structure shown, so that when the RAW image 75A whose image quality is reduced by the factor is input to the learned model 106, the learned model 106 can generate and output an image with higher image quality than the input RAW image 75A. Furthermore, compared with the case where a single image with the least false color and false resolution is used as the example image 94, when the RAW image 75A having a focus area including an image area including a false color and an image area including a false resolution is input to the learned model 106, the learned model 106 can generate and output an image in which the false color and false resolution of the area corresponding to the focus area of ​​the RAW image 75A are suppressed compared with the input RAW image 75A.

[0180] In addition, Fig.14 In the example shown, focal length, aperture value, lens characteristics, spacing characteristics between pixels P, grayscale correction function, gain correction function and noise reduction function are illustrated as examples of factors, but this is only an example. The factor can be one or more of the focal length, aperture value, lens characteristics, spacing characteristics between pixels P, grayscale correction function, gain correction function and noise reduction function. Furthermore, the factor can also be one or more of the focal length, aperture value, lens characteristics, spacing characteristics between pixels P, grayscale correction function, gain correction function, noise reduction function, color space conversion processing, brightness filtering processing, chromatic aberration processing, resizing processing, pseudo color and / or pseudo resolution. Furthermore, in Fig.14 In the example shown, a single image with false color and false resolution is illustrated, but this is only an example, and a single image with false color or false resolution may also be used. Fig.14 In the example shown, the image quality of the focus area 128 is lowered by a factor. However, this is only an example, and the image quality of image areas other than the focus area 128 may be lowered by a factor.

[0181] Furthermore, a plurality of learned models 106 generated by optimizing the model 98 by subjecting the model 98 to machine learning based on the training data 92 including the example question images 94 and the correct answer images 96 reflecting one factor and / or a plurality of factors different from each other may be mounted on the imaging device 10. In this case, the RAW image 75A having a high image quality obtained by the learned model 106 may be generated and output by inputting the RAW image 75A to one or more learned models 106 selected from the plurality of learned models 106 based on instructions and / or various conditions given by the user or the like.

[0182] In this way, one or more learned models 106 generated by optimizing the model 98 through machine learning based on the training data 92 including example images 94 and correct answer images 96 reflecting one factor and / or multiple different factors are mounted on the camera device 10. It is assumed that even if the camera device 10 does not have the above-mentioned function of adjusting the image quality, it is possible to obtain an image with higher image quality than the RAW image 75A input to the learned model 106 by inputting the RAW image 75A to the learned model 106.

[0183] In the above embodiment, an example is given in which the learned model 106 mounted on the camera 10 generates and outputs an image in RAW format, but this is only an example. For example, the learned model 106 mounted on the camera 10 may generate and output an image based on the RAW image 75A (for example, an image file in a predetermined file format). To achieve this purpose, Fig.15 In the example shown, training data 92E is used instead of training data 92. Training data 92E includes example question image 94E and correct answer image 96C. Fig.14 The difference between the example image 94 shown in FIG. 1 and the correct answer image 96C is that the correct answer image 96C is a JPEG file including an image in RGB format. Fig.14 The difference from the correct answer image 96 shown is that it is a JPEG file including an image in RGB format. Here, an image in RGB format is illustrated, but this is only an example, and an image in YCbCr format may also be used.

[0184] By conducting Fig.15 The learned model 106 is optimized by machine learning of the training data 92E having the structure shown, so that when an image in RGB format is input, the learned model 106 can generate and output an image in RGB format with higher image quality than the input RGB format image. Furthermore, when a JPEG file is input to the learned model 106, the learned model 106 can generate and output a JPEG file with higher image quality than the input JPEG file.

[0185] In addition, Fig.15 In the example shown, a JPEG file is illustrated, but this is only an example, and the image file may be in the form of TIFF, JPEG XR, MPEG, or AVI.

[0186] In the above embodiment, the example in which the first single image 108A is used when creating the example image 94 is given because the center closest to the center of gravity G1 among the centers 108A1 to 108D1 is the center 108A1 (see FIG. 108A). Figure 5), but this is only an example, and when creating the example image 94, it is sufficient to use no more than three of the first to fourth single images 108A to 108D.

[0187] In the above embodiment, the example in which the fifth single image 110A is used when creating the example image 94 is given because the center closest to the center of gravity G2 among the centers 110A1 to 110D1 is the center 110A1 (see FIG. 110A). Figure 5 ), but this is only an example, and when creating the example image 94, it is sufficient to use no more than three of the 5th to 8th single images 110A to 110D.

[0188] In the above embodiment, the example in which the ninth single image 112A is used when creating the example image 94 is given because the center closest to the center of gravity G3 among the centers 112A1 to 112D1 is the center 112A1 (see FIG. 112A). Figure 5 ), but this is only an example, and when creating the example image 94, it is sufficient to use no more than three of the 9th to 12th single images 112A to 112D.

[0189] In the above embodiment, the example in which the thirteenth single image 114A is used when creating the example image 94 is given because the center closest to the center of gravity G4 among the centers 114A1 to 114D1 is the center 114A1 (see FIG. 114A). Figure 5 ), but this is only an example, and when creating the example image 94, it is sufficient to use no more than three of the 13th to 16th single images 114A to 114D.

[0190] In the above embodiment, a single image in which R pixels, G pixels, and B pixels are regularly arranged is illustrated, but the single image may also be a monochrome image (for example, an R image composed only of multiple R images, a G image composed only of multiple G images, or a B image composed only of multiple B images).

[0191] In the above embodiment, an example of a form of high-definition processing performed by the processor 62 of the image processing engine 12 included in the camera device 10 is cited for explanation, but the present invention is not limited to this, and the device for high-definition processing may also be a device provided outside the camera device 10 such as a server. For example, the server may also be implemented by cloud computing. In addition, the server may also be implemented by network computing such as fog computing, edge computing, or grid computing. Here, a server is illustrated, but this is only an example, and at least one personal computer may be substituted for the server.

[0192] In the above embodiment, an example in which the high-quality image processing program 124 is stored in the storage device 64 is given for explanation, but the present invention is not limited thereto. For example, the high-quality image processing program 124 may also be stored in a portable computer-readable non-temporary storage medium such as an SSD or a USB memory. The high-quality image processing program 124 stored in the non-temporary storage medium is installed in the image processing engine 12 of the camera device 10. The processor 62 performs high-quality image processing according to the high-quality image processing program 124.

[0193] Alternatively, the image quality enhancement program 124 may be stored in a storage device such as another computer or server device connected to the imaging device 10 via a network, and downloaded in response to a request from the imaging device 10 to be installed in the image processing engine 12 .

[0194] In addition, the image quality enhancement program 124 does not need to be stored in its entirety in a storage device or the storage device 64 of another computer or server device connected to the imaging device 10, and a portion of the image quality enhancement program 124 may be stored. In addition, the image quality enhancement program 124 is described here, but the learning program 90 may be described in the same manner.

[0195] and, Figure 1 and Figure 2 The imaging device 10 shown has the image processing engine 12 built therein, but the present invention is not limited thereto, and for example, the image processing engine 12 may be provided outside the imaging device 10 .

[0196] In the above embodiment, the image processing engine 12 is exemplified, but the present invention is not limited thereto, and a device including an ASIC, an FPGA, and / or a PLD may be used instead of the image processing engine 12. Furthermore, a combination of a hardware structure and a software structure may be used instead of the image processing engine 12.

[0197] As hardware resources for performing the high-quality image processing and / or learning processing described in the above embodiments, various processors shown below can be used. As a processor, for example, a CPU can be cited, which is a general-purpose processor that performs the function of hardware resources for performing high-quality image processing and / or learning processing by executing software (i.e., a program). In addition, as a processor, for example, a dedicated circuit can be cited, which is a processor with a circuit structure specially designed for performing specific processing, such as FPGA, PLD or ASIC. Regardless of the type of processor, a memory is built in or connected, and regardless of the type of processor, the high-quality image processing and / or learning processing is performed by using the memory.

[0198] The hardware resource for performing high-definition processing and / or learning processing may be composed of one of these various processors, or may be composed of a combination of two or more processors of the same type or different types (for example, a combination of multiple FPGAs or a combination of a CPU and an FPGA). Furthermore, the hardware resource for performing high-definition processing and / or learning processing may also be a single processor.

[0199] As an example of a system composed of one processor, first, there is a form in which a processor is composed of a combination of one or more CPUs and software, and the processor performs the function of a hardware resource for performing high-definition processing and / or learning processing. Secondly, there is a form represented by SoC, etc., in which a processor is used to realize the function of the entire system including multiple hardware resources for performing high-definition processing and / or learning processing by one IC chip. In this way, high-definition processing and / or learning processing are realized by using one or more of the above-mentioned various processors as hardware resources.

[0200] Furthermore, as the hardware structure of these various processors, more specifically, a circuit formed by combining circuit elements such as semiconductor elements can be used. In addition, the above-mentioned high-definition processing and / or learning processing is only an example. Therefore, it is of course possible to delete unnecessary steps, add new steps, or change the processing order within the scope of the main purpose.

[0201] The above-mentioned records and illustrated contents are detailed descriptions of the parts involved in the present invention, and are only examples of the present invention. For example, the description of the above-mentioned structure, function, action and effect is a description of an example of the structure, function, action and effect of the parts involved in the present invention. Therefore, it is of course possible to delete unnecessary parts, add new elements or replace the above-mentioned records and illustrated contents without departing from the gist of the present invention. In addition, in order to avoid trouble and facilitate understanding of the parts involved in the present invention, the description of technical common sense that does not need to be particularly explained when implementing the present invention is omitted in the above-mentioned records and illustrated contents.

[0202] All documents, patent applications, and technical standards described in this specification are incorporated herein by reference to the same extent as if each document, patent application, and technical standard was specifically and individually indicated to be incorporated by reference.

[0203] Regarding the above embodiment, the following additional notes are also disclosed.

[0204] (Note 1)

[0205] A training data used for machine learning of a model, which has:

[0206] The correct answer image is a composite of multiple single images; and

[0207] Example images,represent the multiple single images mentioned above.

[0208] (Note 2)

[0209] A learned model, which is generated by optimizing the model by performing the above-mentioned machine learning on the above-mentioned model using the training data described in Note 1.

[0210] (Note 3)

[0211] A program (e.g., image quality enhancement program 124) is used to enable a computer (e.g., image processing engine 12) to perform image quality enhancement processing, the image quality enhancement processing comprising the following steps:

[0212] Inputting a camera image captured by an image sensor into the learned model described in Supplementary Note 2; and

[0213] The inference result output from the learned model based on the input of the camera image is obtained.

[0214] (Note 4)

[0215] A program (e.g., learning program 90) that causes a computer (e.g., learning device 79) to execute a learning process that generates a learned model by performing machine learning on the model using training data including correct answer images and example question images,

[0216] The correct answer image above is an image synthesized from multiple single images.

[0217] The above example images are images representing the above multiple single images.

[0218] The above learning process includes the following steps:

[0219] Input the above example image to the above model;

[0220] Outputting an evaluation object image from the model according to the input of the example image; and

[0221] The model is optimized based on the comparison result between the evaluation object image and the correct answer image.

[0222] (Note 5)

[0223] A computer-readable non-temporary storage medium stores a program (e.g., an image quality enhancement program 124) for causing a computer (e.g., an image processing engine 12) to perform an image quality enhancement process including the following steps:

[0224] Inputting a camera image captured by an image sensor into the learned model described in Supplementary Note 2; and

[0225] The inference result output from the learned model based on the input of the camera image is obtained.

[0226] (Note 6)

[0227] A non-transitory storage medium, which is a computer-readable non-transitory storage medium storing a program (e.g., a learning program 90) that causes a computer (e.g., a learning device 79) to execute a learning process that generates a learned model by performing machine learning on the model using training data including correct answer images and example question images,

[0228] The correct answer image above is an image synthesized from multiple single images.

[0229] The above example images are images representing the above multiple single images.

[0230] The above learning process includes the following steps:

[0231] Input the above example image to the above model;

[0232] Outputting an evaluation object image from the model according to the input of the example image; and

[0233] The model is optimized based on the comparison result between the evaluation object image and the correct answer image.

[0234] Explanation of symbols

[0235] 10, 500-camera device, 12-image processing engine, 16-camera device body, 18-interchangeable lens, 18A-focus ring, 20-image sensor, 22-release button, 24-dial, 26-indicator key, 28-display, 30-touch panel, 32-touch screen display, 36-control device, 37-first actuator, 38-second actuator, 39-third actuator, 40-imaging lens, 40A-objective lens, 40B-focus lens, 40C-zoom lens, 40D-aperture, 40D1-opening, 40D2-aperture blades, 44-system controller, 46-image memory, 48-UI system device, 50-external I / F, 52-communication I / F F, 54-photoelectric conversion element driver, 62, 80-processor, 64, 82-storage device, 66, 84-memory, 68-bus, 70-input / output interface, 72-photoelectric conversion element, 72A-light receiving surface, 74-A / D converter, 75A-RAW image, 75A1-high resolution image, 75B-image file, 76-receiving device, 78-hard key unit, 79-learning device, 90-learning program, 92, 92A, 92B, 92C, 92D, 92E-training data, 94, 94A, 94B, 94C, 94D, 94E-example image, 96, 96A, 96B, 96C-correct answer image, 98-model, 100 - comparison image, 102 - error, 104 - adjustment value, 106 - learned model, 108A - first single image, 108A1, 110A1, 112A1, 114A1, 108B1, 110B1, 112B1, 114B1, 108C1, 110C1, 112C1, 114C1, 108D1, 110D1, 112D1, 114D1 - center, 108B - second single image, 108C - third single image, 108D - fourth single image, 110A - fifth single image, 110B - sixth single image, 110C - seventh single image, 110D - eighth single image, 112A - ninth single image, 112B-10th single image, 112C-11th single image, 112D-12th single image, 114A-13th single image, 114B-14th single image, 114C-15th single image, 114D-16th single image, 116-1st micronized image, 118-2nd micronized image, 120-3rd micronized image, 122-4th micronized image, 124-high image quality program, 126, 130-non-focus area, 128, 132-focus area, 502-pixel shifting device, G1, G2, G3, G4-center of gravity, OA-optical axis, P, P1, P2a-pixel, P2-phase difference pixel, P3-non-phase difference pixel.

Claims

1. A training data used for machine learning of a model, having: The correct answer image is a composite of multiple single images; and The example image represents the multiple single images.

2. The training data according to claim 1, wherein: The correct answer image is an image obtained by synthesizing the plurality of single images to obtain a higher resolution.

3. The training data according to claim 1, wherein: The correct answer image is an image with a higher resolution than the example question image.

4. The training data according to claim 1, wherein: The correct answer image is an image having more pixels than the example question image.

5. The training data according to claim 1, wherein: The correct answer image is an image with a higher visual resolution than the example question image.

6. The training data according to claim 2, wherein: The plurality of single images are respectively pixel-shifted images.

7. The training data according to claim 6, wherein: The multiple single images are respectively images after 1 / 2 pixel shift.

8. The training data according to claim 1, wherein: The multiple single images are respectively pixel-shifted images, The correct answer image is an image with high image quality obtained by synthesizing the plurality of single images.

9. The training data according to claim 8, wherein: The plurality of single images are regularly configured with pixels of different colors, The plurality of single images are images in which pixels are shifted to positions where the pixels of different colors overlap.

10. The training data according to claim 5, wherein: The plurality of single images are obtained by photographing a first image sensor including phase difference pixels and non-phase difference pixels. Each of the plurality of single images is an image where pixels are shifted to a position where a pixel corresponding to the non-phase difference pixel overlaps a pixel corresponding to the phase difference pixel.

11. The training data according to claim 1, wherein: The correct answer image is an image whose image quality is improved compared to the single image by factors affecting image quality. The example question image is an image whose image quality is lowered compared to the correct answer image due to the factor.

12. The training data according to claim 11, wherein: The factors are focal length, aperture value, lens characteristics, pixel spacing characteristics, grayscale correction function, gain correction function and / or noise reduction function.

13. The training data according to claim 1, wherein: The correct answer image and the example question image include a focus area and a non-focus area, The correct answer image is an image in which the degree of image quality improvement is different between the focus area and the non-focus area.

14. The training data according to claim 13, wherein: The correct answer image is an image in which the degree of image quality enhancement in the non-focus area is smaller than that in the focus area.

15. The training data according to any one of claims 1 to 14, wherein: The correct answer image and the example question image are RAW images.

16. The training data according to claim 1, wherein: The correct answer image and the example question image are images based on RAW images.

17. The training data according to claim 1, wherein: The correct answer image and the example question image are images in RGB format or images in YCbCr format.

18. The training data according to claim 1, wherein: The example image is a first image based on the single images whose number is smaller than the number of the single images among the multiple single images.

19. The training data according to claim 1, wherein: The example image is a second image obtained by thinning out pixels of the single images whose number is less than the number of the single images among the plurality of single images.

20. The training data according to claim 1, wherein: The plurality of single images are images captured by the second image sensor from different imaging positions and are shifted by 1 / 2 pixels. The example image is a single image having a center closest to the center of gravity when the multiple single images are overlapped, among the multiple single images.

21. The training data according to claim 1, wherein: The example image is a single image with false color and / or false resolution among the multiple single images, or an image generated based on the single image with false color and / or false resolution among the multiple single images.

22. A learned model generated by optimizing the model by performing the machine learning on the model using the training data of any one of claims 1 to 21.

23. A camera device comprising: the first processor; and The 3rd image sensor, The first processor performs the following processing: inputting a captured image captured by the third image sensor into the learned model according to claim 22; and An inference result output from the learned model based on the input of the camera image is obtained.

24. A learning device comprising a second processor, The second processor optimizes the model by causing the model to perform the machine learning using the training data of any one of claims 1 to 21.

25. A method for creating training data for machine learning of a model, The training data includes correct answer images and example question images. The method for creating training data comprises the following steps: Creating the correct answer image by synthesizing a plurality of single images; and An image representing the plurality of single images is created as the example image.

26. A method for generating a learned model, wherein the learned model is generated by performing machine learning on the model using training data including correct answer images and example question images, The correct answer image is an image synthesized from multiple single images. The example image is an image representing the multiple single images, The method for generating the learned model comprises the following steps: Inputting the example image into the model; Outputting an evaluation object image by the model according to the input of the example image; and The model is optimized according to the comparison result between the evaluation object image and the correct answer image.

Images