Training data, learned model, imaging device, learning device, method for creating training data, and method for generating learned model

Abstract

The invention provides training data, a learned model, an imaging device, a learning device, a method for creating the training data, and a method for generating the learned model, wherein the training data enables the learned model to generate an image having a bit depth deeper than that of an input image. The training data is used for machine learning of the model. The training data includes a correct answer image and an example image having a bit depth shallower than that of the correct answer image.

Description

Technical Field

[0001] The present invention relates to training data, a learned model, a camera device, a learning device, a method for creating training data, and a method for generating a learned model. Background Art

[0002] Patent document 1 discloses an image transceiver system. In the image transceiver system described in Patent document 1, at least one of the single or multiple transmitting devices has a machine learning unit that generates model data through machine learning, and the model data is used to generate an improved image closer to the original image based on a low-bit-rate encoded image that encodes the original image to a low bit rate. In addition, in the image transceiver system described in Patent document 1, at least one of the single or multiple transmitting devices has a transmitting unit that transmits the low-bit-rate encoded image and the model data to the outside of the device. In the image transceiver system described in Patent document 1, the receiving device has an improved image generating unit that generates an improved image of the low-bit-rate encoded image based on the received low-bit-rate encoded image and the model data.

[0003] Patent document 2 discloses an image encoding device for encoding an image. The image encoding device described in Patent document 2 includes: a classification unit that classifies pixels constituting an image into predetermined categories according to their properties; a mapping coefficient storage unit that stores predetermined mapping coefficients by category; an operation unit that calculates correction data for correcting a pixel of interest in the image by performing a predetermined operation using the pixel of interest in the image and the mapping coefficient corresponding to the category of the pixel of interest; and a restriction unit that restricts the level of the correction data as the encoded data obtained by encoding the image.

[0004] In the image encoding device described in Patent Document 2, the mapping coefficient has the following characteristics. That is, the mapping coefficient is generated by learning using the learning image data. Alternatively, the mapping coefficient is obtained by learning in such a way that the prediction error of the prediction result of predicting the original image based on the encoding data relative to the original image becomes the minimum. Alternatively, the mapping coefficient is obtained by learning in such a way that the prediction error of the prediction result of predicting the original image based on the encoding data relative to the original image becomes less than a specified value. Alternatively, the mapping coefficient is obtained by classifying the pixels constituting the learning image into any one of the categories according to their properties, calculating the learning correction data that corrects the learning pixel of interest by performing a specified operation using the learning pixel of interest in the learning image and the specified coefficient corresponding to the category of the learning pixel of interest, finding the learning restriction data that restricts the level of the learning correction data, predicting the prediction value of the learning image based on the learning restriction data, calculating the prediction error of the prediction value of the learning image relative to the learning image, changing the specified coefficient based on the prediction error, repeating this step until the specified coefficient becomes the optimal value, and obtaining the specified coefficient of the optimal value. Alternatively, the mapping coefficients are obtained using appropriate learning correction data and a learning image, wherein the appropriate learning correction data is obtained as follows: compressing the learning image by reducing the number of pixels of the learning image, limiting the level of the compressed learning data obtained by compressing the learning image, correcting the compressed learning data of the limited level to output the learning correction data, predicting the learning image based on the learning correction data and outputting its predicted value, calculating the prediction error of the predicted value of the learning image with respect to the learning image, determining the appropriateness of the learning correction data based on the prediction error, and repeating this step until the learning correction data becomes appropriate.

[0005] Patent Document 1: Japanese Patent Application Publication No. 2020-109897

[0006] Patent Document 2: Japanese Patent Application Publication No. 2006-109499 Summary of the invention

[0007] One embodiment of the present invention provides training data, a learned model, a camera device, a learning device, a method for creating training data, and a method for generating a learned model that enable a learned model to generate an image with a bit depth deeper than an input image.

[0008] The first method involved in the present invention is training data, which is used for machine learning of a model and includes: a correct answer image; and an example question image whose bit depth is shallower than that of the correct answer image.

[0009] According to a second aspect of the present invention, in the training data according to the first aspect, the correct answer image is an image obtained by synthesizing a plurality of single images.

[0010] According to a third aspect of the present invention, in the training data according to the second aspect, the example image is a representative image of a plurality of single images.

[0011] According to a fourth aspect of the present invention, in the training data according to the second aspect, the example image is an image having a shallower bit depth than the plurality of single images.

[0012] According to a fifth aspect of the present invention, in the training data according to any one of the first to fourth aspects, the single image is a shifted image.

[0013] In the sixth mode of the present invention, in the training data involved in the first mode, the correct answer image is an image obtained by synthesizing the processing object area and the corresponding area when a corresponding area corresponding to the arrangement pattern in the shifted image and the processing object area in the image exists in the image; or an image obtained by synthesizing the processing object pixel and the adjacent pixels when adjacent pixels of the same color that are regularly adjacent to the processing object pixel in the shifted image exist in the image.

[0014] According to a seventh aspect of the present invention, in the training data according to the sixth aspect, the example image is an image before bit shift or an image generated based on the image before bit shift.

[0015] According to an eighth aspect of the present invention, in the training data according to any one of the first to seventh aspects, the correct answer image and the example question image are images captured by a first camera device.

[0016] According to a ninth aspect of the present invention, in the training data according to any one of the first to eighth aspects, the correct answer image and the example question image are images in RAW format.

[0017] The tenth method involved in the present invention is that in the training data involved in any one of the first to eighth methods, the model inputs and outputs images in RAW format, and the correct answer image is in Log format. The model is optimized by converting the RAW format image output from the model into an image in Log format and comparing it with the correct answer image.

[0018] The 11th mode involved in the present invention is that in the training data involved in any one of the 1st to 8th modes, the model outputs an image in a second file format other than RAW format in response to the input of an image in a first file format other than RAW format, the example image is in the first file format, and the correct answer image is in the second file format.

[0019] In the twelfth mode of the present invention, in the training data involved in any one of the first to eleventh modes, the example image is divided into a plurality of first image regions, and the correct answer image is divided into a plurality of second image regions corresponding to the plurality of first image regions.

[0020] According to a thirteenth aspect of the present invention, in the training data according to the twelfth aspect, labels capable of identifying image quality are given to the plurality of second image regions.

[0021] According to a fourteenth aspect of the present invention, in the training data according to the twelfth aspect or the thirteenth aspect, the example image is an image assumed to be an image captured by a second camera device, and the image quality is determined according to the characteristics of the second camera device.

[0022] According to a fifteenth aspect of the present invention, in the training data according to the fourteenth aspect, the second imaging device includes an optical system, and the characteristics include an image height related to the optical system.

[0023] The sixteenth aspect of the present invention is a learned model, which is obtained by optimizing the model by performing machine learning on the model using the training data involved in any one of the first to fifteenth aspects.

[0024] The 17th method involved in the present invention is a camera device, which comprises: a first processor; and an image sensor, the first processor inputs a camera image captured by the image sensor to the learned model involved in the 16th method, and obtains an inference result output from the learned model based on the input of the camera image.

[0025] An eighteenth aspect of the present invention is a learning device including a second processor configured to optimize a model by performing machine learning on the model using the training data according to any one of the first to fifteenth aspects.

[0026] The 19th method involved in the present invention is a method for creating training data, which is used for machine learning of a model. The training data includes a correct answer image and an example question image. The method for creating training data includes the following steps: creating a correct answer image; and creating an example question image with a bit depth shallower than the correct answer image.

[0027] The 20th method involved in the present invention is a method for generating a learned model, which generates the learned model by performing machine learning on the model using training data including a correct answer image and an example question image, wherein the example question image is an image having a shallower bit depth than the correct answer image, and the method for generating the learned model includes the following steps: inputting the example question image into the model; outputting an evaluation object image from the model based on the input of the example question image; and optimizing the model based on the comparison result between the evaluation object image and the correct answer image. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a schematic structural diagram showing an example of the overall structure of the imaging device.

[0029] Figure 2 This is a schematic configuration diagram showing an example of the hardware configuration of the optical system and the electrical system of the imaging device.

[0030] Figure 3 This is a schematic diagram showing an example of the content of a RAW image obtained by photographing with an imaging device.

[0031] Figure 4 This is a schematic configuration diagram showing an example of the configuration of a learning device.

[0032] Figure 5 This is a conceptual diagram showing an example of a method for creating training data.

[0033] Figure 6 This is a block diagram showing an example of the function of the image processing engine.

[0034] Figure 7 This is a flowchart showing an example of the flow of the learning process performed by the learning device.

[0035] Figure 8 This is a flowchart showing an example of the flow of image quality improvement processing performed by the imaging device.

[0036] Fig. 9 This is a conceptual diagram showing a first modified example of the method for creating training data.

[0037] Fig.10 This is a conceptual diagram showing a second variation of the method for creating training data.

[0038] Fig.11 This is a conceptual diagram showing a third variation of the method for creating training data.

[0039] Fig.12 This is a conceptual diagram showing a fourth variation of the method for creating training data.

[0040] Fig.13 This is a conceptual diagram showing a fifth variation of the method for creating training data.

[0041] Fig.14 This is a conceptual diagram showing a first variation of training data.

[0042] Fig.15 This is a conceptual diagram showing a first modified example of the image quality improvement process.

[0043] Fig.16 This is a conceptual diagram showing a second variation of training data.

[0044] Fig.17 This is a conceptual diagram showing a third variation of training data.

[0045] Fig.18 This is a conceptual diagram showing a second modified example of the image quality improvement process.

[0046] Fig.19 This is a conceptual diagram showing the fourth variation of training data.

[0047] Fig. 20 This is a conceptual diagram showing a third modified example of the image quality improvement process.

[0048] Fig.21 This is a conceptual diagram showing the fifth variation of training data.

[0049] Fig. 22 This is a conceptual diagram showing a fourth modified example of the high image quality processing. DETAILED DESCRIPTION

[0050] Hereinafter, an example of an implementation of the training data, the learned model, the camera device, the learning device, the training data creation method, and the learned model generation method involved in the present invention is described with reference to the accompanying drawings.

[0051] First, the words and phrases used in the following description are explained.

[0052] CPU is the abbreviation of “Central Processing Unit”. GPU is the abbreviation of “Graphics Processing Unit”. GPGPU is the abbreviation of “General-purpose computing on graphics processing units”. APU is the abbreviation of “Accelerated Processing Unit”. TPU is the abbreviation of “Tensor processing unit”. NVM is the abbreviation of “Non-volatile memory”. RAM is the abbreviation of “Random Access Memory”. IC is the abbreviation of “Integrated Circuit”. ASIC is the abbreviation of “Application Specific Integ rated Circuit”. PLD is the abbreviation of “Programmable Logic Device”. FPGA is the abbreviation of “Field-Programmable Gate Array”. SoC is the abbreviation of “System-on-a-chip”. SSD is the abbreviation of “Solid State Drive”. USB is the abbreviation of “Universal Serial Bus”. HDD is the abbreviation of “Hard Disk Drive”. EEPROM is the abbreviation of “Electrically Erasable and Programmable ReadOnly Memory”. EL is the abbreviation of “Electro-Luminescence”. I / F is the abbreviation of “Interface”. UI is the abbreviation of “User Interface”. fps is the abbreviation of “frame per second”. MF is the abbreviation of “Manual Focus”. AF is the abbreviation of “Auto Focus”. CMOS is the abbreviation of “Complementary Metal Oxide Semiconductor”.CCD is the abbreviation of “Charge Coupled Device”. AI is the abbreviation of “Artificial Intelligence”. A / D is the abbreviation of “Analog / Digital”. FIR is the abbreviation of “Finite Impulse Response”. IIR is the abbreviation of “Infinite Impulse Response”. JPEG is the abbreviation of “Joint Photographic Experts Group”. TIFF is the abbreviation of “Tagged Image File Format”. JPEG XR is the abbreviation of “Joint Photographic Experts Group Extended Range”. MPEG is the abbreviation of “Moving Picture Expert Group”. AVI is the abbreviation of “Audio Video Interleave”.

[0053] In the following description, a processor with a symbol (hereinafter referred to as a "processor") can be a physical or virtual computing device, or a combination of multiple physical or virtual computing devices. In addition, a processor can be a computing device, or a combination of multiple computing devices. As an example of a computing device, a CPU, a GPU, a GPGPU, an APU, or a TPU can be cited.

[0054] In the following description, a memory with a symbol is a memory such as a RAM that temporarily stores information and is used as a working memory by a processor.

[0055] In the following description, the storage device with a symbol is one or more non-volatile storage devices that store various programs and various parameters, etc. As an example of a non-volatile storage device, a flash memory, a magnetic disk, or a magnetic tape, etc., can be cited. And as another example of a storage device, a cloud storage device can be cited.

[0056] In the following embodiments, the external I / F with a symbol controls the exchange of various information between multiple devices connected to each other. As an example of an external I / F, a USB interface can be cited. The external I / F can also be applied to a communication I / F including a communication processor and an antenna. The communication I / F controls the communication between multiple computers. As an example of a communication standard applicable to the communication I / F, wireless communication standards including 5G, Wi-Fi (registered trademark) or Bluetooth (registered trademark) can be cited.

[0057] In the following embodiments, "A and / or B" has the same meaning as "at least one of A and B". That is, "A and / or B" means that it can be only A, only B, or a combination of A and B. In addition, in this specification, when three or more items are expressed in association with "and / or", the same concept as "A and / or B" also applies.

[0058] Figure 1 FIG. 1 is a schematic structural diagram showing an example of the overall structure of a camera device. Figure 1 As shown, the imaging device 10 is a device for photographing a subject, and includes an image processing engine 12, an imaging device body 16, and an interchangeable lens 18. Here, the imaging device 10 is an example of the "imaging device" and "second imaging device" according to the present invention.

[0059] The image processing engine 12 is built into the camera body 16, and controls the entire camera 10. The interchangeable lens 18 is replaceably mounted on the camera body 16. The interchangeable lens 18 is provided with a focus ring 18A. The focus ring 18A is operated by a user of the camera 10 (hereinafter, simply referred to as a "user") or the like when the user manually adjusts the focus of a subject through the camera 10.

[0060] exist Figure 1 In the example shown, a lens-interchangeable digital camera is shown as an example of the imaging device 10. However, this is only an example, and a lens-fixed digital camera may be used, or a digital camera built into various electronic devices such as a smart device, a wearable terminal, an endoscope device, a cell observation device, an ophthalmic observation device, or a surgical microscope may be used.

[0061] The camera body 16 is provided with an image sensor 20. The image sensor 20 is an example of the "image sensor" involved in the present invention. The image sensor 20 is a CMOS image sensor. The image sensor 20 captures a camera range including at least one subject. When the interchangeable lens 18 is mounted on the camera body 16, subject light representing the subject is transmitted through the interchangeable lens 18 and formed on the image sensor 20, and image data representing the image of the subject is generated by the image sensor 20.

[0062] In the present embodiment, a CMOS image sensor is exemplified as the image sensor 20 , but the present invention is not limited thereto. For example, even if the image sensor 20 is another type of image sensor such as a CCD image sensor, the present invention is also applicable.

[0063] A release button 22 and a dial 24 are provided on the upper surface of the camera body 16. The dial 24 is operated when setting the operation mode of the camera system and the operation mode of the playback system, and by operating the dial 24, the camera mode, the playback mode, and the setting mode are selectively set as the operation mode in the camera 10. The camera mode is an operation mode in which the camera 10 performs shooting. The playback mode is an operation mode in which images (for example, still images and / or moving images) obtained by shooting for recording in the camera mode are played. The setting mode is an operation mode set for the camera 10 when setting various setting values ​​used in control related to shooting.

[0064] The release button 22 functions as a shooting preparation indication unit and a shooting indication unit, and can detect pressing operations in two stages: a shooting preparation indication state and a shooting indication state. The shooting preparation indication state refers to, for example, a state in which the button is pressed from the standby position to an intermediate position (half-pressed position), and the shooting indication state refers to a state in which the button is pressed to a final pressed position (full-pressed position) beyond the intermediate position.

[0065] In addition, hereinafter, the "state of being pressed from the standby position to the half-pressed position" is referred to as the "half-pressed state", and the "state of being pressed from the standby position to the fully-pressed position" is referred to as the "fully-pressed state". Depending on the structure of the imaging device 10, the imaging preparation instruction state may be a state in which the user's finger is in contact with the release button 22, and the imaging instruction state may be a state in which the user's finger is in contact with the release button 22 and then released.

[0066] The back of the camera body 16 is provided with a pointer key 26 and a touch panel display 32. The touch panel display 32 includes a display 28 and a touch panel 30 (see also Figure 2 As an example of the display 28, an EL display (for example, an organic EL display or an inorganic EL display) can be cited. The display 28 may also be another type of display such as a liquid crystal display instead of an EL display.

[0067] The display 28 displays images and / or character information, etc. When the camera device 10 is in the camera mode, the display 28 is used to display a live preview image, which is obtained by shooting for live preview images (i.e., continuous shooting). Here, the "live preview image" refers to a dynamic image for display based on image data obtained by shooting by the image sensor 20. The shooting performed to obtain the live preview image (hereinafter also referred to as "shooting for live preview images") is performed, for example, at a frame rate of 60 fps. 60 fps is only an example, and a frame rate of less than 60 fps may be used, or a frame rate of more than 60 fps may be used.

[0068] When the camera 10 is instructed to shoot a still image via the release button 22, the display 28 is used to display a still image obtained by shooting a still image. In addition, the display 28 is used to display a playback image when the camera 10 is in a playback mode. Furthermore, when the camera 10 is in a setting mode, the display 28 is used to display a menu screen for selecting various menus and a setting screen for setting various setting values ​​used in control related to shooting.

[0069] The touch panel 30 is a transmissive touch panel, which is superimposed on the surface of the display area of ​​the display 28. The touch panel 30 receives instructions from the user by detecting contact with an indicator such as a finger or a stylus. In addition, for the sake of convenience, the above-mentioned "full press state" also includes a state in which the user presses a soft key for starting shooting via the touch panel 30.

[0070] In this embodiment, as an example of the touch screen display 32, an external touch screen display in which the touch panel 30 is superimposed on the surface of the display area of ​​the display 28 is cited, but this is only an example. For example, as the touch screen display 32, an external or internal touch screen display may also be applied.

[0071] The instruction key 26 accepts various instructions. Here, the “various instructions” include, for example, instructions for displaying a menu screen, instructions for selecting one or more menus, instructions for confirming selected content, instructions for deleting selected content, various instructions such as zooming in, zooming out, and frame forwarding, etc. Moreover, these instructions can also be performed through the touch panel 30.

[0072] Figure 2 FIG. 1 is a schematic diagram showing an example of the hardware configuration of the optical system and the electrical system of the imaging device. Figure 2 As shown in FIG. 1 , the image sensor 20 includes a photoelectric conversion element 72. The photoelectric conversion element 72 has a light receiving surface 72A. The photoelectric conversion element 72 is arranged in the imaging device body 16 in such a manner that the center of the light receiving surface 72A coincides with the optical axis OA (see also FIG. 1 ). Figure 1 ). The photoelectric conversion element 72 has a plurality of photosensitive pixels arranged in a matrix, and the light receiving surface 72A is formed by a plurality of photosensitive pixels. Each photosensitive pixel has a microlens (not shown). Each photosensitive pixel is a physical pixel having a photodiode (not shown), which performs photoelectric conversion on the received light and outputs an electrical signal corresponding to the amount of received light.

[0073] Furthermore, color filters (not shown) of the three primary colors of light (i.e., red (hereinafter, also referred to as "R"), green (hereinafter, also referred to as "G"), or blue (hereinafter, also referred to as "B")) are arranged in a predetermined pattern on the plurality of photosensitive pixels. In the present embodiment, the Bayer arrangement is used as an example of a predetermined pattern arrangement. However, the Bayer arrangement is only an example, and the present invention is also applicable even if the predetermined pattern arrangement is another type of pattern arrangement such as a G stripe R / G complete grid, an X-Trans (registered trademark) arrangement, or a honeycomb arrangement.

[0074] In addition, hereinafter, for the convenience of explanation, the photosensitive pixel having a microlens and an R color filter is referred to as an R pixel, the photosensitive pixel having a microlens and a G color filter is referred to as a G pixel, and the photosensitive pixel having a microlens and a B color filter is referred to as a B pixel. Also, hereinafter, for the convenience of explanation, the electrical signal output from the R pixel of the photosensitive pixel is referred to as an "R signal", the electrical signal output from the G pixel of the photosensitive pixel is referred to as a "G signal", and the electrical signal output from the B pixel of the photosensitive pixel is referred to as a "B signal". Also, hereinafter, for the convenience of explanation, the R signal, the G signal, and the B signal are also referred to as "RGB color signals". Also, hereinafter, for the convenience of explanation, the R pixels, the G pixels, and the B pixels constituting the RAW image 75A generated based on the RGB color signals are also referred to as "R pixels", "G pixels", and "B pixels". Here, R pixels, G pixels, and B pixels are illustrated, but this is merely an example, and when colors other than R, G, and B (i.e., colors other than the primary colors) are also regularly arranged in the color filter together with R, G, and B, the RAW image also includes pixels of colors other than R pixels, G pixels, and B pixels. Even in this case, the present invention is applicable.

[0075] The interchangeable lens 18 includes an imaging lens 40. The imaging lens 40 includes an objective lens 40A, a focusing lens 40B, a zoom lens 40C, and an aperture 40D. The objective lens 40A, the focusing lens 40B, the zoom lens 40C, and the aperture 40D are arranged in order along the optical axis OA from the subject side (object side) to the imaging device body 16 side (image side).

[0076] Furthermore, the interchangeable lens 18 includes a control device 36, a first actuator 37, a second actuator 38, and a third actuator 39. The control device 36 controls the entire interchangeable lens 18 according to instructions from the camera body 16. The control device 36 is, for example, a device having a computer including a processor, a storage device, and a memory. The storage device of the control device 36 is, for example, an EEPROM. Various programs and various parameters are stored in the storage device of the control device 36. The memory of the control device 36 is, for example, a RAM, which temporarily stores various information and is used as a working memory. In the control device 36, the processor reads out necessary programs from the storage device and executes the various programs read out on the memory, thereby controlling the entire imaging lens 40.

[0077] In addition, here, as an example of the control device 36, a device having a computer is listed, but this is only an example, and a device including an ASIC, FPGA and / or PLD may also be applied. In addition, as the control device 36, for example, a device implemented by a combination of a hardware structure and a software structure may also be used.

[0078] The first actuator 37 includes a focusing slide mechanism (not shown) and a focusing motor (not shown). The focusing slide mechanism is provided with a focusing lens 40B so as to be slidable along the optical axis OA. The focusing motor is connected to the focusing slide mechanism, and the focusing slide mechanism is operated by receiving power from the focusing motor, thereby moving the focusing lens 40B along the optical axis OA.

[0079] The second actuator 38 includes a zoom sliding mechanism (not shown) and a zoom motor (not shown). The zoom sliding mechanism has the zoom lens 40C mounted thereon so as to be slidable along the optical axis OA. The zoom motor is connected to the zoom sliding mechanism, and the zoom sliding mechanism receives power from the zoom motor to operate, thereby moving the zoom lens 40C along the optical axis OA.

[0080] The third actuator 39 includes a power transmission mechanism (not shown) and an aperture motor (not shown). The aperture 40D has an opening 40D1, and is an aperture that can change the size of the opening 40D1. The opening 40D1 is formed by, for example, a plurality of aperture blades 40D2. The plurality of aperture blades 40D2 are connected to the power transmission mechanism. In addition, the aperture motor is connected to the power transmission mechanism, and the power transmission mechanism transmits the power of the aperture motor to the plurality of aperture blades 40D2. The plurality of aperture blades 40D2 receive the power transmitted from the power transmission mechanism and operate, thereby changing the size of the opening 40D1. The aperture 40D adjusts the exposure by changing the size of the opening 40D1.

[0081] The focusing motor, the zoom motor, and the diaphragm motor are connected to the control device 36, and the driving of the focusing motor, the zoom motor, and the diaphragm motor is controlled by the control device 36. In addition, in the present embodiment, a stepping motor is used as an example of the focusing motor, the zoom motor, and the diaphragm motor. Therefore, the focusing motor, the zoom motor, and the diaphragm motor operate in synchronization with the pulse signal according to the command from the control device 36. In addition, here, an example in which the focusing motor, the zoom motor, and the diaphragm motor are provided in the interchangeable lens 18 is shown, but this is only an example, and at least one of the focusing motor, the zoom motor, and the diaphragm motor may be provided in the camera body 16. In addition, the structure and / or operation method of the interchangeable lens 18 may be changed as needed.

[0082] In the imaging device 10, when in imaging mode, the MF mode and the AF mode can be selectively set according to an instruction given to the imaging device body 16. The MF mode is an operation mode for manual focus. In the MF mode, for example, when the user operates the focus ring 18A or the like, the focus lens 40B moves along the optical axis OA by an amount corresponding to the amount of operation of the focus ring 18A or the like, thereby adjusting the focus.

[0083] In the AF mode, the imaging device body 16 calculates a focus position corresponding to the subject distance and moves the focus lens 40B toward the calculated focus position to adjust the focus. Here, the focus position refers to the position of the focus lens 40B on the optical axis OA in the focused state.

[0084] The imaging device body 16 includes an image sensor 20, an image processing engine 12, a system controller 44, an image memory 46, a UI device 48, an external I / F 50, a communication I / F 52, a photoelectric conversion element driver 54, and an input / output interface 70. The image sensor 20 includes a photoelectric conversion element 72 and an A / D converter 74.

[0085] The image processing engine 12, image memory 46, UI system device 48, external I / F 50, photoelectric conversion element driver 54, mechanical shutter driver (not shown), and A / D converter 74 are connected to the input / output interface 70. In addition, the control device 36 of the interchangeable lens 18 is also connected to the input / output interface 70.

[0086] The system controller 44 includes a processor (not shown in the figure), a storage device (not shown in the figure), and a memory (not shown in the figure). In the system controller 44, the storage device is a non-temporary storage medium that is readable by a computer and stores various parameters and various programs. The storage device of the system controller 44 is, for example, an EEPROM. However, this is only an example, and a HDD and / or SSD etc. may be used as the storage device of the system controller 44 instead of or together with the EEPROM. Furthermore, the memory of the system controller 44 temporarily stores various information and is used as a working memory. In the system controller 44, the processor reads out necessary programs from the storage device and executes the various programs read out on the memory, thereby controlling the entire imaging device 10. That is, in Figure 2 In the illustrated example, the image processing engine 12 , the image memory 46 , the UI system device 48 , the external I / F 50 , the communication I / F 52 , the photoelectric conversion element driver 54 , and the control device 36 are controlled by the system controller 44 .

[0087] The image processing engine 12 operates under the control of the system controller 44. The image processing engine 12 includes a processor 62, a storage device 64, and a memory 66. Here, the processor 62 is an example of the "first processor" according to the present invention.

[0088] The processor 62, the storage device 64, and the memory 66 are connected via a bus 68, and the bus 68 is connected to an input / output interface 70. Figure 2 In the example shown, for the sake of convenience, one bus is shown as the bus 68, but a plurality of buses may be provided. The bus 68 may be a serial bus or a parallel bus including a data bus, an address bus, and a control bus.

[0089] The storage device 64 is a non-temporary storage medium readable by a computer, and stores various parameters and various programs different from the various parameters and various programs stored in the storage device of the system controller 44. The storage device 64 is, for example, an EEPROM. However, this is only an example, and a HDD and / or an SSD etc. may be used as the storage device 64 instead of or together with the EEPROM. In addition, the memory 66 is, for example, a RAM, which temporarily stores various information and is used as a working memory.

[0090] The processor 62 reads out a necessary program from the storage device 64, and executes the read program on the memory 66. The processor 62 performs image processing according to the program executed on the memory 66.

[0091] The photoelectric conversion element 72 is connected to the photoelectric conversion element driver 54. The photoelectric conversion element driver 54 supplies an imaging timing signal to the photoelectric conversion element 72 according to an instruction from the processor 62. The imaging timing signal specifies the timing of imaging by the photoelectric conversion element 72. The photoelectric conversion element 72 performs reset, exposure, and output of an electrical signal according to the imaging timing signal supplied from the photoelectric conversion element driver 54. Examples of the imaging timing signal include a vertical synchronization signal and a horizontal synchronization signal.

[0092] When the interchangeable lens 18 is mounted on the imaging device body 16, the subject light incident on the imaging lens 40 is imaged on the light receiving surface 72A by the imaging lens 40. The photoelectric conversion element 72 performs photoelectric conversion on the subject light received by the light receiving surface 72A under the control of the photoelectric conversion element driver 54, and outputs an electrical signal corresponding to the light amount of the subject light as analog image data representing the subject light to the A / D converter 74. Specifically, the A / D converter 74 reads out the analog image data for each horizontal line in units of one frame from the photoelectric conversion element 72 in an exposure sequential readout manner.

[0093] The A / D converter 74 generates a RAW image 75A by digitizing the analog image data. The RAW image 75A is an image in which R pixels, G pixels, and B pixels are arranged in a mosaic shape. The RAW image 75A is an example of a "captured image" according to the present invention.

[0094] In the present embodiment, as an example, the bit depth of the R pixels, B pixels, and G pixels included in the RAW image 75A (i.e., the number of bits that express the grayscale value of each of the R pixels, B pixels, and G pixels included in the RAW image 75A (in other words, the bit length, the number of color bits, or the color depth)) is 14 bits. In addition, 14 bits is only an example, and the bit depth of the R pixels, B pixels, and G pixels included in the RAW image 75A may exceed 14 bits or may be less than 14 bits.

[0095] In the present embodiment, as an example, the processor 62 of the image processing engine 12 acquires a RAW image 75A from the A / D converter 74, and performs image processing including development on the acquired RAW image 75A, thereby generating an image file 75B. Here, development refers to the process of compressing the brightness and color difference signals according to a predetermined compression method. As the predetermined compression method (i.e., the format of the image file), for example, JPEG, TIFF, JPEG XR, MPEG, or AVI can be cited. In addition, the image processing includes image quality adjustment for the RAW image 75A. The image quality adjustment for the RAW image 75A is achieved by adjusting the focal length, adjusting the aperture value, adjusting the lens characteristics, adjusting the interval elimination characteristics between pixels P, a grayscale correction function (for example, a process of correcting the grayscale of an RGB image according to a gamma value), a gain correction function, and a noise reduction function.

[0096] As another example of image quality adjustment for the RAW image 75A, there can be cited color space conversion processing (i.e., processing for converting the color space of the RGB image that has been gamma-corrected from RGB color space to YCbCr color space), brightness filtering processing (i.e., processing for filtering the brightness signal (so-called Y signal) using a brightness filter (not shown)), color difference processing (i.e., processing for filtering the Cb signal and the Cr signal to reduce high-frequency noise) and / or resizing processing (i.e., processing for adjusting the brightness and color difference signals so that the size of the image represented by the brightness and color difference signals corresponds to the size indicated by the user, etc.), etc.

[0097] An example of the image file 75B is an MPEG file. However, the MPEG file is only an example, and the image file 75B may be another type of image file such as a JPEG file, a JPEG XR file, a TIFF file, or an AVI file. The image file 75B is stored in the image memory 46 by the processor 62 .

[0098] The UI system device 48 includes a display 28, and the processor 62 displays various information on the display 28. In addition, the UI system device 48 includes a receiving device 76. The receiving device 76 includes a touch panel 30 and a hard key unit 78. The hard key unit 78 includes the indication key 26 (see Figure 1 The processor 62 operates according to various instructions received through the touch panel 30. In addition, here, the hard key unit 78 is included in the UI system device 48, but the present invention is not limited to this. For example, the hard key unit 78 can also be connected to the external I / F 50.

[0099] The external I / F 50 controls the exchange of various information with a device (hereinafter, also referred to as an "external device") existing outside the camera 10. As an example of the external I / F 50, a USB interface can be cited. External devices such as smart devices, personal computers, servers, USB memories, memory cards, and / or printers (not shown) are directly or indirectly connected to the USB interface.

[0100] The communication I / F 52 is connected to a network (not shown). The communication I / F 52 controls the exchange of information between a communication device (not shown) such as a server on the network and the system controller 44. For example, the communication I / F 52 sends information corresponding to a request from the system controller 44 to the communication device via the network. In addition, the communication I / F 52 receives information sent from the communication device and outputs the received information to the system controller 44 via the input / output interface 70.

[0101] When editing moving images used in movies or television, dynamic range is widely known as one of the very important specifications. As an example of a method of recording moving images with a wide dynamic range, there is the RAW method (for example, a method of recording 14-bit RGB images) or the Log method (for example, a method of recording 10-bit YC images). In the RAW method, gamma correction processing is subsequently performed by software to raise the steep shadow part of the gamma rise (that is, the side with smaller input values), and in the Log method, processing is performed to convert from RGB grayscale to Log grayscale when recording. In either case, if a sufficient bit depth is not ensured, as an example, Figure 3 As shown in FIG. 7A , the image quality of the image may be degraded, such as grayscale jumps called banding or disappearance of fine structures in the subject (in other words, disappearance of details) in the RAW image 75A. In addition, the frame rate is currently required to be increased so as not to cause rolling distortion caused by afterimages and / or rolling shutters. The increase in frame rate and the increase in bit depth are in a trade-off relationship, and therefore a high-performance image processing technology that takes into account both the increase in frame rate and the increase in bit depth is required.

[0102] Therefore, in the present embodiment, the image processing engine 12 applies AI to the RAW image 75A to perform processing to improve the image quality of the RAW image 75A (for example, including processing to increase the bit depth of each pixel). Figures 4 to 6 , a method for generating AI applicable to RAW image 75A is described.

[0103] Figure 4 79 is a conceptual diagram showing an example of the structure of the learning device 79. Figure 4As shown, the learning device 79 includes a processor 80, a storage device 82, and a memory 84. The hardware structure of the processor 80, the storage device 82, and the memory 84 in the learning device 79 is basically the same as the hardware structure of the processor 62, the storage device 64, and the memory 66 described above, so the description is omitted here. Figure 4 In the example shown, the learning device 79 is an example of the “learning device” involved in the present invention.

[0104] A learning program 90 is stored in the storage device 82. The processor 80 is an example of the "second processor" involved in the present invention. The processor 80 reads the learning program 90 from the storage device 82 and executes the read learning program 90 on the memory 84. The processor 80 performs a learning process according to the learning program 90 executed on the memory 84. The learning process is a process of generating a learned model 106 from a model 98. The learned model 106 is generated by performing machine learning on the model 98 by the processor 80. That is, the learned model 106 is generated by optimizing the model 98 through machine learning. For example, the model 98 is a neural network having hundreds of millions to trillions of intermediate layers. As an example of the model 98, a generative AI model that generates and outputs an image with higher image quality than the input image (for example, an image with at least a deeper bit depth than the input image) can be cited.

[0105] The storage device 82 stores a plurality of (e.g., tens of thousands to hundreds of billions) training data 92. The training data 92 is used for machine learning of the model 98. That is, in the learning device 79, the processor 80 acquires the plurality of training data 92 from the storage device 82, and uses the acquired plurality of training data 92 to cause the model 98 to perform machine learning.

[0106] The training data 92 is labeled data. Labeled data is data in which a corresponding association is established between an example question image 94 (in other words, example question data) and a correct answer image 96 (in other words, correct answer data). Here, the training data 92 is an example of "training data" involved in the present invention, the example question image 94 is an example of "example question image" involved in the present invention, and the correct answer image 96 is an example of "correct answer image" involved in the present invention.

[0107] Example image 94 is an image in RAW format, and its bit depth is shallower than that of correct answer image 96. Example image 94 is an image that is assumed to be RAW image 75A. The image that is assumed to be RAW image 75A is, for example, an image that has the same image quality as RAW image 75A including the bit depth. Here, bit depth refers to a value that expresses grayscale (i.e., a value that expresses the fineness of grayscale expression of color). Words and phrases used in the same meaning as bit depth include words and phrases such as bit length, color bit number, bit number, or color depth.

[0108] In this embodiment, as the example image 94, a sample subject (for example, a Figure 4 However, this is only an example, and the image assumed to be the RAW image 75A may also be a virtually generated image. An example of a virtually generated image may be an image generated by a generation AI or the like.

[0109] Similar to the example question image 94, the correct answer image 96 is an image in RAW format. Furthermore, the correct answer image 96 is an image with a higher image quality than the question image 94. An example of a higher image quality image is an image with a bit depth deeper than the question image 94. In the present embodiment, the bit depth of the correct answer image 96 is 14 bits, and the bit depth of the example question image 94 is 12 bits. However, these bit depths are merely examples, and the present invention is applicable to any bit depth as long as the bit depth of the correct answer image 96 is deeper than the bit depth of the question image 94.

[0110] The processor 80 acquires training data 92 one by one from the storage device 82. The processor 80 inputs the example image 94 from the training data 92 acquired from the storage device 82 to the model 98. When the example image 94 is input, the model 98 generates a comparison image 100 which is an image having a bit depth deeper than the question image 94 (here, as an example, a 14-bit image). The comparison image 100 is an image for comparison with the correct answer image 96 associated with the example image 94 input to the model 98. In the present embodiment, the comparison image 100 is an example of the “evaluation target image” involved in the present invention.

[0111] The processor 80 calculates an error 102 between the correct answer image 96 and the comparison image 100, which are associated with the example image 94 input to the model 98. The error 102 is an example of the "comparison result" involved in the present invention. The processor 80 calculates a plurality of adjustment values ​​104 that minimize the error 102. Then, the processor 80 uses the plurality of adjustment values ​​104 to adjust the plurality of optimization variables in the model 98. Here, the plurality of optimization variables refers to, for example, a plurality of connection weights and a plurality of offset values ​​included in the model 98.

[0112] The processor 80 repeatedly performs a series of processes, including input of the example image 94 to the model 98, calculation of the error 102, calculation of the plurality of adjustment values ​​104, and adjustment of the plurality of optimization variables in the model 98, using the plurality of training data 92 stored in the storage device 82. That is, the processor 80 adjusts the plurality of optimization variables in the model 98 using the plurality of adjustment values ​​104 calculated so as to minimize the error 102 for each of the plurality of example images 94 included in the plurality of training data 92 stored in the storage device 82, thereby optimizing the model 98. The processor 80 generates a learned model 106 by optimizing the model 98. When the example image 94 is input to the learned model 106 generated in this way, the learned model 106 generates and outputs an image having the same bit depth as the correct answer image 96 as an image corresponding to the input example image 94.

[0113] Figure 5 94 and the correct answer image 96. Figure 5 As shown, in this embodiment, the imaging device 500 is used to create the example question image 94 and the correct answer image 96. The imaging device 500 is an example of the "first imaging device" according to the present invention.

[0114] Similar to the imaging device 10, the image sensor 20 mounted on the imaging device 500 is provided with a photoelectric conversion element 72. Also, similar to the imaging device 10, pixels P of different colors (i.e., R pixels, G pixels, and B pixels) are regularly arranged on the photoelectric conversion element 72. The arrangement pattern of the pixels P of different colors is the same as that of the imaging device 10. Figure 5 In the example shown, in order to facilitate understanding of the present invention, a number smaller than the actual number of pixels P (in Figure 5 In the example shown, the number of pixels P is illustrated for a 4×4 pixel number).

[0115] The imaging device 500 can generate RAW images with a deeper bit depth than the imaging device 10, and is otherwise an imaging device having the same structure and specifications as the imaging device 10. Therefore, the components of the imaging device 500 are denoted by the same reference numerals as those of the imaging device 10, and their description is omitted.

[0116] The imaging device 500 captures a sample subject to generate a single first image 108A for one frame as a RAW image with a bit depth of 14 bits. The single first image 108A captured by the image sensor 20 of the imaging device 500 is used as the correct answer image 96 .

[0117] The first single image 108A has pixels P1 of different colors arranged regularly. Here, the pixels P1 refer to pixels constituting the RAW image obtained by capturing by the image sensor 20 of the imaging device 500 (in other words, image pixels). The positions of the pixels P1 in the RAW image obtained by capturing by the image sensor 20 of the imaging device 500 coincide with the positions of the pixels P in the photoelectric conversion element 72 of the image sensor 20 mounted on the imaging device 500.

[0118] exist Figure 5 In the example shown, the second single image 108B is created from the first single image 108A. The second single image 108B is an image in RAW format with a bit depth of 12 bits. The second single image 108B is obtained by shifting the first single image 108A. For example, the bit depth of each of the pixels P1 of the first single image 108A is shifted right by 2 bits, "0" is placed in the vacated 2 most significant bits, and the extra 2 least significant bits due to the right shift are discarded, thereby creating the second single image 108B. The second single image 108B obtained in this way is used as the example image 94.

[0119] In addition, Figure 5 In the example shown, a first single image 108A obtained by photographing a sample subject with the camera 500 and a second single image 108B as an image that shifts the first single image 108A are illustrated, but this is only an example. For example, the first single image 108A and / or the second single image 108B may also be virtual images generated by a generation AI. The generation AI may be an AI specifically used to generate images, or may be a generation AI that generates and outputs the first single image 108A and / or the second single image 108B according to input instruction data (so-called prompts), such as ChatGPT using GPT-4 (Internet search <https: / / openai.com / gpt-4>).

[0120] Figure 6 It means that by Figure 4 1 is a conceptual diagram of an example of the application stage of the learned model 106 generated by the learning process of the example shown in FIG. Figure 6As shown in FIG. 1 , in the image pickup device 10, the storage device 64 stores a learned model 106. In addition, the storage device 64 stores a high-quality image processing program 107. In the image pickup device 10, the processor 62 reads the high-quality image processing program 107 from the storage device 64 and executes the read high-quality image processing program 107 on the memory 66. The processor 62 performs high-quality image processing according to the high-quality image processing program 107 executed on the memory 66. The high-quality image processing is as follows: the RAW image 75A is input to the learned model 106 stored in the storage device 64 (while referring to FIG. 1 ). Figure 2 ), the learned model 106 generates and outputs a high-quality image 75A1 as an image with higher quality than the RAW image 75A, and the processor 62 acquires the high-quality image 75A1. Here, the image with higher quality than the RAW image 75A refers to an image with a higher bit depth than the RAW image 75A, for example. Figure 6 An example of the RAW image 75A shown is a 12-bit image, and an example of an image having a higher bit depth than the RAW image 75A (i.e., a high-quality image 75A1) is a 14-bit image. In the present embodiment, the high-quality image 75A1 is an example of the "inference result" involved in the present invention.

[0121] The processor 62 generates an image file 75B by performing image processing including development on the high-quality image 75A1 , and stores the generated image file 75B in the image memory 46 .

[0122] Next, refer to Figure 7 The operation of the portion of the learning device 79 that is related to the present invention will be described. Figure 7 An example of the flow of the learning process executed by the processor 80 is shown in FIG. Figure 7 The illustrated flow of the learning process is an example of the “method for generating a learned model” according to the present invention.

[0123] exist Figure 7 In the learning process shown, first, in step ST10, the processor 80 executes to obtain the unprocessed training data 92 (ie, Figure 7 After the processing of step ST10 of the training data 92 not used in the learning process shown in the figure, the learning process proceeds to step ST12.

[0124] In step ST12, processor 80 inputs example image 94 included in training data 92 acquired in step ST10 to model 98. After executing the process of step ST12, the learning process proceeds to step ST14. By executing the process of step ST12, comparison image 100 is output from model 98.

[0125] In step ST14, processor 80 acquires comparison image 100 output from model 98. After executing the process of step ST14, the learning process proceeds to step ST16.

[0126] In step ST16, processor 80 compares comparison image 100 acquired in step ST14 with correct answer image 96 included in training data 92 acquired in step ST10. After executing the process of step ST16, the learning process proceeds to step ST18.

[0127] In step ST18, processor 80 adjusts model 98 using a plurality of adjustment values ​​104 obtained by comparing comparison image 100 with correct answer image 96 in step ST16. Model 98 is optimized by repeatedly executing the process of step ST18 based on all training data 92 stored in storage device 82. After executing the process of step ST18, the learning process proceeds to step ST20.

[0128] In step ST20, processor 80 determines whether unprocessed training data 92 is stored in storage device 82. In step ST20, if unprocessed training data 92 is stored in storage device 82, the determination is affirmative, and the learning process proceeds to step ST10. In step ST20, if unprocessed training data 92 is not stored in storage device 82, the determination is negative, and the learning process ends.

[0129] Next, refer to Figure 8 The operation of the portion of the imaging device 10 according to the present invention will be described. Here, for the sake of convenience, the description will be given on the premise that the learned model 106 has already been stored in the storage device 64 .

[0130] exist Figure 8 In the high-quality image processing shown, first, in step ST50, the processor 62 determines whether one frame of image is captured by the image sensor 20. In step ST50, if one frame of image is not captured by the image sensor 20, the determination is negative, and the high-quality image processing proceeds to step ST58. If one frame of image is captured by the image sensor 20, the determination is positive, and the high-quality image processing proceeds to step ST52.

[0131] In step ST52, the processor 62 acquires the RAW image 75A from the image sensor 20. After the processing of step ST52 is executed, the image quality improvement processing proceeds to step ST54.

[0132] In step ST54, the processor 62 inputs the RAW image 75A acquired in step ST52 to the learned model 106. After executing the process of step ST54, the high-quality image processing proceeds to step ST56. By executing the process of step ST54, the learned model 106 generates and outputs an image obtained by converting the RAW image 75A into a high-quality image (in other words, an image with a higher bit depth than the RAW image 75A), that is, a high-quality image 75A1.

[0133] In step ST56, processor 62 acquires high-quality image 75A1. Processor 62 then generates image file 75B based on high-quality image 75A1 and stores it in image memory 46. After executing the process of step ST56, the high-quality image processing proceeds to step ST58.

[0134] In step ST58, the processor 62 determines whether a condition for ending the high-quality image processing (hereinafter referred to as "ending condition") is satisfied. As the ending condition, the condition that the receiving device 76 receives an instruction to end the high-quality image processing can be cited. In step ST58, if the ending condition is not satisfied, the determination is negative, and the high-quality image processing proceeds to step ST50. In step ST58, if the ending condition is satisfied, the determination is positive, and the high-quality image processing ends.

[0135] As described above, the training data 92 involved in the present embodiment is used for machine learning of the model 98. The learned model 106 is generated by machine learning of the model 98. The training data 92 includes the example question image 94 and the correct answer image 96, and the example question image 94 and the correct answer image 96 are associated with each other. The example question image 94 is an image with a shallower bit depth than the correct answer image 96. Therefore, by inputting the RAW image 75A to the learned model 106 configured in this way, the high-quality image 75A1 can be generated as an image with a deeper bit depth than the RAW image 75A.

[0136] That is, a high-quality image 75A1 having a deeper bit depth than the RAW image 75A obtained by shooting with the camera 10 (in other words, an image having a higher bit accuracy than the original performance of the camera 10 (i.e., the performance before the learned model 106 is mounted) can be obtained, and as a result, the disappearance of the fine structure of the subject on the image and the occurrence of the banding phenomenon can be suppressed. Furthermore, even when the camera 10 generates a RAW image 75A having a bit depth less than a certain level at a frame rate of a certain level or more, a high-quality image 75A1 having a deeper bit depth than the RAW image 75A can be generated by the learned model 106 mounted on the camera 10 without reducing the frame rate.

[0137] In the present embodiment, both example question image 94 and correct answer image 96 are RAW format images. Therefore, by inputting RAW image 75A to learned model 106, RAW format high-quality image 75A1 can be generated as an image having a deeper bit depth than RAW image 75A.

[0138] Furthermore, in the present embodiment, correct answer image 96 is bit-shifted to create example question image 94 having a shallower bit depth than correct answer image 96. Therefore, example question image 94 can be created more easily than when example question image 94 is created independently of correct answer image 96. This can help reduce the number of man-hours required for creating training data 92.

[0139] Furthermore, in the present embodiment, the correct answer image 96 is generated by photographing a sample subject with the camera 500. The example image 94 is created based on the correct answer image 96 thus generated. Therefore, the example image 94 and the correct answer image 96 can reflect the characteristics of the camera 500. If the camera 500 and the camera 10 are of the same type and the same specification, the example image 94 and the correct answer image 96 can also reflect the characteristics of the camera 10. Thus, a high-quality image 75A1 in which image quality degradation caused by the characteristics of the camera 10 is suppressed can be generated and output for the learned model 106.

[0140] In the above embodiment, the example in which the first single image 108A of one frame obtained by photographing the sample subject by the camera device 500 is used as the correct answer image 96 and the example image 94 is created by shifting the correct answer image 96 is given. However, this is only an example, and the example image 94 and the correct answer image 96 may be created from a plurality of images. To achieve this purpose, for example, Fig. 9 As shown, first, the camera device 500 generates the third to sixth single images 109A to 109D by photographing a sample subject as images in RAW format with a bit depth of 12 bits. Then, the example image 94 is created by synthesizing the third to sixth single images 109A to 109D. Synthesizing the third to sixth single images 109A to 109D means, for example, adding and averaging the third to sixth single images 109A to 109D in units of pixels P1 corresponding to each other in position. Fig. 9 The example image 94 shown is an example of a “representative image” according to the present invention.

[0141] exist Fig. 9In the example shown, the correct answer image 96 is created based on the 7th to 10th single images 110A to 110D. The 7th to 10th single images 110A to 110D correspond to the 3rd to 6th single images 109A to 109D, and are RAW format images with a deeper bit depth than the 3rd to 6th single images 109A to 109D. The bit depth of the 7th to 10th single images 110A to 110D is 14 bits. The 7th single image 110A is obtained by shifting the bits of the 3rd single image 109A. For example, the 7th single image 110A is created as follows: the bit depth of each of all pixels P1 of the 3rd single image 109A is shifted to the left by 2 bits, and "0" is placed in the 2 least significant bits vacated by the left shift. In the same manner, the eighth single image 110B is obtained by shifting the fourth single image 109B, the ninth single image 110C is obtained by shifting the fifth single image 109C, and the tenth single image 110D is obtained by shifting the sixth single image 109D.

[0142] Correct answer image 96 is created by synthesizing the seventh to tenth single images 110A to 110D. Synthesizing the seventh to tenth single images 110A to 110D means, for example, averaging the seventh to tenth single images 110A to 110D in units of corresponding pixels P1.

[0143] So, in Fig. 9 In the example shown, the example question image 94 is obtained by synthesizing the 3rd to 6th single images 109A to 109D (here, as an example, the 3rd to 6th single images 109A to 109D are averaged in units of pixels P1 corresponding to each other). Furthermore, the correct answer image 96 is obtained by synthesizing the 7th to 10th single images 110A to 110D having a deeper bit depth than the 3rd to 6th single images 109A to 109D (here, as an example, the 7th to 10th single images 110A to 110D are averaged in units of pixels P1 corresponding to each other). Therefore, it is easy to obtain the correct answer image 96 having a deeper bit depth than the question image 94.

[0144] And, in Fig. 9 In the example shown, example question image 94 is obtained by synthesizing the third to sixth single images 109A to 109D, which are the basis for the seventh to tenth single images 110A to 110D used to obtain correct answer image 96. That is, example question image 94 is obtained using images representing the third to sixth single images 109A to 109D used to obtain correct answer image 96. Therefore, compared with the case where example question image 94 is obtained independently of correct answer image 96, example question image 94 having high affinity with correct answer image 96 can be effectively obtained.

[0145] Furthermore, in Fig. 9 In the example shown, as the seventh to tenth single images 110A to 110D, images obtained by shifting the third to sixth single images 109A to 109D are used. Therefore, it is possible to easily obtain a correct answer image 96 that is assumed to be a high-quality image 75A1 generated and output by the learned model 106 at the stage where the learned model 106 is applied in the imaging device 10 (that is, a correct answer image 96 having the same bit depth as the high-quality image 75A1).

[0146] exist Fig. 9 In the example shown, the example of creating the example image 94 by synthesizing the third to sixth single images 109A to 109D is given, but this is only an example. Fig.10 As shown, any one of the 3rd to 6th single images 109A to 109D before the shift may be used as the example image 94. In this case, for example, if the 3rd to 6th single images 109A to 109D are images obtained by continuous shooting, an image obtained between the 3rd single image 109A and the 6th single image 109D (i.e., the 4th single image 109B or the 5th single image 109C) may be used as the example image 94.

[0147] Furthermore, an image obtained by synthesizing the fourth single image 109B and the fifth single image 109C (for example, averaging the fourth single image 109B and the fifth single image 109C in units of pixels P1 corresponding to each other) may be used as the example image 94. Furthermore, an image generated from a plurality of images of no more than three of the third to sixth single images 109A to 109D before the shift (that is, an image obtained by synthesizing a plurality of images of no more than three of the third to sixth single images 109A to 109D before the shift) may be used as the example image 94.

[0148] In addition, here, any one of the 3rd to 6th single images 109A to 109D used as example image 94 or an image obtained by synthesizing multiple images of 3 or less of the 3rd to 6th single images 109A to 109D is an example of a "representative image" involved in the present invention.

[0149] The 3rd to 6th single images 109A to 109D before the shift are all images used to obtain the correct answer image 96. Fig.10As shown, by adopting any one of the 3rd to 6th single images 109A to 109D as example image 94, or adopting an image obtained by synthesizing multiple images of no more than 3 of the 3rd to 6th single images 109A to 109D as example image 94, compared with the case where example image 94 is obtained independently of correct answer image 96, example image 94 having a high affinity with correct answer image 96 can be effectively obtained.

[0150] And, in Fig.10 In the example shown, an example is given in which any one of the third to sixth single images 109A to 109D is used as the example image 94, but this is only an example, and an image generated based on at least one of the third to sixth single images 109A to 109D may be used as the example image 94. The image generated based on at least one of the third to sixth single images 109A to 109D refers to, for example, an image obtained by processing at least one of the third to sixth single images 109A to 109D. As an example of an image obtained by processing at least one of the 3rd to 6th single images 109A to 109D, there can be cited an image in which one of the 3rd to 6th single images 109A to 109D is converted from RAW format to Log format, an image obtained by performing some kind of image quality improvement processing (for example, noise reduction and / or edge extraction processing, etc.) on one of the 3rd to 6th single images 109A to 109D, or an image in which multiple images of the 3rd to 6th single images 109A to 109D are converted from RAW format to Log format and then added and averaged.

[0151] exist Fig. 9 In the example shown, an example of creating the correct answer image 96 by synthesizing the seventh to tenth single images 110A to 110D is shown, but this is only an example. Fig.11 As shown in FIG. 1 , the correct answer image 96 having a bit depth of 14 bits can also be created by adding the third to sixth single images 109A to 109D in units of pixels P1 corresponding to each other in positions. By setting it in this way, it is not necessary to create the seventh to tenth single images 110A to 110D from the third to sixth single images 109A to 109D (that is, it is not necessary to shift the third to sixth single images 109A to 109D), so compared with Fig. 9 In the example shown, a correct answer image 96 can be easily created with a bit depth of 14 bits.

[0152] In the above embodiment, the example image 94 is illustrated, but this is only an example. Fig.12As shown, even if example image 94A is used instead of example image 94, the present invention is still valid. Example image 94A is different from example image 94 in that the bit depth is lower. That is, the bit depth of example image 94 is 12 bits, while the bit depth of example image 94A is 10 bits. Fig.12 In the example shown, the example image 94A is created based on the 11th to 14th single images 112A to 112D having a shallower bit depth than the 3rd to 6th single images 109A to 109D.

[0153] exist Fig.12 In the example shown, in the same manner as in the above-described embodiment, in which the second single image 108B having a bit depth two bits shallower than the first single image 108A is created by bit shifting the third to sixth single images 109A to 109D, 11th to 14th single images 112A to 112D having a bit depth two bits shallower than the third to sixth single images 109A to 109D (i.e., the 11th to 14th single images 112A to 112D having a bit depth of 10 bits) are created. Then, the example image 94A is created by synthesizing the 11th to 14th single images 112A to 112D. Synthesizing the 11th to 14th single images 112A to 112D means, for example, adding and averaging the 11th to 14th single images 112A to 112D in units of pixels P1 corresponding to each other in position.

[0154] The difference between the bit depth of the example question image 94A obtained in this way and the bit depth of the correct answer image 96 is greater than the difference between the bit depth of the example question image 94 and the bit depth of the correct answer image 96. Therefore, compared with the learned model 106 generated by using the example question image 94 and the correct answer image 96 for machine learning to optimize the model 98 for the model 98, the learned model 106 generated by using the example question image 94A and the correct answer image 96 for machine learning to optimize the model 98 can make the difference between the bit depths of the input and output images larger. For example, when the RAW image 75A with a bit depth of 10 bits is input to the learned model 106 generated by using the example question image 94A and the correct answer image 96 for machine learning to optimize the model 98 for the model 98, the learned model 106 can generate and output a high-quality image 75A1 with a bit depth of 4 bits deeper than the input RAW image 75A (that is, a 14-bit high-quality image 75A1).

[0155] In addition, Fig.12In the example shown, an example is given of creating the correct answer image 96 by adding and averaging the 7th to 10th single images 110A to 110D, but this is only an example, and the correct answer image 96 with a bit depth of 12 bits can also be created by adding and averaging the 3rd to 6th single images 109A to 109D.

[0156] exist Fig.10 In the example shown, the correct answer image 96 is created by synthesizing the seventh to tenth single images 110A to 110D, but this is only an example. Fig.13 As shown, the present invention is also applicable even if correct answer image 96A is used instead of correct answer image 96. Correct answer image 96A has the same bit depth as correct answer image 96, but differs in that the image size is 1 / 4 of correct answer image 96.

[0157] The correct answer image 96A is created based on a plurality of image regions obtained by dividing the seventh single image 110A. Here, an example of a form of dividing the seventh single image 110A is given, but this is only an example, and the eighth to tenth single images 110B to 110D may also be divided (see Figures 9 to 12 )

[0158] The seventh single image 110A is divided into a processing target region 110A1, a first adjacent region 110A2 adjacent to the right side of the processing target region 110A1, a second adjacent region 110A3 adjacent to the bottom side of the processing target region 110A1, and a third adjacent region 110A4 adjacent to the lower right side of the processing target region 110A1. Here, the processing target region 110A1 is an example of the "processing target region" involved in the present invention, and the first adjacent region 110A2, the second adjacent region 110A3, and the third adjacent region 110A4 are examples of the "corresponding region" involved in the present invention.

[0159] The processing target region 110A1, the first adjacent region 110A2, the second adjacent region 110A3, and the third adjacent region 110A4 are all sets of 2×2 pixels P1 (in other words, blocks), and are defined by R pixels, G pixels, G pixels, and B pixels in corresponding arrangement patterns in the seventh single image 110A. That is, the positions of R pixels, G pixels, G pixels, and B pixels are consistent among the processing target region 110A1, the first adjacent region 110A2, the second adjacent region 110A3, and the third adjacent region 110A4.

[0160] Therefore, in Fig.13In the example shown, an image having the same bit depth as that of the seventh single image 110A is created as the correct answer image 96A by synthesizing the first adjacent region 110A2, the second adjacent region 110A3, and the third adjacent region 110A4 with respect to the processing target region 110A1. Here, synthesizing the first adjacent region 110A2, the second adjacent region 110A3, and the third adjacent region 110A4 means, for example, averaging the processing target region 110A1, the first adjacent region 110A2, the second adjacent region 110A3, and the third adjacent region 110A4 in units of pixels P1 corresponding to each other in position.

[0161] In this way, the bit depth of the correct answer image 96A created by synthesizing the first adjacent region 110A2, the second adjacent region 110A3, and the third adjacent region 110A4 obtained by dividing the seventh single image 110A becomes the same as the bit depth of the seventh single image 110A, and the image size becomes 1 / 4 of the seventh single image 110A. That is, the bit depth of the correct answer image 96A is the same as that of the correct answer image 96, and the image size is 1 / 4 of the correct answer image 96. Therefore, when the RAW image 75A is input to the learned model 106 generated by using the example image 94 and the correct answer image 96A for machine learning of the model 98 to optimize the model 98, the learned model 106 can generate and output a high-quality image 75A1 having a deeper bit depth and a smaller image size than the input RAW image 75A (that is, a high-quality image 75A1 having a bit depth of 14 bits and an image size of 1 / 4). Here, the Bayer arrangement is exemplified, but even with another type of arrangement pattern such as the quad-Bayer arrangement, multiple divisional areas (for example, in the case of the quad-Bayer arrangement, multiple divisional areas composed of multiple pixels adjacent to each other of the same color) corresponding to the arrangement pattern in the image (that is, the image equivalent to the seventh single image 110A) can be synthesized. And, for example, in the case where there are adjacent pixels of the same color that are regularly adjacent to the processing target pixel (that is, one pixel of interest) in the image (that is, the shifted image) as in the quad-Bayer arrangement, an image equivalent to the correct answer image 96A can be created by synthesizing (for example, performing addition averaging) the processing target pixel and at least one adjacent pixel.

[0162] In addition, Fig.13 In the example shown, an example is given in which the processing object area 110A1 is synthesized with all of the first adjacent area 110A2, the second adjacent area 110A3 and the third adjacent area 110A4, but this is only an example, and it is also possible to synthesize less than two of the first adjacent area 110A2, the second adjacent area 110A3 and the third adjacent area 110A4 with the processing object area 110A1 and discard the remaining parts.

[0163] And, in Fig.13 In the example shown, the image size is reduced to 1 / 4, but this is only an example, and the image size may be reduced to a size other than 1 / 4. In this case, the number of divisions of the seventh single image 110A may be changed according to the image size so that a plurality of image regions obtained by dividing the seventh single image 110A may be synthesized.

[0164] In the above embodiment, as an example of improving the image quality of the RAW image 75A, an example of increasing the bit depth of the entire RAW image 75A is cited. However, the RAW image 75A may be further improved in image quality by dividing the RAW image 75A into a plurality of image areas and improving the image quality of a category different from the bit depth for each image area. In this case, for example, Fig.14 As shown in FIG. 1 , the example image 94 is divided into a plurality of first image regions 111. Fig.14 In the example shown, 5×5 first image regions 111 are shown as an example of the plurality of first image regions 111. Image heights 114 related to the optical system of the imaging device 500 are given to the plurality of first image regions 111. As an example of the image height related to the optical system of the imaging device 500, the imaging lens 40 (see Figure 2 Here, the imaging lens 40 is an example of the “optical system” involved in the present invention, and the image height 114 is an example of the “characteristic of the second imaging device” and the “image height” involved in the present invention.

[0165] exist Fig.14 In the example shown, the correct answer image 96 is divided into a plurality of second image regions 116 corresponding to the plurality of first image regions 111. Fig.14 In the illustrated example, 5×5 second image regions 116 are shown as an example of the plurality of second image regions 116 .

[0166] The image quality of each of the example image 94 and the correct answer image 96 includes image quality of a different category from the bit depth. As an example of image quality of a different category from the bit depth, image quality determined by the image height 114, which is one of the characteristics of the camera device 500, can be cited. As an example of image quality determined by the image height 114, the shading of the imaging lens 40 and / or the shading on the light receiving surface 72A can be cited. The shading of the imaging lens 40, for example, refers to a phenomenon in which the light gradually becomes darker along the radial direction from the optical axis OA of the imaging lens 40 (in other words, a phenomenon in which the amount of light gradually decreases from the center to the periphery of the imaging lens 40), and the shading on the light receiving surface 72A refers to a phenomenon in which the signal-to-noise ratio changes from the center to the periphery of the light receiving surface 72A (in other words, a phenomenon in which the signal-to-noise ratio is uneven from the center to the periphery of the light receiving surface 72A). In addition, generally, the peripheral portion of the light receiving surface 72A is easily affected by the aberration of the peripheral portion of the imaging lens 40, and thus artifacts are easily generated.

[0167] Therefore, when the example question image 94 is divided into a plurality of first image regions 111 and the correct answer image 96 is divided into a plurality of second image regions 116, an image height 114 is assigned to each of the plurality of first image regions 111 and the plurality of second image regions 116. Furthermore, a label 118 capable of specifying a type of image quality different from the bit depth of the correct answer image 96 is assigned to each of the plurality of second image regions 116.

[0168] The image height 114 is an indicator of how far away from the center of the imaging lens 40 the image is. The larger the value of the image height 114 is, the darker the image quality is. Therefore, the label 118 is associated with the image height 114 of each second image area 116 as an indicator of the degree of darkness.

[0169] In addition, here, as a category of image quality different from the bit depth, the image quality determined by the image height 114 is exemplified, but this is only an example. For example, since it is important for the learned model 106 to distinguish the dynamic range of the shadow part from the noise and the normal image signal, it is also possible to superimpose noise determined by the characteristics of the camera 500 of the same type and specification as the camera 10 (for example, noise caused by the characteristics of the image sensor 20) on the example image 94 and the correct answer image 96, and make the label 118 include information that can identify the situation where the noise is included and the type of noise.

[0170] Furthermore, even if defective pixels, aberrations, flare, ghosting and / or blurring occur due to the characteristics of the imaging device 500, the label 118 can include information that can identify the defective pixels, aberrations, flare, ghosting and / or blurring in the same manner.

[0171] As an example, Fig.15As shown, the RAW image 75A is divided into a plurality of third image regions 120 in the same manner as the example image 94 is divided into a plurality of first image regions 111. Fig.15 In the example shown, 5×5 third image areas 120 are shown. An image height 122 of the imaging lens 40 mounted on the imaging device 10 is given to each of the plurality of third image areas 120 .

[0172] like Fig.15 As shown, by Fig.14 The example image 94 and the correct answer image 96 shown are used for the machine learning of the model 98 to optimize the model 98. The learned model 106 generated is used in the high-quality image processing performed by the processor 62. If the RAW image 75A is input to the learned model 106, the learned model 106 generates and outputs a high-quality image 75A1. The high-quality image 75A1 is divided into a plurality of fourth image regions 124 in the same manner as the RAW image 75A is divided into a plurality of third image regions 120. Fig.15 In the example shown, a 5×5 fourth image region 124 is shown. Each fourth image region 124 is given the same image height 122 as the third image region 120 at the corresponding position. In addition, image quality information 126 is given to each fourth image region 124. The image quality information 126 is information for determining the image quality of the fourth image region 124 (here, as an example, information for determining the image quality determined by the image height 122).

[0173] The high image quality processing includes image quality adjustment processing. The image quality adjustment processing is a processing for adjusting the image quality of the fourth image area 124 according to the image quality information 126 (for example, a processing for improving the image quality). In the image quality adjustment processing, the image quality of all the fourth image areas 124 is adjusted according to the image quality information 126 assigned to each fourth image area 124 (for example, adjustment of shading, defective pixels, aberration, flare, ghosting and / or blur, etc.). In this way, the overall image quality of the high image quality image 75A1 is adjusted by adjusting the image quality of all the fourth image areas 124. The high image quality image 75A1 after the image quality adjustment is filed in the same manner as in the above-mentioned embodiment, and the filed image file 75B is stored in the image memory 46.

[0174] In this way, the example image 94 is divided into a plurality of first image regions 111, and the correct answer image 96 is divided into a plurality of second image regions 116 corresponding to the plurality of first image regions 111. By constructing the example image 94 and the correct answer image 96 in this way, machine learning can be performed on the model 98 for each first image region 111.

[0175] Furthermore, labels 118 are respectively assigned to the plurality of second image regions 116. Labels 118 are information that can identify the image quality of the first image region 111 corresponding to the second image region 116 assigned with labels 118. Therefore, when the RAW image 75A is input to the learned model 106 optimized by machine learning using the correct answer image 96 and the example image 94 thus constructed, the image quality can be determined for each of the fourth image regions 124 of the high-quality image 75A1 generated by the learned model 106. In this way, if the image quality can be determined for each of the fourth image regions 124, the image quality can be adjusted for each of the fourth image regions 124 by the image quality adjustment process. Furthermore, the image quality determined by the labels 118 assigned to the second image regions 116 is determined according to the characteristics of the imaging device 500. Therefore, it is possible to determine the image quality according to the characteristics of the imaging device 500 (for example, the image height 114 ) for each fourth image region 124 of the high-quality image 75A1 generated by the learned model 106 .

[0176] exist Fig.14 In the example shown in FIG. 1 , a label 118 is assigned to each second image region 116. However, the present invention is not limited thereto, and a label 118 may not be assigned to each second image region 116. Fig.16 As shown in FIG. 1 , an image in which the brightness difference caused by the image height 114 in the example image 94 is eliminated for each second image region 116 may be used as the correct answer image 96. In this case, since an image in which the brightness difference caused by the image height 122 is eliminated is generated and output as the high-quality image 75A1 by the learned model 106, there is no need to Fig.15 The image quality adjustment process shown.

[0177] exist Fig.14 and Fig.16 In the example shown, an example of a form in which the image height 114 is given to each first image region 111 of the example image 94 is given, but this is only an example. Fig.17 As shown in FIG. 1 , the example image 94 may not be divided into a plurality of first image regions 111, and image height 114 may not be assigned. Fig.18 As shown, in the use of model 98 Fig.17When the learned model 106 generated by optimizing the model 98 through machine learning of the example image 94 and the correct answer image 96 shown in the figure inputs an image with shading as the RAW image 75A, the learned model 106 generates and outputs an image divided into a plurality of fourth image regions 124 and each fourth image region 124 is assigned an image height 122 as a high-quality image 75A1. In this case, the high-quality image 75A1 is subjected to image quality adjustment processing by the processor 62, and image quality adjustment corresponding to the image height 122 assigned to each fourth image region 124 is performed. Image quality adjustment corresponding to the image height 122 refers to, for example, image processing (so-called shading correction) that eliminates shading of the imaging lens 40 and / or shading on the light-receiving surface 72A. By setting it as follows Fig.17 and Fig.18 The example shown can suppress the image quality degradation in the high-quality image 75A1 that depends on the image height 122 (ie, the shading of the imaging lens 40 and / or the shading on the light receiving surface 72A).

[0178] In the above embodiment, an image in RAW format is exemplified as the correct answer image 96, but the present invention is not limited to this. Fig.19 As shown, an image in Log format may be used as the correct answer image 96. In this case, for example, an image in RAW format is input to the model 98 as the example image 94, and an image in RAW format is output from the model 98 as the comparison image 100. The learning process includes a first conversion process. The first conversion process is a process of converting the comparison image 100 from a RAW image to a Log format image. The comparison image 100 obtained in the form of an image in Log format by the processor 80 performing the first conversion process is compared with the correct answer image 96, and the model 98 is optimized based on the comparison result.

[0179] By configuring in this way, the amount of data of the image in Log format is less than that of the image in RAW format, so the load required for comparing the correct answer image 96 and the comparison image 100 in RAW format can be reduced. In addition, the shadow part is prone to grayscale jumps due to insufficient bit accuracy. Fig.19 As shown, by using an image in RAW format as the example image 94 and using an image in Log format with a higher accuracy in the shadow portion in the evaluation stage (i.e., the stage of comparing the image output from the model 86 and the correct answer image 96), it is possible to generate and output a learned model 106 that generates and outputs an image in which the grayscale of the shadow portion is corrected with high accuracy from the input RAW format image.

[0180] As an example, Fig. 20As shown in FIG. 1 , the high-quality image processing includes a second conversion processing. Similar to the first conversion processing, the second conversion processing is a process of converting a RAW format image into a Log format image. Fig.19 When the learned model 106 optimized by the method shown in the figure is input with the RAW image 75A, the learned model 106 generates and outputs the high-quality image 75A1 in RAW format. The high-quality image 75A1 in RAW format is converted into the high-quality image 75A1 in Log format by the processor 62 executing the second conversion process. The high-quality image 75A1 in Log format is filed in the same method as the above-mentioned embodiment, and the filed image file 75B is stored in the image memory 46.

[0181] exist Fig.19 In the example shown, a RAW image is shown as the example image 94, but the present invention is not limited to this. Fig.21 As shown, the example image 94 may also be an image in Log format. In this case, the correct answer image 96 is also an image in Log format. The Log format is an example of the "first file format" and the "second file format" involved in the present invention. An image in Log format is input to the model 98 as the example image 94, and an image in Log format is output from the model 98 as the comparison image 100. Then, the comparison image 100 is compared with the correct answer image 96 in the same manner as in the above-mentioned embodiment, and the model 98 is optimized based on the comparison result. By setting it in this way, the amount of data of the image in Log format is less than that of the image in RAW format, so compared with comparing the correct answer image 96 in RAW format and the comparison image 100 in RAW format, the load required for comparing the correct answer image 96 and the comparison image 100 can be reduced.

[0182] As an example, Fig. 22 As shown, in Fig.21 When the learned model 106 generated by the method shown in the figure inputs the Log image 75C obtained by converting the RAW image 75A into the Log format, the learned model 106 generates and outputs the high-quality image 75A1 in the Log format. The high-quality image 75A1 in the Log format is filed in the same method as the above-mentioned embodiment, and the filed image file 75B is stored in the image memory 46.

[0183] Thus, in high-definition processing, by using Fig.21The learned model 106 generated by the method shown can achieve the same high image quality as the above-mentioned embodiment by using Log format images with less data than RAW format images, even if the camera 10 is a model that does not support RAW format images (for example, a model that does not support RAW format images due to the limitation of transmission speed because the data volume of RAW format images is large). In addition, in the case where the transmission speed is limited for the camera 10 due to the problem of image data volume, YC format images after Log conversion or gamma conversion can also be used. In this case, YC format images can be used for learning processing instead of Log format images.

[0184] In the above-mentioned embodiment, various images in which R pixels, G pixels, and B pixels are regularly arranged are illustrated (for example, example image 94 and correct answer image 96, etc.), but the various images may also be monochrome images (for example, an R image composed only of multiple R images, a G image composed only of multiple G images, or a B image composed only of multiple B images).

[0185] In the above embodiment, an example of a form of high-definition processing performed by the processor 62 of the image processing engine 12 included in the camera device 10 is cited for explanation, but the present invention is not limited to this, and the device for high-definition processing may also be a device provided outside the camera device 10 such as a server. For example, the server may also be implemented by cloud computing. In addition, the server may also be implemented by network computing such as fog computing, edge computing, or grid computing. Here, a server is illustrated, but this is only an example, and at least one personal computer may be substituted for the server.

[0186] In the above embodiment, an example in which the high-quality image processing program 107 is stored in the storage device 64 is given for explanation, but the present invention is not limited to this. For example, the high-quality image processing program 107 may also be stored in a portable computer-readable non-temporary storage medium such as an SSD or a USB memory. The high-quality image processing program 107 stored in the non-temporary storage medium is installed in the image processing engine 12 of the camera device 10. The processor 62 performs high-quality image processing according to the high-quality image processing program 107.

[0187] Alternatively, the image quality enhancement program 107 may be stored in a storage device such as another computer or server device connected to the imaging device 10 via a network, and downloaded in response to a request from the imaging device 10 to be installed in the image processing engine 12 .

[0188] In addition, the entire image quality enhancement program 107 does not need to be stored in a storage device or the storage device 64 of another computer or server device connected to the imaging device 10, and a portion of the image quality enhancement program 107 may be stored. In addition, here, the image quality enhancement program 107 is described, but the learning program 90 can also be described in the same manner.

[0189] and, Figure 1 and Figure 2 The imaging device 10 shown has the image processing engine 12 built therein, but the present invention is not limited thereto, and for example, the image processing engine 12 may be provided outside the imaging device 10 .

[0190] In the above embodiment, the image processing engine 12 is exemplified, but the present invention is not limited thereto, and a device including an ASIC, an FPGA, and / or a PLD may be used instead of the image processing engine 12. Furthermore, a combination of a hardware structure and a software structure may be used instead of the image processing engine 12.

[0191] As hardware resources for performing the high-quality image processing and / or learning processing described in the above embodiments, various processors shown below can be used. As a processor, for example, a CPU can be cited, which is a general-purpose processor that performs the function of hardware resources for performing high-quality image processing and / or learning processing by executing software (i.e., a program). In addition, as a processor, for example, a dedicated circuit can be cited, which is a processor with a circuit structure specially designed for performing specific processing, such as FPGA, PLD or ASIC. Regardless of the type of processor, a memory is built in or connected, and regardless of the type of processor, the high-quality image processing and / or learning processing is performed by using the memory.

[0192] The hardware resource for performing high-definition processing and / or learning processing may be composed of one of these various processors, or may be composed of a combination of two or more processors of the same type or different types (for example, a combination of multiple FPGAs or a combination of a CPU and an FPGA). Furthermore, the hardware resource for performing high-definition processing and / or learning processing may also be a single processor.

[0193] As an example of a system composed of one processor, first, there is a form in which a processor is composed of a combination of one or more CPUs and software, and the processor performs the function of a hardware resource for performing high-definition processing and / or learning processing. Secondly, there is a form represented by SoC, etc., in which a processor is used to realize the function of the entire system including multiple hardware resources for performing high-definition processing and / or learning processing by one IC chip. In this way, high-definition processing and / or learning processing are realized by using one or more of the above-mentioned various processors as hardware resources.

[0194] Furthermore, as the hardware structure of these various processors, more specifically, a circuit formed by combining circuit elements such as semiconductor elements can be used. In addition, the above-mentioned high-definition processing and / or learning processing is only an example. Therefore, it is of course possible to delete unnecessary steps, add new steps, or change the processing order within the scope of the main purpose.

[0195] The above-mentioned records and illustrated contents are detailed descriptions of the parts involved in the present invention, and are only examples of the present invention. For example, the description of the above-mentioned structure, function, action and effect is a description of an example of the structure, function, action and effect of the parts involved in the present invention. Therefore, it is of course possible to delete unnecessary parts, add new elements or replace the above-mentioned records and illustrated contents without departing from the gist of the present invention. In addition, in order to avoid trouble and facilitate understanding of the parts involved in the present invention, the description of technical common sense that does not need to be particularly explained when implementing the present invention is omitted in the above-mentioned records and illustrated contents.

[0196] All documents, patent applications, and technical standards described in this specification are incorporated herein by reference to the same extent as if each document, patent application, and technical standard was specifically and individually indicated to be incorporated by reference.

[0197] Regarding the above embodiment, the following additional notes are also disclosed.

[0198] (Note 1)

[0199] A training data used for machine learning of a model, which has:

[0200] Correct answer image; and

[0201] The example image has a lighter bit depth than the correct answer image above.

[0202] (Note 2)

[0203] According to the training data described in Appendix 1,

[0204] The example question image is an image whose bit depth is shallower than that of the correct answer image by shifting the bit depth of the correct answer image.

[0205] (Note 3)

[0206] According to the training data described in Appendix 1,

[0207] The correct answer image described above is an image obtained by adding a plurality of single images.

[0208] (Note 4)

[0209] According to the training data described in Appendix 1,

[0210] The correct answer image is an image obtained by adding and averaging a plurality of single images.

[0211] (Note 5)

[0212] According to the training data described in Appendix 1,

[0213] The above example images are representative images of multiple single images.

[0214] The representative image is at least one of the plurality of single images or an image generated based on at least one of the plurality of single images.

[0215] (Note 6)

[0216] According to the training data described in Appendix 1,

[0217] The correct answer image is an image obtained by averaging the processing target region and at least one region adjacent to the processing target region in the shifted image.

[0218] (Note 7)

[0219] According to the training data described in Appendix 1,

[0220] The above correct answer image is divided into a plurality of image regions.

[0221] (Note 8)

[0222] The training data according to Appendix 7, wherein:

[0223] The correct answer image is an image captured by a camera having an optical system.

[0224] The plurality of image areas are each given an image height associated with the optical system.

[0225] (Note 9)

[0226] A learned model, which optimizes the model by performing the above-mentioned machine learning on the above-mentioned model using the training data described in any one of Supplement 1 to Supplement 8.

[0227] (Note 10)

[0228] A program (e.g., image quality enhancement program 107) is used to cause a computer (e.g., image processing engine 12) to perform image quality enhancement processing, the image quality enhancement processing comprising the following steps:

[0229] Inputting a camera image captured by an image sensor into the learned model described in Supplementary Note 9; and

[0230] The inference result output from the learned model based on the input of the camera image is obtained.

[0231] (Note 11)

[0232] A program (e.g., learning program 90) that causes a computer (e.g., learning device 79) to execute a learning process that generates a learned model by performing machine learning on the model using training data including correct answer images and example question images,

[0233] The above example image has a shallower bit depth than the above correct answer image.

[0234] The above learning process includes the following steps:

[0235] Input the above example image to the above model;

[0236] Based on the input of the example image, the model outputs an evaluation target image (for example, comparison image 100); and

[0237] The model is optimized based on the comparison result between the evaluation object image and the correct answer image.

[0238] (Note 12)

[0239] A computer-readable non-temporary storage medium stores a program (e.g., an image quality enhancement program 107) for causing a computer (e.g., an image processing engine 12) to perform an image quality enhancement process including the following steps:

[0240] Inputting a camera image captured by an image sensor into the learned model described in Supplementary Note 9; and

[0241] The inference result output from the learned model based on the input of the camera image is obtained.

[0242] (Note 13)

[0243] A non-transitory storage medium, which is a computer-readable non-transitory storage medium storing a program (e.g., a learning program 90) that causes a computer (e.g., a learning device 79) to execute a learning process that generates a learned model by performing machine learning on the model using training data including correct answer images and example question images,

[0244] The above example image has a shallower bit depth than the above correct answer image.

[0245] The above learning process includes the following steps:

[0246] Input the above example image to the above model;

[0247] Based on the input of the example image, the model outputs an evaluation target image (for example, comparison image 100); and

[0248] The model is optimized based on the comparison result between the evaluation object image and the correct answer image.

[0249] Explanation of symbols

[0250] 10, 500-camera device, 12-image processing engine, 16-camera device body, 18-interchangeable lens, 18A-focus ring, 20-image sensor, 22-release button, 24-dial, 26-indicator key, 28-display, 30-touch panel, 32-touch screen display, 36-control device, 37-first actuator, 38-second actuator, 39-third actuator, 40-imaging lens, 40A-objective lens, 40B-focusing lens, 40C-zoom lens, 40D-aperture, 40D1-opening, 40D2-aperture blades, 44-system controller, 46-image memory, 48-UI system device, 50-external I / F, 52-communication I / F, 54-photoelectric conversion element driver, 62, 80-processor, 64, 82-storage device, 66, 84-memory, 68-bus, 70-input / output interface, 72-photoelectric conversion element, 72A-light receiving surface, 74-A / D converter, 75A-RAW image, 75A1-high-quality image, 75B-image file, 76-receiving device, 78-hard key unit, 79-learning device, 90-learning program, 92- Training data, 94, 94A-Example image, 96, 96A-Correct answer image, 98-Model, 100-Comparison image, 102-Error, 104-Adjustment value, 106-Learned model, 107-High image quality program, 108A-1st single image, 108B-2nd single image, 109A-3rd single image, 109B-4th single image, 109C-5th single image, 109D-6th single image, 110A-7th single image, 110A1-Processing target area, 110A2-1st adjacent area, 110 A3-the second adjacent area, 110A4-the third adjacent area, 110B-the eighth single image, 110C-the ninth single image, 110D-the tenth single image, 111-the first image area, 112A-the eleventh single image, 112B-the twelfth single image, 112C-the thirteenth single image, 112D-the fourteenth single image, 114, 122-image height, 116-the second image area, 118-label, 120-the third image area, 124-the fourth image area, 126-image quality information, OA-optical axis, P, P1-pixel.

Claims

1. A training data used for machine learning of a model, having: Correct answer image; and The example image has a lighter bit depth than the correct answer image.

2. The training data according to claim 1, wherein: The correct answer image is an image obtained by synthesizing a plurality of single images.

3. The training data according to claim 2, wherein: The example image is a representative image of the multiple single images.

4. The training data according to claim 2, wherein: The example image is an image having a shallower bit depth than the plurality of single images.

5. The training data according to claim 2, wherein: The single image is a shifted image.

6. The training data according to claim 1, wherein: The correct answer image is the following image: An image obtained by synthesizing the processing target region and the corresponding region when a corresponding region corresponding to the arrangement pattern in the shifted image and the processing target region in the image exists in the image; or When adjacent pixels of the same color that are regularly adjacent to a processing target pixel in a shifted image exist in the image, an image is obtained by synthesizing the processing target pixel and the adjacent pixels.

7. The training data according to claim 6, wherein: The example image is an image before bit shift or an image generated based on the image before bit shift.

8. The training data according to claim 1, wherein: The correct answer image and the example question image are images captured by a first imaging device.

9. The training data according to claim 1, wherein: The correct answer image and the example question image are images in RAW format.

10. The training data according to claim 1, wherein: The model inputs and outputs images in RAW format. The correct answer image is in Log format. The model is optimized by converting the RAW format image output from the model into a Log format image and comparing it with the correct answer image.

11. The training data according to claim 1, wherein: The model outputs an image in a second file format other than RAW format in response to input of an image in a first file format other than RAW format, The example image is in the first file format. The correct answer image is in the second file format.

12. The training data according to claim 1, wherein: The example image is divided into a plurality of first image regions. The correct answer image is divided into a plurality of second image regions corresponding to the plurality of first image regions.

13. The training data according to claim 12, wherein: Labels capable of identifying image quality are given to the plurality of second image regions.

14. The training data according to claim 13, wherein: The example image is an image that is assumed to be an image captured by the second camera device. The image quality is determined according to the characteristics of the second imaging device.

15. The training data according to claim 14, wherein: The second camera device has an optical system. The characteristics include an image height associated with the optical system.

16. A learned model, wherein the model is optimized by performing the machine learning on the model using the training data of any one of claims 1 to 15.

17. A camera device comprising: the first processor; and Image sensor, The first processor performs the following processing: inputting a camera image obtained by taking the image sensor into the learned model according to claim 16; and An inference result output from the learned model based on the input of the camera image is obtained.

18. A learning device comprising a second processor, The second processor optimizes the model by causing the model to perform the machine learning using the training data according to any one of claims 1 to 15.

19. A method for creating training data for machine learning of a model, The training data includes correct answer images and example question images. The method for creating training data comprises the following steps: Create a correct answer image; and An example question image is created that has a lighter bit depth than the correct answer image.

20. A method for generating a learned model, wherein the learned model is generated by performing machine learning on the model using training data including correct answer images and example question images, The example question image has a bit depth shallower than that of the correct answer image. The method for generating the learned model comprises the following steps: Inputting the example image into the model; Outputting an evaluation object image by the model according to the input of the example image; and The model is optimized according to the comparison result between the evaluation object image and the correct answer image.

Images