Image data processing device, image data processing method, image data processing program, imaging device, and learning device

Abstract

Provided are an image data processing device, an image data processing method, an image data processing program, an imaging device, and a learning device capable of generating accurate distance image data. The image data processing device comprises at least one processor. The processor acquires first image data captured via an optical system. The processor generates second image data from the first image data by correcting image quality deterioration caused by the optical system. The processor generates first distance image data from the second image data.

Description

Image data processing device, image data processing method, image data processing program, imaging device, and learning device

[0001] The present invention relates to an image data processing device, an image data processing method, an image data processing program, an imaging device, and a learning device, and more particularly to an image data processing device, an image data processing method, an image data processing program, an imaging device, and a learning device that generate distance image data.

[0002] There is known a technique for measuring distance using AI (artificial intelligence) from a single image data captured by a single camera (see, for example, Patent Documents 1 and 2).

[0003] JP 2022-129941 A JP 2020-148483 A

[0004] One embodiment of the technique of the present disclosure provides an image data processing device, an image data processing method, an image data processing program, an imaging device, and a learning device that can generate accurate distance image data.

[0005] (1) An image data processing device comprising at least one processor, which acquires first image data captured through an optical system, corrects image quality degradation caused by the optical system, generates second image data from the first image data, and generates first distance image data from the second image data.

[0006] (2) An image data processing device as described in (1), in which the processor corrects image quality degradation caused by the optical system that affects the generation of the first distance image data, and generates second image data from the first image data.

[0007] (3) The image data processing device according to (1) or (2), wherein the processor generates third image data from the first image data and outputs the third image data to a display destination or a recording destination.

[0008] (4) The image data processing device according to (3), wherein the processor generates the third image data from the first image data by performing a correction different from the correction performed on the second image data.

[0009] (5) An image data processing device described in any one of (1) to (4), wherein the processor generates fourth image data from the first image data, outputs the fourth image data to a display or recording destination, performs processing to reduce the difference between the first distance image data and the fourth image data, generates second distance image data from the first distance image data, and outputs the second distance image data to a display or recording destination.

[0010] (6) An image data processing device described in any one of (1) to (5), wherein the processor outputs first distance image data to a display destination or a recording destination, generates fifth image data from the first image data, performs processing to reduce the difference between the first distance image data and the fifth image data, generates sixth image data from the fifth image data, and outputs the sixth image data to a display destination or a recording destination.

[0011] (7) The image data processing device according to (3), wherein the processor outputs the second image data to a recording destination in association with the third image data.

[0012] (8) The image data processing device according to (7), wherein the processor outputs information used to generate the second image data to a recording destination in association with the second image data.

[0013] (9) An image data processing device described in any one of (1) to (8), wherein the processor acquires first information including information necessary to correct image quality degradation caused by the optical system, and generates second image data based on the first information.

[0014] (10) The image data processing device according to (9), wherein the processor issues a notification when the first information cannot be acquired.

[0015] (11) An image data processing device according to (9) or (10), in which the processor generates second image data when specific information among the plurality of pieces of information included in the first information is acquired.

[0016] (12) The image data processing device according to any one of (9) to (11), wherein the processor limits the generation of the second image data when the first information cannot be acquired.

[0017] (13) An image data processing device described in any one of (9) to (12), in which the processor generates second image data based on predetermined second information when the first information cannot be obtained, and notifies the user that the second image data has been generated based on the second information.

[0018] (14) An image data processing device described in any one of (1) to (13), wherein the processor generates the second image data using one of the components of the first image data or mixed image data in which the components are weighted.

[0019] (15) An image data processing device according to any one of (1) to (14), wherein the processor generates the second image data using the component of the first image data that can express the greatest brightness.

[0020] (16) An image data processing device described in any one of (1) to (15), wherein the processor generates second image data for each component of the first image data, and generates first distance image data for each component from the second image data for each component.

[0021] (17) An image data processing device described in any one of (1) to (16), wherein the processor generates first distance image data from the second image data using a trained model generated by machine learning.

[0022] (18) An image data processing method that acquires first image data captured through an optical system, corrects image quality degradation caused by the optical system, generates second image data from the first image data, and generates first distance image data from the second image data.

[0023] (19) An image data processing program that causes a computer to perform the following functions: acquiring first image data captured through an optical system; correcting image quality degradation caused by the optical system and generating second image data from the first image data; and generating first distance image data from the second image data.

[0024] (20) An imaging device comprising an imaging unit that captures image data via an optical system, and an image data processing device according to any one of (1) to (17) that processes the image data captured by the imaging unit.

[0025] (21) An imaging device as described in (20), comprising a main body having an imaging unit and an image data processing device, and an interchangeable lens having an optical system and capable of communicating with and being detachably attached to the main body, wherein the processor acquires lens information from the interchangeable lens attached to the main body and generates second image data based on the lens information.

[0026] (22) A learning device comprising at least one processor, the processor acquiring a dataset including learning image data captured through an optical system and corrected for image quality degradation due to the optical system, and corrective data corresponding to the learning image data, and performing machine learning using the dataset to generate a model that outputs distance image data when image data is input.

[0027] 1. A diagram showing an example of the system configuration of an interchangeable lens digital camera. 2. A schematic diagram of the hardware configuration of a digital camera. 3. A block diagram of the main functions of a digital camera relating to distance measurement. 4. A block diagram of the main functions of a development processing unit. 5. A conceptual diagram of model learning. 6. A flowchart showing the distance measurement procedure. 7. A diagram showing an example of visualization of distance image data. 8. A block diagram of the main functions of a digital camera relating to live view output and distance measurement. 9. A flowchart showing the processing procedure for live view output. 10. A block diagram of the main functions of a digital camera relating to live view output and distance measurement. 11. A flowchart showing the processing procedure for live view output. 12. A diagram showing the processing flow until a live view image and a distance image are generated. 13. A block diagram of the main functions of a digital camera relating to live view output and distance measurement. 14. A flowchart showing the processing procedure until a live view image and a distance image are generated.

[0028] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0029] First Embodiment With the advancement of deep learning, a technology for measuring distance using AI from a single image data captured by a single camera (monocular camera distance measurement technology) has been attracting attention.

[0030] In distance measurement using AI, machine learning of the model is performed using image data actually captured by a camera.

[0031] However, images captured by a camera are subject to image quality degradation such as distortion, color unevenness, and vignetting due to the optical system. The manner in which this degradation manifests itself varies depending on the lens. For example, even lenses with the same focal length may exhibit different distortions depending on the optical design (combination of positive and negative lenses, number of lenses, lens material, etc.). Therefore, using a lens different from that used during learning may result in a decrease in the accuracy of distance measurement.

[0032] One way to solve this problem is to prepare image data captured with every lens in existence and use all of that data to train a model. However, this method is not realistic. Even if such data were available, the training process would require a significant amount of time. Another drawback is that the model cannot be used with lenses that are sold after the model is generated.

[0033] Therefore, in this embodiment, an image data processing device, method, and program are described that are capable of performing highly accurate distance measurement using AI in distance measurement, even from image data captured with a lens different from that used during learning.

[0034] [Interchangeable Lens Digital Camera] Here, an example in which the present invention is applied to an interchangeable lens digital camera will be described. An interchangeable lens digital camera is a digital camera in which the lens can be changed.

[0035] The interchangeable lens digital camera of this embodiment is a digital camera with a distance measurement function, that is, a digital camera with a function for measuring distance from captured image data.

[0036] The distance measurement results are acquired in the form of distance image data. The distance image data is image data in which the pixel value of each pixel is configured as distance information to the subject. The pixel value of each pixel in the distance image data represents distance information of the corresponding pixel in the captured image data. Because the distance information for each pixel is arranged corresponding to the position of each pixel in the captured image data, the distance image data is also referred to as distance map data, depth map data, etc. The distance image data can be visualized by, for example, representing the distance value for each pixel using color or density.

[0037] [System Configuration] Fig. 1 is a diagram showing an example of the system configuration of an interchangeable lens digital camera. In particular, Fig. 1 shows an example of the system configuration of a mirrorless single-lens camera. A mirrorless single-lens camera is a classification of digital cameras, and refers to an interchangeable lens digital camera in which an image is viewed through an electronic viewfinder (EVF) or a rear monitor instead of the optical viewfinder used in single-lens reflex cameras. Note that, although the example of a mirrorless single-lens camera will be described here, the present invention can also be applied to single-lens reflex cameras.

[0038] A camera system of an interchangeable lens digital camera is configured with at least one interchangeable lens 10 and at least one camera body 100. Figure 1 shows an example of a camera system configured with multiple interchangeable lenses 10 and one camera body 100.

[0039] Each interchangeable lens 10 is made up of a lens with different specifications, such as a different focal length, maximum aperture, zoom magnification, etc.

[0040] Each interchangeable lens 10 is detachably attached to the camera body 100 via a mount. The camera body 100 is provided with a camera-side mount 102 for attaching the interchangeable lens 10. In addition, each interchangeable lens 10 is provided with a lens-side mount 12 for attaching it to the camera body 100.

[0041] The camera-side mount 102 and the lens-side mount 12 each have a plurality of terminals (not shown). When the interchangeable lens 10 is attached to the camera body 100, the corresponding terminals are connected together. This connects the camera body 100 and the interchangeable lens 10 so that they can communicate with each other.

[0042] The digital camera 1 serving as an imaging device is configured by attaching the interchangeable lens 10 to the camera body 100. The digital camera 1 is configured as a single-lens imaging device (monocular camera).

[0043] [Hardware Configuration of Digital Camera] FIG. 2 is a schematic diagram of the hardware configuration of a digital camera.

[0044] [Interchangeable Lens] The interchangeable lens 10 includes an optical system 20, an optical system driver 30, a lens operation unit 40, a lens microcomputer 50, and the like.

[0045] The optical system 20 is configured by combining a plurality of lenses. The optical system 20 also includes a diaphragm. The diaphragm is configured by, for example, an iris diaphragm.

[0046] The optical system driver 30 includes a focus driver and an aperture driver. The focus driver drives a group of lenses used for focus adjustment (focus lens group). The aperture driver drives the aperture. If the interchangeable lens 10 has an optical image stabilization function, the optical system driver 30 is further provided with an image stabilization driver. The image stabilization driver drives the lens used for image stabilization. If the interchangeable lens 10 is a zoom lens, the optical system driver 30 may also include a zoom driver. The zoom driver drives a group of lenses used for zooming (zoom lens group).

[0047] The lens operation unit 40 includes a focus operation unit and an aperture operation unit, etc. The focus operation unit includes a focus operation member (for example, a focus ring) and a sensor that detects operation of the focus operation member, etc. The aperture operation unit includes an aperture operation member (for example, an aperture ring) and a sensor that detects operation of the aperture operation member, etc. If the interchangeable lens 10 is a zoom lens, the lens operation unit 40 is further provided with a zoom operation unit, etc. The zoom operation unit includes a zoom operation member (for example, a zoom ring) and a sensor that detects operation of the zoom operation member, etc.

[0048] The lens microcomputer 50 is composed of a microcomputer including a processor 51, a memory 52, and the like.

[0049] The processor 51 is configured, for example, by a CPU (Central Processing Unit), which is a general-purpose processor that executes programs and functions as various processing units. The processor 51 executes predetermined programs to function as a control unit that comprehensively controls the operation of the interchangeable lens 10.

[0050] The memory 52 includes a random access memory (RAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), etc. The memory 52 stores programs executed by the processor 51 and various data required for control.

[0051] [Camera Body] The camera body 100 includes an image sensor 110, a shutter 120, a storage unit 130, an interface unit (I / F unit) 140 (I / F: interface), a rear monitor 150, an EVF 160, a camera operation unit 170, a camera microcomputer 200, and the like.

[0052] The image sensor 110 receives light that passes through the interchangeable lens 10 and captures an optical image of a subject. In this embodiment, the image sensor 110 is an example of an imaging section.

[0053] The image sensor 110 is a color image sensor having a predetermined color filter array (CFA). The configuration of the image sensor 110 is not particularly limited. For example, a CMOS (Complementary Metal Oxide Semiconductor) type or a CCD (Charge Coupled Device) type can be used.

[0054] As an example, in this embodiment, a CMOS color image sensor equipped with a driver and a signal processor is employed as the image sensor 110. Therefore, signals (image signals) from each pixel are driven by the built-in driver and read out. The image signals read out from each pixel undergo predetermined signal processing in the signal processor and are then output from the image sensor 110. The signal processor includes, for example, a CDS (Correlated Double Sampling) circuit, an AGC (Automatic Gain Control) circuit, and an A / D converter (Analog to Digital Converter). The CDS circuit removes reset noise contained in the image signal. The AGC circuit amplifies the image signal and controls its amplitude to a constant level. The A / D converter converts the analog image signal into a digital image signal. The image signals read out from each pixel are finally converted into digital signals and output from the image sensor 110.

[0055] The arrangement of the color filters in the image sensor 110 is not particularly limited. As an example, in this embodiment, a Bayer arrangement is adopted. In the Bayer arrangement, a red (R) color filter is assigned to one pixel, a green (G) color filter to two pixels, and a blue (B) color filter to one pixel is assigned to four pixels, and the pixels are arranged in a regular pattern. Other color filter arrangements that can be used include, for example, X-Trans (registered trademark).

[0056] The shutter 120 is configured, for example, as a focal plane shutter. The shutter 120 is driven to operate by a shutter driver 122. The shutter driver 122 includes a charging motor, a holding electromagnet, and drive circuits for these components.

[0057] The storage unit 130 mainly stores image data obtained by capturing images. The storage unit 130 includes a storage device and its control circuit, etc. The storage device may be configured, for example, as an EEPROM or an SSD (Solid State Drive). The storage device may be configured as an integral part of the camera body 100 (in the form of a so-called built-in memory), or may be configured as a removable device for the camera body 100 (in the form of a so-called memory card). If the storage unit 130 is configured as a removable device for the camera body 100, the camera body 100 will be provided with an attachment section (a so-called card slot) for a memory card as a storage device. The storage unit 130 is an example of a recording destination in the digital camera 1.

[0058] The interface unit 140 connects the camera body 100 and external devices so that they can communicate with each other. The interface unit 140 includes various connection terminals, such as a Universal Serial Bus (USB) terminal and a High-Definition Multimedia Interface (HDMI) terminal. When external storage is connected to the digital camera 1 via the interface unit 140, the external storage (a so-called external storage device) is another example of a recording destination. Furthermore, when an external display is connected to the digital camera 1 via the interface unit 140, the external display (a so-called external display) is an example of a display destination.

[0059] The rear monitor 150 is a relatively large monitor provided on the rear of the camera body 100. The rear monitor 150 includes a display and its drive circuit, etc. The display is configured, for example, by an LCD (Liquid Crystal Display), an organic EL display (Organic Light Emitting Diode Display), etc. The rear monitor 150 can also be configured by a touch panel. The rear monitor 150 is an example of a display destination in the digital camera 1.

[0060] The EVF 160 has a small monitor, and is configured so that the display on the monitor can be observed through a viewfinder. The EVF 160 is another example of a display destination in the digital camera 1.

[0061] The camera operation unit 170 includes various operation members provided on the camera body 100, sensors that detect operations on the operation members, etc. The operation members include a power button, shutter button, mode dial, cross pad, OK button, cancel button, etc. Furthermore, if the rear monitor 150 is configured with a touch panel, the operation members include the touch panel.

[0062] The camera microcomputer 200 is configured as a microcomputer including a processor 210 and a memory 220. The microcomputer is an example of a computer.

[0063] The processor 210 is configured, for example, by a CPU, which is a general-purpose processor that executes programs and functions as various processing units. By executing a predetermined program, the processor 210 functions as a control unit that comprehensively controls the operation of the digital camera 1. By executing a predetermined program, the processor 210 also functions as an image data processing unit that processes image data obtained by capturing images.

[0064] The memory 220 includes RAM, ROM, EEPROM, etc. The memory 220 stores programs executed by the processor 210 and various data required for control and the like.

[0065] When the interchangeable lens 10 is attached to the camera body 100, the camera microcomputer 200 is connected to the lens microcomputer 50 so as to be able to communicate with it.

[0066] [Functions of the Digital Camera] [Basic Functions] The digital camera 1 has, as basic functions, an image capturing function and a playback function.

[0067] [Image Capture Function] The image capture function is a function for capturing images. Images include both still images and moving images. The user selects one of them and issues an instruction to capture an image. The processor 210 of the camera body 100 performs image capture control in response to the image capture instruction, and captures an image for recording. It also performs recording control, and records image data obtained by capturing an image. Image capture control includes AF (Auto Focus) control, AE (Automatic Exposure) control, etc.

[0068] Image data obtained by capturing an image is converted into image data in a predetermined format and recorded in storage unit 130. For example, the image data is converted into image data in JPEG (Joint Photographic Experts Group) format and recorded in storage unit 130. Processor 210 of camera body 100 processes the signal output from image sensor 110 to generate image data in the predetermined format and record it in storage unit 130.

[0069] Furthermore, image data obtained by capturing an image is recorded in the form of RAW data in the storage unit 130 as needed. RAW data is unprocessed (undeveloped) sensor data output from the image sensor 110.

[0070] During image capture, a live view (through image) is displayed on the rear monitor 150 or the EVF 160. That is, the image captured by the image sensor 110 is displayed on the rear monitor 150 or the EVF 160. The processor 210 of the camera body 100 processes the signal output from the image sensor 110 to generate image data for the live view and outputs it to the rear monitor 150 or the EVF 160.

[0071] [Playback Function] The playback function is a function for playing back and displaying recorded images on the rear monitor 150 or the EVF 160. In response to a playback instruction from the user, the processor 210 of the camera body 100 reads out image data recorded in the storage unit 130 and outputs it to the rear monitor 150 or the EVF 160.

[0072] [Distance Measurement Function] As described above, the digital camera 1 of this embodiment has a function (distance measurement function) for measuring distance from captured image data (monocular image data). Distance measurement is performed using AI. That is, distance measurement is performed using a trained model that has been machine-learned to output corresponding distance image data when image data of a captured scene is input. In the digital camera 1 of this embodiment, predetermined image processing is performed on the image data to be measured before inputting it into the trained model, thereby normalizing or standardizing the image data and enabling highly accurate distance measurement (generation of distance image data). Specifically, as preprocessing, a process is performed to correct image quality degradation caused by the optical system (image quality degradation caused by the characteristics of an interchangeable lens) and normalizing or standardizing the image data. That is, by removing image quality degradation caused by the optical system, the image data input to the trained model has a uniform image quality, thereby improving the accuracy of distance measurement.

[0073] FIG. 3 is a block diagram of the main functions of the digital camera relating to distance measurement.

[0074] 3, with regard to distance measurement, the digital camera 1 has functions such as an image data acquisition unit 210A, a lens information acquisition unit 210B, a preprocessing unit 210C, and a distance image data generation unit 210D. The functions of each unit are realized by a processor 210 in the camera body 100. The processor 210 realizes the functions of each unit by executing a predetermined program (image data processing program).

[0075] [Image Data Acquisition Unit] The image data acquisition unit 210A acquires image data to be processed. The image data to be processed is image data for distance measurement. In this embodiment, image data after so-called development processing is acquired as image data for distance measurement. As an example, image data in YCbCr format (YCbCr data) is acquired. Image data in YCbCr format is image data consisting of luminance data Y representing brightness and color difference data Cb, Cr representing the difference between two colors and their luminance. Of the two color difference data Cb, Cr, one color difference data Cb represents the difference between the blue component and the luminance (B-Y), and the other color difference data Cr represents the difference between the red component and the luminance (R-Y).

[0076] [Development Processing] The development processing is performed by the processor 210 of the camera body 100. The processor 210 executes a predetermined program to function as a development processing unit 210E.

[0077] FIG. 4 is a block diagram showing the main functions of the development processing section.

[0078] The development processing unit 210E has functions such as a white balance correction unit (WB correction unit) 210E1 (WB: White Balance), a gamma correction unit 210E2, a demosaic unit 210E3, a YC conversion unit 210E4, and a contour / color tone correction unit 210E5.

[0079] The development processing unit 210E performs development processing on the image data (RAW data) output from the image sensor 110, and generates image data in YCbCr format (YCbCr data).

[0080] Image data (RAW data) output from a Bayer array image sensor that employs primary color filters is composed of single-color data of either R, G, or B for each pixel.

[0081] The white balance correction unit 210E1 performs white balance correction on the image data (RAW data) output from the image sensor 110. The white balance correction is performed by multiplying each of the R, G, and B data by a gain value for white balance correction. The gain value for white balance correction is set based on the type of light source. The type of light source is identified, for example, by analyzing the RAW data, or it is manually selected by the user.

[0082] The gamma correction unit 210E2 performs gamma correction on the image data after white balance correction. The gamma correction is performed, for example, using a lookup table (LUT). For example, the gamma correction is performed by referencing an LUT having input / output characteristics for gamma correction and reading output data corresponding to input data from the LUT.

[0083] The demosaic unit 210E3 performs demosaic processing on the gamma-corrected image data. Demosaic processing is a process for storing missing color components in each pixel in a digital camera that uses a single-chip image sensor with a predetermined color filter array. The demosaic unit 210E3 interpolates the missing color component data in each pixel using information from surrounding pixels. Demosaic processing generates image data in which R, G, and B data are complete for each pixel (image data of color components consisting of R, G, and B).

[0084] The YC conversion unit 210E4 performs a process (RGB / YC conversion process) of converting the demosaiced image data (RGB data) into image data (YCbCr data) consisting of luminance data Y and color difference data Cb and Cr. The YC conversion unit 210E4 converts the R, G, and B data of each pixel into luminance data Y and color difference data Cb and Cr using, for example, a predetermined conversion formula. The conversion formula is configured, for example, by multiplying each piece of R, G, and B data by a predetermined coefficient and adding them up. Therefore, the generated luminance data Y and color difference data Cb and Cr are configured as data obtained by weighting and mixing the R, G, and B data.

[0085] In this embodiment, the image data after YC conversion (YCbCr data) is an example of first image data.

[0086] The contour / color tone correction unit 210E5 performs a process (contour correction process) on the luminance data Y to emphasize contours (portions with large luminance changes), and also performs a color tone correction process on the color difference data Cb and Cr to achieve good color reproducibility. The color tone correction is performed, for example, by a matrix operation using a predetermined color correction matrix.

[0087] As described above, in the digital camera 1 of this embodiment, development processes include white balance correction, gamma correction, demosaic processing, RGB / YC conversion processing, contour correction processing, and color correction processing, etc. The development processing unit 210E performs development processes on the image data (RAW data) output from the image sensor 110 to generate image data in YCbCr format (YCbCr data) consisting of luminance data Y and color difference data Cb and Cr.

[0088] The image data acquisition unit 210A acquires the developed image data (YCbCr data) and supplies it to the pre-processing unit 210C.

[0089] [Lens Information Acquisition Unit] The lens information acquisition unit 210B acquires lens information of the interchangeable lens 10 attached to the camera body 100.

[0090] Lens information is information that indicates the specifications of an interchangeable lens, and includes, for example, lens model data, lens characteristic data, lens characteristic correction data, and the like.

[0091] The lens model data indicates the main specifications of an interchangeable lens, and includes the lens model name, focal length, maximum aperture, and manufacturer name.

[0092] The lens characteristic data indicates the characteristics of the interchangeable lens, and includes luminance shading data, color shading data, distortion data, aberration data, and the like.

[0093] The lens characteristic correction data is data for correcting image quality degradation (distortion, color unevenness, vignetting, etc.) caused by the characteristics of an interchangeable lens. The lens characteristic correction data includes luminance shading correction data, color shading correction data, distortion correction data, aberration correction data, etc.

[0094] Here, luminance shading is a phenomenon in which, when capturing an image of a surface with uniform luminance, the luminance decreases with increasing distance from the center (optical axis). Luminance shading is also called vignetting or peripheral light falloff. Luminance shading varies depending on the lens conditions (focal length, aperture value, etc.).

[0095] Luminance shading correction refers to correcting luminance shading through image processing. Luminance shading correction data refers to data necessary for luminance shading correction.

[0096] Color shading is a phenomenon in which the intensity ratio of each color signal varies depending on the location when an image of a surface with uniform brightness is captured, and is perceived as color unevenness. The amount of color shading varies depending on the difference in the angle of incidence of light rays on the light-receiving surface, and the greater the difference in the angle of incidence, the greater the amount of color shading that occurs. Since the distribution of the direction of incidence of light rays also changes depending on the lens conditions (focal length, aperture value, etc.), color shading also changes depending on the lens conditions.

[0097] Color shading correction refers to correcting color shading through image processing. Color shading correction data refers to data necessary for color shading correction.

[0098] Distortion is a phenomenon in which distortion occurs on the focal plane, causing straight lines to not appear as straight lines. Distortion is also called distortion aberration or field curvature. Depending on the type of distortion, it appears in the image as barrel distortion, pincushion distortion, etc.

[0099] Distortion correction refers to correcting distortion through image processing, and distortion correction data refers to data necessary for distortion correction.

[0100] Aberration data includes information on various aberrations (such as chromatic aberration of magnification) that occur due to the characteristics of interchangeable lenses. Aberration correction refers to correcting the various aberrations that occur due to the optical characteristics of these interchangeable lenses through image processing. Aberration correction data is the data required for aberration correction.

[0101] The lens information is stored in the memory 52 of the lens microcomputer 50 (for example, in a ROM or an EEPROM).

[0102] The processor 210 of the camera body 100 communicates with the processor 51 of the interchangeable lens 10 and acquires lens information from the processor 51 of the interchangeable lens 10. The lens information is acquired when the camera body 100 is turned on and when the lens is replaced.

[0103] The lens information acquisition unit 210B adds the lens information acquired from the interchangeable lens 10 to the pre-processing unit 210C.

[0104] In the present embodiment, the information included in the lens information is an example of information necessary for correcting image quality degradation caused by the optical system, and the lens information is an example of first information.

[0105] [Pre-processing unit] The pre-processing unit 210C performs predetermined image processing on the image data for distance measurement to correct image quality degradation caused by the optical system. That is, it corrects image quality degradation caused by the characteristics of the interchangeable lens used to capture the image data for distance measurement. In this embodiment, the image data for distance measurement is corrected using lens characteristic correction data included in the lens information acquired by the lens information acquisition unit 210B.

[0106] As described above, the lens characteristic correction data is data for correcting image quality degradation caused by the optical characteristics of the interchangeable lens. The lens characteristic correction data includes luminance shading correction data, color shading correction data, distortion correction data, aberration correction data, etc. The pre-processing unit 210C uses the lens characteristic correction data to correct image data for distance measurement. That is, the lens characteristic correction data is used to correct luminance shading, color shading, distortion, aberration, etc. through image processing. Note that this type of correction technology is well known, so detailed description thereof will be omitted.

[0107] [Distance Image Data Generation Unit] The distance image data generation unit 210D performs image analysis of the image data for distance measurement, performs distance measurement, and generates distance image data. In this embodiment, the distance image data is generated from the image data for distance measurement using a trained model generated by machine learning (hereinafter referred to as "distance image data generation AI").

[0108] The distance image data generation AI is generated by machine learning a model to output distance image data from image data captured via an optical system.

[0109] The range image data generation AI is generated by applying a known machine learning algorithm such as a neural network, which includes a convolutional neural network (CNN), a fully connected neural network, and a recurrent neural network.

[0110] FIG. 5 is a conceptual diagram of model learning.

[0111] Model training is performed, for example, by inputting training image data (training image data) into the model and feeding back the error between the predicted distance value and the correct value (correct label) to the model. The correct value is the actual distance to the subject included in the training image data. Feedback refers to updating the model parameters (e.g., weighting coefficients) so that the error is reduced.

[0112] The generation of the distance image data generation AI is performed using a learning device. The learning device is configured, for example, by a computer. That is, the computer functions as a learning device by executing a predetermined program. The computer that constitutes the learning device has a configuration including at least a processor and a memory. As an example, the computer that constitutes the learning device has a configuration including a processor, memory, an auxiliary storage device, an output device, and an input device.

[0113] The learning device uses a predetermined data set (learning data set) to train a model that constitutes the range image data generation AI. The learning data set is composed of a set of predetermined image data (learning image data) and its corresponding ground truth data.

[0114] The learning image data is composed of, for example, images captured by a digital camera, that is, image data captured by an image sensor via an optical system.

[0115] The correct answer data is data that indicates the correct value of the corresponding training image data. In this embodiment, the correct answer value is data on the distance to the subject (subject distance) included in the corresponding training image data. Therefore, the correct answer data is composed of data on the distance to the subject for each pixel included in the corresponding training image data.

[0116] The device (digital camera) that captures the learning image data is not particularly limited. Therefore, it does not necessarily have to be an imaging device with the same configuration as the imaging device that will be equipped with the distance image data generation AI. However, the image data used for learning image data should be image data that is almost free of image quality degradation caused by the optical system, especially image quality degradation that affects distance measurement.

[0117] There are no particular limitations on the method for suppressing image quality degradation caused by the optical system in training image data. When a lens that is almost free of image quality degradation caused by the optical system (a lens in which distortion, color unevenness, vignetting, etc. are corrected by lens design) is used, the image data obtained by imaging can be used as image data for training as is. On the other hand, when a lens that causes image quality degradation caused by the optical system is used, image processing is performed on the image data obtained by imaging to correct the image quality degradation caused by the optical system.

[0118] As described above, in this embodiment, image data for distance measurement is input to the distance image data generation AI after correcting image quality degradation caused by the optical system. Therefore, even in the learning stage, machine learning of the model is performed using image data with almost no image quality degradation (optically ideal image data).

[0119] The generated distance image data generation AI is stored in memory 220 of camera microcomputer 200. Processor 210 functioning as distance image data generation unit 210D generates distance image data from the image data for distance measurement using the distance image data generation AI stored in memory 220. That is, the image data for distance measurement is input to the distance image data generation AI, and distance image data is obtained that is output as the processing result.

[0120] [Distance Measurement (Generation of Distance Image Data)] Next, a distance measurement method (a method of generating distance image data) by the digital camera 1 of this embodiment will be described.

[0121] 6 is a flowchart showing a procedure for measuring distance. The procedure for measuring distance shown in FIG. 6 is an example of an image data processing method.

[0122] First, image data for distance measurement is acquired (step S101). In this embodiment, the image data for distance measurement is image data obtained by capturing an image and is image data after development processing. In this embodiment, image data in YCbCr format (YCbCr data) is acquired as the image data after development processing.

[0123] Next, preprocessing is performed on the acquired image data for distance measurement (step S102). In the preprocessing, predetermined image processing is performed on the image data for distance measurement to correct image quality degradation caused by the optical system. In this embodiment, image processing is performed using lens information about the interchangeable lens 10 that captured the image data, and image quality degradation caused by the optical system is corrected. It is assumed that the lens information has been acquired in advance.

[0124] Next, the distance image data generation AI is used to generate distance image data from the preprocessed image data for distance measurement (step S103). That is, the preprocessed image data is input to the distance image data generation AI, and distance image data is output as the processing result.

[0125] As described above, the distance image data generation AI of this embodiment generates image data that is almost free of image quality degradation due to the optical system for learning. In the digital camera 1 of this embodiment, image data for distance measurement is corrected for image quality degradation due to the optical system and input to the distance image data generation AI. This eliminates the influence of the optical system and allows for accurate distance image data to be acquired. Furthermore, accurate distance image data can be stably acquired even when the lens is changed.

[0126] The generated distance image data is used according to the application. For example, it is visualized as a distance image (distance map) and output to a display destination (the rear monitor 150 or the EVF 160). For example, when distance image data is generated from image data captured for display or recording, and the visualized distance image is to be displayed together with the captured image, the distance image can be displayed superimposed on the captured image (so-called superimposed display). In superimposed display, for example, the distance image can be displayed semi-transparently. Alternatively, the captured image and the distance image can be displayed side by side. Furthermore, for example, the distance image data is recorded in a recording destination (the storage unit 130) in a predetermined format. For example, the distance image data is associated with the captured image data (image data for recording) and recorded in the recording destination. Additionally, the distance image data can also be used for AF.

[0127] FIG. 7 is a diagram showing an example of visualization of distance image data.

[0128] Fig. 7A shows a captured image. Fig. 7B shows an example of visualization of distance image data. Fig. 7B shows an example of visualization of distance image data by displaying it with a density corresponding to the distance. In this case, the distance value (pixel value) of each pixel is converted into a density value to generate a distance image (distance map).

[0129] As described above, according to this embodiment, image quality degradation (image quality degradation due to the optical system) that occurs in the image data for distance measurement is corrected before the image data for distance measurement is input to the distance image data generation AI. This allows for stable generation of accurate distance image data even when the lens is replaced. That is, even when a lens different from that used during learning is used, accurate distance image data can be stably generated. Furthermore, generation of the distance image data generation AI (model learning) can be facilitated. That is, there is no need to prepare image data captured with every lens currently available in the world as learning image data or to learn using a huge amount of image data, making it easy to generate the distance image data generation AI. Furthermore, it can also be used when lenses that have been sold after learning are used, allowing for accurate distance measurements.

[0130] [Modification] As described above, image quality degradation caused by the optical system varies depending on the focal length (zoom position), aperture value (F-number), and the like. Therefore, in preprocessing, it is preferable to correct each item taking into account this information (information on focal length, aperture value, etc.). That is, it is preferable to obtain information on focal length, aperture value, etc. at the time of image capture and perform correction with a corresponding correction amount. For example, when performing distortion correction, information on focal length and aperture value at the time of image capture is obtained, and distortion correction is performed with a correction amount corresponding to the focal length and aperture value at the time of image capture. This enables more accurate correction.

[0131] Second Embodiment As described above, by using the distance image data generation AI, distance image data can be generated from image data obtained by imaging. Distance image data can be generated not only from image data captured for distance measurement purposes, but also from image data captured for display and recording. Therefore, for example, distance image data can be generated from image data captured for live view, or from image data captured for recording.

[0132] When distance image data is generated using distance image data generation AI from image data captured for display or recording, it is preferable to be able to correct image quality degradation caused by the optical system even for image data for display or recording and output it to the display or recording destination.

[0133] On the other hand, some users regard distortion and shading as the individual characteristics of the lens (the so-called "flavor of the lens") and prefer not to correct them. For example, many users regard luminance shading as the individual characteristics of the lens and use it in their work.

[0134] In this embodiment, when distance image data is generated using distance image data generation AI from image data captured for display or recording, different corrections are possible for the image data for display or recording and the image data for distance measurement.

[0135] In the embodiment described below, an example will be described in which, in parallel with the output of a live view, distance image data is generated from image data captured for the live view.

[0136] FIG. 8 is a block diagram of the main functions of the digital camera relating to live view output and distance measurement.

[0137] 8 , with regard to live view output and distance measurement, the digital camera 1 has functions such as an image data acquisition unit 211A, a lens information acquisition unit 211B, a correction setting information acquisition unit 211C, a first pre-processing unit 211D, a display image data generation unit 211E, a second pre-processing unit 211F, and a distance image data generation unit 211G. The functions of each unit are realized by a processor 210 of the camera body 100. The processor 210 realizes the functions of each unit by executing a predetermined program (image data processing program).

[0138] [Image Data Acquisition Unit] The image data acquisition unit 211A acquires image data to be processed. In this embodiment, the image data to be processed is image data captured for live view. Image data for live view is captured at a predetermined frame rate. The image data acquisition unit 211A acquires image data captured for live view in chronological order. In this embodiment, image data (YCbCr data) that has been captured for live view and has been developed is acquired. The acquired image data (YCbCr data) is applied to the first pre-processing unit 211D and the second pre-processing unit 211F.

[0139] The image data applied to the first pre-processing unit 211D is used for live view, and the image data applied to the second pre-processing unit 211F is used for distance measurement. Hereinafter, as necessary, the image data used for live view will be referred to as image data for live view, and the image data used for distance measurement will be referred to as image data for distance measurement, to distinguish between the two.

[0140] [Lens Information Acquisition Unit] The lens information acquisition unit 211B acquires lens information of the interchangeable lens 10 attached to the camera body 100. The acquired lens information is applied to a first pre-processing unit 211D and a second pre-processing unit 211F.

[0141] [Setting Information Acquisition Unit] The correction setting information acquisition unit 211C acquires setting information (correction setting information) for correction to be performed on image data for live view.

[0142] In this embodiment, corrections made to image data for recording are also reflected in image data for live view, so the correction setting information acquisition unit 211C acquires setting information for corrections made to image data for recording as setting information for corrections made to image data for live view.

[0143] In this embodiment, the correction setting is a setting of correction ON or OFF. The correction setting is performed, for example, on a predetermined setting screen. The user uses the camera operation unit 170 to call up a correction setting screen on the rear monitor 150 or EVF 160 and set correction ON or OFF. When correction is set to ON, correction is enabled. When correction is set to OFF, correction is disabled. Information on the set correction ON or OFF is stored, for example, in memory 220. The correction setting information acquisition unit 211C reads and acquires the correction setting information from the memory 220. The acquired correction setting information is added to the first pre-processing unit 211D.

[0144] [First Pre-Processing Unit] As described above, image data for live view is applied to the first pre-processing unit 211D.

[0145] The first pre-processing unit 211D performs predetermined image processing on the image data for live view to correct image quality degradation caused by the optical system. As with the pre-processing unit 210C of the first embodiment, the first pre-processing unit 211D corrects the image data for live view using lens characteristic correction data included in the lens information acquired by the lens information acquisition unit 211B.

[0146] The first preprocessing unit 211D also corrects the image data for live view based on the correction setting information provided by the correction setting information acquisition unit 211C. That is, the first preprocessing unit 211D corrects the image data for live view only when correction is set to ON. When correction is set to OFF, the image data is output without being corrected.

[0147] [Display Image Data Generation Unit] The display image data generation unit 211E generates image data for display (display image data) from the image data for live view output from the first preprocessing unit 211D. That is, it converts the image data into image data in a format that conforms to the output format of the display destination, and generates display image data. The generated display image data is output to the display destination (rear monitor 150 and / or EVF 160) and displayed as a live view. In this embodiment, the display image data is an example of third image data.

[0148] [Second Pre-Processing Unit] As described above, image data for distance measurement is applied to the second pre-processing unit 211F.

[0149] The second preprocessing unit 211F performs predetermined image processing on the image data for distance measurement to correct image quality degradation caused by the optical system. Similar to the first preprocessing unit 211D, the second preprocessing unit 211F corrects the image data for distance measurement using lens characteristic correction data included in the lens information acquired by the lens information acquisition unit 211B. While the first preprocessing unit 211D allows the user to turn correction on and off, the second preprocessing unit 211F always performs correction. The image data for distance measurement, for which image quality degradation caused by the optical system has been corrected, is sent to the distance image data generation unit 211G.

[0150] [Distance Image Data Generation Unit] The distance image data generation unit 211G performs image analysis of the pre-processed image data for distance measurement, performs distance measurement, and generates distance image data. As with the distance image data generation unit 210D in the first embodiment, the distance image data generation unit 211G uses a distance image data generation AI to generate distance image data from the pre-processed image data for distance measurement. The generated distance image data is used according to the application.

[0151] The distance image data can be used for distance measurement, for example, and can also be used for various image processing. For example, by referencing the distance image data, it is possible to control the blurring or brightness of an image. For example, by referencing the distance image data, it is possible to blur only distant areas of an image and highlight foreground areas. Also, by referencing the distance image data, it is possible to darken distant areas and brighten foreground areas.

[0152] [Operation] Here, an example will be described in which distance measurement (generation of distance image data) is performed from image data captured for live view at the same time as live view output.

[0153] FIG. 9 is a flowchart showing the procedure of the live view output and distance measurement process.

[0154] First, an image for live view is captured (step S201). Image data (RAW data) obtained by capturing the image is developed (step S202) and input to the first pre-processing unit 211D and the second pre-processing unit 211F.

[0155] Here, the first pre-processing unit 211D receives the image data (YCrCb data) after development processing as image data for live view. Meanwhile, the second pre-processing unit 211F receives the image data after development processing as image data for distance measurement. Below, the live view output process and the distance measurement process will be explained separately.

[0156] (1) Live View Output First, it is determined whether correction is ON (step S203).

[0157] If the correction is ON, the first preprocessing unit 211D performs preprocessing on the image data for live view (step S204). That is, image processing corrects image quality degradation caused by the optical system. The preprocessed image data for live view is sent to the display image data generation unit 211E.

[0158] On the other hand, when the correction is turned off, the first preprocessing unit 211D does not perform preprocessing, and the image data is directly sent to the display image data generating unit 211E.

[0159] The display image data generation unit 211E converts the image data for live view into image data in a format that conforms to the output format of the display destination, thereby generating image data (display image data) of an image to be displayed as live view (display image) (step S205).

[0160] The generated display image data is output to the display destination (the rear monitor 150 and / or the EVF 160) (step S206) and displayed as a live view.

[0161] Thereafter, it is determined whether or not the live view has been turned off (step S210). If the live view has been turned off, the process ends. On the other hand, if the live view has not been turned off, the next image to be displayed is captured (step S201).

[0162] Live view imaging is performed at a predetermined frame rate, so the processes from steps S201 to S206 are repeated until the live view is turned off.

[0163] (2) Distance Measurement (Generation of Distance Image Data) Next, the second pre-processing unit 211F performs pre-processing on the image data for distance measurement (step S207).

[0164] Next, distance image data is generated from the preprocessed image data for distance measurement using the distance image data generation AI (step S208). The image data input to the distance image data generation AI has been preprocessed to correct image quality degradation caused by the optical system, so it is possible to obtain highly accurate distance image data without being affected by the optical system.

[0165] The generated distance image data is output to an output destination according to the intended use (step S209).

[0166] Distance measurement is performed in synchronization with capturing an image for live view, so when live view is turned off, distance measurement also ends.

[0167] As described above, according to this embodiment, when generating distance image data from image data captured for live view, correction can be turned on or off as desired for the image data for live view. This allows the user to view an image as intended by the user as live view. Meanwhile, image data that has been subjected to correction processing is always input to the distance image data generation AI. This enables highly accurate distance measurement.

[0168] [Modification] [Correction process for image data for recording] When generating distance image data from image data captured for recording, it is preferable to process it in the same way as in the case of live view. That is, it is preferable to perform preprocessing separately from the image data for distance measurement, and to configure it so that correction can be enabled or disabled as desired.

[0169] The image data captured for recording is preprocessed by the first preprocessing unit 211D, converted into a predetermined recording format, and recorded in the recording destination (storage unit 130). The distance image data generated from the image data captured for recording is associated with the image data for recording and recorded in the recording destination.

[0170] [Correction of image data for display or recording] In the above embodiment, it is only possible to set correction ON / OFF for image data for display and recording, but instead of or in addition to switching ON / OFF, it is also possible to configure the correction strength to be changeable.

[0171] Furthermore, when the correction content is diverse, a configuration may be adopted in which the correction can be individually turned on / off or the correction intensity can be individually set. For example, when correcting image quality degradation caused by the optical system, such as luminance shading correction, color shading correction, distortion correction, and aberration correction, each item can be individually turned on / off and its intensity can be individually set. As an example, in addition to an ON / OFF setting for distortion correction (distortion aberration correction), a configuration may be adopted in which barrel distortion and pincushion distortion can be corrected in three levels: strong, medium, and weak. Furthermore, for example, a configuration may be adopted in which color shading, such as differences in color between the center and periphery of an image, can be individually corrected for each of the four corners of the image. Furthermore, the brightness shading correction (peripheral light amount correction) is configured to be able to correct the peripheral light amount in a total of 11 levels of correction (0 is OFF): -5, -4, -3, -2, -1, 0, +1, +2, +3, +4, and +5.

[0172] Furthermore, the display and recording may be separated, and correction settings may be made separately for each.

[0173] In the above embodiment, the user can set the correction to ON / OFF, but the correction may be always performed or may be automatically turned ON / OFF depending on the conditions.

[0174] [Preprocessing] In the above embodiment, the preprocessing is configured to correct only image quality degradation caused by the optical system, but other corrections or image processing can also be performed. For example, the preprocessing may be configured to correct brightness, saturation, etc., for image data for display and / or recording.

[0175] Furthermore, for example, as preprocessing of image data for distance measurement, the brightness of the image may be corrected so that image data whose brightness is always above a certain level or within a certain range is input to the distance image data generation AI. For example, the photometry results obtained by AE may be used to adjust the brightness.

[0176] [Third embodiment] When distance image data is generated using a distance image data generation AI, a misalignment (a misalignment in the corresponding positional relationship within the image) may occur between the input image (image data for distance measurement) and the output image (distance image data). If a misalignment occurs, for example, the two images cannot be correctly superimposed when superimposed, resulting in reduced visibility.

[0177] The misalignment occurs, for example, when different corrections are made to image data for display or recording and image data for distance measurement. For example, when distortion correction is performed with different correction amounts, misalignment may occur in the peripheral parts of the image (misalignment may occur mainly in the peripheral parts of the image because the enlargement / reduction process is performed locally depending on the image height). In addition, misalignment may occur due to the distance image data generation AI. Misalignment is an example of a difference that occurs between image data for display or recording and distance image data.

[0178] In this embodiment, when distance image data is generated using distance image data generation AI from image data captured for display or recording, post-processing is performed on the distance image data generated by the distance image data generation AI to reduce the discrepancy that occurs between the image data for display or recording.

[0179] In the embodiment described below, an example will be described in which distance image data is generated from image data captured for live view, and the distance image is displayed superimposed on a live view image.

[0180] FIG. 10 is a block diagram of the main functions of the digital camera relating to live view output and distance measurement.

[0181] 10 , with regard to live view output and distance measurement, the digital camera 1 has functions such as an image data acquisition unit 212A, a lens information acquisition unit 212B, a correction setting information acquisition unit 212C, a first pre-processing unit 212D, a display image data generation unit 212E, a second pre-processing unit 212F, a distance image data generation unit 212G, and a post-processing unit 212H. The functions of each unit are realized by a processor 210 of the camera body 100. The processor 210 realizes the functions of each unit by executing a predetermined program (image data processing program).

[0182] [Image Data Acquisition Unit] The image data acquisition unit 212A acquires image data to be processed. In this embodiment, the image data to be processed is image data captured for live view. The image data acquisition unit 212A acquires image data (YCbCr data) that has been captured for live view and has been developed. The acquired image data (YCbCr data) is applied to the first pre-processing unit 212D and the second pre-processing unit 212F.

[0183] The image data applied to the first pre-processing unit 212D is used for live view, and the image data applied to the second pre-processing unit 212F is used for distance measurement. Hereinafter, as necessary, the image data used for live view will be referred to as image data for live view, and the image data used for distance measurement will be referred to as image data for distance measurement, to distinguish between the two.

[0184] In this embodiment, the image data (YCrCb data) acquired by the image data acquisition unit 212A is an example of first image data.

[0185] [Lens Information Acquisition Unit] The lens information acquisition unit 212B acquires lens information of the interchangeable lens 10 attached to the camera body 100. The acquired lens information is applied to a first pre-processing unit 212D and a second pre-processing unit 212F.

[0186] [Setting Information Acquisition Unit] The correction setting information acquisition unit 212C acquires setting information for correction to be performed on image data for live view.

[0187] In this embodiment, corrections made to image data for recording are also reflected in image data for live view, so the correction setting information acquisition unit 211C acquires setting information for corrections made to image data for recording as setting information for corrections made to image data for live view.

[0188] As an example, in this embodiment, correction of luminance shading, color shading, distortion, and aberration is possible. Each correction can be individually set to ON / OFF, and the intensity can be set. Correction settings are performed, for example, on a predetermined setting screen. Setting information for each correction is stored, for example, in memory 220. The correction setting information acquisition unit 212C reads and acquires setting information for each correction from memory 220. The acquired setting information for each correction is applied to the first pre-processing unit 212D and the post-processing unit 212H.

[0189] [First Pre-Processing Unit] The first pre-processing unit 212D performs image processing on image data for live view based on correction setting information. In this embodiment, correction of luminance shading, color shading, distortion, and aberration is possible. The first pre-processing unit 212D performs correction of luminance shading, color shading, distortion, and aberration on image data for live view based on the setting information for each correction. The correction is performed using lens characteristic correction data.

[0190] [Display Image Data Generation Unit] The display image data generation unit 212E generates image data for display (display image data) from the image data for live view output from the first pre-processing unit 212D. The generated display image data is output to the display destination (the rear monitor 150 and / or the EVF 160) and displayed as a live view. In this embodiment, the display image data is an example of fourth image data.

[0191] [Second Pre-Processing Unit] The second pre-processing unit 212F performs predetermined image processing on the image data for distance measurement to correct image quality degradation caused by the optical system. The image data for distance measurement for which image quality degradation caused by the optical system has been corrected is sent to the distance image data generating unit 212G.

[0192] [Distance Image Data Generation Unit] The distance image data generation unit 212G generates distance image data from the pre-processed image data for distance measurement using the distance image data generation unit AI. The generated distance image data is sent to the post-processing unit 212H.

[0193] The post-processing unit 212H performs predetermined image processing on the generated distance image data to reduce misalignment with the display image data. That is, the post-processing unit 212H performs image processing on the distance image data so that when the distance image is displayed superimposed on the display image, the two are displayed without misalignment.

[0194] The deviation is mainly caused by differences in the correction content in pre-processing. In particular, when distortion is corrected as pre-processing, the deviation occurs due to differences in the amount of correction. As an example, the post-processing unit 212H performs image processing on the distance image data based on information (such as information about the amount of correction) about the distortion correction performed on the image data for live view by the first pre-processing unit 212D and information (such as information about the amount of correction) about the distortion correction performed on the image data for distance measurement by the second pre-processing unit 212F.

[0195] The post-processed distance image data is visualized and output as a distance image to a display destination. The distance image is generated, for example, by representing the distance value of each pixel in the distance image data with color or density. In this embodiment, the distance image is displayed superimposed on the live view image. For example, the distance image is made semi-transparent and displayed superimposed on the live view image.

[0196] In this embodiment, the distance image data before post-processing is an example of first distance image data, and the distance image data after post-processing is an example of second distance image data.

[0197] [Function] Here, an example will be described in which all corrections to image data for live view (i.e., corrections to image data for recording) are turned off. In this case, distortion, vignetting (brightness shading), and the like appear in the image displayed as live view due to the characteristics of the interchangeable lens 10.

[0198] FIG. 11 is a flowchart showing the procedure for outputting a live view.

[0199] First, an image for live view is captured (step S301). The image data (RAW data) obtained by capturing the image is developed (step S302) and sent to the first pre-processing unit 212D and the second pre-processing unit 212F. The image data after the development process (YCrCb data) is sent to the first pre-processing unit 212D as image data for live view. Meanwhile, the image data after the development process is sent to the second pre-processing unit 212F as image data for distance measurement.

[0200] The image data for live view sent to the first pre-processing unit 212D is pre-processed in accordance with the correction settings made by the user (step S303). As described above, in this example, all corrections are turned off, so the first pre-processing unit 212D does not perform pre-processing, and the image data is sent directly to the display image data generation unit 212E.

[0201] The display image data generation unit 212E converts the image data for live view into image data in a format that conforms to the output format of the display destination, thereby generating image data (display image data) of an image (display image) to be output as a live view (step S304).

[0202] The generated display image data is output to the display destination (the rear monitor 150 and / or the EVF 160) (step S305) and displayed as a live view. As described above, distortions and the like due to the characteristics of the interchangeable lens 10 appear in the displayed image.

[0203] On the other hand, the image data for distance measurement that has been applied to the second pre-processing section 212F undergoes predetermined pre-processing in the second pre-processing section 212F, and image quality degradation caused by the optical system is corrected (step S306).

[0204] The pre-processed image data for distance measurement is sent to the distance image data generation unit 212G, which uses the distance image data generation AI to generate distance image data from the image data for distance measurement (step S307).

[0205] The generated distance image data is sent to the post-processing unit 212H, where it is post-processed (step S308). The post-processing unit 212H corrects the distance image data to reduce the discrepancy between the distance image data and the display image data. Specifically, image processing is performed on the distance image data based on information about the distortion correction performed on the image data for live view and information about the distortion correction performed on the image data for distance measurement, thereby reducing the discrepancy between the distance image data and the display image data (disparity due to distortion). In this example, correction of the image data for live view is turned off, so image processing is performed on the distance image data based on information about the distortion correction performed on the image data for distance measurement.

[0206] The post-processed depth image data is visualized and output as a depth image to a display destination (step S309). In this embodiment, the depth image is displayed superimposed on the live view image.

[0207] Thereafter, it is determined whether or not the live view has been turned off (step S310). If the live view has been turned off, the process ends. On the other hand, if the live view has not been turned off, the next image to be displayed is captured (step S301).

[0208] Live view imaging is performed at a predetermined frame rate, so the processes from steps S301 to S309 are repeated until the live view is turned off.

[0209] FIG. 12 is a diagram showing the flow of processing up to the generation of a live view image and a distance image.

[0210] FIG. 12 shows an example in which distortion (for example, pincushion distortion) occurs in a captured image, and shows an example in which a live view is output without correcting the distortion.

[0211] The image data Im0 obtained by capturing an image is developed and then input to a first pre-processing unit 212D and a second pre-processing unit 212F.

[0212] The image data (image data for live view) supplied to the first preprocessing unit 212D is subjected to preprocessing according to the settings and then supplied to the display image data generation unit 212E. As described above, in this example, live view is output without correcting distortion. Therefore, in this example, image data Im1 in a distorted state is supplied to the display image data generation unit 212E.

[0213] The display image data generating unit 212E generates display image data Lv from the image data Im1 in a distorted state and outputs the generated display image data to a display destination. Therefore, the display image data Lv is composed of image data in which distortion has occurred.

[0214] On the other hand, the image data (image data for distance measurement) input to the second preprocessing unit 212F is subjected to predetermined preprocessing to correct image quality degradation caused by the optical system. Therefore, the image data Im2 after preprocessing is composed of image data with distortion corrected.

[0215] The image data Im2, for which image quality degradation due to the optical system has been corrected by pre-processing, is sent to the distance image data generation unit 212G. The distance image data generation unit 212G generates distance image data Di1 based on the image data Im2 for which image quality degradation due to the optical system has been corrected (the distance image data Di1 is generated using the distance image data generation AI). Therefore, the generated distance image data Di1 also consists of distance image data without distortion. If this distance image data Di1 is superimposed on display image data Lv for which distortion has not been corrected, a misalignment will occur between the two. For this reason, the distance image data Di1 generated by the distance image data generation unit 212G is sent to the post-processing unit 212H, where it is post-processed.

[0216] The post-processing unit 212H performs post-processing on the distance image data Di1 to reduce the discrepancy between the distance image data Di1 and the display image data Lv. In this example, distortion appears in the display image data Lv. Therefore, image processing is performed on the distance image data Di1 so that the same distortion as in the display image data Lv appears (image processing is performed so that the same pincushion distortion appears).

[0217] The misalignment between the post-processed distance image data Di2 and the display image data Lv is reduced, so that when the distance image is superimposed on an image displayed as a live view (display image), the two can be displayed without misalignment.

[0218] As described above, according to this embodiment, by performing post-processing on the distance image data, it is possible to reduce discrepancies that occur between the distance image data and image data for display or recording. This improves visibility, for example, when a distance image is displayed superimposed on a captured image. Furthermore, it is possible to accurately associate the distances of each subject in the image.

[0219] [Modifications] [Post-processing unit] In the above embodiment, the distance image data is corrected to reduce deviations due to distortion correction, but deviations (differences) between the distance image data output from the distance image data generation AI and the image data for display or recording can also be caused by other factors. Therefore, it is preferable that the post-processing unit 212H be configured to eliminate deviations caused by other factors.

[0220] For example, if a deviation occurs due to the distance image data generation AI, the data necessary for correction is collected in advance, correction information is generated based on the collected data, and the distance image data is corrected using the generated correction information.

[0221] [Processing of Image Data for Recording] When generating distance image data from image data captured for recording, it is preferable to process it in the same way as in the case of live view. For example, when distance image data is generated from image data captured for recording and the generated distance image data is recorded in association with the image data for recording, post-processing is performed on the distance image data to reduce discrepancies that occur with the image data for recording. This reduces discrepancies and improves visibility, for example, when a recorded image is played back and displayed with the distance image superimposed.

[0222] [Fourth Embodiment] In the third embodiment, the misalignment between the image data for display or recording and the distance image data is reduced by performing post-processing on the distance image data.

[0223] In this embodiment, post-processing is performed on image data for display or recording to reduce discrepancies that occur between image data for display or recording and distance image data.

[0224] The following description will be given taking as an example a case where distance image data is generated from image data captured for live view, and the distance image is displayed superimposed on a live view image.

[0225] FIG. 13 is a block diagram of the main functions of the digital camera relating to live view output and distance measurement.

[0226] 13 , with regard to live view output and distance measurement, the digital camera 1 has functions such as an image data acquisition unit 213A, a lens information acquisition unit 213B, a correction setting information acquisition unit 213C, a first pre-processing unit 213D, a display image data generation unit 213E, a second pre-processing unit 213F, a distance image data generation unit 213G, and a post-processing unit 213H. The functions of each unit are realized by a processor 210 in the camera body 100. The processor 210 realizes the functions of each unit by executing a predetermined program (image data processing program).

[0227] Note that the functions other than those of post-processing unit 213H, i.e., the functions of image data acquisition unit 213A, lens information acquisition unit 213B, correction setting information acquisition unit 213C, first pre-processing unit 213D, display image data generation unit 213E, second pre-processing unit 213F, and distance image data generation unit 213G, are substantially the same as the functions of image data acquisition unit 212A, lens information acquisition unit 212B, correction setting information acquisition unit 212C, first pre-processing unit 212D, display image data generation unit 212E, second pre-processing unit 212F, and distance image data generation unit 212G in digital camera 1 of the third embodiment. Therefore, only the functions of post-processing unit 213H will be described here.

[0228] [Post-processing unit] The post-processing unit 213H performs image processing on the image data for live view to reduce the deviation that occurs between the image data for live view and the depth image data. As an example, in this embodiment, the image processing is performed on the image data for live view so that the distortion is approximately the same as that of the image data input to the depth image data generation AI.

[0229] As described above, if the correction amount (strength) of the distortion correction in the first pre-processing unit 213D and the distortion correction in the second pre-processing unit 213F differs, a misalignment may occur between the image displayed as live view (display image) and the distance image. In this embodiment, image processing is performed on the image data for live view so that the distortion is approximately the same as that of the image data input to the distance image data generation AI, thereby reducing the misalignment that occurs between the image data for live view and the distance image data.

[0230] The post-processing unit 213H performs image processing on the image data for live view based on information on the correction performed by the first pre-processing unit 213D and information on the correction performed on the image data for distance measurement by the second pre-processing unit 213F.

[0231] In the present embodiment, the image data before post-processing by the post-processing unit 213H is an example of fifth image data, and the image data after post-processing is an example of sixth image data.

[0232] [Operation] Here, an example will be described in which all corrections for image data to be recorded (corrections for image quality degradation caused by the optical system) are turned off. The settings for corrections for image data to be recorded are also applied to live view.

[0233] FIG. 14 is a flowchart showing the procedure for outputting a live view.

[0234] First, an image for live view is captured (step S401). The image data (RAW data) obtained by capturing the image is developed (step S402) and sent to the first pre-processing unit 213D and the second pre-processing unit 213F. The image data after the development process (YCrCb data) is sent to the first pre-processing unit 213D as image data for live view. Meanwhile, the image data after the development process is sent to the second pre-processing unit 213F as image data for distance measurement.

[0235] The image data for live view that has been applied to the first pre-processing unit 213D is pre-processed in accordance with the correction settings made by the user (step S403). As described above, in this example, all corrections are turned OFF, so no pre-processing is performed in the first pre-processing unit 213D, and the image data is applied directly to the post-processing unit 213H.

[0236] The image data for live view sent to the post-processing unit 213H is subjected to predetermined image processing as post-processing (step S404). In this embodiment, image processing is performed so that the distortion is approximately the same as that of the image data input to the depth image data generation AI. This reduces the discrepancy that occurs between the image data for live view and the depth image data. The post-processed image data for live view is sent to the display image data generation unit 213E.

[0237] The display image data generation unit 213E converts the image data for live view into image data in a format that conforms to the output format of the display destination. As a result, image data (display image data) of the image to be displayed as live view (display image) is generated (step S405). The generated display image data is output to the display destination (rear monitor 150 and / or EVF 160) (step S406) and displayed as live view.

[0238] On the other hand, the image data for distance measurement that has been applied to the second pre-processing section 213F undergoes predetermined pre-processing in the second pre-processing section 213F, and image quality degradation caused by the optical system is corrected (step S407).

[0239] The pre-processed image data for distance measurement is sent to distance image data generation unit 213G, which uses distance image data generation AI to generate distance image data from the image data for distance measurement (step S408).

[0240] The generated depth image data is visualized and output as a depth image to a display destination (step S409), which in this embodiment is displayed superimposed on a live view image.

[0241] Thereafter, it is determined whether or not the live view has been turned off (step S410). If the live view has been turned off, the process ends. On the other hand, if the live view has not been turned off, the next image to be displayed is captured (step S401).

[0242] FIG. 15 is a diagram showing the flow of processing up to the generation of a live view image and a distance image.

[0243] FIG. 15 shows an example in which distortion (for example, pincushion distortion) occurs in the captured image, and shows an example in which the setting for correction of image data for recording (correction of image quality degradation caused by the optical system) is turned OFF.

[0244] The image data Im0 obtained by capturing an image is subjected to a development process and then input to a first pre-processing unit 213D and a second pre-processing unit 213F.

[0245] The image data (image data for distance measurement) sent to the second preprocessing unit 213F is subjected to predetermined preprocessing to correct image quality degradation caused by the optical system. Therefore, the image data Im2 after preprocessing is composed of image data with distortion corrected.

[0246] The image data Im2, for which image quality degradation due to the optical system has been corrected by preprocessing, is sent to the distance image data generation unit 213G. The distance image data generation unit 213G generates distance image data Di based on the image data Im2 for which image quality degradation due to the optical system has been corrected (the distance image data Di is generated using the distance image data generation AI). Therefore, the generated distance image data Di also consists of distance image data without distortion.

[0247] On the other hand, the image data (image data for live view) input to the first pre-processing unit 213D is subjected to pre-processing according to the settings and then input to the post-processing unit 213H. As described above, in this example, correction of the image data for recording is turned OFF, so the image data is input as is to the post-processing unit 213H. Therefore, in this example, image data Im1a in a distorted state is input to the post-processing unit 213H.

[0248] The post-processing unit 213H performs post-processing on the image data Im1a for live view so as to reduce the deviation from the distance image data Di. In this example, image processing is performed so that the distortion is approximately the same as that of the image data Im2 input to the distance image data generation unit AI. As a result of the post-processing, the deviation between the image data Im1b for live view and the distance image data Di is reduced.

[0249] The post-processed live view image data Im1b is added to the display image data generation unit 213E. The display image data generation unit 212E generates display image data Lv from the post-processed live view image data Im1b and outputs it to the display destination. This display image data Lv has reduced deviation from the distance image data Di. Therefore, even when a distance image is superimposed on an image displayed as live view, the two can be displayed without deviation.

[0250] As described above, according to this embodiment, by performing post-processing on image data for live view, it is possible to reduce discrepancies that occur between the image data and depth image data. This improves visibility, for example, when a depth image is superimposed on an image displayed as live view. It also makes it possible to accurately associate the distances of each subject in the image.

[0251] [Modifications] [Post-processing unit] In the above embodiment, the image data for display or recording is corrected so as to reduce deviations due to distortion correction, but deviations (differences) between the image data and the distance image data may also be caused by other factors. Therefore, it is preferable that the post-processing unit 213H be configured to eliminate deviations caused by other factors.

[0252] For example, if a deviation occurs due to distance image data generation AI, the data necessary for correction is collected in advance, correction information is generated based on the collected data, and the generated correction information is used to correct the image data for display or recording.

[0253] [Processing of Image Data for Recording] When generating distance image data from image data captured for recording, similar image processing may be performed to reduce discrepancies that occur with the distance image data.

[0254] In the above embodiment, the first pre-processing unit 213D is configured to perform correction based on the user's settings, but the first pre-processing unit 213D may be omitted. In this case, for example, the post-processing unit 213H performs the same correction as the second pre-processing unit 213F.

[0255] Furthermore, when reproducing recorded image data, similar image processing may be performed to reduce the discrepancy that occurs between the image data and the distance image data.

[0256] [Fifth Embodiment] It is preferable that distance measurement can also be performed from recorded image data. Therefore, in this embodiment, when capturing an image for recording, image data for distance measurement is recorded in association with image data for recording. The image data for distance measurement to be recorded is image data that has been subjected to preprocessing. In other words, it is image data in which image quality degradation due to the optical system has been corrected.

[0257] FIG. 16 is a block diagram of the main functions of a digital camera relating to recording image data obtained by capturing an image.

[0258] 16 , with regard to recording image data obtained by capturing an image, the digital camera has functions such as an image data acquisition unit 214A, a lens information acquisition unit 214B, a correction setting information acquisition unit 214C, a first pre-processing unit 214D, a display image data generation unit 214E, a second pre-processing unit 214F, a distance image data generation unit 214G, a recording image data generation unit 214H, and a recording control unit 214J. The functions of each unit are realized by a processor 210 of the camera body 100. The processor 210 realizes the functions of each unit by executing a predetermined program (image data processing program).

[0259] Note that the functions other than those of the recording image data generation unit 214H and the recording control unit 214J, i.e., the functions of the image data acquisition unit 214A, lens information acquisition unit 214B, correction setting information acquisition unit 214C, first pre-processing unit 214D, display image data generation unit 214E, second pre-processing unit 214F, and distance image data generation unit 214G, are substantially the same as the functions of the image data acquisition unit 211A, lens information acquisition unit 211B, correction setting information acquisition unit 211C, first pre-processing unit 211D, display image data generation unit 211E, second pre-processing unit 211F, and distance image data generation unit 211G in the digital camera 1 of the second embodiment. Therefore, only the functions of the recording image data generation unit 214H and the recording control unit 214J will be described here.

[0260] It should be noted that in the digital camera of this embodiment, depth image data is always generated from image data (including both display and recording data) obtained by capturing an image.

[0261] [Recording Image Data Generation Unit] The recording image data generation unit 214H generates image data for recording (recording image data) from the image data that has been preprocessed by the first preprocessing unit 214D. For example, it generates image data in JPEG (Joint Photographic Experts Group) format. The generated recording image data is sent to the recording control unit 214J.

[0262] [Recording Control Unit] The recording control unit 214J records the recording image data generated by the recording image data generation unit 214H in a recording destination. In this embodiment, the recording is performed in the storage unit 130.

[0263] When recording image data, the recording control unit 214J acquires the corresponding image data for distance measurement and correction information used in preprocessing of the image data for distance measurement (such as information on parameters used in the correction process), associates them with the image data for distance measurement, and records them in the recording destination (the correction information is also associated with the image data for distance measurement). The image data for distance measurement to be recorded is image data that has been preprocessed by the second preprocessing unit 214F (preprocessed image data). In other words, it is image data in which image quality degradation caused by the optical system has been corrected.

[0264] In this embodiment, the recorded image data is an example of third image data, the preprocessed image data is an example of second image data, and the correction information is an example of information used to generate the preprocessed image data (second image data).

[0265] [Operation] FIG. 17 is a flowchart showing the procedure of processing when recording image data.

[0266] It is determined whether or not an instruction to capture an image for recording has been issued (step S501). In response to the instruction, an image for recording is captured (step S502). The image data (RAW data) obtained by capturing the image is developed (step S503) and converted into image data for recording (recorded image data) (step S504).

[0267] As described above, in the digital camera of this embodiment, distance image data is always generated from image data obtained by capturing an image, and therefore distance image data is also generated from image data captured for recording.

[0268] Here, the distance image data is generated by inputting image data (preprocessed image data) generated by performing predetermined preprocessing on image data captured for recording to the distance image data generation AI. When recording the record image data, the corresponding preprocessed image data and correction information used to generate the preprocessed image data are acquired (step S505). The acquired preprocessed image data and correction information are then associated with each other, and the record image data is recorded in the recording destination (storage unit 130) (step S506).

[0269] In this way, according to this embodiment, the preprocessed image data used to generate the distance image data and the correction information used to generate the preprocessed image data are associated with the recording image data and recorded in the recording destination, making it possible to generate distance image data even after capturing an image.

[0270] Furthermore, by recording the preprocessed image data in this way, it can be used, for example, for relearning the distance image data generation AI or for creating a new distance image data generation AI.It can also be used to analyze the cause of a large incorrect distance output by the distance image data generation AI, for example.

[0271] [Modifications] In the above embodiment, the preprocessed image data and correction information are recorded in association with the recording image data, but the distance image data may also be recorded in association with the preprocessed image data. Alternatively, only the preprocessed image data may be recorded in association with the preprocessed image data. Alternatively, instead of the preprocessed image data, the image data before preprocessing and correction information may be recorded in association with the preprocessed image data.

[0272] The form of association is not particularly limited, and any form that can identify the data that is associated with each other may be used. For example, information about the associated data may be recorded in a header or the like as additional information, and the data may be associated. Alternatively, a management file may be created and the associated data may be managed in the management file. Alternatively, the data to be associated with each other may be recorded together in one file, and the associated data may be associated.

[0273] [Other Embodiments] [Preprocessing of Image Data for Distance Measurement (1)] In the above embodiment, luminance shading correction, color shading correction, distortion correction, and aberration correction are performed as preprocessing of image data for distance measurement, but the contents of preprocessing performed on image data for distance measurement are not limited to this. Furthermore, it is not necessary to perform all of these corrections. It is preferable to correct image quality degradation caused by the optical system that affects the generation of distance image data.

[0274] Furthermore, in the above embodiment, the information necessary for correction is acquired from the interchangeable lens 10 and the image data for distance measurement is corrected (pre-processed), but the method for correcting the image data for distance measurement is not limited to this. For example, a configuration may be adopted in which correction is performed with a predetermined correction amount (correction parameter) for a predetermined correction item. For example, a configuration may be adopted in which distortion correction is performed with a predetermined correction amount.

[0275] Furthermore, in a configuration in which information necessary for correction is acquired from a lens or the like and image data for distance measurement is corrected (preprocessed) as in the above embodiment, if the information necessary for correction cannot be acquired, the implementation of preprocessing may be restricted. For example, in the above embodiment, if lens information cannot be acquired from the interchangeable lens 10, preprocessing may not be performed. In this case, distance image data is generated without preprocessing. That is, the image data for distance measurement is input to the distance image data generation AI without preprocessing, and distance image data is generated.

[0276] Furthermore, if the information necessary for correction cannot be acquired, the generation of distance image data may be restricted. For example, in the above embodiment, if lens information cannot be acquired from the interchangeable lens 10, distance image data may not be generated.

[0277] Furthermore, instead of or in addition to restricting the execution of preprocessing or the generation of distance image data when the information necessary for correction cannot be acquired, a notification or warning may be provided to the user that the information necessary for correction could not be acquired. For example, in the above-described embodiment, when lens information cannot be acquired from the interchangeable lens 10, a notification or warning is provided to the user that the lens information cannot be acquired. For example, a message indicating that the lens information cannot be acquired may be output to the rear monitor 150 or the EVF 160. Similarly, when preprocessing is canceled, it is preferable to output to the rear monitor 150 or the EVF 160 a message indicating that preprocessing cannot be performed or that distance image data will be generated without preprocessing. Similarly, when preprocessing is canceled, it is preferable to output to the rear monitor 150 or the EVF 160 a message indicating that distance image data cannot be generated.

[0278] Furthermore, when correcting multiple items, if information necessary for correction can be acquired for only some of the items, only the items for which information can be acquired may be corrected. In this case, it is preferable to notify the user by outputting a message indicating that only some of the items have been corrected, or that some of the items could not be corrected. For example, in the above embodiment, if only distortion correction data can be acquired, only distortion correction is performed as preprocessing. Then, the user is notified that only distortion correction has been performed as preprocessing.

[0279] Alternatively, preprocessing may be performed only when specific conditions are met. For example, when there are multiple correction items, required correction items are determined. Then, preprocessing may be performed only when information necessary for correction for the required correction items is obtained. In this case, correction may be performed only on the required correction items. Furthermore, required items and items for which information necessary for correction has been obtained may be corrected using the obtained information, and items for which information necessary for correction has not been obtained may be corrected under predetermined conditions (e.g., parameters, etc.). For example, in the above embodiment, distortion correction may be set as a required correction item, and preprocessing may be performed only when distortion correction data has been obtained. In this case, correction may be performed only on items other than distortion correction for which information necessary for correction has been obtained. Alternatively, items for which information necessary for correction has not been obtained may be corrected under predetermined conditions (e.g., parameters, etc.). In this example, the information necessary for correction for each correction item is an example of multiple information, and the information necessary for correction for the required correction items is an example of specific information. Furthermore, correction information used when making corrections under predetermined conditions (correction information for items for which information necessary for correction could not be obtained) is an example of second information.

[0280] Furthermore, when correcting multiple items, if the information necessary for correction is acquired for only some of the items, the implementation of preprocessing may be limited (including being stopped) or the generation of distance image data may be limited (including being stopped). In this case, it is preferable to notify the user that the implementation of preprocessing and the generation of distance image data have been limited.

[0281] [Preprocessing of Image Data for Distance Measurement (2)] In the above embodiment, image quality degradation due to lens characteristics (image quality degradation caused by the optical system) is corrected, but image quality degradation due to image sensor characteristics may also be corrected in a similar manner. In this case, the distance image data generation AI preferably performs machine learning using image data in which image quality degradation due to the image sensor characteristics has been corrected as learning image data to generate the image data.

[0282] Furthermore, the brightness of the image data for distance measurement may be measured and the image data may be corrected to a specified brightness. For example, the gain may be corrected to a specified brightness or brightness within a specified range. The brightness of the image data may be determined based on the photometry results obtained by AE.

[0283] [Acquisition of Information Necessary for Correction] In the above embodiment, lens information including lens characteristic correction information is acquired from an interchangeable lens to acquire information necessary for correcting image quality degradation caused by the optical system, but the method for acquiring the information necessary for correction is not limited to this. For example, a configuration may be adopted in which correction information for each interchangeable lens is stored on the camera body side (for example, stored in memory 220). In this case, for example, information identifying the interchangeable lens (for example, identification information such as lens model data) is acquired from the interchangeable lens, and the corresponding correction information is read and acquired from memory. Alternatively, for example, a configuration may be adopted in which correction information is stored on a network, and the correction information for the interchangeable lens attached to the camera body is acquired via the network.

[0284] [Image data for distance measurement] In the above embodiment, YCrCb data is used as image data for distance measurement, but the image data used for distance measurement (image data input to the distance image data generation AI and image data that is subject to preprocessing) is not limited to this.

[0285] For example, only the luminance data Y of the YCrCb data may be used for the image data for distance measurement. Alternatively, only the color difference data Cb or the color difference data Cr may be used for the image data for distance measurement. In this example, the YCrCb data is an example of first image data, and the luminance data Y and the color difference data Cb and Cr are an example of mixed image data (data in which the R, G, and B color components are weighted and mixed).

[0286] Furthermore, for example, when YCrCb data is used for image data for distance measurement, distance image data may be generated from the luminance data Y and the color difference data Cr and Cb. In this case, distance image data generation AI corresponding to each image data is prepared. Furthermore, when distance image data is generated separately from the luminance data Y and the color difference data Cr and Cb, a predetermined integration process may be further performed to generate a single distance image data.

[0287] RGB data (image data of color components consisting of R, G, and B) can also be used as image data for distance measurement. For RGB data, distance image data may be generated individually from each of the R, G, and B color components. In this case, distance image data generation AI corresponding to each component is prepared. Furthermore, when distance image data is generated from each of the R, G, and B color components, a predetermined integration process may be further performed to generate a single piece of distance image data. In this example, the RGB data is another example of first image data, and the R, G, and B color components are examples of components.

[0288] When generating distance image data using one of the components of RGB data, it is preferable to use the component that can express the greatest brightness. For example, with RGB data, it is preferable to generate distance image data from G image data. Also, with YCrCb data, it is preferable to generate distance image data from Y luminance data.

[0289] In addition, RAW data and image data after compression processing can also be used as image data for distance measurement. Thus, image data for distance measurement is not necessarily limited to data that can be visualized directly, but also includes data that can be visualized by performing appropriate processing. Therefore, for example, it also includes data with a structure in which signal values ​​or pixel values ​​are continuously arranged one-dimensionally or two-dimensionally.

[0290] Furthermore, the image data to be processed is not limited to image data of still images, but also includes image data of moving images.

[0291] [System Configuration] In the above embodiment, the present invention has been described as being applied to an interchangeable lens digital camera, but the application of the present invention is not limited to this. The present invention can also be applied to an integrated lens digital camera. In this case, correction information according to the focal length, aperture value, etc. is stored, and preprocessing (processing to correct image quality degradation caused by the optical system) is performed on image data for distance measurement according to the focal length, aperture value, etc.

[0292] The application of the present invention is not limited to digital cameras, but can also be applied to imaging devices such as video cameras, television cameras, and cine cameras.Furthermore, the present invention can also be applied to imaging devices incorporated into other devices.For example, the present invention can be applied to imaging devices incorporated into smartphones, personal computers, etc.

[0293] Furthermore, when the present invention is applied to an imaging device such as a digital camera, some of the functions may be implemented by an external device. For example, the digital camera may be communicably connected to a computer on a network, and some of the functions may be implemented by the computer on the network. For example, the function of generating depth image data may be implemented by the computer on the network.

[0294] Furthermore, in the above embodiment, an example has been described in which image data captured by an imaging device (digital camera) is processed within the imaging device, but the present invention can also be applied to cases in which image data captured by an imaging device is processed by an external device. The present invention can also be applied to cases in which image data captured by an imaging device such as a digital camera is imported into an external computer for processing. In this case, the external computer functions as an image data processing device. In this case, the image data captured by the imaging device may be imported into the external computer as needed for processing, or recorded image data may be imported into the external computer for processing.

[0295] [Other] The processing unit that provides the functions of the image data processing device can be composed of various processors. These include general-purpose processors such as CPUs and GPUs (Graphic Processing Units), as well as programmable logic devices (PLDs) such as FPGAs (Field Programmable Gate Arrays), whose circuit configuration can be changed after manufacture, and dedicated electrical circuits such as ASICs (Application Specific Integrated Circuits), which are processors with circuit configurations specifically designed to perform specific processes. A single processing unit may be composed of one of the various processors, or two or more processors of the same or different types. For example, a single processing unit may be composed of multiple FPGAs or a combination of a CPU and an FPGA. Multiple processing units may also be composed of a single processor. A first example of multiple processing units composed of a single processor is a configuration in which a single processor is composed of a combination of one or more CPUs and software, and this processor functions as multiple processing units, as typified by computers used as clients, servers, etc. Secondly, there is a form using a processor that realizes the functions of an entire system including multiple processing units on a single IC (Integrated Circuit) chip, as typified by a system on chip (SoC), etc. In this way, various processing units are configured as a hardware structure using one or more of the above-mentioned various processors.

[0296] DESCRIPTION OF SYMBOLS 1...Digital camera 10...Interchangeable lens 12...Lens side mount 20...Optical system 30...Optical system driving section 40...Lens operation section 50...Lens microcomputer 51...Processor 52...Memory 100...Camera body 102...Camera side mount 110...Image sensor 120...Shutter 122...Shutter driving section 130...Storage section 140...Interface section 150...Rear monitor 170...Camera operation section 200...Camera microcomputer 210...Processor 210A...Image data acquisition section 210B...Lens information acquisition section 210C...Preprocessing section 210D...Distance image data generation section 210E...Development processing section 210E1...White balance correction section 210E2...Gamma correction section 210E3...Demosaic section 210E4...YC conversion section 210E5...Color correction section 211A...Image data acquisition section 211B...lens information acquisition unit 211C...correction setting information acquisition unit 211D...first pre-processing unit 211E...display image data generation unit 211F...second pre-processing unit 211G...distance image data generation unit 212A...image data acquisition unit 212B...lens information acquisition unit 212C...correction setting information acquisition unit 212D...first pre-processing unit 212E...display image data generation unit 212F...second pre-processing unit 212G...distance image data generation unit 212H...post-processing unit 213A...image data acquisition unit 213B...lens information acquisition unit 213C...correction setting information acquisition unit 213D...first pre-processing unit 213E...display image data generation unit 213F...second pre-processing unit 213G...distance image data generation unit 213H...post-processing unit 214A...image data acquisition unit 214B...lens information acquisition unit 214C...Correction setting information acquisition unit 214D...First pre-processing unit 214E...Display image data generation unit 214F...Second pre-processing unit 214G...Distance image data generation unit 214H...Recording image data generation unit 214J...Recording control unit 220...Memory Di...Distance image data Di1...Distance image data Di2...Distance image data Im0...Image data Im1...Image data Im1a...Image data Im1b...Image data Im2...Image data Lv...Display image data S101 to S103...Distance measurement procedure S201 to S210...Live view output and distance measurement processing procedure S301 to S310...Live view output processing procedure S401 to S410...Live view output processing procedureS501 to S506: Processing procedure for recording image data

Claims

1. An image data processing device comprising at least one processor, which acquires first image data captured through an optical system, corrects image quality degradation caused by the optical system, generates second image data from the first image data, and generates first distance image data from the second image data.

2. The image data processing device according to claim 1, wherein the processor generates the second image data from the first image data by correcting image quality degradation caused by the optical system that affects the generation of the first distance image data.

3. The image data processing device according to claim 1, wherein the processor generates third image data from the first image data, and outputs the third image data to a display destination or a recording destination.

4. The image data processing device according to claim 3, wherein the processor generates the third image data from the first image data by performing a correction different from the correction performed on the second image data.

5. An image data processing device as described in claim 1, wherein the processor generates fourth image data from the first image data, outputs the fourth image data to a display destination or a recording destination, performs processing to reduce a difference between the first distance image data and the fourth image data, generates second distance image data from the first distance image data, and outputs the second distance image data to a display destination or a recording destination.

6. An image data processing device as described in claim 1, wherein the processor outputs the first distance image data to a display destination or a recording destination, generates fifth image data from the first image data, performs processing to reduce a difference between the first distance image data and the fifth image data, generates sixth image data from the fifth image data, and outputs the sixth image data to a display destination or a recording destination.

7. The image data processing device according to claim 3, wherein the processor outputs the second image data to a recording destination in association with the third image data.

8. The image data processing device according to claim 7, wherein the processor outputs information used in generating the second image data to a recording destination in association with the second image data.

9. The image data processing device according to claim 1, wherein the processor acquires first information including information necessary for correcting image quality degradation caused by the optical system, and generates the second image data based on the first information.

10. The image data processing device according to claim 9, wherein the processor issues a notification when the first information cannot be acquired.

11. The image data processing device according to claim 9, wherein the processor generates the second image data when specific information is acquired from among a plurality of pieces of information included in the first information.

12. The image data processing device according to claim 9, wherein the processor limits generation of the second image data when the first information cannot be acquired.

13. The image data processing device according to claim 9, wherein the processor generates the second image data based on predetermined second information when the first information cannot be acquired, and notifies the user that the second image data has been generated based on the second information.

14. The image data processing device according to claim 1, wherein the processor generates the second image data using one of the components of the first image data or mixed image data obtained by weighting the components.

15. The image data processing device according to claim 1, wherein the processor generates the second image data using a component of the first image data that can express the greatest brightness.

16. The image data processing device according to claim 1, wherein the processor generates the second image data for each component of the first image data, and generates the first distance image data for each component from the second image data for each component.

17. The image data processing device according to claim 1, wherein the processor generates the first range image data from the second image data using a trained model generated by machine learning.

18. An image data processing method comprising: acquiring first image data captured through an optical system; correcting image quality degradation caused by the optical system; generating second image data from the first image data; and generating first distance image data from the second image data.

19. An image data processing program that causes a computer to realize the following functions: acquiring first image data captured through an optical system; correcting image quality degradation caused by the optical system and generating second image data from the first image data; and generating first distance image data from the second image data.

20. A non-transitory computer-readable recording medium having the program according to claim 19 recorded thereon.

21. An imaging device comprising: an imaging section which captures image data via an optical system; and an image data processing device according to any one of claims 1 to 17 which processes the image data captured by the imaging section.

22. The imaging device described in claim 21, comprising: a main body having the imaging unit and the image data processing device; and an interchangeable lens having the optical system and communicatively and detachably attached to the main body, wherein the processor acquires lens information from the interchangeable lens attached to the main body, and generates the second image data based on the lens information.

23. A learning device comprising at least one processor, which acquires a dataset including training image data captured via an optical system and in which image quality degradation due to the optical system has been corrected, and ground truth data corresponding to the training image data, and performs machine learning using the dataset to generate a model that outputs distance image data when image data is input.

Images