Phase difference map generation device and generation method, image data acquisition device, focus control method, training method, and phase difference map generator

Abstract

One embodiment pertaining to the technology of the present disclosure provides a phase difference map generation device, a phase difference map generation method, an image data acquisition device, a focus control method, a training method, and a phase difference map generator. This phase difference map generation device pertaining to one embodiment of the present invention includes a processor. The processor: acquires at least first phase-difference image data and second phase-difference image data from an image data acquisition unit including a single optical system and an imaging element having phase difference pixels; performs preprocessing on the first phase-difference image data and the second phase-difference image data to acquire first corrected-image data and second corrected-image data; and generates, from the first corrected-image data and the second corrected-image data, a phase difference map in which the amount of phase difference between the first phase difference image data and the second phase difference image data and the direction of a phase shift are mapped. The preprocessing is processing based on the characteristics of the processing for generating the phase difference map.

Description

Phase difference map generating device and generating method, image data acquiring device, focus control method, learning method, and phase difference map generator

[0001] The present invention relates to a phase difference map generating device and method, an image data acquisition device, a focus control method, a learning method, and a phase difference map generator, and in particular to a technique for handling phase difference image data.

[0002] Regarding techniques for handling image data, for example, Patent Document 1 describes a technique for estimating distance information from defocus blur in a captured image.

[0003] Japanese Patent Application Laid-Open No. 2022-019374

[0004] One embodiment of the disclosed technique provides a phase difference map generation device and method, an image data acquisition device, a focus control method, a learning method, and a phase difference map generator.

[0005] A phase difference map generation device according to a first aspect of the present invention is a phase difference map generation device including a processor, wherein the processor acquires at least first phase difference image data and second phase difference image data from an image data acquisition unit including a single optical system and an imaging element having phase difference pixels, performs preprocessing on the first phase difference image data and the second phase difference image data to acquire first corrected image data and second corrected image data, and generates a phase difference map from the first corrected image data and the second corrected image data, mapping the amount of phase difference and the direction of phase shift between the first phase difference image data and the second phase difference image data, wherein the preprocessing is processing based on the characteristics of the processing for generating the phase difference map.

[0006] In the first aspect, the "phase difference image data" and the "corrected image data" are data distributed two-dimensionally and can be handled in the same way as normal images. Note that these image data are not intended to be displayed or viewed as images, but may be processed as necessary and displayed in the same way as normal images for viewing by a user. Furthermore, the "single optical system" is, for example, a monocular optical system, and the "image data acquisition unit" can be configured in the same way as a normal optical system.

[0007] In the first aspect and each of the following aspects, three or more types of phase difference image data may be acquired, and a phase difference map may be generated from the three or more types of phase difference image data.

[0008] The phase difference map generating device according to the first aspect may be realized as a device that acquires phase difference image data or the like from an external device and generates a phase difference map, or may be realized as a processor part of an image data acquisition device or an imaging device that has an image data acquisition unit.

[0009] A phase difference map generating device according to a second aspect of the present invention is the first aspect, wherein the preprocessing is processing for reducing a difference in image quality between the first phase difference image data and the second phase difference image data.

[0010] In the phase difference map generating device according to the third aspect, in the first or second aspect, the processor acquires, as first phase difference image data, image data generated from signals output by phase difference pixels having one side of the light receiving unit shielded from light, and acquires, as second phase difference image data, image data generated from signals output by phase difference pixels having the other side of the light receiving unit shielded from light.

[0011] A phase difference map generating device according to a fourth aspect is any one of the first to third aspects, wherein the processor performs different processes on the first phase difference image data and the second phase difference image data in the preprocessing.

[0012] A phase difference map generating device according to a fifth aspect is the fourth aspect, wherein the processor performs pre-processing on the first phase difference image data and the second phase difference image data, the pre-processing differing in at least one of content and degree.

[0013] A phase difference map generating device according to a sixth aspect is the third aspect, wherein the processor performs processing as preprocessing to eliminate non-uniformity within the angle of view of the image data caused by the shading technique.

[0014] A phase difference map generation device according to a seventh aspect is any one of the first to sixth aspects, wherein the processor acquires data acquisition conditions for the first phase difference image data and the second phase difference image data, and performs preprocessing according to the data acquisition conditions. In the seventh aspect and each of the following aspects, the "data acquisition conditions" are conditions equivalent to imaging conditions when capturing a normal image.

[0015] In the phase difference map generating device of the eighth aspect, in the seventh aspect, the processor, in preprocessing, enlarges the size of the image indicated by the first phase difference image data and the size of the image indicated by the second phase difference image data according to the data acquisition conditions, and generates first corrected image data and second corrected image data from the enlarged first phase difference image data and second phase difference image data.

[0016] A phase difference map generating device according to a ninth aspect is any one of the first to eighth aspects, wherein the processor performs preprocessing on at least one of resolution, noise, gradation, and image structure. The ninth aspect specifically defines the content of the preprocessing.

[0017] A phase difference map generating device according to a tenth aspect is any one of the first to ninth aspects, in which a processor extracts corresponding points between the first corrected image data and the second corrected image data, which correspond to the same position of the same object in real space, and generates a phase difference map by mapping the amount of phase difference and the direction of phase shift for the corresponding points.

[0018] In the phase difference map generating device of the eleventh aspect, in the tenth aspect, a processor extracts first feature points that are feature points of the first corrected image data and second feature points that are feature points of the second corrected image data, extracts corresponding points from the first feature points and the second feature points, and generates a phase difference map by mapping the amount of phase difference and the direction of phase shift for the corresponding points.

[0019] A phase difference map generating device according to a twelfth aspect is any one of the first to eleventh aspects, wherein the processor generates the phase difference map using a phase difference map generator constructed by machine learning.

[0020] A phase difference map generating device according to a thirteenth aspect is the twelfth aspect, wherein the phase difference map generator is a trained model constructed by providing a neural network with first phase difference image data and second phase difference image data, and distance information corresponding to the first phase difference image data and the second phase difference image data as training data and causing the neural network to train. In the thirteenth aspect, the distance information is correct data during training, and may be the distance itself or other information corresponding to the distance.

[0021] A phase difference map generating device according to a fourteenth aspect is the thirteenth aspect, in which the phase difference map generator is constructed by learning using first phase difference image data and second phase difference image data acquired under at least one identical data acquisition condition.

[0022] Increasing the number of data acquisition conditions assumed in learning makes it possible to generate a phase difference map with high accuracy, but on the other hand, the learning cost (preparation of learning data and correct answer data, learning time, etc.) increases. From this perspective, in the fourteenth aspect, the learning cost is reduced by using, for learning, first phase difference image data and second phase difference image data acquired under at least one identical data acquisition condition. The first phase difference image data and second phase difference image data acquired under all identical data acquisition conditions may also be used for learning. Note that it is preferable to determine "how identical the data acquisition conditions should be (how uniform the data acquisition conditions should be)" in consideration of the accuracy of generating the phase difference map and the learning cost.

[0023] The phase difference map generating device of the 15th aspect is the 14th aspect, wherein the data acquisition conditions for the first phase difference image data and the data acquisition conditions for the second phase difference image data are the same in at least one of the focal length of the optical system, the aperture value of the optical system, the shutter speed, and the focal length.

[0024] A phase difference map generating device according to a sixteenth aspect is any one of the first to fifteenth aspects, in which the processor uses the phase difference map to determine a focus position according to the data acquisition conditions of the first phase difference image data and the second phase difference image data.

[0025] A phase difference map generating device according to a seventeenth aspect is the sixteenth aspect, wherein the processor determines the focus position based on a distribution of phase difference amounts in a focus region set in the phase difference map.

[0026] In the phase difference map generating device according to an eighteenth aspect, in any one of the first to seventeenth aspects, the processor converts the phase difference amount of the phase difference map into distance information in the optical axis direction, thereby generating distance image data composed of distance information. In the eighteenth aspect and the following aspects, the "distance image data" is data in which distance information is distributed two-dimensionally, and can be handled in the same manner as a normal image, similar to the above-mentioned "phase difference image data" and "corrected image data." Note that the distance image data is not intended to be displayed or viewed as an image, but may be displayed in the same manner as a normal image, or after being processed as necessary, and viewed by a user.

[0027] In the phase difference map generating device of the 19th aspect, in the 18th aspect, the processor generates, as distance image data, at least one of a defocus map in which the phase difference amount is converted into a defocus amount as distance information, and a distance map in which the defocus amount is converted into a subject distance as distance information.

[0028] In the phase difference map generating device according to the twentieth aspect, in the 19th aspect, the processor performs post-processing on the phase difference map in accordance with the data acquisition conditions of the first phase difference image data and the second phase difference image data to generate distance image data.

[0029] In the phase difference map generating device of the 21st aspect, in the 20th aspect, the processor performs post-processing while taking into consideration at least one of ray angle information of the first phase difference image data and the second phase difference image data, position information of a focus lens of the optical system used to acquire the first phase difference image data and the second phase difference image data, and optical characteristics of the optical system as data acquisition conditions.

[0030] An image data acquisition device according to the 22nd aspect includes a phase difference map generating device according to the 16th or 17th aspect, an image data acquisition unit, and a drive unit that drives a single optical system, and a processor performs focusing control to drive the single optical system to a focus position using the drive unit.

[0031] The image data acquisition device according to the 23rd aspect is the 22nd aspect, wherein the imaging element includes color pixels in which one of a plurality of optical filters that transmit light of at least a portion of different wavelength bands is arranged.

[0032] A phase difference map generation method according to a 24th aspect is a phase difference map generation method executed by a phase difference map generation device having a processor, in which the processor acquires at least first phase difference image data and second phase difference image data from an image data acquisition unit having a single optical system and an imaging element having phase difference pixels, performs preprocessing on the first phase difference image data and the second phase difference image data to acquire first corrected image data and second corrected image data, and generates a phase difference map from the first corrected image data and the second corrected image data, mapping the amount of phase difference and the direction of phase shift between the first phase difference image data and the second phase difference image data, and the preprocessing is processing based on the characteristics of the processing for generating the phase difference map.

[0033] The phase difference map generating method according to the 24th aspect may have a configuration corresponding to the phase difference map generating devices according to the second to 21st aspects. Also, a phase difference map generating program that causes a computer to execute the phase difference map generating methods of these aspects, and a non-transitory tangible recording medium on which computer-readable code for the phase difference map generating program is recorded can also be cited as aspects of the present invention.

[0034] A focusing control method according to a 25th aspect is a focusing control method executed by an image data acquisition device including a processor, an image data acquisition unit that includes an image sensor having a single optical system and phase difference pixels, and that acquires first phase difference image data and second phase difference image data of a subject, and a drive unit that drives the image data acquisition unit, wherein the processor acquires the first phase difference image data and the second phase difference image data of the subject using the image data acquisition unit, performs preprocessing on the first phase difference image data and the second phase difference image data to acquire first corrected image data and second corrected image data, generates a phase difference map from the first corrected image data and the second corrected image data that maps the amount of phase difference and the direction of phase shift between the first phase difference image data and the second phase difference image data, determines a focusing position according to the data acquisition conditions of the first phase difference image data and the second phase difference image data, and performs focusing control by driving the single optical system to the focusing position using the drive unit, and the preprocessing is processing based on the characteristics of the processing that generates the phase difference map.

[0035] In addition, a focusing control program that causes a computer to execute the focusing control method of the 25th aspect, and a non-transitory, tangible recording medium on which computer-readable code of such a focusing control program is recorded can also be cited as aspects of the present invention.

[0036] A learning method according to a 26th aspect provides a neural network with first phase-difference image data and second phase-difference image data of a subject, which are acquired by an image data acquisition unit including a single optical system and an image sensor having phase-difference pixels, and distance information corresponding to the first phase-difference image data and the second phase-difference image data, as learning data, thereby constructing a phase-difference map generator that outputs a phase-difference map that maps the amount of phase difference and the direction of phase shift between the first phase-difference image data and the second phase-difference image data when the first phase-difference image data and the second phase-difference image data are input. The phase-difference map generator constructed by the learning method according to the 26th aspect is a trained model.

[0037] A learning method according to a 27th aspect is the same as that of the 26th aspect, except that information based on a result of actually measuring a distance from an image data acquisition device that acquires the first phase-contrast image data and the second phase-contrast image data to the subject is provided as the distance information. The information provided in the 27th aspect may be the distance itself or other information corresponding to the distance.

[0038] A phase difference map generator according to a 28th aspect is a phase difference map generator constructed by the learning method according to the 26th or 27th aspect. The phase difference map generator according to the 28th aspect is a trained model.

[0039] FIG. 1 is a diagram illustrating the configuration of an imaging device according to a first embodiment. FIG. 2 is a diagram illustrating the configuration of an image processing unit. FIG. 3 is a diagram illustrating an example of a pixel arrangement in an imaging element. FIG. 4 is a diagram illustrating an example of the configuration of each pixel. FIG. 5 is a diagram illustrating an example of a two-dimensional array of phase difference pixels. FIG. 6 is a diagram illustrating an example in which all pixels of an imaging element are phase difference pixels. FIG. 7 is a diagram illustrating the relationship between phase difference amount and focus deviation. FIG. 8 is a diagram illustrating focus control based on a phase difference image. FIG. 9 is a diagram illustrating the effect of focus deviation on a color image and phase difference pixels. FIG. 10 is a diagram conceptually illustrating how a phase difference map is generated. FIG. 11 is a diagram illustrating an example of a phase difference image before preprocessing. FIG. 12 is a diagram illustrating an example of a phase difference image after preprocessing. FIG. 13 is a diagram illustrating an example of the layer configuration of a convolutional neural network (CNN). FIG. 14 is a diagram illustrating convolution using a filter. FIG. 15 is a diagram illustrating how a phase difference is calculated from a phase difference image after preprocessing. FIG. 16 is a diagram illustrating an example of a phase difference map for a single subject (point light source). Fig. 17 is a diagram showing the configuration of an image processing unit in an imaging device according to the second embodiment. Fig. 18 is a diagram showing how depth image data is generated from a phase difference map in the second embodiment. Fig. 19 is a diagram showing how post-processing is performed in the second embodiment. Fig. 20 is a diagram showing an example of depth image data.

[0040] [Distance Estimation Using Monocular Phase-Difference Images and Generation of Phase-Difference Maps] In recent years, distance estimation technologies have evolved in various fields. Conventional distance estimation using monocular optical systems (single optical systems) is a simple distance measurement method because it can acquire a distance image from a single image. However, because it does not involve physical distance measurement, it can estimate depth even with a flat image. Furthermore, distance estimation using compound-eye optical systems generally uses two cameras to capture images and enables highly accurate distance estimation based on the parallax between the images. However, it requires accurate calibration of the relationship between the positions and imaging directions of the two cameras, making it a less convenient distance measurement method. Furthermore, distance estimation using multiple images captured by moving a monocular optical system requires accurate calibration of the relationship between the camera positions and imaging directions, just as with compound-eye optical systems. Furthermore, because the required images cannot be captured simultaneously, distance estimation can be difficult in situations such as when the subject is moving.

[0041] In consideration of these circumstances, the inventors of the present application have conducted extensive research and have found that "using a phase difference image acquired with a monocular optical system enables distance estimation based on parallax (phase difference) as with a compound eye optical system by dividing the pupil, even though the optical system is monocular, thereby enabling simple and highly accurate distance measurement" and "a phase difference map can be generated from the phase difference image during the distance estimation process and can be used for controlling the imaging device, etc." The present invention was created based on these findings, and specific aspects of the phase difference map generation device and phase difference map generation method, image data acquisition device, focus control method, learning method, and phase difference map generator according to the present invention will be described below.

[0042] [First Embodiment] [Overall Configuration of Imaging Device] Fig. 1 is a diagram showing the configuration of an imaging device 10 (imaging device, image data acquisition device) according to the first embodiment. The imaging device 10 is composed of an interchangeable lens 100 (single optical system, monocular optical system, image data acquisition unit) and an imaging device body 200 (image data acquisition unit), and forms a subject image (optical image) on an imaging element 202 using a photographing lens including a zoom lens 102 (described later). The interchangeable lens 100 and the imaging device body 200 can be attached and detached via a mount (not shown).

[0043] [Configuration of Interchangeable Lens] The interchangeable lens 100 includes a zoom lens 102, a focus lens 104, an aperture 106, and a lens driver 110. The lens driver 110 drives the zoom lens 102 and the focus lens 104 forward and backward in response to commands from the image processor 210 (optical system driver 230 in FIG. 2 ) to perform zoom (optical zoom) adjustment and focus adjustment. The zoom adjustment and focus adjustment may be performed in response to commands from the image processor 210 or in response to zoom and focus operations (such as rotation of a zoom ring or focus ring, not shown) performed by the user. The lens driver 110 also controls the aperture 106 in response to commands from the image processor 210 to adjust exposure. Information such as the positions of the zoom lens 102 and the focus lens 104 and the aperture of the aperture 106 is input to the image processor 210. The interchangeable lens 100 has an optical axis L.

[0044] [Configuration of Imaging Device Main Body] The imaging device main body 200 includes an imaging element 202 (imaging element), an AFE (Analog Front End: AFE) 204, an A / D converter 206 (Analog to Digital: A / D, imaging unit), an image processing unit 210, an operation unit 260, a recording unit 270, and a monitor 280. The imaging device main body 200 may include a shutter (not shown) for blocking light incident on the imaging element 202. If a shutter is included, it is preferable that the shutter speed be variable.

[0045] The image sensor 202 has a light-receiving surface on which a large number of light-receiving elements are arranged in a two-dimensional matrix. The light-receiving surface of the image sensor 202 is provided with color pixels and phase-contrast pixels, allowing it to acquire color images and phase-contrast images (phase-contrast image data) of a subject. Subject light that has passed through the zoom lens 102, focus lens 104, and aperture 106 is focused on the light-receiving surface of the image sensor 202 and converted into electrical signals by each light-receiving element. The detailed configuration of the image sensor 202 and the acquisition of image data will be described later. Note that various photoelectric conversion elements, such as a CMOS (Complementary Metal-Oxide Semiconductor) or a CCD (Charge-Coupled Device), can be used as the image sensor 202.

[0046] The AFE 204 performs noise removal, amplification, etc. on the analog image signal output from the image sensor 202, and the A / D converter 206 converts the captured analog image signal into a digital image signal with a wide gradation range.

[0047] 2 is a diagram showing the configuration of the image processing unit 210. The image processing unit 210 includes a processor 220, a ROM 240 (Read Only Memory), and a RAM 250 (Random Access Memory). The processor 220 includes an image acquisition unit 222, a preprocessing unit 224, a learning control unit 226, a phase difference map generator 228, an optical system driving unit 230, an output control unit 234, and an external input / output unit 236. Details of the processing performed by these functions will be described later.

[0048] The processor 220 is configured with various processors and electrical circuits, such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), a PLD (Programmable Logic Device), etc. When these processors and electrical circuits execute software (programs), computer-readable code of the software to be executed (for example, various processors and electrical circuits that constitute the processor, and / or a combination thereof) is stored in a non-transitory and tangible recording medium such as the ROM 240, and the computer references the software.

[0049] The software stored in the non-transitory, tangible recording medium includes various programs according to the present invention (programs that cause a computer to execute the phase difference map generation method, focus control method, and learning method according to the present invention) and data used in executing the programs. Instead of the ROM 240, the code may be recorded in a non-transitory, tangible recording medium such as a flash ROM or an EEPROM (Electronically Erasable and Programmable Read Only Memory). Note that this "non-transitory, tangible recording medium" does not include non-tangible recording media such as carrier signals or propagation signals themselves. During processing using the software, the RAM 250 is used as a temporary storage area or working area.

[0050] [Configuration of Operation Unit and Monitor] The operation unit 260 has a release button, operation buttons, dials, switches, etc. (not shown), and allows a user to perform various operations such as acquiring color images and phase difference images (phase difference image data), training the phase difference map generator, generating phase difference maps and distance images, and outputting these results. The monitor 280 may be configured as a touch panel device, and this device may be used as the operation unit 260. The operation unit 260 may also include a microphone and speaker (not shown).

[0051] The monitor 280 (display device) is configured with a touch panel type liquid crystal display panel and can display normal moving images, still images, phase difference image data, corrected image data, phase difference maps, distance image data, etc. The monitor 280 can be placed on the back side, top side, etc. of the imaging device main body 200.

[0052] [Configuration of Recording Unit] The recording unit 270 (recording unit) is composed of non-transitory, tangible recording media such as various types of magneto-optical recording media and semiconductor memories, and their control circuits, and records phase difference image data, corrected image data, phase difference maps, distance image data, etc. Data acquired from the external device 300 may be recorded in the recording unit 270. The data recorded in the recording unit 270 can be displayed on the monitor 280 or output to the external device 300 in response to a user instruction via the operation unit 260, or automatically without a user instruction. The recording medium used in the recording unit 270 may be a type that is detachable from the imaging device main body 200, such as various types of memory cards.

[0053] [External Device] An external device 300 can be connected to the imaging device 10 to input and output information. The connection may be via wired or short-range wireless communication, or via a network. Various display devices and recording devices can be used as the external device 300, and an imaging device other than the imaging device 10 may also be used as the external device 300. Furthermore, a vehicle, a moving object, or the like may be used as the external device 300, and data output from the imaging device 10 (normal moving images, still images, phase difference maps, and range image data) may be used to control the equipment of the vehicle, the moving object, or the like (for example, focus control, maintaining a safe distance between vehicles, route control, collision prevention, and avoiding dangerous objects).

[0054] [Configuration of Image Sensor] [Pixel Arrangement Example (Part 1)] FIG. 3 is a diagram showing an example of a pixel arrangement in the image sensor 202 (as viewed from the object side of the light receiving surface of the image sensor 202), and FIG. 4 is a diagram showing an example of the configuration of each pixel. As shown in FIG. 3, the image sensor 202 includes phase-difference pixels and color pixels. In the example of FIG. 3, the image sensor 202 includes color pixels 202R, 202G, and 202B (color pixels), each of which is provided with a color filter (optical filter) that transmits red, green, and blue light. These color filters constitute "multiple optical filters that transmit light of at least some different wavelength bands." Note that while the pixel arrangement in FIG. 3 is a Bayer array, other arrangements, such as a diagonal Bayer array (double Bayer array), X-Trans (registered trademark), or quad Bayer array, may also be used. The image processing unit 210 (processor 220) can generate a color image (RGB image) using signals output from these color pixels.

[0055] In the following description, the color pixels 202R, 202G, and 202B may be referred to as "R pixel, G pixel, and B pixel," respectively. Microlenses are provided in the color pixels and the phase difference pixels (not shown in FIG. 3, see FIG. 4).

[0056] [Arrangement of Phase Difference Pixels (Intermittent Arrangement)] As shown in part (a) of FIG. 3 , the image sensor 202 includes phase difference pixels 201 and 203 (phase difference pixels). The phase difference pixel 201 has an opening 201A on the right side (the left side as you face the figure) of the pixel, which functions as a light receiving unit, and a mask 201B on the left side (the right side as you face the figure; one side of the light receiving unit) of the pixel. On the other hand, the phase difference pixel 203 has an opening 203A on the left side (the right side as you face the figure) of the pixel, which functions as a light receiving unit, and a mask 203B on the right side (the left side as you face the figure; the other side of the light receiving unit) of the pixel. The image acquisition unit 222 (processor) can acquire image data generated from signals output by the phase difference pixel 201 as first phase difference image data, and can acquire image data generated from signals output by the phase difference pixel 203 as second phase difference image data.

[0057] 3A shows an example in which one pixel is divided and shielded in the horizontal direction, but as shown in FIG. 3B, it may also be divided and shielded in the vertical direction. In the example shown in FIG. 3B, the phase difference pixel 201 has an opening 201D on the upper side (upper side as viewed in the figure) of the pixel, which functions as a light receiving unit, and the lower side (lower side as viewed in the figure; one side of the light receiving unit) of the pixel is shielded from light by a mask 201C. On the other hand, the phase difference pixel 203 has an opening 203D on the lower side (lower side as viewed in the figure) of the pixel, which functions as a light receiving unit, and the upper side (upper side as viewed in the figure; the other side of the light receiving unit) of the pixel is shielded from light by a mask 203C. In this case, the image acquisition unit 222 (processor) can acquire image data generated from the signal output by the phase difference pixel 201 as first phase difference image data, and can acquire image data generated from the signal output by the phase difference pixel 203 as second phase difference image data.

[0058] 3, the phase difference pixels are arranged at the positions of the G pixels, but the phase difference pixels may be arranged at the positions of the R pixels or the B pixels. Furthermore, although color filters are not arranged in the phase difference pixels in the example of FIG. 3, color filters may be arranged in the phase difference pixels.

[0059] 4 is a diagram showing the configurations of color pixels and phase difference pixels. As shown in part (a) of FIG. 4, the color pixels include a microlens ML and a photodiode PD (color filters are not shown). On the other hand, as shown in part (b) of FIG. 4, the phase difference pixels include a microlens ML, a photodiode PD, and a mask 202A. As described above with reference to FIG. 3, the position of the mask 202A can be left / right or up / down. The position, shape, and size of the mask 202A can be set according to the position, shape, and size of the pupil to be directed.

[0060] 5 is a diagram showing an example of a two-dimensional array of phase difference pixels. In the example shown in the figure, phase difference pixels 202X divided into left and right as shown in part (a) of FIG. 3 and phase difference pixels 202Y divided into top and bottom as shown in part (b) of FIG. 3 are arranged in orthogonal directions, and the phase difference pixels as a whole are arranged two-dimensionally. In FIG. 5, the phase difference pixels 202X and the phase difference pixels 202Y are shown in one row, but in order to generate a highly accurate phase difference map, it is preferable to arrange the phase difference pixels in the vertical and horizontal directions over the entire surface of the image sensor 202. When it is not necessary to acquire a color image, all pixels of the image sensor 202 may be phase difference pixels.

[0061] It should be noted that, for pixel positions where color pixels cannot be placed due to the placement of phase difference pixels, the signal at that position can be obtained by interpolation calculation using signals from surrounding pixels.

[0062] [Phase Difference Pixels (Full-Screen Arrangement)] FIG. 6 is a diagram showing an example in which all pixels of the image sensor 202 are phase difference pixels. In the example of FIG. 6, the light receiving section (photodiode PD in FIG. 4) of the phase difference pixel 207 is divided in the left-right direction to form phase difference pixels 207-1 and 207-2. By individually extracting the signals of these phase difference pixels 207-1 and 207-2, a phase difference image (phase difference image data) can be generated, similar to the case of shielding light with a mask. Furthermore, when it is not necessary to acquire a phase difference image, a normal image can be generated by adding the signals of the phase difference pixels 207-1 and 207-2. Furthermore, since color filters are arranged on the phase difference pixels 207-1 and 207-2, a color image can be generated whether the pixels are divided and used as phase difference pixels or used as a single pixel. Note that FIG. 6 shows an example in which the phase difference pixels are divided in the left-right direction, but the phase difference pixels may also be divided in the up-down direction, as described above with reference to FIG. 3. Then, by two-dimensionally arranging the pixels divided in the horizontal direction and the pixels divided in the vertical direction, a highly accurate phase difference map can be generated.

[0063] [Relationship Between Phase Difference Amount and Focus Error] FIG. 7 is a conceptual diagram showing the relationship between phase difference amount and focus error. FIG. 7 is a diagram showing the optical system of the interchangeable lens 100 as viewed from a direction perpendicular to the optical axis L, with lens 101 representing a virtual lens included in the optical system. In this state, assuming that light from a subject (point light source) located at point P0 is imaged at point S0 (the position where the imaging surface of the image sensor is located), light from a subject located at point P1 on the -Z side of point S0 is imaged at point S1 on the -Z side of point S0. In this case, the light beam from point P1 (shown by a solid line in the figure) spreads (shifts) in the ±X directions on the imaging surface. The amount of this shift in the ±X directions corresponds to the amount of phase difference, and the direction of the shift corresponds to the direction of the phase shift. In the present invention, as will be described in detail below, the amount of phase difference and the direction of the phase shift are mapped as a phase difference map. Note that light from a subject present at point P2 on the +Z side of point P0 forms an image on the +Z side of point S0, and the direction of the phase shift is opposite to that of the subject present at point P1 (see the description of FIG. 8).

[0064] The focus deviation in the ±Z direction (the amount and direction of deviation) corresponds to distance information, and can be mapped as distance image data, as will be described later in the second embodiment.

[0065] [Focus Control Based on Phase-Contrast Images] Figure 8 illustrates focus control (focusing) based on phase-contrast images (the light-shielding mask is not shown). Parts (a) to (c) of Figure 8 respectively show so-called "back focus" (a state in which the focal point is located behind the light-receiving surface), "just focus" (a state in which the focal point is focused at the target position and located on the light-receiving surface), and "front focus" (a state in which the focal point is located in front of the light-receiving surface). The "just" in "just focus" stands for "just," and the "pin" stands for "pint." In English, "just-focused" can be expressed as "just-focused" or "in perfect focus." As shown in Figure 8, in the cases of "back focus" and "front focus," the direction of the shift (the direction of the phase shift) between the light beam A passing through the +X side of the lens and the light beam B passing through the -X side is opposite. The phase difference amounts for "back focus" and "front focus" are "shift d" and "shift d'," respectively.

[0066] In the imaging device 10 according to the first embodiment, the driving direction of the focus lens 104 is determined based on the direction of the phase shift described above with reference to Figures 7 and 8, and the focus lens 104 is driven so that the amount of phase shift becomes zero, thereby enabling focusing on the target subject.

[0067] [Effect of Out-of-Focus in Color Images and Phase-Contrast Images] Figure 9 illustrates the effect of out-of-focus in color images and phase-contrast images. The subject is a point light source located at the center of the angle of view. Parts (a) to (c) of Figure 9 respectively show a color image, phase-contrast image 1 (e.g., a left-viewpoint image), and phase-contrast image 2 (e.g., a right-viewpoint image). In Figure 9, the images toward the right are in "back focus," as described above, the images toward the left are in "front focus," and the image in the center is in "focus-on-focus." As the focus state deviates from "focus-on-focus," the blur of the subject image increases. Furthermore, as described above with reference to Figures 7 and 8, in the phase-contrast images, the direction of the shift in phase-contrast images 1 and 2 is opposite, and the direction of the shift is also opposite between "front focus" and "back focus." In "focus-on-focus," the blur of the subject image in the phase-contrast image is zero, and the two subject images overlap.

[0068] 10 is a diagram conceptually illustrating how a phase difference map is generated. The image acquisition unit 222 (processor) acquires phase difference images 1 and 2 (first phase difference image data and second phase difference image data) via the interchangeable lens 100, the image sensor 202, etc., and the preprocessing unit 224 (processor) performs preprocessing on the first phase difference image data and the second phase difference image data to acquire first corrected image data and second corrected image data. The phase difference map generator 228 (processor) then generates a phase difference map that maps the amount of phase difference and the direction of phase shift between the first phase difference image data and the second phase difference image data from the first corrected image data and the second corrected image data obtained by the preprocessing.

[0069] [Preprocessing] As described above, the present invention enables the generation of a phase difference map and simple, highly accurate distance measurement using a monocular phase difference optical system. However, phase difference images are affected by the characteristics of the optical system and imaging sensor (image capture element) used for capture, making it difficult to estimate distance directly from disparity images. Distance estimation is possible by training the system to take into account the characteristics of the optical system and imaging sensor. However, in situations where the optical system and imaging conditions vary widely, such as in interchangeable lens digital cameras, where multiple lenses are likely to be attached, the conditions used during training may differ from the actual conditions, making highly accurate distance measurement difficult. Therefore, by performing preprocessing appropriate to the characteristics of the optical system and the imaging sensor during capture, highly accurate distance estimation is possible regardless of the optical system and imaging conditions.

[0070] The above-mentioned "preprocessing" is processing based on the characteristics of the processing for generating a phase difference map, and as will be described below, it is possible to perform processing to absorb differences between acquisition conditions for phase difference image data for learning and acquisition conditions for actual phase difference image data, and processing to reduce differences in image quality between the first phase difference image data and the second phase difference image data. It is preferable to perform at least one of these processing.

[0071] [Example of Preprocessing] Figures 11 and 12 are conceptual diagrams for explaining preprocessing. Specifically, Figure 11 shows phase difference images 900 and 902 (first and second phase difference image data; left and right phase difference images) before preprocessing, and Figure 12 shows phase difference images 900A and 902A (first and second corrected image data) after preprocessing. When phase difference image data (actual image data used to generate a phase difference map) is acquired using phase difference pixels in which a light-shielding mask is arranged as described above (as described above with reference to Figures 3 and 4 ), the brightness distribution is not uniform in the phase difference images 900 and 902 before preprocessing. Therefore, it is preferable to perform processing to make the brightness of the left and right phase difference images (left and right viewpoint images) uniform (processing to eliminate nonuniformity in the image data within the angle of view due to the light-shielding technique), as shown in the phase difference images 900A and 902A.

[0072] Furthermore, when acquiring training images using an optical system other than the monocular phase-contrast optical system configured as described above (for example, when using a compound-eye optical system or when capturing images multiple times using a monocular optical system), the difference in brightness within the phase-contrast images and between the phase-contrast images is small, resulting in a large difference between the training images and the images that will actually be used. For this reason, it is preferable to make the brightness uniform in preprocessing and absorb the influence of the difference between the training images and the actual images.

[0073] Such pre-processing makes it easier to detect corresponding points between images, and enables the generation of a phase difference map and depth image data based on the phase difference map with high accuracy.

[0074] It is preferable that the preprocessing be performed on at least one of resolution, noise, gradation, and image structure, where "gradation" may include brightness and contrast, and "image structure" may include contrast, sharpness, distortion, and shading.

[0075] Furthermore, in the first embodiment, the image acquisition unit 222 and the preprocessing unit 224 (processor) preferably acquire data acquisition conditions for the first phase-difference image data and the second phase-difference image data and perform preprocessing according to the data acquisition conditions. For example, an aperture value (F-number) can be acquired as a data acquisition condition, and the size of the phase-difference image (first and second corrected image data) input to the phase-difference map generator 228 can be enlarged according to the aperture value. Specifically, as the F-number increases, the left and right light rays approach each other and the amount of phase difference decreases. Therefore, when the F-number is large, the image size can be increased to increase the phase difference on the image (apparently), thereby enabling the phase difference to be detected properly. Note that, since the amount of phase difference detected by this processing is large, the phase-difference map generator 228 (processor) preferably reduces the amount of phase difference according to the image size enlargement rate when generating the final phase-difference map.

[0076] Furthermore, in the first embodiment, the preprocessing unit 224 (processor) may perform different preprocessing on the first phase-contrast image data and the second phase-contrast image data. Specifically, the preprocessing unit 224 may perform processing on these phase-contrast image data that differs in at least one of content and degree. Such preprocessing allows a phase-contrast map to be generated with high accuracy, taking into account differences in image quality of the phase-contrast images due to characteristics of the optical system (such as various aberrations). Note that the preprocessing unit 224 may determine the content and degree of preprocessing in response to a user operation via the operation unit 260, or may automatically determine the content and degree of preprocessing without relying on a user operation.

[0077] [Construction of a Phase Difference Map Generator by Machine Learning] In the first embodiment, the phase difference map generator 228 is a phase difference map generator constructed by a machine learning algorithm. Specifically, the phase difference map generator 228 can be constructed by providing a neural network with the first and second phase difference image data and distance information (ground truth data) corresponding to these phase difference images as training data and causing the neural network to learn. Examples of such neural networks include a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), and an autoencoder.

[0078] [Example of CNN Layer Configuration] FIG. 13 is a diagram showing an example of the layer configuration of a convolutional neural network (CNN). In the example shown in part (a) of FIG. 13, the CNN 562 includes an input layer 562A, an intermediate layer 562B, and an output layer 562C. The input layer 562A inputs preprocessed phase contrast images (first corrected image data, second corrected image data) and outputs features. The intermediate layer 562B includes a convolutional layer 564 and a pooling layer 565, and inputs the features output by the input layer 562A to calculate other features. These layers have a structure in which multiple "nodes" are connected by "edges," and hold multiple weight parameters. The values ​​of the weight parameters change as learning progresses. The CNN 562 may also include a fully connected layer 566, as shown in part (b) of FIG. 13. The layer configuration of the CNN 562 is not limited to a case where the convolutional layer 564 and the pooling layer 565 are repeated one by one, but may include any layer (for example, the convolutional layer 564) multiple times in succession. Also, the CNN 562 may include multiple fully connected layers 566 in succession.

[0079] [Processing in the Intermediate Layer] The intermediate layer 562B calculates features through convolution and pooling. The convolution performed in the convolution layer 564 is a process for obtaining a feature map through convolution using a filter, and is responsible for extracting features such as edge extraction from an image. This convolution using a filter generates one channel (one sheet) of "feature map" for each filter. The size of the "feature map" is downscaled by the convolution, becoming smaller as convolution is performed in each layer. The pooling process performed in the pooling layer 565 is a process for reducing (or enlarging) the feature map output by the convolution to create a new feature map, and is responsible for providing robustness to the extracted features so that they are not affected by translation, etc. The intermediate layer 562B can be composed of one or more layers that perform these processes.

[0080] FIG. 14 is a diagram showing the convolution process using a filter in the intermediate layer 562B. In the first convolutional layer of the intermediate layer 562B, a convolution operation is performed between a filter F1 and an image set consisting of multiple phase-contrast images (a training image set during training, and a measurement image set during distance measurement). The image set is composed of images having an image size of, for example, H height and W width. In the case of such images, a two-dimensional filter (3×3), for example, can be used as the filter F1 to be convolved with this image set. Furthermore, in the case of an image having a height of H height, a width of W width, and a depth of D, for example, a three-dimensional filter (3×3×3) can be used. By the convolution operation using this filter F1, one channel (one image) of a "feature map" is generated for each filter F1.

[0081] The size of the filter does not need to be linked to the number of channels of the image, and the number of channels can be freely determined. Also, the number of filters can be determined depending on the number of channels of the next layer.

[0082] Note that, when two phase-contrast images generated by the output of phase-contrast pixels not provided with color filters are input as shown in FIG. 3 described above, the image set has two channels (one channel x two; the first and second phase-contrast images are paired). In this case, one channel each for the left and right may be input and merged midway through the network, or the two channels may be overlapped and input from the beginning. Furthermore, when color phase-contrast images based on the output of phase-contrast pixels provided with color filters are input, the image set is configured of images of 2 x 3 channels (first and second phase-contrast images x 3 (R (red), G (green), B (blue))).

[0083] Similar to the first convolutional layer, the second to nth convolutional layers use filters F2 to F n The size of the "feature map" in the nth convolutional layer is smaller than the size of the "feature map" in the second convolutional layer because it has been downscaled by the previous convolutional layers or pooling layers.

[0084] Of the layers in the intermediate layer 562B, low-level feature extraction (edge ​​extraction, etc.) is performed in the convolutional layers closer to the input side, while higher-level feature extraction (extraction of features related to the shape, structure, etc. of the object; i.e., extraction of feature points and detection of corresponding points) is performed closer to the output side. When segmentation is performed for measurement purposes, upscaling is performed in the latter convolutional layers, and a "feature map" of the same size as the input image set is obtained in the final convolutional layer. On the other hand, when object detection is performed, upscaling is not necessary because position information only needs to be output.

[0085] The intermediate layer 562B may include a layer that performs batch normalization in addition to the convolutional layer 564 and the pooling layer 565. The batch normalization process normalizes the distribution of data in units of mini-batches when performing learning, and plays a role in "accelerating learning," "reducing dependency on initial values," "suppressing overlearning," etc.

[0086] [Processing in the Output Layer] The output layer 562C is a layer that calculates the phase difference amount and phase shift direction of corresponding points (corresponding feature points between phase difference images) in the phase difference image (corrected image) input to the CNN 562 based on the feature map output from the intermediate layer 562B, and outputs the results. A phase difference map can be generated by mapping the phase difference amount and phase shift direction for a large number of corresponding points. In the output layer, for example, a cost feature volume can be constructed by linking or performing correlation calculations while moving one of the left and right feature maps relative to the other, and this cost feature volume can be converted into a cost volume by a three-dimensional convolution operation, after which a phase difference map can be generated by a soft-argmin operation or the like.

[0087] When using the CNN 562 configured as described above, it is preferable that during the learning process, the result output by the output layer 562C is compared with the distance image data (distance information) serving as the correct data for the image set to calculate the loss (error), and a process (error backpropagation) be performed in which the weight parameters in the intermediate layer 562B are updated from the output layer to the input layer so as to reduce the loss.

[0088] [Neural Network Training Method] In the first embodiment, the training control unit 226 (processor) can construct a phase difference map generator 228 (phase difference map generator) by providing phase difference images (pairs of first and second phase difference images) and distance information (distance image data) as ground truth data as training data to a neural network such as the CNN 562. In other words, the phase difference map generator 228 is a trained model constructed by the training method according to the present invention.

[0089] It is also possible to construct a trained model using a device other than the imaging device 10, and then import the constructed model (including the values ​​of weighting parameters, etc.) and use it as the phase difference map generator 228. When using such a trained model, the processor 220 does not need to include the learning control unit 226.

[0090] [Uniformity of Image Data Acquisition Conditions] When acquiring phase-contrast image data for learning, it is preferable to align the acquisition conditions for the first and second phase-contrast image data. Specifically, it is preferable to acquire the first and second phase-contrast images under at least one identical data acquisition condition and construct the phase-contrast map generator 228 through learning using these phase-contrast image data. These "data acquisition conditions" correspond to the imaging conditions used in normal imaging. Specifically, it is preferable that at least one of the focal length of the optical system, the aperture value of the optical system, the shutter speed, and the focal length be the same. The imaging device 10 according to the first embodiment can acquire phase-contrast images using a single optical system and an imaging element having phase-contrast pixels, making it easy to align these data acquisition conditions between phase-contrast images.

[0091] In generating the phase difference map and the distance information, if there is a difference in the data acquisition conditions when acquiring the first and second phase difference image data, or if there is a difference between the "data acquisition conditions when acquiring image data for learning" and the "data acquisition conditions when acquiring image data to be used for actual measurement," the influence of such a difference can be absorbed by the above-described preprocessing. In addition, if there is a difference in brightness between the left and right phase difference images, they may be used for learning with the difference, or they may be used for learning after performing preprocessing to change the state of Figure 11 to the state of Figure 12 and eliminating the difference in brightness.

[0092] Furthermore, in the present invention, phase-contrast images for learning may be acquired using a device other than "a device having a single optical system and phase-contrast pixels" (for example, an imaging device having a compound eye optical system or multiple imaging devices). In this case, it is preferable to absorb the effects of calibration errors and differences in optical characteristics of the imaging devices through preprocessing. Furthermore, learning does not necessarily have to be done using phase-contrast images; normal images may be used as long as the difference between left and right viewpoint images can be learned.

[0093] [Data Acquisition Conditions for Training Phase-Difference Image Data] When acquiring training phase-contrast image data, the same subject is imaged under predetermined data acquisition conditions (imaging conditions) while changing the shooting distance. The focal position (focusing distance) can be fixed. The acquisition conditions for the phase-contrast image data used for training (the number of "predetermined data acquisition conditions") can be one or more sets. High-precision phase-contrast maps and depth image data can be generated by training using a large number of image data acquired under multiple sets of conditions (e.g., conditions with different focal lengths, apertures, and focusing distances). Furthermore, in the present invention, the phase-contrast map generating device may include "multiple phase-contrast map generators with different data acquisition conditions assumed during training," and the processor may switch between these multiple phase-contrast map generators depending on the data acquisition conditions when actually acquiring the phase-contrast image data. However, increasing the number of data acquisition conditions increases the training cost (e.g., the time required for preparing training data and training). Therefore, it is preferable to determine the number of data acquisition conditions by considering both the accuracy required for the phase-contrast map and depth image data and the acceptable training cost.

[0094] In addition, in the present invention, the influence of differences between the data acquisition conditions during learning and the data acquisition conditions during actual measurement is absorbed by preprocessing, so high-precision measurement is possible even with one or a small number of sets of data acquisition conditions.

[0095] [Acquisition of Distance Information as Ground-Point Data] In the learning data, the distance information (distance image data; corresponding to the distance from the imaging device to the subject) as ground-point data can be the results of actual measurements using LiDAR (Light Detection and Ranging, or Laser Imaging Detection and Ranging), etc. LiDAR is a technology that irradiates an object with laser light and measures the distance to the object and the shape of the object from the results of receiving the reflected light, and may be a Time of Flight (TOF) method using pulsed laser light or a Frequency-Modulated Continuous Wave (FMCW) method using continuous-wave laser light.

[0096] In the imaging device 10 according to the first embodiment, the phase difference map can be generated by the above-described method.

[0097] [Construction of a Phase Difference Map Generator Using Multiple Phase Difference Images] In the first embodiment, two phase difference images are input to a neural network for learning, and a phase difference map is generated from the two phase difference images. However, in the present invention, the phase difference map generator is not limited to the use of two phase difference images, and three or more phase difference images may be used. For example, as described above with reference to FIG. 3 , four phase difference images are generated from the outputs of phase difference pixels divided in the horizontal and vertical directions, and these four phase difference images are input to a neural network for learning, thereby constructing a "phase difference map generator that generates a phase difference map from four phase difference images." In this case (when four phase difference images are used), the image set is four channels (when color filters are not provided in the phase difference pixels).

[0098] [Generation and Output of Phase Difference Map] Figure 15 is a diagram showing how a phase difference is calculated from a preprocessed phase difference image. In the example of Figure 15, a corresponding point CP1 (first feature point, corresponding point) and a corresponding point CP2 (second feature point, corresponding point) are detected in preprocessed phase difference images 900A and 902A (first and second corrected image data; the same as the example of Figure 12). These corresponding points are feature points of the subject in Figure 15, and are corresponding points between the phase difference image 900A and the phase difference image 902A, indicating the same position of the same object in real space. The phase difference map generator 228 (processor) then generates a phase difference map by mapping and outputting the phase difference amount (corresponding to distance D1 in the example of Figure 15) and the phase shift direction (left and right direction in Figure 15) for a large number of corresponding points (including corresponding points CP1 and CP2).

[0099] 16 is a diagram showing an example of a phase difference map (a map in a direction perpendicular to the optical axis) for a single subject (point light source). The diagrams on the left show a "front-focus" state, the diagrams in the center show a "pre-focus" state, and the diagrams on the right show a "back-focus" state. In these diagrams, the size of the subject image's spread (blur) indicates the amount of phase difference, and the darkness of the subject image indicates the direction of the phase difference (lighter colors correspond to a "front-focus" state, and darker colors correspond to a "back-focus" state). Since a real subject can be considered to be a collection of multiple point light sources with different distances, brightness, etc., an actual phase difference map will be a superposition of multiple maps like those in FIG. 16.

[0100] Although FIG. 16 shows an example of a two-dimensional display of the phase difference map, the processor 220 can also display the phase difference map in three dimensions. For example, when the phase difference is negative (in FIG. 16 , the blur is close to white), it is displayed as a point below the horizontal plane, and when the phase difference is positive (in FIG. 16 , the blur is close to dark color), it is displayed as a point above the horizontal plane, thereby forming a three-dimensional surface for the entire object field. The processor 220A may simultaneously display the phase difference map in two dimensions and three dimensions, or may switch between them. Furthermore, while FIG. 16 shows the direction of the phase difference in a single shade of color (black and white), multiple colors (for example, purple for the front side and red for the back side) may be assigned to the direction of the phase difference for display.

[0101] The generated phase difference map can be recorded, displayed, output to the outside, etc. by the output control unit 234 and the external input / output unit 236 (processor).

[0102] [Focus Control Using Phase Difference Map] The optical system driver 230 (processor, driver) uses the phase difference map generated by the above-described method to determine a focus position according to the data acquisition conditions of the first and second phase difference image data, and can perform focus control by driving the focus lens 104 (single optical system) of the interchangeable lens 100 to the focus position via the lens driver 110 (driver). The optical system driver 230 may set a focus region in the phase difference map and determine the focus position based on the distribution of phase difference amounts in the focus region. The position, number, and shape of the focus region are not particularly limited, and there may be one or more. Furthermore, the position and size of the focus region may be variable, and an area containing a specific subject (e.g., a person or other specified subject) may be set as the focus region.

[0103] [Second Embodiment] A second embodiment of the present invention will be described. In the second embodiment, the configuration and processing related to generation of a phase difference map are the same as those in the first embodiment, so the same reference numerals are used for the same configuration as in the first embodiment, and detailed description thereof will be omitted. The second embodiment differs from the first embodiment in that depth image data is generated from the phase difference map by post-processing.

[0104] 17 is a diagram showing the configuration of an image processing unit 210A (processor) in an imaging device according to the second embodiment. The image processing unit 210A differs from the image processing unit 210 according to the first embodiment in that a processor 220A includes a post-processing unit 232.

[0105] 18 is a diagram showing how distance image data is generated from a phase difference map in the second embodiment. The processing up to the generation of the phase difference map is the same as in the first embodiment. In the second embodiment, a post-processing unit 232 (processor) converts the phase difference amount in the phase difference map into distance information in the optical axis direction, thereby generating distance image data composed of distance information.

[0106] 19 is a diagram showing post-processing in the second embodiment (calculating distance information from the phase difference amount). As shown in part (b) of FIG. 19, the post-processing unit 232 converts the phase difference amount into a "defocus amount (imaging side)" by using data acquisition conditions such as ray angle information of the first and second phase difference image data. This defocus amount is the defocus amount as distance information.

[0107] The post-processing unit 232 can perform post-processing by taking into consideration, as data acquisition conditions, at least one of ray angle information of the first and second phase-contrast image data, position information of the focus lens 104 (focus lens) of the optical system (interchangeable lens 100 in the second embodiment) used to acquire the first and second phase-contrast image data, and optical characteristics of the interchangeable lens 100 (optical system). The post-processing unit 232 may determine which condition to consider when performing post-processing in accordance with a user instruction via the operation unit 260, or may determine the condition independently of a user instruction. The post-processing unit 232 may also consider characteristics of the subject when determining the data acquisition conditions to be considered in post-processing.

[0108] When the "defocus amount (image side)" is X times the depth of focus, this defocus amount corresponds to a shift of X times the depth of field on the object side. This shift on the object side is the "defocus amount (object side)" in part (a) of FIG. 19, and is the defocus amount as distance information. Since the distance from the current focus position to the focal position is known, the post-processing unit 232 can calculate the distance to the object (the subject distance as distance information obtained by converting the defocus amount) by adding the above-mentioned "defocus amount (object side)" to this distance. Note that it is preferable that the post-processing unit 232 take into account the optical aberration of the interchangeable lens 100 (optical system) when converting the defocus amount between the image side and the object side.

[0109] The post-processing unit 232 can generate at least one of a "defocus map that maps defocus amounts" and a "distance map that maps subject distances" as the depth image data. Which one to generate may be determined in accordance with a user instruction via the operation unit 260, or may be determined independently. The output control unit 234 and the external input / output unit 236 (processor) can record, display, externally output, etc., the generated depth image data.

[0110] Fig. 20 is a diagram showing an example of distance image data (a map of the optical axis direction) generated by the above-mentioned method (the subject is a point light source). The size of the blur corresponds to the distance between the focus position and the object, and the intensity of the blur corresponds to the deviation from the focus position (a light blur corresponds to "fore-focus" and a dark blur corresponds to "back-focus"). The example in Fig. 20 is the above-mentioned "defocus map, which maps the amount of defocus as one aspect of distance image data."

[0111] While FIG. 20 illustrates an example of a two-dimensional display of a defocus map, the processor 220A can display the defocus map and / or the distance map in three dimensions, as described above with reference to FIG. 16 . For example, when the distance is closer to the focus position (the side closer to the image capture device 10; in FIG. 20 , the blur is closer to white), the distance is displayed as a point below the horizontal plane. When the distance is farther from the focus position (the side farther from the image capture device 10; in FIG. 20 , the blur is closer to dark colors), the distance is displayed as a point above the horizontal plane. This allows a three-dimensional surface to be formed as the entire object scene. The processor 220A may simultaneously display the defocus map and / or the distance map in two dimensions and three dimensions, or may switch between them. Furthermore, while FIG. 20 illustrates deviation from the focus position using a single shade of color (black and white), multiple colors may be assigned according to the direction of deviation (for example, purple for the near side and red for the far side).

[0112] [Others] In the first and second embodiments described above, the phase difference map is generated using a machine learning technique, but in the present invention, a technique other than machine learning may be used to generate the phase difference map. For example, the phase difference map can be generated by repeating corresponding point detection and phase difference calculation using normal image processing.

[0113] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described aspects and various modifications are possible.

[0114] 10 Imaging device 100 Interchangeable lens 101 Lens 102 Zoom lens 104 Focus lens 110 Lens driving unit 200 Imaging device body 201 Phase difference pixel 201A Opening 201B Mask 201C Mask 201D Opening 202 Imaging element 202A Mask 202B Color pixel 202G Color pixel 202R Color pixel 202X Phase difference pixel 202Y Phase difference pixel 203 Phase difference pixel 203A Opening 203B Mask 203C Mask 203D Opening 206 A / D converter 207 Phase difference pixel 207-1 Phase difference pixel 207-2 Phase difference pixel 210 Image processing unit 210A Image processing unit 220 Processor 220A Processor 222 Image acquisition unit 224 Pre-processing unit 226 Learning control unit 228 Phase difference map generator 230 Optical system driving unit 232 Post-processing unit 234 Output control unit 236 External input / output unit 260 Operation unit 270 Recording unit 280 Monitor 300 External device 562A Input layer 562B Hidden layer 562C Output layer 564 Convolution layer 565 Pooling layer 566 Fully connected layer 900 Phase difference image 900A Phase difference image 902 Phase difference image 902A Phase difference image d Shift d' Shift F 1 Filter F 2 filter

Claims

1. A phase difference map generating device comprising a processor, wherein the processor: acquires at least first phase difference image data and second phase difference image data from an image data acquisition unit comprising a single optical system and an imaging element having phase difference pixels; performs pre-processing on the first phase difference image data and the second phase difference image data to acquire first corrected image data and second corrected image data; and generates a phase difference map from the first corrected image data and the second corrected image data, mapping the amount of phase difference and the direction of phase shift between the first phase difference image data and the second phase difference image data; and the pre-processing is processing based on characteristics of a processing for generating the phase difference map.

2. The phase difference map generating device according to claim 1, wherein the pre-processing is a process for reducing the difference in image quality between the first phase difference image data and the second phase difference image data.

3. A phase difference map generating device as described in claim 1 or 2, wherein the processor acquires, as the first phase difference image data, image data generated from a signal output from the phase difference pixel having one side of the light receiving section shielded from light, and acquires, as the second phase difference image data, image data generated from a signal output from the phase difference pixel having the other side of the light receiving section shielded from light.

4. The phase difference map generating device according to claim 1 or 2, wherein the processor performs different processes on the first phase difference image data and the second phase difference image data in the pre-processing.

5. The phase difference map generating device according to claim 4, wherein the processor, in the pre-processing, applies processing to the first phase difference image data and the second phase difference image data that differs in at least one of content and degree.

6. The phase difference map generating device according to claim 3, wherein the processor performs, as the pre-processing, a process for eliminating non-uniformity within the angle of view of the image data caused by the light blocking technique.

7. A phase difference map generating device as described in claim 1 or 2, wherein the processor acquires data acquisition conditions for the first phase difference image data and the second phase difference image data, and performs the pre-processing in accordance with the data acquisition conditions.

8. The phase difference map generating device of claim 7, wherein the processor, in the pre-processing, enlarges a size of an image represented by the first phase difference image data and a size of an image represented by the second phase difference image data in accordance with the data acquisition conditions, and generates the first corrected image data and the second corrected image data from the enlarged first phase difference image data and the enlarged second phase difference image data.

9. A phase difference map generating device according to claim 1 or 2, wherein the processor performs the pre-processing for at least one of resolution, noise, gradation, and image structure.

10. A phase difference map generating device as described in claim 1 or 2, wherein the processor extracts corresponding points between the first corrected image data and the second corrected image data, which are at the same position of the same object in real space, and maps the amount of phase difference and the direction of phase shift for the corresponding points to generate the phase difference map.

11. The phase difference map generating device according to claim 10, wherein the processor extracts a first feature point which is a feature point of the first corrected image data and a second feature point which is a feature point of the second corrected image data, extracts the corresponding points from the first feature point and the second feature point, and generates the phase difference map by mapping the amount of phase difference and the direction of phase shift for the corresponding points.

12. A phase difference map generating device as described in claim 1 or 2, wherein the processor generates the phase difference map using a phase difference map generator constructed by machine learning.

13. The phase difference map generating device described in claim 12, wherein the phase difference map generator is a trained model constructed by providing a neural network with the first phase difference image data and the second phase difference image data, and distance information corresponding to the first phase difference image data and the second phase difference image data as training data and training the neural network.

14. A phase difference map generating device as described in claim 13, wherein the phase difference map generator is constructed by learning using the first phase difference image data and the second phase difference image data acquired under at least one identical data acquisition condition.

15. A phase difference map generating device as described in claim 14, wherein the data acquisition conditions for the first phase difference image data and the data acquisition conditions for the second phase difference image data are the same in at least one of the focal length of the optical system, the aperture value of the optical system, the shutter speed, and the focal distance.

16. A phase difference map generating device as described in claim 1 or 2, wherein the processor uses the phase difference map to determine a focus position according to data acquisition conditions of the first phase difference image data and the second phase difference image data.

17. The phase difference map generating device according to claim 16, wherein the processor determines the focus position based on a distribution of phase difference amounts in a focus region set in the phase difference map.

18. A phase difference map generating device as described in claim 1 or 2, wherein the processor generates distance image data constituted by the distance information by converting the amount of phase difference in the phase difference map into distance information in the optical axis direction.

19. The phase difference map generating device of claim 18, wherein the processor generates, as the distance image data, at least one of a defocus map in which the amount of phase difference is converted into a defocus amount as the distance information, and a distance map in which the amount of defocus is converted into a subject distance as the distance information.

20. A phase difference map generating device as described in claim 19, wherein the processor performs post-processing on the phase difference map in accordance with data acquisition conditions of the first phase difference image data and the second phase difference image data to generate the distance image data.

21. The phase difference map generating device described in claim 20, wherein the processor performs the post-processing while taking into account at least one of the following data acquisition conditions: ray angle information of the first phase difference image data and the second phase difference image data, position information of a focus lens of an optical system used to acquire the first phase difference image data and the second phase difference image data, and optical characteristics of the optical system.

22. An image data acquisition device comprising: the phase difference map generating device according to claim 16; the image data acquisition unit; and a drive unit that drives the single optical system, wherein the processor performs focusing control to drive the single optical system to the in-focus position using the drive unit.

23. The image data acquisition device according to claim 22, wherein the imaging element comprises color pixels, at least some of which have arranged thereon one of a number of optical filters that transmit light of different wavelength bands.

24. A phase difference map generating method executed by a phase difference map generating device having a processor, wherein the processor: acquires at least first phase difference image data and second phase difference image data from an image data acquisition unit having a single optical system and an imaging element having phase difference pixels; performs pre-processing on the first phase difference image data and the second phase difference image data to acquire first corrected image data and second corrected image data; and generates a phase difference map from the first corrected image data and the second corrected image data, mapping the amount of phase difference and the direction of phase shift between the first phase difference image data and the second phase difference image data; and the pre-processing is processing based on characteristics of the processing for generating the phase difference map.

25. A focusing control method executed by an image data acquisition device comprising: a processor; an image data acquisition unit including an image sensor having a single optical system and phase difference pixels, and configured to acquire first phase difference image data and second phase difference image data of a subject; and a drive unit configured to drive the image data acquisition unit, wherein the processor: acquires the first phase difference image data and the second phase difference image data of the subject using the image data acquisition unit; performs pre-processing on the first phase difference image data and the second phase difference image data to acquire first corrected image data and second corrected image data; generates a phase difference map from the first corrected image data and the second corrected image data, mapping the amount of phase difference and the direction of phase shift between the first phase difference image data and the second phase difference image data; determines a focusing position according to data acquisition conditions for the first phase difference image data and the second phase difference image data using the phase difference map; and performs focusing control to drive the single optical system to the focusing position using the drive unit, wherein the pre-processing is processing based on characteristics of processing for generating the phase difference map.

26. A learning method for constructing a phase difference map generator that outputs a phase difference map that maps the amount of phase difference and the direction of phase shift between the first phase difference image data and the second phase difference image data, when the first phase difference image data and the second phase difference image data are input, by providing first phase difference image data and second phase difference image data of a subject acquired by an image data acquisition unit equipped with a single optical system and an imaging element having phase difference pixels, and distance information corresponding to the first phase difference image data and the second phase difference image data as learning data to a neural network.

27. The learning method according to claim 26, wherein the distance information is information based on the result of actually measuring the distance from an image data acquisition device that acquires the first phase difference image data and the second phase difference image data to the subject.

28. A phase difference map generator constructed by the learning method of claim 26 or 27.