Image processing apparatus and method, and program, storage medium

Abstract

To accurately acquire a distance image of the entire photographic scenes including a moving subject.SOLUTION: An image processing apparatus has: acquisition means that acquires a plurality of different viewpoint images obtained by photographing a single scene from different positions and at least one pair of pupil-divided parallax images having parallax; first creation means that creates a first distance image from the pair of parallax images; second creation means that creates a second distance image from the plurality of different viewpoint images; and integration means that integrates the first distance image and the second distance image to create an integrated distance image.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image processing apparatus and method, an imaging apparatus and its control method, a program, and a storage medium, and particularly to a technique for generating a distance image from an image obtained by an imaging apparatus.

Background Art

[0002] Conventionally, a technique is known for obtaining a distance image indicating the distance to each subject included in an image using a pair of pupil-segmented images (Patent Document 1). In this technique, in addition to a stationary subject (hereinafter referred to as a "stationary subject"), there is an advantage that a distance image of a scene including a moving subject (hereinafter referred to as a "moving subject") that is the main subject in many shooting scenes can be obtained by one shooting.

[0003] Also, a compound-eye stereo camera has been used for a long time as a camera for distance measurement. The compound-eye stereo camera calculates the distance to a subject from the parallax of the subject that is commonly imaged and the baseline length between cameras using a plurality of cameras whose positional relationship between the cameras is known. However, in a compound-eye camera with a short baseline length, the parallax of the image for a distant subject is small, so the distance measurement accuracy becomes low.

[0004] On the other hand, in a monocular stereo camera, a plurality of images are taken by changing the position and orientation of a single camera, and the relative distance to the subject is calculated from the parallax of the images of the same subject shown in those images. In a monocular stereo camera, since the baseline length can be made large, the relative distance can be accurately calculated even for a distant subject, and the distance measurement accuracy can be improved by using a large number of images, such as several tens to several hundreds of images.

[0005] However, in a monocular stereo camera, since the distance to the subject and the position and orientation of the camera are estimated simultaneously, the distance measurement value is not uniquely determined, and the obtained distance measurement value is a relative distance value rather than an absolute distance value.

[0006] Here, relative distance is a dimensionless quantity, defined, for example, as the ratio of the distance to a subject of interest when the distance to a specific subject is set to 1. In contrast, absolute distance has the dimension of length, and is a value such as 3 [m].

[0007] Therefore, in a rangefinder equipped with both a monocular stereo camera and a compound stereo camera, a technique is known for converting relative distance values ​​acquired by the monocular stereo camera into absolute distance values ​​using information obtained by the compound stereo camera. For example, in Patent Document 2, the absolute distance value of a subject is obtained using the compound stereo camera, the absolute position and orientation of the camera are calculated from there, and then the absolute distance values ​​of all subjects are obtained using these. [Prior art documents] [Patent Documents]

[0008] [Patent Document 1] Patent No. 5192096 [Patent Document 2] Japanese Patent Publication No. 2011-112507 [Non-patent literature]

[0009] [Non-Patent Document 1] Furukawa, Y. and Ponce, J., 2010. "Accurate, dense, and robust multiview stereopsis", IEEE Transactions Actions Pattern Analysis and Machine Intelligence, 32(8), pp. 1362-1376. [Non-Patent Document 2] Abdullah Abuolaim and Michael S. Brown,"Defocus Deblurring Using Dual-Pixel Data",ECCV2020 [Non-Patent Document 3] Liyuan Pan, Shah Chowdhury, Richard Hartley, Miaomiao Liu, Honguang Zhang, Hongdong Li,”Dual Pixel Exploration: Simultaneous Depth Estimation and Image Restoration”,arxiv(CVPR2021) [Non-Patent Document 4] Abhijith Punnappurath, Abdullah Abuolaim, Mahmoud Afifi and Michael S. Brown,"Modeling Defocus-Disparity in Dual-Pixel Sensors",ICCP2020 [Overview of the Initiative] [Problems that the invention aims to solve]

[0010] However, the above-mentioned prior art disclosed in Patent Document 1 had the following problems. Specifically, because the baseline length between the main pixel and the sub-pixel for acquiring a pair of pupil-divided images is short, it is not possible to accurately calculate the distance to subjects within the image that are not close to the imaging device and are outside a predetermined distance range from the focus distance. On the other hand, if the pupil is widened and the baseline length is increased in order to measure the distance to the subject with even a little more accuracy in the configuration of Patent Document 1, the depth of field becomes shallow, and the range in which the distance can be calculated becomes narrower. Thus, when the depth range of the shooting scene is wide, it is difficult to obtain an accurate distance image.

[0011] Furthermore, according to the method described in Patent Document 2, if the absolute distance of the subject acquired by the compound-eye stereo camera is inaccurate, it is not possible to calculate the accurate absolute position and orientation of the camera. As a result, there is a problem in that the absolute distance values ​​of all subjects also become inaccurate.

[0012] The present invention has been made in view of the above problems, and an object thereof is to accurately acquire a distance image of an entire scene including a moving subject.

[0013] Another object is to accurately convert the relative distance value obtained by a monocular camera into an absolute distance value.

Means for Solving the Problems

[0014] To achieve the above object, an image processing apparatus according to the present invention includes acquisition means for acquiring a plurality of different viewpoint images captured from different positions of the same scene, and at least one pair of parallax image pairs having parallax by pupil division; first generation means for generating a first distance image from the parallax image pair; second generation means for generating a second distance image from the plurality of different viewpoint images; and integration means for integrating the first distance image and the second distance image to generate an integrated distance image. 、 and has Furthermore, in integrating the first distance image and the second distance image, the integrating means complements the distance information of the moving subject region within the distance information constituting the second distance image with the distance information of the same region in the first distance image. an image processing apparatus characterized by the above.

Effects of the Invention

[0015] According to the present invention, a distance image of an entire shooting scene including a moving subject can be accurately acquired.

[0016] Also, the relative distance value obtained by a monocular camera can be accurately converted into an absolute distance value.

Brief Description of the Drawings

[0017] [Figure 1] A block diagram showing a schematic configuration of an imaging apparatus in a first embodiment of the present invention. [Figure 2] A diagram for explaining an imaging unit in a first embodiment. [Figure 3] A flowchart showing procedures for shooting and generating an integrated distance image in a first embodiment. [Figure 4] A flowchart of single-shot distance image generation processing in a first embodiment. [Figure 5]A diagram for explaining a method of converting a defocus amount into a distance value. [Figure 6] A diagram for explaining a method of generating a multi-view distance image. [Figure 7] A diagram showing an example of integration of a single-shot distance image and a multi-view distance image in the first embodiment. [Figure 8] A diagram showing another example of integration of a single-shot distance image and a multi-view distance image in the first embodiment. [Figure 9] A diagram for explaining an example of variation in shooting timing in the first embodiment. [Figure 10] A diagram for explaining an example of change in shooting conditions in the first embodiment. [Figure 11] A block diagram showing a schematic configuration of an image processing apparatus in the second embodiment. [Figure 12] A diagram for explaining the selection of a shot for generating a single-shot distance image during post-processing in the second embodiment. [Figure 13] A diagram showing an outline of a generation procedure of an integrated distance image when integrating a plurality of single-shot distance images in a modification example. [Figure 14] A block diagram showing a schematic configuration of an image processing apparatus in the third embodiment. [Figure 15] A flowchart showing the procedures of shooting and generating an integrated distance image in the third embodiment. [Figure 16] A diagram showing the input-output relationship of a network used for focus blur recovery in the third embodiment. [Figure 17] A diagram showing an outline of the procedures from shooting to generating an integrated distance image in the third embodiment. [Figure 18] A diagram showing an outline of the procedures from multi-frame acquisition to generating an integrated distance image in moving image shooting in the third embodiment. [Figure 19A] A block diagram showing the configuration of the imaging apparatus according to the fourth embodiment of the present invention. [Figure 19B] A front view of the appearance of the imaging apparatus according to the fourth embodiment. [Figure 19C] A rear view of the appearance of the imaging apparatus according to the fourth embodiment. [Figure 20]A flowchart illustrating the operation of the distance measurement process in the fourth embodiment. [Figure 21] A flowchart showing the detailed processing in S401 and S402 in Figure 20. [Figure 22] A diagram showing the mode selection screen on the display unit. [Figure 23] A diagram showing the display during distance measurement. [Figure 24] A diagram illustrating feature point tracking using a monocular stereo camera. [Figure 25] A flowchart showing the detailed processing in S403 and S404 in Figure 20. [Figure 26] A diagram illustrating window matching in a compound-eye stereo camera. [Figure 27] A diagram illustrating the structure of a pupil-splitting image sensor and the principle of distance calculation. [Modes for carrying out the invention]

[0018] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0019] <First Embodiment> Figure 1 is a block diagram showing the schematic configuration of the imaging device 100 in the first embodiment of the present invention, and shows only the components necessary for explaining the present invention. The imaging device 100 includes an imaging unit 101 including an optical system 1011 and an image sensor 1012, a memory 102, a viewing image generation unit 103, a single-shot distance image generation unit 104, a multi-view distance image generation unit 105, and a distance image integration unit 106.

[0020] Figure 2 is a diagram illustrating the imaging unit 101. In Figure 2(a), the optical system 1011 is composed of multiple lenses and mirrors, etc., and forms an image of reflected light from the subject 10 onto the light-receiving surface of the image sensor 1012. Note that in Figure 2(a), the optical system 1011 is represented by a single lens. 202 is the optical axis of the optical system 1011. The image sensor 1012 receives the optical image of the subject 10 formed by the optical system 1011, converts it into an electrical signal, and outputs it.

[0021] As shown in Figure 2(b), the image sensor 1012 has a number of pixels 210 arranged in the xy plane, each covered by a so-called Bayer array color filter 222 consisting of red (R), green (G), and blue (B), which will be described later. As a result, each pixel of the image sensor 1012 is given spectral characteristics corresponding to the wavelength band transmitted by the color filter 222, and outputs signals mainly for red, green, and blue light, respectively.

[0022] Figure 2(c) is a cross-sectional view showing the configuration of each pixel, which includes a microlens 211, a color filter 222, photoelectric conversion units 210a and 210b, and a waveguide 213. The substrate 224 is made of a material that absorbs light in the wavelength range transmitted by the color filter 222, such as silicon (Si), and the photoelectric conversion units 210a and 210b are formed in at least a portion of the interior region by ion implantation or the like. Each pixel also has wiring (not shown).

[0023] The photoelectric conversion units 210a and 210b are each incident on a light beam 232a that has passed through a first pupil region 231a, which is a different pupil region of the exit pupil 230 of the optical system 1011, and a light beam 232b that has passed through a second pupil region 231b. As a result, a pupil-divided first signal and a second signal can be obtained from each pixel 210. Alternatively, the first signal and the second signal, which are the signals from the photoelectric conversion units 210a and 210b respectively, may be read out independently from each pixel 210. Or, after reading out the first signal, the signal obtained by adding the first signal and the second signal may be read out, and the second signal may be obtained by subtracting the first signal from the added signal.

[0024] Figure 2(d) shows the arrangement of the photoelectric conversion units 210a and 210b corresponding to each microlens 211 on the image sensor 1012, as viewed from the direction of incidence of the optical axis, and is an example of a case where the units are divided horizontally or vertically. However, the present invention is not limited to this, and the units may be arranged so that pixels divided horizontally and pixels divided vertically are mixed. By arranging them in this way, it becomes possible to obtain the amount of defocus, as described later, with high accuracy not only for subjects whose brightness distribution changes horizontally, but also for subjects whose brightness distribution changes vertically. Furthermore, three or more photoelectric conversion units may be configured for each microlens 211, and Figure 2(e) shows an example in which one pixel has four photoelectric conversion units 210c to 210f divided horizontally and vertically.

[0025] The first and second signals obtained from the photoelectric conversion units 210a and 210b are sent to the arithmetic processing unit 204 included in the imaging unit 101 and converted into electronic information. If the signal acquired by photoelectric conversion is analog, the arithmetic processing unit 204 performs basic processing such as noise reduction by correlated double sampling (CDS), exposure control by gain increase with automatic gain control (AGC), black level correction, and A / D conversion to obtain an image signal converted into a digital signal. Since these calculations mainly involve pre-processing of analog signals, they are generally called AFE (analog front-end) processing. When used in conjunction with a digital output sensor, they are sometimes called DFE (digital front-end) processing.

[0026] Then, an image A is generated by collecting the first signals output from multiple pixels 210 of the image sensor 1012, and an image B is generated by collecting the second signals output from multiple pixels 210. Since images A and B are images with parallax, they will be referred to as "parallax images" respectively, and images A and B together will be referred to as a "parallax image pair".

[0027] The arithmetic processing unit 204 also performs Bayer array interpolation and other operations when the image sensor 1012 is a color sensor. Furthermore, in order to improve the quality of the parallax image and the quality of the viewing image output together with the pair of parallax images (described later), filtering processes such as low-pass and high-pass filters and sharpening processes may be performed. In addition, various processes such as tonal correction including dynamic range expansion such as HDR (High Dynamic Range) processing and color correction such as WB (White Balance) correction may be performed. Note that the processing of the arithmetic processing unit 204 tends to be integrated with the image sensor 1012 at the chip level or unit level, and is therefore not shown in Figure 1.

[0028] In this way, by forming multiple photoelectric conversion units beneath each of the multiple microlenses 211 formed on the light-receiving surface of the image sensor 1012, the multiple photoelectric conversion units each receive the subject light beam that has passed through different pupil regions of the optical system 1011. As a result, even if the optical system 1011 has only one aperture, it becomes possible to obtain a pair of disparity images in a single shot. The generated disparity image pairs are temporarily stored in memory 102.

[0029] Returning to Figure 1, the viewing image generation unit 103 receives the pair of disparity images obtained in a single capture, stored in the memory 102. The first signal and the second signal are added to each pixel to generate a single image. In other words, a viewing image (hereinafter referred to as the "viewing image") is generated for the image formed by the light beam that has passed through the entire pupil area of ​​the optical system 1011.

[0030] Furthermore, if focus detection using parallax images or generation of distance images is not performed, the generation of the viewing image may be performed by the arithmetic processing unit 204 integrated at the chip level with the image sensor 1012, or by adding the first signal and the second signal within each pixel before reading them out. In this case, it is possible to save transmission bandwidth and shorten the time required for reading out. Also, if a viewing image that is ultimately paired with the distance image is not required, the viewing image generation unit 103 may not be explicitly present and may be included in the multi-view distance image generation unit 105.

[0031] The single-shot distance image generation unit 104 receives a pair of parallax images obtained in a single shot, stored in the memory 102. Conversion to a luminance image may be performed at this time. The unit then associates the input parallax images with each other and generates a distance image based on imaging information that includes camera parameters such as focal length and aperture value determined by the zoom state of the optical system 1011, and image sensor information such as the pixel pitch of the image sensor 1012. Hereinafter, the distance image generated by the single-shot distance image generation unit 104 will be referred to as a "single-shot distance image".

[0032] The multi-view distance image generation unit 105 takes as input multiple pairs of parallax images obtained by taking multiple consecutive shots of the same scene from different positions, i.e., multi-shots, and converts each shot into a single image by the viewing image generation unit 103, and generates a distance image using multiple viewing images (different viewpoint images). Hereinafter, the distance image generated by the multi-view distance image generation unit 105 will be referred to as a "multi-view distance image". In the case of a camera whose imaging device is moving, parallax occurs between viewing images obtained by taking multi-shots in a time series. Therefore, if the movement and change in orientation of the imaging device 100 are known, the multi-view distance image can be calculated from the parallax between viewing images (different viewpoint images).

[0033] The distance image integration unit 106 integrates the single-shot distance image and the multi-view distance image to generate an integrated distance image.

[0034] Next, the procedure for image acquisition and integrated distance image generation in this embodiment will be described with reference to the flowchart in Figure 3. In S101, shooting is performed for generating single-shot distance images (single shot) and shooting is performed for generating multi-view distance images (continuous shooting). The shooting order will be described later, referring to Figure 9.

[0035] In S102, a distance measurement process is performed using the pair of disparity images obtained by single-shot imaging to calculate a single-shot distance image. Figure 4 is a flowchart of the single-shot distance image generation process performed in S102. In S201 of Figure 4, the image displacement amount, which is the relative positional displacement between the disparity images, is calculated. Known methods can be used to calculate the image displacement amount. For example, the correlation value is calculated from the signal data A(i) and B(i) of image A and image B using the following equation (1). In equation (1) of TIFF0007884349000001.tif19129, S(r) is the correlation value indicating the degree of correlation between two images at an image shift amount r, i is the pixel number, and r is the relative image shift amount of the two images. p and q indicate the target pixel range used to calculate the correlation value S(r). The image displacement amount can be calculated by finding the image shift amount r that gives the minimum value of the correlation value S(r).

[0036] Furthermore, the method for calculating the amount of image displacement is not limited to the method described above, and other known methods may also be used.

[0037] Next, in S202, the amount of defocus is calculated from the amount of image shift calculated in S201. The image of the subject 10 is formed on the image sensor 1012 via the optical system 1011. In the example shown in Figure 2(a) above, a defocused state is shown where the light beam that passed through the exit pupil 230 is focused at the imaging plane 207. This defocused state is a state in which the imaging plane 207 and the imaging plane (light-receiving surface) of the image sensor 1012 do not coincide and are shifted in the direction of the optical axis 202, and the amount of defocus indicates the distance between the imaging plane of the image sensor 1012 and the imaging plane 207.

[0038] Here, we will explain an example of a method for converting the defocus amount to a distance value using the simplified optical arrangement diagram of the imaging device shown in Figure 5.

[0039] Figure 5 shows the light ray 232 when the image of the subject 10 is defocused relative to the image sensor 1012, where 202 is the optical axis, 208 is the aperture diaphragm, 205 is the front principal point, 206 is the rear principal point, and 207 is the imaging plane. Also, d is the amount of image shift, W is the baseline length, D is the distance between the image sensor 1012 and the exit pupil 230, Z is the distance between the front principal point 205 of the optical system 1011 and the subject 10, L is the distance between the imaging plane of the image sensor 1012 and the rear principal point 206, and ΔL is the amount of defocus.

[0040] In the imaging device of this embodiment, the distance to the subject 10 is detected based on the defocus amount ΔL. The image shift amount d, which indicates the relative positional shift between image A, which relies on the first signal acquired from the photoelectric conversion unit 210a of each pixel 210, and image B, which relies on the second signal acquired from the photoelectric conversion unit 210b, and the defocus amount ΔL have the relationship shown in equation (2). Equation (2) of TIFF0007884349000002.tif34141 can be simplified and written as in equation (3) using the proportionality constant K.

[0041] ΔL≒K·d …(3) The coefficient used to convert the image displacement amount into the defocus amount is hereinafter referred to as the "conversion coefficient." The conversion coefficient is, for example, the proportionality constant K or the baseline length W shown in equation (3). Hereafter, the correction of the baseline length W is equivalent to the correction of the conversion coefficient. Note that the method for calculating the defocus amount is not limited to the method of this embodiment, and other known methods may be used.

[0042] Furthermore, the conversion from the defocus amount to the subject distance can be performed using the following equation (4), which shows the imaging relationship between the optical system 1011 and the image sensor 1012. Alternatively, the image shift amount can be directly converted to the subject distance using a conversion coefficient. In equation (4), f is the focal length. TIFF0007884349000003.tif35145 By determining the amount of defocus for all pixels between multiple input disparity images, for example, between image A and image B, a defocus map can be calculated. By transforming the defocus map using the relationship in equation (4), the corresponding single-image distance image can be calculated.

[0043] By following the distance calculation procedure described above, the pupil-splitting imaging system can calculate a single-shot distance image from a pair of disparity images obtained in a single shot.

[0044] Returning to Figure 3, in the next step S103, the viewing image generation unit 103 takes as input a viewing image corresponding to the light that has passed through the entire pupil area, which is generated from a pair of parallax images obtained for each shot of the continuous shooting sequence, and generates a multi-view distance image from multiple viewing images. It is assumed that the shooting is performed while moving, such as by holding the camera by hand, and the three-dimensional scene is reconstructed from multiple viewing images with overlapping fields of view, and a distance image for each shot is generated in the process.

[0045] The relative position and orientation of the imaging device 100 between shots during continuous shooting can be determined using attitude sensors such as gyro sensors, acceleration sensors, and tilt sensors that are commonly attached to the imaging device 100 in recent years, or by known camera attitude estimation methods that combine vibration detection and image vectors, which are standard features of imaging devices in recent years. Since the method for determining the position and orientation is known, an explanation will be omitted here.

[0046] If the changes in the position and orientation of the imaging device 100 between shots in continuous shooting are known, the estimation of the distance to the subject in the image can be reduced to a simple one-dimensional search problem due to the epipolar geometry constraint, as shown in Figure 6(a). Furthermore, if it is possible to track the point of the subject in the same space across multiple viewing images over multiple frames, this epipolar search can be replaced with a multi-baseline search that finds the same distance between multiple sets of different viewing images. By utilizing this, it becomes possible to determine the distance with high accuracy. For example, as shown in Figure 6(b), by simply adding one more shot, distance image estimation with respect to the reference viewpoint C1 of the first shot becomes possible not only between the shot at viewpoint C2 but also between the shot at viewpoint C3.

[0047] By increasing the number of related shots in this way, it becomes possible to perform multiple searches for the depth value of each point in the image of the shot set as the reference image. This enables robust and highly accurate estimation of distance values. On the other hand, for moving subjects, epipolar geometry no longer holds between the correspondences of each shot, so the distance cannot be calculated in the multi-view distance image.

[0048] There are various known methods for matching shots. For example, there is a patch-based matching method called PMVS, which calculates distance values ​​while also considering the normal direction (Non-Patent Literature 1), and a method called plane sweep, which sets a virtual depth plane and matches shots by back projection. The generation of multiview distance images is also called the multiview stereo method. Using the methods described above, a multiview distance image corresponding to the viewing image of the shot selected as the reference image can be obtained from a group of viewing images, which are aggregated images of light images from the entire pupil area captured in continuous shooting.

[0049] In S104, single-shot distance images and multi-view distance images are integrated to generate a highly accurate integrated distance image for the entire depth direction of the scene.

[0050] As explained above, single-shot distance images contain distance information for both stationary and moving subjects, while multi-view distance images contain only distance information for stationary subjects. This is because the correspondence of moving subjects does not satisfy epipolar geometry when generating multi-view distance images. Therefore, in the integration of distance images in this embodiment, only the distance values ​​for moving subjects are taken from the single-shot distance image, and the distance information for overlapping stationary subjects is obtained from the multi-view distance image and integrated.

[0051] When a single-shot distance image and a multi-view distance image are superimposed, areas where distance information exists in both can be considered as stationary subject areas, while areas where distance information exists only in the single-shot distance image, and where distance information is uncalculated or unreliable in the multi-view distance image, can be considered as moving subject areas. Therefore, by obtaining distance information for stationary subject areas from the multi-view distance image and distance information for moving subject areas from the single-shot distance image and integrating them, a highly accurate integrated distance image can be obtained.

[0052] Figure 7 shows an example of integrating single-shot distance images and multi-view distance images. Figure 7(a) is the viewing image corresponding to the single-shot distance image, Figure 7(b) is the single-shot distance image, Figure 7(c) is the corresponding multi-view distance image, and (d) is the integrated distance image. In the single-shot distance image in Figure 7(b), distance information exists for the moving subject, the person, but in the multi-view distance image in Figure 7(c), distance information does not exist for the moving subject, the person, and is therefore represented in white. Therefore, the distance information for the moving subject area is obtained from the single-shot distance image in Figure 7(b), and the distance information for the overlapping stationary subject area is obtained from the multi-view distance image in Figure 7(c), to generate the integrated distance image in Figure 7(d).

[0053] Furthermore, as mentioned above, a characteristic unique to pupil-splitting imaging systems is that the distance to a subject that is not close to the imaging device and is outside the predetermined distance range from the focus distance cannot be calculated accurately due to the short baseline length of the pupil-splitting optical system. To address this problem, distance information for the moving subject area and the nearby area and the area within the predetermined distance range of the focus distance can be obtained from the single-shot distance image based on the distance information of the single-shot distance image, and distance information for the remaining stationary subject area can be obtained from the multi-view distance image to generate an integrated distance image.

[0054] Figure 8 shows an example of determining from which distance values ​​to obtain the combined distance image based on the distance values ​​of the single-shot distance image. u0 is the focus distance, v1 and v2 indicate a predetermined distance range from the focus distance u0, where v1 is the front-focus side and v2 is the back-focus side. Also, u1 is the distance (threshold) indicating the nearby region from the imaging device 100. If u is the distance value of each position in the distance image, u≦u1 …(5) -v1≦u-u0≦v2 …(6) Distance values ​​(distance information) are obtained from single-shot distance images for regions that satisfy at least one of equations (5) and (6), and from multi-view distance images for all other regions. This makes it possible to obtain a distance image with higher accuracy for the entire depth direction of the scene than using only single-shot distance images.

[0055] Furthermore, in multi-view distance images, distance information for moving subjects is missing as uncalculated regions, while distance information for other stationary subjects overlaps with the same regions in single-shot distance images. However, in distance images, the boundary between moving and stationary subjects tends to be blurred. Therefore, when extracting distance information from each distance image, it is advisable to use moving subject region detection with the viewing image as input. For example, one could detect faces or human bodies and extract those regions, or utilize a CNN network that has improved the accuracy of extracting moving subject regions by leveraging prior knowledge gained through learning.

[0056] Furthermore, the distance information (pixels) that constitute a depth image is not necessarily limited to the subject distance (distance value); it may also be the amount of image shift or defocus before conversion to subject distance. In addition, the reciprocal of the subject distance (inverse distance value) may be used as the distance information (pixels) that constitutes a depth image.

[0057] Furthermore, in S101, single-shot and continuous shooting can be performed in various ways. That is, the shot to obtain a single-shot distance image may be taken as one shot in a continuous shooting sequence, or as a single shot. Also, continuous shooting may be performed before or after the single shot that will obtain the single-shot distance image. In addition, if the movement of the imaging device 100 during continuous shooting is small and the baseline length cannot be obtained, the shot may be taken later with a time delay.

[0058] Figure 9 shows examples of variations in shooting timing. Figure 9(a) shows an example where multiple images are acquired by first performing continuous shooting as pre-shooting, and then single shooting is performed to obtain a single-shot distance image. In Figure 9, SW1 represents the state where the shutter of the imaging device 100 is half-pressed, and SW2 represents the state where the shutter is fully pressed. Pre-shooting is a shooting method that utilizes a cyclic buffer and is a common function that is standard not only in imaging devices such as cameras, but also in the camera functions of smartphones, etc.

[0059] In this shooting method, the image taken before the full shutter press SW2 is temporarily stored, allowing the user to select the best shot when the shutter is fully pressed SW2, or to save a certain number of continuous shots for post-processing. This pre-shot image is acquired as input for generating the multi-view distance image. In this case, since the pre-shot image is only used for generating the multi-view distance image, the signals from the photoelectric conversion units 210a and 210b may be added pixel by pixel during shooting to reduce the image data size and save it as a viewing image. Alternatively, the image obtained when the shutter is fully pressed SW2 is saved as a pair of parallax images and used to generate the single-shot distance image, but it may also be used to generate a viewing image by adding the signals pixel by pixel and using it to generate the multi-view distance image. In that case, the multi-view distance image is generated based on this image.

[0060] Figure 9(b) shows an example where continuous shooting is performed after taking a single-shot distance image. This shooting method involves taking a certain amount of continuous shots for a certain period of time after the shutter of the imaging device 100 is fully pressed SW2, and saving the images. The best shot of the moving subject is taken when the shutter is fully pressed SW2. Then, shooting continues even after the shutter is released to obtain images for generating the multi-view distance image. Since the images taken after the shutter is released are only used for generating the multi-view distance image, they may be added pixel by pixel during shooting, as in the case of pre-shooting, and saved as viewing images with reduced storage capacity. However, the images obtained when the shutter is fully pressed SW2 are saved as a pair of parallax images and used for generating the single-shot distance image. Alternatively, these shots may also be added and used for generating the multi-view distance image. In that case, the multi-view distance image is generated based on this image.

[0061] Figure 9(c) shows the case where pre-shooting and post-shooting are performed before and after fully pressing the shutter SW2. Images obtained at times other than when the shutter SW2 is fully pressed are acquired from a circular buffer. Since these images are only used to generate the multi-view distance image, they may be added pixel by pixel during shooting to reduce storage capacity and save as viewing images. Alternatively, the image obtained when the shutter SW2 is fully pressed is saved as a pair of parallax images and used to generate the single-shot distance image, but it may also be added pixel by pixel to generate a viewing image, which is then used to generate the multi-view distance image. In that case, the multi-view distance image is generated based on this image.

[0062] Furthermore, if the photographer intentionally selects the best shot before generating the distance image, such as by operating the shutter of the imaging device, to obtain a set of viewing image and distance image, the imaging conditions can be explicitly switched between single-shot distance image generation and multi-view distance image generation. For example, as shown in Figure 10, when shooting in burst mode for multi-view distance image generation, the Av (aperture) value is set high, and the aperture is stopped down to achieve pan-focus, making it easier to capture feature points across the entire depth direction of the scene. On the other hand, the Av value can be changed to a lower value only when the shutter is fully pressed (SW2), allowing the pupil to open and enabling accurate distance measurement near the focus distance during single-shot shooting. The Tv value and ISO value should be set appropriately according to the program table, etc., in accordance with the Av value.

[0063] Furthermore, since only the distance value is ultimately integrated, it is not necessary to match the Ev (exposure) value, which is the sum of the Av value, Tv value, and ISO. Therefore, as shown in Figure 10(b), the Ev value of the shooting conditions can be significantly changed between imaging for single-shot distance image generation and multi-view distance image generation. Figure 10(b) shows an example where different Ev values ​​are used between imaging for single-shot distance image generation and multi-view distance image generation. For example, adopting such shooting conditions when shooting in dark places expands the range of situations in which the technology of the present invention can be used.

[0064] For example, when shooting in burst mode, you want to achieve pan-focus but also suppress motion blur. Therefore, even with insufficient light, you lower the Ev value to set the shooting conditions, for example, setting the combined Av, Tv, and ISO to 11. On the other hand, when shooting in single shot mode, you can increase the Ev value by increasing the Av value, but trying to force the Ev value to match that of burst shooting would reduce the amount of light and increase noise, so you intentionally create a mismatch. Even when shooting in this way, problems are unlikely to occur because the captured images themselves are not being merged. Also, when using images obtained from single shots to generate a multi-view distance image, you adjust the tonal gradation of the image pixel values ​​by the difference in Ev values. In this case as well, it only indirectly affects the final merged distance image, so it is unlikely to have an impact.

[0065] As described above, according to this first embodiment, by integrating single-shot distance images and multi-view distance images in a way that complements the shortcomings of each, it is possible to accurately acquire distance images of the entire shooting scene, including moving subjects.

[0066] Although the integrated distance "image" is mentioned, the output information does not have to be in the format of an image. For example, it can be a 2.5-dimensional stereoscopic image projected onto a 3D space according to the distance values, a point cloud or volume data with different storage formats, or stereoscopic data converted into mesh information.

[0067] <Second Embodiment> Next, a second embodiment of the present invention will be described. Figure 11 is a block diagram of the image processing apparatus 200 in the second embodiment.

[0068] The image input unit 201 receives a pair of disparity images captured by an external imaging device (not shown), and if an image for viewing has been obtained by adding the pairs of disparity images pixel by pixel within the imaging device, it receives the image for viewing and stores the input pairs of disparity images and the image for viewing in the memory 102. Then, it processes the input pairs of disparity images and the image for viewing as described in the first embodiment to obtain a combined distance image. Note that the configuration other than the image input unit 201 is the same as that shown in Figure 1, so the same reference numerals are used and their explanation is omitted here.

[0069] As shown in Figure 12(a), when parallax image pairs for all shots are input, it is possible to select the parallax image pair to be used to generate the single-shot distance image and the reference viewing image when generating the multi-view distance image. Therefore, from the multiple parallax image pairs obtained by continuous shooting, the parallax image pair to be used to generate the single-shot distance image and the reference viewing image are selected retrospectively or by a predetermined algorithm. For example, it may be set in advance to select the parallax image pair obtained from the last shot of all shots. Since such cases are particularly anticipated when shooting video, images obtained from the shot corresponding to each frame, or from shots every few frames, are always acquired as parallax image pairs during video shooting.

[0070] Alternatively, as shown in Figure 12(b), the selected pairs of disparity images may be sequentially moved to generate single-shot distance images and multi-view distance images. In this way, for all pairs of disparity images, a highly accurate integrated scene-wide distance image can be obtained by integrating the single-shot distance image and multi-view distance image, even for scenes with a wide depth.

[0071] As described above, according to the second embodiment, an image processing device can obtain a highly accurate distance image using a pair of disparity images obtained from a shooting device.

[0072] <Variation> In the descriptions of the first and second embodiments above, the capture of a pair of disparity images for generating a single-shot distance image was described as a typical single-shot capture. However, in cases where the subject is moving slowly, multiple pairs of disparity images may be captured, and the single-shot distance image generation unit 104 may generate a single-shot distance image from each pair, integrate them to improve the quality of the single-shot distance image, and then integrate it with the multi-view distance image.

[0073] Referring to Figure 13, the process of generating single-shot distance images from multiple pairs of disparity images, integrating them, improving the quality of the single-shot distance images, and then integrating them with the multi-view distance image is explained. For example, if the position or orientation of the imaging device changes between shots, the single-shot distance images are transformed by changing the viewpoint in three dimensions, and then integrated into single-shot distance images that match the coordinates at the time of the reference shot.

[0074] For example, in Figure 13, the central pair of disparity images is set as the reference, and the single-shot distance images obtained from the remaining pairs of disparity images are transformed to match the shooting position and orientation at the time of shooting. Then, the two transformed single-shot distance images and the reference single-shot distance image are combined, and the combined single-shot distance image is combined with the multiview distance image to obtain the final combined distance image.

[0075] As described above, by modifying the image, a more accurate distance image can be obtained.

[0076] <Third Embodiment> Next, a third embodiment of the present invention will be described. In the first and second embodiments described above, we explained the case in which a single-shot distance image generated from a pair of disparity images obtained in a single shot using a pupil-splitting imaging system and a multi-view distance image generated from multiple viewing images obtained by performing multi-shots in a time series are integrated to generate an integrated distance image. However, in order to obtain a single-shot distance image with good accuracy using a pair of disparity images captured by a pupil-splitting imaging system as input, it is necessary to widen the pupil and lengthen the baseline. On the other hand, widening the pupil makes the depth of field shallower, so subjects outside the predetermined distance range from the focus position become blurred, making it difficult to match them with other viewing images. As a result, it becomes impossible to accurately calculate the distance to subjects outside the predetermined distance range from the focus position using the viewing images.

[0077] Thus, in principle, there are limitations to the shooting scenes and conditions under which both accurate single-shot distance images and multi-view distance images can be obtained. For example, in environments such as dark places, the selectable shooting conditions become even more stringent, and it becomes more difficult to obtain both single-shot distance images and multi-view distance images with good accuracy. This is because in dark places, it is necessary to keep the shutter speed short to prevent motion blur, but since there is insufficient light, it is necessary to open the pupil of the optical system and reduce the F-number to increase the amount of light. As mentioned above, opening the pupil of the optical system reduces the depth of field, so the depth range in which multi-view distance images can be generated with good accuracy is narrowed.

[0078] Therefore, in the third embodiment, shooting is performed in a way that allows for the acquisition of parallax image pairs even in multi-shot shooting other than the main shooting where parallax image pairs were acquired by the pupil-splitting imaging system. Then, using the parallax image pairs from each shot, a viewing image is generated in which the focus blur of the image area outside the predetermined distance range from the focus position is restored and the depth of field is expanded, and correspondence between the multiple viewing images obtained from multi-shot shooting is performed without focus blur. Furthermore, by integrating the single-shot distance image and the multi-view distance image to generate an integrated distance image, a more accurate integrated distance image is obtained for the entire scene, including the distance to the subject outside the predetermined distance range from the focus position.

[0079] Figure 14 is a block diagram showing the image processing apparatus 300 in the third embodiment. In Figure 14, components similar to those in the image processing apparatus 200 described in the second embodiment with reference to Figure 11 are given the same reference numerals, and their descriptions are omitted.

[0080] In the third embodiment, the viewing image generation unit 303 can perform the same process as the viewing image generation unit 103: generating a viewing image directly from the input parallax image pair, and generating a viewing image after performing a focus blur recovery process on the input parallax image pair. In other words, focus blur recovery may be applied to all of the multiple viewing images input to the multi-view distance image generation unit 105, or it may be applied to only some of them. Therefore, the images input to the viewing image generation unit 303 may all be parallax image pairs, or a mixture of parallax image pairs and viewing images output after adding the first signal and the second signal pixel by pixel may be used.

[0081] Next, the procedure for image acquisition and integrated distance image generation in the third embodiment will be described with reference to the flowchart in Figure 15.

[0082] In S301, parallax image pairs used for generating single-shot distance images and recovering focus blur are captured multiple times in succession (multi-shot). If single-shot distance image generation and focus blur recovery are performed for all images obtained by multi-shot, all shots are acquired as a set of multiple parallax images (parallax image pairs) consisting of multiple images corresponding to the number of viewpoints under the microlens 211. If single-shot distance image generation and focus blur recovery are not performed for some of the images obtained by multi-shot, pre-generated viewing images may be mixed in. If multiple images are acquired by video recording instead of multi-shot, it is simpler to control by acquiring parallax image pairs for all frames.

[0083] In S302, the reference shot for generating the integrated distance image is selected from multiple images obtained by performing a multi-shot. In the case of continuous shooting of still images, the reference shot may be selected in advance using the GUI of the imaging device at the time of shooting. Alternatively, the reference shot may be selected using the GUI (not shown) of the image processing device 300. In the case of video recording, the reference frame for generating the integrated distance image is sequentially moved in the time direction.

[0084] In S303, a single viewing image is generated by taking as input parallax image pairs, consisting of multiple images or a portion thereof, corresponding to the number of viewpoints under the microlens 211 captured in each shot, and performing focus blur recovery.

[0085] Focus blur recovery processing for each shot's disparity image pair may be implemented by deconvolution processing or MAP estimation processing, which involves calculating blind or distance images to estimate the focus blur kernel. Alternatively, it may be implemented by deep learning processing as an alternative. When implementing focus blur recovery processing for each shot's disparity image pair using deep learning processing, end-to-end processing using an encoder-decoder structure is the first thing to consider. Alternatively, it can be implemented by constructing a network that corresponds to conventional deconvolution processing that estimates non-blind distance images or focus blur kernels. The following describes each of these processes.

[0086] First, we will explain the implementation method using an encoder-decoder structure with end-to-end processing. Figure 16 shows the input-output relationship of the network used for focus blur recovery processing. Figure 16(a) is an example of the input and output of a network that performs focus blur recovery end-to-end. Regarding this approach, multiple network implementations have been proposed in the form of Non-Patent Document 2. Note that any approach is acceptable as long as a focus blur recovery image can be obtained by taking the disparity image pair (image A, image B) of each shot as input. The encoder-decoder network is trained using all-focus images taken with the aperture of the imaging device stopped down as training data. The network is trained using the difference between the output image and the ground truth all-focus image as the loss. Once training is complete, a single viewing image with focus blur recovery performed is obtained by taking the disparity image pair of each shot as input.

[0087] Next, we will describe an example of a method for constructing a network that corresponds to deconvolution processing performed by estimating a distance image or a focus blur kernel. An example of the network is shown in Figure 16(b). The network consists of a network that takes a pair of disparity images from each shot as input and calculates a hypothetical distance image or a hypothetical inverse depth image where the distance value d is 1 / d, and a network that takes the pair of disparity images and this hypothetical distance image or hypothetical inverse depth image as input and outputs a focus blur recovery image and a refined distance image or inverse depth image. The hypothetical distance image or hypothetical inverse depth image can be a parameter map representing the focus blur kernel or an image of it. Note that Figure 16(b) is an example where there are two disparity images (image A and image B) for each shot. The distance calculation network and the focus blur recovery network can be trained independently. The network is trained using the ground truth images of distance values ​​corresponding to the pairs of disparity images and all-focus images taken with the aperture of the imaging device stopped down as training data. This training is performed by backpropagation using the error function of the distance value error, pixel value error, edge sharpness, etc. Such a network is proposed as disclosed in Non-Patent Document 3. Details of the network, training data, and examples of error functions can be implemented by referring to Non-Patent Document 3.

[0088] While the distance calculation network and focus blur recovery network are illustrated with reference to the network configuration described in Non-Patent Document 3, the present invention is not limited to these specific network configurations. They can each be replaced with other deep learning networks or classical methods. For example, the distance calculation network may be replaced with a deep learning network that estimates the model parameters of the focus blur kernel disclosed in Non-Patent Document 4, as shown in Figure 16(c). The focus blur recovery network can also be replaced with a classical shift variant type deconvolution process.

[0089] Non-patent document 4 describes a network that assumes a model of a focus blur kernel, calculates the parameters of the focus blur kernel for each field of view for each pair of disparity images input to create a disparity map, and outputs it as a substitute for a depth image.

[0090] In S304, a single-shot distance image is generated. This process is the same as the process in S102 in Figure 3 of the first embodiment, so the explanation is omitted.

[0091] In S305, a multi-view distance image is generated using multiple viewing images acquired by multi-shot, which include both viewing images with and without focus blur recovery from S303. The method for generating the multi-view distance image is the same as the process in S103 described in the first embodiment, so the explanation is omitted.

[0092] In S306, the single-shot distance image and the multi-view distance image are integrated to generate a highly accurate integrated distance image for the entire depth direction of the scene. The method for generating the integrated distance image is the same as the process in S104 described in the first embodiment, so the explanation is omitted.

[0093] Next, with reference to Figure 17, the procedure from shooting to generating the integrated distance image described above will be outlined. The case where all images are acquired as parallax image pairs using multi-shots will be explained. For example, assuming a still camera is used for shooting, in S301, the photographer may intentionally take a picture by fully pressing the shutter. Then, in S302, a reference parallax image pair (reference shot) is selected from among the multiple parallax image pairs acquired by the multi-shot process. The selected parallax image pair may be the same as the parallax image pair used to generate the single-shot distance image. In S303, focus blur recovery processing is performed on the parallax image pair used to generate the viewing image for generating the multi-view distance image.

[0094] In S304, a single-shot distance image is generated using the selected pair of parallax images. In S305, a multi-view distance image is generated from multiple viewing images obtained by performing focus blur recovery processing. In S306, the single-shot distance image and the multi-view distance image are integrated to obtain an integrated distance image. The resulting integrated distance image is a distance image with higher distance accuracy than the single-shot distance image for the entire scene, including areas outside the predetermined distance range from the focus position. Also, unlike the multi-view distance image, distance values ​​can be obtained even if moving objects are included.

[0095] Figure 18 illustrates the case where you want to generate an integrated distance image for all shots obtained by a multi-shot method, or when you want to generate an integrated distance image corresponding to all frames in video capture. In this case, by sequentially changing the selection of the reference shot in S302, an integrated distance image corresponding to all shots is generated.

[0096] As described above, according to the third embodiment, highly accurate distance images can be obtained in various shooting scenes.

[0097] <Fourth Embodiment> Next, a fourth embodiment of the present invention will be described. Figure 19A is a block diagram showing the configuration of a fourth embodiment of the imaging device of the present invention. In Figure 19A, the imaging device 400 includes an imaging unit 401, an imaging unit 402, a calculation unit 403, a storage unit 404, a shutter button 405, an operation unit 406, a display unit 407, and a control unit 408.

[0098] Figure 19B is an external front view of the imaging device 400 of this embodiment. The imaging device 400 has two imaging units: an imaging unit 401 and an imaging unit 402. The imaging unit 401 includes a lens 401a and an image sensor 401b. The imaging unit 402 includes a lens 402a and an image sensor 402b. The lenses 401a and 402a collect light reflected from the subject and form an optical image on the image sensors 401b and 402b. The image sensors 401b and 402b convert the optical image into an electrical signal and output video data.

[0099] The imaging units 401 and 402 are arranged to photograph a common subject and acquire images with parallax between them. When the user of the imaging device 400 presses the shutter button 405, the imaging units 401 and 402 perform compound stereo imaging. The distance D between the optical axes of imaging unit 401 and imaging unit 402 is assumed to be known.

[0100] Figure 19C is an external rear view of the imaging device 400 of this embodiment. The operation unit 406 is pressed to set shooting conditions or to give instructions such as starting or ending the distance measurement mode. The display unit 407 is, for example, a liquid crystal display, and displays the composition at the time of shooting or various setting items. If the display unit 407 is a touch panel, it can also serve as the shutter button 405 and the operation unit 406, allowing the user to take pictures or make settings by touching the display unit 407. In this case, there is no need to equip hardware parts such as the shutter button 405 and the operation unit 406, making it easier for the user to operate, and a larger display unit 407 can be installed for better visibility.

[0101] The control unit 408 controls the shooting conditions when the imaging unit 401 or imaging unit 402 takes a picture. For example, it controls the aperture diameter of the optical system, the shutter speed, and the ISO sensitivity. The calculation unit 403 performs development processing on the images taken by the imaging unit 401 or imaging unit 402 and calculates the subject distance.

[0102] Figure 20 is a flowchart showing the operation of the distance measurement process in the imaging device 400 of this embodiment. When the user instructs to start the distance measurement mode in S401, the operation shown in this flowchart begins.

[0103] Figure 21(a)A is a flowchart detailing the processing in S401. S401 consists of the processes in S4011 and S4012.

[0104] In S4011, the user selects the distance measurement mode. Figure 22 shows the state of the display unit 407 when the distance measurement mode is selected. The user operates the control unit 406 to display display 501 on the display unit 407. Display 501 shows the shooting mode item, and the user operates the control unit 406 to select the distance measurement mode. In S4012, the control unit 408 of the imaging device 400 starts continuous shooting.

[0105] Returning to Figure 20, in S402, relative distance is acquired using a monocular stereo camera. For shooting with the monocular stereo camera, either the imaging unit 401 or the imaging unit 402 can be used. In this embodiment, it will be explained as shooting using the imaging unit 401. Continuous shooting may have a close frame interval, like video recording, or a sparse frame interval, like continuous shooting of still images.

[0106] Figure 21(b) is a flowchart showing the detailed processing in S402. S402 consists of processes S4021 to S4027.

[0107] S4021 is a loop of frames captured by a monocular stereo camera. Acquiring relative distance with a monocular stereo camera involves first obtaining the 3D relative position of the subject, and then using this to convert it into a relative distance value from the position where the user wants to obtain an absolute distance value. Methods such as SfM (Structure from Motion) and SLAM (Simultaneous Localization And Mapping) can be used to acquire the 3D relative position of the subject. Alternatively, a method called MVS (Multi View Stereo) can be used, where the position and orientation of the imaging device 400 are calculated using SfM or SLAM, and then this position and orientation are used to acquire a dense 3D relative position. The following explanation assumes the use of a method that simultaneously acquires the position and orientation of the imaging device 400 and the 3D relative position of the subject, similar to SfM or SLAM.

[0108] Figure 23 shows the state of the display unit 407 during relative distance measurement. Display 601 displays the subject to be measured relative distance in real time. By taking a picture while viewing this image, the user can perform relative distance measurement including the subject they wish to measure the distance to. Notification 602 is a display that notifies the user that relative distance measurement is being performed by monocular stereo shooting, and is displayed superimposed on display 601. Notification 602 can also be done by an alarm sound or voice guidance, in which case display 601 can be displayed without any loss of information.

[0109] In S4022, the control unit 408 extracts feature points from the image of the current captured frame. Typical methods for extracting feature points include SIFT (Scale Invariant Feature Transform), SURF (Speeded-Up Robust Features), and FAST (Features from Accelerated Segment Test), but other methods may also be used.

[0110] In S4023, the control unit 408 determines whether the current frame is the first frame or not. If it is the first frame, it proceeds to the second frame without performing any further processing; otherwise, it proceeds to S4024.

[0111] In S4024, the control unit 408 associates the feature points extracted in S4022 in the previous frame with the feature points extracted in S4022 in the current frame. When the frame interval is sparse, it is necessary that the common parts of the subject to be measured for distance are sufficiently captured in the images between different frames. If the common parts are insufficient, it will become impossible to associate the feature points, and the calculation of relative distance acquisition will stop.

[0112] Figure 24 shows the correspondence of feature points during relative distance measurement using a monocular stereo camera. Positions 701 and 702 indicate the position of the imaging device 400 at different times. Images 703 and 704 show images taken by the imaging unit 401 at position 701 and position 702, respectively. Feature point 705 is a feature point extracted in image 703 and is correctly associated as feature point 706 in image 704. In contrast, feature point 707 was not extracted at the correct position in image 704 and is therefore incorrectly associated as feature point 708 in image 704.

[0113] If we define the reliability of relative distance values ​​obtained by a monocular stereo camera as the matching accuracy of feature points, then feature point 707 is a feature point with poor matching accuracy, and therefore the reliability of the relative distance value is relatively low. Possible reasons for this decrease in reliability include the lack of texture in the subject and the blurring of the image at the location corresponding to feature point 707 in image 704.

[0114] In step S4025, the control unit 408 calculates the position and orientation of the imaging device 400 and the three-dimensional relative position of the subject.

[0115] In S4026, the control unit 408 calculates the reliability of the relative distance values. It calculates the matching accuracy of the feature points mapped in S4024, and points with higher matching accuracy are considered to have higher reliability. For calculating the matching accuracy, algorithms such as RANSAC (Random Sample Consensus) can be used. RANSAC calculates how much the movement of the feature point of interest deviates from the position and orientation of the imaging device 400 and the average value of the movement of many feature points on the image, and determines the matching accuracy of the feature point.

[0116] In S4027, the control unit 408 determines whether the user has instructed the system to take a photograph using the compound eye stereo camera. If the user has not instructed the system to take a photograph, the system proceeds to the next frame. If the user has instructed the system to take a photograph using the compound eye stereo camera, the system terminates this process and proceeds to S403 in Figure 20. Here, the instruction to take a photograph using the compound eye stereo camera refers to the user instructing the system to begin the shooting operation for calculating the absolute distance value, such as by half-pressing the shutter button 405.

[0117] In S403, the control unit 408 takes images with both the compound stereo camera, i.e., the imaging unit 401 and the imaging unit 402, and acquires absolute distance values.

[0118] Figure 25(a) is a flowchart detailing the process for S403. S403 consists of processes S4031 to S4039.

[0119] In S4031, the control unit 408 focuses on a specific subject. A specific subject is, for example, something the user particularly wants to measure the distance to. One method of focusing is to use the autofocus function, which is activated when the user half-presses the shutter button 405. Focusing can also be done manually. In addition, both the imaging unit 401 and the imaging unit 402 focus on the same subject.

[0120] In S4032, the absolute distance and its reliability are calculated using a compound stereo camera. In calculating the absolute distance using the compound stereo camera, techniques such as stereo matching are used on the images captured by imaging unit 401 and imaging unit 402, respectively.

[0121] Figure 26 shows the stereo matching process during absolute distance measurement using a compound-eye stereo camera. When the user focuses at position 801 using S4031, the images captured by imaging unit 401 and imaging unit 402 are designated as image 802 and image 803, respectively. Image 802 is designated as the reference image and image 803 as the reference image, and windows (regions) 804 and 805 are set within image 802 and image 803, respectively. Window 804 is fixed, and window 805 is moved. By performing a correlation calculation on the images within the windows, the position of window 805 with the highest correlation value is designated as the corresponding position for window 804. The amount of displacement on the image at the corresponding position is defined as parallax, and the absolute distance to the subject captured in window 804 is calculated from the parallax value and the baseline length based on the distance D between the optical axes of imaging unit 401 and imaging unit 402. At this time, the confidence level for the calculated absolute distance is calculated. For example, by changing the position of window 804 in various ways, the one with the highest correlation value obtained from the window matching calculation for each position can be considered to have high confidence. Alternatively, as will be described later, confidence can be expressed in terms of the amount of defocus relative to the subject. Here, the window matching method has been explained, but it is not limited to this, and feature points can be extracted and matched, similar to the relative distance calculation using a monocular stereo camera.

[0122] In S4033, the control unit 408 determines whether there is a nearly identical subject with high confidence in both the relative distance value acquired by the monocular stereo camera and the absolute distance value acquired by the compound stereo camera. If there is a nearly identical subject with high confidence, the process proceeds to S4037; otherwise, it proceeds to S4034.

[0123] In S4034, feature points near the focus point are used to convert the relative distance acquired by the monocular stereo camera into an absolute distance. In the calculation of reliability for both relative and absolute distance, there may not always be a subject with high reliability near the focus point, but here, it is sufficient if the reliability is high enough to obtain an approximate value of the depth of field.

[0124] Next, the confidence level of the absolute distance acquired by the compound stereo camera is defined using the amount of defocus. For example, the confidence level is determined to be the reciprocal of the magnitude of the defocus amount, such that the smaller the defocus amount, the higher the confidence level. This allows the window 804 to be set at various positions to determine the position with high confidence. If a window 804 is set that includes candidate points that could be feature point X among the feature points acquired by the monocular stereo camera, the confidence level for those points can also be determined from the amount of defocus. In photography using a compound stereo camera, if the absolute distance value of a subject where the amount of defocus is within the depth of field is considered to have high confidence, then any subject included in window 804 that is within the depth of field will be a candidate point for feature point X.

[0125] In S4035, the control unit 408 sets the aperture to a small value so that the candidate points of feature point X are within the depth of field. This process allows the candidate points of feature point X to be photographed without blurring, and the absolute distance value can be obtained with high accuracy. When converting the relative distance value to an absolute distance value in S4034, it is not always the case that highly reliable feature points are used, so it is good to set an aperture smaller than the aperture size that brings the image within the depth of field. Here, it is not necessary to set the aperture size; the aperture size can remain unchanged, and the focus can be readjusted to a position where the candidate points of feature point X are within the depth of field. In this way, even if the shooting scene is too dark to set a small aperture, the candidate points of feature point X can be photographed without blurring.

[0126] In S4036, the control unit 408 calculates the confidence level of the absolute distance value for candidate points of feature point X.

[0127] In S4037, feature point X is determined from candidate points for feature point X. The determination method may be, for example, the position with the highest overall ranking of confidence levels calculated by a monocular stereo camera and confidence levels obtained by a bicular stereo camera.

[0128] In the S4038, the user takes a picture by pressing the shutter button 405.

[0129] In S4039, the absolute distance value of feature point X is obtained from the two images captured by the compound eye stereo camera. The method for obtaining the absolute distance value is the same as described above, but when calculating by window matching, you should set the smallest possible window that includes feature point X. When calculating by feature point matching, you simply match feature point X in the two images. From S4039, return to S404 in Figure 20.

[0130] In S404, the relative distance values ​​acquired by the monocular stereo camera are converted to absolute distance values. Figure 25(b) is a flowchart detailing the process for S404. S404 consists of processes S4041 to S4042.

[0131] S4041 calculates a conversion formula (conversion relationship) that converts relative distance values ​​to absolute distance values ​​for all subjects using the relative distance value and absolute distance value for a feature point X. For example, if the relative distance value of feature point X is zr and the absolute distance value is Za[m], then Z = (Za / zr) × z. Here, z is the relative distance value of a certain subject, and Z is the absolute distance value relative to it.

[0132] In S4042, a conversion formula is used to convert relative distance values ​​acquired by the monocular stereo camera into absolute distance values. By limiting the conversion range to the range of the composition captured in S4038, the absolute distance values ​​of the composition desired by the user can be obtained with high accuracy.

[0133] By using the method of this embodiment, a conversion formula from relative distance to absolute distance can be determined at an image position where both the relative distance and absolute distance have high reliability. Then, by applying this conversion formula to other images, highly accurate absolute distance values ​​can be obtained for all subjects.

[0134] <Fifth Embodiment> Next, a fifth embodiment of the present invention will be described. In the fourth embodiment described above, the imaging device was configured as a two-lens stereo camera. However, by using a pupil-splitting type image sensor and having only one of the imaging units, either the imaging unit 401 or the imaging unit 402, the configuration can be simplified by having only one imaging unit.

[0135] Figure 27 shows the structure of a pupil-splitting image sensor and the principle of distance calculation. Figure 27(a) shows the state where the subject is in focus, and Figure 27(b) shows the state where the subject is located in front of the point of focus.

[0136] The image sensor 901 has a pupil-splitting structure, and the inside of the pixel 903 is divided into two sub-pixels 904 and 905. Of the light reflected from the subject, one beam of light passes through the end of the imaging optical system 902 and is received by the sub-pixel 904, and the other beam of light passes through the opposite end of the imaging optical system 902 and is received by the sub-pixel 905. Images 906 and 908 are images generated from the light received by the sub-pixel 904, and images 907 and 909 are images generated from the light received by the sub-pixel 905.

[0137] As shown in Figure 27(a), light from a subject at the focal point is received by subpixels 904 and 905 within the same pixel, so there is no parallax between the subject in image 906 and image 907. In contrast, as shown in Figure 27(b), light from a subject that is out of focus is received by subpixels 904 and 905 in different pixels, so there is parallax between the subject in image 908 and image 909. The absolute distance can be calculated from this parallax.

[0138] When using a pupil-splitting image sensor, the camera becomes a compound eye sharing a lens, and the baseline length of the compound eye stereo camera is determined by the aperture size. Therefore, if the aperture is reduced in an attempt to keep candidate points for feature point X within the depth of field, the baseline length becomes shorter, making it impossible to obtain highly accurate absolute distance values. In this case, the reliability of the absolute distance values ​​obtained by the pupil-splitting image sensor should be considered in relation to both the degree of defocus and the baseline length. Alternatively, as explained above, the aperture can be kept fixed, and the focus can be readjusted to a position that increases the reliability of candidate points for feature point X.

[0139] The above explanation describes a method for calculating highly accurate absolute distance values ​​by combining relative distance values ​​obtained with a monocular stereo camera and absolute distance values ​​obtained with a compound stereo camera. However, it is also possible to combine the distance values ​​from both cameras.

[0140] For example, if you want to obtain the absolute distance value of a scene that includes a moving subject, a monocular stereo camera cannot obtain the relative distance value of the moving subject. In this case, it is best to use the absolute distance value of the moving subject obtained with a compound stereo camera.

[0141] In monocular stereo cameras, corresponding points in images are calculated using images from frames taken at different times, so in images containing moving subjects, corresponding points may not be found. Also, it may be difficult to distinguish between the movement of a moving subject and changes in the camera's position and orientation. In contrast, with a compound stereo camera, the time of capture can be matched, so moving subjects can be treated the same as stationary subjects between two images, and absolute distance values ​​can be calculated. When creating an absolute distance image represented by grayscale using this, for stationary subjects, a distance image is used which is obtained by converting the relative distance value acquired by the monocular stereo camera into an absolute distance value using feature points X extracted from the stationary subject. For moving subjects, the absolute distance value acquired by the compound stereo camera is used. Whether or not a subject is moving can be determined using machine learning, or a subject that is not in the relative distance image acquired by the monocular stereo camera but is in the absolute distance image acquired by the compound stereo camera may be considered a moving subject, or a combination of both may be used.

[0142] <Other Embodiments> Furthermore, the present invention may be applied to a system consisting of multiple devices or to a device consisting of a single device.

[0143] Furthermore, the present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0144] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0145] 100: Imaging device, 101: Imaging unit, 102: Memory, 103, 303: Viewing image generation unit, 104: Single-shot distance image generation unit, 105: Multi-view distance image generation unit, 106: Distance image integration unit, 1011: Optical system, 1012: Image sensor, 204: Processing unit, 400: Imaging device, 401, 402: Imaging unit, 403: Processing unit, 404: Memory unit, 405: Shutter button, 406: Operation unit, 407: Display unit, 408: Control unit

Claims

1. An acquisition means for acquiring multiple images from different viewpoints of the same scene taken from different positions, and at least one pair of disparity images having parallax due to pupil division. A first generation means for generating a first distance image from the pair of disparity images, A second generation means for generating a second distance image from the aforementioned multiple images from different viewpoints, The system includes an integration means for integrating the first distance image and the second distance image to generate an integrated distance image, The integration means is an image processing device characterized in that, in integrating the first distance image and the second distance image, it supplements the distance information of the moving subject region within the distance information constituting the second distance image with the distance information of the same region in the first distance image.

2. The image processing apparatus according to claim 1, characterized in that the moving subject region is distance information of a region in the second distance image where distance information has not been calculated or where distance information exists but is of low reliability.

3. The image processing apparatus according to claim 1 or 2, characterized in that the second generation means generates the second distance image based on the parallax between the plurality of different viewpoint images.

4. The image processing apparatus according to claim 3, characterized in that the second generation means generates the second distance image from the plurality of different viewpoint images using epipolar geometry.

5. The acquisition means receives multiple pairs of disparity images taken of the same scene from different positions, The image processing apparatus according to any one of claims 1 to 4, further comprising a third generation means for generating the plurality of different viewpoint images from the plurality of pairs of disparity images.

6. The image processing apparatus according to claim 5, characterized in that the third generation means generates the plurality of different viewpoint images by using one of the plurality of disparity image pairs as a reference disparity image pair and adding the plurality of disparity image pairs excluding the reference disparity image pair pixel by pixel.

7. The image processing apparatus according to claim 5, characterized in that the third generation means generates the plurality of different viewpoint images by leaving one of the plurality of disparity image pairs as a reference disparity image pair and adding the plurality of disparity image pairs, including the reference disparity image pair, pixel by pixel.

8. The third generation means generates multiple different viewpoint images by sequentially changing the reference disparity image pair among the plurality of disparity image pairs. The integration means integrates a first distance image generated from sequentially shifted reference disparity image pairs with a second distance image generated from the corresponding plurality of different viewpoint images to generate a plurality of integrated distance images. The image processing apparatus according to claim 6 or 7.

9. The image processing apparatus according to any one of claims 5 to 8, characterized in that the third generation means performs a focus blur recovery process on the plurality of disparity image pairs and generates the plurality of different viewpoint images from the plurality of disparity image pairs that have undergone the focus blur recovery process.

10. The image processing apparatus according to claim 9, characterized in that the focus blur recovery process includes a deconvolution process or MAP estimation process performed by estimating a focus blur kernel, and a deep learning process by end-to-end processing using an encoder-decoder structure.

11. The acquisition means receives multiple pairs of disparity images taken of the same scene from different positions, The first generation means generates a first distance image for each of a predetermined number of pairs of disparity images from among the plurality of pairs of disparity images, and uses one of the generated first distance images as a reference first distance image, and integrates the other first distance images by performing a viewpoint transformation. The image processing apparatus according to any one of claims 1 to 4, characterized in that the integration means integrates the integrated first distance image and the second distance image.

12. The image processing apparatus according to any one of claims 1 to 11, characterized in that the acquisition means is an imaging means for photographing a subject.

13. The image processing apparatus according to claim 12, characterized in that the imaging means captures the plurality of different viewpoint images and then captures the at least one pair of disparity images.

14. The image processing apparatus according to claim 12, characterized in that the imaging means captures the plurality of different viewpoint images after capturing the at least one pair of disparity image pairs.

15. The image processing apparatus according to claim 12, characterized in that the imaging means captures the plurality of different viewpoint images before and after capturing the at least pair of disparity image pairs.

16. The image processing apparatus according to any one of claims 12 to 15, characterized in that the imaging means uses a smaller aperture when capturing the plurality of different viewpoint images than when capturing the at least one pair of disparity images.

17. The image processing apparatus according to any one of claims 12 to 15, characterized in that the imaging means sets shooting conditions when capturing the at least pair of disparity image pairs and when capturing the plurality of different viewpoint images, respectively.

18. The acquisition means includes an acquisition step of acquiring multiple images from different viewpoints of the same scene taken from different positions, and at least one pair of disparity images having parallax due to pupil division, The first generation means includes a first generation step of generating a first distance image from the pair of disparity images, The second generation means includes a second generation step of generating a second distance image from the plurality of different viewpoint images, The integration means includes an integration step of integrating the first distance image and the second distance image to generate an integrated distance image, The image processing method is characterized in that, in the integration step of the first distance image and the second distance image, the distance information of the moving subject region among the distance information constituting the second distance image is supplemented with the distance information of the same region in the first distance image.

19. A program for causing a computer to function as one of the means of an image processing apparatus according to any one of claims 1 to 11.

20. A computer-readable storage medium storing the program described in claim 19.

Images