Image processing device and method, camera device and control method thereof, and storage medium

Abstract

The present invention provides an image processing device and method, a camera device and a control method thereof, and a storage medium. The image processing device includes: an acquisition component configured to acquire multiple different viewpoint images obtained by photographing the same scene from different viewpoints, and configured to acquire at least one parallax image pair having parallax through pupil division; a first generation component configured to generate a first distance image from the parallax image pair; a second generation component configured to generate a second distance image from the multiple different viewpoint images; and an integration component configured to integrate the first distance image and the second distance image and generate an integrated distance image.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus and method, an imaging apparatus and a control method thereof, and a storage medium, and more particularly to a technique for generating a distance image from an image obtained from the imaging apparatus. Background Art

[0002] Conventionally, there is known a technique for obtaining a distance image showing the distance to each subject included in an image by using a pair of pupil-divided images (Japanese Patent 5192096). This technique has an advantage that a distance image of a scene including not only a subject that does not move (hereinafter referred to as a "still subject") but also a subject that moves and is the main subject in many shooting scenes (hereinafter referred to as a "moving subject") can be obtained by performing one shooting.

[0003] In addition, a compound eye stereo camera has been used for a long time as a camera for distance measurement. The compound eye stereo camera uses a plurality of cameras whose positional relationships between the cameras are known, and calculates the distance to the subject based on the parallax of the subject that is commonly present in the captured image and the baseline length between the cameras. However, in a compound eye camera with a short baseline length, if the subject is present at a long distance, the parallax of the subject in the captured image is small, and the distance measurement accuracy becomes low.

[0004] On the other hand, in a monocular stereo camera, one camera captures a plurality of images while changing its position and posture, and the relative distance to the subject is calculated based on the parallax of the same subject in these images. Since a monocular stereo camera can adopt a long baseline length, the relative distance can be accurately calculated even for a subject at a long distance, and the distance measurement accuracy can be improved by using a large number of images of tens to hundreds.

[0005] However, in a monocular stereo camera, since the distance to the subject and the position and posture of the camera are estimated simultaneously, the distance measurement value is not uniquely determined, and the obtained distance measurement value is not an absolute distance value but a relative distance value.

[0006] Here, the relative distance value is a dimensionless quantity and is defined, for example, as a ratio of the distance to the object of interest when the distance to a specific object is defined as 1. On the other hand, the absolute distance value has a dimension of length and is a value such as 3 [m].

[0007] Therefore, in a distance measuring device equipped with both a monocular stereo camera and a compound eye stereo camera, there is known a technique for converting a relative distance value acquired by a monocular stereo camera into an absolute distance value using information obtained by the compound eye stereo camera. For example, in Japanese Patent Laid-Open No. 2011-112507, an absolute distance value to a subject is obtained using a compound eye stereo camera, an absolute position and posture of the camera are calculated based on the absolute distance value, and the absolute distance values ​​to all subjects are calculated using the absolute position and posture of the camera.

[0008] However, the above-mentioned prior art disclosed in Japanese Patent 5192096 has the following problems. That is, since the baseline length between the main pixel and the sub-pixel used to obtain a pair of pupil-divided images is short, if the subject in the image is not at a short distance relative to the camera device and if the subject is outside the predetermined range relative to the focus distance, the distance to the subject cannot be accurately calculated. On the other hand, if the structure shown in Japanese Patent 5192096 is used to widen the pupil to extend the baseline length so that the distance to the subject can be measured with better accuracy, the depth of field becomes shallower instead, and the range of distance that can be calculated becomes narrower. As described above, in the case where the depth of the shooting scene is deep, it is difficult to obtain an accurate distance image.

[0009] Furthermore, according to the method described in Japanese Patent Laid-Open No. 2011-112507, if the absolute distance to the subject acquired by the compound-eye stereo camera is inaccurate, the absolute position and orientation of the camera cannot be accurately calculated. As a result, there is a problem that the absolute distance values ​​to all subjects become inaccurate. Summary of the invention

[0010] The present invention has been made in view of the above-described circumstances, and acquires a range image of the entire scene including a moving subject with high accuracy.

[0011] Furthermore, according to another configuration of the present invention, the relative distance value acquired by the monocular camera is accurately converted into an absolute distance value.

[0012] According to the present invention, there is provided an image processing device, comprising: an acquisition component configured to acquire a plurality of different viewpoint images obtained by photographing the same scene from different viewpoints, and configured to acquire at least one parallax image pair having parallax through pupil division; a first generating component configured to generate a first distance image from the parallax image pair; a second generating component configured to generate a second distance image from the plurality of different viewpoint images; and an integrating component configured to integrate the first distance image and the second distance image and generate an integrated distance image.

[0013] Furthermore, according to the present invention, there is provided an imaging device including: the image processing device as described above; and an image sensor as at least a part of the acquisition means for capturing the at least one parallax image pair.

[0014] In addition, according to the present invention, there is provided an image processing method, comprising: an acquisition step for acquiring a plurality of different viewpoint images obtained by photographing the same scene from different viewpoints, and acquiring at least one parallax image pair having parallax through pupil division; a first generation step for generating a first distance image from the parallax image pair; a second generation step for generating a second distance image from the plurality of different viewpoint images; and an integration step for integrating the first distance image and the second distance image to generate an integrated distance image.

[0015] Furthermore, according to the present invention, there is provided a computer-readable storage medium for storing a program for causing a computer to function as the image processing apparatus as described above.

[0016] In addition, according to the present invention, there is provided an imaging device, comprising: at least two imaging components, wherein the distance between the optical axes of the imaging components is known; and a calculation component, which is configured to calculate a plurality of absolute distance values ​​to a subject and the reliability of the plurality of absolute distance values ​​based on a plurality of first images simultaneously captured by the imaging components and having parallaxes with each other, and to calculate a plurality of relative distance values ​​to the subject and the reliability of the plurality of relative distance values ​​based on a plurality of second images captured by the imaging components at different times and having parallaxes with each other, wherein the calculation component uses the absolute distance values ​​and the relative distance values, which have relatively high reliability and correspond to substantially the same area of ​​the subject, among the plurality of absolute distance values ​​and the plurality of relative distance values, to determine a conversion relationship between the absolute distance values ​​and the relative distance values.

[0017] In addition, according to the present invention, there is provided a control method for an imaging device, wherein the imaging device has at least two imaging components, and the distance between the optical axes of the imaging components is known, and the control method comprises: a calculation step for calculating a plurality of absolute distance values ​​to a photographed object and the reliability of the plurality of absolute distance values ​​based on a plurality of first images simultaneously captured by the imaging components and having parallax with each other, and calculating a plurality of relative distance values ​​to the photographed object and the reliability of the plurality of relative distance values ​​based on a plurality of second images captured by the imaging components at different times and having parallax with each other, wherein in the calculation step, the absolute distance values ​​and the relative distance values ​​corresponding to substantially the same areas as the photographed object among the plurality of absolute distance values ​​and the plurality of relative distance values ​​having relatively high reliability are used to determine a conversion relationship between the absolute distance values ​​and the relative distance values.

[0018] Furthermore, according to the present invention, there is provided a computer-readable storage medium for storing a program for causing a computer to execute the control method as described above.

[0019] Further features of the present invention will become apparent from the following description of exemplary embodiments (with reference to the attached drawings). BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0021] Figure 1 is a block diagram showing a schematic structure of an image pickup apparatus according to a first embodiment of the present invention.

[0022] FIG. 2A to FIG. 2E is a diagram for explaining an image sensing unit according to the first embodiment.

[0023] Figure 3 : is a flowchart showing the process of image capturing and integrated range image generation according to the first embodiment.

[0024] Figure 4 is a flowchart of the single viewpoint distance image generation process according to the first embodiment.

[0025] Figure 5 This is a diagram illustrating a method of converting a defocus amount into a distance value.

[0026] Fig. 6A and Figure 6B The diagram exemplifies a method of generating a multi-viewpoint range image.

[0027] 7A to 7D is a diagram showing an example for integrating a single-viewpoint distance image and a multi-viewpoint distance image according to the first embodiment.

[0028] Figure 8 is a diagram showing another example for combining a single-viewpoint distance image and a multi-viewpoint distance image according to the first embodiment.

[0029] 9A to 9C is a diagram illustrating an example of changes in photographing timing according to the first embodiment.

[0030] Fig. 10A and Fig. 10B is a diagram illustrating an example of changing the shooting conditions according to the first embodiment.

[0031] Fig.11 is a block diagram showing a schematic structure of an image processing apparatus according to a second embodiment.

[0032] Fig. 12Aand Fig. 12B : is a diagram for explaining selection of a shot for generating a single viewpoint range image during post-processing according to the second embodiment.

[0033] Fig.13 is a diagram showing an outline of a process for generating an integrated distance image when a plurality of single-viewpoint distance images are integrated.

[0034] Fig.14 is a block diagram showing a schematic structure of an image processing apparatus according to a third embodiment.

[0035] Fig.15 : is a flowchart showing the process of image capturing and integrated range image generation according to the third embodiment.

[0036] FIG. 16A to FIG. 16C is a diagram showing an input / output relationship of a network used for defocus deblurring according to the third embodiment.

[0037] Fig.17 is a diagram showing an outline of a process from photographing to integrated range image generation according to the third embodiment.

[0038] Fig.18 : is a diagram showing an outline of a process from acquisition of a plurality of frame images in moving image capturing to generation of an integrated distance image according to the third embodiment.

[0039] Fig.19A is a block diagram showing the configuration of an image pickup apparatus according to a fourth embodiment.

[0040] Fig.19B is a front view of the appearance of an image pickup apparatus according to a fourth embodiment.

[0041] Fig.19C is a rear view of the external appearance of an image pickup apparatus according to the fourth embodiment.

[0042] Fig. 20 is a flowchart showing the operation of the distance measurement process according to the fourth embodiment.

[0043] Fig.21A and Fig. 21B It is shown separately Fig. 20 Flowchart of detailed processing of steps S401 and S402.

[0044] Fig. 22 It is a diagram showing a mode selection screen of the display unit.

[0045] Fig.23 is a diagram showing a screen of a display unit during which distance measurement is in progress.

[0046] Fig.24 This is a diagram showing a state of feature point tracking using a monocular stereo camera.

[0047] Fig.25A and Fig.25B It is shown separately Fig. 20 Flowchart of detailed processing of steps S403 and S404.

[0048] Fig.26 This is a diagram showing a state of window matching used in a compound eye stereo camera.

[0049] Fig.27A and Fig.27B : is a diagram showing the structure of a pupil division image sensor and the principle of distance calculation. DETAILED DESCRIPTION

[0050] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claimed invention, and are not intended to limit the invention of the combination of all features that need to be described in the embodiments. Two or more than two of the multiple features described in the embodiments may be appropriately combined. In addition, the same reference numerals are given to the same or similar structures, and their redundant descriptions are omitted. The various embodiments of the present invention described below may be implemented separately, or as a combination of multiple embodiments or features of these embodiments when necessary or when it is beneficial to combine elements or features from the various embodiments in a single embodiment.

[0051] <First Embodiment>

[0052] Figure 1 1 is a block diagram showing a schematic structure of an image pickup apparatus 100 according to a first embodiment of the present invention, and only components necessary for the description of the present invention are shown. The image pickup apparatus 100 includes an image sensing unit 101 including an optical system 1011 and an image sensor 1012, a memory 102, a viewing image generation unit 103, a single viewpoint distance image generation unit 104, a multi-viewpoint distance image generation unit 105, and a distance image integration unit 106.

[0053] FIG. 2A to FIG. 2E is a diagram for explaining the image sensing unit 101 .

[0054] exist Figure 2A In FIG. 1 , the optical system 1011 is composed of a plurality of lenses, a reflecting mirror, and the like, and forms an image of reflected light from the subject 10 on the light receiving surface of the image sensor 1012. Figure 2A, the optical system 1011 is shown by a single lens. Reference numeral 202 denotes an optical axis of the optical system 1011. The image sensor 1012 receives an optical image of the subject 10 formed by the optical system 1011, converts the optical image into an electric signal, and outputs the electric signal.

[0055] like Figure 2B As shown, the image sensor 1012 has a large number of pixels 210 arranged on the xy plane, which are arranged in a two-dimensional manner and covered by a so-called Bayer arrangement color filter 222 of red (R), green (G), and blue (B) to be described later. With this arrangement, each pixel of the image sensor 1012 is given a spectral characteristic corresponding to the wavelength band of light passing through the color filter 222, and a signal corresponding mainly to any one of red light, green light, and blue light is output.

[0056] Figure 2C 2 is a cross-sectional view showing the structure of each pixel, and includes a microlens 211, a color filter 222, photoelectric converters 210a and 210b, and an optical waveguide 213. The substrate 224 is made of a material (e.g., silicon (Si)) that absorbs light in a wavelength band that passes through the color filter 222, and the photoelectric converters 210a and 210b are formed in at least a portion of an inner region of the substrate 224 by ion implantation or the like. In addition, each pixel includes wiring (not shown).

[0057] A luminous flux 232a having passed through the first pupil region 231a and a luminous flux 232b having passed through the second pupil region 231b are incident on the photoelectric converters 210a and 210b, respectively, wherein the first pupil region 231a and the second pupil region 231b are different pupil regions of the exit pupil 230 of the optical system 1011. As a result, a first signal and a second signal as pupil division signals can be obtained from each pixel 210. The first signal and the second signal as signals from the photoelectric converters 210a and 210b, respectively, can be read out independently from each pixel 210. Alternatively, after the first signal is read out, a signal obtained by adding the first signal and the second signal can be read out, and the first signal can be subtracted from the added signal to obtain the second signal.

[0058] Figure 2D1 is a diagram showing the arrangement of the photoelectric conversion units 210a and 210b corresponding to each microlens 211 on the image sensor 1012 as seen from the light incident direction along the optical axis, and showing an example in which each pixel is divided horizontally or vertically. The present invention is not limited to this, and pixels divided in the horizontal direction and pixels divided in the vertical direction may be mixed. By arranging in this manner, the defocus amount described later can be accurately obtained not only for a subject whose brightness distribution changes in the horizontal direction, but also for a subject whose brightness distribution changes in the vertical direction. In addition, three or more photoelectric conversion units may be formed for each microlens 211, and Figure 2E A case where one pixel has four photoelectric conversion portions 210 c to 210 f divided in the horizontal direction and the vertical direction is shown as an example.

[0059] The first signal and the second signal obtained from the photoelectric conversion parts 210a and 210b are sent to the arithmetic processing unit 204 included in the image sensing unit 101 and converted into electronic information. In the case where the signal acquired by the photoelectric conversion is an analog signal, the arithmetic processing unit 204 performs basic processing such as noise reduction by correlated double sampling (CDS), exposure control using gain in automatic gain control (AGC), black level correction, and A / D conversion, and obtains an image signal converted into a digital signal. Since the arithmetic processing unit 204 mainly pre-processes the analog signal, these arithmetic processes are also generally referred to as AFE (analog front end) processing. When used in combination with a digital output sensor, the processing may be referred to as DFE (digital front end) processing.

[0060] Then, an image A formed by collecting first signals output from the plurality of pixels 210 of the image sensor 1012 and an image B formed by collecting second signals output from the plurality of pixels 210 are generated. Since the A image and the B image are images having parallax with each other, hereinafter, the two images are each referred to as a “parallax image”, and the A image and the B image are collectively referred to as a “parallax image pair”.

[0061] If the image sensor 1012 is a color sensor, the arithmetic processing unit 204 also performs Bayer array interpolation and the like. Furthermore, in order to improve the quality of the parallax image and the quality of the viewing image to be described later that is output together with the parallax image pair, filtering processing such as low-pass and high-pass filtering processing and sharpening processing may be performed. Furthermore, various processing such as grayscale correction including dynamic range expansion (such as HDR (high dynamic range) processing) and tone correction (such as WB (white balance) correction) may be performed. Note that the processing of the arithmetic processing unit 204 is often integrated with the processing performed in the image sensor 1012 at the chip level or the unit level, and thus the processing is not performed at the chip level or the unit level. Figure 1 Not shown in FIG.

[0062] In this way, by forming a plurality of photoelectric converters under each of the plurality of microlenses 211 formed on the light receiving surface of the image sensor 1012, the plurality of photoelectric converters respectively receive light beams of the subject that have passed through different pupil regions of the optical system 1011. This makes it possible to obtain a parallax image pair through a single shooting operation even if the optical system 1011 has only one aperture.

[0063] The generated parallax image pair is temporarily stored in the memory 102 .

[0064] Back to Figure 1 , the parallax image pair obtained by one shooting operation and stored in the memory 102 is sent to the viewing image generation unit 103, and the first signal and the second signal are added for each pixel to generate one image. That is, an image to be viewed (hereinafter referred to as a "viewing image") is generated for an image formed by a light beam passing through the entire pupil area of ​​the optical system 1011.

[0065] If the focus state is not detected or the distance image is not generated using the parallax image, the viewing image can be generated by using the arithmetic processing unit 204 integrated with the image sensor 1012 at the chip level, or by adding the first signal and the second signal in each pixel and then reading the signal. In the latter case, it can help save the transmission belt and shorten the time required for readout. In addition, if the viewing image that is ultimately paired with the distance image is not required, the viewing image generation unit 103 may not exist explicitly and may be included in the multi-viewpoint distance image generation unit 105.

[0066] The parallax image pair obtained by one shooting operation and stored in the memory 102 is sent to the single viewpoint distance image generation unit 104. At this time, conversion to a brightness image can be performed. Then, the pixels between the parallax images of the input parallax image pair are associated with each other, and a distance image is generated based on image shooting information, which includes camera parameters such as focal length and aperture value defined based on the zoom state of the optical system 1011, and image sensor information such as pixel pitch of the image sensor 1012. A distance image is generated based on the image shooting information. Hereinafter, the distance image generated by the single viewpoint distance image generation unit 104 is referred to as a "single viewpoint distance image".

[0067] The multi-view distance image generation unit 105 takes as input a plurality of viewing images (images seen from different viewpoints), each of which is obtained by converting, using the viewing image generation unit 103, each of a plurality of parallax image pairs obtained by taking a plurality of continuous shots of the same scene from different positions (i.e., acquired by multi-shot), and generates a distance image. Hereinafter, the distance image generated by the multi-view distance image generation unit 105 is referred to as a "multi-view distance image". If the camera is a moving imaging device, there is parallax between the viewing images obtained in time sequence by taking multi-shots. Therefore, if the movement and posture change of the imaging device 100 are known, the multi-view distance image can be calculated based on the parallax of the viewing images (between different viewpoint images).

[0068] In the distance image integration unit 106 , the single viewpoint distance image and the multi-viewpoint distance image are integrated, and an integrated distance image is generated.

[0069] Next, refer to Figure 3 The flowchart of FIG. 1 is used to illustrate the process of image capture and integrated distance image generation in this embodiment.

[0070] In step S101, shooting (single shooting) for generating a single viewpoint distance image and shooting (continuous shooting) for generating a multi-viewpoint distance image are performed. 9A to 9C To illustrate the shooting order.

[0071] In step S102 , the parallax image pair obtained by single shooting is used for distance measurement processing, and a single viewpoint distance image is generated. Figure 4 4 is a flowchart of the single viewpoint distance image generation process performed in step S102. Figure 4 In step S201, an image shift amount as a relative shift amount of the position between the parallax images is calculated. The image shift amount can be calculated using a known method. For example, the following formula (1) is used to calculate a correlation value based on the signal data A(i) and B(i) of the image A and the image B.

[0072]

[0073] In formula (1), S(r) is a correlation value indicating the degree of correlation between two images having an image offset r, i is a pixel number, and r is a relative image offset of the two images. p and q represent target pixel ranges for calculating the correlation value S(r). The image offset can be obtained by finding the image offset r that minimizes the correlation value S(r).

[0074] The method for calculating the image shift amount is not limited to the above method, and other known methods may be used.

[0075] Next, in step S202, the defocus amount is calculated based on the image shift amount calculated in step S201. An optical image of the subject 10 is formed on the image sensor 1012 via the optical system 1011. Figure 2A The example shown shows a defocused state in which the light beams passing through the exit pupil 230 converge on the focal plane 207. The defocused state indicates a state in which the focal plane 207 and the imaging surface (light receiving surface) of the image sensor 1012 do not coincide with each other and are shifted in the direction of the optical axis 202, and the defocus amount is the distance between the imaging surface of the image sensor 1012 and the focal plane 207.

[0076] Here, we will refer to Figure 5 An example of a method of converting a defocus amount into a distance value is explained using a simplified optical layout of an imaging device shown.

[0077] Figure 5 A light ray 232 is shown in a state where the image of the object 10 is defocused relative to the image sensor 1012, wherein: reference numeral 202 denotes an optical axis; reference numeral 208 denotes an aperture stop; reference numeral 205 denotes a front principal point; reference numeral 206 denotes a rear principal point; and reference numeral 207 denotes a focal plane. In addition, d denotes an image shift amount, W denotes a baseline length, D denotes a distance between the image sensor 1012 and an exit pupil 230, Z denotes a distance between the front principal point 205 of the optical system 1011 and the object 10, L denotes a distance between an imaging plane of the image sensor 1012 and the rear principal point 206, and ΔL denotes a defocus amount.

[0078] In the imaging apparatus of the present embodiment, the distance to the subject 10 is detected based on the defocus amount ΔL. The image shift amount d, which represents the relative positional deviation between the image A based on the first signal acquired from the photoelectric conversion section 210a of each pixel 210 and the image B based on the second signal acquired from the photoelectric conversion section 210b, and the defocus amount ΔL have a relationship as shown in equation (2).

[0079]

[0080] By using the scaling factor K, equation (2) can be simplified and written as equation (3).

[0081] ΔL≈K×d...(3)

[0082] The coefficient used to convert the image offset amount into the defocus amount is referred to as the "conversion coefficient" hereinafter. The conversion coefficient refers to, for example, the proportional factor K shown in equation (3), or the baseline length W. In the following description, the correction of the baseline length W is equivalent to the correction of the conversion coefficient. The method for calculating the defocus amount is not limited to the method of this embodiment, and other known methods may be used.

[0083] In addition, the conversion from the defocus amount to the distance to the subject can be performed by using the following formula (4) showing the imaging relationship between the optical system 1011 and the image sensor 1012. Alternatively, the image shift amount can be directly converted to the distance to the subject using a conversion coefficient. In formula (4), f is the focal length.

[0084]

[0085] A defocus map can be generated by obtaining defocus amounts for all pixels between multiple input parallax images (eg, between image A and image B). By converting the defocus map using the relationship of equation (4), a corresponding single viewpoint distance image can be generated.

[0086] Through the procedure of the distance calculation processing as described above, in the pupil division imaging system, a single viewpoint distance image can be calculated from a parallax image pair obtained in one imaging operation.

[0087] Return to Figure 3 In the next step S103, a multi-viewpoint distance image is generated from a plurality of viewing images corresponding to the light that has passed through the entire pupil region and generated by the viewing image generation unit 103 from the parallax image pair obtained in each shot of continuous shooting. It should be noted that, for example, it is assumed that shooting is performed while the imaging apparatus 100 is moved in a state held by a hand, and three-dimensional information of a scene is reconstructed from a plurality of viewing images whose fields of view overlap, and a distance image of each shot is generated during this process.

[0088] The relative position and posture of the camera 100 between shots in continuous shooting can be acquired by a posture sensor such as a gyro sensor, an accelerometer, and a tilt sensor that are standardly attached to the camera 100 in recent years, and by known camera posture estimation using camera shake detection and an image vector, etc. in an image stabilization function that is a standard function of the camera in recent years. Since the method for obtaining the position and posture is known, its description is omitted here.

[0089] If the changes in the position and orientation of the imaging apparatus 100 between shots in continuous shooting are known, then Fig. 6AAs shown, by considering the epipolar geometry constraint, the estimation of the distance to the subject captured in the image can be solved as a simple one-dimensional search problem. In addition, if the points of the subject in the same space can be tracked between multiple viewing images in multiple frames, the epipolar search can be performed as a multi-baseline search for the same distance between multiple different groups of viewing images. By using this, the distance can be obtained with high accuracy. For example, Figure 6B As shown, by increasing the number of shots by 1, the distance image based on the first shot reference viewpoint C1 can be estimated not only between images shot at viewpoints C1 and C2 but also between images shot at viewpoints C1 and C3.

[0090] By increasing the number of related images in this way, multiple searches can be performed for the depth value of each point in the captured image set as the reference image. This enables robust and highly accurate estimation of the distance value. On the other hand, for a moving subject, epipolar geometry does not hold when taking correspondence between consecutively captured images, so the distance to the moving subject cannot be calculated in the multi-view distance image.

[0091] There are various known methods for taking correspondence between images captured continuously. Examples of the method include a method for performing a patch-based mapping called PMVS and calculating a distance value in consideration of the normal direction ("Accurate, dense, and robust multiview stereopsis", IEEE Transactions Pattern Analysis and Machine Intelligence, 32 (8), pp. 1362-1376 by Furukawa, Y. and Ponce, J. 2010.), and a method for setting a virtual depth plane called plane scanning and performing correspondence by back projection from each image. Multi-view distance image generation is also referred to as a multi-view stereo method, etc. By the above method, a multi-view distance image corresponding to a captured viewing image selected as a reference image in a viewing image group captured continuously as a polymerized image of light from the entire pupil area can be obtained.

[0092] In step S104 , the single-viewpoint distance image and the multi-viewpoint distance image are integrated to generate an integrated distance image with high accuracy in the entire range in the depth direction of the scene.

[0093] As described above, the single-view distance image includes distance information of the stationary subject area and the moving subject area, but the multi-view distance image includes only the distance information of the stationary subject area. This is because when generating the multi-view distance image, the mapping of the moving subject area does not satisfy the epipolar geometry. Therefore, in the integration of the distance images in this embodiment, only the distance value of the moving subject area is obtained from the single-view distance image, and the distance value of the stationary subject area overlapping between the single-view distance image and the multi-view distance image is obtained from the multi-view distance image, and the obtained distance values ​​are integrated.

[0094] When the single-view distance image and the multi-view distance image are superimposed, the area where the distance information exists in both the single-view distance image and the multi-view distance image is the stationary subject area, and the area where the distance information exists only in the single-view distance image and the distance information does not exist in the multi-view distance image or the reliability of the distance information (when obtained) is low can be regarded as the moving subject area. Therefore, by acquiring the distance information of the stationary subject area from the multi-view distance image and acquiring the distance information of the moving subject area from the single-view distance image and integrating the two distance information, an integrated distance image with highly accurate distance information can be obtained.

[0095] 7A to 7D An example of integrating a single-viewpoint distance image and a multi-viewpoint distance image is shown. Fig. 7A is the viewing image corresponding to the single viewpoint distance image, Figure 7B is a single viewpoint range image, Figure 7C is the corresponding multi-view range image, and Fig.7D is the integrated range image. Figure 7B In the single-view distance image of Figure 7C In the multi-view distance image of , there is no distance information of the area of ​​the person as the moving subject, so that the area is shown in white. Figure 7B The single viewpoint distance image is used to obtain the distance information of the moving subject area, and Figure 7C The multi-view distance image of the image is used to obtain the distance information of the overlapping static subject area to generate Fig.7D The integrated distance image.

[0096] Furthermore, as described above, as a characteristic peculiar to the pupil division imaging system, since the base line length of the pupil division optical system is short, the distance to the subject which exists at a short distance relative to the imaging device and is outside the predetermined distance range relative to the focusing distance cannot be accurately calculated. In order to solve this problem, the integrated distance image can be generated by obtaining the distance information of a portion of the stationary subject region at a short distance and within the predetermined distance range relative to the focusing distance and the moving subject region from the single viewpoint distance image based on the distance information of the single viewpoint distance image, and obtaining the distance information of the region other than the above region in the stationary subject region from the multi-viewpoint distance image.

[0097] Figure 8 An example is shown of determining from which of the single-viewpoint distance image and the multi-viewpoint distance image to obtain the distance value to be used in the integrated distance image based on the distance value of the single-viewpoint distance image. u0 represents the focus distance, and v1 and v2 represent a predetermined distance range relative to the focus distance u0, where v1 is on the front focus side and v2 is on the rear focus side. In addition, u1 is a distance (threshold value) representing the vicinity range relative to the camera 100. If u is a distance value of each position in the distance image, a distance value (distance information) of an area satisfying at least one of equations (5) and (6) is obtained from the single-viewpoint distance image.

[0098] u≤u1…(5)

[0099] -v1≤u-u0≤v2...(6)

[0100] The distance values ​​of other areas are acquired from the multi-viewpoint distance image. As a result, a distance image with higher accuracy over the entire range in the depth direction of the scene can be acquired compared to a distance image whose distance values ​​are acquired only from a single-viewpoint distance image.

[0101] In the multi-view distance image, the distance information of the moving subject area does not exist as an uncalculated area, and the distance information of the stationary subject area other than the moving subject area exists in both the multi-view distance image and the single-view distance image as the same overlapping area. However, the boundary between the moving subject area and the stationary subject area is often unclear. Therefore, in order to determine the boundary between the area using the distance information obtained from the single-view distance image and the area using the distance information obtained from the multi-view distance image, the moving subject area detection using the viewing image as input can be used in combination. For example, the face and the body can be detected and their areas can be extracted, or a CNN network can be used in which the accuracy for extracting the moving subject area is improved by using prior knowledge by means of learning or the like.

[0102] The distance information (pixel) constituting the distance image is not necessarily limited to the distance to the subject (distance value), and may be the image offset or defocus amount before conversion into the distance to the subject. In addition, the reciprocal of the distance to the subject (reciprocal distance value) may be used as the distance information (pixel) constituting the distance image.

[0103] In addition, in the single shooting and continuous shooting in step S101, the single shooting and continuous shooting can be performed in various ways. That is, the shooting for obtaining the single viewpoint distance image can be performed as one shooting of the continuous shooting or as a single shooting. In addition, the continuous shooting can be performed before or after the single shooting for obtaining the single viewpoint distance image. In addition, if the movement amount of the camera device 100 during the continuous shooting is small and the baseline length is not enough, the shooting can be performed later after a period of time.

[0104] 9A to 9C Shows changes in shooting timing. Fig.9A An example is shown in which continuous shooting is first performed as a pre-shooting to acquire a plurality of images, and then a single shooting is performed to acquire a single viewpoint distance image. 9A to 9C , SW1 indicates a state where the shutter button of the imaging device 100 is half-pressed, and SW2 indicates a state where the shutter button is fully pressed. Note that pre-shooting is a shooting method using a ring buffer, and is a general function installed as a standard not only in an apparatus dedicated to imaging such as a camera, but also in a camera function of a smartphone.

[0105] In this shooting method, an image before the shutter button is fully pressed at SW2 is temporarily stored in advance, and the user can select the real best photo when the shutter button is fully pressed at SW2, or store the image obtained by repeatedly shooting for a certain period of time for post-processing. The pre-shot image is acquired as an input for generating a multi-view distance image. At this time, since the pre-shot image is only used to generate a multi-view distance image, the signals of the photoelectric conversion units 210a and 210b can be added for each pixel at the time of shooting, so that the amount of image data can be reduced and saved as a viewing image. In addition, the image obtained at SW2 when the shutter button is fully pressed is saved as a parallax image pair and used to generate a single view distance image, and these images can be added for each pixel to generate a viewing image and also used to generate a multi-view distance image. In this case, the viewing image is used as a reference image to generate a multi-view distance image.

[0106] Fig. 9BAn example of continuous shooting after an image for generating a single viewpoint distance image is captured is shown. In this shooting method, continuous shooting is performed within a certain period of time after the shutter button of the camera 100 is fully pressed at SW2, and the captured images are saved. The best photo of a moving subject is captured at SW2 when the shutter button is fully pressed. Then, even after the shutter button is released, shooting is continued to obtain images for generating a multi-viewpoint distance image. Since the image obtained in the post-shooting after the shutter button is released is only used to generate a multi-viewpoint distance image, as in the case of pre-shooting, the signals of the photoelectric conversion units 210a and 210b can be added for each pixel at the time of shooting and stored as a viewing image to save storage capacity. However, the image obtained at SW2 when the shutter button is fully pressed is stored as a parallax image pair and used to generate a single viewpoint distance image. In addition, the captured parallax image pair can also be added for each pixel and used to generate a multi-viewpoint distance image. In this case, the multi-viewpoint distance image is generated using the image as a reference image.

[0107] Fig. 9C The case where pre-shooting and post-shooting are performed before and after the shutter button is fully pressed at SW2 is shown. Images shot except when the shutter button is fully pressed at SW2 are acquired from the ring buffer. Since these images are used only to generate the multi-view distance image, the signals of the photoelectric conversion units 210a and 210b can be added for each pixel at the time of shooting and stored as a viewing image to reduce the storage capacity. In addition, the image shot at SW2 when the shutter button is fully pressed is saved as a parallax image pair and used to generate a single view distance image, but the parallax images can be added for each pixel to generate a viewing image and used to generate a multi-view distance image. In this case, the multi-view distance image is generated using this image as a reference image.

[0108] In addition, when the photographer intentionally selects the best photo for obtaining the viewing image and the distance image as a set by operating the shutter button of the imaging device before generating the distance image, the imaging conditions can be explicitly changed between the shooting for single-viewpoint distance image generation and the shooting for multi-viewpoint distance image generation. Fig. 10A As shown, in continuous shooting for multi-view distance image generation, a large Av (aperture) value is set so that the aperture becomes small, and pan focus shooting is performed so that feature points can be easily extracted in the entire range in the depth direction of the scene. On the other hand, it is conceivable that a small Av value is used only when the shutter button is fully pressed at SW2 so that the pupil is fully open, and distance measurement in single shooting can be accurately performed near the focus distance. The Tv value and ISO value are appropriately set according to the Ev table or the like to follow the Av value.

[0109] Furthermore, since only the distance value is actually integrated, it is not necessary to use the same Ev (exposure) value as the total value of the Av value, Tv value, and ISO value. Fig. 10B As shown, the Ev value as a shooting condition may significantly change between shooting for single-viewpoint distance image generation and shooting for multi-viewpoint distance image generation. Fig. 10B This is an example of using different Ev values ​​when shooting for generating a single viewpoint distance image and when shooting for generating a multi-viewpoint distance image. For example, when such shooting conditions are used for shooting in a dark environment, the situations where the technology of the present invention can be used can be increased.

[0110] For example, if continuous shooting with pan focus is desired and motion blur is desired to be suppressed at the same time, the shooting conditions are set to have a low Ev value, for example, a total value of Av value, Tv value and ISO of 11, even if the amount of light becomes insufficient. On the other hand, in single shooting, since the less amount of light will increase noise if the same Ev value as that used in continuous shooting is used in single shooting, the Ev value is increased by increasing the Av value (which is not equal to the Ev value used in continuous shooting). Even if an image is captured in this way, the captured image itself is not integrated, making it unlikely that a problem will occur. In the case of generating a multi-viewpoint distance image using an image obtained by a single shot, a scaling correction of grayscale is applied to the pixel value of the image by the difference in Ev value to adjust the pixel value. In this case as well, since the difference in Ev value is only indirectly related to the integrated distance image as the final result, the influence of the difference in Ev value is unlikely to occur.

[0111] As described above, according to the first embodiment, a single-viewpoint distance image and a multi-viewpoint distance image are integrated to supplement their respective shortcomings, and a distance image of the entire scene including a moving subject can be acquired with high accuracy.

[0112] Although the integrated distance information is represented by the word "image", the output format of the information is not limited to an image. For example, the format may be a 2.5-dimensional stereo image obtained by projecting a distance image into a three-dimensional space according to its distance value, point cloud or volume data having different storage formats, or stereo data converted into mesh information.

[0113] <Second Embodiment>

[0114] Next, a second embodiment of the present invention will be described. Fig.11 is a block diagram showing an image processing apparatus 200 according to the second embodiment.

[0115] The image input unit 201 receives a parallax image pair captured by an external camera (not shown) and a viewing image obtained by adding the parallax image pair for each pixel in the camera, and stores the input parallax image pair and viewing image in the memory 102. Then, as described in the first embodiment, the input parallax image pair and viewing image are processed to obtain an integrated distance image. Since the structure other than the image input unit 201 is the same as that of the first embodiment, the image input unit 201 is different from the first embodiment. Figure 1 The structures shown are the same, so the same reference numerals are assigned and their description will be omitted here.

[0116] like Fig. 12A As shown, if a parallax image pair for every shot is input, a parallax image pair to be used for generating a single viewpoint distance image and a viewing image to be used as a reference image when generating a multi-viewpoint distance image can be selected. Therefore, from a plurality of parallax image pairs obtained by continuous shooting, a parallax image pair to be used for generating a single viewpoint distance image and a viewing image to be used as a reference image are selected using an algorithm determined before or after shooting. For example, it can be preset to select a parallax image pair obtained in the last shot of all shots. Since this case is particularly assumed to be a case of motion image shooting, images corresponding to each frame or images shot every several frames are acquired as parallax image pairs during animation shooting.

[0117] In addition, if Fig. 12B As shown, a single viewpoint distance image and a multi-viewpoint distance image can be generated while sequentially shifting the parallax image pair to be selected. By doing so, for each parallax image pair, even in a scene with a wide depth, a highly accurate integrated distance image can be obtained by integrating the single viewpoint distance image and the multi-viewpoint distance image.

[0118] As described above, according to the second embodiment, in the image processing apparatus, a highly accurate distance image can be obtained by using a parallax image pair obtained from an image pickup apparatus.

[0119] <Modification>

[0120] In the description of the first and second embodiments described above, the shooting of the parallax image pairs for generating the single viewpoint distance image has been described as a typical single-shot shooting. However, if the subject moves slowly, a plurality of parallax image pairs may be shot, and the single viewpoint distance image generation unit 104 may generate a single viewpoint distance image from each parallax image pair and integrate these single viewpoint distance images, thereby improving the quality of the single viewpoint distance image, which may be further integrated with the multi-viewpoint distance image.

[0121] refer to Fig.13 , the following process will be described: a single viewpoint distance image is generated from a plurality of parallax image pairs and integrated to improve the quality of the single viewpoint distance image, and then the single viewpoint distance image is integrated with the multi-viewpoint distance image. For example, if there is a change in the position and / or posture of the camera device between shooting, the viewpoint of the single viewpoint distance image is changed three-dimensionally so that the coordinates of the single viewpoint distance image are consistent with the coordinates of the reference image, and then the single viewpoint distance image is integrated.

[0122] For example, in Fig.13 In the method, among three parallax image pairs, the central parallax image pair is set as a reference, and the viewpoints of the single viewpoint distance images obtained from the remaining parallax image pairs are changed according to the shooting position and posture when the parallax image pair set as the reference is shot. Then, the two single viewpoint distance images whose viewpoints are converted are integrated with the reference single viewpoint distance image, and the integrated single viewpoint distance image is further integrated with the multi-viewpoint distance image to obtain the final integrated distance image.

[0123] According to the above-described modification, a more accurate distance image can be obtained.

[0124] <Third Embodiment>

[0125] Next, a third embodiment of the present invention will be described.

[0126] In the first and second embodiments described above, it is explained as follows: a single viewpoint distance image generated from a parallax image pair obtained by a single shot using a pupil division camera system is integrated with a multi-viewpoint distance image generated from a plurality of viewing images obtained by taking multiple shots in time sequence to generate an integrated distance image. However, in order to obtain a single viewpoint distance image with high accuracy by using a parallax image pair captured by a pupil division camera system as an input, it is necessary to widen the distance between the pupils and increase the baseline length. On the other hand, if the pupil is widened, the depth of field becomes shallower, so that a subject existing outside a predetermined distance range relative to the focus position becomes blurred, and it becomes difficult to associate the subject in the viewing image. Therefore, it becomes impossible to accurately calculate the distance to a subject existing outside a predetermined distance range relative to the focus position using the viewing image.

[0127] As described above, in principle, there are limitations on the shooting scenes and shooting conditions in which both a single viewpoint distance image and a multi-viewpoint distance image can be obtained with high accuracy. For example, in an environment such as a dark place, the selectable shooting conditions become worse, and it becomes more difficult to obtain both a single viewpoint distance image and a multi-viewpoint distance image with high accuracy. This is because in a dark place, the shutter speed needs to be kept short to prevent motion blur, but due to insufficient light, it is necessary to open the pupil of the optical system and reduce the F value to increase the amount of light. As described above, when the pupil of the optical system is opened, the depth of field becomes shallow, and the range in the depth direction in which a multi-viewpoint distance image can be accurately generated becomes narrower.

[0128] Therefore, in the third embodiment, shooting is performed so that a parallax image pair can be acquired even in multi-shot shooting other than the main shooting in which the parallax image pair is acquired using the pupil division camera system. Then, using the parallax image pairs of each shooting, a viewing image with an expanded depth of field is generated by restoring the defocus blur in the image area outside the predetermined distance range relative to the focus position, and an operation for finding corresponding points between the plurality of viewing images obtained by the multi-shot shooting is performed in a state without the defocus blur. Then, by integrating the single viewpoint distance image and the multi-viewpoint distance image, a more accurate integrated distance image can be obtained in the entire scene including the distance to the subject outside the predetermined distance range relative to the focus position.

[0129] Fig.14 is a block diagram showing an image processing apparatus 300 according to the third embodiment. Fig.14 In the embodiment, the same reference numerals are assigned to the Fig.11 The structure of the image processing apparatus 200 is the same as that of the above-described image processing apparatus, and a description thereof will be omitted.

[0130] The viewing image generation unit 303 in the third embodiment can generate a viewing image from the input parallax image pair in the same manner as in the viewing image generation unit 103, and can generate a viewing image from the input parallax image pair after performing defocus deblurring processing. That is, the defocus deblurring processing can be applied to all of the multiple viewing images input to the multi-viewpoint distance image generation unit 105, or can be applied to only a part of these viewing images. Therefore, all of the images input to the viewing image generation unit 303 can be parallax image pairs, or both of the parallax image pairs and the viewing image in which the first signal and the second signal are added for each pixel and then output can be input to the viewing image generation unit 303.

[0131] Next, refer to Fig.15The process of photographing and integrating the distance image generation in the third embodiment is described with reference to the flowchart of FIG.

[0132] In step S301, continuous multiple shooting is used to generate a single viewpoint distance image and a parallax image pair for defocus deblurring (multi-shooting). In the case of generating a single viewpoint distance image and performing defocus deblurring on all images obtained by performing multiple shooting, all shooting is performed to obtain a set of parallax images (parallax image pairs) consisting of a plurality of images, where the number of images corresponds to the number of viewpoints under the microlens 211. If a single viewpoint distance image is not generated and defocus deblurring is not performed for some of the images obtained by performing multiple shooting, the generated viewing image may be input instead. In the case of acquiring a plurality of images by motion image shooting rather than multi-shooting, control of acquiring a parallax image pair in all frames is simpler than control of not acquiring a parallax image pair in some frames.

[0133] In step S302, an image (i.e., a reference image) used as a reference for generating an integrated distance image is selected from a plurality of images obtained by performing multiple shooting. In the case of continuous shooting of still images, the reference image may be pre-selected using the GUI of the camera device at the time of shooting. Alternatively, the reference image may be selected using the GUI (not shown) of the image processing device 300. In the case of moving image shooting, the frames used as a reference for generating the integrated distance image are sequentially moved in the time direction.

[0134] In step S303 , a single viewing image to which defocus deblurring is applied is generated using parallax image pairs consisting of a plurality of images obtained in each shooting or a portion thereof (where the number of images corresponds to the number of viewpoints under the microlens 211 ).

[0135] The defocus deblurring processing for the disparity image pairs taken for each time can be implemented by a deconvolution process or a MAP estimation process for calculating a blind image or a distance image and estimating a defocus kernel. Alternatively, it can be implemented by a deep learning process instead. If the defocus deblurring processing for the disparity image pairs taken for each time is implemented by using a deep learning process, an end-to-end process using an encoder-decoder structure is first considered. Optionally, this can be implemented by constructing a network corresponding to a deconvolution process, which includes estimating a traditional non-blind distance image or a defocus kernel. Each process will be described below.

[0136] First, a method of end-to-end processing using an encoder-decoder configuration will be described. FIG. 16A to FIG. 16C is a diagram showing the input-output relationship of the network used for defocus deblurring processing. Fig.16Ais an input-output example of a network that performs defocus deblurring end-to-end. Multiple network implementations have been proposed for this method, such as in the form of Abdullah Abuolaim and Michael S. Brown, "Defocus Deblurring Using Dual-Pixel Data", ECCV1020. It should be noted that any method can be used as long as the disparity image pairs (image A, image B) of each shot are input and the defocus deblurred image can be obtained. The network is learned by using the all-in-focus image captured by reducing the aperture of the camera device as the training data for the network constructed by the encoder-decoder. The network is learned by using the difference between the output image and the ground truth all-in-focus image as the loss. When the learning is completed, for each disparity image pair of each shot considered as input, a viewing image on which defocus deblurring has been performed is obtained.

[0137] Next, an example of a method for constructing a network corresponding to a deconvolution process accompanied by an estimated range image or a defocus kernel will be described. Fig. 16B An example of a network is shown. The network consists of a network that inputs a pair of disparity images taken at each time and calculates a temporary distance image or a temporary inverse depth image that takes the inverse of the distance value d (i.e., 1 / d), and a network that inputs a pair of disparity images and a temporary distance image or a temporary inverse depth image and outputs a defocused deblurred image and a refined distance image or a refined inverse depth image. The temporary distance image or the temporary inverse depth image can be a parameter map representing the defocus kernel or its image. Note that Fig. 16B The case where the number of disparity images captured each time is two (image A and image B) is illustrated. The distance calculation network and the defocus deblurring network can be learned independently. The network is learned using the correct image of the distance value corresponding to the disparity image pair and the full-focus image captured by reducing the aperture of the camera as training data. The learning is performed by performing back propagation using the error of the distance value, the error of the pixel value, and the edge sharpness as the error function. An example of such a network is disclosed in "Dual Pixel Exploration: Simultaneous Depth Estimation and Image Restoration", CVPR2021 by Conducted Liyuan Pan, Shah Chowdhury, Richard Hartley, Miaomiao Liu, Hongguang Zhang, Hongdong Li (hereinafter referred to as "Pan et al."). The details of the network, the details of the learning data, an example of the error function, etc. can be realized by referring to Pan et al.

[0138] Although the distance calculation network and the defocus deblurring network are exemplified with reference to the form of the network of Pan et al., the present invention is not limited to the specific form of the network exemplified by Pan et al. The above network can be replaced by other deep learning networks or classical methods. For example, the distance calculation network can be as follows Fig. 16C The deep learning network shown in FIG. 1 estimates the model parameters of the defocus kernel as disclosed in “Modeling Defocus-Disparity in Dual-Pixel Sensors” by Abhijith Punnappurath, Abdullah Abuolaim, Mahmoud Afifi, and Michael S. Brown (hereinafter referred to as Punnappurath et al.), ICCP 2020. In addition, the defocus deblurring network can be replaced by a shift-variant deconvolution process as a classical process.

[0139] Punnappurath et al. disclose a network that estimates a defocus kernel model, creates a disparity map by finding the parameters of the defocus kernel for each view of each input disparity image captured, and outputs the disparity map as a substitute for a range image.

[0140] In step S304, a single viewpoint distance image is generated, and since this process is similar to that of the first embodiment, Figure 3 The processing in step S102 is the same as that in step S102, so its description will be omitted.

[0141] In S305, a multi-view distance image is generated by using a plurality of viewing images acquired by multi-shooting, the plurality of viewing images including the viewing images defocused and not defocused in step S303. Since the method of generating the multi-view distance image is the same as the process in step S103 described in the first embodiment, the description thereof will be omitted.

[0142] In step S306, the single viewpoint distance image and the multi-viewpoint distance image are integrated to generate an integrated distance image with high accuracy in the entire range in the depth direction of the scene. Since the method of generating the integrated distance image is the same as the processing in step S104 described in the first embodiment, its description will be omitted.

[0143] Next, refer to Fig.17, the above-mentioned process from shooting to the generation of the integrated distance image will be outlined. Here, the case where all images are acquired as parallax image pairs through multiple shooting will be described. For example, assuming that a still camera is used for shooting, in step S301, the photographer can fully press the shutter button to intentionally shoot. Thereafter, in step S302, a parallax image pair (reference image pair) used as a reference is selected from the multiple parallax image pairs acquired by performing multiple shooting. The parallax image pair to be selected may be the same as the parallax image pair to be used to generate a single viewpoint distance image. In step S303, a defocus deblurring process is performed on the parallax image pair of the viewing image to be used for generating a multi-viewpoint distance image.

[0144] In step S304, a single viewpoint distance image is generated using the selected parallax image pair. Then, in step S305, a multi-viewpoint distance image is generated based on a plurality of viewing images obtained by performing defocus deblurring processing. In step S306, the single viewpoint distance image and the multi-viewpoint distance image are integrated to obtain an integrated distance image. The obtained integrated distance image is a distance image having higher distance accuracy than the single viewpoint distance image in the entire scene including an area outside a predetermined distance range relative to the focus point. In addition, unlike the multi-viewpoint distance image, a distance value can be obtained even if a moving subject is included.

[0145] Fig.18 This is an explanatory diagram for the case where it is desired to generate an integrated distance image for each image obtained by multiple shooting, or when it is desired to generate an integrated distance image corresponding to each frame in moving image shooting. In this case, by sequentially shifting the selection of the reference image pair performed in step S302, an integrated distance image corresponding to all shooting is generated.

[0146] As described above, according to the third embodiment, a highly accurate distance image can be obtained in various photographing scenes.

[0147] <Fourth Embodiment>

[0148] Next, a fourth embodiment of the present invention will be described.

[0149] Fig.19A 4 is a block diagram showing the structure of an image pickup apparatus according to a fourth embodiment of the present invention. Fig.19A , the camera device 400 includes an image sensing unit 401, an image sensing unit 402, a calculation unit 403, a storage unit 404, a shutter button 405, an operation unit 406, a display unit 407, and a control unit 408.

[0150] Fig.19B4 is a front view of the appearance of the imaging device 400 of the present embodiment. The imaging device 400 has two imaging units, an image sensing unit 401 and an image sensing unit 402. The image sensing unit 401 includes a lens 401a and an image sensor 401b. The image sensing unit 402 includes a lens 402a and an image sensor 402b. The lenses 401a and 402a converge light reflected by the subject and form an optical image on the image sensors 401b and 402b. The image sensors 401b and 402b convert the optical image into an electrical signal and output image data.

[0151] The image sensing unit 401 and the image sensing unit 402 are arranged to be able to capture a common subject and capture an image with parallax. When the user of the camera 400 presses the shutter button 405, the image sensing unit 401 and the image sensing unit 402 perform compound eye stereoscopic photography. It is assumed that the distance D between the optical axis of the image sensing unit 401 and the optical axis of the image sensing unit 402 is known.

[0152] Fig.19C 406 is a rear view of the appearance of the image pickup device 400 of the present embodiment. The operation unit 406 is pressed to set shooting conditions or give instructions such as the start or end of the subject distance measurement mode. The display unit 407 is, for example, a liquid crystal display, which displays the composition at the time of shooting and displays various setting items. If the display unit 407 is a touch panel, the display unit 407 can be used as the shutter button 405 and the operation unit 406, and the user can instruct to take an image and make settings by touching the display unit 407. In this case, the image pickup device 400 is not necessarily equipped with hardware components such as the shutter button 405 and the operation unit 406, operability can be promoted, and a large display unit 407 that makes it easier to see can be provided.

[0153] The control unit 408 controls shooting conditions when shooting with the image sensing unit 401 and the image sensing unit 402. For example, the aperture diameter of the diaphragm of the optical system, the shutter speed, the ISO sensitivity, etc. are controlled. The calculation unit 403 develops the images shot by the image sensing unit 401 and the image sensing unit 402, and calculates the distance to the subject.

[0154] Fig. 20 400 is a flowchart showing the operation of the distance measurement process in the image pickup apparatus 400 of the present embodiment. When the user instructs the start of the distance measurement mode in step S401, the operation of the flowchart is started.

[0155] Fig.21A 4 is a flowchart showing the detailed processing of step S401. Step S401 includes the processing of step S4011 and step S4012.

[0156] In step S4011, the user selects a distance measurement mode. Fig. 22 The state of the display unit 407 when the distance measurement mode is selected is shown. The user operates the operation unit 406 to display the menu 501 on the display unit 407. The menu 501 shows items of the shooting mode, and the user operates the operation unit 406 to select the distance measurement mode. In step S4012, the control unit 408 of the imaging apparatus 400 starts continuous shooting.

[0157] Return to Fig. 20 In step S402, the relative distance is acquired using a monocular stereo camera. When shooting using a monocular stereo camera, the image sensing unit 401 or the image sensing unit 402 can be used. In this embodiment, it is assumed that the image sensing unit 401 is used for shooting. Continuous shooting can be performed when the frame interval is short as in the shooting of moving images, or can be performed when the frame interval is long as in the continuous shooting of still images.

[0158] Fig. 21B 4 is a flowchart showing the detailed processing of step S402. Step S402 includes the processing of steps S4021 to S4027.

[0159] Step S4021 represents a loop for processing captured frames using a monocular stereo camera. In order to obtain the relative distance using a monocular stereo camera, first, the three-dimensional relative position of the subject is obtained, and then converted into a relative distance value relative to the position where the user wants to obtain the absolute distance value. The three-dimensional relative position of the subject can be obtained using methods such as SfM (Structure from Motion) or SLAM (Simultaneous Localization and Mapping). Alternatively, a method called MVS (Multi-View Stereo) can also be used, in which the position and posture of the camera device 400 are calculated by SfM or SLAM, and then the position and posture are used to obtain a dense three-dimensional relative position. In the following description, it is assumed that a method (such as SfM and SLAM) is used to simultaneously obtain the position and posture of the camera device 400 and the three-dimensional relative position of the subject.

[0160] Fig.23 601 is a diagram showing the state of the display unit 407 during the measurement of the relative distance. The image 601 shows the subject whose relative distance is to be measured in real time. By shooting the image while viewing the image, the user can perform relative distance measurement including the subject whose distance the user wishes to measure. The notification 602 is used to notify the user that the relative distance measurement is being performed by monocular stereo image shooting, and is superimposed on the image 601. The notification 602 can also be performed by an alarm sound or voice guidance, etc., and in this case, the image 601 can be displayed without any hidden part.

[0161] In step S4022, the control unit 408 extracts feature points from the image of the current captured frame. Typical methods for extracting feature points include SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features), and FAST (Speeded Up Segment Test Features); however, other methods may be used.

[0162] In step S4023, the control unit 408 determines whether the current frame is the first frame, and if the current frame is the first frame, the process enters the second frame without subsequent processing, and if the current frame is not the first frame, the process enters step S4024.

[0163] In step S4024, the control unit 408 associates the feature points extracted in step S4022 in the immediately preceding frame with the feature points extracted in step S4022 in the current frame. If the frame interval is long, it is necessary to include a sufficient amount of common parts of the subject whose distance is to be measured in the images of different frames. This is because, if the amount of the common parts is small, the feature points may not be associated between the images, and the calculation of relative distance acquisition may stop.

[0164] Fig.24 701 and 702 represent the positions of the camera device 400 at different times. Image 703 and image 704 respectively show images captured at positions 701 and 702 using the image sensing unit 401. Feature point 705 is a feature point extracted from image 703 and is correctly associated with feature point 706 in image 704. On the other hand, the point corresponding to feature point 707 is not correctly extracted from image 704, and feature point 707 is incorrectly associated with feature point 708 in image 704.

[0165] Now, if the reliability of the relative distance value using the monocular stereo camera is defined as the matching accuracy of the feature point, the feature point 707 has poor matching accuracy and the reliability of the relative distance value is relatively low. The reasons for this reduction in reliability include the small amount of texture of the subject and the image blurring at the position in the image 704 corresponding to the feature point 707.

[0166] In step S4025 , the control unit 408 calculates the position and orientation of the imaging apparatus 400 and the three-dimensional relative position of the object.

[0167] In step S4026, the control unit 408 calculates the reliability of the relative distance value. The matching accuracy of the feature points associated between the images in step S4024 is calculated, and it is assumed that the higher the matching accuracy, the higher the reliability. The matching accuracy can be calculated using an algorithm such as RANSAC (Random Sampling Consensus). RANSAC calculates how much the movement of the feature point of interest deviates based on the position and orientation of the camera 400 and the average of the movement of many feature points in the image, and determines the matching accuracy of each feature point.

[0168] In step S4027, the control unit 408 determines whether the user has instructed to shoot using the compound eye stereo camera. If the user has not instructed, the process continues to the next frame, and if the user has instructed, the process of this flow is terminated and the process enters Fig. 20 Here, the instruction for photographing using the compound-eye stereo camera is an instruction issued by the user by pressing the shutter button 405 halfway to start a photographing operation for calculating an absolute distance value.

[0169] In step S403 , the control unit 408 performs photographing using the compound eye stereo camera (ie, by using the image sensing unit 401 and the image sensing unit 402 ), and obtains an absolute distance value.

[0170] Fig.25A 4 is a flowchart showing the detailed processing of step S403. Step S403 includes the processing of steps S4031 to S4039.

[0171] In step S4031, the control unit 408 focuses on a specific subject. The specific subject is, for example, a subject that the user particularly wants to measure the distance to. As a focusing method, for example, there is a method of focusing by an automatic focusing function performed in response to the user half-pressing the shutter button 405. Or manual focusing can be performed. Here, both the image sensing unit 401 and the image sensing unit 402 focus on the same subject.

[0172] In step S4032, the absolute distance using the compound eye stereo camera and its reliability are calculated. In the absolute distance calculation using the compound eye stereo camera, a technique such as stereo matching is used for the images respectively captured by the image sensing unit 401 and the image sensing unit 402.

[0173] Fig.264 is a diagram showing the state of stereo matching during calculation of absolute distance using a compound eye stereo camera. When the user focuses at position 801 in step S4031, the images captured by the image sensing unit 401 and the image sensing unit 402 are shown as image 802 and image 803, respectively. Image 802 is used as a base image and image 803 is used as a reference image, and a window (area) 804 and a window 805 are set in image 802 and image 803, respectively. A correlation operation between the images in the windows is performed with window 804 fixed and window 805 moved, and the position of window 805 having the highest correlation value is determined as the corresponding position of window 804. Then, by adopting the offset between the corresponding points on the image as parallax, the absolute distance to the subject in the window is calculated based on the parallax value and the baseline length based on the distance D between the optical axis of the image sensing unit 401 and the optical axis of the image sensing unit 402. At this time, the reliability of the calculated absolute distance is calculated. For example, the position of the window 804 can be moved to various positions, and the position having a relatively high maximum correlation value among the correlation values ​​obtained by performing the window matching calculation at the moved position can be regarded as having high reliability. In addition, as will be described later, the reliability can be represented by the defocus amount relative to the subject. Here, the window matching method is described, but the present invention is not limited thereto, and feature points can be extracted and associated with each other as in the relative distance calculation using a monocular stereo camera.

[0174] In step S4033, the control unit 408 determines whether there is a substantially identical subject for which both the relative distance value acquired by the monocular stereo camera and the absolute distance value acquired by the compound-eye stereo camera have high reliability. If there is a substantially identical subject with high reliability, the process proceeds to step S4037, and if there is no substantially identical subject with high reliability, the process proceeds to step S4034.

[0175] In step S4034, the relative distance acquired by the monocular stereo camera is converted into an absolute distance by using feature points near the focus position. It is not always the case that there is a subject with high reliability near the focus position in the reliability calculation in both the relative distance calculation and the absolute distance calculation, and here, the reliability that enables the approximate value of the depth of field to be obtained is sufficient.

[0176] Next, the reliability of the absolute distance acquired by the compound eye stereo camera is defined by using the defocus amount. For example, the reliability is determined as the inverse of the magnitude of the defocus amount, so that the smaller the defocus amount, the higher the reliability. This makes it possible to move the window 804 at various positions and determine a position with high reliability. If the window 804 is set to include a candidate point that can be a feature point X among the feature points acquired by the monocular stereo camera, the reliability of the candidate point can be known from the defocus amount. When shooting with a compound eye stereo camera, if it is assumed that the absolute distance value of the subject within the depth of field with the defocus amount is highly reliable, then when the subject included in the window 804 is within the depth of field, the subject can be a candidate point for the feature point X.

[0177] In step S4035, the control unit 408 sets the aperture to be small so that the candidate point of the feature point X is within the depth of field. Through this processing, the candidate point of the feature point X can be photographed without blur, and the absolute distance value can be obtained with high accuracy. When the relative distance value is converted into an absolute distance value in step S4034, it is not always the case that a highly reliable feature point is used, and it is advantageous to set the aperture to be smaller than the aperture at which the candidate point falls within the depth of field. Here, there is no need to set the size of the aperture, and the focus can be adjusted to a position where the candidate point of the feature point X can be within the depth of field where the size of the aperture remains unchanged. By doing so, even if the shooting scene is too dark to set the aperture small, the candidate point of the feature point X can be photographed without blur.

[0178] In step S4036 , the control unit 408 calculates the reliability of the absolute distance value for each candidate point of the feature point X.

[0179] In step S4037 , the feature point X is determined based on the candidate points of the feature point X. The determination method may be, for example, the position with the highest total ranking of the reliability calculated by the monocular stereo camera and the reliability acquired by the compound eye stereo camera.

[0180] In step S4038, the user presses the shutter button 405 to capture an image.

[0181] In step S4039, the absolute distance value of the feature point X is obtained from the two images captured by the compound eye stereo camera. The method of obtaining the absolute distance value is the same as the above method, but if the calculation is performed by window matching, a window as small as possible and including the feature point X can be set. In the case of calculating the absolute distance value by feature point matching, the feature point X in the two images can be matched. From step S4039, the process returns to Fig. 20 Step S404.

[0182] In step S404, the relative distance value acquired by the monocular stereo camera is converted into an absolute distance value. Fig.25B 4 is a flowchart showing the detailed processing of step S404. Step S404 includes the processing of steps S4041 and S4042.

[0183] In step S4041, a conversion formula (conversion relationship) for converting relative distance values ​​into absolute distance values ​​for all objects is calculated by using the relative distance value and the absolute distance value of the feature point X. For example, if the relative distance value of the feature point X is zr and its absolute distance value is Za[m], the relationship between the two can be represented by Z=(Za / zr)×z. Here, z is the relative distance value of a certain object, and Z is its absolute distance value.

[0184] In step S4042, the relative distance value obtained by the monocular stereo camera is converted into an absolute distance value using a conversion formula. Here, if the range to be converted is limited to the range of the composition of the image captured in step S4038, the absolute distance value of the composition desired to be obtained by the user can be obtained with high accuracy.

[0185] By using the method of this embodiment, a conversion formula from a relative distance value to an absolute distance value can be determined at a position of an image where the relative distance value has high reliability and the absolute distance value has high reliability. Then, by using the conversion formula to convert other objects, highly accurate absolute distance values ​​can be obtained for all objects.

[0186] <Fifth Embodiment>

[0187] Next, a fifth embodiment of the present invention will be described.

[0188] In the fourth embodiment described above, the imaging apparatus is configured as a compound eye stereo camera. In contrast, by using a pupil division image sensor, one of the image sensing unit 401 and the image sensing unit 402 can be eliminated, thereby reducing the number of image sensors to 1 and simplifying the configuration.

[0189] Fig.27A and Fig.27B The structure of a pupil-divided image sensor and the principle of distance calculation are illustrated. Fig.27A Indicates the state of the subject in focus, and Fig.27B A state in which the subject is located closer to the image sensor than the in-focus position is shown.

[0190] The image sensor 901 has a pupil division configuration, and the inside of a pixel 903 is divided into a sub-pixel 904 and a sub-pixel 905. Of the light reflected by the subject, one light beam passes through an end of the imaging optical system 902 and is received by the sub-pixel 904, and the other light beam passes through the opposite end of the imaging optical system 902 and is received by the sub-pixel 905. Images 906 and 908 are images generated from the light received by the sub-pixel 904, and images 907 and 909 are images generated from the light received by the sub-pixel 905.

[0191] like Fig.27A As shown in FIG. 1 , since light from a subject at a focused position is received by sub-pixel 904 and sub-pixel 905 of the same pixel, there is no parallax between the subjects in image 906 and image 907. Fig.27B As shown, light from a subject that is not in focus is received by sub-pixel 904 and sub-pixel 905 of different pixels, and there is a parallax between the subject in image 908 and the subject in image 909. The absolute distance can be calculated based on the parallax.

[0192] In the case of using a pupil segmentation image sensor, the sub-pixels of each pixel share a lens, so the baseline length of the compound eye stereo camera is determined by the size of the aperture. Therefore, if the aperture is reduced to keep the candidate point of the feature point X within the depth of field, the baseline length becomes shorter and an absolute distance value with high accuracy cannot be obtained. In this case, the reliability of the absolute distance value acquired by the pupil segmentation image sensor can be adjusted in consideration of both the size of the defocus and the baseline length. In addition, as described above, the focus position can be adjusted to a position where the reliability of the candidate point of the feature point X becomes high when the aperture is fixed.

[0193] In the above description, a method for calculating a highly accurate absolute distance value by matching a relative distance value acquired by a monocular stereo camera with an absolute distance value acquired by a compound-eye stereo camera is described. In addition, the distance values ​​may be combined.

[0194] For example, if it is desired to obtain the absolute distance value of a scene including a moving object, the relative distance value of the moving object cannot be obtained using a monocular stereo camera. In this case, as the absolute distance value of the moving object, the absolute distance value obtained by the compound eye stereo camera can be used as it is.

[0195] In a monocular stereo camera, the corresponding points of the subject are calculated using images captured at different times, so that the corresponding points may not be found in the case of an image including a moving subject. In addition, the movement of the moving subject may not be distinguished from the change in the position and posture of the camera. On the other hand, in a compound eye stereo camera, since the shooting timing can be controlled to be the same, the moving subject between the two images can be processed in the same way as the stationary subject, and the absolute distance value can be calculated. In the case of creating an absolute distance image represented by shading, with respect to the stationary subject, a value obtained by converting the relative distance value acquired by the monocular stereo camera into the absolute distance value at the feature point X extracted from the stationary subject is used. Then, for the moving subject, the absolute distance value acquired by the compound eye stereo camera is used, and the absolute distance value is synthesized with the absolute distance image using the monocular stereo camera. Whether the subject is a moving subject can be determined by using machine learning, etc., or by finding a subject that does not exist in the relative distance image acquired by the monocular stereo camera but exists in the absolute distance image acquired by the compound eye stereo camera, or by combining these methods.

[0196] Other embodiments

[0197] The embodiments of the present invention may also be implemented by providing software (program) for performing the functions of the above-described embodiments to a system or device via a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0198] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications, equivalent structures and functions.

Claims

1. An image processing device, comprising: an acquisition component configured to acquire a plurality of different viewpoint images obtained by photographing the same scene from different viewpoints, and configured to acquire at least one parallax image pair having parallax through pupil division; A first generating component configured to generate a first distance image from the parallax image pair; a second generating component configured to generate a second distance image from the plurality of different viewpoint images; as well as An integration component is configured to integrate the first range image and the second range image and generate an integrated range image.

2. The image processing apparatus according to claim 1, wherein: The integration means integrates the first range image and the second range image by supplementing the distance information of the same area in the distance information constituting the second range image with the distance information of the area where the subject is moving in the first range image.

3. The image processing apparatus according to claim 1, wherein: The integration component integrates the first range image and the second range image by supplementing the distance information of the same area in the distance information constituting the first range image with the distance information of the following area in the second range image, which is an area other than an area indicating a distance shorter than a predetermined distance and an area indicating a predetermined range relative to a focus distance.

4. The image processing apparatus according to any one of claims 1 to 3, wherein: The second generating means generates the second distance image based on the parallax between the plurality of different viewpoint images.

5. The image processing apparatus according to claim 4, wherein: The second generating means generates the second distance image using epipolar geometry based on the plurality of different viewpoint images.

6. The image processing apparatus according to any one of claims 1 to 3, wherein: The acquisition component acquires a plurality of parallax image pairs obtained by photographing the same scene at different positions, and The image processing apparatus further includes a third generating section configured to generate the plurality of different viewpoint images from the plurality of parallax image pairs.

7. The image processing apparatus according to claim 6, wherein: The third generating means selects one of the plurality of parallax image pairs as a reference parallax image pair, and generates the plurality of different viewpoint images by adding images constituting each of the plurality of parallax image pairs other than the reference parallax image pair in units of pixels.

8. The image processing apparatus according to claim 6, wherein: The third generating means maintains one of the plurality of parallax image pairs as it is as a reference parallax image pair, and generates the plurality of different viewpoint images by adding images constituting each of the plurality of parallax image pairs including the reference parallax image pair in units of pixels.

9. The image processing apparatus according to claim 7, wherein: the third generating means sequentially shifts selection of the reference parallax image pairs among the plurality of parallax image pairs, and generates the plurality of different viewpoint images for each of the sequentially selected reference parallax image pairs, and The integration component integrates, for each of the sequentially selected reference parallax image pairs, a first distance image generated from the reference parallax image pair and a second distance image generated from a plurality of different viewpoint images corresponding to the reference parallax image pair, and generates a plurality of integrated distance images.

10. The image processing apparatus according to claim 8, wherein: the third generating means sequentially shifts selection of the reference parallax image pairs among the plurality of parallax image pairs, and generates the plurality of different viewpoint images for each of the sequentially selected reference parallax image pairs, and The integration component integrates, for each of the sequentially selected reference parallax image pairs, a first distance image generated from the reference parallax image pair and a second distance image generated from a plurality of different viewpoint images corresponding to the reference parallax image pair, and generates a plurality of integrated distance images.

11. The image processing apparatus according to claim 6, wherein: The third generating means performs defocus deblurring processing on the plurality of parallax image pairs, and generates the plurality of different viewpoint images from the plurality of parallax image pairs subjected to the defocus deblurring processing.

12. The image processing apparatus according to claim 11, wherein: The defocus deblurring process includes a deconvolution process or a MAP estimation process performed by estimating a defocus kernel, and a deep learning process by an end-to-end process using an encoder-decoder structure.

13. The image processing apparatus according to any one of claims 1 to 3, wherein: The acquisition component acquires a plurality of parallax image pairs obtained by photographing the same scene at different positions, wherein the first generating unit generates the first distance image from each of a plurality of predetermined parallax image pairs among the plurality of parallax image pairs, selects one of the generated plurality of first distance images as a reference first distance image, changes the viewpoint of the first distance images other than the reference first distance image, and integrates the first distance image with the changed viewpoint with the reference first distance image, and The integration component integrates the integrated first distance image with the second distance image.

14. A camera device, comprising: The image processing device according to any one of claims 1 to 3; as well as The image sensor as at least a part of the acquisition component is used to capture the at least one parallax image pair.

15. The imaging device according to claim 14, wherein: The image sensor captures the at least one parallax image pair after capturing the plurality of different viewpoint images.

16. The imaging device according to claim 14, wherein: The image sensor captures the plurality of different viewpoint images after capturing the at least one parallax image pair.

17. The imaging device according to claim 14, wherein: The image sensor captures the plurality of different viewpoint images before and after capturing the at least one parallax image pair.

18. The imaging device according to claim 14, wherein: The image sensor controls an aperture of an aperture used for capturing the plurality of different viewpoint images to be smaller than an aperture of an aperture used for capturing the at least one parallax image pair.

19. The imaging device according to claim 14, wherein: The image sensor independently sets photographing conditions for photographing the at least one parallax image pair and for photographing the plurality of different viewpoint images.

20. An image processing method, comprising: An acquisition step for acquiring a plurality of different viewpoint images obtained by photographing the same scene from different viewpoints, and acquiring at least one parallax image pair having parallax through pupil division; A first generating step, for generating a first distance image from the parallax image pair; A second generating step is used to generate a second distance image from the plurality of different viewpoint images; as well as An integration step is used to integrate the first range image and the second range image to generate an integrated range image.

21. A computer-readable storage medium storing a program for causing a computer to execute the image processing method according to claim 20.

Images