Information processing device, information processing method, and program

The information processing device addresses parallax miscalculation by integrating parallax information with sensitivity-based area adjustments, enhancing accuracy in distance measurement.

JP2024169101APending Publication Date: 2024-12-05CANON KK
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2023086299
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-25
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Existing methods for calculating parallax information are prone to miscalculation due to the size of the matching area, which can be influenced by noise or the inclusion of objects at different distances within the area.

Method used

An information processing device that acquires images with different sensitivities and integrates parallax information based on sensitivity information for each pixel, adjusting the size of the matching area to minimize noise and ensure accurate parallax calculation.

Benefits of technology

The method suppresses miscalculation of parallax information by optimizing the matching area size based on sensitivity, resulting in highly accurate distance information.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024169101000001_ABST
    Figure 2024169101000001_ABST
Patent Text Reader

Abstract

To enable suppression of miscalculation of parallax information.SOLUTION: An information processing device disclosed herein acquires an image with different sensitivity for each pair of images with parallax and sensitivity information indicative of sensitivity of each pixel of the pair of images, obtains parallax information indicative of the magnitude of parallax based on the pair of images and the sensitivity information for each sensitivity, and integrates the parallax information for each sensitivity.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an information processing technique for acquiring parallax information from a captured image. [Background technology]

[0002] A known technique for acquiring distance is to calculate disparity information (parallax amount) representing the magnitude of disparity from a pair of images obtained by a pair of imaging devices, and convert the disparity amount into distance information using a predetermined coefficient. When calculating the disparity amount using a pair of images, a region-based corresponding point search called template matching is often used. In the corresponding point search, one of the pair of images is used as a standard image and the other is used as a reference image. A local matching area is set in the standard image with a point of interest as the center, and a matching area is set in the reference image with a reference point corresponding to the point of interest as the center. Then, while sequentially moving the reference point, a point where the correlation (i.e. similarity) between the pair of images is highest is searched for within the matching area, and the disparity amount is calculated based on the relative positional shift amount between the search point and the point of interest.

[0003] However, when the matching area is small, the influence of noise contained in the image may cause an erroneous calculation of the amount of parallax. On the other hand, when the matching area is large, the chances that subjects at different distances exist within the matching area increase, which may cause an erroneous calculation of the amount of parallax. Patent Document 1 discloses a technology for changing the size of the matching area according to depth information (distance information in the depth direction).

[0004] In recent years, imaging elements capable of simultaneously acquiring a signal for generating a high dynamic range image and a phase difference signal that can be used to calculate the distance to an imaged subject have also been proposed as information acquisition sensors for vehicle-mounted cameras and surveillance cameras. In particular, Patent Document 2 discloses a technology in which photodiodes are arranged on the left and right sides of a single on-chip lens to acquire a phase difference signal that can be used to calculate distance. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 10-283474 [Patent Document 2] International Publication No. 2019 / 102887 Summary of the Invention [Problem to be solved by the invention]

[0006] As described above, when calculating disparity information (amount of disparity) by performing a region-based corresponding point search, erroneous calculation of disparity information may occur depending on the size of the matching region. In response to this, in the above-mentioned Patent Document 1, the size of the matching region is changed, but merely changing the size of the matching region is often not enough to prevent erroneous calculation of disparity information.

[0007] Therefore, an object of the present invention is to make it possible to prevent the occurrence of erroneous calculation of disparity information. [Means for solving the problem]

[0008] The information processing device of the present invention is characterized by having an image acquisition means for acquiring images having different sensitivities for each pair of images having parallax, a sensitivity acquisition means for acquiring sensitivity information representing the sensitivity of each pixel of the pair of images, and a parallax acquisition means for acquiring, for each sensitivity, parallax information representing the magnitude of parallax based on the pair of images and the sensitivity information, and integrating the parallax information for each sensitivity. Effect of the Invention

[0009] According to the present invention, it is possible to prevent erroneous calculation of disparity information. [Brief description of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating an example of a system configuration according to a first embodiment. [Diagram 2] 4 is a flowchart of information processing according to the first embodiment. [Diagram 3] 11A and 11B are diagrams illustrating examples of the relationship between exposure time and sensitivity information, and between exposure time and the size of a matching area. [Figure 4] FIG. 11 is an explanatory diagram of parallax amount calculation using corresponding points search. [Diagram 5] FIG. 11 is a diagram illustrating an example of a system configuration according to a second embodiment. [Figure 6] 10 is a flowchart of information processing according to the second embodiment. [Figure 7] FIG. 1 is an explanatory diagram of an HDR image synthesis coefficient and HDR image synthesis. [Figure 8] FIG. 11 is an explanatory diagram of sensitivity distribution in the second embodiment. [Figure 9] FIG. 13 is an explanatory diagram of another example of an element that determines the collation area size. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The following embodiments do not limit the present invention, and not all of the combinations of features described in the present embodiments are necessarily essential to the solution of the present invention. The configuration of the embodiment may be appropriately modified or changed depending on the specifications of the device to which the present invention is applied and various conditions (conditions of use, environment of use, etc.). In addition, the embodiment may be configured by appropriately combining parts of the embodiments described below. In the following embodiments, the same or similar configurations and processing steps are given the same reference symbols, and duplicated explanations are omitted.

[0012] <First embodiment> FIG. 1 is a diagram showing an example of the configuration of a system according to the first embodiment. In the first embodiment, a configuration example is given in which a pair of captured images having parallax is acquired, the information processing unit 120 performs a corresponding point search based on the captured images, calculates parallax information indicating the magnitude of parallax, and obtains distance information based on the parallax information. In this embodiment, in order to acquire a pair of captured images having parallax, a pair of imaging units 100 and 110 is provided, the optical axes of which are arranged parallel to each other. In addition, in the first embodiment, the imaging units 100 and 110 each capture images with a plurality of exposure times to obtain captured images with a plurality of different sensitivities, and the information processing unit 120 also has a function of generating an image with a high dynamic range using the plurality of captured images. That is, in the system of this embodiment, a high dynamic range image is generated based on a plurality of images with different sensitivities obtained by capturing images with different exposure times, and distance information is obtained by calculating parallax information using the plurality of images with different exposure times (different sensitivities).

[0013] Since the imaging units 100 and 110 have the same configuration, only the configuration of the imaging unit 100 will be described here, and a description of the imaging unit 110 will be omitted. The lens 101 of the imaging unit 100 is configured with a plurality of lens groups, and forms a light image of a subject or the like on the imaging element 102. The optical system including the lens 101 also includes an aperture stop and the like. The optical system also includes a color filter disposed in front of the imaging surface of the imaging element 102. The color filter is a filter that transmits light of a specific wavelength according to its transmission characteristics, for example, a filter that transmits only visible light or non-visible light, or a filter that transmits only light of a specific color wavelength.

[0014] The imaging element 102 is composed of a CMOS (complementary metal oxide semiconductor) or a CCD (charge-coupled device), and their peripheral circuits. The imaging element 102 converts the optical image of the subject or the like formed by the lens 101 into an analog electric signal by photoelectric conversion for each of a plurality of pixels arranged in a lattice pattern, and further converts the signal into a digital signal by A / D (analog / digital) conversion. The imaging element 102 is controlled by an imaging control unit 122 of an information processing unit 120, which will be described later. Note that the image acquired by the imaging element 102 may be an image captured over the entire imaging surface of the imaging element 102, or may be an image captured over a partial region, such as the center, of the imaging surface. The transmission unit 103 outputs a digital signal of a captured image made up of a plurality of pixel signals acquired for each pixel by the image sensor 102, in accordance with control from an imaging control unit 122 of the information processing unit 120. In the following description, the digital signal of the captured image will be referred to simply as a captured image or an image.

[0015] The receiving section 121 of the information processing section 120 is an image acquisition section that receives the captured images output from the imaging section 100 and the imaging section 110, respectively. An imaging control unit 122 of the information processing unit 120 controls the imaging units 100 and 110 and the transmission unit 103. In order to ensure a dynamic range required for imaging in a dark place, backlight, or under varying illumination, the imaging control unit 122 controls exposure (charge accumulation) of the imaging element 102 so as to obtain a plurality of images (a plurality of images with different sensitivities) with a plurality of different exposure times.

[0016] In this embodiment, three types of exposure times are given as an example of the multiple different exposure times. In the following description, among the three types of exposure times, the long exposure time is called a long-second exposure, the short exposure time is called a short-second exposure, and the intermediate exposure time is called a medium-second exposure, and further, these are collectively referred to as the three exposure times. In the following description, an image captured by a long-second exposure is called a long-second exposure image, an image captured by a medium-second exposure is called a medium-second exposure image, and an image captured by a short-second exposure is called a short-second exposure image, and further, these are collectively referred to as the three exposure images. In this embodiment, the long-second exposure may be determined based on, for example, an exposure time that allows shooting at the lowest subject illuminance, and the medium-second exposure may be set to, for example, an exposure time of one or more cycles of the flicker cycle of the illumination light source. In this embodiment, the long-second exposure image by the long-second exposure can also be rephrased as a high-sensitivity image captured in a low-illuminance environment, while the short-second exposure image by the short-second exposure can also be rephrased as a low-sensitivity image captured in a high-illuminance environment. In this way, in the system of this embodiment, images are captured with three exposure times, namely, long exposure, medium exposure, and short exposure, so that images with three different sensitivities can be obtained. Note that the exposure time may be specified by a user or the like via an external instruction device (not shown) depending on, for example, the image capture scene. Also, the image capture control unit 122 may dynamically determine the exposure time according to the dynamic range of the image capture scene by acquiring, for example, a histogram relating to the brightness of the entire image or a part of the image.

[0017] The imaging control unit 122 controls the timing of reading out pixel signals from the imaging element 102 according to each of the above-mentioned long-second exposure, medium-second exposure, and short-second exposure times. The imaging control unit 122 can control the timing of reading out, for example, in units of one pixel, in units of an area consisting of a group of multiple pixels, or in units of rows. When controlling in units of rows, the imaging control unit 122 can, for example, thin out and read out the signals in the row (vertical) direction by one row at a time, or read out only the rows specified by a register or the like.

[0018] The correction unit 125 performs image correction processing such as correction of uneven brightness and aberration caused by peripheral light falloff due to the optical characteristics of the lens 101 of the imaging unit 100, and correction of uneven sensitivity of the imaging element 102. In addition, when the optical axes of the imaging units 100 and 110 are not parallel and one image of a pair of captured images is distorted relative to the other image, the correction unit 125 also performs image correction processing such as image deformation to correct the distortion of the image.

[0019] The parallax calculation unit 126 performs a corresponding point search using a pair of images obtained for each of the above-mentioned long-second exposure, medium-second exposure, and short-second exposure times and after image correction processing, and calculates parallax information (parallax amount) that indicates the magnitude of parallax. That is, the parallax calculation unit 126 calculates the parallax amount by the corresponding point search using a pair of long-second exposure images obtained by a pair of imaging units 100 and 110, the parallax amount by the corresponding point search using a pair of medium-second exposure images, and the parallax amount by the corresponding point search using a pair of short-second exposure images. In this embodiment, as an example of the corresponding point search, a process called template matching, which searches for corresponding points on a region basis from a pair of images, is given. When searching for corresponding points, the parallax calculation unit 126 sets one of the pair of images as a standard image and the other as a reference image, sets a local matching area centered on a point of interest in the standard image, and sets a matching area centered on a reference point corresponding to the point of interest in the reference image. Then, the disparity calculation unit 126 searches for a point in the matching area where the image correlation (similarity) between the standard image and the reference image is highest while sequentially moving the reference point, and calculates the amount of disparity based on the relative positional shift amount between the search point and the point of interest. The details of the corresponding point search process using the standard image and the reference image and the disparity calculation process in the disparity calculation unit 126 will be described later.

[0020] The sensitivity information determination unit 123 is a sensitivity acquisition unit that acquires information indicating each of the above-mentioned long-second exposure, medium-second exposure, and short-second exposure times from the imaging control unit 122, and generates information (referred to as sensitivity information) indicating the sensitivity of each pixel based on the information indicating each exposure time. In the case of this embodiment, the sensitivity information is used as element information for the subsequent size determination unit 124 to determine the size of the matching area (referred to as matching area size) in the corresponding point search process used when the parallax calculation unit 126 calculates the amount of parallax. The details of the generation process of the sensitivity information for each pixel based on the exposure time by the sensitivity information determination unit 123 will be described later.

[0021] Size determination unit 124 determines the size of the matching area (matching area size) when disparity calculation unit 126 searches for corresponding points, based on the sensitivity information determined by sensitivity information determination unit 123. Details of the process of determining the matching area size based on the sensitivity information for each pixel by size determination unit 124 will be described later. In this manner, in this embodiment, the disparity calculation unit 126 calculates the amount of disparity by searching for corresponding points as described above, using the matching region size determined by the size determination unit 124 based on the sensitivity information.

[0022] The synthesis processing unit 130 generates a high dynamic range image (HDR image) by synthesizing the three exposure images obtained for each of the above-mentioned long second exposure, medium second exposure, and short second exposure times using a predetermined synthesis coefficient (referred to as an image synthesis coefficient). The synthesis processing unit 130 has an image coefficient calculation unit 131 and an image synthesis unit 132 as components for generating an HDR image.

[0023] The image coefficient calculation unit 131 receives three exposure images acquired by either the image capture unit 100 or the image capture unit 110 for each exposure time of long exposure, medium exposure, and short exposure, and after image correction processing. In the case of this embodiment, the image coefficient calculation unit 131 receives three exposure images acquired by the image capture unit 100, namely, a long exposure image, a medium exposure image, and a short exposure image. Then, the image coefficient calculation unit 131 performs a ratio acquisition process to calculate an image synthesis coefficient that determines an image synthesis ratio for each pixel of each of the three exposure images, based on pixel signals at the same coordinates of one or more of the three exposure images. The image synthesis coefficient calculation process in the image coefficient calculation unit 131 will be described in detail later.

[0024] The image synthesis unit 132 performs a process of selecting pixels from one of the three exposure images (i.e., images of three sensitivities) described above according to an image synthesis coefficient for each pixel, and generates an HDR image from each selected pixel. Alternatively, the image synthesis unit 132 performs a process of synthesizing pixels of two or more images among the three exposure images (images of three sensitivities) according to an image synthesis coefficient for each pixel, thereby generating an HDR image. Details of the HDR image generation process in the image synthesis unit 132 will be described later. The HDR image generated in this manner is then output from the information processing unit 120.

[0025] Moreover, the synthesis processing unit 130 integrates the parallax amounts acquired by the parallax calculation unit 126 according to the three exposure times, long second exposure, medium second exposure, and short second exposure, by synthesizing them using a predetermined synthesis coefficient (hereinafter referred to as a parallax synthesis coefficient). In other words, the synthesis processing unit 130 integrates the parallax amounts according to the three sensitivities corresponding to the three exposure times by synthesizing them using a parallax synthesis coefficient. The synthesis processing unit 130 has a parallax coefficient calculation unit 133 and a parallax synthesis unit 134 as components for integrating the parallax amounts.

[0026] The parallax coefficient calculation unit 133 calculates a parallax synthesis coefficient for synthesizing the amount of parallax of the same pixel in images corresponding to the three exposure times of long exposure, medium exposure, and short exposure (i.e., images of three sensitivities) in the parallax calculation unit 126. Details of the parallax synthesis coefficient calculation process in the parallax coefficient calculation unit 133 will be described later.

[0027] The parallax synthesis unit 134 performs a parallax amount integration process to select the most likely parallax amount for each pixel based on the parallax synthesis coefficient calculated by the parallax coefficient calculation unit 133. Alternatively, the parallax synthesis unit 134 performs a parallax amount integration process to combine two or more parallax amounts for each pixel based on the parallax synthesis coefficient. The parallax amount integration process (parallax amount integration process) in the parallax synthesis unit 134 will be described in detail later. That is, in this embodiment, the size determination unit 124, the disparity calculation unit 126, the disparity coefficient calculation unit 133, and the disparity synthesis unit 134 constitute a disparity acquisition unit that acquires and integrates disparity information for each sensitivity based on each pixel signal of a pair of images and sensitivity information for each pixel.

[0028] The distance calculation unit 127 is a distance acquisition unit that calculates distance information by performing a process of converting the amount of parallax after synthesis (integration) by the parallax synthesis unit 134 into a defocus amount using a predetermined conversion coefficient. The distance information calculated by the distance calculation unit 127 is then output from the information processing unit 120.

[0029] As described above, the information processing unit 120 according to the first embodiment generates an HDR image by combining three exposure images (images with three sensitivities) acquired for each pixel according to the three exposure times. The information processing unit 120 also combines (integrates) the amounts of parallax calculated corresponding to the three exposure images (images with three sensitivities) and acquires distance information based on the combined amounts of parallax.

[0030] FIG. 2 is a flowchart showing an example of the flow of information processing executed by the information processing unit 120 according to the first embodiment. In step S201, the imaging control unit 122 determines (sets) an exposure time of N seconds in order to ensure a dynamic range required for imaging in a dark place, against light, or under varying illumination. In this embodiment, the imaging control unit 122 sequentially sets the above-mentioned three exposure times, long exposure, medium exposure, and short exposure, as the N-second exposure time for the imaging units 100 and 110. In this embodiment, it is assumed that the long exposure is set first.

[0031] In the next step S202, the imaging control unit 122 performs exposure (charge accumulation) control on the image sensors 102 of the imaging units 100 and 110 based on the exposure time of the long exposure. As a result, the imaging units 100 and 110 each perform long exposure, and the receiving unit 121 acquires (receives) these long exposure images as a pair of images. Next, in step S203, the sensitivity information determination unit 123 acquires sensitivity information based on the information on the exposure time determined in step S201.

[0032] Fig. 3(a) is a diagram showing the correspondence between exposure time and sensitivity information, and the sensitivity information determination unit 123 acquires 8 levels (3 bits) of sensitivity information according to the exposure time determined by the imaging control unit 122. For example, when the long-second exposure is 1 / 30 seconds, the sensitivity information determination unit 123 acquires "5" as a value indicating the sensitivity information corresponding to the long-second exposure based on the correspondence shown in Fig. 3(a). Also, when the short-second exposure is 1 / 300 seconds, the sensitivity information determination unit 123 acquires "1" as a value indicating the sensitivity information corresponding to the short-second exposure based on the correspondence in Fig. 3(a).

[0033] Next, in step S204, the size determination unit 124 determines the matching area size for searching for corresponding points, which is used when the disparity calculation unit 126 calculates the amount of disparity in subsequent step S205, based on the sensitivity information determined by the sensitivity information determination unit 123.

[0034] Here, it is desirable that the matching area used in the corresponding point search when the parallax calculation unit 126 calculates the amount of parallax in the subsequent step S205 is the minimum necessary size including the texture of the captured object or the like. However, if the size of the matching area is small, the amount of parallax may be erroneously calculated due to the influence of noise generated in the image. On the other hand, if the size of the matching area is increased, the chances that objects with different distances exist in the matching area increase, and the amount of parallax may be erroneously calculated. Also, for example, in a high-sensitivity image obtained by capturing an image of a low-illumination object or the like with long-second exposure, the noise level relative to the signal level is likely to be high, which makes it easy for erroneous search to occur in the corresponding point search. In this case, for example, if the matching area size is increased by a factor of n, the noise level relative to the signal level can be expected to improve by the square root of n (√(n)). For this reason, in step S204, the size determination unit 124 determines the matching area size to be larger as the exposure time increases (i.e., as the image becomes more sensitive). On the other hand, in a low-sensitivity image obtained by capturing an image of a high-illumination subject with a short-second exposure, the noise level relative to the signal level is low, and the possibility of an erroneous search occurring in the corresponding point search is reduced. Therefore, in step S204, the size determination unit 124 determines the matching region size to be the minimum necessary size that includes the texture of the subject, etc., as the exposure time becomes shorter (i.e., as the image becomes less sensitive).

[0035] FIG. 3(b) is a diagram showing the correspondence between the sensitivity information and the matching area size. The size determination unit 124 uses the sensitivity information determined according to the exposure time as described above as element information when determining the matching area size, and determines the matching area size by referring to the correspondence in FIG. 3(b) based on the sensitivity information. For example, when the value indicating the sensitivity information is "5", the size determination unit 124 determines the matching area size (15×15) of 15 pixels in the X direction (horizontal direction) and 15 pixels in the Y direction (vertical direction) based on the correspondence in FIG. 3(b). Also, for example, when the value of the sensitivity information is "1", the size determination unit 124 determines the matching area size (7×7) of 7 pixels in the X direction and 7 pixels in the Y direction based on the correspondence in FIG. 3(b).

[0036] Next, in step S205, the disparity calculation unit 126 uses a pair of images obtained by long-second exposure in the pair of imaging units 100, 110 to perform a corresponding point search based on the matching area size determined by the size determination unit 124, and obtains an evaluation value. Then, in the next step S206, the parallax calculation unit 126 obtains the amount of parallax corresponding to the exposure time of N seconds (long exposure at this point) based on the evaluation value obtained in the corresponding point search in step S205.

[0037] 4(a) to 4(d) are diagrams used to explain an example of the evaluation value acquisition process by searching for corresponding points in step S205 and the parallax amount acquisition process in step S206. When searching for corresponding points, the parallax calculation unit 126 sets one of a pair of images as a base image 400 and the other as a reference image 401. FIG. 4(a) shows an example of the base image 400, and FIG. 4(b) shows an example of the reference image 401.

[0038] The disparity calculation unit 126 sequentially moves a matching area 405 set within a search range 404 of the reference image 401 relative to a matching area 403 including a focus point 402 of the base image 400, and sequentially acquires evaluation values ​​representing the image correlation between the matching area 403 and the matching area 405. Note that a known method can be used to calculate the evaluation value representing the image correlation. For example, there is a method called SAD (Sum of Absolute Difference), which uses the sum of absolute differences between pixel values ​​in the matching area as the evaluation value, and SSD (Sum of Squared Difference), which uses the sum of squared differences as the evaluation value.

[0039] Fig. 4(c) is a diagram showing an example of a matching area 403 set for the reference image 400. In this example, the size of the matching area 403 is determined to be, for example, 5 pixels in the X direction and 5 pixels in the Y direction (5 x 5). Fig. 4(d) is a diagram showing an example of a search range 404 and a matching area 405 in the reference image 401. In this example, the search range 404 is, for example, "64", that is, 64 pixels in the Y direction (horizontal direction).

[0040] In step S205, disparity calculation unit 126 sequentially acquires evaluation values ​​representing the image correlation between matching region 403 and matching region 405 while sequentially moving matching region 405 by one pixel in the Y direction within search range 404 of reference image 401. That is, disparity calculation unit 126 sequentially acquires evaluation values ​​representing the image correlation between matching region 405(0), matching region 405(1), ..., matching region 405(63), which are shifted by one pixel in search range 404 in FIG. 4(d), and matching region 403 of standard image 400.

[0041] Furthermore, in step S206, the disparity calculation unit 126 identifies the position of the matching region 405 corresponding to the evaluation value with the highest image correlation among the evaluation values ​​acquired in that order. Then, the disparity calculation unit 126 acquires the horizontal distance between the reference point of the identified matching region 405 and the point of interest 402 as the amount of disparity. Note that in the case of evaluation values ​​based on SAD or SSD exemplified in this embodiment, the matching region 405 with the highest image correlation is the region corresponding to the smallest evaluation value among the evaluation values ​​acquired in step S205.

[0042] Next, in step S207, the imaging control unit 122 judges whether the exposures for the three exposure times, long second exposure, medium second exposure, and short second exposure, have been completed. In the description up to this point, since only the long second exposure has been completed, it is judged that the exposure is incomplete at this point, and therefore the imaging control unit 122 returns the process to step S201 and sets the exposure time of N seconds to the next exposure time (medium second exposure). Thereafter, the processes of steps 202 to S206 are performed in the case of medium second exposure as well, as described above. Then, if it is judged that the exposure is incomplete in step S207, the imaging control unit 122 returns to the process of step S201 and sets the exposure time to the short second exposure. Similarly, the processes of steps 202 to S206 are performed in the case of short second exposure. Then, if the processes of steps 202 to S206 have been performed for the short second exposure, it is judged that the exposure is completed in step S207, and the process of the information processing unit 120 proceeds to step 208.

[0043] In step S208, the image coefficient calculation unit 131 of the synthesis processing unit 130 calculates an image synthesis coefficient for each pixel based on pixel signals at the same coordinates of one or more of the three exposure images obtained by the three exposure times described above. For example, the image coefficient calculation unit 131 refers to the signal level of the medium-second exposure image, sets three divided sections using two predetermined thresholds, and calculates an image synthesis coefficient for selecting one of the three exposure images in each of the divided sections.

[0044] Here, if the two predetermined thresholds are a first threshold and a second threshold, the first threshold and the second threshold are set as follows. The first threshold value is set to a value equivalent to the signal level before the long exposure image from the minimum environmental illuminance reaches the saturation signal level. The second threshold value is set to a value that corresponds to the signal level before the medium-second exposure image reaches the saturation signal level.

[0045] Furthermore, the image coefficient calculation unit 131 sets the gain coefficients so that the signal levels of the three exposure images have a linear relationship with the environmental illuminance. Note that since it is expected that the first threshold value and the second threshold value may deviate due to individual differences in the image sensor 102, the image coefficient calculation unit 131 may calculate the image synthesis coefficient so as to obtain a weighted average of the signal levels of the respective exposure images near these threshold values.

[0046] Next, in step S209, the image synthesis unit 132 synthesizes pixel signals of the three exposure images (images with three sensitivities) based on the image synthesis coefficient calculated by the image coefficient calculation unit 131, to generate an HDR image.

[0047] Next, in step S210, the parallax coefficient calculation unit 133 of the synthesis processing unit 130 calculates a parallax synthesis coefficient for determining a parallax synthesis ratio for the parallax amount of the same pixel in each of the three exposure images described above. For example, the parallax coefficient calculation unit 133 calculates a parallax synthesis coefficient for selecting the parallax amount corresponding to the exposure image determined to have the most texture, using the amount of texture in the reference image used in the corresponding point search as an index for each of the three exposure images. Alternatively, as a simpler method, the image synthesis coefficient calculated in step S208 may be used as the parallax synthesis coefficient. In this case, the parallax synthesis coefficient is set to a parallax synthesis coefficient for selecting one of the parallax amounts of the same pixel corresponding to the three exposure times in each divided section obtained by dividing each of the three exposure images into three sections by the first threshold value and the second threshold value. In this case, it is also assumed that the first threshold value and the second threshold value may deviate due to individual differences in the image sensor 102, and therefore the parallax coefficient calculation unit 133 may weightedly average the parallax amounts corresponding to each exposure image near those threshold values.

[0048] Next, in step S211, the parallax synthesis unit 134 selects parallax amounts corresponding to the three exposure times for each pixel, or synthesizes two or more parallax amounts for each pixel, based on the parallax synthesis coefficient calculated by the parallax coefficient calculation unit 133, to integrate the parallax amounts. In other words, the parallax synthesis unit 134 selects parallax amounts corresponding to the three sensitivities according to the three exposure times for each pixel, or synthesizes two or more parallax amounts for each pixel, based on the parallax synthesis coefficient, to integrate the parallax amounts. Thereafter, the distance calculation unit 127 converts the parallax amounts into defocus amounts using a predetermined conversion coefficient, thereby calculating distance information.

[0049] As described above, the information processing unit 120 of this embodiment can calculate the amount of parallax with reduced noise in the parallax calculation using three exposure images acquired with three different exposure times (long exposure, medium exposure, and short exposure). This allows the information processing unit 120 to acquire highly accurate distance information.

[0050] The process of the flowchart in FIG. 2 described in this embodiment may be applied to an image obtained from a partial area of ​​the image sensor. The matching area size described in this embodiment is an example, and is not limited to the above example. The size determination unit 124 may determine a matching area size of 7 pixels in the X direction and 3 pixels in the Y direction (7×3) according to the shape of the subject, or a matching area size of 3 pixels in the X direction and 7 pixels in the Y direction (3×7). In addition, the search range may be set to a search range appropriate for, for example, the configuration of the imaging unit (focal length, the installation distance between a pair of imaging units, etc.) or the ranging range, or the search direction may be determined, for example, according to the positions and optical axes of multiple imaging units. Furthermore, the number of imaging units is not limited to two, and three or more imaging units may be combined. In this case, disparity information is acquired based on images from the multiple imaging units. The number of imaging units is not limited to two or more, and may be one imaging unit having an imaging element in which pixels capable of acquiring a phase difference are arranged.

[0051] <Second embodiment> Next, a system according to a second embodiment will be described. In the first embodiment, an example is given in which disparity information (amount of disparity) is acquired using a pair of images captured by a pair of imaging units 100 and 110. In the second embodiment, an example is given in which a single imaging unit is provided, and the imaging unit has an imaging element that simultaneously acquires a phase difference signal that can be used to calculate the distance to a subject, and pixel signals with multiple sensitivities that can be used to generate a high dynamic range HDR image.

[0052] Here, it is generally known that pixels with higher sensitivity are more susceptible to noise, and pixels with lower sensitivity are less susceptible to noise. For this reason, in an HDR image in which pixel signals of different sensitivities are combined, it is considered that the influence of noise differs for each pixel. When disparity information (amount of disparity) is calculated by performing corresponding point search using such an HDR image in which the influence of noise differs for each pixel, disparity information including errors due to the influence of noise that differs for each pixel is calculated. For this reason, in this embodiment, an example in which the influence of noise on the calculation of disparity information can be reduced in an example in which disparity information (amount of disparity) is calculated by performing corresponding point search using an HDR image will be described.

[0053] 5 is a diagram showing an example of the configuration of a system including an imaging unit 500 and an information processing unit 520 according to the second embodiment. In the case of the system of the second embodiment, the imaging unit 500 acquires a plurality of images having a plurality of different sensitivities and a phase difference signal for each pixel, and outputs the images to the information processing unit 520. The information processing unit 520 generates an HDR image based on the plurality of images, and further calculates parallax information (parallax amount) representing the magnitude of parallax based on the HDR image and the sensitivity information. In FIG. 5, the same functional units as those in the configuration of FIG. 1 described above are assigned the same reference symbols as those in FIG. 1, and duplicated explanations will be omitted.

[0054] The lens 101 of the imaging section 500 according to the second embodiment is configured similarly to the lens 101 in FIG. In the second embodiment, the image sensor 502 is configured such that one pixel is composed of multiple photodiodes (PDs), and the light receiving area and exposure time per pixel can be varied by combining the multiple PDs. That is, the image sensor 502 is configured as an image sensor capable of acquiring pixel signals corresponding to multiple different sensitivities by varying the light receiving area and exposure time per pixel. In this embodiment, three sensitivities, high sensitivity, medium sensitivity, and low sensitivity, are given as examples of the multiple different sensitivities.

[0055] Furthermore, the image sensor 502 is configured such that at least two PDs are arranged under one on-chip lens, and light beams passing through different regions on the pupil of the lens 101 are guided to different PDs. This allows the image sensor 502 to acquire optical images generated by light beams passing through different pupil regions, and to acquire pixel signals having a phase difference (parallax) according to the defocus amount along the pupil division direction. Hereinafter, an optical image generated by a light beam passing through one of the different pupil regions will be referred to as image A, and an optical image generated by a light beam passing through the other different pupil region will be referred to as image B.

[0056] In this way, the imaging element 502 according to the second embodiment is capable of acquiring pixel signals corresponding to three sensitivities (high sensitivity, medium sensitivity, and low sensitivity) and having a phase difference, and acquiring images A and B having parallax from each pixel signal. Note that the imaging element 502 of the imaging unit 500 is not limited to the above-mentioned configuration, and may have any configuration that can acquire pixel signals having a phase difference. For example, it may be configured to capture images A and B using a plurality of imaging elements, such as a stereo camera.

[0057] The transmission section 103 of the imaging section 500 outputs pixel signals of the A image and the B image acquired by the imaging element 502 under control of the imaging control section 122 of the information processing section 520 . The receiving unit 521 of the information processing unit 520 of the second embodiment receives the pixel signals output from the imaging unit 500. That is, the receiving unit 521 receives pixel signals having parallax between images A and B corresponding to three sensitivities, namely, high sensitivity, medium sensitivity, and low sensitivity. The correction unit 125 performs the image correction process for the images A and B in the same manner as described above.

[0058] The imaging control unit 122 of the information processing unit 520 controls the imaging element 502 and the transmission unit 103 of the imaging unit 500. In this embodiment, the imaging control unit 122 controls the light receiving area per pixel of the imaging element 502 and the exposure time so that images with different sensitivities can be acquired in order to ensure a dynamic range required for imaging in a dark place, against light, or under varying illumination. Note that in the second embodiment, as in the above, the exposure time may be specified by a user or the like via an external instruction device (not shown) depending on the imaging scene, or may be dynamically determined according to the dynamic range of the imaging scene by acquiring a histogram related to the brightness of the entire image or a part of it.

[0059] The synthesis processing unit 530 of the second embodiment synthesizes the A images at the three sensitivities described above using a predetermined synthesis coefficient (image synthesis coefficient) to generate an HDR image with a high dynamic range in the A image. Similarly, the synthesis processing unit 530 synthesizes the B images at the three sensitivities using a predetermined synthesis coefficient (image synthesis coefficient) to generate an HDR image with a high dynamic range in the B image. The synthesis processing unit 530 according to the second embodiment includes an image coefficient calculation unit 531 and an image synthesis unit 532.

[0060] The image coefficient calculation unit 531 calculates an image synthesis coefficient that determines a pixel synthesis ratio for each pixel for three sensitivities, high sensitivity, medium sensitivity, and low sensitivity, for the image A. Similarly, the image coefficient calculation unit 531 calculates an image synthesis coefficient that determines an image synthesis ratio for each pixel for the three sensitivities for the image B. Details of the calculation process of the image synthesis coefficient in the image coefficient calculation unit 531 will be described later.

[0061] The image synthesis unit 532 generates an HDR image of the image A by selecting one of the pixels of the three sensitivities of the image A for each pixel based on the image synthesis coefficient calculated by the image coefficient calculation unit 531. Similarly, the image synthesis unit 532 generates an HDR image of the image B by selecting one of the pixels of the three sensitivities of the image B for each pixel based on the image synthesis coefficient. Alternatively, the image synthesis unit 532 generates an HDR image of the image A by synthesizing pixels of two or more sensitivities among the pixels of the three sensitivities of the image A according to the image synthesis coefficient. Similarly, the image synthesis unit 532 generates an HDR image of the image B by synthesizing pixels of two or more sensitivities among the pixels of the three sensitivities of the image B according to the image synthesis coefficient. Then, one of the HDR images of the images A and B (the HDR image of the image A in this embodiment) is output from the information processing unit 520 as an HDR image. Moreover, the pair of HDR images of the images A and B generated by the image synthesis unit 532 is sent to the parallax calculation unit 526 .

[0062] The sensitivity information determination unit 523 of the second embodiment acquires one of the image synthesis coefficients calculated for each pixel for the A image and the B image by the image coefficient calculation unit 531 (the image synthesis coefficient calculated for the A image in this embodiment). Then, the sensitivity information determination unit 523 acquires a sensitivity distribution using the image synthesis coefficients for each pixel for the three sensitivities, and determines sensitivity information based on information on the sensitivity distribution. Note that the sensitivity distribution may be generated by the image coefficient calculation unit 531, and in this case, the sensitivity information determination unit 523 determines sensitivity information based on the sensitivity distribution. Details of the sensitivity information determination process in the sensitivity information determination unit 523 will be described later.

[0063] Size determination unit 124 determines the matching region size for the corresponding point search in the same manner as described above, based on the sensitivity information determined by sensitivity information determination unit 523. That is, size determination unit 124 determines the matching region size so that the larger the value of the sensitivity information, the larger the matching region size. The disparity calculation unit 526 performs a corresponding point search in the same manner as described above based on the pair of HDR images of images A and B generated by the synthesis processing unit 530 and the matching area size determined by the size determination unit 124, and calculates disparity information (amount of disparity). The distance calculation unit 127 calculates distance information by converting the disparity information calculated by the disparity calculation unit 526 into a defocus amount using a predetermined conversion coefficient. Then, the calculated distance information is output from the information processing unit 520.

[0064] FIG. 6 is a flowchart showing an example of the flow of information processing executed by the information processing unit 520 according to the second embodiment. In step S601, the imaging control unit 122 determines a plurality of exposure times as described above in order to ensure a dynamic range required for imaging in a dark place, against light, or under varying illumination.

[0065] Next, in step S602, the synthesis processing unit 530 acquires pixel signals of three sensitivities, namely high sensitivity, medium sensitivity, and low sensitivity, in the A image and the B image, which have been output from the imaging unit 500 described above, received by the receiving unit 521, and further subjected to image correction processing by the correction unit 125. Furthermore, in step S603, the synthesis processing unit 530 branches the processing into the A-image and the B-image, and proceeds to step S604 and subsequent steps for the pixel signals of the A-image, and proceeds to step S608 and subsequent steps for the pixel signals of the B-image.

[0066] When proceeding to step S604, the image coefficient calculation unit 531 of the synthesis processing unit 530 calculates image synthesis coefficients for each of the high-sensitivity, medium-sensitivity, and low-sensitivity pixels of the image A. Hereinafter, the high-sensitivity pixels will be referred to as high-sensitivity pixels, the medium-sensitivity pixels as medium-sensitivity pixels, and the low-sensitivity pixels as low-sensitivity pixels, and these will also be collectively referred to as three sensitivity pixels.

[0067] Here, the image coefficient calculation unit 531 determines the blending ratios by which the signal values ​​of the three sensitivity pixels are multiplied, and the offset values ​​by which the signal values ​​of the medium and low sensitivity pixels are added, based on the signal value of the medium sensitivity pixel. Hereinafter, the blending ratio by which the signal value of the high sensitivity pixel is multiplied is defined as αH, the blending ratio by which the signal value of the medium sensitivity pixel is multiplied is defined as αM, and the blending ratio by which the signal value of the low sensitivity pixel is multiplied is defined as αL. Also, the offset value to be added to the signal value of the medium sensitivity pixel is defined as OM, and the offset value to be added to the signal value of the low sensitivity pixel is defined as OL.

[0068] FIG. 7(a) is a graph showing the signal values ​​of three sensitivity pixels (high sensitivity pixel 700H, medium sensitivity pixel 700M, low sensitivity pixel 700L) when the horizontal axis is the environmental illuminance. In this embodiment, the maximum value of the pixel signal value is set to "4095", and is expressed as "2^12-1" in FIG. 7(a). In FIG. 7(a), the threshold value TH1 represents the first threshold value described in the first embodiment, and the threshold value TH2 represents the second threshold value. The threshold value TH1(M) in FIG. 7(a) is a threshold value that is set based on the signal value of the medium sensitivity pixel 700M within a range where the signal value of the medium sensitivity pixel 700M rises and before the signal value of the high sensitivity pixel 700H is saturated. The threshold value TH1(H) in FIG. 7(a) is a threshold value that is set based on the signal value of the high sensitivity pixel 700H at the same environmental illuminance when the threshold value TH1(M) is set. The threshold value TH2(M) in Fig. 7(a) is a threshold value that is set based on the signal value of the medium sensitivity pixel 700M within a range where the signal value of the low sensitivity pixel 700L rises and before the signal value of the medium sensitivity pixel 700M becomes saturated. The threshold value TH2(L) in Fig. 7(a) is a threshold value that is set based on the signal value of the low sensitivity pixel 700L at the same environmental illuminance as when the threshold value TH2(M) was set. Each of these threshold values ​​is calculated based on information indicating the correspondence relationship obtained in advance by measuring the signal values ​​of the three sensitivity pixels with respect to the environmental illuminance.

[0069] FIG. 7(b) is a diagram showing, as a graph, the ideal signal values after performing the synthesis process of the HDR image with the environmental illuminance on the horizontal axis. In the present embodiment, the maximum signal value in the ideal signal values after the synthesis process of the HDR image is set to "1048575", and in FIG. 7(b), it is expressed as "2^20 - 1". As shown in FIG. 7(b), the change amount of the signal value is approximately linear with respect to the environmental illuminance. As shown in FIG. 7(b), the image coefficient calculation unit 531 determines the synthesis ratios (αH, αM, αL) corresponding to the three respective sensitivities (high sensitivity 710H, medium sensitivity 710M, low sensitivity 710L) such that the change amount of the signal value of each sensitivity pixel is linear with respect to the environmental illuminance.

[0070] More specifically, when the signal value A(H) of the high-sensitivity pixel 700H is less than the threshold value TH1(H) (A(H) < TH1(H)), the image coefficient calculation unit 531 determines the synthesis ratio αH such that the signal value A(H) approaches the ideal signal value. Also, when the signal value A(M) of the medium-sensitivity pixel 700M is greater than or equal to the threshold value TH1(M) (A(M) ≧ TH1(H)) and less than the threshold value TH2(M) (A(M) < TH2(M)), the image coefficient calculation unit 531 determines the synthesis ratio αM such that the signal value A(M) approaches the ideal signal value. Further, the image coefficient calculation unit 531 determines the offset value OM to be added to the signal value A(M) of the medium-sensitivity pixel 700M so that the signal values of the threshold values TH1(H) and TH1(M) in the ideal signal values approximately match. Also, when the signal value A(L) of the low-sensitivity pixel 700L is greater than or equal to the threshold value TH2(L) (A(L) ≧ TH2(L)), the image coefficient calculation unit 531 determines the synthesis ratio αL such that the signal value A(L) approaches the ideal signal value. Further, the image coefficient calculation unit 531 determines the offset value OL to be added to the signal value A(L) of the low-sensitivity pixel 700L so that the signal values of the threshold values TH2(M) and TH2(L) in the ideal signal values approximately match.

[0071] In the above manner, the image coefficient calculation unit 531 determines the synthesis ratios (αH, αM, αL) to be multiplied respectively to the signal values of the three sensitivity pixels, and the offset values (OM, OL) to be added. These may be determined in advance by measuring the signal values of the respective sensitivity pixels with respect to the ambient illuminance and according to the dynamic range after the synthesis process of the HDR image.

[0072] Furthermore, the image coefficient calculation unit 531 determines whether to directly select the synthesis ratios (αH, αM, αL) and the offset values (OM, OL) determined in advance as described above, or to replace the values with "0". Specifically, the image coefficient calculation unit 531 determines whether to select or replace based on the following conditions.

[0073] When A(M) < TH1(M), αM, αL, OM, OL are replaced with "0". When TH1(M) ≤ A(M) < TH2(M), αH, αL, OL are replaced with "0". When TH2(M) > A(M), αH, αM, OM are replaced with "0".

[0074] Next, in step S605, the image synthesis unit 532 performs the synthesis process of the HDR image according to the image synthesis coefficients calculated by the image coefficient calculation unit 531. Here, assuming that the signal value after the synthesis process of the HDR image is A(HDR), the synthesis process of the HDR image in the image synthesis unit 532 can be expressed by the following formula (1).

[0075] A(HDR) = A(H) * αH + A(M) * αM + OM + A(L) * αL + OL Formula (1)

[0076] The threshold value, the combination ratio, and the offset value set as described above are not limited to the above-mentioned examples. For example, the image coefficient calculation unit 531 may obtain temperature information around the image sensor and change the threshold value, the combination ratio, and the offset value in accordance with the sensitivity characteristics that change according to the temperature characteristics. The image coefficient calculation unit 531 may change the threshold value, the combination ratio, and the offset value in accordance with the exposure time set by the image capture control unit 122 in step S601. In addition, in the present embodiment, an example in which the combination ratio and the offset value are one for the signal of each sensitivity pixel have been given, but they may have multiple values ​​according to the signal value of each sensitivity pixel, and this may further improve the linearity. In addition, in the present embodiment, an example in which the combination ratio and the offset value are replaced with "0" according to the threshold value have been given, but the combination ratio may be determined so that the signal values ​​of two or more sensitivity pixels are combined, and this makes it possible to suppress the nonlinearity of the signal value that is expected to occur when each sensitivity is switched. In this case, the above-mentioned threshold value is not limited to two, but it is preferable to set three or more.

[0077] Next, in step S606, the sensitivity information determination unit 523 obtains a sensitivity distribution based on the image synthesis coefficient acquired from the image coefficient calculation unit 531, and determines sensitivity information for an area including all or a part of the matching area used in the corresponding point search based on the information on the sensitivity distribution. In this example of the present embodiment, the sensitivity information determination unit 523 refers to the image synthesis coefficient of the image A calculated by the image coefficient calculation unit 531, and calculates the sensitivity information of the most frequent value among the three sensitivities mentioned above.

[0078] FIG. 8 is a diagram showing an example of the sensitivity distribution obtained by the sensitivity information determination unit 523 in the second embodiment, and shows an area (sensitivity information calculation area) of 11 pixels in the X direction (horizontal direction) and 11 pixels in the Y direction (vertical direction) including the collation area. In FIG. 8, the pixel in which the low sensitivity pixel 801 is selected in step S604 is indicated by "1", the pixel in which the medium sensitivity pixel 802 is selected is indicated by "2", and the pixel in which the high sensitivity pixel 803 is selected is indicated by "3". In the case of the example of FIG. 8, the number of pixels in which the high sensitivity pixel 803 is selected is 85, the number of pixels in which the medium sensitivity pixel 802 is selected is 23, and the number of pixels in which the low sensitivity pixel 801 is selected is 13. Here, if the number of high sensitivity pixels is NUM(H), the number of medium sensitivity pixels is NUM(M), and the number of low sensitivity pixels is NUM(L), and 3-bit sensitivity information is determined based on the following formula (2), for example, the sensitivity information AVE in the sensitivity distribution of FIG. 8 is "6" (rounded off to the nearest whole number).

[0079] AVE = (7*NUM(H) + 3.5*NUM(M) + 0*NUM(L)) / (number of pixels in sensitivity information calculation area) Formula (2)

[0080] Next, in step S607, size determination unit 124 determines the size of the matching area for the corresponding point search based on the sensitivity information determined by sensitivity information determination unit 523, with reference to the correspondence relationship in FIG. 3(b) in the same manner as in the first embodiment described above. In the case of this embodiment, the matching area size is determined based on the sensitivity information for each pixel, so the matching area size is determined based on the sensitivity information of the point of interest (pixel of interest) in the corresponding point search and the pixels surrounding it. After the processing of step S607, the processing of information processing unit 520 proceeds to step S610.

[0081] Here, it is desirable that the matching area size is the minimum size necessary to include the texture of the object to be captured, but as explained in the first embodiment, when the matching area size is small, the effect of noise occurring in the image may cause erroneous calculation of the parallax amount. Similarly, as explained in the first embodiment, when the matching area size is increased, the chances that objects with different distances exist in the matching area increase, and erroneous calculation of the parallax amount may occur. Furthermore, when capturing an object with low illumination using high sensitivity pixels, the noise level relative to the signal level increases, and erroneous search is likely to occur in the corresponding point search. In this case, for example, when the matching area size is multiplied by n, the noise level relative to the signal level can be improved by the square root of n (√(n)). Therefore, in the second embodiment, the size determination unit 124 determines the matching area size so that the higher the sensitivity of the pixel, the larger the matching area size. On the other hand, when capturing an object with high illumination using low sensitivity pixels, the noise level relative to the signal level decreases, so the size determination unit 124 determines the area matching size to be the minimum necessary to include the texture of the object. In addition, while it is desirable to perform the calculation for each pixel in the process of step S607, it may be calculated for each region, and in that case, the effect of improving the noise level can also be expected. In addition, in this embodiment, an example was given in which the collation region size is determined based on the sensitivity information calculated as an average value as in formula (2), but the present invention is not limited to this, and may be determined, for example, according to the number of pixels.

[0082] The processing up to this point is from calculation of the image synthesis coefficient using the A image to the processing for determining the matching region size. On the other hand, if the image is determined to be a B image in step S603 described above and the process proceeds to step S608, the image coefficient calculation unit 531 performs processing to calculate an image synthesis coefficient for each pixel based on the medium sensitivity signal level of the B image, similar to the example of the A image described above. Furthermore, in the next step S609, the image synthesis unit 532 performs synthesis processing of the HDR image of the image B based on the image synthesis coefficient calculated in step S608. After the processing of this step S609, the processing of the information processing unit 520 proceeds to step S610.

[0083] When the process proceeds to step S610, the parallax calculation unit 526 calculates parallax information using the HDR image of the image A generated in the process of step S605 and the HDR image of the image B generated in the process of step S609. That is, the parallax calculation unit 526 uses the pair of HDR images of the images A and B to obtain an evaluation value by searching for corresponding points in the same manner as described in the first embodiment, and calculates parallax information (amount of parallax) based on the evaluation value.

[0084] As described above, according to the second embodiment, disparity information (amount of disparity) with reduced noise effects is obtained using an HDR image generated for image A consisting of pixel signals with three different sensitivities, and an HDR image similarly generated for image B. The second embodiment also makes it possible to obtain highly accurate distance information.

[0085] In this embodiment, the matching area size is determined based on the sensitivity information of the point of interest (pixel of interest) in the corresponding point search and its surrounding pixels as described above, but the matching area size may be determined based on the pixel signal value of the HDR image. In this case, the size determination unit 124 may determine that the higher the level of the pixel signal value, the lower the sensitivity, and reduce the matching area size. For example, the size determination unit 124 determines the sensitivity information by using the pixel signal values ​​of the HDR image corresponding to the thresholds TH1 and TH2 described in FIG. 7(a) as the respective thresholds. As an even simpler example, the size determination unit 124 may vary the matching area size depending on the level of the pixel signal value of the HDR image. The process of the flowchart of FIG. 6 described in this embodiment may also be applied to an image obtained from a partial area of ​​the image sensor.

[0086] <Another example of determining the collation area size> In the above-mentioned embodiment, an example was given in which sensitivity information was used as an element for determining the matching area size, but the matching area size may be determined based on various other elements different from the sensitivity information. For example, it is also possible to determine the matching area size based on element information related to image capture as various other elements other than the sensitivity information. In this case, the information processing unit in FIG. 1 includes an element acquisition unit that acquires at least one of the element information exemplified in FIG. 9 instead of the above-mentioned sensitivity information determination unit, and the size determination unit determines the matching area size based on the element information.

[0087] Fig. 9 is a diagram showing an example of the correspondence relationship between each element information and the size of the matching area. In Fig. 9, the element information includes, for example, exposure time, pixel sensitivity (S / N), the amount of defective pixels of the sensor, lens F-number, the amount of optical distortion, environmental illuminance, contrast of the subject, distance to the subject, weather, size of the subject, amount of movement of the subject, temperature and humidity around the sensor, etc.

[0088] For example, the longer the exposure time, which is one of the factors related to image capture, the greater the likelihood of subject blur or camera shake, and the lower the contrast, which is thought to have a greater impact on the calculation of the parallax amount. In this case, the size determination unit uses the exposure time as a factor in determining the matching area size, and determines the matching area size to be larger the longer the exposure time, and conversely, determines the matching area size to be smaller the shorter the exposure time.

[0089] For example, it is believed that the lower the pixel signal-to-noise ratio (pixel sensitivity (S / N)), which is one of the factors related to image capture, the greater the impact on the calculation of the amount of parallax, and the pixel sensitivity (S / N) can be determined from the characteristics of the image sensor. In this case, the size determination unit uses the pixel sensitivity (S / N) as a factor in determining the matching area size, and determines the matching area size to be larger the lower the pixel sensitivity (S / N), and conversely, determines the matching area size to be smaller the higher the pixel sensitivity (S / N).

[0090] For example, it is considered that the greater the number of defective pixels in a sensor (imaging element), which is one of the elements involved in capturing an image, the greater the impact on the calculation of the parallax amount. In this case, the size determination unit uses the number of defective pixels as a factor in determining the matching area size, and determines a larger matching area size the greater the number of defective pixels, and conversely, determines a smaller matching area size the fewer the number of defective pixels.

[0091] For example, if the F-number of the lens, which is one of the elements related to capturing an image, is small (narrow depth of field), the background will be blurred and it will be difficult to calculate the amount of parallax. In this case, the size determination unit uses the F-number of the lens as a factor in determining the matching area size, and determines the matching area size to be larger as the F-number is smaller, and conversely, determines the matching area size to be smaller as the F-number is larger.

[0092] For example, it is considered that the larger the optical distortion of the optical system, which is one of the elements related to image capture, the greater the impact on the calculation of the parallax amount. In this case, the size determination unit uses the optical distortion as a factor in determining the matching area size, and determines the matching area size to be larger as the optical distortion amount is smaller, and conversely, determines the matching area size to be smaller as the optical distortion amount is larger.

[0093] For example, it is believed that the brighter the environmental illumination, which is one of the factors related to image capture, the better the S / N ratio of the image. In this case, the size determination unit uses the environmental illumination as a factor in determining the matching area size, and determines the matching area size to be larger as the environmental illumination is darker, and conversely, determines the matching area size to be smaller as the environmental illuminance is brighter.

[0094] For example, it is considered that noise has little effect on the calculation of the parallax amount when the contrast of the imaging scene or the subject, which is one of the elements related to image capture, is high. In this case, the size determination unit uses the contrast of the imaging scene or the subject as an element when determining the matching region size, and determines the matching region size to be larger as the contrast is lower, and conversely, determines the matching region size to be smaller as the contrast is higher.

[0095] For example, it is considered that the greater the distance from the imaging unit, which is one of the elements related to image capture, the more difficult it becomes to calculate the amount of parallax. In this case, the size determination unit uses the distance of the subject as a factor in determining the matching area size, and determines a larger matching area size as the subject distance becomes greater, and conversely, determines a smaller matching area size as the subject distance becomes closer.

[0096] For example, when the weather, which is one of the factors related to image capture, is night, rain, backlight, or fog, it is considered that the S / N ratio and contrast of the image are likely to decrease. In this case, the size determination unit uses the weather conditions as a factor in determining the matching region size, and determines the matching region size to be larger as the weather conditions become worse, and on the other hand, determines the matching region size to be smaller as the weather conditions become better (for example, sunny weather).

[0097] For example, it is considered that the effect of noise differs when the subject, which is one of the factors related to image capture, is large compared to when the subject is small. In this case, the size determination unit uses the size of the subject as a factor in determining the matching area size, and determines a larger matching area size for a larger subject, and conversely, determines a smaller matching area size for a smaller subject.

[0098] For example, since a large amount of motion of a subject, which is one of the factors related to image capture, may cause the texture to become blurred, leading to deterioration of parallax calculation, it is also possible to detect the motion of the subject and use the amount of motion of the subject as a factor in determining the matching region size. In this case, the size determination unit determines the matching region size to be larger as the amount of motion of the subject increases, and conversely, determines the matching region size to be smaller as the amount of motion decreases.

[0099] For example, when the temperature or humidity around a sensor such as an image sensor, which is one of the elements involved in capturing an image, is high, there is a high possibility that noise will be easily generated. In this case, the size determination unit uses the temperature and humidity as elements for determining the matching area size, and determines the matching area size to be larger the higher at least one of the temperature and humidity, and conversely, determines the matching area size to be smaller the lower the temperature or humidity.

[0100] In addition, the size of the matching area is not limited to being determined using only one of the various element information illustrated in Figure 9, and the size determination unit may determine the size of the matching area based on a combination of two or more element information.

[0101] The information processing device of the above-mentioned embodiment can be realized by a personal computer or the like. That is, the information processing described in each of the above-mentioned embodiments is performed in the computer. The computer in this example executes the program code of the software that realizes the information processing of the present embodiment. Although the illustration of the hardware configuration is omitted, the computer that realizes the information processing device of the present embodiment is configured with a CPU, a ROM, a RAM, an auxiliary storage device, a display unit, an operation unit, a communication I / F, and a bus. The CPU uses computer programs and data stored in the ROM and RAM to control the entire computer and execute the above-mentioned information processing. The information processing device of the present embodiment may also have one or more dedicated hardware different from the CPU, and may be configured such that at least a part of the processing by the CPU is executed by the dedicated hardware. The ROM stores programs that do not require modification. The RAM temporarily stores programs and data supplied from the auxiliary storage device, and data supplied from an external device via the communication I / F. The auxiliary storage device is configured with an HDD, an SSD, or the like, and stores various data such as image data. The display unit is configured with, for example, a liquid crystal display or an LED display, and displays a GUI for the user to operate the information processing device. The operation unit is composed of, for example, a keyboard, a mouse, a joystick, a touch panel, etc., and inputs various instructions to the CPU in response to operations by a user. The CPU also operates as a display control unit that controls the display unit and an operation control unit that controls the operation unit. The communication I / F is used for communication with devices outside the information processing device. For example, when the information processing device is connected to the above-mentioned imaging unit or an external device by wire, a communication cable is connected to the communication I / F. When the information processing device has a function of wirelessly communicating with the imaging unit or an external device, the communication I / F is equipped with an antenna. The bus connects each unit of the information processing device to transmit information. In this embodiment, the display unit and the operation unit are assumed to be present inside the information processing device, but they may be present as separate devices outside.

[0102] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for implementing one or more of the functions. The above-mentioned embodiments are merely examples of the implementation of the present invention, and the technical scope of the present invention should not be interpreted as being limited by these. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features.

[0103] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) an image acquisition means for acquiring images having different sensitivities for each pair of images having parallax; A sensitivity acquisition means for acquiring sensitivity information representing a sensitivity of each pixel of the pair of images; a parallax acquiring means for acquiring parallax information representing a magnitude of parallax based on the pair of images and the sensitivity information for each of the sensitivities, and integrating the parallax information for each of the sensitivities; 13. An information processing device comprising: (Configuration 2) The parallax acquisition means A matching area having a size determined based on the sensitivity information is set for the pair of images; obtaining an image correlation between the set matching region for the pair of images; The information processing device according to configuration 1, wherein the disparity information for each sensitivity is obtained based on the obtained image correlation. (Configuration 3) The information processing device according to configuration 2, characterized in that the disparity acquisition means acquires the disparity information based on the highest image correlation among multiple image correlations acquired between the set matching area for the pair of images. (Configuration 4) The information processing device described in any one of configurations 1 to 3, characterized in that the disparity acquisition means acquires a disparity synthesis ratio based on an amount of texture in at least one of the pair of images, and integrates the disparity information for each sensitivity in accordance with the disparity synthesis ratio. (Configuration 5) The information processing device according to configuration 4, wherein the disparity acquisition means acquires the integrated disparity information by selecting, for each pixel, the most probable disparity information from the disparity information for each sensitivity based on the disparity synthesis ratio, or by synthesizing, for each pixel, disparity information for two or more sensitivities. (Configuration 6) 6. The information processing apparatus according to any one of configurations 1 to 5, wherein the sensitivity acquisition means acquires the sensitivity information based on an exposure time when the image is captured. (Configuration 7) 7. The information processing device according to any one of configurations 1 to 6, further comprising a distance acquisition means for acquiring distance information based on the integrated parallax information. (Configuration 8) 8. The information processing device according to any one of configurations 1 to 7, further comprising an image synthesis means for generating an image by synthesizing the images having different sensitivities for each pixel based on an image synthesis ratio based on pixel signals at the same coordinates. (Configuration 9) 9. The information processing apparatus according to configuration 8, wherein the sensitivity acquisition means acquires the image synthesis ratio as the sensitivity information. (Configuration 10) 9. The information processing device according to configuration 8, wherein the disparity acquisition means integrates the disparity information for each sensitivity based on the image synthesis ratio. (Configuration 11) The information processing device according to configuration 10, characterized in that the disparity acquisition means divides the images for each sensitivity into a plurality of sections using a plurality of threshold values, and acquires the integrated disparity information by selecting disparity information corresponding to the sensitivity for each section based on the image synthesis ratio. (Configuration 12) The information processing device according to any one of configurations 1 to 11, characterized in that the image acquisition means acquires a pair of images captured by at least a pair of imaging elements arranged so that their optical axes are parallel to each other, as the pair of images having parallax. (Configuration 13) an image acquisition means for acquiring a plurality of images captured at a plurality of different sensitivities and having a phase difference for each pixel; a ratio acquisition means for acquiring an image synthesis ratio for each pixel of the plurality of images based on sensitivity information representing a sensitivity for each pixel; a synthesis means for synthesizing the plurality of images for each pixel in accordance with the image synthesis ratio; a parallax acquiring means for acquiring parallax information representing a magnitude of parallax based on the image synthesized by the synthesizing means and the image synthesis ratio; 13. An information processing device comprising: (Configuration 14) The parallax acquisition means A matching area having a size determined based on the image synthesis ratio is set for the plurality of images synthesized by the synthesis means; Obtaining image correlation between the set matching regions for the plurality of synthesized images; 14. The information processing device according to configuration 13, wherein the disparity information is acquired based on the acquired image correlation. (Configuration 15) The information processing device according to configuration 14, characterized in that the disparity acquisition means acquires the disparity information based on the highest image correlation among multiple image correlations acquired between the set matching areas for the multiple synthesized images. (Configuration 16) 16. The information processing device according to any one of configurations 13 to 15, further comprising a distance acquisition unit that acquires distance information based on the parallax information acquired by the parallax acquisition unit. (Configuration 17) An image acquisition means for acquiring at least a pair of images having parallax; An element acquisition means for acquiring element information relating to capturing the pair of images; a parallax acquisition means for setting a matching area for the pair of images, the matching area having a size determined based on the element information, and acquiring parallax information based on the highest image correlation among a plurality of image correlations acquired between the set matching areas for the pair of images; having The information processing device is characterized in that the element information is at least one of the following information: exposure time for each pixel, pixel signal-to-noise ratio, number of defective pixels on the imaging element, F-number of the lens used for imaging, amount of optical distortion of the optical system used for imaging, environmental illuminance at the time of imaging, contrast of the imaged subject, distance to the subject, weather at the time of imaging, size of the imaged subject, amount of movement of the imaged subject, and temperature and humidity of the imaging element. (Method 1) an image acquiring step of acquiring images having different sensitivities for each pair of images having parallax; a sensitivity acquisition step of acquiring sensitivity information representing a sensitivity of each pixel of the pair of images; a parallax acquiring step of acquiring parallax information representing a magnitude of parallax based on the pair of images and the sensitivity information for each of the sensitivities, and integrating the parallax information for each of the sensitivities; 13. An information processing method comprising: (Method 2) an image acquiring step of acquiring a plurality of images captured at a plurality of different sensitivities and having a phase difference for each pixel; a ratio acquisition step of acquiring an image synthesis ratio for each pixel of the plurality of images based on sensitivity information representing a sensitivity for each pixel; a synthesis step of synthesizing the plurality of images for each pixel in accordance with the image synthesis ratio; a parallax acquiring step of acquiring parallax information representing a magnitude of parallax based on the image synthesized in the synthesis step and the image synthesis ratio; 13. An information processing method comprising: (Method 3) An image acquisition step of acquiring at least a pair of images having parallax; An element acquisition step of acquiring element information related to capturing the pair of images; a parallax acquisition step of setting a matching area for the pair of images, the matching area having a size determined based on the element information, and acquiring parallax information based on the highest image correlation among a plurality of image correlations acquired between the set matching areas for the pair of images; having The information processing method is characterized in that the element information is at least one of the following information: exposure time for each pixel, pixel signal-to-noise ratio, number of defective pixels on the image sensor, F-number of the lens used for imaging, amount of optical distortion of the optical system used for imaging, environmental illuminance at the time of imaging, contrast of the imaged subject, distance to the subject, weather at the time of imaging, size of the imaged subject, amount of movement of the imaged subject, and temperature and humidity of the image sensor. (Program 1) A program for causing a computer to function as the information processing device according to any one of configurations 1 to 17. [Explanation of symbols]

[0104] 101: lens, 102: imaging element, 122: imaging control unit, 123: sensitivity information determination unit, 124: size determination unit, 126: parallax calculation unit, 127: distance calculation unit, 130: synthesis processing unit, 133: parallax coefficient calculation unit, 134: parallax synthesis unit

Claims

1. an image acquisition means for acquiring images having different sensitivities for each pair of images having parallax; A sensitivity acquisition means for acquiring sensitivity information representing a sensitivity of each pixel of the pair of images; a parallax acquiring means for acquiring parallax information representing a magnitude of parallax based on the pair of images and the sensitivity information for each of the sensitivities, and integrating the parallax information for each of the sensitivities; 13. An information processing device comprising:

2. The parallax acquisition means A matching area having a size determined based on the sensitivity information is set for the pair of images; obtaining an image correlation between the set matching region for the pair of images; The information processing apparatus according to claim 1 , wherein the parallax information for each sensitivity is obtained based on the obtained image correlation.

3. The information processing apparatus according to claim 2 , wherein the parallax acquisition means acquires the parallax information based on the highest image correlation among a plurality of image correlations acquired between the set matching region for the pair of images.

4. The information processing device according to claim 1 , wherein the parallax acquisition means acquires a parallax synthesis ratio based on an amount of texture in at least one of the pair of images, and integrates the parallax information for each sensitivity in accordance with the parallax synthesis ratio.

5. The information processing device according to claim 4, characterized in that the disparity acquisition means acquires the integrated disparity information by selecting, for each pixel, the most probable disparity information from the disparity information for each sensitivity based on the disparity synthesis ratio, or by synthesizing, for each pixel, disparity information for two or more sensitivities.

6. 2 . The information processing apparatus according to claim 1 , wherein the sensitivity acquisition means acquires the sensitivity information based on an exposure time when the image is captured.

7. The information processing apparatus according to claim 1 , further comprising a distance acquisition unit that acquires distance information based on the integrated parallax information.

8. 2. The information processing apparatus according to claim 1, further comprising an image synthesis unit for generating an image by synthesizing the images having different sensitivities for each pixel on the basis of an image synthesis ratio based on pixel signals at the same coordinates.

9. 9. The information processing apparatus according to claim 8, wherein the sensitivity acquisition means acquires the image synthesis ratio as the sensitivity information.

10. The information processing apparatus according to claim 8 , wherein the parallax acquisition means integrates the parallax information for each sensitivity based on the image synthesis ratio.

11. The information processing device according to claim 10, characterized in that the disparity acquisition means acquires the integrated disparity information by dividing the images for each sensitivity into a plurality of sections using a plurality of threshold values, and selecting disparity information corresponding to the sensitivity for each section based on the image synthesis ratio.

12. 12. The information processing device according to claim 1, wherein the image acquisition means acquires, as the pair of images having parallax, a pair of images captured by at least a pair of imaging elements arranged so that their optical axes are parallel to each other.

13. an image acquisition means for acquiring a plurality of images captured at a plurality of different sensitivities and having a phase difference for each pixel; a ratio acquisition means for acquiring an image synthesis ratio for each pixel of the plurality of images based on sensitivity information representing a sensitivity for each pixel; a synthesis means for synthesizing the plurality of images for each pixel in accordance with the image synthesis ratio; a parallax acquiring means for acquiring parallax information representing a magnitude of parallax based on the image synthesized by the synthesizing means and the image synthesis ratio; 13. An information processing device comprising:

14. The parallax acquisition means A matching area having a size determined based on the image synthesis ratio is set for the plurality of images synthesized by the synthesis means; Obtaining image correlation between the set matching regions for the plurality of synthesized images; The information processing apparatus according to claim 13 , wherein the parallax information is acquired based on the acquired image correlation.

15. The information processing device according to claim 14 , characterized in that the disparity acquisition means acquires the disparity information based on the highest image correlation among a plurality of image correlations acquired between the set matching areas for the plurality of synthesized images.

16. 16. The information processing apparatus according to claim 13, further comprising a distance acquisition unit that acquires distance information based on the parallax information acquired by the parallax acquisition unit.

17. An image acquisition means for acquiring at least a pair of images having parallax; An element acquisition means for acquiring element information relating to capturing the pair of images; a parallax acquisition means for setting a matching area for the pair of images, the matching area having a size determined based on the element information, and acquiring parallax information based on the highest image correlation among a plurality of image correlations acquired between the set matching areas for the pair of images; having The information processing device is characterized in that the element information is at least one of the following information: exposure time for each pixel, pixel signal-to-noise ratio, number of defective pixels on the imaging element, F-number of the lens used for imaging, amount of optical distortion of the optical system used for imaging, environmental illuminance at the time of imaging, contrast of the imaged subject, distance to the subject, weather at the time of imaging, size of the imaged subject, amount of movement of the imaged subject, and temperature and humidity of the imaging element.

18. an image acquiring step of acquiring images having different sensitivities for each pair of images having parallax; a sensitivity acquisition step of acquiring sensitivity information representing a sensitivity of each pixel of the pair of images; a parallax acquiring step of acquiring parallax information representing a magnitude of parallax based on the pair of images and the sensitivity information for each of the sensitivities, and integrating the parallax information for each of the sensitivities; 13. An information processing method comprising:

19. an image acquiring step of acquiring a plurality of images captured at a plurality of different sensitivities and having a phase difference for each pixel; a ratio acquisition step of acquiring an image synthesis ratio for each pixel of the plurality of images based on sensitivity information representing the sensitivity for each pixel; a synthesis step of synthesizing the plurality of images for each pixel in accordance with the image synthesis ratio; a parallax acquiring step of acquiring parallax information representing a magnitude of parallax based on the image synthesized in the synthesis step and the image synthesis ratio; 13. An information processing method comprising:

20. An image acquisition step of acquiring at least a pair of images having parallax; An element acquisition step of acquiring element information related to capturing the pair of images; a parallax acquisition step of setting a matching area for the pair of images, the matching area having a size determined based on the element information, and acquiring parallax information based on the highest image correlation among a plurality of image correlations acquired between the set matching areas for the pair of images; having The information processing method is characterized in that the element information is at least one of the following information: exposure time for each pixel, pixel signal-to-noise ratio, number of defective pixels on the imaging element, F-number of the lens used for imaging, amount of optical distortion of the optical system used for imaging, environmental illuminance at the time of imaging, contrast of the imaged subject, distance to the subject, weather at the time of imaging, size of the imaged subject, amount of movement of the imaged subject, and temperature and humidity of the imaging element.

21. Computer, an image acquisition means for acquiring images having different sensitivities for each pair of images having parallax; A sensitivity acquisition means for acquiring sensitivity information representing a sensitivity of each pixel of the pair of images; a parallax acquiring means for acquiring parallax information representing a magnitude of parallax based on the pair of images and the sensitivity information for each of the sensitivities, and integrating the parallax information for each of the sensitivities; A program that causes the device to function as an information processing device having the above-mentioned configuration.

22. Computer, an image acquisition means for acquiring a plurality of images captured at a plurality of different sensitivities and having a phase difference for each pixel; a ratio acquisition means for acquiring an image synthesis ratio for each pixel of the plurality of images based on sensitivity information representing a sensitivity for each pixel; a synthesis means for synthesizing the plurality of images for each pixel in accordance with the image synthesis ratio; a parallax acquiring means for acquiring parallax information representing a magnitude of parallax based on the image synthesized by the synthesizing means and the image synthesis ratio; A program that causes the device to function as an information processing device having the above-mentioned configuration.

23. Computer, An image acquisition means for acquiring at least a pair of images having parallax; an element acquiring means for acquiring at least one of the following element information related to capturing the pair of images: an exposure time for each pixel, a pixel signal-to-noise ratio, the number of defective pixels of the image sensor, an F-number of the lens used for capturing the image, the amount of optical distortion of the optical system used for capturing the image, the environmental illuminance at the time of capturing the image, the contrast of the captured subject, the distance to the subject, the weather at the time of capturing the image, the size of the captured subject, the amount of movement of the captured subject, and the temperature and humidity of the image sensor; a parallax acquisition means for setting a matching area for the pair of images, the matching area having a size determined based on the element information, and acquiring parallax information based on the highest image correlation among a plurality of image correlations acquired between the set matching areas for the pair of images; A program that causes the device to function as an information processing device having the above-mentioned configuration.

Citation Information

Patent Citations

  • Depth information extracting device and depth information extracting method

    JP1998283474A

  • Solid-state imaging element and electronic device

    WO2019102887A1