Distance measuring device and distance measuring method

Abstract

To provide a distance measuring device and a distance measuring method that can acquire accurate distance information even when a subject is moving.SOLUTION: A distance measuring device has: an imaging unit that picks up first and second images with parallax at a first exposure time and picks up third and fourth images with parallax at a second exposure time longer than the first exposure time; a distance image creation unit that creates a first distance image representing distance information based on the parallax between the first image and the second image, and a second distance image representing distance information based on the parallax between the third image and the fourth image; and a distance image composition unit that divides the first distance image and the second distance image into a plurality of areas, and selects one of the first distance image and the second distance image in each of the plurality of areas on the basis of the distance information corresponding to each of the plurality of areas to create a composite distance image in which the first distance image and the second distance image are combined with each other.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a distance measuring device and a distance measuring method. [Background technology]

[0002] There are known techniques for detecting the distance to a subject using a digital camera. For example, one method involves placing two digital cameras at a predetermined distance and comparing images captured at the same time by these two digital cameras to detect the distance to the subject. The subjects captured in the two images thus captured have a positional shift (parallax) depending on the distance to the subject, and the distance to the subject can be calculated by measuring this parallax.

[0003] Patent document 1 proposes arranging two imaging devices, each having two cameras that capture images with different exposure times, at a predetermined distance, and capturing a wide dynamic range distance image from images with short and long exposure times while complementing areas with saturation or crushed blacks. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2021-156579 Summary of the Invention [Problem to be solved by the invention]

[0005] Generally, when capturing an image of a moving subject, the longer the exposure time and the closer the distance to the subject, the more likely it is that subject blur will occur in the captured image data. However, the technology described in Patent Document 1 does not take moving subjects into particular consideration, and subject blur can cause errors in parallax calculation, which in turn can reduce the accuracy of subject distance measurement.

[0006] An object of the present invention is to provide a distance measuring device and a distance measuring method that can obtain highly accurate distance information even when the subject is moving. [Means for solving the problem]

[0007] According to one disclosure of the present specification, there is provided a distance measuring device having an imaging unit that captures a first image and a second image with parallax using a first exposure time and captures a third image and a fourth image with parallax using a second exposure time longer than the first exposure time; a distance image generation unit that generates a first distance image representing distance information based on the parallax between the first image and the second image and a second distance image representing distance information based on the parallax between the third image and the fourth image; and a distance image synthesis unit that divides the first distance image and the second distance image into a plurality of regions and generates a synthetic distance image by synthesizing the first distance image and the second distance image by selecting one of the first distance image and the second distance image in each of the plurality of regions based on the distance information corresponding to each of the plurality of regions.

[0008] According to another disclosure of this specification, there is provided an information processing device having an image receiving unit that receives input of a first image and a second image with parallax captured using a first exposure time, and a third image and a fourth image with parallax captured using a second exposure time longer than the first exposure time; a distance image generation unit that generates a first distance image representing distance information based on the parallax between the first image and the second image, and a second distance image representing distance information based on the parallax between the third image and the fourth image; and a distance image synthesis unit that divides the first distance image and the second distance image into a plurality of regions and generates a synthetic distance image by synthesizing the first distance image and the second distance image by selecting one of the first distance image and the second distance image in each of the plurality of regions based on the distance information corresponding to each of the plurality of regions.

[0009] Furthermore, according to yet another disclosure of the present specification, there is provided a program that causes a computer to function as: a means for receiving a first image and a second image with parallax captured using a first exposure time, and a third image and a fourth image with parallax captured using a second exposure time longer than the first exposure time; a means for generating a first distance image representing distance information based on the parallax between the first image and the second image, and a second distance image representing distance information based on the parallax between the third image and the fourth image; and a means for dividing the first distance image and the second distance image into a plurality of regions and selecting one of the first distance image and the second distance image in each of the plurality of regions based on the distance information corresponding to each of the plurality of regions, thereby generating a synthetic distance image by synthesizing the first distance image and the second distance image.

[0010] Furthermore, according to yet another disclosure of the present specification, there is provided a ranging method for generating a synthetic distance image by combining the first distance image and the second distance image, by acquiring a first image and a second image with parallax captured using a first exposure time and a third image and a fourth image with parallax captured using a second exposure time longer than the first exposure time, generating a first distance image representing distance information based on the parallax between the first image and the second image and a second distance image representing distance information based on the parallax between the third image and the fourth image, dividing the first distance image and the second distance image into a plurality of regions, and selecting one of the first distance image and the second distance image in each of the plurality of regions based on the distance information corresponding to each of the plurality of regions. [Effects of the Invention]

[0011] According to the present invention, even when a subject is moving, blurring of the subject can be suppressed and highly accurate distance information can be obtained. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a block diagram showing a schematic configuration of a distance measuring device according to a first embodiment. [Figure 2]3 is a timing chart schematically showing the imaging operation in the imaging section of the distance measuring device according to the first embodiment. FIG. [Figure 3] 4 is a timing chart schematically showing the signal processing operation in the signal processing unit of the distance measuring device according to the first embodiment. FIG. [Figure 4] 2 is a schematic diagram showing an example of an image acquired by an imaging unit of the distance measuring device according to the first embodiment. FIG. [Figure 5] 4 is a flowchart showing a distance image synthesis processing method in the distance measuring device according to the first embodiment. [Figure 6] FIG. 10 is a block diagram showing a schematic configuration of a distance measuring device according to a second embodiment. [Figure 7] 10 is a schematic diagram showing an example of an image acquired by an imaging unit of a distance measuring device according to a second embodiment. FIG. [Figure 8] 10 is a flowchart showing a distance image synthesis processing method in the distance measuring device according to the second embodiment. [Figure 9] FIG. 10 is a diagram illustrating an example of the configuration of a moving body according to a third embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0013] [First embodiment] A distance measuring device according to a first embodiment of the present invention will be described with reference to Fig. 1. Fig. 1 is a block diagram showing a schematic configuration of the distance measuring device according to this embodiment.

[0014] As shown in FIG. 1 , a distance measuring device 100 according to this embodiment may include an imaging unit 10 and a signal processing unit 20. The imaging unit 10 includes multiple image acquisition units 10L and 10R. The image acquisition unit 10L includes a lens 12L and an image sensor 14L. The image acquisition unit 10R includes a lens 12R and an image sensor 14R. The signal processing unit 20 includes an image receiving unit 22, an image correction unit 24, a parallax calculation unit 26, a distance calculation unit 28, a memory unit 30, and a distance image synthesis unit 32.

[0015] The image sensors 14L and 14R are connected to an image receiving unit 22. The image receiving unit 22 is connected to an image correcting unit 24. The image correcting unit 24 is connected to a parallax calculating unit 26. The parallax calculating unit 26 is connected to a distance calculating unit 28. The distance calculating unit 28 is connected to a storage unit 30. The storage unit 30 is connected to a distance image combining unit 32.

[0016] The imaging unit 10 has a role of acquiring multiple images with parallax. To fulfill this role, the image acquisition units 10L and 10R are arranged at a predetermined interval. The lens 12L forms an optical image of the subject 110 on the imaging surface of the imaging element 14L. The imaging element 14L converts the optical image of the subject 110 formed by the lens 12L into an electrical signal by photoelectric conversion, and outputs the signal to the signal processing unit 20 as image data. Similarly, the lens 12R forms an optical image of the subject 110 on the imaging surface of the imaging element 14R. The imaging element 14R converts the optical image of the subject 110 formed by the lens 12R into an electrical signal by photoelectric conversion, and outputs the signal to the signal processing unit 20 as image data.

[0017] The image sensors 14L and 14R are image sensors having a plurality of pixels arranged two-dimensionally. The image sensors 14R and 14L may be configured with a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor, a CCD (Charge Coupled Device) image sensor, a SPAD (Single Photon Avalanche Diode) image sensor, or the like. In the following description, the image acquired by the image sensor 14L will be referred to as a left image, and the image acquired by the image sensor 14R will be referred to as a right image. Of the left and right images, the left image will be a reference image, and the right image will be an image that is displaced (parallax) from the left image.

[0018] The signal processing unit 20 has a role of generating a depth map, a so-called distance image, which indicates the distance to the subject 110 for each pixel, based on the image acquired by the imaging unit 10. More specifically, the signal processing unit 20 acquires the distance to the subject 110 for each pixel using the principle of triangulation based on the pair of left and right images received from the imaging unit 10.

[0019] The image receiving unit 22 has a role of receiving image data output from the image sensors 14L, 14R and outputting it to the downstream image correction unit 24. The image data that the image receiving unit 22 receives from the image sensors 14L, 14R includes image data with a first exposure time and image data with a second exposure time that is different from the first exposure time.

[0020] The image correction unit 24 performs preprocessing required to generate a distance image on the image data received from the image receiving unit 22, and outputs the preprocessing to the subsequent parallax calculation unit 26. The processing performed by the image correction unit 24 may include shading correction to correct uneven brightness due to peripheral light loss caused by the lenses 12L and 12R, and filtering to emphasize the correlation of the subject 110 between the left and right images. Note that the image correction unit 24 is not necessarily provided.

[0021] The parallax calculation unit 26 has a role of generating parallax image data based on the image data received from the image correction unit 24 and outputting the data to the distance calculation unit 28. The parallax image data can be generated, for example, by searching for a correlated subject in a set of left image data and right image data with the same exposure time using a template matching process and calculating the positional shift (parallax) at each pixel position across the entire screen. A known method such as SSD (Sum of Squared Difference) can be used to search for the correlation.

[0022] In template matching, one of the left or right image is used as the base image, and the other as the reference image. A base area is set on the base image, with a focus point at its center, and a reference area is set on the reference image, with a reference point corresponding to the focus point at its center. The reference point is then moved sequentially to search for the point at which the correlation between the base area on the left image and the reference area on the right image is highest. The positional shift, or disparity, determined to be highly correlated can be converted into a distance value using a predetermined formula based on the distance between the two image sensors. By calculating this distance value for the entire image, a distance map for the entire image can be generated.

[0023] In this embodiment, the left image is used as a base image and the right image is used as a reference image, and for each region of a predetermined size, correlation is searched for while horizontally shifting the reference region of the right image relative to the base region of the left image, and the pixel position with the highest correlation is identified as the amount of parallax. When searching for correlation, for regions where the correlation is less than a predetermined threshold, no parallax is detected, and a parallax value of, for example, 0 is output.

[0024] Distance calculation unit 28 generates distance image data indicating the distance value to subject 110 at each pixel position based on the parallax image data acquired by parallax calculation unit 26, and stores the data in storage unit 30. More specifically, distance calculation unit 28 calculates the distance value at each pixel position based on the principle of triangulation using the parallax value calculated by parallax calculation unit 26 and device-specific values ​​such as the base length, which is the arrangement distance between image sensors 14L and 14R, and the focal length. At this time, for pixel positions where no parallax amount can be detected, a distance value of, for example, 0 is output, indicating that the object has not been detected. Note that parallax calculation unit 26 and distance calculation unit 28 can also be considered a distance image generation unit that generates distance image data based on image data.

[0025] The storage unit 30 serves to store the distance image data generated by the distance calculation unit 28. The distance image data stored in the storage unit 30 may include distance image data corresponding to imaging with a first exposure time and distance image data corresponding to a second exposure time.

[0026] Distance image synthesis unit 32 has the role of generating synthetic distance image data based on distance image data corresponding to imaging with the first exposure time and distance image data corresponding to the second exposure time. Specifically, distance image synthesis unit 32 selects distance values ​​to use for each region of a predetermined size from the distance image data corresponding to imaging with the first exposure time and the distance image data corresponding to imaging with the second exposure time, and generates synthetic distance image data corresponding to the entire captured image. The synthetic distance image data generated by distance image synthesis unit 32 becomes the distance image output from distance measuring device 100. Details of the processing in distance image synthesis unit 32 will be described later.

[0027] The signal processing unit 20 can be realized by a hardware configuration similar to that of a general information processing device. For example, the signal processing unit 20 can be configured to include a CPU (Central Processing Unit), a main memory, a communication unit, an input / output interface unit, etc. Each functional block of the signal processing unit 20 can be realized in hardware by implementing circuit components, which are hardware components such as an LSI (Large Scale Integration) with a program embedded therein. Alternatively, it can be realized in software by loading a program that provides the function into the main memory and executing it on the CPU. The configuration of the signal processing unit 20 is not particularly limited as long as it can realize the functions described in this embodiment.

[0028] FIG. 2 is a timing diagram showing a schematic representation of the imaging operation of the imaging unit 10. In the distance measuring device 100 of this embodiment, one synthetic distance image is acquired for each frame period, which is one unit period of the imaging operation. FIG. 2 assumes video capture and shows three consecutive frame periods. Here, the Nth frame is a frame that starts at time T0(N) and ends at time T0(N+1). The (N-1)th frame is the frame period immediately before the Nth frame and ends at time T0(N). The (N+1)th frame is the frame period immediately after the Nth frame and starts at time T0(N+1). The operations in the (N-1)th and (N+1)th frames are the same as the operations for the Nth frame, which will be described later. Note that N can be any integer.

[0029] During one frame period, at least a first imaging operation, the exposure period of which is a first exposure time, and a second imaging operation, the exposure period of which is a second exposure time that is longer than the first exposure time, are performed in parallel in each of the image sensors 14L and 14R. Taking the Nth frame as an example, the exposure period of the first imaging operation starts at time T0(N) and ends at time T1(N). The exposure period of the second imaging operation starts at time T0(N) and ends at time T2(N), which is later than time T1(N).

[0030] After the first imaging operation ends at time T1(N), pixel signals are sequentially read out from each pixel of the image sensor 14L and the image sensor 14R. The pixel signals read out from each pixel of the image sensor 14L in this manner constitute left image data 40SL for the first exposure time. Similarly, the pixel signals read out from each pixel of the image sensor 14R constitute right image data 40SR for the first exposure time. The left image data 40SL and the right image data 40SR are input to the signal processing unit 20 as a set of image data 40S.

[0031] After the second imaging operation ends at time T2(N), pixel signals are sequentially read from each pixel of the image sensor 14L and the image sensor 14R. The pixel signals read from each pixel of the image sensor 14L in this manner constitute left image data 40LL for the second exposure time. The pixel signals read from each pixel of the image sensor 14R constitute right image data 40LR for the second exposure time. The left image data 40LL and the right image data 40LR are transferred to the signal processing unit 20 as a set of image data 40L.

[0032] In this way, the image data 40S, 40L acquired by the imaging unit 10 are output to the signal processing unit 20 in order of the image data with the shortest exposure time. Note that, although two types of image data 40S, 40L with a first exposure time and a second exposure time are configured to be output here, three or more types of image data with different exposure times may be configured to be output.

[0033] 2, when the first and second imaging operations are started simultaneously, it is preferable to use SPAD sensors as the imaging elements 14L and 14R that make up the imaging unit 10. For example, a SPAD sensor configured to count pulse signals corresponding to the number of incident photons using a counter can nondestructively read out the count value during the exposure period, allowing a single SPAD pixel to perform the first and second imaging operations in parallel. This simplifies the pixel configuration and operation control of the imaging elements.

[0034] 2, the first and second imaging operations start simultaneously, but they do not necessarily have to start simultaneously. However, from the perspective of improving distance measurement accuracy, it is preferable that the time center of the exposure period of the first imaging operation and the time center of the exposure period of the second imaging operation are closer to each other. In this sense, it is preferable to execute the first and second imaging operations so that at least a portion of the period overlaps, rather than executing the first and second imaging operations sequentially in a time-division manner.

[0035] Fig. 3 is a timing diagram showing a signal processing operation in the signal processing unit 20. As in Fig. 2, Fig. 3 shows operations in the (N-1)th frame, the Nth frame, and the (N+1)th frame, which are three consecutive frame periods. The operations in the (N-1)th frame and the (N+1)th frame are the same as the operation in the Nth frame, which will be described later.

[0036] After time T1(N), when the image receiving unit 22 receives the left image data 40SL and right image data 40SR for the first exposure time from the imaging unit 10, the image receiving unit 22 transfers the received image data to the image correcting unit .

[0037] Upon receiving the left image data 40SL and right image data 40SR, the image correction unit 24 performs preprocessing on these image data, such as shading correction and filtering, necessary for generating distance images, and outputs the processed image data to the parallax calculation unit 26. Here, the image data obtained by performing the preprocessing on the left image data 40SL is referred to as left image correction data 42SL. Also, the image data obtained by performing the preprocessing on the right image data 40SR is referred to as right image correction data 42SR.

[0038] Upon receiving the left image correction data 42SL and the right image correction data 42SR, the parallax calculation unit 26 calculates a parallax value at each pixel position based on the image data, for example, by using a template matching technique. The parallax calculation unit 26 generates parallax image data 50S including data on the parallax value at each pixel position, and outputs the generated parallax image data 50S to the distance calculation unit 28.

[0039] Upon receiving the parallax image data 50S, the distance calculation unit 28 uses the principle of triangulation to calculate the distance value at each pixel position based on the parallax value at each pixel position of the parallax image data 50S. The distance calculation unit 28 generates distance image data 60S including data on the distance value at each pixel position, and stores the generated distance image data 60S in the storage unit 30.

[0040] After the next time T2(N), when the image receiving unit 22 receives the left image data 40LL and right image data 40LR for the second exposure time from the imaging unit 10, the image receiving unit 22 transfers the received image data to the image correcting unit .

[0041] Upon receiving the left image data 40LL and right image data 40LR, the image correction unit 24 performs preprocessing on these image data, such as shading correction and filtering, necessary for generating distance images, and transfers the processed image data to the parallax calculation unit 26. Here, the image data obtained by performing the preprocessing on the left image data 40LL is referred to as left image correction data 42LL. Similarly, the image data obtained by performing the preprocessing on the right image data 40LR is referred to as right image correction data 42LR.

[0042] Upon receiving the left image correction data 42LL and the right image correction data 42LR, the disparity calculation unit 26 calculates a disparity value at each pixel position based on the image data, for example, by using a template matching technique. The disparity calculation unit 26 generates disparity image data 50L including data on the disparity value at each pixel position, and outputs the generated disparity image data 50L to the distance calculation unit 28.

[0043] When distance calculation unit 28 receives parallax image data 50L, it uses the principle of triangulation to calculate a distance value at each pixel position based on the parallax value at each pixel position of parallax image data 50L. Distance calculation unit 28 generates distance image data 60L including data on the distance value at each pixel position and stores the generated distance image data 60L in storage unit 30.

[0044] After the generation of distance image data 60S and distance image data 60L is completed, distance image synthesis unit 32 generates synthesized distance image data 60 by synthesizing distance value information at each pixel position based on distance image data 60S, 60L stored in memory unit 30. Details of the processing by distance image synthesis unit 32 will be described later.

[0045] Here, issues that may arise when generating synthetic distance image data 60 will be described with reference to FIG. 4. FIG. 4 is a schematic illustration of an image of another vehicle moving from left to right near a camera mounted on the front of the vehicle, as an example of an image of a subject 110 in front of the imaging unit 10. FIG. 4(a) is an image captured with a first exposure time (corresponding to left image data 40SL and right image data 40SR), and FIG. 4(b) is an image captured with a second exposure time (corresponding to left image data 40LL and right image data 40LR) that is longer than the first exposure time. The subject 70a in FIG. 4(a) and the subject 70b in FIG. 4(b) are images of the same vehicle. In this specification, when a common description is given for the left image data 40SL and the right image data 40SR, these may be referred to as image data 40SL and 40SR. When a common description is given for the left image data 40LL and the right image data 40LR, these may be referred to as image data 40LL and 40LR.

[0046] As shown in Figures 4(a) and 4(b), subject 70b, captured using the relatively long second exposure time, moves more within the angle of view within one frame period than subject 70a, captured using the relatively short first exposure time, making it more susceptible to subject blur. In particular, the closer a subject is to the camera, the greater the subject blur. In a pair of left and right image data with such subject blur, an error occurs in the positional shift (parallax) searched for in the pattern matching process, potentially reducing the accuracy of distance information to the subject.

[0047] Therefore, in distance measuring device 100 according to this embodiment, distance image synthesis processing in distance image synthesis section 32 is performed according to the flowchart shown in Fig. 5. Fig. 5 is a flowchart illustrating the processing flow of distance image synthesis in distance image synthesis section 32.

[0048] In the distance image synthesis process in distance image synthesis unit 32, the captured image represented by distance image data 60S, 60L is divided into multiple reference areas of a predetermined size, and the multiple divided reference areas are selected in order to perform the following process. Note that each of the multiple reference areas only needs to contain at least one pixel, and there are no particular limitations on the size of the reference area.

[0049] First, in step S101, distance image synthesis unit 32 reads out distance values ​​for each pixel position in the selected reference area from distance image data 60S for the first exposure time stored in storage unit 30.

[0050] Next, in step S102, distance image synthesis unit 32 determines whether or not the distance image data 60S of the reference area read from storage unit 30 contains distance values ​​at pixel positions where parallax was detected. If the determination result shows that the reference area contains distance values, that is, if any pixel position in the reference area contains a distance value other than 0 ("YES" in step S102), the process proceeds to step S106. On the other hand, if the determination result shows that the reference area does not contain distance values, that is, if the distance values ​​at all pixel positions in the reference area are 0 ("NO" in step S102), the process proceeds to step S107.

[0051] In step S106, distance image synthesis unit 32 selects and outputs the distance value acquired from distance image data 60S of the first exposure time as the distance value of the reference region. After outputting the distance value, the process proceeds to step S111.

[0052] In step S107, distance image synthesis unit 32 reads out distance values ​​for each pixel position in the selected reference area from distance image data 60L for the second exposure time stored in storage unit 30.

[0053] Next, in step S108, distance image synthesis unit 32 determines whether any of the distance image data 60L of the reference area read from storage unit 30 contains distance values ​​at pixel positions where parallax was detected. If the determination result shows that the reference area contains distance values, that is, if any pixel position in the reference area contains a distance value other than 0 ("YES" in step S108), the process proceeds to step S109. On the other hand, if the determination result shows that the reference area does not contain distance values, that is, if the distance values ​​at all pixel positions in the reference area are 0 ("NO" in step S108), the process proceeds to step S110.

[0054] In step S109, distance image synthesis unit 32 selects and outputs the distance value acquired from distance image data 60L of the second exposure time as the distance value of the reference region. After outputting the distance value, the process proceeds to step S111.

[0055] In step S110, distance image synthesis unit 32 outputs 0 (no distance value) as the distance value for all pixel positions within the reference area. In other words, distance values ​​of 0 for all pixel positions within the reference area indicate that no subject with parallax has been detected within this area. Therefore, in such a case, distance values ​​of 0 are output for all pixel positions. After the distance values ​​have been output, the process proceeds to step S111.

[0056] In step S111, distance image synthesis unit 32 determines whether there are any unprocessed reference areas in the captured image represented by distance image data 60S, 60L. If the determination shows that there are any unprocessed reference areas ("YES" in step S111), the next reference area to be processed is selected from the unselected reference areas, and the process returns to step S101. On the other hand, if the determination shows that processing of all reference areas has been completed ("NO" in step S111), distance image synthesis processing in distance image synthesis unit 32 is completed.

[0057] As described above, in this embodiment, distance image data 60S with a first exposure time, which has a relatively short exposure time, and distance image data 60L with a relatively long exposure time are generated. Then, based on these distance image data, distance information from either the distance image data 60S or the distance image data 60L is appropriately selected for each predetermined region to generate a composite distance image. Therefore, this embodiment can reduce the effects of subject blur and perform highly accurate distance measurements according to the position of the subject.

[0058] [Second embodiment] A distance measuring device according to a second embodiment of the present invention will be described with reference to Fig. 6. Components similar to those in the distance measuring device according to the first embodiment are given the same reference numerals, and their description will be omitted or simplified. Fig. 6 is a block diagram showing the schematic configuration of the distance measuring device according to this embodiment.

[0059] The distance measuring device according to this embodiment is the same as the distance measuring device according to the first embodiment, except that distance threshold information used for determining when selecting distance measurement information is stored in distance image synthesis unit 32. In this embodiment, differences from the distance measuring device of the first embodiment will be mainly described, and descriptions of similarities to the distance measuring device of the first embodiment will be omitted as appropriate.

[0060] Another problem that may arise when generating synthetic distance image data 60 will now be described with reference to FIG. 7. FIG. 7 is a schematic illustration of an example of an image of a subject 110 in front of the imaging unit 10, showing a vehicle operating near the subject's vehicle and a vehicle operating in the distance, captured by a camera mounted on the front of the subject's vehicle. FIG. 7(a) is an image captured with a first exposure time (corresponding to image data 40SL and 40SR), and FIG. 7(b) is an image captured with a second exposure time longer than the first exposure time (corresponding to image data 40LL and 40LR). Subject 70a in FIG. 7(a) and subject 70b in FIG. 7(b) are images of the same vehicle. Subject 72a in FIG. 7(a) and subject 72b in FIG. 7(b) are images of the same vehicle.

[0061] Focusing on nearby moving subjects, subject 70b, imaged with a relatively long second exposure time, moves more within the angle of view within one frame period than subject 70a, imaged with a relatively short first exposure time, and is therefore more susceptible to subject blur. In contrast, subjects 72a and 72b, moving at a distance, move less within the angle of view within one frame period and are therefore less susceptible to subject blur. On the other hand, subject 72a, imaged with a relatively short first exposure time, does not receive enough light compared to subject 72b, imaged with a relatively long second exposure time, and is therefore more likely to be blurred. In a pair of left image data and right image data in which the subject is blurred or blurred, an error occurs in the positional shift (parallax) searched for in the pattern matching process, which may reduce the accuracy of the distance information to the subject.

[0062] A similar problem can occur not only when a subject moves nearby, but also when, for example, an image is captured under conditions where the subject is dark and the exposure time is relatively long.

[0063] Therefore, in distance measuring device 100 according to this embodiment, distance image synthesis processing in distance image synthesis section 32 is performed according to the flowchart shown in Fig. 8. Fig. 8 is a flowchart illustrating the processing flow of distance image synthesis in distance image synthesis section 32.

[0064] As in the first embodiment, in the distance image synthesis process in the distance image synthesis unit 32, the captured image represented by the distance image data 60S, 60L is divided into multiple reference areas of a predetermined size, and the multiple divided reference areas are selected sequentially to perform the following process.

[0065] First, in step S101, distance image synthesis unit 32 reads out distance values ​​for each pixel position in the selected reference area from distance image data 60S for the first exposure time stored in storage unit 30.

[0066] Next, in step S102, distance image synthesis unit 32 determines whether or not the distance image data 60S of the reference area read from storage unit 30 contains distance values ​​at pixel positions where parallax was detected. If the determination result shows that the reference area contains distance values, that is, if any pixel position in the reference area contains a distance value other than 0 ("YES" in step S102), the process proceeds to step S103. On the other hand, if the determination result shows that the reference area does not contain distance values, that is, if the distance values ​​at all pixel positions in the reference area are 0 ("NO" in step S102), the process proceeds to step S107.

[0067] In step S103, distance image synthesis unit 32 compares the distance value within the reference area acquired in step S102 with a preset distance threshold. If the comparison shows that the distance value within the reference area is less than the distance threshold, that is, the subject is closer than the distance set in the distance threshold ("YES" in step S103), the process proceeds to step S106. On the other hand, if the comparison shows that the distance value within the reference area is equal to or greater than the distance threshold, that is, the subject is farther away than the distance set in the distance threshold ("NO" in step S103), the process proceeds to step S104.

[0068] If the reference area includes multiple pixels, and two or more of these pixels have distance information, the average of the distance values ​​of these pixels can be used as a representative value for the distance value to be compared with the distance threshold. However, the distance value to be compared with the distance threshold is not limited to this, and for example, the maximum or minimum value of the multiple distance values ​​may be selected.

[0069] In step S104, distance image synthesis unit 32 reads out distance values ​​for each pixel position in the selected reference area from distance image data 60L for the second exposure time stored in storage unit 30.

[0070] Next, in step S105, distance image synthesis unit 32 determines whether or not the distance image data 60L of the reference area read from storage unit 30 contains distance values ​​at pixel positions where parallax was detected. If the determination result shows that the reference area contains distance values, that is, if any pixel position in the reference area contains a distance value other than 0 ("YES" in step S105), the process proceeds to step S109. On the other hand, if the determination result shows that the reference area does not contain distance values, that is, if the distance values ​​at all pixel positions in the reference area are 0 ("NO" in step S105), the process proceeds to step S106.

[0071] The processes from step S106 to step S111 are the same as those in the first embodiment. According to the above processing flow, for a subject determined to be closer than a preset distance threshold, a distance value calculated from an image captured with a short exposure time is selected, and for a subject determined to be farther away, a distance value calculated from an image captured with a long exposure time is selected. This makes it possible to suppress the influence of subject blur for subjects moving near the distance measuring device, and to suppress subject blur for subjects moving far away, thereby preventing a decrease in the accuracy of distance information to the subject.

[0072] As described above, according to this embodiment, the influence of subject blurring can be suppressed and distance measurement with high accuracy according to the position of the subject can be performed, as in the first embodiment. Furthermore, because the distance information of distance image data 60S is selected for areas where the distance information is less than a predetermined threshold, and the distance information of distance image data 60L is selected for areas where the distance information is equal to or greater than the predetermined threshold, blurring of a subject moving at a distance can be suppressed, and distance measurement with higher accuracy can be performed.

[0073] In this embodiment, a distance threshold is set for the distance image data 60S with the first exposure time, but if three or more types of image data with different exposure times are acquired, a different distance threshold may be set for each of the distance image data with different exposure times. For example, the distance threshold can be set to a larger value for distance image data with a longer exposure time.

[0074] [Third embodiment] A moving body according to a third embodiment of the present invention will be described with reference to Fig. 9. Fig. 9 is a diagram showing an example of the configuration of a moving body according to this embodiment.

[0075] 9(a) shows an example of the configuration of a device mounted on a vehicle as an on-board camera. The device 300 has a distance measurement unit 303 that measures the distance to an object, and a collision determination unit 304 that determines whether or not there is a possibility of a collision based on the distance measured by the distance measurement unit 303. The distance measurement unit 303 is configured by the distance measuring device 100 described in either the first or second embodiment. Here, the distance measurement unit 303 is an example of a distance information acquisition means that acquires distance information to the object. In other words, the distance information is information related to the distance to the object, etc.

[0076] The device 300 is connected to a vehicle information acquisition device 310 and can acquire vehicle information such as vehicle speed, yaw rate, and steering angle. The device 300 is also connected to a control ECU 320, which is a control device that outputs a control signal to generate a braking force for the vehicle based on the determination result of the collision determination unit 304. The device 300 is also connected to an alarm device 330 that issues an alarm to the driver based on the determination result of the collision determination unit 304. For example, if the collision determination unit 304 determines that a collision is highly likely, the control ECU 320 performs vehicle control to avoid a collision and mitigate damage by applying the brakes, releasing the accelerator, suppressing engine output, etc. The alarm device 330 warns the user by sounding an alarm, displaying alarm information on a screen of a car navigation system, etc., or vibrating a seat belt or steering wheel. These devices of the device 300 function as a mobile object control unit that controls the operation of controlling the vehicle as described above.

[0077] In this embodiment, the device 300 measures the distance around the vehicle, for example, the front or rear. Fig. 9(b) shows the device when measuring the distance in front of the vehicle (distance measurement range 350). The vehicle information acquisition device 310, which serves as a distance measurement control means, sends an instruction to the device 300 or the distance measurement unit 303 to perform a distance measurement operation. This configuration can further improve the accuracy of distance measurement.

[0078] In this embodiment, an example of control to prevent collision with other vehicles has been described, but the present invention can also be applied to control of automatic driving by following other vehicles, control of automatic driving to prevent deviation from a lane, etc. Furthermore, the device is not limited to vehicles such as automobiles, but can be applied to moving bodies (moving devices) such as ships, aircraft, artificial satellites, industrial robots, and consumer robots. In addition, the present invention can be applied to a wide range of devices that use object recognition or biometric recognition, such as intelligent transport systems (ITS) and surveillance systems, not limited to moving bodies.

[0079] [Modified embodiment] The present invention is not limited to the above-described embodiment, and various modifications are possible. For example, an example in which part of the configuration of any one of the embodiments is added to another embodiment, or an example in which part of the configuration of another embodiment is substituted therefor, is also an embodiment of the present invention.

[0080] Furthermore, in the above embodiment, the imaging unit 10 is configured with an imaging element 14L for acquiring a left image and an imaging element 14R for acquiring a right image, but the imaging unit 10 does not necessarily have to be configured with multiple imaging elements. That is, the imaging unit 10 may be configured with an imaging element that can simultaneously acquire image information with multiple parallaxes. For example, if each of the multiple pixels included in the imaging element 14 is configured as a pupil-divided pixel in which multiple photoelectric conversion elements share a single microlens and are capable of detecting a phase difference, the imaging unit 10 can also be configured with a single imaging element.

[0081] In addition, in the above embodiment, an example is shown in which image data 40SL, 40SR for the first exposure time are generated before image data 40LL, 40LR for the second exposure time are generated, but the order in which these image data are generated is not limited to this example. For example, image data 40S for the first exposure time and image data 40L for the second exposure time may be output simultaneously from imaging unit 10, and distance image data 60S, 60L may be generated in parallel in signal processing unit 20.

[0082] Furthermore, in the second embodiment, a preset distance threshold is used in distance image synthesis unit 32, but it is also possible to configure the distance threshold to be set appropriately depending on the image capturing conditions, etc. For example, it is also possible to configure the distance threshold to be automatically set depending on the brightness of the captured image, by measuring the brightness using a separate means.

[0083] Furthermore, although the first and second embodiments have been described with respect to distance measuring devices, the algorithms described in the above embodiments can also be applied to an information processing device for processing signals output from the imaging unit 10. In this case, the information processing device can include the functions of the signal processing unit 20. The information processing device can be a device such as a personal computer including a processor (for example, a CPU or an MPU). Alternatively, the information processing device can be a circuit such as an ASIC that realizes the functions of the signal processing unit 20.

[0084] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0085] It should be noted that the above-described embodiments are merely examples of specific embodiments for carrying out the present invention, and the technical scope of the present invention should not be construed as being limited by these embodiments. In other words, the present invention can be carried out in various forms without departing from its technical concept or main features.

[0086] The disclosure of the above embodiment includes the following configurations and methods. (Configuration 1) an imaging unit that captures a first image and a second image having parallax with a first exposure time, and captures a third image and a fourth image having parallax with a second exposure time that is longer than the first exposure time; a distance image generation unit that generates a first distance image representing distance information based on the parallax between the first image and the second image, and a second distance image representing distance information based on the parallax between the third image and the fourth image; a distance image synthesis unit that divides the first distance image and the second distance image into a plurality of regions, and generates a synthesized distance image by synthesizing the first distance image and the second distance image by selecting one of the first distance image and the second distance image in each of the plurality of regions based on the distance information corresponding to each of the plurality of regions; A distance measuring device comprising: (Configuration 2) When the distance information corresponding to the region of the first distance image includes a distance value to a subject, the distance image synthesis unit selects the distance information of the first distance image as the distance information of the region of the synthesized distance image. 2. The distance measuring device according to configuration 1, (Configuration 3) When the first distance image does not have a distance value corresponding to the region, the distance image synthesis unit selects the distance information of the second distance image as distance information of the region of the synthesized distance image. 2. The distance measuring device according to configuration 1, (Configuration 4) When the first distance image and the second distance image do not have a distance value corresponding to the region, the distance image synthesis unit selects information indicating that there is no distance value as distance information for the region of the synthesized distance image. 2. The distance measuring device according to configuration 1, (Configuration 5) The distance image synthesis unit selects the distance information of the first distance image for the region where the distance information of the first distance image is a distance value less than a predetermined threshold, and selects the distance information of the second distance image for the region where the distance information of the first distance image is a distance value equal to or greater than the predetermined threshold. 2. The distance measuring device according to configuration 1, (Configuration 6) When the second distance image does not have a distance value as the distance information in the region where the distance information in the first distance image is a distance value equal to or greater than the predetermined threshold, the distance image synthesis unit selects the distance information in the first distance image as the distance information in the region. 6. The distance measuring device according to configuration 5. (Configuration 7) The image correcting unit further includes an image correcting unit that performs predetermined correction on the first image, the second image, the third image, and the fourth image. 7. The distance measuring device according to any one of configurations 1 to 6. (Configuration 8) An exposure period of the first exposure time when the first image and the second image are captured and an exposure period of the second exposure time when the third image and the fourth image are captured at least partially overlap with each other. 8. The distance measuring device according to any one of configurations 1 to 7. (Configuration 9) The imaging unit has a plurality of imaging elements arranged at predetermined intervals. 9. The distance measuring device according to any one of configurations 1 to 8. (Configuration 10) The imaging unit has an imaging element including pupil-dividing pixels. 9. The distance measuring device according to any one of configurations 1 to 8. (Configuration 11) The imaging element is a SPAD sensor 11. The distance measuring device according to configuration 9 or 10. (Configuration 12) A mobile object, a distance measuring device according to any one of configurations 1 to 11; a control means for controlling the moving object based on distance information acquired by the distance measuring device; A moving object characterized by having: (Configuration 13) an image receiving unit to which a first image and a second image with parallax captured with a first exposure time and a third image and a fourth image with parallax captured with a second exposure time longer than the first exposure time are input; a distance image generation unit that generates a first distance image representing distance information based on the parallax between the first image and the second image, and a second distance image representing distance information based on the parallax between the third image and the fourth image; a distance image synthesis unit that divides the first distance image and the second distance image into a plurality of regions, and generates a synthesized distance image by synthesizing the first distance image and the second distance image by selecting one of the first distance image and the second distance image in each of the plurality of regions based on the distance information corresponding to each of the plurality of regions; An information processing device comprising: (Configuration 14) The distance image synthesis unit selects the distance information of the first distance image for the region where the distance information of the first distance image is a distance value less than a predetermined threshold, and selects the distance information of the second distance image for the region where the distance information of the first distance image is a distance value equal to or greater than the predetermined threshold. 14. The information processing device according to configuration 13. (Configuration 15) Computer, a means for receiving a first image and a second image with parallax captured with a first exposure time, and a third image and a fourth image with parallax captured with a second exposure time longer than the first exposure time; a means for generating a first distance image representing distance information based on the parallax between the first image and the second image, and a second distance image representing distance information based on the parallax between the third image and the fourth image; and means for dividing the first distance image and the second distance image into a plurality of regions, and selecting one of the first distance image and the second distance image in each of the plurality of regions based on the distance information corresponding to each of the plurality of regions, thereby generating a composite distance image by combining the first distance image and the second distance image; A program that functions as a (Method 1) acquiring a first image and a second image with parallax captured with a first exposure time, and a third image and a fourth image with parallax captured with a second exposure time longer than the first exposure time; generating a first distance image representing distance information based on the parallax between the first image and the second image, and a second distance image representing distance information based on the parallax between the third image and the fourth image; Dividing the first distance image and the second distance image into a plurality of regions, and selecting one of the first distance image and the second distance image in each of the plurality of regions based on the distance information corresponding to each of the plurality of regions, thereby generating a composite distance image by combining the first distance image and the second distance image. A distance measuring method characterized by: (Method 2) When generating the synthetic distance image, the distance information of the first distance image is selected for the area where the distance information of the first distance image is a distance value less than a predetermined threshold, and the distance information of the second distance image is selected for the area where the distance information of the first distance image is a distance value equal to or greater than the predetermined threshold. 2. The distance measuring method according to claim 1, [Explanation of symbols]

[0087] 10...imaging unit 20...Signal processing unit 22...Image receiving unit 24...Image correction section 26…Parallax calculation unit 28...Distance calculation unit 30...Storage section 32...Distance image synthesis unit 100…Distance measuring device

Claims

1. an imaging unit that captures a first image and a second image having parallax with a first exposure time, and captures a third image and a fourth image having parallax with a second exposure time that is longer than the first exposure time; a distance image generating unit that generates a first distance image representing distance information based on the parallax between the first image and the second image, and a second distance image representing distance information based on the parallax between the third image and the fourth image; a distance image synthesis unit that divides the first distance image and the second distance image into a plurality of regions, and generates a synthesized distance image by synthesizing the first distance image and the second distance image by selecting one of the first distance image and the second distance image in each of the plurality of regions based on the distance information corresponding to each of the plurality of regions; A distance measuring device comprising:

2. When the distance information corresponding to the region of the first distance image includes a distance value to a subject, the distance image synthesis unit selects the distance information of the first distance image as the distance information of the region of the synthesized distance image.

2. The distance measuring device according to claim 1.

3. When the first distance image does not have a distance value corresponding to the region, the distance image synthesis unit selects the distance information of the second distance image as distance information of the region of the synthesized distance image.

2. The distance measuring device according to claim 1.

4. When the first distance image and the second distance image do not have a distance value corresponding to the region, the distance image synthesis unit selects information indicating that there is no distance value as distance information for the region of the synthesized distance image.

2. The distance measuring device according to claim 1.

5. The distance image synthesis unit selects the distance information of the first distance image for the region where the distance information of the first distance image is a distance value less than a predetermined threshold, and selects the distance information of the second distance image for the region where the distance information of the first distance image is a distance value equal to or greater than the predetermined threshold.

2. The distance measuring device according to claim 1.

6. When the second distance image does not have a distance value as the distance information in the region where the distance information in the first distance image is a distance value equal to or greater than the predetermined threshold, the distance image synthesis unit selects the distance information in the first distance image as the distance information in the region.

6. The distance measuring device according to claim 5.

7. The image correcting unit further includes an image correcting unit that performs predetermined correction on the first image, the second image, the third image, and the fourth image.

7. A distance measuring device according to claim 1, wherein the distance measuring device is a distance measuring device.

8. An exposure period of the first exposure time when the first image and the second image are captured and an exposure period of the second exposure time when the third image and the fourth image are captured at least partially overlap with each other.

7. A distance measuring device according to claim 1, wherein the distance measuring device is a distance measuring device.

9. The imaging unit has a plurality of imaging elements arranged at predetermined intervals.

7. A distance measuring device according to claim 1, wherein the distance measuring device is a distance measuring device.

10. The imaging unit has an imaging element including pupil-dividing pixels.

7. A distance measuring device according to claim 1, wherein the distance measuring device is a distance measuring device.

11. The imaging element is a SPAD sensor 10. The distance measuring device according to claim 9.

12. A mobile object, A distance measuring device according to any one of claims 1 to 6; a control means for controlling the moving object based on distance information acquired by the distance measuring device; A moving object characterized by having:

13. an image receiving unit to which a first image and a second image having parallax captured with a first exposure time and a third image and a fourth image having parallax captured with a second exposure time longer than the first exposure time are input; a distance image generating unit that generates a first distance image representing distance information based on the parallax between the first image and the second image, and a second distance image representing distance information based on the parallax between the third image and the fourth image; a distance image synthesis unit that divides the first distance image and the second distance image into a plurality of regions, and generates a synthesized distance image by synthesizing the first distance image and the second distance image by selecting one of the first distance image and the second distance image in each of the plurality of regions based on the distance information corresponding to each of the plurality of regions; An information processing device comprising:

14. The distance image synthesis unit selects the distance information of the first distance image for the region where the distance information of the first distance image is a distance value less than a predetermined threshold, and selects the distance information of the second distance image for the region where the distance information of the first distance image is a distance value equal to or greater than the predetermined threshold.

14. The information processing apparatus according to claim 13,

15. Computer, a means for receiving first and second images with parallax captured with a first exposure time, and third and fourth images with parallax captured with a second exposure time longer than the first exposure time; a means for generating a first distance image representing distance information based on the parallax between the first image and the second image, and a second distance image representing distance information based on the parallax between the third image and the fourth image; and a means for dividing the first distance image and the second distance image into a plurality of regions, and selecting one of the first distance image and the second distance image in each of the plurality of regions based on the distance information corresponding to each of the plurality of regions, thereby generating a composite distance image by combining the first distance image and the second distance image; A program that functions as a

16. acquiring a first image and a second image with parallax captured with a first exposure time, and a third image and a fourth image with parallax captured with a second exposure time longer than the first exposure time; generating a first distance image representing distance information based on the parallax between the first image and the second image, and a second distance image representing distance information based on the parallax between the third image and the fourth image; Dividing the first distance image and the second distance image into a plurality of regions, and selecting one of the first distance image and the second distance image in each of the plurality of regions based on the distance information corresponding to each of the plurality of regions, thereby generating a composite distance image by combining the first distance image and the second distance image. A distance measuring method characterized by:

17. When generating the synthetic distance image, the distance information of the first distance image is selected for the area where the distance information of the first distance image is a distance value less than a predetermined threshold, and the distance information of the second distance image is selected for the area where the distance information of the first distance image is a distance value equal to or greater than the predetermined threshold.

17. The distance measuring method according to claim 16.

Images