Distance measuring device

Abstract

To provide a distance measuring device with which it is possible to highly accurately detect parallax even when using an imaging element differing in pixel pitch, a lens differing in focal length, or a distorted lens.SOLUTION: The distance measuring device according to one embodiment detects image positions on an imaging element of at least two images of the same object having been imaged on the imaging element by determination of coincidence degree so as to measure a distance. The images on the imaging element are sampled with the pixel pitch of the imaging element, and geometric correction and resampling process are carried out when determining the coincidence degree of images at image position detection time, with corrections made to the degradation of image signals that occur at resampling processing time.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present invention relates to a distance measuring device.

Background Art

[0002] Conventionally, a distance measuring method and a distance measuring device that use a plurality of cameras such as a stereo camera to accurately measure the distance from a predetermined position to a subject or the three-dimensional position of the subject are known. In a stereo camera, a correlation value between a reference image block including an image of a subject captured by one camera and a comparison image block including an image of the same subject captured by the other camera is calculated, and the degree of image matching is determined based on the magnitude of the correlation value. When it is determined that they match, the difference in the image positions on the imaging elements in each camera at that time is regarded as the parallax, and distance information to the subject is calculated using the principle of triangulation. At this time, for the determination of the degree of image matching, since it is assumed that the images of the two cameras are the same, the two cameras need to have the same characteristics as much as possible.

[0003] International Publication No. 2011 / 010438 (Patent Document 1) discloses "a parallax detection device (3) that calculates the parallax generated between a plurality of optical systems, and a PSF equalization unit (5) that corrects at least one of a plurality of images obtained from each of the plurality of optical systems so that the point image distributions of the plurality of optical systems are the same as the point image distribution of a predetermined optical system, and a parallax calculation unit (9) that calculates the parallax generated between the plurality of optical systems using the image corrected by the PSF equalization unit (5)".

[0004] In recent years, in in-vehicle cameras, the recognition target has expanded to a wide-angle range and a wide distance range, and in order to ensure safety over a wider range, the use of wide-angle lens cameras with large distortion has increased, and the number of cameras also tends to increase.

[0005] Therefore, in order to minimize the number of cameras to be coordinated, it is important to study stereo cameras (multi-viewpoint) with wide-angle high-distortion lenses or sensors with different pixel pitches.

[0006] In addition, technologies that make full use of the optical system and utilize the projection methods of lenses and mirrors to meet the requirements of a wide-angle range and a wide-distance range have also been developed. In Japanese Patent Application Laid-Open No. 2021-012075 (Patent Document 2), it is disclosed that "after the first light R1 from the subject is reflected by the upper hyperbolic mirror 102, it is further reflected by the lower hyperbolic mirror 103 (inner hyperbolic mirror 103A), and then enters the image sensor 105 through the imaging optical system 104. Also, the second light R2 from the subject, which is different from the first light R1, is reflected by the lower hyperbolic mirror 103 (outer hyperbolic mirror 103B) and enters the image sensor 105 through the imaging optical system 104. Since the first light R1 and the second light R2 enter different positions on the image sensor 105 from different directions, the distance to the subject and the like can be calculated by the stereo camera 101."

Prior Art Documents

Patent Documents

[0007]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0008] In a distance measurement device, when the sizes of the imaging elements and the pixel pitches of the pixels of the imaging elements of two cameras used for distance measurement using the principle of a stereo camera are different, there is a problem that the pixel pitches (sampling pitches · sampling frequencies) of the images of the two cameras are different. Also, when the focal lengths of the lenses of the two cameras are different, the sizes of the images formed on the sensor are different, and there is also a problem that it is necessary to compare images with different sampling pitches to obtain the parallax.

[0009] At this time, the images on the two imaging elements are once interpolated so as to become images with the same sampling pitch. However, when converting an image with a coarse pixel pitch into an image with a fine pixel pitch, the original image size increases, and there is no meaning in using an imaging element with a coarse pixel pitch that originally requires a small amount of data processing. Further, when converting an image with a fine pixel pitch into an image with a coarse pixel pitch, not only does the size of the parallax image become smaller and the image density of the parallax image decrease, but also there arises a problem that a high-resolution image required for other recognition processes cannot be used.

[0010] In addition, an image with a coarse pixel pitch (low sampling frequency) has only signal components up to a frequency that is half of the low sampling frequency originally determined by the coarse pixel pitch according to the sampling theorem. On the other hand, an image with a fine pixel pitch (high sampling frequency) has signal components up to a frequency that is half of the high sampling frequency determined by the fine pixel pitch. Even when they are converted to the same pixel pitch by interpolation processing, there is a problem that they do not become exactly the same image.

[0011] Furthermore, in a stereo camera including an optical system having special projection characteristics using a wide-angle lens or a curved mirror with a large distortion, etc., it is calculated as a lattice point on a new image space by geometric conversion operations such as image enlargement, reduction, and rotation called geometric correction or resampling, resampling.

[0012] The luminance value on the newly generated image by resampling is obtained by interpolation processing from the geometrically transformed point of the original sampling point and the luminance value on the original sampling point because the lattice points in the newly generated image space are not necessarily on the geometrically transformed points of the original sampling points. Since the amount of geometric deformation such as distortion varies depending on the points on the image, the sampling frequency conversion in sampling / resampling is not spatially uniform and has a distribution on the imaging surface. to change

[0013] ​In a stereo camera, although geometric correction is performed because the block positions of the reference image and the search image are different, even for images of the same subject, differences in signals based on the sampling theorem occur, resulting in problems such as degradation of parallax performance such as parallax accuracy and parallax rate.

[0014] An object of the present invention is to provide a distance measuring device capable of performing high-precision parallax detection even when using image sensors with different pixel pitches, lenses with different focal lengths, or lenses with distortion, in view of the above.

[0015] The above and other objects and novel features of the present invention will become apparent from the description of this specification and the accompanying drawings.

Means for Solving the Problems

[0016] The outline of a typical example of the present invention will be briefly described as follows.

[0017] A distance measuring device according to an embodiment is For the same object, On the image sensor between the first image and the second image For consistency determination more A distance measuring device that performs distance measurement, For at least one of the first image and the second image Resampling A first correction processing unit that performs a process of adding interpolation points , A second correction processing unit that performs a geometric correction process on at least one of the image data of the resampled first image and the image data of the second image.

[0018] Problems, configurations, and effects other than those described above will be clarified by the description of the embodiments for carrying out the following invention.

Effects of the Invention

[0019] According to the present invention, a distance measuring device capable of performing high-precision parallax detection can be provided.

Brief Description of the Drawings

[0020]

Figure 1

Figure 2A

Figure 2B

Figure 3A

Figure 3B

Figure 3C

Figure 4A

Figure 4B

Figure 4C

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Best Mode for Carrying Out the Invention

[0021] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The embodiments or examples are illustrative for explaining the present invention, and for the sake of clarity of explanation, appropriate omissions and simplifications are made. The present invention can be implemented in various other forms. Unless otherwise particularly limited, each component may be in a single number or a plurality.

[0022] In the drawings, the positions, sizes, shapes, ranges, etc. of the respective components shown may not represent the actual positions, sizes, shapes, ranges, etc. in order to facilitate understanding of the invention. For this reason, the present invention is not necessarily limited to the positions, sizes, shapes, ranges, etc. disclosed in the drawings.

[0023] When there are a plurality of components having the same or similar functions, they may be described by attaching different subscripts to the same reference numeral. Further, when it is not necessary to distinguish these plurality of components, the subscripts may be omitted in the description.

[0024] In all the drawings for explaining the embodiments, the same components and the same members are generally given the same reference numerals, and the repeated description thereof may be omitted. The drawings may be represented schematically compared with the actual aspect in order to make the explanation clearer, but it is merely an example and does not limit the interpretation of the present invention.

Examples

[0025] The distance measuring device according to the first embodiment of the present invention is a distance measuring device 6 having a stereo camera SC composed of an image pickup device 30a and an image pickup device 30b with different pixel pitches, as shown in FIG. 3A described later.

[0026] First, an example of distance calculation to a subject by a stereo ranging method will be described with reference to FIG. 1. FIG. 1 is a diagram for explaining an example of distance calculation to a subject 4 by a stereo ranging method when using two cameras 2, i.e., camera 2a and camera 2b. Camera 2a and camera 2b that constitute the stereo camera SC of the distance measurement device (also referred to as a ranging device) 6r are installed side by side with a distance B (baseline length) between them such that the optical axes 3a and 3b are parallel to each other.

[0027] Light rays emitted from the subject 4 pass through the optical centers of the lens 5a of camera 2a and the lens 5b of camera 2b, and are imaged on the imaging element 1a of camera 2a and the imaging element 1b of camera 2b. The optical axis 3a and the optical axis 3b represent the optical axes 3 of the lens 5a and the lens 5b. At this time, the subject 4 is imaged at a position on the imaging element 1a that is separated from the optical axis 3a by a distance La, and is also imaged at a position on the imaging element 1b that is separated from the optical axis 3b by a distance Lb, and a parallax L (= La - Lb) occurs between camera 2a and camera 2b. This parallax L changes depending on the magnitude of the distance D from the cameras 2a and 2b of the distance measurement device 6r to the subject 4. If the focal lengths of both camera 2a and camera 2b are the focal length f, the distance D to the subject 4 is represented by the following (Equation 1).

[0028] D = f(B / L) (Equation 1) The imaging elements 1a and 1b are usually composed of light receiving elements arranged at equal pitch points in a grid pattern and semiconductor devices such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). Therefore, the luminance signal of the subject image is sampled at points discretized at equal intervals on a two-dimensional plane.

[0029] Next, problems when using image sensors with different pixel pitches in a stereo camera will be described with reference to FIGS. 2A and 2B. In FIGS. 2A and 2B, a case where an image sensor 10a with a coarse pixel pitch and an image sensor 10b with a finer pixel pitch compared to the image sensor 10a are used as image sensors with different pixel pitches will be described. FIG. 2A is a diagram showing a state where the same image is formed on a two-dimensional plane when image sensors with different pixel pitches are used and they are sampled and resampled. FIG. 2B is a diagram for explaining by one-dimensionally showing the state of FIG. 2A for simplicity.

[0030] In FIG. 2A, a two-dimensional plane 22 shows a state where an image formed on the two-dimensional plane of the image sensor 10a with a coarse pixel pitch is sampled. A two-dimensional plane 21 shows a state where an image formed on the two-dimensional plane of the image sensor 10b with a finer pixel pitch compared to the image sensor 10a is sampled. It is assumed that the same image is formed on the two-dimensional plane of the image sensor 10a and the two-dimensional plane of the image sensor 10b. A two-dimensional plane 21r shows a state where the sampled two-dimensional plane 21 is resampled. A two-dimensional plane 24 shows a state where the sampled two-dimensional plane 22 is resampled. By resampling, the coarse pixel pitch of the two-dimensional plane 22 is converted to match the finer pixel pitch of the two-dimensional plane 21r (21), and a two-dimensional plane 24 with a finer pixel pitch is configured.

[0031] In FIG. 2B, a one-dimensional sampled state 26 of the image sensor 10a with a coarse pixel pitch and a one-dimensional sampled state 25 of the image sensor 10b with a finer pixel pitch are shown. Also, a resampled state 25r obtained by resampling the sampled state 25 and a resampled state 28 obtained by resampling the sampled state 26 are shown. In the resampled state 28, interpolation points indicated by black circles are added by resampling.

[0032] In the sampling states 25 and 26 of FIG. 2B, as indicated by the dotted line as the "image on the imaging device", the image itself formed on the surfaces of the imaging devices 10a and 10b by the lenses (for example, the lenses 5a and 5b in FIG. 1) has a continuous light distribution, but these are sampled according to the pixel pitch of the imaging devices 10a and 10b. According to the sampling theorem, assuming that the sampled signal is composed of only frequency components below half of the sampling frequency, the imaging device 10b with a fine pixel pitch has higher spatial frequency components and maintains a shape closer to the original continuous light distribution. On the other hand, since the imaging device 10a with a coarse pixel pitch cannot have high frequency components, it deteriorates from the signal of the original continuous light distribution shape (signal deterioration occurs due to sampling).

[0033] In stereo matching, the comparison between image blocks is performed using the technique of template matching. However, since the pixel pitches of the imaging device 10a and the imaging device 10b are different, it is necessary to convert them to the same pixel pitch by interpolation processing or resampling processing. In a system of a distance measurement device in which imaging devices (10a, 10b) with different pixel pitches are mixed, the reason for adopting the imaging device (10b) with a fine pixel pitch is that generally an image with a fine pixel pitch is required, so the sampling frequency of resampling often uses the sampling frequency of the imaging device (10b) with a fine pixel pitch. However, as described above, since the signal itself of the imaging device (10a) with a coarse pixel pitch has only signal components with a frequency that is half of the sampling frequency determined by the coarse pixel pitch, the image (28) after resampling of the imaging device (10a) with a coarse pixel pitch becomes a deteriorated image with few frequency components.

[0034] Here, the resampling process may be performed on the entire image at once, or may be performed each time the reference image and the search image are selected during the stereo matching process. In the latter case, the required storage area can be smaller.

[0035] In template matching, since the correlation value between two images is calculated to determine the match between the two images, if there are differences in the error due to the conversion of resampling processing, the correlation value will decrease and the parallax performance will deteriorate.

[0036] Next, with reference to FIGS. 3A, 3B, and 3C, an image correction method after resampling in Example 1 will be described. FIG. 3A is a diagram showing a configuration example of the distance measurement device in Example 1. FIG. 3B is a diagram showing an example of the image correction method after resampling in Example 1. FIG. 3C is a diagram showing another example of the image correction method after resampling in Example 1.

[0037] The difference between the distance measurement device 6 shown in FIG. 3A and the distance measurement device 6r in FIG. 1 is that the distance measurement device 6 employs a stereo camera composed of an image sensor 30a and an image sensor 30b with different pixel pitches. Other configurations of the distance measurement device 6 shown in FIG. 3A are the same as those of the distance measurement device 6r in FIG. 1.

[0038] As shown in FIG. 3A, the cameras 2a and 2b that constitute the stereo camera SC of the distance measurement device (also referred to as a distance measuring device) 6 are arranged side by side with a distance B (baseline length) between them so that the optical axes 3a and 3b are parallel. The light rays emitted from the subject 4 pass through the optical centers of the lenses 5a of the camera 2a and the lens 5b of the camera 2b and form images on the image sensors 30a of the camera 2a and the image sensor 30b of the camera 2b. The optical axis 3a and the optical axis 3b represent the optical axes 3 of the lenses 5a and 5b. At this time, the subject 4 forms an image at a position separated from the optical axis 3a by a distance La on the image sensor 30a, and also forms an image at a position separated from the optical axis 3b by a distance Lb on the image sensor 30b, and a parallax L (= La - Lb) occurs between the cameras 2a and 2b. This parallax L changes depending on the magnitude of the distance D from the cameras 2a and 2b of the distance measurement device 6 to the subject 4. If the focal lengths of both the camera 2a and the camera 2b are the focal length f, the distance D to the subject 4 is represented by the previously described (Equation 1).

[0039] In the resampling (also referred to as resampling) described with reference to FIGS. 2A and 2B, since it is known which components of the image signal above which frequency are deteriorated by the resampling, in this embodiment, a correction process for correcting the difference in signal deterioration associated with the resampling is performed. In the example shown in FIG. 3B, a correction process for restoring the signal reduction of the image sensor 30a with a coarse pixel pitch by super-resolution processing is performed. Further, in the example shown in FIG. 3C, correction processing such as decimation filter (DEF) processing for blocking high-frequency components is performed on the image data from the image sensor 30b with a fine pixel pitch.

[0040] FIG. 3B shows the one-dimensional sampled state 32 of the image sensor 30a with a coarse pixel pitch and the one-dimensional sampled state 31 of the image sensor 30b with a fine pixel pitch. Thereafter, resampling is performed on the sampling state 32 and the sampling state 31. Here, it is assumed that the sampling frequency of resampling uses the sampling frequency of the image sensor (30b) with a fine pixel pitch. The resampled state 33 obtained by resampling the sampling state 32 is shown. The resampled state 31r obtained by resampling the sampling state 31 is shown. Thereby, the pixel pitches are aligned between the resampled state 33 and the resampled state 31r.

[0041] In the resampled state 33, interpolation points indicated by black circles are added by resampling. On the other hand, the resampled state 31r obtained by resampling the sampling state 31 is basically the same.

[0042] Thereafter, super-resolution processing is performed on the image data in the resampled state 33. In the super-resolution processing, a correction process for restoring the image data of the signal reduction of the image sensor 30a with a coarse pixel pitch is performed, and the resampled state 33 is converted into the state 34 of the image data after the super-resolution processing. As a result, the error difference between the resampled state 31r and the state 34 of the image data after the super-resolution processing is corrected.

[0043] In FIG. 3C, similar to FIG. 3B, a one-dimensional sampling state 32 of the imaging device 30a with a coarse pixel pitch, a one-dimensional sampling state 31 of the imaging device 30b with a fine pixel pitch, and a resampling state 33 obtained by resampling the sampling state 32 are shown. A resampling state 31r obtained by resampling the sampling state 31 is shown. Here, it is assumed that the sampling frequency for resampling is the sampling frequency of the imaging device (30b) with a fine pixel pitch. Thereby, the pixel pitches are aligned between the resampling state 33 and the resampling state 31r.

[0044] After resampling, decimation filter (DEF) processing is performed on the image data in the resampling state 31r. In the decimation filter (DEF) processing, correction processing is performed to block the high-frequency components of the image data from the imaging device 30b with a fine pixel pitch, and the resampling state 31r is converted into an image data correction state 36 after the decimation filter (DEF) processing. Thereby, the error difference between the image data correction state 36 after the decimation filter (DEF) processing and the resampling state 33 is corrected.

[0045] That is, after resampling, by performing correction processing (restoration processing of signal reduction) by super-resolution processing and correction processing (blocking processing of high-frequency components) by decimation filter processing, the difference in signal degradation between the left and right images (31r and 33) generated during resampling is corrected. Thereby, for the left and right image blocks compared during the stereo matching process, the error between the left and right image blocks due to left and right sampling and interpolation by resampling of the block image portion is corrected, and the stereo matching process can accurately calculate the parallax L, and a distance measuring device 6 that can obtain the distance D to the subject 4 with high precision can be provided.

[0046] After the correction process by super-resolution processing (restoration process for signal degradation) or the correction process by decimation filter processing (cut-off process for high-frequency components), as described with reference to FIG. 3A, the distance measurement device 6 calculates the parallax L.

[0047] In the following description, the image data after the correction process by super-resolution processing (restoration process for signal degradation) or the correction process by decimation filter processing (cut-off process for high-frequency components) (for example, states 31r and 34 in FIG. 3B, and states 36 and 33 in FIG. 3C) will be referred to as the image after resampling or the image after the resampling process. Also, the correction process by super-resolution processing (restoration process for signal degradation) or the correction process by decimation filter processing (cut-off process for high-frequency components) will be referred to as the resampling process.

[0048] As described above, the resampling process may be performed on the entire image at once, or may be performed each time a reference image and a search image are selected during the stereo matching process. Hereinafter, as a representative example, an example will be described in which the stereo matching process is performed after the resampling process is performed on the entire screen in advance.

[0049] The stereo matching process of this embodiment will be described with reference to FIGS. 4A, 4B, and 4C. FIG. 4A is a diagram for explaining the calculation of the SAD, which is the correlation value for each partial image region. FIG. 4B is a diagram showing the transition of the SAD when the image block is moved one pixel at a time. FIG. 4C is a diagram for explaining the sub-pixel parallax of the equiangular line fitting.

[0050] As shown in FIG. 4A, for the image after resampling from the image sensor 30a and the image after resampling from the image sensor 30b, the SAD (Sum of Absolute Difference), which is the correlation value for each partial image region (block region, template image), is calculated. The SAD is represented by Equation 2. One block region is, for example, an 8-pixel × 8-pixel image region.

[0051] Then, the distance measurement device 6 calculates the displacement of the image on the image at the pixel pitch of the resampled images of the imaging device 30a and the imaging device 30b, that is, the parallax L, using the calculated correlation value. Note that SAD is an example of a correlation value, and the distance measurement device 6 can also use generally known SSD (Sum of Squared Difference) or NCC (Normalized Cross-Correlation) as the correlation value.

[0052] First, a partial region of the reference image REF is selected from the resampled image from the imaging device 30a. Next, a partial region of the comparison image CMP is selected from the resampled image from the imaging device 30b, and a correlation value such as SAD is calculated for each partial image region (REF and CMP). Next, the partial region of the comparison image CMP is shifted, for example, to the right by one pixel pitch of the resampled pixel pitch from the resampled image from the imaging device 30b, and the correlation value such as SAD of the partial image region (REF and CMP) after the shift is calculated again. FIG. 4B shows the transition of SAD when the image block is moved by one pixel at a time. The pixel shift amount (shift amount) SG at which SAD becomes minimum becomes the parallax (L) in units of one pixel. Here, SG is the shift amount at which SAD becomes minimum (integer parallax), S(0) is the correlation value (SAD) at the shift amount SG at which SAD becomes minimum, and the SAD at the adjacent shift amounts are S(-1) and S(1).

[0053] In the cases of FIGS. 4A and 4B, since the detection accuracy of the parallax (L) is one pixel, the parallax with a precision of less than one pixel (hereinafter referred to as sub-pixel parallax) cannot be obtained.

[0054] Distance measurement accuracy (ranging accuracy), that is, parallax detection resolution. As a method for further obtaining high precision not in units of one pixel, a method for estimating sub-pixel level parallax has been proposed. For example, in the sub-pixel parallax estimation method called equiangular line fitting, as shown in Fig. 4C, by assuming that the transition of SAD has the same absolute value of the slope on the left and right with respect to the actual parallax, the actual parallax is estimated at the sub-pixel level by linear interpolation. The sub-pixel parallax calculation formula of equiangular line fitting, that is, the interpolation formula, is shown in (Equation 3). Here, δ is the sub-pixel parallax, SG is the shift amount at which SAD is minimized (integer parallax), S(0) is the correlation value (SAD) at the shift amount SG at which SAD is minimized, and the SAD at the adjacent shift amounts are S(-1) and S(1).

[0055] By multiplying the parallax calculated by (Equation 3) by the pixel pitch size, the parallax L in (Equation 1) is obtained, and the distance D to the subject 4 is calculated.

[0056] Using Fig. 5, a configuration example of the distance measurement device 6 will be described. Fig. 5 is a diagram showing a configuration example of the distance measurement device according to the first embodiment. Note that the distance measurement device 6 in Fig. 5 is an example in which the correction process (decimation filter process) described in Fig. 3C is performed on the search image (comparison image) every time a reference image is selected and a search image (comparison image) is selected during the stereo matching process.

[0057] The distance measurement device 6 includes a first camera 2a, a second camera 2b, a first resampling circuit 40a, a second resampling circuit 40b, and a parallax calculation circuit 50.

[0058] The first camera 2a includes a first optical system including a lens 5a and a first imaging device 30a. The second camera 2b includes a second optical system including a lens 5b and a second imaging device 30b. The pixel pitch of the first imaging device 30a is set to be a coarser pixel pitch compared to the pixel pitch of the second imaging device 30b. That is, the pixel pitch of the second imaging device 30b is set to be a finer pixel pitch compared to the pixel pitch of the first imaging device 30a. In this example, the focal lengths of the first camera 2a and the second camera 2b are the same, both being the focal length f.

[0059] The first resampling circuit 40a is provided to resample the image formed on the two-dimensional plane of the imaging device 30a after sampling by the imaging device 30a (see the resampling state 33 in FIG. 3C). The second resampling circuit 40b is provided to resample the image formed on the two-dimensional plane of the imaging device 30b after sampling by the imaging device 30b (see the resampling state 31r in FIG. 3C). Here, the sampling frequency of resampling uses the sampling frequency of the imaging device (30b) with a finer pixel pitch. Thereby, the pixel pitch of the image (33) from the first imaging device 30a and the pixel pitch of the image (31r) from the second imaging device 30b are made to be the same pixel pitch. Here, the images from the first imaging device 30a and the second imaging device 30b with the same pixel pitch will be referred to as the images after resampling.

[0060] The parallax calculation circuit 50 includes a reference position detection circuit (also referred to as a reference image position detection circuit) 51, a comparison image position detection circuit 52, a comparison image correction circuit 53, and a parallax detection circuit 54. Images (33) from the first imaging device 30a and images (31r) from the second imaging device 30b, each having the same pixel pitch, are input to the reference position detection circuit 51 and the comparison image position detection circuit 52. The reference position detection circuit 51 detects or selects a partial region of the reference image REF from the resampled image (31r) from the first imaging device 30a. The comparison image position detection circuit 52 detects or selects a partial region of the comparison image CMP from the resampled image (33) from the imaging device 30b.

[0061] The comparison image correction circuit 53 calculates a correction amount resulting from the difference between the position of the reference image REF and the position of the comparison image CMP, and corrects the comparison image CMP. Here, correction processing (high-frequency component cutoff processing) by decimation filter processing is performed on the comparison image CMP.

[0062] The parallax detection circuit 54 detects the parallax L from the reference image REF and the corrected comparison image CMP. Thereby, the distance measurement device 6 can calculate the accurate distance D to the subject 4 from the detected parallax L using (Equation 1).

[0063] When performing correction processing (restoration processing of signal degradation) by super-resolution processing in FIG. 3B, the comparison image correction circuit 53 is deleted from FIG. 5, and instead, a reference image correction circuit is provided between the output of the reference position detection circuit 51 and one input of the parallax detection circuit 54. And in this case, the output of the comparison image position detection circuit 52 is connected to the other input of the parallax detection circuit 54.

[0064] Before the stereo matching process, when performing correction processing (restoration processing of signal degradation) by super-resolution processing and correction processing (high-frequency component cutoff processing) by decimation filter processing on the entire image as shown in FIGS. 3B and 3C, the following can be done.

[0065] In the case of FIG. 3B, a correction circuit that performs correction processing by super-resolution processing is provided between the first resampling circuit 40a and the reference position detection circuit 51.

[0066] In the case of FIG. 3C, a correction circuit that performs correction processing by decimation filter processing is provided between the second resampling circuit 40b and the comparison image position detection circuit 52.

[0067] An example of the distance measurement method of the distance measurement device 6 in FIG. 5 will be described with reference to FIG. 6. FIG. 6 is a flowchart showing the distance measurement method according to the first embodiment.

[0068] Step S1: An image of the subject 4 is acquired using the first image sensor 30a of the first camera 2a. Also, an image of the subject 4 is acquired using the second image sensor 30b of the second camera 2b. The pixel pitch of the first image sensor 30a is coarser compared to the pixel pitch of the second image sensor 30b.

[0069] Step S2: In the image of the subject 4 acquired using the first image sensor 30a of the first camera 2a, a reference image block position is set. The reference image block position is set, for example, by the reference position detection circuit 51.

[0070] Step S3: In order to align the left and right pixel pitches, resampling is performed using the first resampling circuit 40a and the second resampling circuit 40b. As a result, the pixel pitch of the image (33) from the first image sensor 30a and the pixel pitch of the image (31r) from the second image sensor 30b are aligned to be the same pixel pitch.

[0071] Step S4: In each of the images after resampling, the reference image position and the comparison image position are detected. The reference position detection circuit 51 detects a partial region of the reference image REF from the image (33) after resampling from the first image sensor 30a. The comparison image position detection circuit 52 detects a partial region of the comparison image CMP from the image (31r) after resampling from the image sensor 30b.

[0072] Step S5: The comparison image correction circuit 53 calculates a correction amount resulting from the difference between the reference image position and the comparison image position detected in step S4. Here, for the image (31r) after resampling from the image sensor 30b, a correction amount for performing correction processing (high-frequency component cutoff processing) by decimation filter processing is calculated.

[0073] Step S6: The comparison image correction circuit 53 performs correction processing by decimation filter processing on the image (31r) after resampling from the image sensor 30b with the correction amount calculated in step S5.

[0074] Step S7: The disparity detection circuit 54 detects the disparity L from the reference image REF and the corrected comparison image CMP.

[0075] Step S8: The disparity detection circuit 54 determines whether the block position of the reference image REF used in step S7 is the final reference image block position. If the block position of the reference image REF used in step S7 is the final reference image block position (Yes), the flow of the distance measurement method ends. On the other hand, if the block position of the reference image REF used in step S7 is not the final reference image block position (No), the process proceeds to step S2. Thereafter, steps S2 to S7 are repeatedly executed until the block position of the reference image REF becomes the final reference image block position.

[0076] According to the first embodiment, one or more of the following effects can be obtained.

[0077] 1) Even when image sensors with different pixel pitches are used in a stereo camera, the pixel pitches of the left and right images can be made uniform by resampling.

[0078] 2) After resampling, by performing correction processing (restoration processing of signal degradation) by super-resolution processing or correction processing (cut-off processing of high-frequency components) by decimation filter processing, the difference in signal degradation between the left and right images (31 and 33) that occurs during resampling can be corrected.

[0079] 3) By the above 2), for the left and right image blocks to be compared during stereo matching processing, the error between the left and right image blocks due to left and right sampling and interpolation by resampling of the block image portion can be corrected.

[0080] 4) Even when image sensors with different pixel pitches are used in a stereo camera, a distance measuring device 6 that can accurately calculate the parallax L by stereo matching processing and obtain the distance D to the subject 4 with high precision can be provided.

Example

[0081] The second embodiment (Example 2) of the present invention relates to a distance measuring device using different lenses. FIG. 7 is a diagram showing a configuration example of a distance measuring device when two lenses with different focal lengths are used in a stereo camera in Example 2. FIG. 8 is a diagram showing a configuration example of the distance measuring device in Example 2.

[0082] FIG. 7 shows a configuration example of a distance measuring device 6a having lenses 70a and 70b with different focal lengths in two cameras 200a and 200b constituting a stereo camera SC. Here, a stereo camera SC using a camera 200a having a lens 70a with a long focal length fa and a camera 200b having a lens 70b with a short focal length fb is shown.

[0083] The same subject 4 is imaged on the imaging device 1a through the lens 70a, and the size of the image (Sa) is different from the size of the image (Sb) imaged on the imaging device 1b through the lens 70b (Sb < Sa). These are sampled as image data by the imaging devices 1a and 1b with the same pixel pitch.

[0084] Since these images cannot be directly used for stereo matching as they are, it is necessary to perform enlargement or reduction conversion on the images so that they have the same size with the same pixel pitch. When converting to the same image size, there are more sampling points in the original larger camera 200b and fewer sampling points in the original smaller camera 200a, resulting in a difference in the effective sampling frequency and a difference in image characteristics due to resampling.

[0085] Therefore, similar to the first embodiment, by performing correction processing to compensate for the difference in image characteristics due to resampling, it is possible to improve parallax performance such as parallax accuracy and parallax rate.

[0086] As shown in FIG. 8, the distance measurement device 6a uses the first camera 200a and the second camera 200b as a stereo camera SC. has The first camera 200a includes a first optical system including a lens 70a with a long focal length fa and an imaging device 1a. The second camera 200b includes a second optical system including a lens 70b with a short focal length fb (fb < fa) and an imaging device 1b.

[0087] Since the first resampling circuit 41a and the second resampling circuit 41b have different image sizes for the image formed on the imaging device 1a and the image formed on the imaging device 1b, they are configured to perform resampling after performing enlargement or reduction conversion on the images so that they have the same size with the same pixel pitch.

[0088] The comparison image correction circuit 53a performs correction processing to compensate for the difference in image characteristics due to resampling on the comparison image.

Embodiment

[0089] The third embodiment (Example 3) of the present invention will be described in detail below with reference to the drawings. The third embodiment (Example 3) of the present invention relates to a distance measuring device using a distorted lens. FIG. 9 is a diagram for explaining the influence of lens distortion. FIG. 10 is a diagram for explaining the influence of lens distortion. FIG. 11 is a diagram for explaining stereo matching in Example 3.

[0090] FIG. 9 shows the dot pattern DA (indicated by white circles) that should originally be imaged and the actual dot pattern DB (indicated by black circles) when using a distorted lens. The square grid shape formed by the actual dot pattern DB has less distortion in the central part CEN of the image, and the square shape is distorted as it goes towards the edge PER of the image. Also, at the edge PER of the image, the area of the quadrilateral formed by the four grid points is smaller compared to the area of the quadrilateral in the central part CEN. That is, a phenomenon occurs where the image becomes smaller when the same subject 4 is imaged at the peripheral part of the imaging device than when it is imaged at the central part of the imaging device (size of the image of subject 4 imaged at the central part of the imaging device > size of the image of subject 4 imaged at the peripheral part of the imaging device).

[0091] Using FIG. 10, the error caused by this phenomenon will be explained. FIG. 10 schematically shows one-dimensionally an image with large distortion that is imaged at the peripheral part PER of the imaging device away from the optical axis and an image that is imaged at the central part CEN of the imaging device near the optical axis. It is desirable that the image that should originally be imaged (referred to as the original image) is the same in both the peripheral part PER and the central part CEN. However, the peripheral part PER is greatly affected by lens distortion and is actually imaged as a small image (referred to as the actual image) (see state 61). Since the central part CEN is less affected by lens distortion, the original image and the actual image are imaged almost the same (see state 62).

[0092] As shown in states 63 and 64, since the pixel pitch of the imaging device is constant, the small image of the peripheral part PER and the large image of the central part CEN are sampled at the same sampling frequency. Even for images from the same subject 4, the number of sampling points in the peripheral region PER with large distortion is small (3 points in state 63), and the number of sampling points in the central region CEN with small distortion is large (7 points in state 64).

[0093] When performing lens distortion correction, it is necessary to perform image enlargement / reduction correction according to the amount of distortion (in this case, enlargement correction). Since the peripheral part PER with large distortion is highly reduced, it is necessary to enlarge it a lot (see state 65), and the central part CEN with almost no distortion is enlarged at approximately the same magnification (see state 66). However, for the number of sampling points for images from the same subject 4, the peripheral part PER has few (3 points) and the central part CEN has many (7 points), so in order to increase the pixel points in the peripheral part, it is necessary to perform resampling by interpolation processing (see states 67 and 68). At this time, the image after conversion of the peripheral part PER deteriorates significantly.

[0094] In stereo matching using an optical system with a lens having such a large distortion or a special projection method, even for cameras using lenses with the same focal length and imaging devices with the same pixel pitch, problems occur because the positions of the reference image block and the search image block are different, and thus the distortion correction at each position is different.

[0095] The stereo matching in Example 3 will be described with reference to FIG. 11. FIG. 11 illustrates, as an example, stereo matching in the case of using a stereo camera including two imaging devices (1a, 1b) as in Examples 1 and 2. It should be understood by those skilled in the art that Example 3 is also applicable to the case of a single monocular camera including one imaging device.

[0096] As shown in FIG. 11 to When the image positions of the reference block (partial region of the reference image) REF and the search block (partial region of the comparison image) CMP (CMP1) are the same (i.e., when the pixel shift amount is 0), the lens distortion of the reference block REF and the search block CMP1 is the same, so the same resampling process is performed. However, as the image shift amount increases, the difference in position between the position of the reference block REF and the positions of the search blocks CMP2 and CMP3 becomes larger, and the resampling process of the reference block REF and the resampling process of the search block will be different. Therefore, during the stereo matching process, the image position of the reference image block REF and the image position of the comparison image block CMP being searched are detected, and by correcting only the error difference in resampling caused by that image position difference, the difference in the correction amount due to resampling can be minimized.

[0097] That is, in a distance measuring device using a distorted lens, a detection circuit for detecting the image position of the reference image block REF and the image position of the comparison image block CMP being searched, and a correction circuit for correcting only the error difference in resampling caused by that image position difference are provided.

[0098] Next, with reference to FIG. 12, a configuration example of a distance measuring device using a monocular camera with a distorted lens in Example 3 will be described. FIG. 12 is a diagram showing a configuration example of the distance measuring device in Example 3.

[0099] As shown in FIG. 12, the distance measuring device 6b includes a camera 300, a viewpoint separation circuit 90, a first viewpoint imaging unit resampling circuit 41c, a second viewpoint imaging unit resampling circuit 41d, and a parallax calculation circuit 50. The parallax calculation circuit 50 includes a reference position detection circuit (reference image position detection circuit) 51, a comparison image position detection circuit 52, a comparison image correction circuit 53b, a parallax detection circuit 54, and a reference image correction circuit 55.

[0100] The camera 300 includes an optical system including a distorted lens 80 and an imaging device 1c. An image formed on the two-dimensional plane of the imaging device 1c via the lens 80 is sampled by the imaging device 1c.

[0101] The viewpoint separation circuit 90 selects a first viewpoint imaging region and a second viewpoint imaging region in the image sampled by the imaging device 1c. The viewpoint separation circuit 90 supplies first viewpoint image data corresponding to the first viewpoint imaging region to the resampling circuit 41c of the first viewpoint imaging region portion. The viewpoint separation circuit 90 also supplies second viewpoint image data corresponding to the second viewpoint imaging region to the resampling circuit 41d of the second viewpoint imaging region portion.

[0102] The resampling circuit 41c of the first viewpoint imaging region portion detects the position above the imaging device 1c of the first viewpoint image data, and performs resampling processing of the first viewpoint image data based on that position. The resampling circuit 41c performs geometric transformations such as enlargement, reduction, and rotation, and interpolation processing based on the position above the imaging device 1c of the first viewpoint image data during the resampling process in order to correct the distortion of the lens 80.

[0103] The resampling circuit 41d of the second viewpoint imaging region portion detects the position above the imaging device 1c of the second viewpoint image data, and performs resampling processing of the second viewpoint image data based on that position. The resampling circuit 41d performs geometric transformations such as enlargement, reduction, and rotation, and interpolation processing based on the position above the imaging device 1c of the second viewpoint image data during the resampling process in order to correct the distortion of the lens 80. As a result, the pixel pitch of the resampled first viewpoint image data and the resampled second viewpoint image data is made the same. The resampled first viewpoint image data is supplied to the reference position detection circuit 51, and the resampled second viewpoint image data is supplied to the comparison position detection circuit 52.

[0104] The reference position detection circuit 51 detects or selects a partial region of the reference image REF from the first viewpoint image data after resampling. The comparison image position detection circuit 52 detects or selects a partial region of the comparison image CMP from the second viewpoint image data after resampling.

[0105] The reference image correction circuit 55 calculates a correction amount caused by the position of the reference image REF on the first viewpoint image data after resampling, and corrects the reference image REF. That is, the reference image correction circuit 55 performs a correction process for correcting the signal degradation of the image of the first viewpoint image data generated by the resampling process.

[0106] The comparison image correction circuit 53b calculates a correction amount caused by the position of the comparison image CMP on the second viewpoint image data after resampling, and corrects the comparison image CMP. That is, the comparison image correction circuit 53b performs a correction process for correcting the signal degradation of the image of the second viewpoint image data generated by the resampling process.

[0107] The disparity detection circuit 54 detects the disparity L from the reference image REF corrected by the reference image correction circuit 55 and the comparison image CMP corrected by the comparison image correction circuit 53b. Thus, the distance measurement device 6b using the distorted lens 80 can calculate the accurate distance D to the subject 4 from the detected disparity L using (Equation 1).

[0108] Note that the reference image correction circuit 55 can be deleted. In this case, the comparison image correction circuit 53b is configured to calculate a correction amount caused by the difference between the position of the reference image REF on the first viewpoint image data after resampling and the position of the comparison image CMP on the second viewpoint image data after resampling, and perform a correction process for correcting the comparison image CMP. The disparity detection circuit 54 detects the disparity L from the reference image REF and the comparison image CMP corrected by the comparison image correction circuit 53b.

[0109] As a result, the distance measuring device 6b using the distorted lens 80 can calculate the accurate distance D to the subject 4 from the detected parallax L using (Equation 1).

[0110] The configuration examples of the distance measuring devices in Embodiments 1, 2, and 3 can be summarized as follows.

[0111] 1) A distance measuring device (6, 6a, 6b) that images on an imaging device, detects the image positions of at least two images (REF, CMP) of the same object (subject 4) on the imaging device by determining the degree of coincidence of the two images, and performs distance measurement (measurement of distance), The images on the imaging device are sampled at the pixel pitch of the imaging device (30a, 30b, 1a, 1b, 1c), When determining the degree of coincidence of the images (REF, CMP) at the time of detecting the image positions, geometric correction (Embodiment 1: super-resolution processing, decimation filter processing, Embodiment 2: magnification conversion or reduction conversion, Embodiment 3: geometric conversions such as magnification, reduction, and rotation) and resampling processing (interpolation processing) are performed, Correct the signal degradation of the image generated during the resampling process.

[0112] 2) In the distance measuring device of 1) above (see Embodiment 3), The two images (reference image REF, comparison image CMP) for determining the degree of coincidence of the above images are on one imaging device (1c).

[0113] 3) In the distance measuring device of 1) above (see Embodiment 1), The distance measuring device according to claim 1, The two images (reference image REF, comparison image CMP) for determining the degree of coincidence of the above images are on two imaging devices (30a, 30b) having different pixel pitches, respectively.

[0114] 4) In the distance measuring device of 1) above (see Embodiment 2), The two images (reference image REF, comparison image CMP) for determining the degree of coincidence of the above images are formed by optical systems each composed of lenses (70a, 70b) having different focal lengths.

[0115] 5) In the distance measuring device of 1) above (Examples 1, 2, and 3), When performing the degree of coincidence determination of the images (reference image REF, comparison image CMP) at the time of detecting the image position, geometric correction and resampling processing are performed on the image data on the image sensor necessary for the determination of at least one of the images. A distance measuring device that corrects signal degradation of the image generated during the resampling process.

[0116] 6) In the distance measuring device of 1) above (Example 3), Detect the respective positions on the image sensor of the two images (reference image REF, comparison image CMP) for which the degree of coincidence is determined. Correct the signal degradation of the images generated by the respective geometric corrections and resampling processes at the two image positions.

[0117] In other words, the distance measuring device of the present invention has a geometric correction unit that makes the same object (subject 4) have the same image size during or before the corresponding point search of the stereo camera, and a resampling processing unit that enables comparison at the same pixel pitch. The resampling processing unit performs resampling processing and correction processing for correcting the difference in signal degradation associated with the resampling of the reference image and the search image.

[0118] Furthermore, when performing the corresponding point search of the images in the parallax detection, extract the image data on the image sensor necessary for the degree of coincidence evaluation for at least one of the reference image and the search image, perform geometric correction and resampling processing, and correct the difference in signal degradation associated with the resampling of the reference image and the search image.

[0119] Furthermore, when performing the corresponding point search, detect the respective positions on the image sensors of the two images of the reference image and the search image, and correct the signal degradation of the images generated by the respective geometric corrections and resampling processes at the two image positions.

[0120] As a result, the following effects can be obtained.

[0121] 1) It is possible to provide a stereo camera composed of an optical system with special projection characteristics using a stereo camera with different types of cameras or a wide-angle lens, a curved mirror, etc.

[0122] 2) It is possible to eliminate problems caused by image enlargement / reduction by different camera lenses and image sensors with different pixel pitches, improve the performance of matching image search in a stereo camera, and improve the parallax detection accuracy using different cameras.

[0123] 3) In a stereo camera with cameras having different sizes, pixel pitches, etc. of image sensors (sensors), when performing stereo matching and converting the left and right to the same image space by geometric correction, since the sampling frequencies of the sensors are different, the characteristics of the converted images will be different. Also, in a stereo camera with an optical system such as a high-distortion lens, a lens with a special projection method, or a hyperbolic mirror, when performing pixel position conversion by geometric transformation (distortion correction / geometric correction) for stereo matching, the original sampling points on the sensor do not exist on the new pixel positions in a grid pattern at equal intervals but exist at non-uniform intervals, so the characteristics of the image change depending on the image position after geometric transformation. In the distance measurement device of the present invention, for the image block after geometric correction used in stereo matching, data on the sensor before geometric correction is taken out, geometric interpolation processing is performed so that it becomes a block image with the same image size and pixel pitch on the left and right, and then the difference in the original sampling frequencies is corrected for matching evaluation. Also, the pixel positions of the reference block and the search block are detected, and the sampling difference due to the difference in the positions on the image where two stereo matchings are performed is corrected.

[0124] As described above, the invention made by the present inventor has been specifically described based on examples. However, it goes without saying that the present invention is not limited to the above-described embodiments and examples and can be variously modified.

Explanation of Reference Numerals

[0125] 1a, 1b, 1c, 30a, 30b: imaging elements, 2a, 2b, 200a, 200b, 300: cameras, 3a, 3b: optical axes, 4: subject, 5a, 5b, 70a, 70b, 80: lenses, 6, 6a, 6b: distance measurement devices, 40a, 40b, 41a, 41b: resampling circuits, 40c: first viewpoint imaging unit resampling circuit, 40d: second viewpoint imaging unit resampling circuit, 50: disparity calculation circuit, 51: reference position detection circuit, 52: comparison image position detection circuit, 53, 53a: comparison image correction circuits, 54: disparity detection circuit, 55: reference image correction circuit, 90: viewpoint separation circuit

Claims

1. A distance measurement device that performs distance measurement by determining the degree of coincidence between a first image and a second image on an image sensor, which are captured of the same object, A first correction processing unit that performs a process of adding interpolation points by resampling at least one of the first image and the second image; A second correction processing unit that performs a geometric correction process on at least one of the image data of the resampled first image and the image data of the second image; A distance measurement device, characterized by comprising the above.

2. The distance measurement device according to claim 1, characterized in that the first image and the second image for determining the degree of coincidence are on one image sensor.

3. The distance measurement device according to claim 1, characterized in that the first image and the second image for determining the degree of coincidence are on two image sensors having different pixel pitches respectively.

4. The distance measurement device according to claim 1, characterized in that the first image and the second image for determining the degree of coincidence are formed by optical systems each composed of lenses having different focal lengths.

5. The distance measurement device according to claim 1, wherein the geometric correction process includes a super-resolution process for restoring a signal degradation component, or a decimation filter process for blocking high-frequency components of image data, the geometric correction process is performed on one of the image data of the resampled first image and the image data of the second image, and the one image data after the geometric correction process is adjusted to the other image data. A distance measurement device characterized by this.

6. The distance measurement device according to claim 1, wherein the first image and the second image for determining the degree of coincidence are formed by an optical system composed of a lens having distortion, the first correction processing unit has a detection circuit that detects respective positions on the image sensor of the first image and the second image for determining the degree of coincidence, and performs a process of geometric transformation on the image data of the first image and the image data of the second image and a process of adding interpolation points based on the positions detected by the detection circuit. A distance measurement device characterized by this.

7. It includes a camera, a viewpoint separation circuit, a first viewpoint imaging unit resampling circuit, a second viewpoint imaging unit resampling circuit, and a parallax calculation circuit. The parallax calculation circuit includes a reference position detection circuit, a comparison image position detection circuit, a comparison image correction circuit, a parallax detection circuit, and a reference image correction circuit. The camera includes an optical system including a distorted lens and an imaging device. An image formed on a two-dimensional plane of the imaging device via the lens is sampled by the imaging device. The viewpoint separation circuit selects a first viewpoint imaging region and a second viewpoint imaging region in the image sampled by the imaging device, supplies first viewpoint image data corresponding to the first viewpoint imaging region to the first viewpoint imaging unit resampling circuit, and supplies second viewpoint image data corresponding to the second viewpoint imaging region to the second viewpoint imaging unit resampling circuit. The first viewpoint imaging unit resampling circuit detects a position on the imaging device of the first viewpoint image data, performs resampling processing of the first viewpoint image data based on the position, and performs geometric transformation and interpolation processing based on the position on the imaging device of the first viewpoint image data in order to correct the distortion of the lens during the resampling processing. The second viewpoint imaging unit resampling circuit detects a position on the imaging device of the second viewpoint image data, performs resampling processing of the second viewpoint image data based on the position, and performs geometric transformation and interpolation processing based on the position on the imaging device of the second viewpoint image data in order to correct the distortion of the lens during the resampling processing. The pixel pitch of the resampled first viewpoint image data and the resampled second viewpoint image data is made the same. The resampled first viewpoint image data is supplied to the reference position detection circuit, and the resampled second viewpoint image data is supplied to the comparison image position detection circuit. The reference position detection circuit detects or selects a partial region of the reference image from the resampled first viewpoint image data. The comparison image position detection circuit detects or selects a partial region of the comparison image from the resampled second viewpoint image data. The reference image correction circuit calculates a correction amount resulting from the position of the reference image on the first viewpoint image data after resampling, and corrects the reference image, thereby performing a correction process for correcting signal degradation of the image of the first viewpoint image data generated by the resampling process. The comparison image correction circuit calculates a correction amount resulting from the position of the comparison image on the second viewpoint image data after resampling, and corrects the comparison image, thereby performing a correction process for correcting signal degradation of the image of the second viewpoint image data generated by the resampling process. The parallax detection circuit detects parallax from the reference image corrected by the reference image correction circuit and the comparison image corrected by the comparison image correction circuit. A distance measurement device characterized by this.

Images