Stereo image processing device and processing method therefor

Abstract

To realize a stereo image processing device with a wide angle and high accuracy.SOLUTION: A stereo image processing device mounted on a moving object comprises: a first image acquisition unit and a second image acquisition unit that acquire images; a first parallax generation unit that performs stereo matching between a first image acquired from the first image acquisition unit and a second image acquired from the second image acquisition unit, to generate a first parallax image; an image recording unit that records at least images; a second parallax generation unit that performs stereo matching between the first image and a third image recorded in the image recording unit at a different time from the first image, to generate a second parallax image; a parallax conversion unit that converts the amount of parallax of the second parallax image to obtain a converted second parallax image; a stereo parallax comparison unit that creates a parallax correction table based on the first parallax image and the converted second parallax image; and a stereo parallax correction processing unit that corrects a parallax error of the first parallax image based on the parallax correction table.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a stereo image processing device and a processing method for the device. [Background technology]

[0002] A stereo image processing device (hereafter referred to as a stereo camera) is known as a device for recognizing objects in three dimensions. A stereo camera uses the differences in how images are captured by multiple cameras placed in different positions to detect the parallax between multiple images based on trigonometry, and then uses this parallax to detect the depth and position of an object, thereby enabling accurate detection of the position of the target object.

[0003] Stereo cameras are used in a wide range of fields, including automotive and robotics. Their application in these fields also requires a wider viewing angle. However, using a fisheye lens with a lower resolution in the wide-angle area compared to the central resolution poses challenges in terms of environmental resistance.

[0004] For example, if the sensor shifts due to temperature changes or aging as part of its environmental resistance, the image processed in stereo will have a larger image shift in the wide-angle area than in the center. When a stereo camera detects parallax in the wide-angle area, this effect causes an error in the wide-angle area (hereafter referred to as parallax error), resulting in a large distance error in the stereo processed image.

[0005] Furthermore, when the focal length changes due to factors such as the temperature characteristics of the lens as a means of environmental resistance, the image processed in stereo will have a large deviation (hereafter referred to as a shift) in the wide-angle portion of the image, even though there is no image deviation in the center. This causes a parallax error in the wide-angle portion, and a distance error in the wide-angle portion of the stereo processed image.

[0006] Patent Document 1 discloses that a common feature such as a traffic light can be used for camera calibration, eliminating restrictions on the calibration location. Patent Document 2 discloses updating calibration parameters for calculating distance using a stereo camera device equipped with two cameras.

[0007] In this way, the techniques disclosed in Patent Documents 1 and 2 correct image shift. However, it is difficult to correct image shift in the wide-angle portion due to sensor shift or changes in focal length. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Japanese Patent Application Publication No. 10-341458 [Patent Document 2] Japanese Patent Application Publication No. 2017-201334 Summary of the Invention [Problem to be solved by the invention]

[0009] The present invention has been made in view of the above-mentioned problems, and aims to provide a wide-angle, high-precision stereo image processing device. [Means for solving the problem]

[0010] a first parallax generation unit that performs stereo matching between a first image acquired from the first image acquisition unit and a second image acquired from the second image acquisition unit to generate a first parallax image; an image recording unit that records at least images; a second parallax generation unit that performs stereo matching between the first image and a third image recorded in the image recording unit at a different time from the first image to generate a second parallax image; a parallax conversion unit that converts the parallax amount of the second parallax image to obtain a converted second parallax image; a stereo parallax comparison unit that creates a parallax correction table based on the first parallax image and the converted second parallax image; and a stereo parallax correction processing unit that corrects a parallax error of the first parallax image based on the parallax correction table. [Effects of the Invention]

[0011] According to the stereo image processing device of the present invention, it is possible to provide a wide-angle, high-precision stereo image processing device. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a hardware block diagram of a stereo image processing device according to a first embodiment. [Figure 2] 1 is a configuration diagram illustrating a processing method of a stereo image processing device according to a first embodiment. [Figure 3] FIG. 3 is a diagram illustrating a parallax error of the stereo image processing device according to the first embodiment. [Figure 4] 3A to 3C are diagrams illustrating a detection method of the stereo image processing device according to the first embodiment. [Figure 5] 4A and 4B are diagrams illustrating a parallax error of the stereo image processing device according to the first embodiment. [Figure 6] 4A to 4C are diagrams illustrating a parallax error using two images obtained by moving the stereo image processing device according to the first embodiment. [Figure 7] 4A and 4B are diagrams illustrating the angle difference and field of view dependency of the stereo image processing device according to the first embodiment. [Figure 8]FIG. 2 is a diagram illustrating an example of processing performed by the stereo image processing device according to the first embodiment. [Figure 9] FIG. 10 is a configuration diagram illustrating a processing method of a stereo image processing device according to a second embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of processing performed by the stereo image processing device according to the second embodiment. [Figure 11] FIG. 10 is a diagram illustrating an example of disparity after correction by the stereo image processing device according to the second embodiment. [Figure 12] FIG. 4 is a configuration diagram illustrating another processing method of the stereo image processing device according to the first embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, the present embodiment will be described with reference to the accompanying drawings. In the drawings and description of the present embodiment, functionally identical elements may be indicated by the same numerals. Note that the following description shows an embodiment based on the principles, but this is for understanding the present embodiment and is not to be used to interpret the present embodiment in a restrictive manner. The description of the present embodiment is merely a typical example and does not limit the scope or application of the claims in any sense.

[0014] Although the present embodiment has been described in sufficient detail for those skilled in the art to practice it, it should be understood that other embodiments are possible, and that changes in configuration and structure and substitutions of various elements are possible without departing from the scope and spirit of the technical concept. Therefore, the following description should not be interpreted as limiting.

[0015] First, we will explain the image misalignment in the wide-angle portion. Figure 3 is a diagram showing the parallax error of a stereo image processing device. Figure 3(a) illustrates the field of view of the horizontal angle of view and the amount of parallax error when a sensor position shift occurs. Figure 3(b) illustrates the field of view of the horizontal angle of view and the amount of parallax error when a lens focal length shift occurs. The lens focal length is 5.0 mm, and the sensor pixel pitch is 3 μm.

[0016] Figure 3(a) shows the amount of parallax error when the amount of positional deviation of the sensor of one of the cameras is 300 nm (equivalent to 0.1 pixel), and Figure 3(b) shows the amount of parallax error when the amount of change in the lens focal length of one of the cameras is 0.01%.

[0017] The sensor position shift of 300 nm and the change in lens focal length of 0.01% are very small shifts, and shifts of this magnitude can easily occur due to temperature changes, aging, etc. The lens used is a fisheye orthogonal projection (f sin θ) lens.

[0018] Parallax detection is performed by detecting sub-pixel parallax using block matching parallax processing. It is preferable to keep the parallax error at this time to 0.5 pixels or less. This parallax is then used to correct the misalignment.

[0019] In the cases shown in Figures 3(a) and 3(b), the amount of parallax error is kept to 0.5 pixels or less near the 0-degree field of view (hereinafter referred to as the center), which is satisfactory. On the other hand, the amount of parallax error in the wide-angle portion of the field of view of 40 degrees or more is large and unsatisfactory. [Example 1] Fig. 1 is a hardware block diagram of a stereo image processing device of this embodiment. As shown in Fig. 1, a stereo camera 50, which is a stereo image processing device, includes a camera 1 and a camera 2. The stereo camera 50 may also include a CPU (Central Processing Unit) 101, a ROM 102, a RAM 103, an HDD (Hard Disk Drive) 104 as a storage device 6, and an HDD controller 105.

[0020] The stereo camera 50 also includes a display 106, which is a display device that displays a screen, and a network I / F (Interface) 107. The stereo camera 50 may also include an external device connection I / F 108, a keyboard 109 as a key information input unit 15, a bus line 110, a DVD (Digital Versatile Disk) drive 111, a media I / F 113, and the like.

[0021] Of these, camera 1 and camera 2 acquire images. CPU 101 controls the overall operation of stereo camera 50. ROM 102 stores programs used to drive CPU 101. RAM 103 is used as a work area for CPU 101. HDD 104 stores various data including programs. HDD controller 105 controls reading and writing of various data from HDD 104 under the control of CPU 101. Display 106 displays various information such as a cursor, menu, window, text, or image. Network I / F 107 is an interface for data communication using a network.

[0022] The external device connection I / F 108 is an interface for connecting various external devices. In this case, the external devices are, for example, a USB (Universal Serial Bus) memory, a printer, etc. The keyboard 109 is a type of input means having multiple keys for inputting characters, numbers, various instructions, etc. The bus line 110 is an address bus, a data bus, etc. for electrically connecting each component such as the CPU 101.

[0023] A DVD drive 111 controls reading of various data from a DVD 112, which is an example of a removable recording medium. A media I / F 113 controls reading or writing (storing) of data from a recording medium 114, such as a flash memory.

[0024] In this embodiment, the hardware configuration shown in FIG. 1 is used, but even with a different configuration, the same effects as in this embodiment can be obtained.

[0025] 2 is a diagram illustrating a processing method of a stereo camera 50, which is a stereo image processing device of this embodiment. The stereo camera 50 is mounted on a moving object and detects surrounding three-dimensional objects based on images obtained by a camera 1 having a lens that serves as a first image acquisition unit and a camera 2 having a lens that serves as a second image acquisition unit aligned horizontally with the camera 1.

[0026] The stereo camera 50 includes an image processing unit 100, an image recording unit 150, a stereo parallax image generation unit 200b, a direction conversion unit and conversion unit 210, a movement amount detection unit 220, a stereo vision comparison unit 250, and a correction table recording unit 260. The stereo camera 50 also includes a stereo parallax image generation unit 200a, a stereo vision correction processing unit 270, a three-dimensional object detection unit 600, an alarm control unit 700, etc. The processing or control of these functions is performed using the CPU 101 in Fig. 1 as a processor, using programs in the ROM 102 and RAM 103, data in the HDD 104, etc.

[0027] The stereo camera 50 acquires an image P1, which is the third image, from camera 1, and an image P11, which is the first image, a predetermined time after the acquisition of image P1, and also acquires an image P21, which is the second image, from camera 2 at the same time as the acquisition of image P11.

[0028] Image P1 is an image projected by camera 1 of a three-dimensional object in front of camera 1 in the direction in which cameras 1 and 2 are aligned. Image P11 is an image projected by camera 1 after camera 1 has moved for a predetermined time in a direction perpendicular to the direction in which cameras 1 and 2 are aligned. Image P21 is an image projected by camera 2 of a three-dimensional object in front of camera 1 in the direction in which cameras 1 and 2 are aligned, at the same time that image P11 is acquired.

[0029] The affine processing means 20a of the image processing unit 100 performs affine processing on the image P1 acquired from the camera 1 to acquire an image P3. The affine processing is a process of performing projective transformation on the image by performing processes such as translation, scaling, and rotation on the image. Furthermore, the affine processing means 20a performs affine processing on the image P11 acquired from the moved camera 1 a predetermined time after the acquisition of the image P1 to acquire an image P31.

[0030] Similarly, the affine processing means 20b performs affine processing on image P21 acquired from camera 2 at the same time as image P11 is acquired, thereby acquiring image P41. Note that the affine processing means 20a and affine processing means 20b may be the same affine processing means, or may perform distortion transformation processing in addition to affine processing.

[0031] Specifically, the affine processing in this embodiment is a process of projectively transforming f sin θ of the fisheye lens projection system into a coordinate system of f tan θx, f tan θy, where f is the focal length, θ is the angle of view incident on the lens, and θx and θy are the horizontal and vertical components of the angle of view incident on the lens.

[0032] The luminance correction means 21a corrects the luminance of each pixel in the images P3 and P31. For example, the luminance correction means 21a performs the correction based on a gain that adjusts the brightness of the camera 1. The luminance correction means 21a also performs the correction based on the gain of each pixel in the images P3 and P31.

[0033] The luminance correction means 21b, which has a similar function, corrects the luminance of each pixel of the image P41. Note that the luminance correction means 21a and the luminance correction means 21b may be the same luminance correction means.

[0034] The pixel interpolation means 22a performs demosaicing processing as a complementation process on the images P3 and P31 corrected by the brightness correction means 21a by referring to the RGB pixel information of the images P3 and P31.

[0035] For example, the pixel interpolation means 22a converts a RAW color image from the image sensor (camera 1) into a color image in a luminance image format with different RGB sensitivities, i.e., converts RGGB sensor information into an RGB color image.

[0036] Similarly, the pixel interpolation means 22b performs demosaicing processing on the image P41 corrected by the luminance correction means 21b. Note that the pixel interpolation means 22a and the pixel interpolation means 22b may be the same pixel interpolation means.

[0037] The luminance information generating means 23a generates images PM3 and PM31 by converting the image P3 and image P31 interpolated by the pixel interpolation means 22a into luminance information. For example, the luminance information generating means 23a converts information representing a color image into luminance information for generating parallax images. In this way, the luminance information generating means 23a converts color image information into black-and-white monochrome image information.

[0038] Therefore, the image processing unit 100 determines whether the image acquired by the camera 1 is a monochrome image, and if it is a monochrome image, the processes of the pixel interpolation unit 22a, the brightness information generation unit 23a, etc. may be omitted.

[0039] Similarly, the luminance information generating means 23b generates an image PM41 by converting the image P41 interpolated by the pixel interpolation means 22b into luminance information. The luminance information generating means 23a and the luminance information generating means 23b may be the same luminance information generating means.

[0040] Furthermore, when stereo camera 50 can obtain images having brightness information from camera 1 and camera 2, the processing in image processing unit 100 may be omitted.

[0041] The image recording unit 150 records the image PM3 obtained by the image processing unit 100 in a memory or the like for a predetermined period of time. The image recording unit 150 may also record and store the image PM31 in a memory or the like.

[0042] The stereo parallax image generating unit 200a, which is a first parallax generating unit, performs stereo matching using the images PM31 and PM41 obtained by the image processing unit 100 to generate a parallax image d1s, which is a first parallax image.

[0043] In addition, the stereo parallax image generation unit 200b, which is the second parallax generation unit, performs stereo matching using two images, image PM3 recorded in the image recording unit 150 and image PM31 recorded at a different time, to generate a parallax image dd, which is the second parallax image.

[0044] If the direction of the parallax image d1s differs from the direction of the parallax image dd, the parallax direction converter and converter 210 converts the direction of the parallax of the parallax image dd to the same direction as the parallax image d1s or to the projection direction of a predetermined camera. Furthermore, the parallax direction converter and converter 210 converts the parallax of the parallax image dd based on the movement amount information from the movement amount detector 220 to generate a parallax image ds, which is a converted second parallax image.

[0045] Next, the stereoscopic comparison unit 250 calculates the difference in the amount of parallax between the parallax image ds and the parallax image d1s and generates a parallax correction table dds that serves as a difference image.

[0046] When generating the parallax correction table dds, the stereoscopic comparison unit 250 may obtain the magnification of the parallax image ds from the parallax image ds and the parallax image d1s.

[0047] This parallax correction table dds is data information that presents the differences (xy coordinates and distance information) between two parallax images in a table, and includes correction values ​​that change in the horizontal direction in which camera 1 and camera 2 are aligned and correction values ​​that change in the vertical direction. The correction table recording unit 260 records the generated parallax correction table dds in a memory or the like.

[0048] Then, the stereoscopic correction processing unit 270 uses the parallax correction table dds stored in the correction table recording unit 260 to correct the parallax error in at least a portion of the parallax image d1s obtained by the stereoscopic parallax image generating unit 200a, thereby obtaining a stereoscopic parallax image sds.

[0049] The three-dimensional object detection unit 600 detects three-dimensional objects according to the stereo parallax image sds corrected by the stereo vision correction processing unit 270.

[0050] The three-dimensional object detection unit 600 identifies the type of three-dimensional object (pedestrian, bicycle, vehicle, building, etc.) detected from the corrected stereo disparity image SDS. The three-dimensional object detection unit 600 identifies moving three-dimensional objects such as pedestrians, bicycles, and vehicles among the three-dimensional objects, and uses this information for preventive safety.

[0051] For example, if the object identified by the three-dimensional object detection unit 600 is a vehicle, the moving stereo image processing device performs preventive safety control such as tracking control for the identified vehicle and braking control in an emergency.

[0052] If the identified object is a pedestrian or a bicycle, the moving stereo image processing device performs emergency braking control. In particular, if the pedestrian or bicycle suddenly appears, the warning control unit 700 performs warning control of a warning device such as a speaker.

[0053] If the identified object is a moving object (e.g., a soccer ball) compared to a stationary object, the moving stereo image processing device performs emergency braking control. In particular, if the object suddenly jumps out, the warning control unit 700 performs warning control.

[0054] In this way, the moving stereo image processing device is capable of highly accurate detection of three-dimensional objects in a wide angle of view, particularly objects in the wide-angle area, and can perform preventive safety control early.

[0055] Furthermore, the three-dimensional object detection unit 600 may measure the distance to the detected object and calculate the moving speed of the object being tracked in time series, thereby implementing more appropriate preventive safety control and warning control.

[0056] As described above, the moving stereo image processing device (stereo camera 50) can correct the parallax error of the wide-angle portion when the camera (sensor) shifts due to temperature change or aging, for example, or when the focal length changes. The stereo camera 50 corrects the parallax of the wide-angle portion of the stereo camera using images obtained by moving the camera 1.

[0057] The correction method of this embodiment will be further described with reference to Figure 4. In Figure 4, camera 1 at time t1 in the moving stereo camera 50 is shown as camera t11, and camera 2 at time t1 is shown as camera t12. Also, camera 1 that has moved to time t2, a time t after time t1, is shown as camera t21, and camera 2 at time t2 is shown as camera t22.

[0058] 4 shows the image taken by camera t11 at time t1 as image g11(P1), and the image taken by camera t12 as image g12. Also, Fig. 4 shows the image taken by camera t21 at time t2 as image g21(P11), and the image taken by camera t22 as image g22(P21).

[0059] In Figure 4(b), stereo camera 50 obtains image g11 (P1) captured by camera t11 at time t1. Then, stereo camera 50 moves a baseline length Bd, which becomes the first interval, after time t has elapsed. Camera t21, which has moved the baseline length Bd from camera t11, captures the measurement object O10 and obtains image g21 (P11). Camera t22, which has moved the baseline length Bd from camera t12, captures the measurement object O10 and obtains image g22 (P21).

[0060] 4A, the camera t21 (camera 1) of the stereo camera 50 is fixed to a joint base 51, and the camera t22 (camera 2) is fixed to the joint base 51. That is, the two cameras are installed at positions separated by a base length Bs, which is the second interval.

[0061] Using such camera positions and captured images, the stereo camera 50 detects parallax and measures distance.

[0062] That is, image g21 (P11) and image g22 (P21) are images projected in the horizontal direction where camera 1 (camera t21) and camera 2 (camera t22) are lined up. Therefore, the distance Ls to the measurement object O10 in the vertical direction can be detected by stereoscopic vision of the images from the two cameras.

[0063] The distance Ls in the direction perpendicular to the direction in which camera 1 (camera t21) and camera 2 (camera t22) of the stereo camera 50 are aligned can be expressed by Equation 1.

[0064]

number

[0065] Note that f represents the lens focal length of camera 1 (camera t21) and camera 2 (camera t22), and ds represents the parallax between image g21 (P11) and image g22 (P21) of camera 1 (camera t21) and camera 2 (camera t22).

[0066] Next, the distance Ld in Figure 4(b) is calculated. Since images g11 (P1) and g21 (P11) are images taken in a direction perpendicular to the direction in which camera 1 (camera t21) and camera 2 (camera t22) are lined up, the distance Ld in the direction horizontal to the direction in which the two shooting positions are lined up can be detected by stereoscopic vision.

[0067] From FIG. 4(b), the distance Ld between the camera 1 (camera t11) and the measurement object O10 in the direction perpendicular to the direction connecting the camera 1 (camera t21) can be expressed by the following equation (2).

[0068]

number

[0069] Note that dd indicates the parallax between the image g11 (P1) of the camera 1 (camera t11) and the image g21 (P11) of the camera 1 (camera t21).

[0070] Here, a distance L1d in the traveling direction from the angle α of the measurement object O10 with respect to the traveling direction of the camera t21 to the measurement object O10 can be expressed as Equation 3 using the distance Ld.

[0071]

number

[0072] Here, under ideal conditions with no deviation, the distance Ls and the distance L1d are the same, and therefore the ideal parallax image ds can be expressed by Equation 4.

[0073]

number

[0074] Here, the base line length Bs is a known length. By detecting the base line length Bd and the parallax image dd, the stereo camera 50 (stereo image processing device) can obtain the parallax image ds.

[0075] Furthermore, if the sensor shifts due to temperature changes or changes over time, the parallax image d1s can be detected by the stereo camera 50.

[0076] Therefore, the difference in parallax of the stereo camera 50 can be expressed by Equation 5.

[0077]

number

[0078] The stereo camera 50 calculates this difference in parallax as a parallax correction table dds, and performs parallax correction on the images of the stereo camera 50 using the parallax correction table dds.

[0079] By performing the above-described correction process, the stereo camera 50 can achieve highly accurate correction. That is, the stereo camera 50 handles the stereoscopic parallax obtained by moving the cameras 1 and 2, thereby achieving highly accurate correction of the images of the stereo camera 50.

[0080] 5A and 5B are diagrams illustrating the parallax error of the stereo image processing device, and explain the processing method of the stereo parallax image generating unit 200a. Fig. 5A shows an image obtained by the camera 1 in which the wide-angle portion has been shifted.

[0081] FIG. 5(a) shows image g21 (P11) from camera 1 (camera t21) and image g22 (P21) from camera 2 (camera t22), which are accompanied by a shift, i.e., the pixel shift amount. In FIG. 5(a), the dashed-dotted line represents image g21, and the dotted line represents image g22. FIG. 5(b) shows the parallax error between images g21 and g22. The parallax error can be expressed as the difference in the shift amount between images g21 and g22. Therefore, when a shift occurs in camera 1 as shown in FIG. 5(a), the shift amount becomes the parallax error shown in FIG. 5(b).

[0082] This parallax error becomes larger as the angle becomes wider, as shown in Fig. 3. This is because the shifts of the images from camera 1 and camera 2 generated by the stereo camera 50 are different, which indicates that the pixel-converted parallax image d1s generated by the stereo parallax image generating unit 200a will have an error.

[0083] FIG. 6 is a diagram for explaining a parallax error using two images obtained by moving the camera 1 of the stereo image processing device, and explains the processing method of the stereo parallax image generating unit 200b.

[0084] Figure 6(a) shows the image g11(P1) of camera 1 (camera t11) due to the displacement and the image g21(P11) of camera 1 (camera t21) after the movement, i.e., the pixel shift amount. In Figure 6(a), the dashed-dotted line indicates image g11(P1), and the dotted line indicates image g21(P11).

[0085] Figure 6(b) shows the parallax error between images g11 and g21. The pixel shift characteristics of images g11 and g21 are slightly shifted in the horizontal direction. The amount of horizontal shift corresponds to the difference between the angle α of camera 1 (camera t21) and the angle α1 of camera 1 (camera t11) shown in Figure 4.

[0086] 6(b) is a small value. Therefore, the pixel-converted parallax image dd generated by the stereo parallax image generating unit 200b has a very small parallax error.

[0087] FIG. 7 is a diagram for explaining the angle difference (angle α−angle α1) and field of view dependency of the stereo image processing device of FIG. 4(b), and further explains the processing method of the stereo parallax image generating unit 200b.

[0088] Figure 7 shows the results obtained when the baseline length Bd is 1 m and the distance Ld is 30, 50, and 70 m. Figure 7 shows that correction using an object with a large distance Ld, for example 70 m, can reduce the deviation in the horizontal angle of view (angle α - angle α1) between the two pixel shifts compared to correction using an object with a small distance Ld, for example 30 m. Furthermore, the deviation in each distance Ld decreases as the angle of view increases, i.e., as the angle becomes wider.

[0089] As shown in Figure 6(a), the amount of pixel shift occurring in the center where the angle of view is small is small, while the amount of pixel shift occurring in the wide-angle portion is large. However, as shown in Figure 7, the difference in the horizontal angle of view between the two pixel shifts in the wide-angle portion (angle α - angle α1) is small. Therefore, the difference in the characteristic amount of pixel shift between image g11 and image g21 is small, and almost no parallax error occurs, which is detected from the difference.

[0090] From the above, it can be inferred that the parallax image dd obtained by the stereo parallax image generating unit 200b due to the movement of the camera 1 has highly accurate values.

[0091] As shown in FIG. 2, the stereo camera 50 generates a parallax image ds in the direction conversion unit and conversion unit 210 using the parallax image dd of an image obtained by moving the camera 1 equipped with a lens.

[0092] Furthermore, the stereo disparity image d1s obtained by the stereo disparity image generation unit 200a using the images from camera 1 and camera 2 is corrected by the stereo disparity correction processing unit 270 using a disparity correction table dds generated from the disparity image ds to obtain a stereo disparity image sds.

[0093] That is, the process of FIG. 2 is controlled using a highly accurate parallax image ds obtained by moving the camera 1.

[0094] Note that the parallax image ds obtained by moving the camera 1 cannot be detected before the movement, but can be detected during the movement. Therefore, the stereo camera 50 corrects the parallax image d1s of the stereo camera 50 after the movement using the parallax image ds obtained by the movement of the camera 1. Furthermore, the stereo camera 50 can also determine the distance to the measurement object after a predetermined time even in a situation where the camera 1 or the stereo camera 50 itself does not move.

[0095] The stereo vision correction processing unit 270 may correct the parallax image d1s continuously or discontinuously over time. For example, the correction of the parallax image d1s may be triggered by the activation of the stereo image processing device (stereo camera 50) of the vehicle or a change in the environment.

[0096] Furthermore, the stereo vision comparison unit 250 may create a parallax correction table dds by applying Equation 4 to the parallax image d1s and the parallax image ds obtained by moving the camera 1 on a pixel-by-pixel basis. Then, the stereo vision correction processing unit 270 may obtain the parallax correction amount (stereo parallax image sds) on a pixel-by-pixel basis.

[0097] On the other hand, the stereo camera 50 may produce pixel-level errors due to the effects of detection variations, sensor noise, etc. For this reason, pixel-level errors may be reduced by performing curve approximation or filter processing, for example.

[0098] Furthermore, instead of using one disparity correction table dds, the stereo vision correction processing unit 270 may store multiple correction tables in the correction table recording unit 260 and use a disparity correction table dds that uses the average of the multiple correction tables or the results of statistical processing.

[0099] Since the center of the parallax correction table dds corresponds to the moving direction of the stereo camera 50, not only is it not possible to obtain parallax accuracy, but it may also be impossible to detect parallax itself.

[0100] Therefore, the stereoscopic comparison unit 250 may obtain the parallax correction table dds in an area excluding the central portion of the correction table, for example, an area excluding the area from the horizontal angle of view of about -10 degrees to 10 degrees. Fig. 8 is a diagram showing an example of processing by the stereoscopic image processing device.

[0101] Fig. 8(a) shows the parallax image d1s of the stereo camera 50 before correction by the stereo vision correction processing unit 270 and the parallax correction table dds, and Fig. 8(b) shows the stereo parallax image sds after processing by the stereo vision correction processing unit 270. The vertical axis of Fig. 8(a) and Fig. 8(b) represents the parallax, and the horizontal axis represents the horizontal angle of view.

[0102] 8 shows the parallax when planes are placed at equal distances, and the parallax of the stereo parallax image sds is a constant value. That is, as shown in FIG. 8(a), the stereo camera 50 corrects at least a part of the parallax image d1s of the stereo camera 50, excluding the central part, i.e., at least a region corresponding to the field of view of the wide-angle part of the lens.

[0103] This allows the stereoscopic correction processing unit 270 to set the stereoscopic parallax image sds after correction processing to a constant value.

[0104] The stereo camera 50 corrects the parallax image d1s of the stereo camera 50 using the parallax image ds obtained by moving the camera 1. At this time, a base line length Bd, which is the first distance, and a base line length Bs, which is the second distance, are required for the correction. A design value is used for the base line length Bs of the stereo camera 50. The base line length Bd is determined using distance information from the movement amount detection unit 220, as shown in FIG. 2.

[0105] The distance information of the movement amount detection unit 220 is, for example, information about the distance to the measurement object O10 detected using a distance sensor of the stereo camera 50. Then, the movement amount detection unit 220 obtains the base length Bd as the distance between the camera 1 (camera t11) and the camera 1 (camera t21) that are the image acquisition positions, using the amount of change in the movement distance.

[0106] In addition, the distance information may be obtained from location information of a global positioning system (GPS) that detects the position of the stereo camera 50, speed information output by a moving body that is equipped with the stereo camera 50, information on the number of tire rotations, acceleration information from an acceleration sensor, etc.

[0107] When the image from camera 1 is shifted as shown in Figure 5, stereo camera 50 corrects it using a parallax image rather than directly correcting the images from each camera. The correction using the parallax image can be performed in the same way even if the image from camera 2 is shifted. Furthermore, the correction using the parallax image can be performed in the same way even if the images from both camera 1 and camera 2 are shifted.

[0108] The stereo parallax image generating unit 200a and the stereo parallax image generating unit 200b detect stationary objects when generating the parallax images. For example, stationary objects are structures such as buildings and utility poles.

[0109] Since the stereo camera 50 performs correction using the movement of the stereo camera 50, it can correct a wide field of view within the image by continuously detecting stationary objects such as the same structure as time-series images. Generally, there are more stationary objects than moving objects in a space. Therefore, the entire field of view, including stationary and moving objects, may be continuously processed, and the parallax image d1s or the parallax image dd may be detected using the statistically peak parallax value.

[0110] Furthermore, in the stereo camera 50, cameras 1 and 2 are installed horizontally, and are not in a so-called offset arrangement. Therefore, there is little offset error due to tilt in the direction in which the two cameras are aligned. To perform correction with high precision, the stereo parallax image generation unit 200a or the stereo vision correction processing unit 270 may correct the offset parallax error due to tilt in the horizontal direction of the cameras.

[0111] For example, correction may be performed by searching for a convergence point at infinity using a stationary object such as a white line on a road. If the baseline length Bd can be accurately determined, the stereo vision correction processing unit 270 may determine that the offset shown in Fig. 8 is the tilt of the direction in which the two cameras are aligned and correct it.

[0112] Furthermore, although the stereo camera 50 has been described as having two cameras with the same lenses, the lenses may have different shapes or performance.

[0113] Furthermore, the movement direction of the stereo camera 50 is not limited to a direction perpendicular to the direction in which the cameras 1 and 2 are aligned. In a direction other than the vertical direction, the stereo camera 50 can obtain a parallax image ds by converting the direction using the direction conversion unit and conversion unit 210.

[0114] For example, the parallax image ds can be obtained even when the movement direction of the stereo camera 50 is parallel to the direction in which the cameras 1 and 2 are aligned. In the case of parallel movement, the process of changing the direction by the direction conversion unit and conversion unit 210 can be omitted. Furthermore, the stereo parallax image generation unit 200b may perform a direction conversion process on the image P31 generated by the affine processing means 20a.

[0115] The direction conversion unit and conversion unit 210 converts the projection direction of the camera to the same direction as the parallax image d1s, but is not limited to this. The direction conversion unit and conversion unit 210 may also perform processing so that the parallax of corresponding objects matches. [Example 2] Fig. 9 is a configuration diagram illustrating a processing method of a stereo image processing device according to Example 2. In Fig. 9, the direction conversion unit and conversion unit 210 of Example 1 in Fig. 2 is changed to a direction conversion unit and Bd estimation unit 215. That is, a disparity image ds is obtained by performing direction conversion and Bd estimation that estimates the base line length Bd.

[0116] 9, the parallax image ds is generated using information from the movement amount detection unit 220 of the first embodiment shown in FIG. 2 and information from the parallax image d1s generated by the stereo parallax image generation unit 200a.

[0117] The direction conversion unit and Bd estimation unit 215 converts the direction of the parallax image dd generated by the stereo parallax image generation unit 200b or the projection direction of the camera into the same direction as the parallax image d1s. Then, Bd is estimated based on the parallax image d1s generated by the stereo parallax image generation unit 200a, and the stereo camera 50 generates a parallax image ds.

[0118] Next, the stereoscopic comparison unit 250 creates a parallax correction table dds, which is a difference image between the parallax image ds and the parallax image d1s, and the correction table recording unit 260 records the parallax correction table dds.

[0119] The stereoscopic correction processing unit 270 corrects the parallax image d1s obtained by the stereoscopic parallax image generating unit 200a using the parallax correction table dds recorded in the correction table recording unit 260.

[0120] 2, the stereoscopic comparison unit 250 generates a parallax correction table dds. The correction table recording unit 260 records the generated parallax correction table dds in a memory or the like. The stereoscopic correction processing unit 270 obtains the stereoscopic parallax image sds and performs processing of the three-dimensional object detection unit 600 and the alarm control unit.

[0121] In this way, in the first embodiment of FIG. 2, the base line length Bd is detected by the movement amount detection unit 220, but in the second embodiment of FIG. 9, an estimated value is used as the base line length Bd.

[0122] 10A and 10B are diagrams showing an example of disparity processing in the stereoscopic comparison unit 250 of the stereoscopic image processing device. Fig. 10A shows a case where the estimated base line length Bd is greater than the optimum value. Fig. 10B shows a case where the estimated base line length Bd is equal to the optimum value. Fig. 10C shows a case where the estimated base line length Bd is smaller than the optimum value.

[0123] 10(a) to 10(c), the solid lines indicate the parallax image d1s, and the dotted lines indicate the parallax image ds. For simplicity, the parallax is shown assuming that two planes are placed at equal distances.

[0124] Fig. 11 shows an example of disparity after correction by the stereoscopic comparison unit 250 of the stereoscopic image processing device. Fig. 11(a) shows a case where the estimated base line length Bd is greater than the optimum value. Fig. 11(b) shows a case where the estimated base line length Bd is equal to the optimum value. Fig. 11(c) shows a case where the estimated base line length Bd is smaller than the optimum value.

[0125] Here, when the baseline length Bd is optimal as shown in Fig. 10(b), a linear, constant parallax is obtained as shown in Fig. 11(b). On the other hand, when the baseline length Bd is different from the optimal value, the parallax image d1s and the parallax image ds do not match as shown in Fig. 10(a) and (c). Therefore, the parallax after correction by the stereo vision correction processor 270 becomes nonlinear and discontinuous in the direction in which the two cameras are arranged, as shown in Fig. 11(a) and (c).

[0126] When the object to be measured moves across such discontinuous points, for example, the distance to the object to be measured (a vehicle or person in front) changes instantaneously, which may cause a malfunction in the control of the stereo image processing device (vehicle).

[0127] Therefore, the direction conversion unit and Bd estimation unit 215 estimates the base length Bd so that the disparity image d1s and the disparity image ds match in the field of view θs as shown in Fig. 10(b). The direction conversion unit and Bd estimation unit 215 performs correction using the base length Bd estimated in the field of view θs, thereby smoothing the disparity after correction for the field of view.

[0128] Here, the field of view θs can be determined arbitrarily. However, if it is too small, the wide-angle portion of 45 degrees or more will not be included, reducing the accuracy of the parallax image ds. If it is too large, the parallax error after correction will be large. For this reason, the field of view θs should be in the range of 10 degrees to 60 degrees, preferably 20 degrees to 50 degrees, as shown in FIG. 3. Within this range, the amount of parallax error is small, allowing the direction conversion unit and Bd estimation unit 215 to obtain a parallax image ds with small parallax error. Note that the direction conversion unit and Bd estimation unit 215 and the stereoscopic comparison unit 250 may perform processing using an arbitrary field of view θs after determining that the parallax after correction by the stereoscopic correction processing unit 270 is nonlinear.

[0129] Furthermore, the correction values ​​of the parallax correction table dds obtained by the stereoscopic comparison unit 250 may be nonlinear with respect to the direction in which the two cameras are aligned.

[0130] The method for estimating the base line length Bd is not limited to the above-described method, and may be, for example, estimated by performing statistical processing. Furthermore, the base line length Bd may be estimated using information on multiple angles of view.

[0131] Furthermore, for example, offset parallax error due to horizontal tilt of the camera may be corrected by searching for a convergence point at infinity using white lines on the road. However, if the correction is not complete, the offset parallax error remains. In this case, if the correct baseline length Bd is used, the parallax after correction for the field of view will be discontinuous. Therefore, the offset parallax error can be corrected for the field of view using the estimated baseline length Bd, resulting in a smoother parallax.

[0132] Note that there is a possibility that a slight parallax error in the wide-angle portion may remain. However, the impact of the parallax error in the wide-angle portion is small even if the parallax after correction for the field of view becomes discontinuous. Therefore, in the case of a parallax error in the wide-angle portion, correction for the field of view can be performed using the estimated base line length Bd.

[0133] The affine processing means 20a also generates an image P31. The affine processing means 20a may perform direction conversion processing using the image P3.

[0134] Then, the direction conversion unit and Bd estimation unit 215 converts the projection direction of the camera to the same direction as the parallax image d1s. The direction conversion unit and Bd estimation unit 215 may perform processing so that the parallax of the corresponding objects matches. <Modification> The present invention is not limited to the above-described embodiment, and includes various modifications other than those described above. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to an embodiment having all of the described configurations.

[0135] It is also possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment, or to add, delete, or replace part of the configuration of each embodiment with the configuration of another embodiment.

[0136] In addition, in the stereo camera 50, the direction in which the two cameras are aligned and the direction of movement are approximately perpendicular, but this is not limiting. The direction in which the two cameras are aligned and the direction of movement of the stereo camera 50 mounted on the moving object may be approximately parallel, or may have a predetermined angle therebetween.

[0137] Fig. 12 is a configuration diagram illustrating another processing method of the stereo image processing device. When the alignment direction of the two cameras and the movement direction of the stereo camera 50 are approximately parallel, the stereo camera 50 may omit the functions of the parallax direction conversion unit and the direction conversion unit of the conversion unit 210 in Fig. 2 and may be a parallax conversion unit 210a that performs parallax conversion, as shown in Fig. 12.

[0138] 9 and the function of the direction conversion unit of Bd estimation unit 215 may be omitted, and the stereo camera 50 may be replaced with a Bd estimation unit 215a. In the case of a predetermined angle, the direction conversion unit that converts the direction as shown in FIGS. 2 and 9 converts the direction so as to obtain the distance in the same direction. That is, the stereo camera 50 as shown in FIGS. 2 and 9 may perform processing by stopping the function of the direction conversion unit depending on the moving direction of the stereo camera 50.

[0139] Although the two cameras of the stereo camera 50 are oriented in the same direction, they may be oriented in different directions. This allows the stereo camera 50 to be configured with a simple structure regardless of the direction in which the cameras are mounted. [Explanation of symbols]

[0140] 1, 2: Camera, 50: Stereo camera, 51: Joint stand 100: Image processing unit, 150: Image recording unit, 200a: stereo disparity image generation unit, 200b: stereo disparity image generation unit, 210: direction conversion unit and conversion unit, 210a: conversion section, 215: Direction conversion unit and Bd estimation unit 220: movement amount detection unit, 250: Stereo vision comparison unit, 260: Correction table recording unit 270: stereoscopic correction processing unit, 600: Three-dimensional object detection unit, 700: Alarm control unit.

Claims

1. A stereo image processing device mounted on a moving body, a first image acquisition unit and a second image acquisition unit that acquire images; a first parallax generation unit that performs stereo matching between the first image acquired from the first image acquisition unit and the second image acquired from the second image acquisition unit to generate a first parallax image; an image recording unit that records at least an image; a second parallax generating unit that performs stereo matching between the first image and a third image recorded in the image recording unit at a different time from the first image, and generates a second parallax image; a parallax conversion unit that converts the parallax amount of the second parallax image to obtain a converted second parallax image; a stereoscopic comparison unit that creates a parallax correction table based on the first parallax image and the converted second parallax image; a stereoscopic correction processing unit that corrects a parallax error of the first parallax image based on the parallax correction table; A stereo image processing device comprising:

2. 2. The stereo image processing device according to claim 1, the parallax conversion unit includes a direction conversion unit that converts the direction of the second parallax image to the same direction as the first parallax image when the direction of the first parallax image and the direction of the second parallax image are different.

3. 2. The stereo image processing device according to claim 1, The stereo image processing device, wherein the parallax correction table includes a correction value that changes in a direction in which the first image acquisition unit and the second image acquisition unit are aligned.

4. 2. The stereo image processing device according to claim 1, The stereo image processing device, wherein the stereo vision correction processing unit corrects the parallax error of the first parallax image by correcting at least a part of the first parallax image.

5. 2. The stereo image processing device according to claim 1, The stereo image processing device, wherein the correction of the parallax error of the first parallax image in the stereo vision correction processing unit corrects an area of ​​the first parallax image corresponding to a field of view of a wide-angle part of a lens.

6. 2. The stereo image processing device according to claim 1, a movement amount detection unit that detects a first distance between an acquisition position of the third image and an acquisition position of the first image, The stereo image processing device, wherein the parallax conversion unit converts the amount of parallax using the first distance and a second distance between the first image acquisition unit and the second image acquisition unit.

7. 2. The stereo image processing device according to claim 1, The stereo image processing device is characterized in that the stereoscopic comparison unit calculates a magnification of the converted second parallax image from the converted second parallax image and the first parallax image, and creates the parallax correction table using the magnification.

8. 2. The stereo image processing device according to claim 1, the third image recorded in the image recording unit is an image acquired from the first image acquiring unit, the first parallax image generated by the first parallax generating unit is a parallax image based on the first image, and the second parallax image generated by the second parallax generating unit is a parallax image based on the third image, The stereo image processing device is characterized in that the stereo parallax comparison unit calculates a difference in the amount of parallax between the converted second parallax image and the first parallax image to create the parallax correction table.

9. 2. The stereo image processing device according to claim 1, a correction table recording unit that records the parallax correction table, a stereo image processing device, characterized in that the stereo vision correction processing unit corrects the parallax error of the first parallax image by using a plurality of the parallax correction tables recorded in the correction table recording unit.

10. 7. The stereo image processing device according to claim 6, The stereo image processing device according to claim 1, wherein the movement amount detection unit obtains the first distance using information stored in the stereo image processing device.

11. 7. The stereo image processing device according to claim 6, The stereo image processing device according to claim 1, wherein the movement amount detection unit obtains the first distance using GPS information.

12. 7. The stereo image processing device according to claim 6, The stereo image processing device according to claim 1, wherein the movement amount detection unit obtains the first distance using information from an acceleration sensor.

13. 7. The stereo image processing device according to claim 6, The stereo image processing device according to claim 1, wherein the movement amount detection unit obtains the first distance using speed information output by the moving object.

14. A processing method for a stereo image processing device mounted on a moving body, comprising: a first image acquisition unit and a second image acquisition unit that acquire images; a first parallax generation unit that performs stereo matching between the first image acquired from the first image acquisition unit and the second image acquired from the second image acquisition unit to generate a first parallax image; an image recording unit that records at least an image; a second parallax generating unit that performs stereo matching between the first image and a third image recorded in the image recording unit at a different time from the first image, and generates a second parallax image; a parallax conversion unit that converts the parallax amount of the second parallax image to obtain a converted second parallax image; a stereoscopic comparison unit that creates a parallax correction table based on the first parallax image and the converted second parallax image; a stereoscopic vision correction processing unit that corrects a parallax error of the first parallax image based on the parallax correction table, a correction value of the parallax correction table being nonlinear with respect to the direction in which the first image acquisition unit and the second image acquisition unit are arranged, and the stereo vision correction processing unit corrects the parallax error of the first parallax image based on the parallax correction table.

15. A processing method for a stereo image processing device according to claim 14, comprising: a parallax conversion unit that converts the orientation of the second parallax image to the same orientation as the first parallax image when the orientation of the first parallax image and the orientation of the second parallax image are different.

Images