Stereo Image Processing Device

Abstract

Provided is a stereo image processing device that has a simple structure. The stereo image processing device for processing a stereo image is characterized by comprising a first reflecting unit and a second reflecting unit that reflect an image of a subject, and a camera that images the subject, wherein: a sensor of the camera has thereon at least three regions of sensor regions made up of a first sensor region, a second sensor region, and a third sensor region; the third sensor region is interposed between the first sensor region and the second sensor region; light that is from the subject and is reflected by the first reflecting unit is incident to the first sensor region; and light that is from the subject and is reflected by the second reflecting unit is incident to the second sensor region.

Description

[Technical Field]

[0001] The present invention relates to a stereo image processing device. [Background technology]

[0002] A stereo image processing device (hereafter referred to as a stereo camera) is known as a device for recognizing objects three-dimensionally. A stereo camera uses the differences in how images are captured by multiple cameras placed at different positions to detect the parallax between multiple images based on trigonometry, and then uses this parallax to detect the depth and position of an object, thereby enabling accurate detection of the position of the target object.

[0003] On the other hand, a stereo camera requires two camera modules equipped with lenses and sensors, which poses a cost issue. To address this issue, Patent Documents 1 and 2 have been proposed.

[0004] In Patent Document 1, in response to the problem of providing an inexpensive stereo camera that is not affected by changes in lens distortion due to temperature changes or changes in the mounting position of the mounting fixture, the stereo camera obtains images with parallax of a subject, and includes a polarization combining means including a cross prism 205 and a polarizing beam splitter that combine the optical paths of two images with parallax that have different polarization directions in orthogonal directions into one, an image sensor 202 that detects the brightness of at least two polarization directions, and a lens 204 that forms the combined image on the image sensor 202.

[0005] Furthermore, in Patent Document 2, in response to the problem of how to obtain multiple images with sufficient parallax using a single imaging means in a three-dimensional imaging device with a relatively simple configuration, the device comprises one imaging element 1, a pair of imaging element-side mirrors 5a, 5b arranged so as to face diagonally outward relative to the two different imaging areas 1a, 1b, a pair of subject-side mirrors 6a, 6b arranged so as to face diagonally forward outward relative to the imaging element-side mirrors 5a, 5b and reflect light from the subject to the corresponding imaging element-side mirrors 5a, 5b, a pair of lenses 3a, 3b that focus the light from the same subject reflected by the mirrors 5a, 5b, 6a, 6b on the different imaging areas 1a, 1b of the imaging element 1, and a pair of apertures 4a, 4b arranged between the imaging element-side mirrors 5a, 5b and the lenses 3a, 3b. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2000-227332 [Patent Document 2] Japanese Patent Application Laid-Open No. 2014-199241 Summary of the Invention [Problem to be solved by the invention]

[0007] Patent Documents 1 and 2 have at least three issues. Regarding the first issue, Patent Documents 1 and 2 streamline the optical system to integrate a single camera module. However, Patent Document 1 uses an expensive prism, and Patent Document 2 uses four mirrors, so the cost issue remains.

[0008] The second problem is the large size of the mirror. Both Patent Document 1 and Patent Document 2 require two reflections, which increases the optical path length within the stereo camera, making these configurations impossible to achieve without using large mirrors.

[0009] The third issue is environmental resistance. When the mirror rotates, the reflected light rotates twice as much as the mirror rotates. For this reason, a large number of reflections is undesirable from the standpoint of environmental resistance.

[0010] The present invention has been made in view of the above-mentioned problems, and provides a stereo image processing device with a simple structure. [Means for solving the problem]

[0011] In view of the above, the present invention is defined as "a stereo image processing device for processing stereo images, comprising a first reflecting unit and a second reflecting unit that reflect an image of an object, and a camera that photographs the object, wherein there are at least three sensor areas on the sensor of the camera, namely a first sensor area, a second sensor area, and a third sensor area, the third sensor area being sandwiched between the first sensor area and the second sensor area, and wherein light from the object that is reflected by the first reflecting unit is incident on the first sensor area, and light from the object that is reflected by the second reflecting unit is incident on the second sensor area." [Effects of the Invention]

[0012] According to the stereo image processing device of the present invention, it is possible to provide a stereo image processing device with a simple structure. Specifically, according to the embodiment of the present invention, it is possible to provide a stereo image processing device that is low cost, small in size, and highly environmentally resistant. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a configuration diagram showing an example of an optical system of a stereo image processing device according to a technique related to the present invention. [Figure 2A] A diagram showing light rays as viewed from the positive position on the vertical axis. [Figure 2B] A diagram showing light rays as viewed from the positive position on the horizontal axis. [Figure 3A] A diagram showing regions divided by each light ray in Figures 2A and 2B. [Figure 3B]A diagram showing regions divided by each light ray in Figures 2A and 2B. [Figure 4] 2 is a diagram showing an example of an image P0 in which an object 5 is detected by a sensor 100 in a camera module 2. FIG. [Figure 5] 1 is a diagram showing a processing method of a stereo image processing device according to a technique related to the present invention. [Figure 6A] FIG. 10 is a diagram showing a common area (plane coincidence) in stereoscopic vision between images P3 and P4. [Figure 6B] FIG. 10 is a diagram showing a common area (plane mismatch) in stereo vision between images P3 and P4. [Figure 7] 1A and 1B are diagrams showing examples of the shape of a plane mirror. [Figure 8A] FIG. 10 is a diagram showing an example of an image when the sensor is arranged vertically. [Figure 8B] FIG. 10 shows an example of an image captured when the stereo camera is rotated 90 degrees around the depth axis. [Figure 8C] FIG. 10 shows an example of an image captured when the stereo camera is rotated 90 degrees around the depth axis. [Figure 9] FIG. 10 is a diagram showing an example of detecting an image outside a housing through a glass plate or the like. [Figure 10A] 3 is a three-dimensional diagram showing an example of the arrangement of structural components of the stereo camera 10. [Figure 10B] FIG. 2 is a plan view showing an example of the arrangement of structural components of the stereo camera 10. [Figure 11] FIG. 10 is a configuration diagram showing another example of an optical system of a stereo image processing device according to a technique related to the present invention. [Figure 12] 1A and 1B are diagrams showing detected images of a stereo image processing device according to a technique related to the present invention. [Figure 13A] FIG. 10 is a diagram showing a case where the chief ray is incident from the negative side in the depth direction. [Figure 13B] FIG. 10 is a diagram showing a case where the chief ray is incident from the negative side in the depth direction. [Figure 14A] FIG. 10 is a diagram showing an example of an image when the sensor is arranged vertically. [Figure 14B] FIG. 10 shows an example of an image captured when the stereo camera is rotated 90 degrees around the depth axis. [Figure 14C]FIG. 10 shows an example of an image captured when the stereo camera is rotated 90 degrees around the depth axis. [Figure 15] FIG. 1 is a configuration diagram illustrating an example of an optical system of a stereo image processing device according to a first embodiment. [Figure 16] 3A to 3C are diagrams showing detected images of the stereo image processing device according to the first embodiment. [Figure 17A] FIG. 4 is a configuration diagram showing another example of an optical system of the stereo image processing device according to the first embodiment. [Figure 17B] FIG. 4 is a configuration diagram showing another example of an optical system of the stereo image processing device according to the first embodiment. [Figure 18A] FIG. 10 is a diagram showing an example of an image when the sensor is arranged vertically. [Figure 18B] FIG. 10 shows an example of an image captured when the stereo camera is rotated 90 degrees around the depth axis. [Figure 18C] FIG. 10 shows an example of an image captured when the stereo camera is rotated 90 degrees around the depth axis. [Figure 19] FIG. 10 is a configuration diagram showing an example of an optical system of a stereo image processing device according to a fourth embodiment. [Figure 20] 10A and 10B are diagrams showing detected images of a stereo image processing device according to a fourth embodiment. [Figure 21A] FIG. 10 is a diagram showing an example of an image when the sensor is arranged vertically. [Figure 21B] FIG. 10 shows an example of an image captured when the stereo camera is rotated 90 degrees around the depth axis. [Figure 21C] FIG. 10 shows an example of an image captured when the stereo camera is rotated 90 degrees around the depth axis. [Figure 22] FIG. 10 is a configuration diagram showing another example of an optical system of a stereo image processing device according to a technique related to the present invention. [Figure 23] 1A and 1B are diagrams showing detected images of a stereo image processing device according to a technique related to the present invention. [Figure 24A] FIG. 10 is a diagram showing an example of an image when the sensor is arranged vertically. [Figure 24B] FIG. 10 shows an example of an image captured when the stereo camera is rotated 90 degrees around the depth axis. [Figure 24C] FIG. 10 shows an example of an image captured when the stereo camera is rotated 90 degrees around the depth axis. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings.

[0015] In this specification, before describing the embodiments, related technical matters that are the premise of the present invention will be described using Figures 1 to 13 and Figures 22 to 25, and the embodiments of the present invention will be described using Figures 14 to 21. Note that the related arts described below include technical matters that are not publicly known.

[0016] 1 is a diagram showing an example of the configuration of the optical system of a stereo camera 10 according to technology related to the present invention. The stereo camera 10 can be configured to include a camera module 2, which is made up of a lens 99 and a sensor 100, and two plane mirrors 40R and 40L, and further includes a calculation unit 3 that processes images acquired by the camera module 2. Here, the camera module 2 is positioned so that the chief ray of light intersects with the boundary between the plane mirrors 40R and 40L at a distance d.

[0017] In Patent Documents 1 and 2, the mirror reflects twice, so the imaging direction of the stereo camera and the facing direction of the camera are the same. In contrast, stereo camera 10 according to the technology related to the present invention is characterized in that the mirror reflects once, so the imaging direction of stereo camera 10 and the facing direction of camera module 2 are approximately reversed. Furthermore, because the mirror reflects once, there is an advantage that the environmental resistance is higher than in Patent Documents 1 and 2.

[0018] Here, we define the vertical, horizontal, and depth axes as shown in Figure 1. The stereo camera of the related art measures the distance in the depth direction.

[0019] The stereo camera 10 of the related art detects images from two viewpoints using one camera module, and therefore defines two optical axes OA directed toward two virtual viewpoints (not shown). Here, these are defined as an optical axis OA1 that reflects off the plane mirror 40R and an optical axis OA2 that reflects off the plane mirror 40L. The optical axes OA1 and OA2 outside the stereo camera 10 are substantially parallel. Here, the optical axis outside the stereo camera 10 refers to the optical axis before reflection by the plane mirror 40R and the plane mirror 40L. The optical axis inside the stereo camera 10 refers to the optical axis between the plane mirror 40R, the plane mirror 40L, and the camera module 2.

[0020] According to Fig. 1 showing an example of the basic configuration of the related technology, the related technology is configured as "a stereo image processing device that processes stereo images, and includes at least one or more reflecting units 40R, 40L that reflect an image of an object, and one camera 99 that photographs the object, and the camera 99 detects the image reflected only once by the reflecting units 40R, 40L." Note that in Fig. 1, the plane mirror 40R and the plane mirror 40L can be configured as one unit, but there are two reflecting units.

[0021] The relationship between the optical axis and the angle of view will be described with reference to Figures 2A and 2B. Figure 2A shows light rays when viewed in a cross section formed by optical axes OA1 and OA2 outside the stereo camera 10 (as viewed from the positive position on the vertical axis), and Figure 2B shows light rays when viewed in a cross section perpendicular to the cross section formed by optical axes OA1 and OA2 on optical axis OA2 and outside the stereo camera 10 (as viewed from the positive position on the horizontal axis). Here, dotted lines indicate light rays directed toward the camera module 2.

[0022] 2A and 2B show an example of the definition of the vertical, horizontal, and depth axes, in which, for example, when stereo camera 10 is installed on road 9 (or on a vehicle not shown) and the distance to vehicle 5 traveling on road 9 is measured, the distance to the vehicle ahead is taken as the depth direction and the arrangement direction of the two plane mirrors is taken as the horizontal direction. In the following description in this specification, this arrangement example is taken as a reference arrangement example, and if an arrangement example differs, this will be indicated as appropriate in the drawings.

[0023] As shown in FIGS. 2A and 2B, light rays (dotted lines in the figures) in the vertical and horizontal angles of view are reflected by the plane mirror 40L and the plane mirror 40R, pass through the lens 99 in the camera module 2, and form an image on the sensor 100.

[0024] Here, the dashed-dotted lines indicate extensions of the light rays (dotted lines in the figure) that are incident on the camera module 2 and head toward the flat mirror 40R and the flat mirror 40L. The approximate intersection points of the dashed-dotted lines are defined as the virtual viewpoint VP1 for the flat mirror 40R and the virtual viewpoint VP2 for the flat mirror 40L.

[0025] These virtual viewpoints VP1 and VP2 indicate the effective positions of the camera module 2. This is the same as detecting light rays from virtual viewpoints VP1 and VP2, respectively, because the light rays are reflected by the plane mirror 40R and the plane mirror 40L. Strictly speaking, the virtual viewpoints VP1 and VP2 correspond to the pupil positions of the lens 99 of the camera module 2. In an actual lens, the pupil position differs depending on the angle of view, so the virtual viewpoints VP1 and VP2 do not overlap at a single point, but because the amount of deviation is small, on the order of millimeters, they can be considered to be a single point when measuring distances of several meters or more, for example.

[0026] As shown in Figure 2B, the stereo camera 10 of the related art is characterized in that the angle between the plane formed by the optical axes OA1 and OA2 inside the stereo camera 10 and the plane formed by the optical axes OA1 and OA2 outside the stereo camera 10 is a predetermined angle α. If these two planes were to coincide, the image of the camera module itself would be formed on the sensor, reducing the number of usable pixels. Therefore, by setting the angle between these two planes to the predetermined angle α, it is possible to efficiently detect an image in the depth direction with just one reflection on the plane mirror.

[0027] In this configuration, as shown in FIG. 2B, the pupil position of the camera module 2 is located below the plane formed by the normal 40MA of the plane mirror 40L and the normal 40MA of the plane mirror 40R (not shown).

[0028] 3A and 3B are diagrams dividing the regions according to the light rays in Figures 2A and 2B. Here, the region consisting of the light rays incident on the flat mirror 40R (first reflecting portion) and the flat mirror 40L (second reflecting portion) is defined as a first region R1, the region consisting of the line (dashed line) when the light rays incident on the flat mirror 40R (first reflecting portion) are extended to a virtual viewpoint VP1 (first virtual viewpoint) is defined as a second region R2, the region consisting of the line (dashed line) when the light rays incident on the flat mirror 40L (second reflecting portion) are extended to a virtual viewpoint VP2 (second virtual viewpoint) is defined as a third region R3, and the smallest convex region including the second region R2 and the third region R3 is defined as a fourth region R4. In the related art, the lens pupil position of the camera module 2 is located outside the region R consisting of the first region R1 and the fourth region R4. In the related art, the camera module 2 is arranged outside the area formed by the first area R1 and the fourth area R4 in the vertical direction.

[0029] 4 shows an image P0 of an object 5 detected by the sensor 100 in the camera module 2. It is assumed that the object 5 is located on a road 9 parallel to the cross section of the horizontal and depth axes in FIG. 2 (also parallel to the plane formed by the optical axes OA1 and OA2 outside the stereo camera 10).

[0030] Image P0 is divided into two horizontal image areas IM1 (first sensor area) and IM2 (second sensor area) by a dividing line SL. A feature of the related technology is that the road surface 9 is detected at an angle on the sensor. This is because the angle between the plane formed by optical axes OA1 and OA2 inside the stereo camera 10 shown in FIG. 2B and the plane formed by optical axes OA1 and OA2 outside the stereo camera 10 is a predetermined angle α. A feature of the stereo camera 10 of the related technology is that image areas IM1 and IM2 within the image are tilted in opposite directions. The tilt direction differs depending on the placement position of the camera module 2. For example, if the camera module 2 is placed below the area R consisting of the first area R1 and the fourth area R4, the result will be as shown in FIG. 4. If the camera module 2 is placed above the area R, the center of the road surface 9 will have a convex shape.

[0031] 1 and 2. If these parameters are the stereoscopic viewing angle of the stereo camera 10, the angle formed by the plane mirrors 40L and 40R, and d, respectively, and the distance between the camera module 2 and the mirror, then the required viewing angle of the camera module 2 of the stereo camera 10 is (θ-β+180) degrees or more, the base length B is 2×d×sin(180-γ), and the required mirror size (horizontal direction) is d×{sin(90-γ / 2)+cos(90-γ / 2)×tan(θ-γ / 2+90)} or more.

[0032] Here, the required angle of view and required mirror size are equal when the mirror surfaces of plane mirror 40R and plane mirror 40L are connected and the specified angle α approaches 0. As this condition deviates, the required angle of view and required mirror size must increase.

[0033] A processing device and processing method for the stereo camera 10 will be described with reference to Fig. 5. The stereo camera 10 is mounted on a vehicle such as an automobile or a motorcycle, and is used to detect the distance from the vehicle to surrounding three-dimensional objects (other automobiles, buildings, pedestrians, etc.). The following description will be given using an example in which the stereo camera 10 is mounted on a vehicle, but the present invention is not limited to this.

[0034] The stereo camera 10 is configured to be able to detect surrounding three-dimensional objects based on images obtained by the camera module 2 and issue an alarm as necessary.

[0035] The calculation unit 3 in the stereo camera 10 includes an image processing unit 200, a stereo parallax image generation unit 300, a stereoscopic three-dimensional object detection unit 500, and an alarm control unit 700, for example.

[0036] In the stereo camera 10, an image P0 shown in Fig. 4 is detected by a sensor 100 in the camera module 2. The image P0 is provided to an image processing unit 200, which is the first processing function of the calculation unit 3, and the image P0 is then separated by an image separation means 60 into two images P1 (first detected image) and P2 (second detected image) according to the virtual viewpoints VP1 and VP2.

[0037] The image processing unit 200 further includes, for example, affine processing means 20a and 20b, luminance correction means 21a and 21b, pixel interpolation means 22a and 22b, and luminance information generation means 23a and 23b. The image processing unit 200 applies these predetermined image processes to each of the images P1 and P2, and supplies the images to the stereo parallax image generation unit 300.

[0038] First, the affine processing means 20a applies affine processing to image P1. The affine processing is, for example, a linear coordinate transformation process, but may also include non-linear calculations. As a result of performing this affine processing, the affine processing means 20a obtains image P3 (third image). Similarly, the affine processing means 20b applies affine processing to image P2 to obtain image P4 (fourth image).

[0039] The affine processing means 20a and affine processing means 20b correct the tilt of the images shown in Figure 3. Here, tilt processing in opposite directions is performed on image P1 (first detected image) and image P2 (second detected image). As a result, the horizontal axes of images P3 and P4 become parallel, allowing matching processing to be performed on each line, which makes it easy to perform parallax processing.

[0040] The luminance correction means 21a corrects the luminance of each pixel of the image P3. For example, the luminance of each pixel of the image P3 is corrected based on the difference in gain of each pixel in the image P3. Similarly, the luminance correction means 21b corrects the luminance of each pixel of the image P4.

[0041] The pixel interpolation means 22a performs demosaicing on the image P3 (e.g., converting the image from a RAW image to a color image). Similarly, the pixel interpolation means 22b performs demosaicing on the image P4.

[0042] The luminance information generating means 23a generates luminance information for the image P3. For example, the luminance information generating means 23a converts information representing a color image into luminance information for generating a parallax image. Similarly, the luminance information generating means 23b generates luminance information for the image P4.

[0043] Next, the stereo parallax image generating unit 300 generates a stereo parallax image using the images of the stereo common area of ​​the obtained images P3 and P4. Here, the stereo common area will be explained. Figures 6A and 6B show the stereo common area of ​​the images P3 and P4 in the cases of plane coincidence and plane mismatch.

[0044] 6A shows a case where the plane formed by the optical axes OA1 and OA2 inside the stereo camera 10 coincides with the plane formed by the optical axes OA1 and OA2 outside the stereo camera 10. As mentioned above, in this case, the number of available pixels decreases because the image of the camera module itself is formed on the sensor, but for simplicity's sake, we will not discuss this reduction in pixel count here.

[0045] Figure 6B shows the case where the angle between the plane formed by the optical axes OA1 and OA2 inside the stereo camera 10 and the plane formed by the optical axes OA1 and OA2 outside the stereo camera 10 is a predetermined angle α (plane misalignment).

[0046] In Figure 6B, image P3, indicated by the dotted line, and image P4, indicated by the dashed line, are tilted in opposite directions compared to Figure 6A. Therefore, it can be seen that the stereo common area CA of images P3 and P4, indicated by the diagonal line, changes depending on the angle α. From the perspective of the stereo common area, it is desirable that the angle α be small. However, the angle α may be set according to requirements such as detecting the bottom side using one of the images detected in image P3 or image P4.

[0047] 5 generates a stereo parallax image using an image of the stereo common view area CA from among images P3 and P4. The stereo parallax image generation unit 300 includes an exposure adjustment unit 210 and a sensitivity adjustment unit 220, and can perform feedback control of the exposure amount, sensitivity, etc. of the camera module 2 to the camera module 2.

[0048] The stereo parallax image generation unit 200 further includes a geometric correction unit 230 that performs geometric correction on the two images, a matching unit 240 that performs matching processing on the left and right images, and a pixel shift amount calculation unit 260 that calculates the pixel shift amount. The pixel shift amount calculation unit 260, together with the affine processing means 20a and 20b, constitutes a pixel shift correction processing unit.

[0049] The stereoscopic three-dimensional object detection unit 500 detects three-dimensional objects in the stereoscopic region according to the stereo disparity images generated by the stereo disparity image generation unit 200. It also applies stereo matching to the detected three-dimensional objects to detect disparity and identify the type of the three-dimensional object (pedestrian, bicycle, vehicle, building, etc.).

[0050] By detecting three-dimensional objects and identifying the type of three-dimensional object, the type to be used for preventive safety can be further specified. When a vehicle is detected, the detection result can be used for following control of the preceding vehicle or emergency braking control. If the detected three-dimensional object is a pedestrian or bicycle, emergency braking control or warning control can be executed. By measuring the distance to these detected objects and estimating the moving speed of the object being tracked over time, the warning control unit 700 can implement more appropriate warnings and control.

[0051] The optical system and processing method described above make it possible to realize a stereo camera using only one camera module and two plane mirrors, thereby resolving the conventional issues of low cost, small size, and environmental resistance.

[0052] In principle, the detection method used here does not depend on the lens projection method. For example, it can be central projection (f tan θ), which is the projection used by normal lenses, or orthogonal projection (f sin θ), which is the projection used by fisheye lenses, equidistant projection (f θ), equistereographic projection (2f sin(θ / 2)), or stereographic projection (2f tan(θ / 2)). However, of these projections, equidistant projection (f θ) is the most efficient way to achieve stereoscopic vision. The reason for this is explained below.

[0053] In other projections, the two optical axes OA1 and OA2 on the sensor are equidistant from the lens optical axis (the axis extending from the center of the lens), and therefore have the same resolution. However, the images in the right (or left) regions of the two images P3 and P4 are not the same distance from the lens optical axis, and therefore have different lens resolutions. Different resolutions can easily cause parallax errors. For example, parallax errors can be reduced by processing using a digital filter that changes the resolution according to the angle of view, but the accuracy of the parallax is determined by the lower resolution of the two images P3 and P4, which results in a decrease in the accuracy of distance measurement.

[0054] In contrast, with equidistant projection (fθ), the resolution is constant regardless of the position from the lens optical axis. Therefore, the resolution of the two images P3 and P4 on the right (or left) side is the same. This allows for the most efficient stereoscopic vision. Another advantage is that if there is a tilt deviation in the mirror, the angle of rotation deviation can be corrected as a fixed value for all images in the detected image.

[0055] Furthermore, although the mirror has been described as having a rectangular shape, the shape of the mirror is not limited thereto. For example, the flat mirror may have a shape such as that shown in FIG. 7 in order to achieve compactness. The reason for such a shape is that the required mirror size differs depending on the angle of view, which is the distance between the pupil position of the camera module 2 and the flat mirror 40L (or flat mirror 40R). For example, when the distance between the lens pupil position of the lens 99 in the camera module 2 and the flat mirror 40L (or flat mirror 40R) is close, the mirror portion is small, and when the distance is far, the mirror portion is large.

[0056] Furthermore, although the horizontally elongated sensor is arranged horizontally, the sensor 100 may be arranged vertically as shown in FIG. 8A. In this case, there is an advantage that the angle of view in the vertical direction (vertical direction in the figure) can be widened. The stereo camera may also be rotated, for example, by 90 degrees about the depth axis. In this case, images like those shown in FIGS. 8B and 8C are obtained. In the cases of FIGS. 4 and 8A, parallax in the horizontal direction is detected, but in the cases of FIGS. 8B and 8C, parallax in the vertical direction is detected. FIG. 8B has the advantage that the angle of view in the vertical direction can be widened. Furthermore, FIG. 8C has the advantage that the angle of view in the horizontal direction can be widened.

[0057] On the other hand, when the sensor detection is a rolling shutter type, there is a good detection method from the viewpoint of high-speed detection. For example, when the sensor detection is performed by scanning in the longitudinal direction of the sensor and then repeating the detection in the lateral direction of the sensor, Figs. 4 and 8B are advantageous from the viewpoint of high-speed detection. In Figs. 4 and 8B, parallax is detected using the detection signal in the longitudinal direction. Therefore, the detection time difference between the two images for parallax detection is small. On the other hand, in Figs. 8A and 8B, parallax detection is performed using the detection signal in the lateral direction, so the detection time difference is larger than in Figs. 4 and 8B. If the detection time difference is small, high distance measurement accuracy can be achieved when a car or motorcycle equipped with stereo camera 10 is traveling at high speed.

[0058] Here, the two viewpoints on the sensor 100 are equally divided from the viewpoint of increasing the angle of view of the stereoscopic view, but this is not limiting. The angle of view of the virtual viewpoint VP1 or the virtual viewpoint VP2 can be increased by shifting the plane mirror 40R and the plane mirror 40L in FIG. 2A in the horizontal direction relative to the camera module 2. For example, when using an in-vehicle stereo camera to grasp the driving situation of a vehicle traveling in the left lane, the angle that needs to be detected at an intersection differs depending on whether the vehicle is turning left or right. To accommodate this, the plane mirror 40R and the plane mirror 40L may be shifted in the horizontal direction relative to the camera module.

[0059] Furthermore, the two mirrors in this example are flat. However, to widen the field of view, the mirrors may be curved. This configuration makes it possible to realize a wide-angle stereo camera without using wide-angle lenses.

[0060] Furthermore, by using a curved surface, it is possible to reduce the angle between the plane formed by the optical axes OA1 and OA2 inside the stereo camera 10 and the plane formed by the optical axes OA1 and OA2 outside the stereo camera 10. This makes it possible to reduce the tilt of the two images and increase the common stereoscopic viewing area CA. Note that the same effect can be obtained whether the curved surface is in a one-dimensional direction or a two-dimensional direction.

[0061] On the other hand, flat mirrors have higher distance measurement accuracy than curved mirrors. The reason for this is the effect of aberration. With a curved mirror, the light beam that makes up one pixel on the sensor is tilted at a different angle, causing aberration. This reduces resolution. Another reason is that curved mirrors are less robust against rotational and positional misalignment of the mirror. While flat mirrors are almost independent of position, curved mirrors are highly dependent on position. For this reason, from a practical standpoint, a highly robust flat mirror is preferable.

[0062] Although the configuration has been described using the camera module 2 and the two plane mirrors 40R and 40L, it is also possible to place part or all of the optical system inside the housing and detect an image outside the housing via a glass plate, etc. In this case, the glass plate may be tilted with respect to the optical axis in consideration of the effect of stray light due to reflection from the glass plate.

[0063] In particular, reflected stray light can be efficiently avoided by tilting the glass plate 65 by 90 degrees with respect to the plane formed by the optical axes OA1 and OA2 outside the stereo camera 10, as shown in Fig. 9. In contrast, if the glass plate is tilted about a vertical axis, the horizontal angle of view is large, so the glass must be tilted significantly to avoid the stray light, which poses issues such as the influence of refraction by the glass and the size of the housing.

[0064] Therefore, for example, when the sensor 100 is arranged vertically as shown in FIG. 8A, reflected stray light can be efficiently avoided by tilting the glass plate 65 horizontally with respect to the optical axes OA1 and OA2 outside the stereo camera 10. (When the sensor is longer horizontally than vertically, the transparent medium is tilted about the horizontal axis, and when the sensor is shorter horizontally than vertically, the glass plate 65 (transparent medium) is tilted about the vertical axis.) Although the plane mirror 40R and the plane mirror 40L are arranged as two mirrors, they may be formed as an integrated unit.

[0065] 10A and 10B show the layout of the structural components of the stereo camera 10. Here, the camera module 2, the plane mirror 40R, and the plane mirror 40L are held by a holder 198. These components are protected by a housing 298 and a glass plate 65.

[0066] By attaching the camera module 2, the plane mirror 40R, and the plane mirror 40L to the holder 198, the stereo camera 10 can stably fix these components and reduce the effects of temperature changes and changes over time. Note that in Figures 10A and 10B, the cameras are placed below the plane formed by the optical axes OA1 and OA2 outside the stereo camera 10, but the same effect can be obtained by placing them above the plane.

[0067] FIG. 11 is a diagram showing another example of the configuration of the optical system of stereo camera 10. Similar to FIG. 1, this stereo camera 10 is characterized by a single mirror reflection. Here, the two optical axes that realize the stereo camera are defined as follows: optical axis OA3 that reflects from plane mirror 30R and optical axis OA4 that reflects from plane mirror 30L. The plane formed by optical axes OA3 and OA4 inside stereo camera 10 and optical axes OA3 and OA4 outside stereo camera 10 approximately coincides. While the camera module was previously placed outside the area formed by the first and fourth areas in the vertical direction, the stereo camera 10 of FIG. 11 differs in that the camera module is placed outside the area in the horizontal direction.

[0068] The relationship between the optical axis and the angle of view will be described with reference to Fig. 11. Fig. 11 shows the configuration when viewed from a cross section formed by optical axes OA3 and OA4 outside the stereo camera 10 (as viewed from the positive position of the vertical axis). Here, dotted lines indicate light rays directed toward the camera module 2.

[0069] 11, light rays (dotted lines in the figure) of the vertical and horizontal angles of view are reflected by the plane mirror 30R and the plane mirror 30L, pass through the lens 99 in the camera module 2, and form an image on the sensor 100. Here, the light rays that are incident on the camera module 2 and are directed toward the plane mirror 30R and the plane mirror 30L (dotted lines) are extended as dashed-dotted lines.

[0070] The points where the dashed lines intersect are defined as virtual viewpoints VP3 and VP4. These virtual viewpoints VP3 and VP4 indicate the effective positions of the camera module 2. This is equivalent to detecting light rays from virtual viewpoints VP3 and VP4, respectively, because the light rays are reflected by the flat mirrors 30R and 30L. Therefore, the distance and position to the target object are detected using images detected from the two horizontal virtual viewpoints VP3 and VP4. In the case of Figure 11, the reference axes SA3 and SA4, which are perpendicular to the direction of the base length B, are different from the optical axes OA3 and OA4. Even in this case, distance measurement is possible, even if the initial angle of view is shifted.

[0071] FIG. 12 shows image P0 detected by sensor 100 in camera module 2. Image P0 is divided into two parts horizontally by a dividing line SL. In FIG. 4, the image was detected tilted, but in the configuration of FIG. 11, the plane formed by optical axes OA3 and OA4 inside stereo camera 10 and optical axes OA3 and OA4 outside stereo camera 10 approximately coincides, so the image is not tilted. For this reason, FIG. 11 can enlarge the stereoscopic common area CA compared to FIG. 1.

[0072] In Figure 1, the image is tilted, so the vertical position required to detect parallax differs within image P0. In this case, if a rolling shutter sensor is used, the detected object position will differ between images IM1 and IM2, and the detection time will differ, resulting in parallax errors when driving at high speeds. For example, Figure 4 shows that the vertical positions of the right side of the vehicle in images IM1 and IM2 are different. On the other hand, with the configuration in Figure 11, the image is not tilted, so even when a rolling shutter sensor is used, the detection time difference between virtual viewpoints VP3 and VP4 is small, which has the advantage of making it less likely for parallax errors to occur when driving at high speeds.

[0073] In Fig. 11, a stereo camera can be realized with only one camera module and two plane mirrors by performing the same processing on image P0 as in Fig. 5. This solves the conventional problems of low cost, small size, and environmental resistance.

[0074] The positions and angles of the plane mirrors 30R and 30L of the stereo camera 10 shown in Fig. 11 are not limited. For example, the optical axes OA3 and OA4 may be perpendicular to the direction of the base line length B, as shown in Figs. 13A and 13B. Fig. 13A shows the case where the chief ray of the camera module 2 is incident from the negative side in the depth direction, and Fig. 13B shows the case where the chief ray of the camera module 2 is incident from the positive side in the depth direction.

[0075] 13A, the angle of the optical axis OA3 reflected by the plane mirror 30R with respect to the horizontal axis is defined as angle θ3, and the distance of the optical axis OA3 from the plane mirror 30R to the pupil of the camera module 2 is defined as d3. The angle of the optical axis OA4 reflected by the plane mirror 40R with respect to the horizontal axis is defined as angle θ4, and the distance of the optical axis OA4 from the plane mirror 40R to the pupil of the camera module 2 is defined as d4. In order to make the directions of the optical axes OA3 and OA4 perpendicular to the direction of the base length B, it is sufficient to set d1(1+sinθ1)=d2(1+sinθ2).

[0076] Although the plane mirrors 30R and 30L are arranged close to each other, the base length can be changed by arranging the plane mirrors 30R and 30L apart, which allows for distant detection.

[0077] Although the horizontally long sensor is arranged horizontally, the sensor 100 may be arranged vertically as shown in FIG. 14A. In this case, there is an advantage that the angle of view in the vertical direction (vertical direction in the figure) can be widened. The stereo camera 10 may be rotated, for example, by 90 degrees about the depth axis. In this case, images like those shown in FIGS. 14B and 14C are obtained. In the cases of FIGS. 12 and 14A, parallax in the horizontal direction is detected, but in the cases of FIGS. 14B and 14C, parallax in the vertical direction is detected. FIG. 14B has the advantage that the angle of view in the vertical direction can be widened. In addition, FIG. 14C has the advantage that the angle of view in the horizontal direction can be widened.

[0078] Furthermore, in the above description, the two mirrors are flat. However, the mirrors may be curved to widen the field of view. With this configuration, a wide-angle stereo camera can be realized without using a wide-angle lens. Furthermore, by using curved surfaces, the angle between the plane formed by the optical axes OA3 and OA4 inside the stereo camera 10 and the plane formed by the optical axes OA3 and OA4 outside the stereo camera 10 can be reduced. This reduces the tilt of the two images and increases the common stereoscopic area CA.

[0079] The same effect can be achieved whether the direction of the curvature is one-dimensional or two-dimensional. However, flat mirrors have higher distance measurement accuracy than curved mirrors. The reason for this is the effect of aberration. With a curved mirror, the light beam that makes up one pixel on the sensor is tilted at a different angle, causing aberration. This reduces resolution. Another reason is that curved mirrors are less robust against rotational and positional misalignment of the mirror. While flat mirrors are almost independent of position, curved mirrors are highly dependent on position. For this reason, from a practical perspective, a highly robust flat mirror is preferable.

[0080] Furthermore, the sizes of the plane mirror 30R and the plane mirror 30L are not limited. For example, the plane mirror 30L closer to the camera module 2 can be made smaller than the plane mirror 30R farther away. The sizes of the plane mirrors may also be asymmetric. This allows the stereo camera device to be made smaller.

[0081] 22 is a diagram showing another example of the configuration of the optical system of the stereo camera 10. This stereo camera 10 does not use a plane mirror on one side. Here, the two optical axes that realize the stereo camera are defined as follows: the optical axis that enters the camera module is defined as OA9, and the optical axis that reflects off the plane mirror 80 is defined as OA10.

[0082] The relationship between the optical axis and the angle of view will be described with reference to Fig. 22. Fig. 22 shows the configuration when viewed from a cross section formed by optical axes OA9 and OA10 outside the stereo camera 10 (as viewed from the positive position of the vertical axis). Here, dotted lines indicate light rays directed toward the camera module 2.

[0083] 22, some of the light rays (dotted lines in the figure) in the vertical and horizontal angles of view are reflected by the plane mirror 80, pass through the lens 99 in the camera module 2, and form an image on the sensor 100, while others pass directly through the lens 99 and form an image on the sensor 100. Here, the light rays that are incident on the camera module 2 and are directed toward the plane mirror 80 are shown by dashed dotted lines.

[0084] The approximate intersection point of the dashed dotted lines is defined as the virtual viewpoint VP10. These virtual viewpoints VP9 and VP10 indicate the effective position of the camera module 2. This is the same as detection from the virtual viewpoint VP10, because the light rays are reflected by the plane mirror 80. For this reason, the distance to and position of the target are detected using images detected from the two horizontal virtual viewpoints VP9 and VP10.

[0085] This example is characterized by the fact that stereo vision is achieved using only one plane mirror 80. The reference axes SA9 and SA10, which are perpendicular to the direction of the base line length B in the figure, are different from the optical axes OA9 and OA10. Even in this case, distance measurement is possible, as the initial angle of view is simply shifted.

[0086] FIG. 23 shows image P0 detected by sensor 100 in camera module 2. Image P0 is divided into two parts horizontally by a dividing line SL. In FIG. 1, images IM1 and IM2 are divided into approximately equal parts, but in FIG. 23, images IM1 and IM2 are different in size. In reality, the sizes of images IM1 and IM2 can be matched by enlarging the plane mirror 80, but this does not result in a reduction in the size of the stereo camera 10. For this reason, in this embodiment, the plane mirror 80 is set to a predetermined size, and images IM1 and IM2 are configured to be different in size.

[0087] Then, images IM1 and IM2 are generated from image P0 by image branching means 60, and by performing the same processing as in Fig. 5, a stereo camera can be realized with only one camera module and a plane mirror. This solves the conventional problems of low cost, small size, and environmental resistance.

[0088] 5, only the necessary portion of the image IM1 may be extracted and parallax may be detected, which has the advantage of reducing the processing load in the stereo processed image generating unit.

[0089] Although the horizontally elongated sensor is arranged horizontally here, the sensor 100 may also be arranged vertically as shown in FIG. 24A. In this case, there is an advantage that the angle of view in the vertical direction (vertical direction in the figure) can be widened. The stereo camera may also be rotated, for example, by 90 degrees about the depth axis. In this case, images like those shown in FIGS. 24B and 24C are obtained. In the cases of FIGS. 23 and 24A, horizontal parallax is detected, but in the cases of FIGS. 24B and 24C, vertical parallax is detected. FIG. 24B has the advantage of widening the angle of view in the vertical direction. Also, FIG. 24C has the advantage of widening the angle of view in the horizontal direction. Also, in FIGS. 24A and 24B, one side in the horizontal direction can be detected at a wide angle. For example, when an on-board stereo camera is used to grasp the driving situation of a vehicle traveling in the left lane, the angle required for detection differs depending on whether the vehicle is turning left or right at an intersection. The angle can also be widened in a corresponding direction. In addition, in the cases of FIGS. 24B and 24C, the lower side can be detected, so that the vicinity can be detected.

[0090] In the accompanying drawings, functionally identical elements may be designated by the same numerals. The accompanying drawings illustrate examples according to the principles of the present disclosure, but these are for the purpose of understanding the present disclosure and are not intended to limit the scope of the present disclosure. The descriptions in this specification are merely typical examples and are not intended to limit the scope or application of the present disclosure in any way.

[0091] Although the present embodiments are described in sufficient detail to enable those skilled in the art to practice the present disclosure, it should be understood that other embodiments are possible, and that changes in the configuration and structure and substitutions of various elements are possible without departing from the scope and spirit of the technical ideas of the present disclosure. Therefore, the following description should not be interpreted as being limited thereto. [Example]

[0092] The above has described the related technical matters that are the premise of the present invention. Here, the technical concept of "a stereo image processing device that processes stereo images, which includes a plurality of reflecting units 40R, 40L that reflect images of an object, and one camera 99 that captures an image of the object, and the camera 99 detects the image reflected only once by the reflecting units 40R, 40L" and its application development has been described.

[0093] In the present invention, following this concept, three sensor areas are formed on the sensor of the camera's imaging unit, thereby acquiring three images, measuring the distance from the information of two of the sensor areas, and obtaining other image information from the third area. This will be explained below with reference to the examples.

[0094] 15 is a diagram illustrating an example of the configuration of an optical system of a stereo camera 10 according to Example 1 of the present invention. The stereo camera 10 of this example is characterized in that the number of reflections by the mirror is one, similar to that of FIG.

[0095] Here, the two optical axes that realize the stereo camera are defined as follows: optical axis OA5 that reflects off the plane mirror 60R and optical axis OA6 that reflects off the plane mirror 60L. In the first embodiment, the angle between the plane formed by the optical axes OA1 and OA2 inside the stereo camera 10 and the plane formed by the optical axes OA1 and OA2 outside the stereo camera 10 was a predetermined angle α. However, in this embodiment, the plane formed by the optical axes OA5 and OA6 inside the stereo camera 10 and the plane formed by the optical axes OA5 and OA6 outside the stereo camera 10 approximately coincide with each other. In this embodiment, the imaging direction of the stereo camera 10 and the facing direction of the camera module 2 are approximately reversed.

[0096] The relationship between the optical axis and the angle of view will be described with reference to Fig. 15. Fig. 15 shows the configuration when viewed from a cross section formed by optical axes OA5 and OA6 outside the stereo camera 10 (as viewed from the positive position of the vertical axis). Here, dotted lines indicate light rays directed toward the camera module 2.

[0097] 15, light rays (dotted lines in the figure) of the vertical and horizontal angles of view are reflected by the plane mirror 60R and the plane mirror 60L, pass through the lens 99 in the camera module 2, and form an image on the sensor 100. Here, the light rays that are incident on the camera module 2 and are extensions of the light rays (dotted lines) that are directed toward the plane mirrors 60R and 60L are shown by dashed-dotted lines. The approximate points where the dashed-dotted lines intersect are defined as virtual viewpoints VP5 and VP6.

[0098] These virtual viewpoints VP5 and VP6 indicate the effective positions of the camera module 2. This is the same as detecting light rays from the virtual viewpoints VP5 and VP6, respectively, because the light rays are reflected by the plane mirror 60R and the plane mirror 60L. For this reason, in this embodiment, the distance to and position of the target object are detected using images detected from the two horizontal virtual viewpoints VP5 and VP6.

[0099] In this embodiment, the positions and angles of the plane mirrors 60R and 60L are determined so that the light rays have an angle of view such that the camera module 2 itself does not form an image on the sensor.

[0100] Here, the configuration of this embodiment shown in Figure 15 is characterized in that the pupil position of lens 99 in camera module 2 is between the area (range indicated by dotted lines and dashed dotted lines) consisting of light rays incident on plane mirror 60R (including optical axis OA5) and their extensions, and the area (range indicated by dotted lines and dashed dotted lines) consisting of light rays incident on plane mirror 60L (including optical axis OA6) and their extensions.

[0101] In Fig. 15, the plane mirror 60R and the plane mirror 60L are separated from each other and placed at distant positions. As a result, the camera 99 can receive light rays from three directions. Two of these rays are from the plane mirror 60R and the plane mirror 60L, and the remaining one is from the front direction of the camera 99 (direction indicated by symbol B). However, in the case of Fig. 15, the front direction of the camera 99 is normally covered by the case of the stereo camera body, so no light enters from this direction.

[0102] FIG. 16 shows image P0 detected by sensor 100 in camera module 2. Image P0 is divided horizontally into three by division lines SL1 and SL2, resulting in the formation of three image areas. Object 5 is projected onto the image areas at both ends that receive reflected light from plane mirrors 60R and 60L, but no object is reflected in the central image area because no light is received. Note that the image areas in FIG. 16 mean that there are at least three sensor areas on the sensor of the imaging unit: a first sensor area, a second sensor area, and a third sensor area.

[0103] In FIG. 4, the image is detected as tilted, but in the configuration of this embodiment, the plane formed by the optical axes OA5 and OA6 inside the stereo camera 10 and the optical axes OA5 and OA6 outside the stereo camera 10 approximately coincides, so the image is not tilted.

[0104] In this embodiment, for image P0, images IM1 and IM2 are generated from the image regions on both ends by image branching means 60, and a stereo camera can be realized with only one camera module and two plane mirrors by performing the same processing as in Fig. 5. This solves the conventional problems of low cost, small size, and environmental resistance.

[0105] In this embodiment, the plane formed by the optical axes OA5 and OA6 inside the stereo camera 10 coincides with the plane formed by the optical axes OA5 and OA6 outside the stereo camera 10, so the angle of view is reduced by the amount of the camera module 2. On the other hand, if the camera module is small, the effect is very small, and the same effect as in FIG. 1 can be obtained.

[0106] In the present invention, image formation is actively performed by utilizing the central image area in Fig. 16. For example, by configuring the stereo camera 10 as shown in Fig. 17A and Fig. 17B and performing processing such as recognition, it is possible to use pixels effectively.

[0107] In the configuration example of Fig. 17A, a hole 30 is drilled in the case of the stereo camera body in front of the camera 99 to allow external light to enter. As shown in Fig. 17A, image IM3 can detect the opposite direction to the stereoscopic viewing direction (positive depth direction) of images IM1 and IM2. For example, in cars and motorcycles, it is desirable to detect not only the front but also the rear to ensure safety.

[0108] By performing detection as shown in Figure 17A, it is possible to detect not only the front but also the rear. In this way, for example, it is possible to detect automobiles or motorcycles behind or to the side using image IM3. In addition, by comparing image IM1, image IM2 (front) and image IM3 (rear), it is possible to detect the vehicle's inclination in the roll, pitch and yaw directions. In particular, motorcycles have large inclinations in the roll, pitch and yaw directions relative to automobiles, so these must be detected in order to provide accurate warnings and control, and this information can be obtained.

[0109] Furthermore, for example, image IM3 can be used to monitor the interior of a car. This information can be used to add functions such as driver status, engine start using facial recognition, and detection of items left behind inside the car. In the case of a motorcycle, this information can also be used to determine whether the vehicle is the owner, adding functions such as anti-theft. To achieve these functions, for example, the lens 99 may have different characteristics between its inner and outer peripheries. In this case, the stereoscopic view area (images IM1 and IM2) may have high resolution in the distance, while the monocular view area (IM3) may have high resolution in the near distance.

[0110] 17B shows a plane mirror 429 installed in front of the camera 99. Depending on the direction the plane mirror 429 faces, it becomes possible to monitor directions other than just the front and rear. For example, by using the plane mirror 429 to detect the negative vertical side in the image IM3, it is possible to detect nearby objects and road surfaces relative to the stereo camera 10. Furthermore, by using the plane mirror 429 to detect the positive vertical side in the image IM3, it is also possible to recognize traffic signals and signs.

[0111] In this embodiment, the horizontally long sensor is arranged horizontally, but the sensor 100 may also be arranged vertically as shown in FIG. 18A. In this case, there is an advantage that the angle of view in the vertical direction (vertical direction in the figure) can be widened. The stereo camera 10 may also be rotated, for example, by 90 degrees about the depth axis. In this case, images like those shown in FIGS. 18B and 18C are obtained. In the cases of FIGS. 16 and 18A, parallax in the horizontal direction is detected, but in the cases of FIGS. 18B and 18C, parallax in the vertical direction is detected. FIG. 18B has the advantage that the angle of view in the vertical direction can be widened. In addition, FIG. 18C has the advantage that the angle of view in the horizontal direction can be widened.

[0112] 15, the camera module 2, the plane mirror 60R, and the plane mirror 60L are on the same plane (a plane formed by the depth direction and the horizontal direction), but the same effect can be obtained even if the camera module 2 is shifted in the vertical direction. In this case, the configuration is similar to that in FIG.

[0113] Furthermore, for the imaging lens of the camera module 2, an equidistant projection is desirable for the same reasons as in Figure 1. However, rather than an equidistant projection, a projection that compresses the image at the central angle of view and enlarges the image at the wide-angle side can expand the field of view for stereoscopic viewing of images IM1 and IM2. For example, in the case of a general projection, the wide-angle side is used, such as central projection (f tan θ) or stereographic projection (2f tan(θ / 2)).

[0114] The stereo image processing device described in Example 1 is configured as follows: "It comprises a first reflecting unit 60R and a second reflecting unit 60L that reflect an image of the object 5, and an imaging unit 99 that captures an image of the object 5, and the sensor of the imaging unit 99 has at least three sensor areas, namely, a first sensor area IM1, a second sensor area IM2, and a third sensor area IM3, and the third sensor area IM3 is sandwiched between the first sensor area IM1 and the second sensor area IM2, and light from the object 5 that is reflected by the first reflecting unit 60R is incident on the first sensor area IM1, and light from the object 5 that is reflected by the second reflecting unit 60L is incident on the second sensor area IM2." [Example]

[0115] 19 is a diagram illustrating an example of the configuration of an optical system of a stereo camera 10 according to Example 2 of the present invention. The stereo camera 10 of this example is characterized in that the number of reflections by the mirror is one, similar to that of FIG.

[0116] Here, the optical axis reflected by the plane mirror 70R is designated as OA7, and the optical axis reflected by the plane mirror 70L is designated as OA8. In FIG. 1 , the angle formed by the plane formed by the optical axes OA1 and OA2 inside the stereo camera 10 and the plane formed by the optical axes OA1 and OA2 outside the stereo camera 10 is a predetermined angle α, but in this embodiment, the plane formed by the optical axes OA7 and OA8 inside the stereo camera 10 and the plane formed by the optical axes OA7 and OA8 outside the stereo camera 10 are substantially aligned. Also, while in the first embodiment the imaging direction of the stereo camera 10 and the facing direction of the camera module 2 are substantially reversed, the imaging direction of the stereo camera 10 of this embodiment are substantially aligned with the facing direction of the camera module 2.

[0117] The relationship between the optical axis and the angle of view will be described with reference to Fig. 19. Fig. 19 shows the configuration when viewed from a cross section formed by optical axes OA7 and OA8 outside the stereo camera 10 (as viewed from the positive position of the vertical axis). Here, dotted lines indicate light rays directed toward the camera module 2.

[0118] 19, light rays (dotted lines in the figure) of the vertical and horizontal angles of view are reflected by the plane mirror 70R and the plane mirror 70L, pass through the lens 99 in the camera module 2, and form an image on the sensor 100. Here, the light rays that are incident on the camera module 2 and are directed toward the plane mirror 70R and the plane mirror 70L (dotted lines) are extended as dashed-dotted lines.

[0119] The approximate intersecting points of the dashed dotted lines are defined as virtual viewpoints VP7 and VP8. These virtual viewpoints VP7 and VP8 indicate the effective positions of the camera module 2. This is equivalent to detecting light rays from the virtual viewpoints VP7 and VP8, respectively, because the light rays are reflected by the flat mirror 70R and the flat mirror 70L. For this reason, in this embodiment, the distance and position to the target are detected using images detected from the two horizontal virtual viewpoints VP7 and VP8. This embodiment is characterized by performing stereoscopic viewing using the wide-angle portion of the camera module 2.

[0120] FIG. 20 shows image P0 detected by sensor 100 in camera module 2. Image P0 is divided horizontally into three by division lines SL1 and SL2, forming three image areas. Object 5 is projected in the image areas at both ends that receive reflected light from plane mirrors 70R and 70L, and in the central image area. Note that the image areas in FIG. 20 mean that there are at least three sensor areas on the sensor of the imaging unit: a first sensor area, a second sensor area, and a third sensor area.

[0121] In FIG. 1, the image is detected as tilted, but in the configuration of this embodiment, the plane formed by the optical axes OA7 and OA8 inside the stereo camera 10 and the optical axes OA7 and OA8 outside the stereo camera 10 approximately coincides, so the image is not tilted.

[0122] Here, the configuration of this embodiment shown in Figure 19 is characterized in that the pupil position of lens 99 in camera module 2 is between the area (range indicated by dotted lines and dashed dotted lines) consisting of light rays incident on plane mirror 70R (including optical axis OA7) and their extensions, and the area (range indicated by dotted lines and dashed dotted lines) consisting of light rays incident on plane mirror 70L (including optical axis OA8) and their extensions.

[0123] In this embodiment, images IM1 and IM2 are generated from image P0 by image branching means 60, and a stereo camera can be realized with only one camera module and two plane mirrors by performing the same processing as in Fig. 5. This solves the conventional problems of low cost, small size, and environmental resistance.

[0124] In this embodiment, images IM1, IM2, and IM3 may have different resolutions. For example, the peripheral and central portions of the lens 99 may have different resolutions. Increasing the resolution of the peripheral portion of the lens 99 makes it possible to measure the distance to a distant object. Lowering the resolution of the central portion of the lens 99 makes it possible to achieve a wide angle of view. Conversely, increasing the resolution of the central portion makes it possible to detect a distant object, albeit with a monocular vision. Lowering the resolution of the peripheral portion of the lens 99 makes it possible to achieve a wide angle of view.

[0125] In this embodiment, the horizontally elongated sensor is arranged horizontally, but the sensor 100 may also be arranged vertically as shown in FIG. 21A. In this case, there is an advantage that the angle of view in the vertical direction (vertical direction in the figure) can be widened. The stereo camera 10 may also be rotated, for example, by 90 degrees about the depth axis. In this case, images like those shown in FIGS. 21B and 21C are obtained. In the cases of FIGS. 20 and 21A, parallax in the horizontal direction is detected, but in the cases of FIGS. 21B and 21C, parallax in the vertical direction is detected. FIG. 21B has the advantage that the angle of view in the vertical direction can be widened. In addition, FIG. 21C has the advantage that the angle of view in the horizontal direction can be widened.

[0126] Furthermore, by using a mirror as in the first embodiment, information on objects, road surfaces, traffic signals, signs, and other objects in the upward and downward directions relative to the stereo camera 10 can be detected in the image IM3.

[0127] In this embodiment shown in Figure 19, the camera module 2, flat mirror 70R, and flat mirror 70L are on the same plane (a plane formed by the depth direction and horizontal direction), but the same effect can be obtained even if the camera module 2 is configured to be shifted in the vertical direction.

[0128] Furthermore, for the imaging lens of the camera module 2, an equidistant projection is desirable for the same reasons as in Figure 1. However, rather than an equidistant projection, a projection that compresses the image at the central angle of view and enlarges the image at the wide-angle side can expand the field of view for stereoscopic viewing of images IM1 and IM2. For example, in the case of a general projection, the wide-angle side is used, such as central projection (f tan θ) or stereographic projection (2f tan(θ / 2)).

[0129] The present invention is not limited to the above-described embodiments and includes various other modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations. [Explanation of symbols]

[0130] 10: stereo camera, 20a: affine processing means, 20b: affine processing means, 30R: mirror, 30L: mirror, 40R: mirror, 40L: mirror, 60: image separation means, 300: stereo parallax image generation unit, 500: stereoscopic three-dimensional object detection unit

Claims

1. A stereo image processing device for processing stereo images, The device includes a first reflecting section and a second reflecting section that reflect an image of an object, and a camera that photographs the object, the camera has at least three sensor areas on its sensor, including a first sensor area, a second sensor area, and a third sensor area; the third sensor area is sandwiched between the first sensor area and the second sensor area; light from the object and reflected by the first reflecting portion is incident on the first sensor area; light from the object and reflected by the second reflecting portion is incident on the second sensor area; a first detected image detected in the first sensor area and a second detected image detected in the second sensor area are both images that detect the object in a first direction, and a third detected image detected in the third sensor area is an image that detects an area that includes a second direction different from the first direction and that is not detected by the first sensor area and the second sensor area.

2. 2. The stereo image processing device according to claim 1, A stereo image processing device characterized in that a first image is detected in the first sensor area, a second image is detected in the second sensor area, and a parallax calculation unit is provided to calculate parallax using the first image and the second image.

3. 2. The stereo image processing device according to claim 1, A stereo image processing device comprising a recognition unit that processes an image of a person detected in the third sensor area.

4. 2. The stereo image processing device according to claim 1, A stereo image processing device comprising a traffic sign recognition unit for processing images of traffic signals or road signs detected in at least the third sensor area.

5. 2. The stereo image processing device according to claim 1, The stereo image processing device is characterized in that the camera is composed of an imaging lens and a sensor, and the projection of the imaging lens is an equidistant projection.

6. 2. The stereo image processing device according to claim 1, The camera is composed of an imaging lens and a sensor, and the projection of the imaging lens is a projection in which the image is compressed at the central angle of view and enlarged at the wide-angle side compared to an equidistant projection.

7. 2. The stereo image processing device according to claim 1, The stereo image processing device, wherein the first reflecting section and the second reflecting section are disposed at positions spaced apart from each other.

Images