Stereo camera, stereo camera system, and arrangement method

Abstract

To provide a stereo camera capable of achieving a long distance and a wide angle.SOLUTION: Each of a first rotation axis of a first rotation mechanism and a second rotation axis of a second rotation mechanism is orthogonal to a plane including both of a first optical axis of a first camera and a second optical axis of a second camera. The first rotation axis intersects with the first optical axis and the second rotation axis intersects with the second optical axis so that the distance from the intersection point of the first rotation axis and the first optical axis to a first entrance pupil position of the first camera is equal to the distance from the intersection point of the second rotation axis and the second optical axis to a second entrance pupil position of the second camera.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] The present invention relates generally to a stereo camera for object detection, ranging and / or identification. [Background technology]

[0002] In recent years, a technology has been disclosed for a stereo camera having a mechanism capable of controlling camera settings such as zoom, pan, and tilt, which has at least two cameras and estimates the camera parameters of the cameras when the lens position of the cameras is changed by an instruction to operate the cameras (see Patent Document 1).

[0003] Also disclosed is an image capture device equipped with a rotation mechanism that rotates an imaging camera in at least one direction, wherein image data output from the image capture device has fθ characteristics, and the center of rotation of the rotation mechanism is located approximately at the center of the entrance pupil of the optical system (see Patent Document 2). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-135495 A [Patent Document 2] JP 2004-289367 A Summary of the Invention [Problem to be solved by the invention]

[0005] Object detection systems using cameras need to be able to detect objects at longer distances and wider angles in response to demands for wider detection ranges. Cameras play an important role as visual sensors, and so longer distances can be achieved by lengthening the focal length of the lens, but the image sensor size cannot be increased proportionally, resulting in a smaller and narrower angle of view. Conversely, to achieve a wider angle of view, the focal length must be shortened, resulting in a lower resolution and difficulty in detecting objects at longer distances.

[0006] One way to deal with this is to use a lens with distortion, such as a fisheye lens, to achieve high resolution near the optical axis and a wide angle by compressing the image at the periphery. However, the projection methods used in fisheye lenses, equidistant projection (f θ), equisolid angle projection (2f sin(θ / 2)), and orthogonal projection (f sinθ), do not provide sufficient results unless there is a certain angle of view.

[0007] On the other hand, the pan and tilt operations that mechanically change the orientation of the camera as described in Patent Documents 1 and 2 are a rational method of ensuring long-distance performance with the optical performance of the camera and realizing a wide angle by pan and tilt operations. However, Patent Documents 1 and 2 do not mention cases where the entire angle of view needs to be detected simultaneously, and the techniques described in Patent Documents 1 and 2 make it difficult to realize long distance and wide angle.

[0008] The present invention has been made in consideration of the above points, and aims to propose a stereo camera etc. that can realize long distance and wide angle shooting. [Means for solving the problem]

[0009] In order to solve this problem, the present invention provides a stereo camera including a first camera, a first rotation mechanism that rotates the first camera, a second camera, and a second rotation mechanism that rotates the second camera, wherein a first rotation axis of the first rotation mechanism and a second rotation axis of the second rotation mechanism are each perpendicular to a plane that includes both a first optical axis of the first camera and a second optical axis of the second camera, the first rotation axis intersects with the first optical axis, and the second rotation axis intersects with the second optical axis, such that a distance from an intersection of the first rotation axis and the first optical axis to a first entrance pupil position of the first camera is equal to a distance from an intersection of the second rotation axis and the second optical axis to a second entrance pupil position of the second camera.

[0010] In the above configuration, the distance from the rotation axis of the rotation mechanism to the entrance pupil position of each camera is the same, so that, for example, by performing coordinate transformation of the image using rotation angle information, it is possible to achieve a longer distance and wider angle detection range while maintaining the distance measurement accuracy of the detected object. Effect of the Invention

[0011] According to the present invention, a stereo camera capable of handling longer distances and wider angles can be realized. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating an example of a stereo camera for explaining a first embodiment. [Diagram 2] FIG. 1 is a diagram illustrating an example of a stereo camera for explaining a first embodiment. [Diagram 3] FIG. 1 is a diagram illustrating an example of a stereo camera for explaining a first embodiment. [Figure 4] FIG. 1 is a diagram illustrating an example of a stereo camera for explaining a first embodiment. [Diagram 5] FIG. 1 is a diagram illustrating an example of a stereo camera for explaining a first embodiment. [Figure 6] FIG. 1 is a diagram illustrating an example of a stereo camera for explaining a first embodiment. [Figure 7] FIG. 4 is a diagram showing distance measurement errors for explaining the first embodiment. [Figure 8] 1 is a diagram illustrating an example of a configuration of an object detection system according to a first embodiment. [Figure 9] FIG. 2 is a diagram showing the positional relationship between a camera and a rotation mechanism according to the first embodiment. [Figure 10] FIG. 2 is a diagram for explaining coordinate transformation according to the first embodiment. [Figure 11] FIG. 4 is a diagram for explaining distance calculation according to the first embodiment. [Figure 12]5A to 5C are diagrams for explaining inconveniences that arise when the rotation axis is not on the entrance pupil in the first embodiment. [Figure 13] FIG. 2 is a diagram for explaining calibration according to the first embodiment. [Figure 14] FIG. 2 is a diagram for explaining calibration according to the first embodiment. [Figure 15] FIG. 11 is a diagram showing the positional relationship between a camera and a rotation mechanism according to the second embodiment. [Figure 16] 13 is a diagram for explaining a method for solving the problem that occurs when the rotation axis is not located at the entrance pupil according to the second embodiment. FIG. [Figure 17] FIG. 11 is a diagram for explaining coordinate transformation according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] (I) First embodiment An embodiment of the present invention will be described in detail below, however, the present invention is not limited to the embodiment.

[0014] In this embodiment, a stereo camera having a long-distance, wide FOV (Field of view) detection range and high-precision distance measurement performance, and a stereo camera system using the stereo camera will be described in particular.

[0015] The stereo camera system described in this embodiment includes a plurality of means for solving the above problems. For example, the stereo camera system is characterized in that "camera 1 and camera 2 constituting the stereo camera each have a rotation mechanism for changing the shooting direction, and when the entrance pupil positions of each camera are P1 and P2 and a line including P1 and P2 is L, the rotation axes R1 and R2 of the rotation mechanisms are perpendicular to the line L passing through P1 and P2 and the optical axes O1 and O2 of each camera, and are parallel to each other." Also, the stereo camera system is characterized in that "based on the rotation angle θ of the rotation mechanism, correction processing is performed on at least one of the captured images of each camera or processed images obtained by performing image processing on the captured images."

[0016] By having a rotation mechanism for each stereo camera that changes the shooting direction, aligning the rotation axis of the rotation mechanism with the entrance pupil position of each camera, and performing coordinate conversion of the image using the rotation angle information, it is possible to achieve a long-distance, wide-angle detection range and space savings while maintaining the distance measurement accuracy of the detected object.

[0017] The designations "first," "second," "third," and the like in this specification are given to identify components and do not necessarily limit the number or order. Furthermore, numbers for identifying components are used in each context, and a number used in one context does not necessarily indicate the same configuration in another context. Furthermore, a component identified by a certain number is not prevented from also having the function of a component identified by another number.

[0018] Next, an embodiment of the present invention will be described based on the drawings. The following description and drawings are examples for explaining the present invention, and are omitted and simplified as appropriate for clarity of explanation. The present invention can be implemented in various other forms. Unless otherwise specified, each component may be singular or plural.

[0019] In the following description, the same elements in the drawings are given the same numbers, and the description is omitted as appropriate. When describing elements of the same type without distinction, the common part (part excluding the branch number) of the reference sign including the branch number is used, and when describing elements of the same type while distinguishing between them, the reference sign including the branch number may be used. For example, when describing cameras without distinction between them, they may be described as "camera 401," and when describing individual cameras while distinguishing between them, they may be described as "camera 401-1," "camera 401-2," and so on.

[0020] [Each camera rotates to save space. The diagonal model has distance measurement errors.] Generally, changing the camera orientation causes some inconveniences. In central projection (f·tanθ), which is the usual camera projection method, the image appears larger the further away from the optical axis. For example, a person standing at the corner of the camera screen appears wider on the image than a person standing directly in front of the camera. This can be solved by using equidistant projection (f·θ). When equidistant projection is used, the image of an object is equal regardless of the angle of view, so the image size of the same object can be kept constant even if the camera orientation is changed by panning or tilting. In addition, by aligning the camera's rotation axis with the camera's entrance pupil, the distance to the object can be kept accurately constant even if the camera orientation is changed, and the variation in image size due to the angle of view can be further reduced.

[0021] Here, when changing the shooting direction of the stereo camera, it is necessary to accurately maintain the relative positions and orientations of the two cameras. For this reason, as shown in Figs. 1, 2, and 3, it is easier to achieve a wide angle by changing the orientation of the stereo camera 100 as a whole and maintaining the relative positions of the cameras 101. However, unlike a monocular camera, when changing the orientation of the stereo camera 100, it is necessary to secure a large space where the cameras 101 and the fixing mechanism for the cameras 101 do not interfere with other members. This increases according to the required rotation angle and baseline length of the cameras 101, so it is difficult to install the stereo camera 100, which is particularly suitable for long distances and has a long baseline, in a narrow driver's seat or cab of an automobile, train, etc.

[0022] Therefore, as shown in Figures 4, 5, 6, etc., by configuring the stereo camera 400 so that each camera 401 has an independent rotation axis, it is possible to realize a wide angle without providing a space larger than the horizontal projection area of ​​the device. However, this method has the disadvantage that an error occurs in the distance measurement result when the orientation of the camera 401 is tilted. For example, if the case of Figure 5 is replaced with that of Figure 7, the image sizes I1 and I2 of the camera 401-1 (camera 1) and the camera 401-2 (camera 2) are respectively: I1 = Ob / L1 fc Equation (1) I2=Ob / L2 fc=Ob / (L1-B sin(Δθ)) fc...Equation (2) It becomes. Here, Ob is the size of the target object, L1 and L2 are the distances from camera 1 and camera 2 to the target object, respectively, fc is the focal length of camera 1 and camera 2, B is the baseline length, and Δθ is the orientation (rotation angle) of camera 1 and camera 2. Both equations show that the image sizes of the target object obtained by camera 1 and camera 2 are different. In addition, since the image size ratio also differs depending on the distance to the target object, simply enlarging or reducing one of the images will result in a distance measurement error.

[0023] In this embodiment, as an example of a stereo camera system, a stereo camera mounted on a train and an object detection system 800 using the stereo camera will be described. FIG. 8 shows a schematic configuration of the object detection system 800. First, the role of each block will be described. Reference numerals 801 and 802 indicate cameras constituting the stereo camera. Reference numerals 811 and 812 indicate a rotation mechanism (ROTATION MECH.) that changes the shooting direction of the cameras 801 and 802. Reference numeral 820 indicates a controller unit (CAMERA & ANGLE CONTROLLER) that controls the shooting timing of the cameras 801 and 802 and the operation of the rotation mechanisms 811 and 812. Reference numeral 830 indicates a geometric correction unit (GEOMETRIC CORRECTION) that corrects image distortion, image orientation, image size, etc. for the images acquired by the cameras 801 and 802. Reference numeral 840 indicates a coordinate conversion unit (COORDINATE CORRECTION) that mainly performs coordinate conversion of pixel values ​​based on the rotation angle of the rotation mechanisms 811 and 812. Reference numeral 850 denotes a disparity image generating unit (DISPARITY) that generates a disparity image using an image that has been subjected to geometric correction and coordinate transformation. Reference numeral 860 denotes an object detecting unit (IMAGE ANALYSIS OBJECT DETECTION) that performs distance measurement, image analysis, object detection, etc. using the disparity image and the original image.

[0024] The functions of the object detection system 800 (controller unit 820, geometric correction unit 830, coordinate conversion unit 840, parallax image generation unit 850, object detection unit 860, etc.) may be realized, for example, by a processor reading a program stored in an auxiliary storage device into a main storage device and executing it (software), or may be realized by hardware such as a dedicated circuit, or may be realized by a combination of software and hardware. Note that one function of the object detection system 800 may be divided into multiple functions, or multiple functions may be integrated into one function. For example, the geometric correction unit 830, the coordinate conversion unit 840, and the parallax image generation unit 850 may be a correction unit. Also, some of the functions of the object detection system 800 may be provided as separate functions, or may be included in other functions. For example, geometric correction and coordinate conversion may be performed simultaneously. In addition, the stereo camera including the cameras 801 and 802 may include, for example, a controller unit 820, a geometric correction unit 830, a coordinate conversion unit 840, and a parallax image generation unit 850, and a computer capable of communicating with the stereo camera may include an object detection unit 860. Also, for example, the stereo camera including the cameras 801 and 802 may include, for example, a controller unit 820, and a computer capable of communicating with the stereo camera may include, for example, a geometric correction unit 830, a coordinate conversion unit 840, a parallax image generation unit 850, and an object detection unit 860. Also, some of the functions of the object detection system 800 may be realized by another computer capable of communicating with the object detection system 800.

[0025] Next, the relationship between each block will be described. The rotation mechanisms 811 and 812 rotate at a rotation angle instructed by the controller unit 820, and change the shooting direction of the cameras 801 and 802 fixed to the rotation mechanisms 811 and 812, respectively. The cameras 801 and 802 shoot images at the timing instructed by the controller unit 820, and send the acquired images (image data) to the geometric correction unit 830. For each image sent to the geometric correction unit 830, the geometric correction unit 830 performs geometric correction processing including lens distortion correction and parallelization processing, and sends the processed geometrically corrected image to the coordinate conversion unit 840. The coordinate conversion unit 840 performs coordinate conversion of the geometrically corrected image sent from the geometric correction unit 830 based on the rotation angle information sent from the controller unit 820, and sends the coordinate converted image to the parallax image generation unit 850 and the object detection unit 860. The parallax image generating unit 850 generates parallax images using coordinate transformation images based on the images sent from the cameras 801 and 802, and sends the generated parallax images to the object detecting unit 860. The object detecting unit 860 performs distance measurement, object detection, object identification, and the like using the sent parallax images and coordinate transformation images, and sends information necessary for vehicle control to a downstream system (not shown). For example, if the downstream system is a brake control system, the brake control system performs vehicle brake control based on the received information (for example, information indicating that an obstacle has been detected on the track).

[0026] FIG. 9 is a diagram showing the positional relationship between the cameras 801, 802 and the rotation mechanisms 811, 812. Reference numerals 901, 902 indicate the lenses of the cameras 801, 802. Note that the cameras 801, 802 are provided with one or more lenses 901, 902, and typically, a plurality of lenses are provided, but FIG. 9 shows only one lens for convenience of explanation. Reference numerals 911, 912 indicate the image pickup elements of the cameras 801, 802. Reference numerals 921, 922 indicate the entrance pupil positions (entrance pupils) of the cameras 801, 802. Reference numerals 931, 932 indicate the optical axes of the lenses 901, 902 (cameras 801, 802). Reference numerals 941, 942 indicate the rotation axes of the rotation mechanisms 811, 812 with dashed lines. These rotation axes 941, 942 pass through the entrance pupils 921, 922, respectively, and are disposed so as to be perpendicular to the optical axes 931, 932 and the straight line L. Here, the straight line L is a straight line that passes through both the entrance pupils 921, 922. In general, the optical axes of both cameras of a stereo camera are disposed so as to be parallel to each other. In other words, the above can be said to be such that the rotation axes 941, 942 are disposed so as to be perpendicular to a plane that includes both optical axes 931, 932, and to pass through each of the entrance pupils 921, 922.

[0027] Moreover, the rotation angle θ is based on a state in which the optical axes 931, 932 are perpendicular to the straight line L (θ=0 deg), and the following description will be given on this premise.

[0028] [Conversion formula] Next, the detailed operation of the coordinate conversion unit 840 will be described with reference to FIG. 10. FIG. 10 is a schematic diagram of the camera 801 facing in the direction of a rotation angle θ=Δθ, and also shows the arrangement of the camera 801 when θ=0 using a dotted line. The camera 801 in the figure is a central projection model after geometric correction, and is equivalent to a pinhole model. Reference numeral 1001 denotes a lens modeled after the lens 901. As shown in FIG. 10, basically, in the pinhole model, the pinhole position and the entrance pupil position of the camera 801 coincide with each other. Note that the camera 802 is similar to the camera 801, and a description thereof will be omitted.

[0029] The coordinate conversion by the coordinate conversion unit 840 is an operation of converting the value of real pixel B into the value of virtual pixel A, where pixel B is a pixel that a light ray incident from a certain direction reaches when the rotation angle of the camera 801 is Δθ, and pixel A is a pixel that the light ray reaches when the rotation angle θ=0. Specifically, when the coordinates of each pixel in the pre-conversion image are B(Ub,Vb), the coordinates of a reference pixel (e.g., center pixel) in the pre-conversion image are B(Ub0,Vb0), the coordinates of each pixel in the post-conversion image (post-correction image) are A(Ua,Va), the coordinates of a reference pixel (e.g., center pixel) in the post-conversion image are A(Ua0,Va0), the focal length of the camera 801 is fc, and the pixel pitches in the x and y directions are px,py, the coordinates after conversion are expressed as follows using the coordinates before conversion and the rotation angle Δθ: Ua=fc·tan(Δθ+α) / px+Ua0...Equation (3) Va=Vb-Vb0+Va0...Equation (4) α=atan((Ub-Ub0)·px / fc) ··· Formula (5) This can be expressed as:

[0030] When the rotation angle θ=0, the light ray incident from the angle of view (Δθ+α) reaches outside the range of the image sensor 911, and therefore the image sensor 911 cannot capture the image. Therefore, by changing the camera orientation by the rotation angle Δθ, the light ray reaches within the range of the image sensor 911, and α can be obtained from the coordinates of the pixel where the image is formed. Therefore, the angle of incidence (Δθ+α) of the light ray can be obtained using the known rotation angle Δθ, and the coordinates of the pixel where the image is formed by the light ray when the rotation angle θ=0 can be calculated.

[0031] [Disparity and distance obtained by transformation] Next, an example of distance calculation of a target object will be shown using Fig. 11. Here, the target object is a point light source, and the cameras 801 and 802 are not shown, but the entrance pupils 921 and 922 of the cameras 801 and 802 are indicated by black circles. The U coordinate of the image-forming pixel of the light ray when the rotation angle θ = 0 is expressed as (Ua - Ua0) from the above formula, so if the same coordinates of the cameras 801 and 802 are expressed as (Ua1 - Ua10) and (Ua2 - Ua20), then (Ua1-Ua10)=tan(Δθ+α) fc / px Formula (6) (Ua2-Ua20)=tan(Δθ) fc / px...Equation (7) Here, Y / L = tan(Δθ) Equation (8) (B+Y) / L=tan(Δθ+α)...Equation (9) Since the disparity D is (Ua1-Ua10)-(Ua2-Ua20), from the above formula, D = B / L fc / px Equation (10) Using the measured value of this parallax D, the distance L of the light spot can be obtained. L = B / D fc / px Equation (11)

[0032] [What happens if the entrance pupil is not the rotation axis?] Here, the inconvenience that occurs when the rotation axis 941 is not the entrance pupil 921 will be described using the camera 801. FIG. 12 is a diagram in which the rotation axis 941 is provided on the image sensor 911 and the camera orientation is set to a rotation angle Δθ. The dashed and dotted line indicates the optical axis 931 of the rotation angle Δθ of the camera 801, and the solid thick line indicates the image plane 1201 of the image sensor 911. The light ray direction β can be calculated from the coordinates of the image of the light point formed on the image plane 1201 and the focal length. However, unlike the case of FIG. 11, the light ray direction (Δθ+α) in the state where the camera orientation θ=0 is not calculated even if the light ray direction β is used. This is because, even if it is known that the target object exists on the light ray direction β, the light ray direction in the state where the camera orientation θ=0 is set to a different value such as α0 or α1 depending on the distance of the target object. This occurs because the pinhole positions in the camera orientations θ=0 and Δθ do not match.

[0033] [Calibration] The above parallax and distance can be obtained accurately when the rotation mechanisms 811 and 812 operate ideally, but in reality, there are mechanical rotation errors and mechanical errors caused by the inclination and backlash of the rotation axes 941 and 942, which can cause non-negligible parallax errors and distance measurement errors, or problems such as the parallax not being obtained at all. In this case, calibration of the conversion formula according to the rotation angle Δθ is required. Calibration requires initial calibration to correspond to the initial mechanical error at the time of system manufacture, and time-dependent calibration to correspond to the mechanical error caused by changes over time. The purpose of time-dependent calibration can be achieved using the same equipment and method as the initial calibration, but it may also be operated so that the calibration operation is performed in the background during business hours. Below, the methods of initial calibration and time-dependent calibration will be explained in order.

[0034] [Proofreading (initial, with chart)] An example of initial calibration of the conversion equation according to the rotation angle Δθ will be described with reference to Figs. 13 and 14. In both figures, a stereo camera 1300 is shown at the bottom of the paper, and charts 1301 and 1302 are shown at the top of the paper. It is desirable that the charts 1301 and 1302 have a high-contrast pattern all over so that parallax can be obtained. For example, checker marks, evenly spaced dots or doughnut-shaped marks, random patterns, etc. are available. The distance between the charts 1301 and 1302 and the cameras 801 and 802 is ideally far enough to focus and to allow both cameras 801 and 802 to capture a common area on the charts 1301 and 1302.

[0035] First, the chart image is acquired in a state where the cameras 801 and 802 are oriented so that the straight line L described in Fig. 9 is perpendicular to the optical axes 931 and 932 of the cameras 801 and 802. At this time, for example, the area a1 shown in Fig. 13 is captured on the left side of the camera 801 (left camera), and the area a2 is captured on the right side of the camera 801.

[0036] Next, the orientation of the cameras 801 and 802 is changed by half the angle of view to the state shown in FIG. 14. If the orientation of the cameras 801 and 802 moves ideally, an image obtained by geometrically and coordinate-transforming area a2, which is the left half of the image acquired in the state shown in FIG. 14, should match an image obtained by geometrically transforming area a2 acquired in the state shown in FIG. 13. However, if there is a mechanical error during rotation, the two images will not match, so the amount of misalignment between corresponding pixels in both images is found, and calibration is performed to eliminate the amount of misalignment. The amount of misalignment for each pixel can be obtained by block matching, feature point matching, or the like of both images, but any means may be used as long as the amount of misalignment can be obtained.

[0037] Area a3 can be calibrated by extrapolating the calibration of area a2. Thereafter, the amount of deviation between the right half area a(n) of the image rotated n times the half angle of view and the left half area a(n+1) of the image rotated (n+1) times is calibrated. The area to the left of area a1 is also calibrated by changing the rotation angle to the negative side. The camera 802 on the right side of the paper in Figures 13 and 14 can be calibrated in the same way. The initial calibration is completed through the above procedure.

[0038] [Calibration (over time, no chart)] The method of time-dependent calibration can be basically the same as that of initial calibration. In a maintenance yard where maintenance and repairs are performed, a dedicated chart can be provided or the shaking of the cameras 801 and 802 can be eliminated, and the accuracy can be almost the same as that of the initial calibration. On the other hand, ingenuity is required when performing calibration on a vehicle in operation. In particular, when the cameras 801 and 802 are shaken, it is not possible to determine whether the mismatch of the images used for calibration is due to the shaking or mechanical error. Therefore, it is desirable to perform the calibration operation when there are no passengers or luggage getting on or off the vehicle, when there is no movement of passengers or luggage inside the vehicle, or when the same situation is detected. In addition, it is desirable to avoid environments that include low-contrast images that are unsuitable for block matching, feature point matching, etc., set conditions to obtain images with a certain level of contrast, accumulate calibration information, and build a mechanism to perform calibration after obtaining information of a certain level or more.

[0039] Discrete Values Considering the above calibration method, it is desirable that the camera orientation (rotation angle Δθ) has discrete values ​​and that a calibration value is provided for each angle. It is also desirable that the actual shooting direction is changed discretely and that a calibration value is applied for each angle.

[0040] [Calibration does not have to be in units of half angle] However, as described above, the discrete values ​​do not need to match the half angle of view of the cameras 801 and 802. The finer the discrete values, the longer the calibration takes, but the higher the accuracy of extrapolation is expected, enabling highly accurate distance measurement in practical use. Conversely, if the discrete interval is made larger, the calibration time can be shortened, but there is a concern that errors during extrapolation may increase.

[0041] [The accuracy of the extrapolated parts other than the left and right half pairs is questionable. Adjust the linearity and equal spacing as a whole.] If errors due to extrapolation cause actual harm, calibration may be performed by placing straight lines, evenly spaced marks, or orthogonal lines over the entire area of ​​the charts 1301 and 1302 to ensure straight lines, even spacing, and orthogonality in the calibrated image. This can eliminate practical problems.

[0042] [Landing from the same direction] Furthermore, mechanical errors during rotation can be reduced by devising control over the rotation mechanisms 811, 812. In other words, the controller unit 820 controls the rotation mechanisms 811, 812 so that they always move in one direction and then stop, regardless of the rotation angle at which they are stopped. This reduces the hysteresis of the rotation mechanisms 811, 812 and the backlash of the rotation mechanisms 811, 812, and can reduce mechanical errors.

[0043] [Calibration formula] Let the calibration value in the x direction be Ka and the calibration value in the y direction be Kb. Ua' = Ua + Ka Equation (12) Va' = Va + Kb (13) Here, Ua=fc tan(Δθ+α) / px+Ua0. Va=Vb-Vb0+Va0.

[0044] As described above, by arranging the rotation axis that changes the camera orientation so that it passes through the entrance pupil and is perpendicular to the line connecting each optical axis and the entrance pupil, and performing geometric transformation to central projection and coordinate transformation based on the rotation angle information on the images captured by both cameras, it is possible to provide a stereo camera that can achieve long distances, a wide angle of view, and space savings while providing high-precision ranging performance, and an object detection system using this stereo camera.

[0045] (II) Second embodiment [An example where the entrance pupil is not equal to the rotation axis. However, the rotation angles of the left and right cameras must be the same.] In the first embodiment, the rotation axes 941, 942 of the rotation mechanisms 811, 812 of the cameras 801, 802 pass through the entrance pupils 921, 922 of the cameras 801, 802, but this is not limiting and similar effects can be obtained. That is, in Fig. 9, the rotation axes 941, 942 pass through the entrance pupils 921, 922, but in this embodiment, the rotation axes 941, 942 do not pass through the entrance pupils 921, 922, but instead pass through points on the optical axes 931, 932 that are equidistant and equidirectional from the entrance pupils 921, 922 as shown in Fig. 15, and accurate distance measurement can also be performed.

[0046] The components of the object detection system 800 are the same as those in FIG. 8 which describes the first embodiment, so they will be omitted. The relationship between the components is also the same as in the first embodiment, so they will be omitted. The difference from the first embodiment is in the way of thinking about the entrance pupil position that moves when the cameras 801 and 802 face the rotation angle Δθ. That is, in the first embodiment, it was shown using FIG. 12 that the entrance angle α cannot be obtained from the entrance angle β, but in this embodiment, it is assumed that the stereo camera itself has moved the same amount as the entrance pupils 921 and 922. Specifically, as shown in FIG. 16, when θ=0, the cameras 801 and 802 are disposed at positions where their optical axes 931 and 932 overlap with the dashed line, but when they face the direction of the rotation angle Δθ, it is assumed that the stereo camera has shifted by fc·tan(Δθ) to the right of the paper and by (1-fc·cos(Δθ)) down the paper. At this time, the orientation of the camera 801 and the orientation of the camera 802 are in the same state.

[0047] Considering this, the light ray directions β and γ are calculated from the coordinates of the light point imaged by camera 801 as shown in Fig. 17, and using these values ​​and the rotation angle Δθ, it is possible to obtain the coordinates of the imaged pixels of the light point in the stereo camera after shifting so that the black circles in the figure coincide with the entrance pupils 921, 922 and the optical axes 931, 932 coincide with the thick dotted lines. The shift amount of the stereo camera is minute compared to the calculated object position, so in many cases it can be ignored.

[0048] In this embodiment, even if the rotation axis of the rotation mechanism does not pass through the entrance pupil, it passes through a position equally shifted from the entrance pupil, and by controlling so that the orientations of both cameras are equal, accurate distance measurement can be performed using a coordinate conversion unit similar to that of the first embodiment. Therefore, it is possible to provide a stereo camera that realizes a wide angle of view, long distances, and has highly accurate distance measurement performance, and an object detection system using the stereo camera.

[0049] As described above, the rotation axis is perpendicular to the plane including the optical axis of each camera and passes through each optical axis. Furthermore, the point where each rotation axis intersects with the optical axis can be equally expressed as the distance ΔX from the entrance pupil. The value of ΔX can be reduced by appropriate correction whether it is zero or a non-zero value, but the former allows coordinate conversion with simpler calculations. However, in general, it may be advantageous in terms of mechanical error, stability, etc. to have the rotation axis closer to the center of gravity of the rotated body. From this point of view, it can be said that the range of ΔX is appropriate to be from the imaging surface of the image sensor to the frontmost surface of the lens.

[0050] (III) Supplementary Note The above-described embodiment includes, for example, the following contents.

[0051] In the above embodiment, the present invention has been described as being applied to an object detection system for trains. However, the present invention is not limited to this, and can be widely applied to a method of arranging the cameras and rotation mechanisms in a stereo camera by a worker, a manufacturing machine, or other operator, as well as various other systems, devices, and programs.

[0052] In the above-described embodiment, a part or all of the programs may be installed in the device from a program source. The program source may be, for example, a program distribution server connected via a network or a computer-readable recording medium (for example, a non-transitory recording medium). In the above description, two or more programs may be realized as one program, and one program may be realized as two or more programs.

[0053] The above-described embodiment has the following characteristic configurations, for example.

[0054] (1) A stereo camera (e.g., stereo camera 1300) including a first camera (e.g., camera 801), a first rotation mechanism (e.g., rotation mechanism 811) for rotating the first camera, a second camera (e.g., camera 802), and a second rotation mechanism (e.g., rotation mechanism 812) for rotating the second camera, wherein a first rotation axis (e.g., rotation axis 941) of the first rotation mechanism and a second rotation axis (e.g., rotation axis 942) of the second rotation mechanism are each The first rotation axis is perpendicular to a plane containing both a first optical axis (e.g., optical axis 931) and a second optical axis (e.g., optical axis 932) of the second camera, the first rotation axis intersects with the first optical axis, and the second rotation axis intersects with the second optical axis, and the distance from the intersection of the first rotation axis and the first optical axis to a first entrance pupil position (e.g., entrance pupil 921) of the first camera is equal to the distance from the intersection of the second rotation axis and the second optical axis to a second entrance pupil position (e.g., entrance pupil 922) of the second camera.

[0055] (2) The first rotation axis passes through the first entrance pupil position, and the second rotation axis passes through the second entrance pupil position (see, for example, FIG. 9). In the above configuration, since the rotation axes pass through the entrance pupil position, for example, coordinate conversion can be performed with high accuracy.

[0056] (3) A stereo camera system (e.g., object detection system 800) using the stereo camera includes a correction unit (e.g., coordinate conversion unit 840) that performs correction processing (e.g., coordinate conversion) on at least one of a first image of the first camera or a processed image (e.g., a geometrically corrected image) obtained by applying image processing to the first image (e.g., an image captured by camera 801) based on a rotation angle of rotation angle information sent from a controller unit (e.g., controller unit 820) to the first rotation mechanism and the second rotation mechanism, and performs correction processing (e.g., coordinate conversion) on at least one of a second image of the second camera (e.g., an image captured by camera 801) or a processed image (e.g., a geometrically corrected image) obtained by applying image processing to the second image based on the rotation angle.

[0057] (4) The correction unit includes a geometric correction unit (e.g., geometric correction unit 830) that performs geometric correction on each of the first image and the second image, a coordinate conversion unit (e.g., coordinate conversion unit 840) that performs coordinate conversion of pixel values ​​based on the rotation angle for each of the first geometrically corrected image and the second geometrically corrected image that have been geometrically corrected by the geometric correction unit, and a parallax image generation unit (e.g., parallax image generation unit 850) that compares the first coordinate converted image and the second coordinate converted image that have been coordinate converted by the coordinate conversion unit to generate a parallax image including a parallax value for each pixel value.

[0058] (5) The correction unit performs geometric correction and coordinate conversion of pixel values ​​based on the rotation angle on the first image, and performs geometric correction and coordinate conversion of pixel values ​​based on the rotation angle on the second image.

[0059] (6) The coordinate conversion unit calculates, according to a rotation angle Δθ from a reference angle θ=0, the following: where each pixel value of the image before correction is B(Ub,Vb), a reference pixel of the image before correction is B(Ub0,Vb0), each pixel value of the image after correction is A(Ua,Va), a reference pixel of the image after correction is A(Ua0,Va0), a focal length is fc, and pixel pitches in the x and y directions are px,py. Ua=fc·tan(Δθ+α) / px+Ua0 (Formula 1) Va=Vb-Vb0+Va0 (Formula 2) α=atan((Ub-Ub0)·px / fc) ···(Formula 3) Coordinate transformation is performed according to (see, for example, FIG. 10).

[0060] (7) A rotation angle at which the first optical axis and the second optical axis are perpendicular to a line containing the first entrance pupil position and the second entrance pupil position is set as a reference angle, and the stereo camera system includes a controller unit (e.g., controller unit 820) that sends angle information indicating a rotation angle of a discrete value to the first rotation mechanism and the second rotation mechanism.

[0061] (8) When at least one of the first camera and the second camera is used to acquire a reference image at a reference angle, a first image is acquired at a first rotation angle of the camera, and a second image is acquired at a second rotation angle of the camera, the correction unit calibrates the corrected image of the first image based on the reference image, and calibrates the corrected image of the second image based on the calibrated image of the first image (see, for example, FIG. 13).

[0062] (9) The angle difference between the n-th rotation angle and the (n+1)-th rotation angle at which the images used for the calibration are acquired is equal to or smaller than a half angle of view (see FIG. 13, for example).

[0063] (10) The correction unit uses the overlapping area between the first image and the second image to calibrate the correction equation for each pixel so that the corrected image of the second image matches the corrected image of the first image, and configures the area of ​​the second image other than the overlapping area by extrapolating from the calibration result of the overlapping area (e.g., see FIG. 13).

[0064] (11) The stereo camera system includes a controller unit (e.g., controller unit 820) that rotates the first camera and the second camera in one direction when positioning the first rotation mechanism and the second rotation mechanism at any rotation angle.

[0065] Furthermore, the above-described configurations may be modified, rearranged, combined, or omitted as appropriate without departing from the spirit and scope of the present invention.

[0066] It should be understood that items listed in the format "at least one of A, B, and C" can mean (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). Similarly, items listed in the format "at least one of A, B, or C" can mean (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). [Explanation of symbols]

[0067] 800...object detection system, 801...camera, 802...camera, 811...rotation mechanism, 812...rotation mechanism, 921...entrance pupil (entrance pupil position), 922...entrance pupil (entrance pupil position), 931...optical axis, 932...optical axis, 941...rotation axis, 942...rotation axis.

Claims

1. A stereo camera system using a stereo camera including a first camera, a first rotation mechanism that rotates the first camera, a second camera, and a second rotation mechanism that rotates the second camera, a first rotation axis of the first rotation mechanism and a second rotation axis of the second rotation mechanism are each perpendicular to a plane including both a first optical axis of the first camera and a second optical axis of the second camera; the first rotation axis intersects the first optical axis, and the second rotation axis intersects the second optical axis; a distance from an intersection of the first rotation axis and the first optical axis to a first entrance pupil position of the first camera is equal to a distance from an intersection of the second rotation axis and the second optical axis to a second entrance pupil position of the second camera; a correction unit that performs a correction process on at least one of a first image of the first camera or a processed image obtained by performing image processing on the first image, based on a rotation angle of rotation angle information sent from a controller unit to the first rotation mechanism and the second rotation mechanism, and performs a correction process on at least one of a second image of the second camera or a processed image obtained by performing image processing on the second image, based on the rotation angle; The correction unit is a geometric correction unit that performs geometric correction on each of the first image and the second image; a coordinate conversion unit that performs coordinate conversion of pixel values ​​on each of the first geometrically corrected image and the second geometrically corrected image that have been geometrically corrected by the geometric correction unit, based on the rotation angle; Equipped with The coordinate conversion unit calculates, according to a rotation angle Δθ from a reference angle θ=0, the following: where each pixel value of the image before correction is B (Ub, Vb), a reference pixel of the image before correction is B (Ub0, Vb0), each pixel value of the image after correction is A (Ua, Va), a reference pixel of the image after correction is A (Ua0, Va0), a focal length is fc, and pixel pitches in the x and y directions are px, py: Ua=fc・tan(Δθ+α) / px+Ua0...(Formula 1) Va=Vb-Vb0+Va0...(Formula 2) α=atan((Ub-Ub0)・px / fc)...(Formula 3) Perform coordinate transformation according to Stereo camera system.

2. 2. The stereo camera system according to claim 1, the first axis of rotation passes through the first entrance pupil position and the second axis of rotation passes through the second entrance pupil position. Stereo camera system.

3. 2. The stereo camera system according to claim 1, The correction unit is a parallax image generating unit that compares a first coordinate transformed image, which has been subjected to coordinate transformation by the coordinate transformation unit, with a second coordinate transformed image, to generate a parallax image including a parallax value for each pixel value; Stereo camera system.

4. 2. The stereo camera system according to claim 1, the correction unit performs geometric correction and coordinate conversion of pixel values ​​based on the rotation angle on the first image, and performs geometric correction and coordinate conversion of pixel values ​​based on the rotation angle on the second image. Stereo camera system.

5. 2. The stereo camera system according to claim 1, a rotation angle at which the first optical axis and the second optical axis are perpendicular to a line including the first entrance pupil position and the second entrance pupil position is set as a reference angle; a controller unit that transmits angle information indicating a rotation angle of a discrete value to the first rotation mechanism and the second rotation mechanism; Stereo camera system.

6. 2. The stereo camera system according to claim 1, when at least one of the first camera and the second camera is used to acquire a reference image at a reference angle, a first image is acquired at a first rotation angle of the camera, and a second image is acquired at a second rotation angle of the camera, the correction unit calibrates the corrected image of the first image based on the reference image, and calibrates the corrected image of the second image based on an image after calibration of the first image; Stereo camera system.

7. 7. The stereo camera system according to claim 6, an angle difference between the n-th rotation angle and the (n+1)-th rotation angle for acquiring the image used for the calibration is equal to or smaller than a half angle of view; Stereo camera system.

8. 7. The stereo camera system according to claim 6, the correction unit calibrates a correction equation for each pixel by using an overlapping area between the first image and the second image so that a corrected image of the second image coincides with a corrected image of the first image, and configures an area of ​​the second image other than the overlapping area by extrapolating from the calibration result of the overlapping area. Stereo camera system.

9. 2. The stereo camera system according to claim 1, a controller unit that rotates the first camera and the second camera in one direction when the first rotation mechanism and the second rotation mechanism are positioned at any rotation angle; Stereo camera system.

10. A method for arranging a camera and a rotation mechanism in a stereo camera used in a stereo camera system, comprising: the stereo camera includes a first camera, a first rotation mechanism that rotates the first camera, a second camera, and a second rotation mechanism that rotates the second camera; a first rotation axis of the first rotation mechanism and a second rotation axis of the second rotation mechanism are each perpendicular to a plane including both a first optical axis of the first camera and a second optical axis of the second camera; the first rotation axis intersects the first optical axis, and the second rotation axis intersects the second optical axis; a distance from an intersection of the first rotation axis and the first optical axis to a first entrance pupil position of the first camera is equal to a distance from an intersection of the second rotation axis and the second optical axis to a second entrance pupil position of the second camera; the stereo camera system includes a correction unit that performs a correction process on at least one of a first image of the first camera or a processed image obtained by performing image processing on the first image based on a rotation angle of rotation angle information sent from a controller unit to the first rotation mechanism and the second rotation mechanism, and performs a correction process on at least one of a second image of the second camera or a processed image obtained by performing image processing on the second image based on the rotation angle; The correction unit is a geometric correction unit that performs geometric correction on each of the first image and the second image; a coordinate conversion unit that performs coordinate conversion of pixel values ​​on each of the first geometrically corrected image and the second geometrically corrected image that have been geometrically corrected by the geometric correction unit, based on the rotation angle; Equipped with The coordinate conversion unit calculates, according to a rotation angle Δθ from a reference angle θ=0, the following: where each pixel value of the image before correction is B (Ub, Vb), a reference pixel of the image before correction is B (Ub0, Vb0), each pixel value of the image after correction is A (Ua, Va), a reference pixel of the image after correction is A (Ua0, Va0), a focal length is fc, and pixel pitches in the x and y directions are px, py: Ua=fc・tan(Δθ+α) / px+Ua0...(Formula 1) Va=Vb-Vb0+Va0...(Formula 2) α=atan((Ub-Ub0)・px / fc)...(Formula 3) Perform coordinate transformation according to Placement method.

Images