STEREO CAMERA DEVICE AND CONTROL DEVICE

Abstract

To provide a stereo camera device in which it is possible to eliminate a parallax error that is difficult to predict in advance.SOLUTION: A stereo camera device comprises a stereo camera, a speed sensor, and a control device. The processor of the control device is constituted to execute: detecting a plurality of corresponding points from a first image pair and a second image pair imaged at different times of day by the stereo camera; dividing each of the first image pair and the second image pair into a plurality of regions; calculating, for each of the plurality of regions, a movement speed based on an external parameter using the corresponding points included in each region; and calculating, for each of the plurality of regions, the correction value of parallax so that the difference between the movement speed based on the external parameter and the movement speed detected by the speed sensor is less than a prescribed threshold.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a stereo camera device and a control device. [Background technology]

[0002] A stereo camera device can measure the distance to an object based on images obtained by a pair of cameras. The stereo camera device uses the parallax between the pair of images to calculate the distance to the object. Parallax errors can lead to a decrease in measurement accuracy. Therefore, various techniques have been proposed to improve the accuracy of parallax (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2014-85120 A Summary of the Invention [Problem to be solved by the invention]

[0004] The parallax may include errors due to various factors. For example, when a stereo camera device is installed in a vehicle, errors may include errors that depend on the environment in which the individual stereo camera devices are installed, such as distortion of the windshield. Such errors may be difficult to predict in advance.

[0005] An object of the present invention is to provide a stereo camera device and a control device that can eliminate disparity errors that are difficult to predict in advance. [Means for solving the problem]

[0006] A stereo camera device according to one aspect of the present invention includes: A stereo camera and A speed sensor; A control device; Equipped with the control device includes one or more processors and one or more storage media that store instructions to be executed by the one or more processors; The processor, in accordance with the instructions, Detecting a plurality of corresponding points from a first image pair and a second image pair captured at different times by the stereo camera; Dividing each image of the first image pair and the second image pair into a plurality of regions; calculating a moving speed based on an external parameter for each of the plurality of regions using corresponding points included in each region; calculating a parallax correction value for each of the plurality of regions such that a difference between a moving speed based on the external parameters and a moving speed detected by the speed sensor is less than a predetermined threshold; The apparatus is configured to execute the following steps:

[0007] A stereo camera device according to another aspect of the present invention includes: A stereo camera and A control device; Equipped with the control device includes one or more processors and one or more storage media that store instructions to be executed by the one or more processors; The processor, in accordance with the instructions, Detecting a plurality of corresponding points from a first image pair and a second image pair captured at different times by the stereo camera; Dividing each image of the first image pair and the second image pair into a plurality of regions; selecting a reference region from among the plurality of regions; calculating a moving speed based on an external parameter for each of the plurality of regions using corresponding points included in each region; Calculating an individual parallax correction value for each area other than the reference area so that a difference between the movement speed of the reference area and the movement speed of the other areas other than the reference area is less than a predetermined threshold value; The apparatus is configured to execute the following steps:

[0008] Yet another aspect of the present invention is a control device capable of communicating with a stereo camera and a speed sensor provided in a moving object, The control device includes: one or more processors; one or more storage media that store instructions for execution by the one or more processors; Equipped with The processor, in accordance with the instructions, Detecting a plurality of corresponding points from a first image pair and a second image pair captured at different times by the stereo camera; Dividing each image of the first image pair and the second image pair into a plurality of regions; calculating a moving speed based on an external parameter for each of the plurality of regions using corresponding points included in each region; calculating a parallax correction value for each of the plurality of regions such that a difference between a moving speed based on the external parameters and a moving speed detected by the speed sensor is less than a predetermined threshold; The apparatus is configured to execute the following steps: Effect of the Invention

[0009] According to the present invention, it is possible to eliminate parallax errors that are difficult to predict in advance. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic plan view showing a vehicle equipped with a stereo camera device according to an embodiment. [Diagram 2] FIG. 2 is an example of a first image pair and a second image pair. [Diagram 3] FIG. 3 is a functional block diagram of the ECU. [Figure 4] FIG. 4 is a schematic diagram illustrating an example of calculation of the moving speed based on external parameters. [Diagram 5] FIG. 5 is a flowchart showing an example of the operation of the ECU. [Figure 6] FIG. 6 is a flowchart showing another example of the operation of the ECU. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. The specific dimensions, materials, values, etc. shown in the embodiment are merely examples for easy understanding, and do not limit the present invention unless otherwise specified. In the specification and drawings, elements having substantially the same functions and configurations are given the same reference numerals to avoid repeated explanation. In addition, elements not directly related to the present invention are not shown.

[0012] 1 is a schematic plan view showing a vehicle 500 equipped with a stereo camera device 100 according to an embodiment. In this embodiment, the stereo camera device 100 is applied to the vehicle 500. However, in other embodiments, the stereo camera device 100 may be applied to a moving body other than the vehicle 500, for example, an aircraft such as a helicopter. The stereo camera device 100 measures a distance to an object based on images obtained by a pair of cameras 1L, 1R. The stereo camera device 100 includes a stereo camera 1, a vehicle speed sensor (speed sensor) 2, and an ECU (control device) 50.

[0013] The stereo camera 1 includes a first camera 1L and a second camera 1R. For example, each of the first camera 1L and the second camera 1R may be a CCD (Charge-Coupled Device) camera or a CMOS (Complementary Metal Oxide Semiconductor) camera, and may be a color camera or a monochrome camera. For example, each of the first camera 1L and the second camera 1R is provided in the interior of the vehicle 500 so as to capture an image in front of the vehicle 500. For example, each of the first camera 1L and the second camera 1R may be attached to the bottom surface of the roof of the vehicle 500, slightly spaced from the windshield 9. The first camera 1L and the second camera 1R are arranged to have a predetermined interval in the vehicle width direction.

[0014] Each of the first camera 1L and the second camera 1R captures an image at a predetermined frame rate, for example. Each of the first camera 1L and the second camera 1R is connected to the ECU 50 in a wired or wireless manner so as to be able to communicate with the ECU 50, and transmits the captured image to the ECU 50.

[0015] The vehicle speed sensor 2 detects a speed (moving speed) V0 of the vehicle 500. For example, the vehicle speed sensor 2 detects the speed V0 based on the rotation speed of an output shaft of the vehicle 500. The vehicle speed sensor 2 is communicatively connected to the ECU 50 by wire or wirelessly, and transmits the detected speed V0 to the ECU 50. In another embodiment, for example, when the moving body is an aircraft, the speed sensor may be a sensor that detects the flight speed.

[0016] The ECU (Electronic Control Unit) 50 has one or more processors 51 (e.g., CPU, etc.), one or more storage media 52 (e.g., ROM and RAM, etc.), and one or more connectors 53. The ECU 50 may further have other components. The components of the ECU 50 are communicatively connected to each other by a bus. The storage medium 52 stores one or more programs executed by the processor 51. The programs include instructions for the processor 51. The operation of the ECU 50 shown in the present disclosure is realized by the processor 51 executing the instructions stored in the storage medium 52. The storage medium 52 also stores images received from the first camera 1L and the second camera 1R, and a speed V0 received from the vehicle speed sensor 2. The storage medium 52 may also store an external parameter E, which will be described later, calculated by the processor 51, a speed V1 based on the external parameter E, and parallax correction values ​​θ, θ0, and θ1. The ECU 50 is connected to the first camera 1L, the second camera 1R and the vehicle speed sensor 2 via a connector 53 so as to be able to communicate with each other.

[0017] Next, the images captured by the stereo camera 1 and the operation of the ECU 50 will be described.

[0018] 2 is an example of a first image pair 10 and a second image pair 20. The first image pair 10 and the second image pair 20 are captured at different times by a first camera 1L and a second camera 1R. For example, the first image pair 10 and the second image pair 20 can be images of successive frames.

[0019] Specifically, image 10L of the first image pair 10 is captured by the first camera 1L, and image 10R of the first image pair 10 is captured by the second camera 1R. The images 10L and 10R are captured simultaneously. Also, image 20L of the second image pair 20 is captured by the first camera 1L a predetermined interval after image 10L, and image 20R of the second image pair 20 is captured by the second camera 1R a predetermined interval after image 10R. The images 20L and 20R are captured simultaneously.

[0020] 3 is a functional block diagram of the ECU 50. A processor 51 that processes the images 10L, 10R, 20L, and 20R as described above functions as a detection unit 54, a division unit 55, a selection unit 56, a first calculation unit 57, a second calculation unit 58, and a third calculation unit 59 in accordance with instructions stored in a storage medium 52.

[0021] Returning to FIG. 2, the detection unit 54 detects a plurality of corresponding points P1, P2, P3, P4, and P5 from the images 10L, 10R, 20L, and 20R. Note that the "corresponding points" may also be referred to as "feature points." Although only five corresponding points P1, P2, P3, P4, and P5 are shown in FIG. 2, more corresponding points may be detected from the images 10L, 10R, 20L, and 20R. For example, each of the corresponding points P1, P2, P3, P4, and P5 may be an edge commonly included in the images 10L, 10R, 20L, and 20R. For example, each of the corresponding points P1, P2, P3, P4, and P5 may be detected as a point where the luminance changes.

[0022] For example, the corresponding points P1 and P2 can be included in the contour of a block 31 arranged along the road R. Also, for example, the corresponding point P3 can be a point where two parallel lines L1 and L2 on the road R intersect on the horizontal line H. That is, the corresponding point P3 is a point at infinity. Also, for example, the corresponding point P4 can be included in the contour of a depression on the road R. Also, for example, the corresponding point P5 can be included in the contour of a wall W arranged along the road R. Such corresponding points can be detected by various known stereo matching methods.

[0023] The division unit 55 divides each of the images 10L, 10R, 20L, and 20R into a plurality of regions A1, A2, A3, A4, and A5. Specifically, the division unit 55 divides each of the images 10L, 10R, 20L, and 20R so that each of the regions A1, A2, A3, A4, and A5 includes at least one identical corresponding point P1, P2, P3, P4, and P5 throughout all of the images 10L, 10R, 20L, and 20R. For example, the region A1 includes the corresponding point P1 throughout all of the images 10L, 10R, 20L, and 20R. Furthermore, the region A2 includes the corresponding point P2 throughout all of the images 10L, 10R, 20L, and 20R. Furthermore, the region A3 includes the corresponding point P3 throughout all of the images 10L, 10R, 20L, and 20R. Moreover, the region A4 includes the corresponding point P4 throughout all of the images 10L, 10R, 20L, and 20R, and the region A5 includes the corresponding point P5 throughout all of the images 10L, 10R, 20L, and 20R.

[0024] In Fig. 2, the regions A1, A2, A3, A4, and A5 each include only one identical corresponding point P1, P2, P3, P4, and P5, but each region A1, A2, A3, A4, and A5 may include two or more identical corresponding points. In Fig. 2, each image 10L, 10R, 20L, and 20R is divided into five regions A1, A2, A3, A4, and A5, but in other embodiments, each image 10L, 10R, 20L, and 20R may be divided into a number of regions other than five.

[0025] The selection unit 56 selects one reference region from among the regions A1, A2, A3, A4, and A5. The reference region can be selected based on various concepts. For example, the reference region may be the edge region A1 or A5. Also, for example, the reference region may be a region that includes the point at infinity. Also, for example, the reference region may be a region that is predicted in advance to have a small parallax error.

[0026] The general equations related to measuring distance using a stereo camera are given below:

[0027]

number

[0028]

number

[0029] Where: X, Y, Z: Coordinates of a 3D point in the world coordinate system x,y: Coordinates of the projection point in the camera coordinate system s: coefficient f x ,f y :focal length c x ,c y :principal point r ij :Rotational component t x ,t y ,t z : Translational component E: External parameters

[0030] The first calculation unit 57 calculates the speed (movement speed) V1 of the vehicle 500 based on the above-mentioned external parameter E for each of the regions A1, A2, A3, A4, and A5, using corresponding points P1, P2, P3, P4, and P5 included in each of the regions A1, A2, A3, A4, and A5. For example, the first calculation unit 57 can calculate the external parameter E and the speed V1 by stereo SLAM (Simultaneous Localization and Mapping) using the images 10L, 10R, 20L, and 20R of the stereo camera 1 and monocular SLAM using the images 10L and 20L of a single camera 1L (or the images 10R and 20R of a single camera 1R), for example, an ORB (Oriented Fast and Rotated Brief) SLAM method, a PTAM (Parallel Tracking and Mapping for Small AR Workspaces) method, or a DTAM (Dense Tracking and Mapping in Real-Time) method. The external parameter E and the velocity V1 can be calculated, for example, based on various known solving methods that can be used in a PnP (Perspective-n-points) problem, such as the 8-point method, the 5-point method, or the DLT (Direct Linear Transformation) method.

[0031] Fig. 4 is a schematic diagram for explaining an example of calculation of a velocity V1 based on an external parameter E. In Fig. 4, for example, the velocity V1 is calculated using the corresponding point P3 in Fig. 2. The stereo camera 1 captures images 10L and 10R at time t = t1, and then captures images 20L and 20R at time t = t2.

[0032] For example, the first calculation unit 57 first calculates the extrinsic parameters E for the corresponding point P3 using the images 10R and 20R captured by a single camera 1R (that is, monocular SLAM).

[0033] The external parameter E calculated from the images 10R, 20R captured by the single camera 1R can be considered as the movement vector of the camera 1R from time t1 to time t2, i.e., the movement vector of the vehicle 500 from time t1 to time t2. However, in this case, the external parameter E has a unit length (e.g., 1 km, 1 m, or 1 mm, etc.). Therefore, the pseudo corresponding point P3f calculated using this external parameter E is different from the actual corresponding point P3. Therefore, the pseudo distance Df to the pseudo corresponding point P3f is different from the distance D to the actual corresponding point P3.

[0034] However, the distance D to the actual corresponding point P3 can be calculated based on the principle of triangulation using the images 10L, 10R, 20L, and 20R captured by the pair of cameras 1L and 1R. In this case, the following formula (3) holds between the actual distance D and the pseudo distance Df.

[0035]

number

[0036] The following equation (4) holds between the translation component of the above external parameter E and the movement distance v (km) of the camera 1R from time t1 to time t2, i.e., the movement distance v (km) of the vehicle 500 from time t1 to time t2.

[0037]

number

[0038] Therefore, the speed V1 (km / h) based on the external parameter E is calculated by the following equation (5).

[0039]

number

[0040] Where: n(FPS): Frame rate of stereo camera 1

[0041] According to the above-described method, the first calculation unit 57 uses each of the corresponding points P1, P2, P3, P4, and P5 to calculate the speed V1 of the vehicle 500 based on the external parameter E for each of the areas A1, A2, A3, A4, and A5.

[0042] 1, the speed V1 based on the external parameter E may include an error depending on the environment in which the stereo camera 1 is installed. Such an error may be caused by various factors, such as distortion of the windshield 9 and an assembly error of the first camera 1L and the second camera 1R. In other words, such an error can be removed depending on the environment in which the stereo camera 1 is installed by correcting the parallax in each of the areas A1, A2, A3, A4, and A5 so that the speed V1 based on the external parameter E matches the actual speed V0 of the vehicle 500.

[0043] Based on this idea, the processor 51 calculates the parallax correction values ​​θ1, θ0, and θ. The correction values ​​θ1, θ0, and θ are expressed in terms of the number of pixels.

[0044] For example, the second calculator 58 calculates an individual parallax correction value θ1 for each area other than the reference area so that the speed V1 of the other area other than the reference area coincides with the speed V1 of the reference area. With this configuration, the variation in error in the areas A1, A2, A3, A4, and A5 is reduced.

[0045] The third calculation unit 59 calculates a correction value θ0 for the overall parallax so that the speed V1 based on the external parameters of the reference area matches the speed V0 measured by the vehicle speed sensor 2. For the reference area, the overall correction value θ0 becomes the correction value θ actually used for correcting the parallax (θ=θ0). For each area other than the reference area, the sum of the individual correction value θ1 calculated for each area and the overall correction value θ0 becomes the correction value θ actually used for correcting the parallax (θ=θ0+θ1). The processor 51 calculates the distance to the object using the parallax corrected by the correction value θ.

[0046] In other embodiments, the processor 51 may not calculate the overall correction value θ0.

[0047] For example, as described above, the area A3 including the point at infinity P3 can be selected as the reference area. For example, Japanese Patent Application Laid-Open Publication No. 2017-44573, filed by the same applicant as the present application, discloses a method for correcting disparity using the point at infinity. According to this method, it is possible to remove the error of the disparity of the reference area A3 including the point at infinity P3. In other words, if the disparity of the reference area A3 is corrected by this method, the speed V1 based on the external parameters of the reference area A3 after the correction should be close to the actual speed V0 of the vehicle 500. Therefore, the second calculation unit 58 may calculate individual correction values ​​θ1 for each of the areas A1, A2, A4, and A5 so that the speeds V1 of the areas A1, A2, A4, and A5 other than the reference area A3 match the speed V1 of the reference area A3. In this case, since the speed V1 of the reference area A3 after the correction is close to the actual speed V0 of the vehicle 500, it is not necessary to refer to the speed V0 measured by the vehicle speed sensor 2. Therefore, the processor 51 does not need to calculate the overall correction value θ0. In this case, the individual correction value θ1 calculated for each region becomes the correction value θ actually used for parallax correction (θ=θ1).

[0048] Also, for example, as described above, a region predicted in advance to have a small parallax error can be selected as the reference region. In other words, the speed V1 based on the external parameters of this reference region should be close to the actual speed V0 of the vehicle 500 without any parallax correction. In this case, the second calculation unit 58 may calculate an individual correction value θ1 for each region so that the speed V1 of other regions other than the reference region coincides with the speed V1 of the reference region. In this case, the speed V0 measured by the vehicle speed sensor 2 may not be referred to. Therefore, the processor 51 may not need to calculate an overall correction value θ0. In this case, the individual correction value θ1 calculated for each region becomes the correction value θ actually used for parallax correction (θ=θ1).

[0049] In yet other embodiments, the processor 51 may not select a reference region.

[0050] Specifically, the second calculation unit 58 may calculate an individual correction value θ1 for each of the areas A1, A2, A3, A4, and A5 so that the speed V1 of each of the areas A1, A2, A3, A4, and A5 coincides with the speed V0 measured by the vehicle speed sensor 2. In this case, the processor 51 does not need to select a reference area. In addition, the processor 51 does not need to calculate an overall correction value θ0. In this case, the individual correction value θ1 calculated for each area becomes the correction value θ actually used for correcting the parallax (θ=θ1).

[0051] Next, the operation of the ECU 50 will be described with reference to a flowchart.

[0052] Fig. 5 is a flowchart showing an example of the operation of the ECU 50. In the example of Fig. 5, the processor 51 does not function as the selection unit 56 and the third calculation unit 59. That is, the selection of the reference region is not performed, and the overall correction value θ0 is not calculated.

[0053] For example, the operation shown in FIG. 5 may be performed only once after the start button of the vehicle 500 is pressed and the ECU 50 is started, or may be repeated at a predetermined interval after the ECU 50 is started.

[0054] The processor 51 reads the images 10L, 10R, 20L, and 20R captured by the first camera 1L and the second camera 1R (step S100).

[0055] Next, the processor 51 detects corresponding points P1, P2, P3, P4, and P5 from the images 10L, 10R, 20L, and 20R (step S102).

[0056] Next, the processor 51 divides each of the images 10L, 10R, 20L, and 20R into a plurality of regions A1, A2, A3, A4, and A5 (step S104).

[0057] Next, the processor 51 calculates a velocity V1 based on the external parameter E for each of the regions A1, A2, A3, A4, and A5 using the corresponding points P1, P2, P3, P4, and P5 (step S106).

[0058] Next, the processor 51 calculates a parallax correction value θ for each of the regions A1, A2, A3, A4, and A5 so that the difference between the speed V1 of each of the regions A1, A2, A3, A4, and A5 and the speed V0 measured by the vehicle speed sensor 2 is less than a predetermined threshold (step S108). For example, the processor 51 calculates a parallax correction value θ for each of the regions A1, A2, A3, A4, and A5 so that the speed V1 of each of the regions A1, A2, A3, A4, and A5 matches the speed V0 measured by the vehicle speed sensor 2.

[0059] Next, the processor 51 determines whether the correction value θ for each of the regions A1, A2, A3, A4, and A5 has converged (step S110). For example, the processor 51 may execute step S110 by determining whether the difference between the speed V1 corrected by the correction value θ and the speed V0 measured by the vehicle speed sensor 2 is less than a predetermined threshold value.

[0060] In step S110, when it is determined that the correction value θ of each of the areas A1, A2, A3, A4, and A5 has not converged (NO), the processor 51 repeats steps S106 to S110.

[0061] In step S110, if it is determined that the correction value θ of each of the regions A1, A2, A3, A4, and A5 has converged (YES), the processor 51 stores the calculated correction value θ of each of the regions A1, A2, A3, A4, and A5 in the storage medium 52 (step S112) and ends the series of operations. After that, the processor 51 calculates the distance to the object using the parallax corrected by the correction value θ.

[0062] Fig. 6 is a flowchart showing another example of the operation of the ECU 50. In the example of Fig. 6, the processor 51 functions as a selection unit 56 and a third calculation unit 59. That is, the selection of the reference region is performed, and the overall correction value θ0 is calculated.

[0063] For example, the operation shown in FIG. 6 may be performed only once after the start button of the vehicle 500 is pressed and the ECU 50 is started, or may be repeated at a predetermined interval after the ECU 50 is started.

[0064] The processor 51 reads the images 10L, 10R, 20L, and 20R captured by the first camera 1L and the second camera 1R (step S200).

[0065] Next, the processor 51 detects corresponding points P1, P2, P3, P4, and P5 from the images 10L, 10R, 20L, and 20R (step S202).

[0066] Next, the processor 51 divides each of the images 10L, 10R, 20L, and 20R into a plurality of regions A1, A2, A3, A4, and A5 (step S204).

[0067] Next, the processor 51 selects a reference region from among the multiple regions A1, A2, A3, A4, and A5 (step S206). In this example, for example, the edge region A1 is selected as the reference region.

[0068] Next, the processor 51 calculates a velocity V1 based on the external parameter E for each of the regions A1, A2, A3, A4, and A5 using the corresponding points P1, P2, P3, P4, and P5 (step S208).

[0069] Next, the processor 51 calculates an individual correction value θ1 for each of the regions A2, A3, A4, and A5 so that the difference between the speed V1 of the reference region A1 and the speed V1 of each of the regions A2, A3, A4, and A5 other than the reference region is less than a predetermined threshold (step S210). For example, the processor 51 calculates an individual correction value θ1 for each of the regions A2, A3, A4, and A5 so that the speed V1 of the reference region A1 coincides with the speed V1 of each of the regions A2, A3, A4, and A5 other than the reference region.

[0070] Next, the processor 51 determines whether or not the individual correction value θ1 of each of the regions A2, A3, A4, and A5 has converged (step S212). For example, the processor 51 may execute step S212 by determining whether or not the difference between the speed V1 of each of the regions A2, A3, A4, and A5 after correction by the correction value θ1 and the speed V0 of the reference region A1 is less than a predetermined threshold value.

[0071] In step S212, when it is determined that the individual correction values ​​θ1 of the regions A2, A3, A4, and A5 have not converged (NO), the processor 51 repeats steps S208 to S212.

[0072] If it is determined in step S212 that the individual correction values ​​θ1 for the regions A2, A3, A4, and A5 have converged (YES), the processor 51 calculates an overall correction value θ0 (step S214) so ​​that the difference between the speed V1 in the reference region A1 and the speed V0 measured by the vehicle speed sensor 2 becomes less than a predetermined threshold. For example, the processor 51 calculates the correction value θ0 so that the speed V1 in the reference region A1 coincides with the speed V0 measured by the vehicle speed sensor 2.

[0073] Next, the processor 51 determines whether or not the overall correction value θ0 has converged (step S216). For example, the processor 51 may execute step S216 by determining whether or not the difference between the speed V1 in the reference area A1 after being corrected by the correction value θ0 and the speed V0 measured by the vehicle speed sensor 2 is less than a predetermined threshold value.

[0074] If it is determined in step S216 that the overall correction value θ0 has not converged (NO), the processor 51 repeats steps S208 to S216.

[0075] If it is determined in step S216 that the overall correction value θ0 has converged (YES), the processor 51 stores θ=θ0 for the reference area A1 and θ=θ0+θ1 for the areas A2, A3, A4, and A5 other than the reference area in the storage medium 52 (step S218), and ends the series of operations. After that, the processor 51 calculates the distance to the object using the parallax corrected by the correction value θ.

[0076] The stereo camera device 100 as described above includes the stereo camera 1, the vehicle speed sensor 2, and the ECU 50. The ECU 50 includes one or more processors 51 and one or more storage media 52 that store instructions executed by the one or more processors 51. In the example of FIG. 5, the processor 51 is configured to execute the following operations according to the instruction: detect a plurality of corresponding points P1, P2, P3, P4, and P5 from a first image pair 10 and a second image pair 20 captured by the stereo camera 1 at different times t1 and t2; divide each of images 10L, 10R, 20L, and 20R of the first image pair 10 and the second image pair 20 into a plurality of regions A1, A2, A3, A4, and A5; calculate a speed V1 based on an external parameter E for each of the plurality of regions A1, A2, A3, A4, and A5 using the corresponding points P1, P2, P3, P4, and P5 included in each of the plurality of regions; and calculate a parallax correction value θ for each of the plurality of regions A1, A2, A3, A4, and A5 such that a difference between the speed V1 based on the external parameter E and a speed V0 detected by the vehicle speed sensor 2 is less than a threshold value. According to this configuration, the correction value θ for each of the areas A1, A2, A3, A4, and A5 in the image can be calculated according to the state of each of the areas A1, A2, A3, A4, and A5 in the environment in which the stereo camera 1 is actually installed. Therefore, the parallax error, which is difficult to predict in advance, can be removed according to the environment in which the stereo camera 1 is actually installed.

[0077] 6, the processor 51 is configured to select the reference area A1 from among the multiple areas A1, A2, A3, A4, and A5, and to calculate an individual parallax correction value θ1 for each area A2, A3, A4, and A5 so that the difference between the speed V1 of the reference area A1 and the speed V1 of the areas A2, A3, A4, and A5 other than the reference area A1 is less than a predetermined threshold. With this configuration, the variation in error in the areas A1, A2, A3, A4, and A5 is reduced. Also, with this configuration, parallax errors that are difficult to predict in advance can be removed according to the environment in which the stereo camera 1 is actually installed.

[0078] 6, the processor 51 is further configured to calculate an overall parallax correction value θ0 common to the multiple areas A1, A2, A3, A4, and A5 such that a difference between a speed V1 based on the external parameter E of the reference area A1 and a speed V0 measured by the vehicle speed sensor 2 is less than a threshold value. With this configuration, the processor 51 can use the sum of the individual parallax correction value θ1 and the overall parallax correction value θ0 as the correction value θ actually used for parallax correction.

[0079] Although the embodiment has been described above with reference to the accompanying drawings, the present invention is not limited to such an embodiment. It is clear that a person skilled in the art can conceive of various modifications or alterations within the scope of the claims, and it is understood that such modifications and alterations are also within the technical scope of the present invention. In addition, the steps of the ECU 50 in the above embodiment do not have to be performed in the above order, and may be performed in a different order as long as no technical contradiction occurs. [Explanation of symbols]

[0080] 1 Stereo camera 1L First camera 1R 2nd camera 2 Vehicle speed sensor (speed sensor) 10 First Image Pair 10L Images 10R Image 20 Second Image Pair 20L Images 20R Image 51 Processors 52 Storage medium 100 Stereo camera device A1~A5 area P1~P5 corresponding points

Claims

1. A stereo camera and A speed sensor; A control device; Equipped with the control device includes one or more processors and one or more storage media that store instructions executed by the one or more processors; The processor, in accordance with the instructions, Detecting a plurality of corresponding points from a first image pair and a second image pair captured by the stereo camera at different times; Dividing each image of the first image pair and the second image pair into a plurality of regions; calculating a moving speed based on an external parameter for each of the plurality of regions using corresponding points included in each region; calculating a parallax correction value for each of the plurality of regions such that a difference between a moving speed based on the external parameters and a moving speed detected by the speed sensor is less than a predetermined threshold; A stereo camera device configured to execute the steps described above.

2. A stereo camera and A control device; Equipped with the control device includes one or more processors and one or more storage media that store instructions executed by the one or more processors; The processor, in accordance with the instructions, Detecting a plurality of corresponding points from a first image pair and a second image pair captured by the stereo camera at different times; Dividing each image of the first image pair and the second image pair into a plurality of regions; selecting a reference region from among the plurality of regions; calculating a moving speed based on an external parameter for each of the plurality of regions using corresponding points included in each region; Calculating an individual parallax correction value for each area other than the reference area so that a difference between the movement speed of the reference area and the movement speed of the other areas other than the reference area is less than a predetermined threshold value; A stereo camera device configured to execute the steps described above.

3. The stereo camera device further includes a speed sensor. The processor, in accordance with the instructions, calculating a correction value of an overall disparity common to the plurality of regions such that a difference between the moving speed of the reference region and the moving speed measured by the speed sensor is less than a predetermined threshold value; The stereo camera device according to claim 2 , further configured to execute:

4. A control device capable of communicating with a stereo camera and a speed sensor provided in a moving object, The control device includes: one or more processors; one or more storage media that store instructions for execution by the one or more processors; Equipped with The processor, in accordance with the instructions, Detecting a plurality of corresponding points from a first image pair and a second image pair captured by the stereo camera at different times; Dividing each image of the first image pair and the second image pair into a plurality of regions; calculating a moving speed based on an external parameter for each of the plurality of regions using corresponding points included in each region; calculating a parallax correction value for each of the plurality of regions such that a difference between a moving speed based on the external parameters and a moving speed detected by the speed sensor is less than a predetermined threshold; A control device configured to execute the steps of:

Images