Object detection method and object detection device

Abstract

To correctly detect an object by accurately performing clustering even in a region where the accuracy of position detection by a stereo camera decreases.SOLUTION: An object detection method acquires two images captured at different positions from an own vehicle toward a prescribed direction, extracts a plurality of feature points from each of the two images, sets a first weight applied to such a first possibility that the plurality of feature points calculated on the basis of a distance between the plurality of feature points in at least a real space are the feature points of the same object, and a second weight applied to such a second possibility that the plurality of feature points calculated on the basis of the distance between the plurality of feature points in the image and a movement speed of each feature point are the feature points of the same object on the basis of the distance in a prescribed direction from the own vehicle to the plurality of feature points, and performs clustering on the plurality of feature points to detect the object by using the first possibility applied with the first weight and the second possibility applied with the second weight.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to an object detection method and an object detection device. [Background technology]

[0002] Conventionally, there has been known an object detection method that extracts candidate points (feature points) of an object from images captured by a stereo camera, and detects the object by clustering the extracted candidate points (Patent Document 1). The object detection method disclosed in Patent Document 1 acquires coordinates of candidate points of an object from two images captured by a stereo camera, votes each candidate point of the object on a voting plane based on the acquired coordinates, and performs clustering by grouping the candidate points of the object into adjacent groups on the voting plane. The clustering described in Patent Document 1 uses a shortest distance method using width and depth as feature quantities. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2006-53756 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the accuracy of object position detection by a stereo camera decreases as the distance between the stereo camera and the object increases. Therefore, the object detection method disclosed in Patent Document 1 may not be able to cluster candidate points of a detected object in an area where the accuracy of position detection by the stereo camera decreases, or may perform erroneous clustering, and may not be able to accurately detect the object.

[0005] The present invention has been made in consideration of the above-mentioned problems, and aims to provide an object detection method and an object detection device that can perform accurate clustering even in areas where the accuracy of position detection by a stereo camera decreases, and can accurately detect objects. [Means for solving the problem]

[0006] According to an embodiment of the present invention, an object detection method obtains two images by capturing images from different positions in a predetermined direction from a vehicle, and extracts a plurality of feature points from each of the two images. From the positions of the plurality of feature points on the two images, the distances in the predetermined direction from the vehicle to the plurality of feature points and the distances between the plurality of feature points in real space are calculated. A first weight is assigned to a first possibility indicating that the plurality of feature points calculated based on at least the distances between the plurality of feature points in real space are feature points of the same object, and a second weight is assigned to a second possibility indicating that the plurality of feature points calculated based on the distances between the plurality of feature points in the image and the moving speed of each of the feature points are feature points of the same object, based on the distances in the predetermined direction from the vehicle to the plurality of feature points. The plurality of feature points are clustered using the first possibility to which the first weight is assigned and the second possibility to which the second weight is assigned. Effect of the Invention

[0007] According to the present invention, it is possible to perform clustering with high accuracy even in an area where the accuracy of position detection by a stereo camera decreases, and it is possible to accurately detect an object. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing the configuration of an object detection device according to an embodiment. [Diagram 2] FIG. 2 is a schematic diagram showing a theoretical error that occurs when the stereo camera 10 measures the position of an object. [Figure 3A] FIG. 3A is a schematic diagram showing first weights and second weights that are set based on distances in the z direction from the host vehicle to multiple feature points (first example). [Figure 3B]FIG. 3B is a schematic diagram showing first weights and second weights that are set based on the distances in the z direction from the host vehicle to multiple feature points (second example). [Figure 3C] FIG. 3C is a schematic diagram showing first weights and second weights that are set based on distances in the z direction from the host vehicle to multiple feature points (third example). [Figure 3D] FIG. 3D is a schematic diagram showing first weights and second weights that are set based on distances in the z direction from the host vehicle to multiple feature points (fourth example). [Figure 3E] FIG. 3E is a schematic diagram showing first weights and second weights that are set based on distances in the x direction from the host vehicle to multiple feature points. [Figure 4A] FIG. 4A is a schematic diagram showing filtering according to an embodiment (part 1). [Figure 4B] FIG. 4B is a schematic diagram showing filtering according to the embodiment (part 2). [Diagram 5] FIG. 5 is a diagram illustrating a match cost matrix according to the embodiment. [Figure 6A] FIG. 6A is a schematic diagram showing a first possibility-based clustering result. [Figure 6B] FIG. 6B is a schematic diagram showing the clustering result based on the second possibility. [Figure 7] FIG. 7 is a flowchart illustrating an example of the operation of the object detection device according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The embodiments will be described with reference to the drawings. In the description of the drawings, the same parts are given the same reference numerals and the description will be omitted.

[0010] [Configuration of object detection device] The configuration of an object detection device 1 according to an embodiment will be described with reference to Fig. 1. The object detection device 1 includes a stereo camera 10 and a control unit 20. The object detection device 1 can be used for object detection in, for example, Advanced Driver-Assistance Systems (ADAS). In this embodiment, the object detection device 1 is mounted on a vehicle and detects an object present in front of the vehicle.

[0011] The stereo camera 10 includes a camera 11 and a camera 12. The camera 11 and the camera 12 are disposed at different positions in the housing of the stereo camera 10. The stereo camera 10 is disposed in the front of the vehicle, and the camera 11 and the camera 12 capture images of the front of the vehicle from different positions to obtain two images. Specifically, when the stereo camera 10 is installed on the vehicle, the camera 11 is disposed on the right side of the vehicle, and the camera 12 is disposed on the left side of the vehicle. The cameras 11 and 12 capture images at the same timing. The camera 11 captures a first image of the front of the vehicle from the right side of the vehicle, and the camera 12 captures a second image of the front of the vehicle from the left side of the vehicle. The stereo camera 10 repeatedly captures images of the front of the vehicle at a predetermined cycle, obtains two images of the front of the vehicle captured at the same timing, and transmits the obtained images to the control unit 20.

[0012] In this embodiment, the stereo camera 10 (camera 11 and camera 12) is a camera that captures an image in front of the vehicle, but is not limited to this. The stereo camera 10 may be a camera that captures an image on the side of the vehicle. The cameras 11 and 12 are arranged at different positions in the left-right direction of the vehicle, but if the stereo camera 10 is a camera that captures an image on the side of the vehicle, the cameras are arranged at different positions in the front-rear direction of the vehicle. That is, the stereo camera 10 may be a camera that captures an image in a predetermined direction from the vehicle, and the cameras 11 and 12 may be arranged at different positions in a direction having an angle with respect to a predetermined direction that is the imaging direction of the stereo camera 10. In the following, an example will be described in which the predetermined direction that is the imaging direction of the stereo camera 10 is the front-rear direction, and the different positions at which the cameras 11 and 12 are arranged are different positions in the left-right direction of the vehicle.

[0013] In the present embodiment, the stereo camera 10 is exemplified as a stereo camera having two cameras, camera 11 and camera 12, arranged at different positions, but is not limited thereto. For example, the stereo camera 10 may have three or more cameras arranged at different positions. Also, the stereo camera 10 may be a so-called motion stereo camera having only one camera and acquiring two images captured by one camera at different times during the movement of the vehicle, that is, a plurality of images captured by one camera at different positions. In other words, the stereo camera 10 is not limited in the number of cameras, and may be a camera that acquires a plurality of images captured at different positions. Hereinafter, in the present embodiment, for the sake of simplicity, the stereo camera 10 will be described as a stereo camera having two cameras, camera 11 and camera 12, arranged at different positions.

[0014] The control unit 20 detects an object in front of the vehicle by extracting a plurality of feature points from each of the two images acquired by the stereo camera 10 and clustering the extracted feature points. In this embodiment, the object may be a moving object including another vehicle, a motorcycle, a bicycle, and a pedestrian, or a stationary object including a parked vehicle. The control unit 20 is a general-purpose microcomputer equipped with a CPU (Central Processing Unit), memories (storage units) such as RAM and ROM, and an input / output unit. A computer program for functioning as an object detection device is installed in the microcomputer. By executing the computer program, the microcomputer functions as a plurality of information processing circuits (21, 22, 23, 24, 25, 26) included in the object detection device. Note that in this embodiment, an example is shown in which the plurality of information processing circuits (21, 22, 23, 24, 25, 26) included in the object detection device are realized by software, but it is also possible to configure the information processing circuits by preparing dedicated hardware for executing each information processing. In addition, the plurality of information processing circuits may be configured by individual hardware.

[0015] The control unit 20 includes a feature point extraction unit 21, a distance image generation unit 22, an optical flow calculation unit 23, a vehicle behavior measurement unit 24, a three-dimensional flow calculation unit 25, and an object detection unit 26 as a plurality of information processing circuits.

[0016] The feature point extraction unit 21 extracts a plurality of feature points, which are pixels having characteristics distinguishable from surrounding pixels, from each of the images acquired by the camera 11 and the camera 12. A known method can be applied to extract the feature points, and for example, the method described in the non-patent document "Jianbo Shi and Carlo Tomasi, "Good Features to Track," 1994 IEEE Conference on Computer Vision and Pattern Recognition (CVPR'94), 1994, pp.593-600." is used. The feature point extraction unit 21 outputs the extracted plurality of feature points to the distance image generation unit 22 and the optical flow calculation unit 23. In the following, unless otherwise specified, it is assumed that there are a plurality of feature points.

[0017] The distance image generating unit 22 calculates the distances from the host vehicle to the multiple feature points in the forward / rearward direction of the host vehicle and the distances between the multiple feature points in real space, and generates a distance image displaying the relative positions of the multiple feature points with respect to the host vehicle. Specifically, the distance image generating unit 22 calculates the three-dimensional coordinates (relative coordinates) of the multiple feature points with respect to the host vehicle from the parallax (shift amount) for the same feature point among the feature points of the first image and the feature points of the second image. Next, the distance image generating unit 22 calculates the distances from the host vehicle to the multiple feature points in the forward / rearward direction of the host vehicle and the distances between the multiple feature points in real space from the three-dimensional coordinates of the multiple feature points. Note that the distance image generating unit 22 may calculate the distances from the host vehicle to the multiple feature points in the forward / rearward direction of the host vehicle from the parallax between the feature points extracted from the first image acquired by the camera 11 and the feature points extracted from the second image acquired by the camera 12. Distance image generation unit 22 also calculates the angle that the feature point makes with respect to the straight-ahead direction (predetermined direction) of the vehicle from the three-dimensional coordinates of the multiple feature points with respect to the vehicle. Next, distance image generation unit 22 arranges each feature point on a projection plane based on the three-dimensional coordinates of the multiple feature points, and generates a distance image showing the distance from the vehicle to the multiple feature points in the front-rear direction of the vehicle. Distance image generation unit 22 outputs the generated distance image to three-dimensional information evaluation unit 28 and evaluation method determination unit 29.

[0018] The optical flow calculation unit 23 calculates an optical flow from the positions of feature points in a past image of either the camera 11 or the camera 12 extracted by the feature point extraction unit 21 and the positions of feature points in a current image. The optical flow is a vector that represents the two-dimensional movement (movement speed) of feature points on an image. The optical flow calculation unit 23 detects, as related feature points, current feature points that correspond to the same object in real space as the object corresponding to the past feature point. The optical flow calculation unit 23 calculates a combination of past feature points and current feature points that are related to each other as an optical flow. This allows the optical flow calculation unit 23 to calculate the distance between multiple feature points in an image and the movement speed of each feature point. A known method can be applied to calculate the optical flow, for example, the method described in the non-patent document "Bruce D. Lucasand Takeo Kanade. An Iterative Image Registration Technique with an Application to Stereo Vision. International Joint Conference on Artificial Intelligence, pages 674-679, 1981" is used. The optical flow calculation unit 23 outputs the calculated optical flow to the three-dimensional flow calculation unit 25 and the two-dimensional information evaluation unit 27.

[0019] The vehicle behavior measurement unit 24 measures the behavior of the vehicle. Specifically, the vehicle behavior measurement unit 24 measures the behavior of the vehicle by performing odometry and dead reckoning. The behavior of the vehicle refers to the change in the relative position, relative attitude, and speed of the vehicle at a predetermined time. The vehicle behavior measurement unit 24 can measure the behavior of the vehicle at a predetermined time by measuring the relative position, relative attitude, and speed of the vehicle with respect to a predetermined reference point based on information acquired from a wheel speed sensor that detects the wheel speed of each wheel of the vehicle and a steering angle sensor that detects the steering angle of the steering wheel. The vehicle behavior measurement unit 24 may acquire the absolute position and absolute attitude of the vehicle in the earth coordinates from a receiving device (not shown) that receives a GPS (Global Positioning System) signal, and measure the behavior of the vehicle from the amount of change in each at a predetermined time.

[0020] The three-dimensional flow calculation unit 25 calculates the three-dimensional movement (flow) of the feature points in real space. Specifically, first, the three-dimensional flow calculation unit 25 converts the optical flow calculated by the optical flow calculation unit 23 into a movement in real space. As described above, the optical flow is a two-dimensional movement of the feature points on the image. Therefore, the three-dimensional flow calculation unit 25 converts the two-dimensional movement of the multiple feature points on the image into a two-dimensional movement (velocity) in real space. Next, the three-dimensional flow calculation unit 25 adds the longitudinal movement with respect to the host vehicle to the calculated two-dimensional movement of the multiple feature points in real space. The longitudinal movement of the multiple feature points with respect to the host vehicle can be calculated from the change per unit time in the longitudinal distance from the host vehicle to the multiple feature points calculated by the distance image generation unit 22. Note that the three-dimensional movement of the multiple feature points calculated up to this point is a relative movement with respect to the host vehicle. Therefore, the three-dimensional flow calculation unit 25 corrects the relative movements of the multiple feature points with respect to the host vehicle using the behavior of the host vehicle calculated by the host vehicle behavior measurement unit 24, and calculates three-dimensional movements (absolute values) in the real space. The three-dimensional flow calculation unit 25 outputs the calculated three-dimensional movements (three-dimensional flows) of the feature points in the real space to the three-dimensional information evaluation unit .

[0021] The object detection unit 26 clusters the feature points and detects the clustered feature points as one object. Specifically, the object detection unit 26 sets a first weight to be assigned to a first possibility calculated based on at least the distance between the feature points in real space, the first possibility indicating the possibility that the feature points are feature points of the same object. The object detection unit 26 also sets a second weight to be assigned to a second possibility calculated based on the distance between the feature points in the image and the moving speed of each feature point, the second possibility indicating the possibility that the feature points are feature points of the same object. Specifically, the object detection unit 26 sets the first weight and the second weight according to the distance from the vehicle to the feature points in the front-rear direction of the vehicle and the angle that the feature points make with respect to the straight direction of the vehicle. Then, the object detection unit 26 clusters the feature points using the first possibility to which the first weight is assigned and the second possibility to which the second weight is assigned, to detect an object.

[0022] The first possibility is calculated based on at least the distance between a plurality of feature points in real space. Therefore, the first possibility takes into consideration the three-dimensional positional relationship between the feature points. Therefore, the first possibility is effective in an area where the accuracy of position detection by the stereo camera 10 is good. The second possibility is calculated based on the distance between a plurality of feature points in the image and the moving speed of each of the feature points. Therefore, the second possibility takes into consideration the two-dimensional positional relationship between the feature points and the moving speed of the feature points in the two-dimensional space. That is, the second possibility is effective in an area where the accuracy of position detection by the stereo camera 10 is reduced by not taking into consideration the distance between the feature points in the front-rear direction (depth direction) of the host vehicle.

[0023] Here, referring to FIG. 2, a theoretical error that occurs when calculating the distance in the front-rear direction of the host vehicle 2 from the host vehicle 2 to the multiple feature points (i1, i2, i3, i4) and the position in the left-right direction of the host vehicle 2 of each feature point from two images acquired by the stereo camera 10 will be described. In this embodiment, the distance image generating unit 22 calculates the distance in the front-rear direction of the host vehicle 2 from the multiple feature points (i1, i2, i3, i4) and the three-dimensional coordinates (relative coordinates) of the multiple feature points with respect to the host vehicle 2, but the same applies when the stereo camera 10 calculates the distance to an object. Here, the description is based on the premise that the stereo camera 10 calculates the distance in the front-rear direction of the host vehicle 2 from the host vehicle 2 to the multiple feature points (i1, i2, i3, i4) and the position in the left-right direction (x direction) of each feature point. The theoretical error that occurs when the stereo camera 10 calculates the distance in the front-rear direction of the host vehicle 2 from the host vehicle 2 to the feature point i1 is tez1. Also, the theoretical error that occurs when the stereo camera 10 calculates the distance in the front-rear direction of the host vehicle 2 from the host vehicle 2 to the feature point i2 that is farther away than the feature point i1 is tez2. In this way, when the stereo camera 10 calculates the distance in the front-rear direction of the host vehicle 2 from the host vehicle 2 to the feature point, the greater the distance in the front-rear direction of the host vehicle 2 from the stereo camera 10 to the feature point, the greater the theoretical error. In addition, in the case of feature points that have an angle with respect to the straight-ahead direction (z direction) of the host vehicle, such as feature points i3 and i4, the greater the angle that the feature point makes with respect to the straight-ahead direction of the host vehicle (θ1<θ2), the greater the theoretical error when calculating the position in the left-right direction (x direction) of the host vehicle. Therefore, the theoretical error at the position (x1) in the x direction of the feature point i3 is tex1, whereas the theoretical error at the position (x2) in the x direction of the feature point i4 is tex2. Therefore, when the feature point i4 is on the broken line l, the feature point i4 is calculated as the same x coordinate.

[0024] The object detection unit 26 includes a two-dimensional information evaluation unit 27, a three-dimensional information evaluation unit 28, an evaluation method determination unit 29, and a clustering unit 30. Specific processing of each element constituting the object detection unit 26 will be described below.

[0025] The evaluation method determination unit 29 sets a first weight to be assigned (multiplied) to the first possibility and a second weight to be assigned (multiplied) to the second possibility based on the distance from the host vehicle to the multiple feature points in the front-rear direction (z direction) of the host vehicle. Here, the process of the evaluation method determination unit 29 setting the first weight w1 and the second weight w2 based on the distance from the host vehicle to the multiple feature points in the front-rear direction (z direction) of the host vehicle will be specifically described with reference to Figures 3A to 3D.

[0026] (First example) FIG. 3A shows a first example in which the evaluation method determination unit 29 sets the first weight w1 and the second weight w2. In the first example, the evaluation method determination unit 29 judges whether or not the distance from the host vehicle to the multiple feature points in the longitudinal direction (z direction) of the host vehicle is equal to or greater than a predetermined distance zc. Then, when the evaluation method determination unit 29 judges that the distance from the host vehicle to the multiple feature points in the longitudinal direction of the host vehicle is equal to or greater than the predetermined distance zc, the evaluation method determination unit 29 sets the first weight w1 to 0.0 and the second weight w2 to 1.0. Note that 1.0 is the maximum value of the weights that can be set. That is, when the distance from the host vehicle to the multiple feature points in the longitudinal direction of the host vehicle is equal to or greater than the predetermined distance zc, the evaluation method determination unit 29 sets the weights to be determined to be equal to or greater than the second possibility, using only the second possibility. Furthermore, when the evaluation method determination unit 29 determines that the distance from the host vehicle to the multiple feature points in the longitudinal direction of the host vehicle is less than the predetermined distance zc, it sets the first weight w1 to 1.0 and the second weight w2 to 0.0. That is, when the distance from the host vehicle to the multiple feature points in the longitudinal direction of the host vehicle is less than the predetermined distance zc, the evaluation method determination unit 29 sets the weights to determine whether the multiple feature points are feature points of the same object using only the first possibility. The predetermined distance zc is a distance (e.g., 50 m) at which the accuracy of position detection by the stereo camera 10 decreases. In this way, in the first example, the evaluation method determination unit 29 discontinuously changes the first weight w1 and the second weight w2.

[0027] (Second example) 3B shows a second example in which the evaluation method determination unit 29 sets the first weight w1 and the second weight w2. The difference from the first example is that when the evaluation method determination unit 29 determines that the distance from the host vehicle to the multiple feature points in the front-rear direction of the host vehicle is equal to or greater than a predetermined distance zc, the evaluation method determination unit 29 sets the first weight w1 to 0.1 and the second weight w2 to 0.9. This allows the first possibility to be taken into consideration even in an area where the accuracy of position detection by the stereo camera 10 decreases, and it is possible to prevent feature points that are far apart in real space and are not feature points of the same object from being determined to be feature points of the same object.

[0028] (Third example) FIG. 3C shows a third example in which the evaluation method determination unit 29 sets the first weight w1 and the second weight w2. In the third example, when the distance from the host vehicle to the multiple feature points in the longitudinal direction of the host vehicle is less than a predetermined distance z1, the evaluation method determination unit 29 sets the first weight w1 to 1.0 and the second weight w2 to 0.0. When the distance from the host vehicle to the multiple feature points in the longitudinal direction of the host vehicle is equal to or greater than the predetermined distance z1 and less than the predetermined distance z2, the evaluation method determination unit 29 continuously changes the first weight w1 and the second weight w2 based on the distance from the host vehicle to the multiple feature points in the longitudinal direction of the host vehicle. More specifically, the farther the distance from the host vehicle to the multiple feature points in the longitudinal direction of the host vehicle, the more the first weight w1 is decreased from 1.0 and the second weight w2 is increased from 0.0. When the distance from the host vehicle to the multiple feature points in the longitudinal direction of the host vehicle is equal to or greater than a predetermined distance z2, the evaluation method determination unit 29 sets the first weight w1 to 0.0 and the second weight w2 to 1.0. The predetermined distance z1 is a distance at which the theoretical error when calculating the distance from the host vehicle to the multiple feature points in the longitudinal direction of the host vehicle is 2 m. The predetermined distance z1 is, for example, 30 m. The predetermined distance z2 is a distance at which the theoretical error when calculating the distance from the host vehicle to the multiple feature points in the longitudinal direction of the host vehicle is 10 m. The predetermined distance z2 is, for example, 70 m. In this way, in the third example, the evaluation method determination unit 29 continuously changes the first weight w1 and the second weight w2 between the predetermined distance z1 and the predetermined distance z2.

[0029] (Example 4) 3D shows a fourth example in which the evaluation method determination unit 29 sets the first weight w1 and the second weight w2. In the fourth example, when the distance from the host vehicle to the multiple feature points in the longitudinal direction of the host vehicle is less than a predetermined distance z3, the evaluation method determination unit 29 sets the first weight w1 to 1.0 and the second weight w2 to 0.0. The evaluation method determination unit 29 sets the value obtained by dividing the theoretical error in the longitudinal direction of the host vehicle at the predetermined distance z3 by the theoretical error in the distance from the host vehicle to the multiple feature points in the longitudinal direction of the host vehicle as the first weight w1. The second weight w2 is set to a value obtained by subtracting the first weight w1 from 1.0.

[0030] The evaluation method determination unit 29 sets a first weight w1 to be assigned to the first possibility and a second weight w2 to be assigned to the second possibility based on the angle that the feature point makes with respect to the straight-ahead direction of the vehicle. Here, referring to FIG. 3E, a process in which the evaluation method determination unit 29 sets the first weight w1 and the second weight w2 based on the angle that the feature point makes with respect to the straight-ahead direction of the vehicle will be specifically described. As described with reference to FIG. 2, in the case of a feature point having an angle with respect to the straight-ahead direction (z direction) of the vehicle, the greater the angle that the feature point makes with the straight-ahead direction of the vehicle, the greater the theoretical error in calculating the position in the left-right direction (x direction) of the vehicle. Therefore, the evaluation method determination unit 29 reduces the first weight w1 and increases the second weight w2 in the region r1 where the angle that the feature point makes with respect to the straight-ahead direction of the vehicle is larger than in the region r0. Similarly, in region r3 where the angle that the feature point makes with respect to the straight-ahead direction of the host vehicle is larger, the first weight w1 is decreased and the second weight w2 is increased compared to region r2. As described above, the evaluation method determination unit 29 sets the first weight w1 assigned to the first possibility and the second weight w2 assigned to the second possibility based on the distance from the host vehicle to the feature point in the front-rear direction of the host vehicle and the angle that the feature point makes with respect to the straight-ahead direction of the host vehicle.

[0031] The three-dimensional information evaluation unit 28 filters the feature points, and calculates a first possibility indicating the possibility that the first weighted feature points are feature points of the same object based on the distance between the feature points in the real space after filtering and the moving speed of each feature point in the real space. Here, the filtering of the feature points will be specifically described with reference to Figs. 4A and 4B. The filtering of the feature points is a process that prohibits clustering of feature points whose distance between the feature points in the real space is longer than a predetermined allowable distance as feature points of the same object, taking into account a theoretical error that occurs when calculating the distance from the vehicle to the feature points in the front-rear direction of the vehicle.

[0032] FIG. 4A shows feature point i, feature point j, theoretical error te that occurs when calculating the distance from the vehicle to feature point i and the distance from the vehicle to feature point j in the front-rear direction (z direction) of the vehicle, and a predetermined allowable distance pd. In FIG. 4A and FIG. 4B, for convenience, the theoretical error te of feature point i and feature point j is the same value. The three-dimensional information evaluation unit 28 judges whether or not the distance d1 between feature point i and feature point j in the real space considering the theoretical error te is within a predetermined allowable distance pd. The distance d1 between multiple feature points in the real space considering the theoretical error is the distance at which the distance between feature point i and feature point j is the shortest when the theoretical error te is considered. In FIG. 4A, the distance d1 between multiple feature points in the real space considering the theoretical error is longer than the predetermined allowable distance pd. Therefore, the three-dimensional information evaluation unit 28 prohibits clustering of feature point i and feature point j as feature points of the same object. More specifically, the calculation of the possibility 1 indicating the possibility that feature point i and feature point j are feature points of the same object is not performed. In contrast to Fig. 4A, the distance d1 between multiple feature points in real space taking into account the theoretical error te between feature point i and feature point j in Fig. 4B is within a predetermined allowable distance pd. Therefore, the three-dimensional information evaluation unit 28 calculates a first possibility that indicates the possibility that feature point i and feature point j are feature points of the same object.

[0033] The three-dimensional information evaluation unit 28 calculates a first possibility indicating the possibility that the first weighted feature points are feature points of the same object, based on the distance between the feature points in real space after filtering and the moving speed of each feature point in real space. Specifically, the first possibility is calculated by the following formula (1). Note that the first possibility f3d(i, j) in the following formula (1) is calculated as a smaller value as the possibility that feature point i and feature point j are feature points of the same object increases.

[0034]

number

[0035] Xi, Yi, Zi are the coordinates of feature point i in the real space, and vXi, vYi, vZi are the velocities of feature point i in the x-, y-, and z-directions in the real space. Xj, Yj, Zj are the coordinates of feature point j in the real space, and vXj, vYj, vZj are the velocities of feature point j in the x-, y-, and z-directions in the real space. a to f are normalization parameters for stabilizing the theoretical error included in the evaluation formula. The three-dimensional information evaluation unit 28 outputs the result of applying a first weight to the first possibility calculated by formula (1) to the clustering unit 30.

[0036] The two-dimensional information evaluation unit 27 calculates a second possibility indicating the possibility that the feature points with the second weighting are feature points of the same object, based on the distance between the feature points and the moving speed of each feature point. Specifically, the second possibility is calculated by the following formula (2). Note that the second possibility f2d(i, j) in the following formula (2) is calculated as a smaller value as the possibility that the feature point i and the feature point j are feature points of the same object increases.

[0037]

number

[0038] where xi, yi are the coordinates of feature point i in the image, and vxi, vvy are the moving speeds of feature point i in the x and y directions in the image. xj, yj are the coordinates of feature point j in the image, and vxj, vyj are the moving speeds of feature point i in the x and y directions in the image. a to d are normalization parameters for stabilizing the theoretical error included in the evaluation formula. The two-dimensional information evaluation unit 28 outputs the result of applying a second weight to the second possibility calculated by formula (2) to the clustering unit 30.

[0039] The clustering unit 30 adds the value obtained by multiplying the first possibility by the first weight w1 and the value obtained by multiplying the second possibility by the second weight w2, and calculates the result as a matching cost. The clustering unit 30 creates a matching cost matrix for each combination of feature points shown in FIG. 5. In FIG. 5, each feature point is indicated by "A" to "F", and the matching cost for each combination of feature points is indicated. For example, the matching cost of feature point "A" and feature point "B" is 16. The clustering unit 30 performs clustering by connecting the combinations in ascending order of cost, and detects the clustered feature points as one object by combining feature points whose matching cost is equal to or less than a predetermined threshold. Hereinafter, a plurality of feature points clustered as one object will be called a "cluster".

[0040] As described above, the first possibility (f3d(i, j)) and the second possibility (f2d(i, j)) are calculated as values ​​that are smaller the more likely that feature point i and feature point j are feature points of the same object, and therefore the matching cost is smaller the more likely that feature point i and feature point j are feature points of the same object. However, for example, the first possibility may be the inverse (1 / f3d(i, j)) and the second possibility may be the inverse (1 / f2d(i, j)), and the first possibility and the second possibility may be values ​​that are larger the more likely that feature point i and feature point j are feature points of the same object. Also, the sign of the first possibility may be a negative sign (-f3d(i, j)) and the sign of the second possibility may be a negative sign (-f2d(i, j)), and the first possibility and the second possibility may be values ​​that are larger the more likely that feature point i and feature point j are feature points of the same object. Alternatively, the inverse of the value obtained by weighting and adding the first possibility (f3d(i, j)) and the second possibility (f2d(i, j)) may be set as the cost, so that the more likely it is that feature point i and feature point j are feature points of the same object, the higher the cost. In this case, the more likely it is that feature point i and feature point j are feature points of the same object, the higher the matching cost becomes. Therefore, the clustering unit 30 performs clustering by linking combinations in descending order of cost, and combines feature points whose matching cost is equal to or greater than a predetermined threshold. In this way, it is possible to detect the clustered feature points as one object. Therefore, it is possible to appropriately change whether the first possibility and the second possibility are set to a small value or a large value as the more likely it is that feature point i and feature point j are feature points of the same object. In other words, it is sufficient that the first possibility and the second possibility are parameters that change according to the possibility that feature point i and feature point j are feature points of the same object, and it is sufficient that the matching cost is a value obtained by weighting and adding the first possibility and the second possibility, and the value itself is not limited to being positive or negative.

[0041] Here, referring to FIG. 6A and FIG. 6B, a process of the clustering unit 30 clustering the feature points to detect an object will be described. FIG. 6A shows a cluster c1 in a case where a plurality of feature points ig are present in an area where the accuracy of position detection by the stereo camera 10 is good. The distance from the host vehicle to the plurality of feature points ig in the forward / backward direction of the host vehicle and the distance between the feature points in real space are shown in the distance image di in FIG. 6A. The plurality of feature points ig shown in the distance image di are present in an area where the accuracy of position detection by the stereo camera 10 is good. Therefore, the clustering unit 30 performs clustering c1 based on a matching cost matrix in which the first possibility is reflected more than the second possibility. This makes it possible to detect the object 3 by clustering c1 the plurality of feature points ig. FIG. 6B shows a cluster c2 in a case where a plurality of feature points ip are present in an area where the accuracy of position detection by the stereo camera 10 is reduced. The distance from the host vehicle to the feature points in the forward / backward direction of the host vehicle and the distance between the feature points in real space are shown in the distance image di in FIG. 6B. The feature points i p shown in the distance image di are present in an area where the accuracy of position detection by the stereo camera 10 is reduced. Therefore, the clustering unit 30 performs clustering c2 based on a matching cost matrix in which the second possibility is reflected more than the first possibility. This makes it possible to detect the object 4 by clustering c2 the feature points i p.

[0042] [Object detection method] Next, an example of the processing of the object detection device shown in Fig. 1 will be described with reference to Fig. 7. The operation of the object detection device shown in the flowchart of Fig. 7 starts at the same time that the ignition switch or power switch of the vehicle is turned ON, and ends when the ignition switch or power switch is turned OFF.

[0043] In step S10, the feature point extraction unit 21 acquires two images of the area in front of the vehicle captured from different positions by the stereo camera 10, and the process proceeds to step S20. In step S20, the feature point extraction unit 21 extracts a plurality of feature points, each of which is a pixel having a feature that can be distinguished from surrounding pixels, and the process proceeds to step S30.

[0044] In step S30, distance image generation unit 22 calculates three-dimensional coordinates (relative coordinates) of the multiple feature points with respect to the host vehicle from the parallax (amount of deviation) for the same feature points among the feature points in the first image and the feature points in the second image, and calculates the distances in the front-rear direction of the host vehicle from the multiple feature points to the multiple feature points from the three-dimensional coordinates of the multiple feature points.The process then proceeds to step S40, where distance image generation unit 22 calculates the distances between the multiple feature points in real space from the three-dimensional coordinates of the multiple feature points with respect to the host vehicle, and the process proceeds to step S50.

[0045] In step S50, the optical flow calculation unit 23 detects current feature points corresponding to the same object in real space as the object corresponding to the past feature point as related feature points, and calculates a combination of the mutually related past feature points and current feature points as an optical flow. The optical flow calculation unit 23 calculates the distance between multiple feature points in the image from the optical flow, and the process proceeds to step S60. In step S60, the optical flow calculation unit 23 calculates the moving speed of each feature point in the image from the optical flow, and the process proceeds to step S70.

[0046] In step S70, the three-dimensional information evaluation unit 28 filters the feature points. Specifically, the three-dimensional information evaluation unit 28 determines whether or not the distance between the feature points in the real space, taking into account the theoretical error that occurs when calculating the distance from the vehicle to the feature points in the front-rear direction of the vehicle, is within a predetermined allowable distance. Then, the three-dimensional information evaluation unit 28 prohibits the feature points, whose distance between the feature points in the real space, taking into account the theoretical error, is longer than the predetermined allowable distance, from being clustered as feature points of the same object, and the process proceeds to step S80.

[0047] In step S80, the evaluation method determination unit 29 sets a first weight to be assigned to the first possibility and a second weight to be assigned to the second possibility based on the longitudinal distances from the vehicle to the multiple feature points, and the processing proceeds to step S90.

[0048] In step S90, the three-dimensional information evaluation unit 28 calculates a first possibility indicating that the feature points to which a first weight has been applied (multiplied) are feature points of the same object based on the distance between the feature points in the real space of each of the feature points after filtering and the moving speed of each of the feature points in the real space. The two-dimensional information evaluation unit 27 calculates a second possibility indicating that the feature points to which a second weight has been applied (multiplied) are feature points of the same object based on the distance between the feature points and the moving speed of each of the feature points. Then, the clustering unit 30 adds the first possibility and the second possibility to calculate a matching cost. The clustering unit 30 creates a matching cost matrix for each combination of feature points. Then, the process proceeds to step S100, where the clustering unit 30 performs clustering by linking the combinations in ascending order of cost, and detects the clustered feature points as one object by combining feature points whose matching cost is equal to or less than a predetermined threshold.

[0049] [Effects] As described above, according to this embodiment, the following effects can be obtained.

[0050] A first possibility is calculated based on at least the distance between the multiple feature points in real space, and a first weight is assigned to the first possibility indicating the possibility that the multiple feature points are feature points of the same object, and a second possibility is calculated based on the distance between the multiple feature points in the image and the moving speed of each feature point, and a second weight is assigned to the second possibility indicating the possibility that the multiple feature points are feature points of the same object, and is set based on the distance from the host vehicle to the multiple feature points in the forward / backward direction (a specified direction) of the host vehicle, and the multiple feature points are clustered using the first possibility with the first weight assigned and the second possibility with the second weight assigned.

[0051] The first possibility is the possibility that a plurality of feature points calculated based on the distances between the plurality of feature points in real space are feature points of the same object, and is effective in an area where the accuracy of position detection by the stereo camera 10 is good. The second possibility is the possibility that a plurality of feature points calculated based on the distances between the plurality of feature points in the image and the moving speed of each of the feature points are feature points of the same object, and is effective in an area where the accuracy of position detection by the stereo camera 10 is reduced. Therefore, by setting the first weight and the second weight based on the distances from the host vehicle to the plurality of feature points in the front-rear direction of the host vehicle, clustering can be performed with high accuracy even in an area where the accuracy of position detection by the stereo camera is reduced, and objects can be accurately detected.

[0052] It is determined whether or not the distance between the multiple feature points in the real space, taking into consideration a theoretical error that occurs when calculating the distance in the front-rear direction of the vehicle from the multiple feature points to the multiple feature points, is within a predetermined allowable distance. Then, feature points whose distance between the multiple feature points in the real space, taking into consideration the theoretical error, is longer than the predetermined allowable distance are prohibited from being clustered as feature points of the same object. As a result, feature points whose distance between the multiple feature points in the real space, taking into consideration the theoretical error, is determined not to be feature points of the same object, and clustering is prohibited, thereby preventing erroneous clustering. In other words, an object can be detected with high accuracy.

[0053] It is determined whether the distance in the longitudinal direction of the host vehicle from the host vehicle to the multiple feature points is equal to or greater than a predetermined distance, and if the distance in the longitudinal direction is equal to or greater than the predetermined distance, the first weight is set to 0, and if the distance in the longitudinal direction is less than the predetermined distance, the second weight is set to 0. As a result, if the distance in the longitudinal direction of the host vehicle from the host vehicle to the multiple feature points is equal to or greater than a predetermined distance, it is possible to determine whether the multiple feature points are feature points of the same object using the second possibility that is effective in an area where the accuracy of position detection by the stereo camera 10 is reduced. Also, if the distance in the longitudinal direction of the host vehicle from the host vehicle to the multiple feature points is less than a predetermined distance, it is possible to determine whether the multiple feature points are feature points of the same object using the first possibility that is effective in an area where the accuracy of position detection by the stereo camera 10 is good. Therefore, it is possible to more accurately determine the possibility that the multiple feature points are feature points of the same object.

[0054] The first weight and the second weight are continuously changed based on the distance from the vehicle to the multiple feature points in the forward / backward direction of the vehicle and the angle the feature points make with respect to the straight-ahead direction (predetermined direction) of the vehicle. This makes it possible to determine whether the multiple feature points are feature points of the same object using both the first possibility and the second possibility while changing the first weight and the second weight in response to the change in the distance from the vehicle to the multiple feature points in the forward / backward direction of the vehicle and the angle the feature points make with respect to the straight-ahead direction of the vehicle. In other words, clustering can be performed with higher accuracy.

[0055] The first weight decreases as the distance from the host vehicle to the multiple feature points in the longitudinal direction of the host vehicle increases, and the second weight increases as the distance from the host vehicle to the multiple feature points in the longitudinal direction of the host vehicle increases. As a result, the influence of the first possibility when calculating the possibility that the multiple feature points are feature points of the same object can be reduced and the influence of the second possibility can be increased as the distance from the host vehicle to the multiple feature points in the longitudinal direction of the host vehicle increases. Therefore, the influence of the first possibility and the influence of the second possibility can be appropriately changed according to the distance from the host vehicle to the multiple feature points in the longitudinal direction of the host vehicle, and the possibility that the multiple feature points are feature points of the same object can be calculated. In other words, clustering can be performed with higher accuracy.

[0056] The first weight decreases as the angle of the feature point relative to the straight-line direction of the vehicle increases, and the second weight increases as the angle of the feature point relative to the straight-line direction of the vehicle increases. In position detection by the stereo camera 10, in the case of a feature point having an angle relative to the straight-line direction of the vehicle, the greater the angle of the feature point relative to the straight-line direction of the vehicle, the greater the theoretical error in calculating the left-right position of the vehicle. Therefore, the weights of the first possibility and the second possibility can be changed in accordance with the characteristic that the greater the angle of the feature point relative to the straight-line direction of the vehicle, the greater the theoretical error in calculating the left-right position of the vehicle. As a result, the influence of the first possibility and the influence of the second possibility can be appropriately changed according to the angle of the feature point relative to the straight-line direction of the vehicle, and the possibility that multiple feature points are feature points of the same object can be calculated. In other words, clustering can be performed with higher accuracy.

[0057] As described above, the embodiment of the present invention has been described, but the description and drawings forming a part of this disclosure should not be understood as limiting this invention. From this disclosure, various alternative embodiments, examples and operating techniques will become apparent to those skilled in the art. [Explanation of symbols]

[0058] 1. Object detection device 10 Stereo Camera 20 Control section 21 Feature point extraction unit 22 Distance image generation unit 23 Optical flow calculation unit 24 Vehicle behavior measurement unit 25 3D flow calculation section 26 Object detection unit 27 2D Information Evaluation Section 28 3D Information Evaluation Department 29 Evaluation Method Decision Division 30 Clustering Department

Claims

1. Obtaining two images by capturing images from different positions in a predetermined direction from the vehicle; Extracting a plurality of feature points from each of the two images; calculating distances from the vehicle to the plurality of feature points in the predetermined direction from the positions of the plurality of feature points on the two images, and distances between the plurality of feature points in real space; a first weighting applied to a first possibility calculated based on at least a distance between the plurality of feature points in the real space, the first weighting applied to the first possibility indicating a possibility that the plurality of feature points are feature points of a same object, and a second weighting applied to the second possibility calculated based on a distance between the plurality of feature points in the image and a moving speed of each of the feature points, the second weighting applied to the second possibility indicating a possibility that the plurality of feature points are feature points of a same object ... calculated based on a distance between the plurality of feature points in the image and a moving speed of each of the feature points, the second weighting applied to the second possibility indicating a possibility that the plurality of feature points are feature points of a same object, Detecting an object by clustering the plurality of feature points using the first possibility with the first weight and the second possibility with the second weight.

13. An object detection method comprising:

2. determining whether or not a distance between the plurality of feature points in the real space, taking into consideration a theoretical error that occurs when calculating a distance from the host vehicle to the plurality of feature points in the predetermined direction, from the positions of the plurality of feature points on the two images, is within a predetermined allowable distance; When the distance between the plurality of feature points in the real space taking into account the theoretical error is longer than the predetermined allowable distance, the feature points are prohibited from being clustered as feature points of the same object. The object detection method according to claim 1 .

3. determining whether a distance from the vehicle to the plurality of characteristic points in the predetermined direction is equal to or greater than a predetermined distance; If the distance in the predetermined direction is equal to or greater than a predetermined distance, the first weight is set to 0; If the distance in the predetermined direction is less than a predetermined distance, the second weight is set to 0. The object detection method according to claim 1 .

4. The first weight and the second weight are continuously changed based on distances from the host vehicle to the plurality of characteristic points in the predetermined direction. The object detection method according to claim 1 .

5. the first weight decreases as the distance in the predetermined direction increases, The second weight increases as the distance in the predetermined direction increases. The object detection method according to claim 4 .

6. Calculating angles that the plurality of feature points make with respect to the predetermined direction the first weight decreases as the angle increases; The second weight increases as the angle increases. The object detection method according to claim 4 .

7. Two cameras that capture two images from different positions toward a predetermined direction from the vehicle; A control unit, The control unit is Extracting a plurality of feature points from each of the two images; calculating distances from the vehicle to the plurality of feature points in the predetermined direction from the positions of the plurality of feature points on the two images, and distances between the plurality of feature points in real space; a first weighting applied to a first possibility calculated based on at least a distance between the plurality of feature points in the real space, the first weighting applied to the first possibility indicating a possibility that the plurality of feature points are feature points of a same object, and a second weighting applied to the second possibility calculated based on a distance between the plurality of feature points in the image and a moving speed of each of the feature points, the second weighting applied to the second possibility indicating a possibility that the plurality of feature points are feature points of a same object ... calculated based on a distance between the plurality of feature points in the image and a moving speed of each of the feature points, the second weighting applied to the second possibility indicating a possibility that the plurality of feature points are feature points of a same object, Detecting an object by clustering the plurality of feature points using the first possibility with the first weight and the second possibility with the second weight.

1. An object detection device comprising:

Images