Speed ​​estimation method and speed estimation device

Abstract

To estimate the speed of an object with good accuracy even in a region where the accuracy of detecting positions by a stereo camera decreases.SOLUTION: A prescribed direction from an own vehicle is imaged from different positions to acquire two or more images, a plurality of feature points are extracted from each of the two or more images, the plurality of extracted feature points are clustered with respect to feature points of the same object. One of a first measurement method or a second measurement method is selected as a method for measuring speed of the clustered object in a prescribed direction, and the speed of the object in the prescribed direction is estimated using the selected measurement method. The first measurement method calculates a distance from the own vehicle to the object in the prescribed direction on the basis of positions of the feature points of the object in the two or more images, and calculates speed from a change rate of the distance. The second measurement method calculates size of the object on the basis of the positions of feature points of the object in the images and calculates the speed from a change rate of the size.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present invention relates to a speed estimation method and a speed estimation device.

Background Art

[0002] Conventionally, a speed estimation method is known that detects an object (stereoscopic object) from a pair of images captured by a stereo camera and estimates the moving speed of the detected object (Patent Document 1). The speed estimation method disclosed in Patent Document 1 calculates distance data, which is the parallax for each corresponding region of a pair of images, by stereo matching, and detects an object based on the distance data. Then, it detects an optical flow corresponding to the detected object and estimates the speed of the object based on the distance data and the optical flow.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, the accuracy of position detection by a stereo camera decreases as the distance between the stereo camera and the object increases. Therefore, in the speed estimation method disclosed in Patent Document 1, in a region where the accuracy of position detection of the stereo camera decreases, it may not be possible to accurately calculate the distance data and the optical flow, and the accuracy of the speed of the object estimated based on the distance data and the optical flow decreases.

[0005] The present invention has been made in view of the above problems, and an object thereof is to provide a speed estimation method and a speed estimation device that can accurately estimate the speed of an object even in a region where the accuracy of position detection by a stereo camera decreases.

Means for Solving the Problems

[0006] A velocity estimation method according to one aspect of the present invention involves capturing images from different positions in a predetermined direction from the vehicle to acquire two or more images, extracting multiple feature points from each of the two or more images, and clustering the extracted multiple feature points with the feature points of the same object. Either the first measurement method or the second measurement method is selected as the method for measuring the velocity of the clustered object in a predetermined direction, and the velocity of the object in the predetermined direction is estimated using the selected measurement method. The first measurement method calculates the distance from the vehicle to the object in a predetermined direction based on the positions of the object's feature points in the images, and calculates the velocity from the rate of change of the distance. The second measurement method calculates the size of the object based on the positions of the object's feature points in the images, and calculates the velocity from the rate of change of the size. [Effects of the Invention]

[0007] According to the present invention, the velocity of an object can be estimated with high accuracy even in regions where the accuracy of position detection by a stereo camera decreases. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is a block diagram showing the configuration of a speed estimation device according to an embodiment. [Figure 2A] Figure 2A is a schematic diagram illustrating the process of calculating the size of an object. [Figure 2B] Figure 2B is a schematic diagram illustrating the process for calculating the rate of change in the size of an object. [Figure 3A] Figure 3A is a schematic diagram showing the first blind spot region in the imaging unit 10. [Figure 3B] Figure 3B is a schematic diagram showing the second blind spot region in the imaging unit 10. [Figure 4A] Figure 4A is a schematic diagram showing the process of selecting a measurement method (first example). [Figure 4B] Figure 4B is a schematic diagram showing the process of selecting a measurement method (second example). [Figure 4C] Figure 4C is a schematic diagram showing the process of selecting a measurement method (third example). [Figure 5] Figure 5 is a flowchart showing the process by which the velocity estimation device according to the first embodiment estimates the velocity of an object in a predetermined direction. [Figure 6] Figure 6 is a flowchart showing the process by which the velocity estimation device according to the second embodiment estimates the velocity of an object in a predetermined direction. [Figure 7] Figure 7 is a flowchart showing the process by which the velocity estimation device according to the third embodiment estimates the velocity of an object in a predetermined direction. [Modes for carrying out the invention]

[0009] The embodiments will be described with reference to the drawings. In the drawings, the same parts are denoted by the same reference numerals and their descriptions are omitted.

[0010] [Configuration of the speed estimation device] Referring to Figure 1, the configuration of the speed estimation device 1 according to an embodiment will be described. The speed estimation device 1 comprises an imaging unit 10 and a control unit 20. The speed estimation device 1 is mounted on a vehicle and estimates the velocity of objects present around the vehicle in a predetermined direction. The speed estimation device 1 can be used in vehicle control systems such as ADAS (Advanced Driver-Assistance Systems) and autonomous driving systems that control the vehicle using the velocity of objects.

[0011] The imaging unit 10 is a stereo camera equipped with cameras 11 and 12. The imaging unit 10 is installed on the vehicle so that the imaging direction is a predetermined direction. Cameras 11 and 12 are positioned at different locations within the housing of the imaging unit 10. Specifically, cameras 11 and 12 are positioned at different locations in directions that are at an angle to the predetermined direction, which is the imaging direction for the imaging unit 10. For example, if the predetermined direction is the front of the vehicle, cameras 11 and 12 are positioned at different locations to the left and right of the vehicle, facing forward. In this case, camera 11 is positioned on the left side of the vehicle, and camera 12 is positioned on the right side of the vehicle. Therefore, camera 11 images the predetermined direction from the left side facing forward of the vehicle, and camera 12 images the predetermined direction from the right side facing forward of the vehicle. The predetermined direction may be any direction of the vehicle. Cameras 11 and 12 acquire two images by imaging the predetermined direction from different positions relative to the vehicle. Cameras 11 and 12 perform imaging at the same time. Camera 11 acquires a first image, and camera 12 acquires a second image. The imaging unit 10 repeatedly captures images in a predetermined direction at a predetermined period, acquiring multiple images of the vehicle in the predetermined direction captured at the same time, and transmits the acquired images to the control unit 20. In this embodiment, the imaging unit 10 uses a stereo camera with two cameras, but it may also be a stereo camera with one or three or more cameras.

[0012] In the following description of this embodiment, an example is given in which the predetermined direction, which is the imaging direction of the imaging unit 10, is the front-rear direction, and the different positions where cameras 11 and 12 are arranged are different in the left-right direction of the vehicle.

[0013] The control unit 20 extracts a plurality of feature points from each of the two images acquired by the imaging unit 10, and detects an object in front of the vehicle by clustering the extracted plurality of feature points as feature points of the same object. The control unit 20 selects either the first measurement method or the second measurement method as a method for estimating the speed of the clustered object in a predetermined direction, and estimates the speed of the object in the predetermined direction using the selected measurement method. The object in the present embodiment may be a moving object including other vehicles, motorcycles, bicycles, pedestrians, or a stationary object including parked vehicles. Details of the first measurement method and the second measurement method, and a method for selecting either the first measurement method or the second measurement method will be described later. The control unit 20 is a general-purpose microcomputer including a CPU (central processing unit), a memory (storage unit) such as a RAM and a ROM, and an input / output unit. A computer program for functioning as the speed estimation device 1 is installed in the microcomputer. By executing the computer program, the microcomputer functions as a plurality of information processing circuits (210, 220) included in the speed estimation device 1. In the present embodiment, an example in which a plurality of information processing circuits (210, 220) included in the speed estimation device 1 are realized by software is shown, but it is also possible to prepare dedicated hardware for executing each information processing to constitute the information processing circuit. Further, a plurality of information processing circuits may be constituted by individual hardware. The control unit 20 includes an object detection unit 210 and a speed estimation unit 220 as a plurality of information processing circuits.

[0014] (Configuration of the object detection unit 210) The object detection unit 210 includes a feature point extraction unit 211, a distance image generation unit 212, an optical flow calculation unit 213, a host vehicle behavior measurement unit 214, a three-dimensional flow calculation unit 215, and a clustering unit 216.

[0015] The feature point extraction unit 211 extracts multiple feature points from the two images acquired by cameras 11 and 12, each of which is a pixel with features that can be distinguished from surrounding pixels. While well-known methods can be applied to feature point extraction, for example, the method described in the non-patent document "Jianbo Shi and Carlo Tomasi, "Good Features to Track," 1994 IEEE Conference on Computer Vision and Pattern Recognition (CVPR'94), 1994, pp.593 - 600." is used. The feature point extraction unit 211 outputs the extracted feature points to the distance image generation unit 212 and the optical flow calculation unit 213. In the following, unless otherwise specified, it is assumed that there are multiple feature points.

[0016] The distance image generation unit 212 calculates the distance from the vehicle to multiple feature points in the longitudinal direction, and the distance between the multiple feature points in real space, respectively, and generates a distance image that displays the relative positions of the multiple feature points with respect to the vehicle. Specifically, the distance image generation unit 212 calculates the 3D coordinates (relative coordinates) of the multiple feature points with respect to the vehicle from the parallax (amount of shift) of the same feature points between the feature points of the first image and the feature points of the second image. Next, the distance image generation unit 212 calculates the distance from the vehicle to the multiple feature points in the longitudinal direction, and the distance between the multiple feature points in real space, from the 3D coordinates of the multiple feature points. Alternatively, the distance image generation unit 212 may directly calculate the distance from the vehicle to the multiple feature points in the longitudinal direction from the parallax between the feature points extracted from the first image acquired by camera 11 and the feature points extracted from the second image acquired by camera 12. Furthermore, the distance image generation unit 212 calculates the angle that the feature points make with respect to the vehicle's longitudinal direction (a predetermined direction) from the 3D coordinates of the multiple feature points relative to the vehicle. Next, the distance image generation unit 212 places each feature point on a projection plane based on the 3D coordinates of the multiple feature points and generates a distance image showing the distance from the vehicle to the multiple feature points in the longitudinal direction of the vehicle. Although well-known methods can be applied to generate the distance image, for example, the method described in the non-patent document "Andreas Geiger, Julius Ziegler and Christoph Stiller “StereoScan: Dense 3d reconstruction in real-time” 2011 IEEE Intelligent Vehicles Symposium (IV), 2011, pp. 963-968" can be used. The distance image generation unit 212 outputs the generated distance image to the 3D flow calculation unit 215.

[0017] The optical flow calculation unit 213 calculates an optical flow from the positions of feature points in a past image of either the camera 11 or the camera 12 extracted by the feature point extraction unit 211 and the positions of feature points in the current image. The optical flow represents, in vector form, the two-dimensional movement (moving speed) of feature points on the image. The optical flow calculation unit 213 detects, as related feature points, the current feature points corresponding to the same object in real space as the object corresponding to the past feature points. The optical flow calculation unit 213 calculates, as an optical flow, a combination of the past feature points and the current feature points that are related to each other. Thereby, the optical flow calculation unit 213 can calculate the distances between a plurality of feature points in the image and the moving speed of each feature point. For the calculation of the optical flow, well-known methods can be applied. For example, the method described in the non-patent document “Bruce D. Lucasand and Takeo Kanade “An Iterative Image Registration Technique with an Application to Stereo Vision” International Joint Conference on Artificial Intelligence, pages 674-679, 1981” is used. The optical flow calculation unit 213 outputs the calculated optical flow to the three-dimensional flow calculation unit 215.

[0018] The vehicle behavior measurement unit 214 measures the behavior of the vehicle. Specifically, the vehicle behavior measurement unit 214 measures the behavior of the vehicle by performing odometry and dead reckoning. The behavior of the vehicle refers to the changes in the relative position, relative attitude, and speed of the vehicle over a predetermined period of time. Based on information obtained from wheel speed sensors that detect the wheel speed of each wheel of the vehicle and steering angle sensors that detect the steering angle of the steering wheels, the vehicle behavior measurement unit 214 can measure the behavior of the vehicle over a predetermined period of time by measuring the relative position, relative attitude, and speed of the vehicle with respect to a predetermined reference point. Alternatively, the vehicle behavior measurement unit 214 may obtain the absolute position and absolute attitude of the vehicle in Earth coordinates from a receiving device (not shown) that receives GPS (Global Positioning System) signals and measure the behavior of the vehicle from the amount of change of each over a predetermined period of time. The vehicle behavior measurement unit 214 outputs the measured behavior of the vehicle to the 3D flow calculation unit 215.

[0019] The 3D flow calculation unit 215 calculates the 3D motion (flow) of feature points in real space. Specifically, the 3D flow calculation unit 215 converts the optical flow calculated by the optical flow calculation unit 213 into motion in real space. As mentioned above, optical flow is the 2D motion of feature points on an image. Therefore, the 3D flow calculation unit 215 converts the 2D motion of multiple feature points on an image into 3D motion (velocity) in real space. First, the 3D flow calculation unit 215 calculates the 2D motion in real space from the 2D motion of feature points on an image. Next, the 3D flow calculation unit 215 adds the longitudinal motion relative to the vehicle to the calculated 2D motion of the feature points in real space. The longitudinal motion of the feature points relative to the vehicle can be calculated from the change in the longitudinal distance of the vehicle from the vehicle to multiple feature points per unit time, which is calculated by the distance image generation unit 212. Note that the three-dimensional movement of the feature points calculated up to this point is relative to the vehicle itself. Therefore, the three-dimensional flow calculation unit 215 corrects the relative movement of the feature points with respect to the vehicle using the vehicle behavior calculated by the vehicle behavior measurement unit 214, and calculates the three-dimensional movement (absolute value) in real space. The three-dimensional flow calculation unit 215 outputs the calculated three-dimensional movement (absolute value) of the feature points in real space to the clustering unit 216.

[0020] The clustering unit 216 clusters multiple feature points extracted from each of the two images based on the likelihood that they belong to the same object, and detects the clustered feature point group (cluster) as a single object. Specifically, the clustering unit 216 can cluster feature points based on the distance from the vehicle to multiple feature points in the longitudinal direction, the distance between multiple feature points in real space, and the three-dimensional movement (absolute value) of the feature points in real space. For example, the clustering unit 216 extracts feature points as cluster candidate points if the difference in the longitudinal distance from the vehicle to multiple feature points is less than a first predetermined difference distance, and the distance between multiple feature points in real space is less than a second predetermined difference distance. The extracted cluster candidate points are feature points that are judged to have a high probability of belonging to the same object based on the positional relationship of the feature points. Then, the clustering unit 216 clusters feature points for which the difference in the three-dimensional movement (absolute value) of the extracted cluster candidate points in real space is less than a third predetermined difference. As a result, the clustering unit 216 can cluster feature points that are highly likely to belong to the same object based on the positional relationship of the feature points and the three-dimensional movement of the feature points. Note that the method for clustering multiple feature points is not limited to this, and various known methods can be used. The clustering unit 216 adds information indicating the clusters to the first image and the second image and outputs it to the measurement method selection unit 221.

[0021] (Configuration of the speed estimation unit 220) The speed estimation unit 220 includes a measurement method selection unit 221 and a speed calculation unit 222.

[0022] The measurement method selection unit 221 selects either the first measurement method or the second measurement method as the method for measuring the velocity of the clustered objects in the forward / backward direction (a predetermined direction), and outputs the selected measurement method to the velocity calculation unit 222. Details of the process for selecting either the first or second measurement method will be described later with reference to Figures 2A to 4C.

[0023] The velocity calculation unit 222 calculates the velocity of the object in the longitudinal direction using either the first measurement method or the second measurement method selected by the measurement method selection unit 221. The first measurement method and the second measurement method will now be described in detail.

[0024] (First measurement method) The first measurement method calculates the longitudinal distance (a predetermined direction) from the vehicle to the object based on the positions of the object's feature points on two images, and calculates the longitudinal velocity of the object from the rate of change of the calculated distance. The process of calculating the predetermined distance from the vehicle to the object is the same as the process by which the distance image generation unit 212 calculates the longitudinal distance of the vehicle from the vehicle to multiple feature points. Therefore, the velocity calculation unit 222 obtains the distances from the distance image generation unit 212 to multiple feature points included in the cluster and identifies the predetermined distance from the vehicle to the object. More specifically, the shortest distance among the distances from the vehicle to each feature point included in the cluster is taken as the longitudinal distance from the vehicle to the object. The velocity calculation unit 222 calculates the longitudinal velocity of the object from the rate of change of the distance from the vehicle to the object per unit time. Note that the method of calculating the predetermined distance from the vehicle to the object is not limited to the above method. For example, the predetermined distance from the vehicle to the object may be the distance to the feature point located at the center of the cluster, or it may be the average value of the distances from the vehicle to each feature point included in the cluster.

[0025] (Second measurement method) The second measurement method calculates the size of an object based on the position of its feature points on the image, and calculates the velocity of the object in the forward and backward direction from the rate of change of the object's size. Here, referring to Figures 2A and 2B, the process by which the velocity calculation unit 222 calculates the size of an object and calculates the velocity from the rate of change of the object's size will be explained. First, referring to Figure 2A, the process by which the velocity calculation unit 222 calculates the size of an object will be explained. As shown in Figure 2A, the velocity calculation unit 222 acquires image p1, which was captured at time t=0, and image p2, which was captured at time t=1. Images p1 and p2 are images captured by either camera 11 or camera 12. Time t=1 is the time after a predetermined time Δt has elapsed from time t=0. Images p1 and p2 show the vehicle V1 (object) and its feature points (feature points i1 to feature points i8). The vehicle V1 shown in images p1 and p2 is the same vehicle. Feature points i1 to i8 are clustered, and feature points i1 to i8 are displayed as cluster C in images p1 and p2. The velocity calculation unit 222 sets a frame f1 surrounding the clustered feature points (feature points i1 to feature points i8) in image p1, and a frame f2 surrounding the clustered feature points (feature points i1 to feature points i8) in image p2. More specifically, the velocity calculation unit 222 sets rectangular frames (f1 and f2) surrounding the clustered feature points (feature points i1 to feature points i8). The velocity calculation unit 222 calculates the area of ​​the region enclosed by frames f1 and f2. Specifically, the velocity calculation unit 222 calculates the area on the image of the region enclosed by frames f1 and f2 based on the coordinates of frames f1 and f2. Note that the area may also be the number of pixels located within the region enclosed by the border lines f1 and f2.

[0026] Next, the speed calculation unit 222 calculates the rate of change of the object's size. As shown in Figure 2B, the speed calculation unit 222 calculates the rate of change of area e1 when the area A1 enclosed by the frame line f1 changes to the area A2 enclosed by the frame line f2. Alternatively, the speed calculation unit 222 may calculate the rate of change of area A3 relative to area A1, i.e., the rate of change of area A1 in the vertical direction e2. The speed calculation unit 222 multiplies the reciprocal of the rate of change of area (e1 or e2) by the previously calculated longitudinal distance of the object to calculate the amount of change per unit time Δt of the longitudinal distance between the vehicle and vehicle V1 (object). The speed calculation unit 222 calculates the longitudinal speed of vehicle V1 from the amount of change per unit time Δt of the longitudinal distance between the vehicle and vehicle V1 to calculate the longitudinal speed of vehicle V1.

[0027] (Method for selecting measurement methods) (Example 1) Next, a first example of the process by which the measurement method selection unit 221 selects either the first measurement method or the second measurement method will be described. As shown in Figure 4A, in the first example, the measurement method selection unit 221 selects either the first measurement method M1 or the second measurement method M2 as a method for measuring the longitudinal velocity of the clustered objects, based on the longitudinal distance Z from the vehicle V0 to the object. Specifically, the measurement method selection unit 221 selects the first measurement method M1 if the distance Z from the vehicle V0 to the object is less than a predetermined distance (first predetermined distance) Zp, and selects the second measurement method M2 if the distance Z from the vehicle V0 to the object is greater than or equal to the predetermined distance Zp. The predetermined distance Zp is the distance at which the accuracy of the object's velocity calculated using the second measurement method M2 is higher than the accuracy of the object's velocity calculated using the first measurement method.

[0028] In the first measurement method, the change in distance in the front-to-back direction when the parallax changes by 1 pixel is calculated by the following equation (1). Z is the distance in the front-to-back direction from the vehicle to the object, D is the parallax between camera 11 and camera 12, b is the distance from the center of the objective lens of camera 11 to the center of the objective lens of camera 12 (baseline length), and f is the focal length.

number

[0029] In the second measurement method, the change in the vertical length Y (size) of an object when the vertical coordinate y of the object changes by 1 pixel is calculated by the following equation (2).

number

[0030] Therefore, the measurement method selection unit 221 compares the change in distance when the parallax D changes by one pixel with the change in the vertical length Y of the object when the vertical coordinate y of the object changes by one pixel, and selects the measurement method with the smaller change. In other words, the measurement method selection unit 221 selects the measurement method with higher resolution per pixel from the first measurement method and the second measurement method. Therefore, the predetermined distance Zp is the distance at which the resolution of the second measurement method is higher than that of the first measurement method.

[0031] Furthermore, the measurement method selection unit 221 selects either the first measurement method or the second measurement method as the measurement method based on the number of feature points of an object in a blind spot area that is visible in the first image but not in the second image. There are two patterns of blind spots: a first blind spot area Ab1, which occurs when the imaging area AL of the first image and the imaging area AR of the second image are different, as shown in Figure 3A, and a second blind spot area Ab2, which occurs due to an obstruction S, as shown in Figure 3B. If feature points of an object cannot be extracted from either image (p1L or p1R) due to the blind spots (Ab1 and Ab2), the distance from the vehicle to the object cannot be calculated. Therefore, the first measurement method cannot calculate the longitudinal velocity of the object. In addition, if some of the feature points of the object are obscured, it may not be possible to accurately calculate the distance to the object, and the first measurement method may not be able to accurately calculate the longitudinal velocity of the object. Therefore, the measurement method selection unit 221 counts the number of feature points of an object present in the first blind spot region Ab1 or the second blind spot region Ab2, and if the number of feature points of an object present in the blind spot regions (Ab1 and Ab2) is greater than or equal to a predetermined number, it calculates the velocity of the object in the forward and backward direction using the second measurement method.

[0032] Furthermore, the measurement method selection unit 221 calculates the degree of coincidence correlation for each of the clustered feature points and selects either the first measurement method or the second measurement method as the method for measuring the velocity of the object in the forward-backward direction based on the calculated degree of coincidence correlation. The degree of coincidence correlation is the degree to which the 3D coordinates and 3D motion of the same feature points in the first image and the second image match. For example, if the measurement method selection unit 221 extracts a cluster in which part of the feature points are occluded in a blind spot region (Ab1 or Ab2), it calculates the degree of coincidence correlation for each of the feature points extracted from the two images among the feature points included in the cluster. Then, if the degree of coincidence correlation is less than a predetermined correlation, the measurement method selection unit 221 selects the second measurement method as the method for calculating the velocity of the object in a predetermined direction. In this way, the measurement method selection unit 221 can select either the first measurement method or the second measurement method as the method for measuring the velocity of the object in the forward-backward direction based on the number of occluded feature points of the object, as well as the degree of coincidence correlation of the feature points. Alternatively, the measurement method selection unit 221 may calculate the degree of coincidence and correlation for each of the clustered feature points without counting the number of feature points of the object that are obscured by the blind spots (Ab1 and Ab2), and select either the first measurement method or the second measurement method as the method for measuring the velocity of the object in the longitudinal direction based on the degree of coincidence and correlation.

[0033] (Second example) Next, a second example of the process by which the measurement method selection unit 221 selects either the first measurement method or the second measurement method will be described. In the second example, the measurement method selection unit 221 selects either the first measurement method or the second measurement method as a method for measuring the velocity of the clustered objects in the longitudinal direction, based on the size of the objects. Specifically, as shown in Figure 4B, if the size Y of the objects is less than a predetermined size (first predetermined size) Yp, the first measurement method M1 is selected, and if the size Y of the objects is greater than or equal to the predetermined size Yp, the second measurement method M2 is selected. The size Y of the objects is the vertical length of the objects, as mentioned above. The second example is the same as the first example, but with the predetermined distance Zp replaced by the first predetermined size Yp, and the position where the first and second measurement methods switch is the same as in the first example.

[0034] (Third example) Next, a third example of the process in which the measurement method selection unit 221 selects either the first measurement method or the second measurement method will be described. As shown in Figure 4C, in the third example, a predetermined distance (second predetermined distance) Zp1 shorter than a predetermined distance Zp is set, and if the longitudinal distance Z from the vehicle V0 to the object is less than the predetermined distance Zp1, the first measurement method is selected as the method for measuring the longitudinal velocity of the object. If the longitudinal distance Z from the vehicle V0 to the object is greater than or equal to the predetermined distance Zp1, the measurement method selection unit 221 obtains the size Y of the object. If the size Y of the object is greater than or equal to a predetermined size (second predetermined size) Yp1, the measurement method selection unit 221 selects the second measurement method, and if the size Y of the object is less than the predetermined size Yp1, it continues to select the first measurement method. The predetermined size Yp1 is a value greater than the predetermined size Yp.

[0035] [Speed ​​estimation method] (First Embodiment) Next, with reference to Figure 5, an example of the processing of the speed estimation device shown in Figure 1 will be explained. The processing of the speed estimation device shown in Figure 5 corresponds to the case where the first example is executed in the process in which the measurement method selection unit 221 selects either the first measurement method or the second measurement method. The operation of the speed estimation device shown in the flowchart of Figure 5 starts simultaneously when the vehicle's ignition switch or power switch is turned on, and ends when the ignition switch or power switch is turned off.

[0036] In step S10, cameras 11 and 12 repeatedly capture images of the area in front of the vehicle (in a predetermined direction) at a predetermined interval, acquiring multiple images of the area in front of the vehicle captured at the same time, and transmitting the acquired images to the control unit 20. The process proceeds to step S20, where the feature point extraction unit 211 extracts multiple feature points from the two images acquired by cameras 11 and 12, each of which are pixels with features that can be distinguished from surrounding pixels. The process proceeds to step S30, where the distance image generation unit 212 calculates the distance from the vehicle to the multiple feature points in the front-to-back direction, and the distance between the multiple feature points in real space, respectively, and generates a distance image that displays the relative positions of the multiple feature points with respect to the vehicle. Specifically, the distance image generation unit 212 calculates the three-dimensional coordinates (relative coordinates) of the multiple feature points with respect to the vehicle from the parallax (amount of shift) for the same feature points between the feature points of the first image and the feature points of the second image. Next, the distance image generation unit 212 calculates the distance from the vehicle to the feature points in the longitudinal direction, and the distance between the feature points in real space, from the three-dimensional coordinates of the feature points.

[0037] The process proceeds to step S40, where the optical flow calculation unit 213 calculates the optical flow from the positions of feature points in past images of either camera 11 or camera 12 extracted by the feature point extraction unit 211 and the positions of feature points in the current image. Specifically, the optical flow calculation unit 213 detects current feature points corresponding to the same real-space object corresponding to the past feature points as related feature points. The optical flow calculation unit 213 calculates the optical flow as a combination of related past and current feature points. The process proceeds to step S50, where the vehicle behavior measurement unit 214 measures the behavior of the vehicle by performing odometry and dead reckoning. The behavior of the vehicle refers to the changes in the relative position, relative attitude, and speed of the vehicle over a predetermined time. The vehicle behavior measurement unit 214 may acquire the absolute position and absolute attitude of the vehicle in Earth coordinates from a receiving device (not shown) that receives GPS (Global Positioning System) signals, and measure the behavior of the vehicle from the respective changes over a predetermined time.

[0038] The process proceeds to step S60, where the 3D flow calculation unit 215 calculates the 3D motion (flow) of the feature points in real space. Specifically, the 3D flow calculation unit 215 converts the optical flow calculated by the optical flow calculation unit 213 into motion in real space. Optical flow is the 2D motion of the feature points on the image. Therefore, the 3D flow calculation unit 215 converts the 2D motion of multiple feature points on the image into 2D motion (velocity) in real space. Next, the 3D flow calculation unit 215 adds the longitudinal motion of the vehicle relative to the calculated 2D motion of the multiple feature points in real space. The longitudinal motion of the multiple feature points relative to the vehicle can be calculated from the change per unit time of the longitudinal distance of the vehicle from the vehicle to the multiple feature points, which is calculated by the distance image generation unit 212. The 3D flow calculation unit 215 corrects the relative movement of multiple feature points with respect to the vehicle using the vehicle behavior measurement unit 214's calculation of the vehicle's behavior (absolute value) in real space.

[0039] The process proceeds to step S70, where the clustering unit 216 clusters the multiple feature points extracted from each of the two images based on the likelihood of them being the same object, and detects the clustered feature point group (cluster) as a single object. The clustering unit 216 extracts feature points as cluster candidate points where the difference in the longitudinal distance from the vehicle to the multiple feature points is less than a first predetermined difference distance, and the distance between the multiple feature points in real space is less than a second predetermined difference distance. Then, the clustering unit 216 clusters the feature points extracted as cluster candidate points where the difference in the three-dimensional movement (absolute value) of the feature points in real space is less than a third predetermined difference. The process proceeds to step S80, where the measurement method selection unit 221 counts the number of feature points of objects present in the first blind spot region Ab1 or the second blind spot region Ab2, and determines whether the number of feature points of objects present in the blind spot regions (Ab1 and Ab2) is greater than or equal to a predetermined number. In step S80, if the number of feature points of objects in the blind spot areas (Ab1 and Ab2) is greater than or equal to a predetermined number (YES in step S80), the process proceeds to step S120. If the number of feature points of objects in the blind spot areas (Ab1 and Ab2) is greater than or equal to a predetermined number (NO in step S80), the process proceeds to step S90.

[0040] In step S90, the measurement method selection unit 221 calculates the degree of agreement correlation for each of the clustered feature points and determines whether the degree of agreement correlation is less than a predetermined correlation. If a cluster is extracted in which part of the feature points is obscured by a blind spot region (Ab1 or Ab2), the measurement method selection unit 221 calculates the degree of agreement correlation for each of the feature points included in the cluster that have been extracted from the two images. The degree of agreement correlation is the degree to which the 3D coordinates and 3D motion of the same feature point between the feature points of the first image and the feature points of the second image match. In step S90, if the measurement method selection unit 221 determines that the degree of agreement correlation is less than a predetermined correlation (YES in step S90), the process proceeds to step S120. In step S90, if the measurement method selection unit 221 determines that the degree of agreement correlation is equal to or greater than a predetermined correlation (NO in step S90), the process proceeds to step S100. In step S100, the measurement method selection unit 221 determines whether the distance in the longitudinal direction from the vehicle to the object is greater than or equal to a predetermined distance (first predetermined distance) Zp. If the measurement method selection unit 221 determines in step S100 that the distance in the longitudinal direction from the vehicle to the object is greater than or equal to the first predetermined distance (YES in step S100), the process proceeds to step S120. If the measurement method selection unit 221 determines in step S100 that the distance in the longitudinal direction from the vehicle to the object is less than the first predetermined distance (NO in step S100), the process proceeds to step S110.

[0041] In step S110, the measurement method selection unit 221 selects a first measurement method, and the process proceeds to step S130. In step S120, the measurement method selection unit 221 selects a second measurement method, and the process proceeds to step S130. In step S130, the velocity calculation unit 222 calculates the velocity of the object in the longitudinal direction using either the first measurement method or the second measurement method selected by the measurement method selection unit 221.

[0042] In step S130, if the first measurement method is selected, the speed calculation unit 222 calculates the longitudinal distance from the vehicle to the object based on the positions of the object's feature points on two images. The speed calculation unit 222 calculates the longitudinal velocity of the object from the rate of change per unit time of the distance from the vehicle to the object.

[0043] In step S130, if the second measurement method is selected, the speed calculation unit 222 sets a rectangular frame around the clustered feature points and calculates the area of ​​the region enclosed by the frame. Specifically, the speed calculation unit 222 calculates the area of ​​the region enclosed by the frame on the image based on the coordinates of the frame, and calculates the rate of change of area when the area changes. The speed calculation unit 222 may also calculate the rate of change of the area in the vertical direction. The speed calculation unit 222 multiplies the reciprocal of the rate of change of area by the previously calculated longitudinal distance of the object to calculate the amount of change per unit time of the longitudinal distance between the vehicle and the object. The speed calculation unit 222 calculates the longitudinal velocity of the object from the amount of change per unit time of the longitudinal distance between the vehicle and the object.

[0044] (Second Embodiment) Next, with reference to Figure 6, an example of the processing of the speed estimation device shown in Figure 1 will be described. The processing of the speed estimation device shown in Figure 6 corresponds to the case where the measurement method selection unit 221 performs the second example in the process of selecting either the first measurement method or the second measurement method. Therefore, the second embodiment differs from the first embodiment in that it performs the processing of step S101 instead of the processing of step S100. Thus, only the differences will be explained, and other similar processing will be omitted from the explanation.

[0045] In step S101, the measurement method selection unit 221 determines whether the size of the object is less than a predetermined size (first predetermined size) Yp. If the measurement method selection unit 221 determines in step S101 that the size of the object is equal to or greater than the first predetermined size (YES in step S101), the process proceeds to step S120. If the measurement method selection unit 221 determines in step S101 that the size of the object is less than the first predetermined size (NO in step S110), the process proceeds to step S110.

[0046] (Third embodiment) Next, with reference to Figure 7, an example of the processing of the speed estimation device shown in Figure 1 will be described. The processing of the speed estimation device shown in Figure 7 corresponds to the case where the measurement method selection unit 221 performs the third example in the process of selecting either the first measurement method or the second measurement method. Therefore, the third embodiment differs from the first embodiment in that the processing of step S102 is performed instead of the processing of step S100, and the processing of step S103 is added. Thus, only the differences will be explained, and other similar processing will be omitted from the explanation.

[0047] In step S102, the measurement method selection unit 221 determines whether the distance in the longitudinal direction from the vehicle to the object is greater than or equal to a predetermined distance Zp1 (second predetermined distance). In step S102, if the measurement method selection unit 221 determines that the distance in the longitudinal direction from the vehicle to the object is greater than or equal to the second predetermined distance (YES in step S102), the process proceeds to step S103. In step S102, if the measurement method selection unit 221 determines that the distance in the longitudinal direction from the vehicle to the object is less than the second predetermined distance (NO in step S102), the process proceeds to step S110. In step S103, the measurement method selection unit 221 determines whether the size of the object is greater than or equal to a predetermined size (second predetermined size) Yp1. The second predetermined size is a larger value than the first predetermined size. In step S103, if the measurement method selection unit 221 determines that the size of the object is greater than or equal to the second predetermined size (YES in step S103), the process proceeds to step S120. In step S103, if the measurement method selection unit 221 determines that the size of the object is less than the second predetermined size (NO in step S103), the process proceeds to step S110.

[0048] [Effects and Effects] As described above, the following effects and advantages can be obtained according to this embodiment.

[0049] The speed estimation device 1 acquires two or more images by capturing images from different positions in a predetermined direction from the vehicle, extracts multiple feature points from each of the two or more images, and clusters the extracted feature points with the feature points of the same object. The speed estimation device 1 selects either the first measurement method or the second measurement method as the method for measuring the velocity of the clustered object in a predetermined direction, and estimates the velocity of the object in a predetermined direction using the selected measurement method. In this way, the speed estimation device 1 selects either the first measurement method, which calculates the velocity of the object in a predetermined direction from the rate of change of the distance from the vehicle to the object in a predetermined direction, or the second measurement method, which calculates the velocity of the object in a predetermined direction from the rate of change of the size of the object, as the method for calculating the velocity of the clustered object in a predetermined direction. In this way, the velocity of the object in a predetermined direction can be calculated by appropriately switching between the first and second measurement methods. Therefore, the velocity of the object in a predetermined direction can be estimated with greater accuracy compared to when only one of the measurement methods is used.

[0050] The accuracy of measuring the distance from the vehicle to an object in a predetermined direction using a stereo camera varies depending on the distance from the vehicle to the object. Therefore, the speed estimation device 1 selects either the first measurement method or the second measurement method as the measurement method based on the distance from the vehicle to the object in a predetermined direction. This allows the speed estimation device 1 to estimate the velocity of the object in a predetermined direction by appropriately switching between the first measurement method and the second measurement method according to the distance from the vehicle to the object in a predetermined direction.

[0051] The first measurement method calculates the distance from the vehicle to the object in a predetermined direction based on the positions of the object's feature points on two images, and calculates the velocity of the object in that predetermined direction from the rate of change of the calculated distance. The distance from the vehicle to the object in the processing direction is calculated based on the parallax between the two images. Therefore, in the first measurement method, the resolution of the distance per pixel decreases in proportion to the square of the distance from the vehicle to the object in the predetermined direction. In contrast, the second measurement method calculates the size of the object based on the positions of the object's feature points on the image, and calculates the velocity from the rate of change of the object's size. In the second measurement method, the resolution of the object's size per pixel decreases in proportion to the distance from the vehicle to the object in the predetermined direction. Therefore, the velocity estimation device 1 selects the first measurement method when the distance from the vehicle to the object in the predetermined direction is less than the predetermined distance, and selects the second measurement method when the distance from the vehicle to the object in the predetermined direction is greater than or equal to the predetermined distance. In this way, the velocity estimation device 1 can select the first measurement method in areas where the resolution of the first measurement method is high, and select the second measurement method in areas where the resolution of the second measurement method is high. Therefore, based on the distance from the vehicle to the object in a predetermined direction, the velocity of the object in a predetermined direction can be estimated with high accuracy.

[0052] The velocity estimation device 1 selects either the first measurement method or the second measurement method as the method for measuring the velocity of an object in a predetermined direction, based on the size of the object. The size of the object serves as an indicator for determining which of the first and second measurement methods is more suitable for calculating the velocity of the object in a predetermined direction. Therefore, by selecting either the first or second measurement method as the method for measuring the velocity of an object in a predetermined direction based on the size of the object, the velocity estimation device 1 can accurately estimate the velocity of an object in a predetermined direction.

[0053] The speed estimation device 1 selects either the first or second measurement method as the method for measuring the velocity of an object in a predetermined direction, based on the number of feature points of an object that are visible in the first image included in two or more images, but are in a blind spot area that is not visible in the second image excluding the first image included in the two or more images. If the feature points of an object are not extracted from at least two or more images, the distance from the vehicle to the object in the predetermined direction cannot be calculated. Also, if some of the feature points of an object are extracted from two or more images, the accuracy of calculating the distance from the vehicle to the object in the predetermined direction decreases. This reduces the accuracy of calculating the velocity of the object in the predetermined direction. The speed estimation device 1 selects either the first or second measurement method as the method for measuring the velocity of an object in a predetermined direction, based on the number of feature points of an object that are present in the blind spot area. This allows the speed estimation device 1 to appropriately switch the measurement method depending on the detection status of the object's feature points.

[0054] The velocity estimation device 1 calculates the degree of agreement correlation between two or more images for each of the clustered feature points, and selects either the first measurement method or the second measurement method based on the degree of agreement correlation. This allows the velocity estimation device 1 to switch between the first and second measurement methods based on the degree to which the three-dimensional coordinates and three-dimensional motion of the same feature points in the first image and the second image match. Therefore, the velocity estimation device 1 can select the second measurement method if there is a possibility of reduced accuracy in calculating the velocity of an object in a given direction. This allows the velocity estimation device 1 to estimate the velocity of an object in a given direction with greater accuracy.

[0055] The second measurement method involves setting a frame around multiple clustered feature points and estimating the velocity of the object in a predetermined direction from the rate of change of the area within the frame. This allows the velocity estimation device 1 to estimate the velocity of the object in a predetermined direction based on the rate of change of the object's size.

[0056] The second measurement method involves setting a rectangular frame. This allows the velocity estimation device 1 to always calculate the area of ​​the object using a consistent evaluation method, resulting in a more accurate calculation of the object's area. Consequently, the velocity estimation device 1 can more accurately estimate the velocity of the object in a given direction from the rate of change of the object's area.

[0057] The second measurement method calculates the velocity of an object in a predetermined direction from the vertical rate of change of the area enclosed by the frame. This allows the change in the object's area to be calculated without being affected by the vehicle's behavior. Therefore, the velocity estimation device 1 can estimate the velocity of an object in a predetermined direction with greater accuracy from the rate of change of the object's area.

[0058] As described above, embodiments of the present invention have been presented, but the statements and drawings that constitute part of this disclosure should not be understood as limiting the invention. Various alternative embodiments, examples, and operational techniques will become apparent to those skilled in the art from this disclosure. [Explanation of Symbols]

[0059] 1 Speed ​​estimation device 10 Imaging Unit 20 Control Unit

Claims

1. By capturing images from different positions in a predetermined direction from the vehicle, two or more images are acquired. Multiple feature points are extracted from each of the two or more images mentioned above. The extracted feature points are clustered using feature points of the same object. Either the first measurement method or the second measurement method is selected as the method for measuring the velocity of the clustered objects in the predetermined direction. The velocity of the object in a predetermined direction is estimated using the selected measurement method. The first measurement method described above is: Based on the positions of the feature points of the object on the two or more images, the distance from the vehicle to the object in the predetermined direction is calculated, and the speed is calculated from the rate of change of the distance. The second measurement method described above is: Based on the position of the characteristic points of the object on the image, the size of the object is calculated, and the velocity is calculated from the rate of change of the size. For each of the clustered feature points, calculate the degree of agreement and correlation in the two or more images. Based on the aforementioned correlation, one of the first measurement method and the second measurement method is selected as the measurement method. Speed ​​estimation method.

2. If the coincident correlation is less than a predetermined correlation, the second measurement method is selected as the measurement method. If the coincident correlation is equal to or greater than the predetermined correlation, then, based on the distance, either the first measurement method or the second measurement method is further selected as the measurement method. The speed estimation method according to claim 1.

3. If the distance is less than a predetermined distance, the first measurement method is selected; if the distance is greater than or equal to a predetermined distance, the second measurement method is selected. The speed estimation method according to claim 2.

4. The predetermined distance is the distance at which the accuracy of the speed calculated using the second measurement method becomes higher than the accuracy of the speed calculated using the first measurement method. The speed estimation method according to claim 3.

5. If the aforementioned coincident correlation is less than a predetermined correlation, the second measurement method is selected as the measurement method; if the aforementioned coincident correlation is equal to or greater than the predetermined correlation, then, based on the magnitude, either the first measurement method or the second measurement method is further selected as the measurement method. The speed estimation method according to claim 1 or 2.

6. If the number of feature points of the object in a blind spot region that is visible in the first image included in the two or more images, but not visible in the second image excluding the first image included in the two or more images, is greater than or equal to a predetermined number, the second measurement method is selected as the measurement method. If the number of feature points of the object in the blind spot region is less than the predetermined number, then, based on the correlation coefficient, either the first measurement method or the second measurement method is further selected as the measurement method. The speed estimation method according to claim 1.

7. The second measurement method involves setting a frame around the clustered feature points and estimating the velocity from the rate of change of the area of ​​the region enclosed by the frame. The speed estimation method according to claim 1.

8. The second measurement method involves setting the rectangular frame lines. The speed estimation method according to claim 7.

9. The second measurement method calculates the velocity from the rate of change of the area in the vertical direction. The speed estimation method according to claim 8.

10. An imaging unit that captures images from different positions in a predetermined direction from the vehicle and acquires two or more images, It comprises a control unit and, The control unit, Multiple feature points are extracted from each of the two or more images mentioned above. The extracted feature points are clustered using feature points of the same object. Either the first measurement method or the second measurement method is selected as the method for measuring the velocity of the clustered objects in the predetermined direction. The velocity of the object in a predetermined direction is estimated using the selected measurement method. The first measurement method described above is: Based on the positions of the feature points of the object on the two or more images, the distance from the vehicle to the object in the predetermined direction is calculated, and the speed is calculated from the rate of change of the distance. The second measurement method described above is: Based on the position of the characteristic points of the object on the image, the size of the object is calculated, and the velocity is calculated from the rate of change of the size. For each of the clustered feature points, calculate the degree of agreement and correlation in the two or more images. Based on the aforementioned correlation, one of the first measurement method and the second measurement method is selected as the measurement method. Speed ​​estimation device.

Images