Object detection method and object detection device

Abstract

Provided is an object detection method, wherein: the optical flow of a feature point is detected from a captured image obtained by photographing the surroundings of a host vehicle with a camera (S2); a relative distance from the host vehicle to the feature point is measured (S3); a vehicle behavior of the host vehicle is measured (S4); a first feature point speed, which is the speed of the feature point on a spatial coordinate system, is estimated on the basis of the relative distance (S5); a second feature point speed, which is a speed component of the feature point in a lateral direction orthogonal to an optical axis direction of the camera, is calculated on the basis of the optical flow, the relative distance, and the vehicle behavior (S6); the feature point is clustered into a feature point group on the basis of the second feature point speed (S7); and the object is detected from the feature point group.

Description

Object detection method and object detection device

[0001] The present invention relates to an object detection method and an object detection device.

[0002] The following Patent Document 1 describes a movement speed detection device that detects an object within a monitoring area by detecting optical flow from an image captured in the monitoring area, calculating a movement speed based on the optical flow, and grouping pixel blocks based on the movement speed.

[0003] Japanese Patent Application Laid-Open No. 2005-214914

[0004] When detecting objects by clustering feature points on an image based on the movement speed detected from the captured image, there is a risk that objects moving at different speeds in the optical axis direction of the camera may be mistakenly recognized as the same object. An object of the present invention is to improve the detection accuracy when detecting objects from captured images.

[0005] In one aspect of the object detection method of the present invention, the optical flow of feature points is detected from captured images obtained by photographing the area around the host vehicle with a camera, the relative distance from the host vehicle to the feature points is measured, the vehicle behavior of the host vehicle is measured, a first feature point velocity, which is the velocity of the feature point in a spatial coordinate system, is estimated based on the relative distance, a second feature point velocity, which is the lateral velocity component of the feature point that is perpendicular to the optical axis direction of the camera, is calculated based on the optical flow, the relative distance, and the vehicle behavior, the feature points are clustered into feature point groups based on the second feature point velocity, and an object is detected from the feature point groups.

[0006] According to the present invention, it is possible to improve the detection accuracy when detecting an object from a captured image. The objects and advantages of the present invention are realized and attained by using the elements and combinations thereof set forth in the claims. It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not intended to limit the invention as claimed.

[0007] FIG. 1 is a schematic configuration diagram of an example of a vehicle control device according to an embodiment; (a) is a schematic diagram of an image coordinate system and a spatial coordinate system, and (b) is a schematic diagram of the influence of the measurement accuracy of the longitudinal speed on the lateral speed; (b) is a flowchart of an example of an object detection method according to a first embodiment; (c) is a block diagram of an example of the functional configuration of a controller according to a second embodiment; (d) is a flowchart of an example of an object detection method according to a second embodiment; (e) is a flowchart of an example of an object detection method according to a modified example; (f) is a block diagram of an example of the functional configuration of a controller according to a third embodiment; (a) is a schematic diagram of an example of an intersection area, and (b) is a schematic diagram for explaining a method of calculating the intersection speed; and (f) is a flowchart of an example of a method of calculating the intersection time.

[0008] (First embodiment) (Configuration) Fig. 1 is a schematic configuration diagram of an example of a vehicle control device according to an embodiment. A host vehicle 1 includes a vehicle control device 10 that controls the traveling of the host vehicle 1. For example, the vehicle control device 10 may execute autonomous driving control, which automatically drives the host vehicle 1 without the involvement of a driver, based on the traveling environment around the host vehicle 1, or driving assistance control, which assists the driver in driving the host vehicle 1 by controlling at least one of the steering mechanism, driving force, or braking force of the host vehicle 1. The driving assistance control may be, for example, automatic braking, preceding vehicle following control, constant speed traveling control, merging assistance control, etc.

[0009] The vehicle control device 10 includes an external sensor 11, a vehicle sensor 12, a positioning device 13, a map database (map DB) 14, an actuator 17, and a controller 18. The external sensor 11 includes a camera 11a mounted on the host vehicle 1 to capture images of the surroundings of the host vehicle 1. The camera 11a may be, for example, a stereo camera. The external sensor 11 may also include a plurality of different types of object detection sensors that detect objects around the host vehicle 1, such as laser radar, millimeter-wave radar, or LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging).

[0010] The vehicle sensors 12 are mounted on the host vehicle 1 and detect various information (vehicle signals) obtained from the host vehicle 1. The vehicle sensors 12 include, for example, a vehicle speed sensor that detects the vehicle speed of the host vehicle 1, a wheel speed sensor that detects the rotational speed of the tires of the host vehicle 1, an acceleration sensor that detects the acceleration and deceleration of the host vehicle 1, a steering angle sensor that detects the steering angle of the steering wheel, a turning angle sensor that detects the turning angle of the steered wheels, a gyro sensor that detects the angular velocity of the host vehicle 1, a yaw rate sensor that detects the yaw rate, an accelerator sensor that detects the accelerator opening of the host vehicle, and a brake sensor that detects the amount of brake operation.

[0011] The positioning device 13 includes a Global Navigation System (GNSS) receiver and receives radio waves from multiple navigation satellites to measure the current position of the vehicle 1. The GNSS receiver may be, for example, a Global Positioning System (GPS) receiver. The positioning device 13 may be, for example, an inertial navigation system. The map database 14 stores road map data. For example, the map database 14 may store high-precision map data (hereinafter simply referred to as "high-precision map") suitable as map information for autonomous driving.

[0012] The actuator 17 operates the steering wheel, accelerator opening, and brake device of the host vehicle in response to control signals from the controller 18 to generate vehicle behavior of the host vehicle. The actuator 17 includes a steering actuator, an accelerator opening actuator, and a brake control actuator. The steering actuator controls the steering direction and steering amount of the steering of the host vehicle. The accelerator opening actuator controls the accelerator opening of the host vehicle. The brake control actuator controls the braking operation of the brake device of the host vehicle.

[0013] The controller 18 is an electronic control unit that controls the driving of the vehicle 1. The controller 18 includes a processor 18a and peripheral components such as a storage device 18b. The processor 18a may be, for example, a CPU or an MPU. The storage device 18b may include a semiconductor storage device, a magnetic storage device, an optical storage device, etc. The storage device 18b may include memories such as a register, a cache memory, and a ROM and a RAM used as a main storage device. The functions of the controller 18 described below are realized, for example, by the processor 18a executing a computer program stored in the storage device 18b. Note that the controller 18 may be formed by dedicated hardware for executing each of the information processes described below.

[0014] When controlling the running of the vehicle 1, the controller 18 detects the optical flow on the image captured by the camera 11a and detects moving objects around the vehicle 1 based on the optical flow detection results. Specifically, the controller 18 detects feature points of the objects from the image captured by the camera 11a and calculates the optical flow of the feature points. The controller 18 calculates the lateral velocity V of the feature points on the image coordinate system of the image captured by the camera 11a. ｘ is calculated based on the optical flow. Fig. 2(a) is a schematic diagram of the image coordinate system SI and the spatial coordinate system SS. The image coordinate system SI is a coordinate system that has a reference point (e.g., the center of the image) on the image IM captured by the camera 11a as its origin and represents the two-dimensional coordinates of the pixels of the captured image. In Fig. 2(a), the horizontal direction and the vertical direction on the image coordinate system SI are represented by the symbols "x" and "y", respectively.

[0015] The spatial coordinate system SS is a coordinate system that represents three-dimensional coordinates in a three-dimensional space. In this specification, a camera coordinate system having the viewpoint of the camera 11a (the center point of the image sensor) as the origin O is exemplified as the spatial coordinate system SS. However, a stationary coordinate system having a fixed point as the origin, such as a map coordinate system, may also be used as the spatial coordinate system SS. In FIG. 2( a), the longitudinal direction of the spatial coordinate system SS is represented by the symbol "Z". The longitudinal direction is the optical axis direction AO of the camera 11a. The vertical direction is represented by the symbol "Y", and the lateral direction perpendicular to the longitudinal direction and the vertical direction is represented by the symbol "X". For example, the longitudinal direction and lateral direction of the spatial coordinate system SS may coincide with the longitudinal direction and the vehicle width direction of the host vehicle 1, respectively. The optical axis direction AO may be inclined in the pitching direction with respect to the longitudinal direction of the host vehicle 1.

[0016] The controller 18 measures the relative distance and azimuth angle from the host vehicle 1 to the feature point on the spatial coordinate system SS. Hereinafter, the relative distance from the host vehicle 1 to the feature point will be simply referred to as the "relative distance." For example, the controller 18 may measure the relative distance based on the parallax between the feature point on a pair of captured images taken by the camera 11a, which is a stereo camera, and measure the azimuth angle from the coordinate position of the feature point. Furthermore, for example, the controller 18 may measure the relative distance and the azimuth angle based on a laser radar, millimeter-wave radar, LIDAR, or the like of the external sensor 11. The controller 18 estimates a first feature point velocity, which is the velocity of the feature point of the object on the spatial coordinate system SS, based on the change over time in the relative distance and the azimuth angle.

[0017] The measurement accuracy of a stereo camera deteriorates in inverse proportion to the square of the distance to the object being measured. For this reason, it is difficult to accurately calculate the velocity of a feature point on the spatial coordinate system SS for a distant object. For this reason, the controller 18 calculates the lateral velocity V of the feature point on the spatial coordinate system SS using the movement velocity of the feature point on the image coordinate system SI. Ｘ However, as shown by the arrow AR in FIG. 2B, the distance from the origin O (the viewpoint of the camera) to the feature point P ｆ Depth direction D ｄ Feature point P ｆ Even if the feature point P on the image coordinate system SI moves, ｆTherefore, the accuracy of the vertical moving speed on the spatial coordinate system SS affects the accuracy of calculating the horizontal speed.

[0018] Specifically, the feature point P on the image coordinate system SI ｆ The lateral position and lateral velocity of the ｘ " and the vertical position and vertical velocity on the spatial coordinate system SS are represented as "Z" and "V" respectively. Ｚ " and the focal length of the camera 11a is expressed as "f", the lateral velocity V Ｘ is expressed by the following equation (1): Ｘ = (ZV ｘ +xV Ｚ ) / f ... (1) From the above equation (1), the lateral velocity V Ｘ The accuracy of the vertical velocity V Ｚ In this specification, an example is shown in which the "front-rear direction Z" is used as the "vertical direction" on the spatial coordinate system SS, but instead, the "depth direction D" shown in FIG. ｄ " may also be used.

[0019] For this reason, it is difficult to accurately determine the moving speed on the spatial coordinate system SS for the feature points of a distant object, where the measurement accuracy of the stereo camera decreases. Note that when the vehicle 1 is stopped, the longitudinal speed of the object is approximated to be "0", and the lateral speed V on the spatial coordinate system SS is calculated. Ｘ is calculated using the following equation (2) to obtain the lateral velocity V of the object. Ｘ can be obtained. V Ｘ = ZV ｘ / f (2) However, when the host vehicle 1 is moving, the background and objects approach due to the vehicle behavior of the host vehicle, and this approximation is inappropriate for the actual observed values.

[0020] Therefore, the controller 18 measures the vehicle behavior of the host vehicle 1 based on the vehicle signals detected by the vehicle sensor 12, and approximates the longitudinal speed based on the vehicle behavior. For example, the controller 18 measures, as the vehicle behavior, the amount of movement and the amount of change in attitude (e.g., the amount of change in yaw angle) of the host vehicle 1 during a processing cycle of the controller 18. The controller 18 estimates the longitudinal speed of the feature point by calculating a homogeneous transformation matrix T relating to the time change in the position and attitude of the camera 11a based on the measured vehicle behavior.

[0021] For example, the coordinate vector x of a feature point at a certain time t ｔ = (X ｔ ,Y ｔ , Z ｔ , 1) is given, the estimated value x' of the coordinate vector of the feature point at time (t-1) one time before time t ｔ－１ = (X' ｔ－１ , Y' ｔ－１ , Z' ｔ－１ , 1) into x' ｔ－１ = Tx ｔ If the time width (t-(t-1)) of the processing cycle of the controller 18 is represented as Δt, the estimated value V of the vertical velocity of the characteristic point on the spatial coordinate system SS is ｅＺ can be calculated using the following equation (3): ｅＺ = (Z ｔ -Z' ｔ－１ ) / Δt (3) Note that the observed value x of the coordinate vector of the feature point at time (t-1) ｔ－１ = (X ｔ－１ ,Y ｔ－１ , Z ｔ－１ , 1), the estimated value of the coordinate vector of the characteristic point at time t is calculated to obtain the estimated value of the longitudinal velocity V ｅＺ may be calculated.

[0022] The estimated longitudinal velocity V ｅＺ means the relative velocity due to the vehicle behavior of the host vehicle 1 when the feature point of the object is stationary in the longitudinal direction. The controller 18 calculates the estimated value V ｅＺ Based on this, the lateral velocity V of the feature point on the spatial coordinate system SS, which is the velocity component of the feature point in the lateral direction perpendicular to the optical axis direction of the camera 11a, is calculated.２Ｘ is calculated based on the following formula (4): ２Ｘ = (ZV ｘ +xV ｅＺ ) / f ... (4) This allows the horizontal velocity V ２Ｘ The lateral velocity V ２Ｘ is an example of a "second feature point velocity" as defined in the claims. The controller 18 clusters the feature points extracted from the image captured by the camera 11a into feature point groups based on the second feature point velocity. Then, individual objects are detected from the feature point groups.

[0023] FIG. 3 is a flowchart of an example of the object detection method of the first embodiment. In step S1, the camera 11a captures an image of the surroundings of the host vehicle 1. In step S2, the controller 18 detects the optical flow of the feature points from the captured image of the surroundings of the host vehicle 1. In step S3, the controller 18 measures the relative distance from the host vehicle to the feature points. In step S4, the controller 18 measures the vehicle behavior of the host vehicle 1 based on the vehicle signal detected by the vehicle sensor 12. In step S5, the controller 18 estimates a first feature point velocity based on the change over time in the relative distance. In step S6, the controller 18 calculates a second feature point velocity (e.g., lateral velocity V ) based on the optical flow, the relative distance, and the vehicle behavior. ２Ｘ In step S7, the controller 18 clusters the feature points into feature point groups based on the second feature point velocities, and detects an object from the feature point groups.

[0024] Second Embodiment FIG. 4 is a block diagram showing an example of the functional configuration of the controller 18 according to the second embodiment. The controller 18 includes a spatial position calculation unit 20, an optical flow calculation unit 21, a first feature point velocity estimation unit 22, a second feature point velocity calculation unit 23, a moving object detection unit 24, and a vehicle control unit 25. The spatial position calculation unit 20 measures the relative distance and azimuth angle from the vehicle 1 to the feature point. For example, the spatial position calculation unit 20 may measure the relative distance and azimuth angle based on the parallax between a pair of images captured by the camera 11a, which is a stereo camera, or may measure the relative distance and azimuth angle based on laser radar, millimeter-wave radar, LIDAR, or the like. The spatial position calculation unit 20 calculates the position (X, Y, Z) of the feature point on the spatial coordinate system SS based on the relative distance and azimuth angle.

[0025] The optical flow calculation unit 21 detects the optical flow of the feature point from the captured image of the surroundings of the vehicle 1. The first feature point velocity estimation unit 22 estimates the lateral velocity V of the feature point on the spatial coordinate system SS based on the time change of the position calculated by the spatial position calculation unit 20. Ｘ and the longitudinal velocity V Ｚ For example, the first feature point velocity estimation unit 22 estimates the vertical direction velocity V Ｚ In particular, when the relative distance to the feature point is greater than a predetermined threshold Th, which will be described later, the longitudinal velocity V may be estimated using a Kalman filter. Ｚ That is, the vertical velocity V may be estimated based on the relative distance and azimuth angle measured from multiple frames of images captured by the stereo camera used as the camera 11a. Ｚ In addition, the longitudinal velocity V in the Kalman filter may be estimated. Ｚ As the initial state of ｅＺ Using this, the estimation can be stabilized.

[0026] The second feature point velocity calculation unit 23 calculates the lateral velocity V of the feature point on the image coordinate system SI based on the optical flow calculated by the optical flow calculation unit 21. ｘ The second feature point velocity calculation unit 23 calculates the lateral velocity V ｘand the lateral velocity V given by the above equations (3) and (4) based on the longitudinal position Z of the characteristic point and the vehicle behavior of the host vehicle 1. ２Ｘ is calculated as the second feature point velocity.

[0027] The moving object detection unit 24 detects the lateral velocity V estimated by the first feature point velocity estimation unit 22. Ｘ and longitudinal velocity V Ｚ and the lateral velocity V calculated by the second feature point velocity calculation unit 23. ２Ｘ Based on this, feature points extracted from the captured image are classified into groups, and the classified groups are detected as individual moving objects. The moving object detection unit 24 includes a clustering unit 24a and a separating unit 24b. The clustering unit 24a determines whether the relative distance from the vehicle 1 to the feature points is equal to or less than a predetermined threshold Th. The predetermined threshold Th may be set to, for example, a threshold distance at which the detection speed of the stereo camera used as the camera 11a falls within a predetermined allowable error. For example, the distance at which the detection speed falls within an allowable error can be calculated by determining the speed at which the parallax changes by one tone from the parallax characteristics and observation period of the stereo camera.

[0028] If the relative distance is equal to or smaller than the threshold value Th, the clustering unit 24a calculates the lateral velocity V estimated by the first feature point velocity estimation unit 22. Ｘ First, the clustering unit 24a performs labeling and clustering of the feature points extracted from the captured image based on the lateral velocity V Ｘ Based on this, it is determined whether the feature point is moving to the right or left, or is stationary in the horizontal direction, and labeling is performed to classify the feature point into a feature point moving to the right, a feature point moving to the left, or a feature point that is stationary. For example, when the x-coordinate of the feature point in the image coordinate system SI is moving to the right, the horizontal velocity V Ｘ is defined as positive, and the lateral velocity V when the x-coordinate of the feature point moves to the left in the image coordinate system SI is Ｘ The sign of the lateral velocity V may be defined as negative. Then, the clustering unit 24a clusters the feature points that are labeled as moving to the right with respect to the lateral velocity V ＸFeature points are clustered based on the following: Feature points labeled as moving to the left and feature points labeled as stationary are clustered together in the same way.

[0029] On the other hand, if the relative distance is not equal to or less than the threshold value Th, the clustering unit 24a calculates the lateral direction velocity V ２Ｘ The labeling and clustering of the feature points extracted from the captured image is performed based on the lateral velocity V Ｘ Instead of the lateral velocity V ２Ｘ The process is the same as the above except that the separation unit 24b uses the vertical direction velocity V estimated by the first feature point velocity estimation unit 22. Ｚ The clustering unit 24a separates the generated feature point groups based on the vertical velocity V Ｚ For example, the first feature point velocity estimation unit 22 may use a Kalman filter to estimate the vertical velocity V Ｚ When estimating the longitudinal velocity V based on the error covariance matrix, Ｚ The estimation accuracy of may be calculated.

[0030] The vehicle control unit 25 executes vehicle control to control at least one of the steering mechanism, the drive device, and the braking device of the host vehicle 1 based on the moving object detected by the moving object detection unit 24. For example, the vehicle control unit 25 may control at least one of the steering mechanism, the drive device, and the braking device of the host vehicle 1 so as to avoid the detected moving object.

[0031] 5 is a flowchart of an example of an object detection method according to the second embodiment. The processes of steps S10 to S14 are the same as steps S1 to S5 in FIG. 3. In step S15, the moving object detection unit 24 determines whether the relative distance from the vehicle 1 to the feature point is equal to or less than a predetermined threshold Th. If the relative distance is equal to or less than the predetermined threshold Th (step S15: Y), the process proceeds to step S19. If the relative distance is not equal to or less than the predetermined threshold Th (step S15: N), the process proceeds to step S16. The process of step S16 is the same as step S6 in FIG. 3.

[0032] In step S17, the clustering unit 24a calculates the lateral velocity V ２Ｘ In step S18, the clustering unit 24a groups feature points labeled as moving to the right, moving to the left, or standing still in the lateral direction based on the lateral velocity V ２Ｘ Feature points labeled as moving leftward and feature points labeled as stationary are similarly clustered together. The process then proceeds to step S21. The processes in steps S19 and S20 are performed based on the lateral velocity V ２Ｘ The lateral velocity V Ｘ The process in step S21 is the same as that in step S8 in FIG.

[0033] (Modification) In a modification of the second embodiment, the clustering unit 24 a estimates the lateral velocity V Ｘ and the lateral velocity V calculated as the second characteristic point velocity. ２Ｘ and a weighted sum of the velocity Vw Ｘ For example, the feature points are clustered based on the lateral velocity V Ｘ is expressed by the above equation (1), and the lateral velocity V ２Ｘ is expressed by the above equation (4). Therefore, for example, by weighting the above equations (1) and (4) with a weighting coefficient a as in the following equation (5), the weighted velocity VwＸ Vw Ｘ = (ZV ｘ +x(aV ｅＺ +(1-a)V Ｚ ) / f (5) The weighting coefficient a is set to be larger when the relative distance is large than when it is small. For example, the weighting coefficient a may be set to be larger as the relative distance increases. The clustering unit 24a uses the vertical direction velocity V estimated by the first feature point velocity estimation unit 22 Ｚ and V calculated by the above formula (3) ｅＺ Based on the weighting coefficient a, the weighted velocity Vw is calculated using the following equation (5). Ｘ Calculate.

[0034] 6 is a flowchart of an example of the object detection method according to the modified example. The processes of steps S30 to S35 are the same as steps S1 to S6 in FIG. 3. In step S36, the clustering unit 24a calculates the weighted velocity Vw Ｘ In step S37, the clustering unit 24a calculates the weighted velocity Vw Ｘ In step S38, the clustering unit 24a groups the feature points labeled as moving to the right, the feature points labeled as moving to the left, or the feature points labeled as standing still in the horizontal direction, based on the weighted velocity Vw Ｘ Feature points are clustered based on the above. Feature points labeled as moving leftward are clustered together, and feature points labeled as stationary are clustered together in the same way. The processing in step S39 is the same as step S8 in FIG. 3.

[0035] Third Embodiment In a third embodiment, when a moving object detected by the moving object detection unit 24 intersects with the path of the host vehicle 1, the controller 18 calculates an intersection time T from the current time until the moving object intersects with the path of the host vehicle 1. ｃｒｏｓｓAs described above, when the speed of a distant moving object in the spatial coordinate system SS is calculated using a stereo camera, the measurement accuracy becomes unstable. On the other hand, when the coordinates in the spatial coordinate system SS are observed using a Kalman filter to calculate the speed, the delay of the Kalman filter becomes a problem when detecting a road user (vehicle or pedestrian) running out. Therefore, the controller 18 of the third embodiment estimates the lateral velocity V calculated as described above. ２Ｘ The speed at which the moving object moves in the direction intersecting with the path of the vehicle 1 (hereinafter referred to as the "intersection speed V ｃｒｏｓｓ ") and calculate the crossing time T ｃｒｏｓｓ Estimate.

[0036] 7 is a block diagram of an example of the functional configuration of the controller 18 of the third embodiment. In addition to the configuration shown in FIG. 4, the controller 18 of the third embodiment includes an intersection area extraction unit 26, an intersection determination unit 27, and an intersection time estimation unit 28. The intersection area extraction unit 26 extracts an intersection area R that is a candidate area around the vehicle 1 where an object that is expected to intersect with the path of the vehicle 1 may exist. ｃｒｏｓｓ FIG. 8(a) shows the intersection region R ｃｒｏｓｓ For example, the route T ｒ When the crosswalk CW intersects with the crosswalk CW, the area occupied by the crosswalk CW is called the intersection area R ｃｒｏｓｓ For example, the route T ｒ When the lane crosses the oncoming lane, the area occupied by the oncoming lane is called the intersection area R ｃｒｏｓｓ For example, the area occupied by the crossing lane that intersects with the lane in which the vehicle 1 is traveling may be extracted as an intersection area R ｃｒｏｓｓ The vehicle 1 may extract the intersection area R ｃｒｏｓｓ may be extracted from the image captured by the camera 11a, and the intersecting region R ｃｒｏｓｓ may be extracted.

[0037] The intersecting region extraction unit 26 extracts the intersecting region R ｃｒｏｓｓ The expected direction of movement of road users present at ｃｒｏｓｓFor example, the intersection area extraction unit 26 acquires the intersection direction D from high-precision map data suitable as map information for autonomous driving. ｃｒｏｓｓ The intersection determination unit 27 may acquire information on the path T of the vehicle 1 from among the moving objects detected by the moving object detection unit 24. ｒ For example, the intersection determination unit 27 determines whether the moving object MO detected by the moving object detection unit 24 is in the intersection area R. ｃｒｏｓｓ The moving object MO is located within the intersection area R ｃｒｏｓｓ The distance between the moving direction Dm and the crossing direction D ｃｒｏｓｓ If the difference between the current position of the moving object MO and the intersection direction D is equal to or smaller than the threshold value, the intersection determination unit 27 determines that the moving object MO is an intersection object. ｃｒｏｓｓ and the path T of the vehicle 1. ｒ The intersection point with ｃｒｏｓｓ It is calculated as follows.

[0038] The intersection time estimation unit 28 estimates the lateral speed V ２Ｘ The crossing direction D ｃｒｏｓｓ The crossover speed V is the speed above. ｃｒｏｓｓ Figure 8(b) shows the crossing speed V ｃｒｏｓｓ 1 is a schematic diagram for explaining a method of calculating the vertical direction (Z direction) and the crossing direction D on the spatial coordinate system SS. ｃｒｏｓｓ If the angle between the two is θ, the crossing speed V ｃｒｏｓｓ The horizontal (X-direction) component V Ｘ and the vertical (Z) component V Ｚ is expressed by the following equations (6) and (7): Ｘ = sinθ×V ｃｒｏｓｓ …(6) V Ｚ = cos θ × V ｃｒｏｓｓ …(7)

[0039] On the other hand, the lateral velocity V on the image coordinate system SI ｘ is expressed by the following equation (8): ｘ = (f / Z) × V Ｘ −(fX / Z ２ ) × V Ｚ ...(8) Furthermore, if the vehicle speed of the host vehicle 1 is approximated to be 0, the lateral speed V ２Ｘis given by the following equation (9): ２Ｘ = ZV ｘ / f (9) The following equation (10) can be derived from the above equations (8) and (9): V ２Ｘ =V Ｘ -(fX / Z) x V Ｚ …(10)

[0040] Substituting the above equations (6) and (7) into the above equation (10), the lateral velocity V ２Ｘ From the intersection speed V ｃｒｏｓｓ The conversion formula to V is derived from the following formula (11): ｃｒｏｓｓ =V ２Ｘ / (sin θ−(X / Z)cos θ) (11) The intersection time estimation unit 28 estimates the lateral direction velocity V calculated by the second feature point velocity calculation unit 23. ２Ｘ and the longitudinal position Z and lateral position X of the object calculated by the spatial position calculation unit 20, and the intersection speed V based on the above equation (11). ｃｒｏｓｓ The intersection time estimation unit 28 calculates the current position of the moving object MO and the intersection point P ｃｒｏｓｓ The distance between the intersection speed V ｃｒｏｓｓ By dividing by ｃｒｏｓｓ The vehicle control unit 25 estimates the intersection time T ｃｒｏｓｓ For example, when the vehicle 1 is stopped and then starts moving, the vehicle 1 may move toward the intersection point P ｃｒｏｓｓ The time to reach and the crossing time T ｃｒｏｓｓ If the difference is equal to or greater than the threshold, the host vehicle 1 may be permitted to start, and if the difference is less than the threshold, the host vehicle 1 may be prohibited from starting. ｃｒｏｓｓ The time to reach and the crossing time T ｃｒｏｓｓ If the difference is less than the threshold value, a braking force may be generated on the host vehicle 1, or automatic steering control may be performed to avoid the moving object MO.

[0041] FIG. 9 shows the crossing time T ｃｒｏｓｓ 10 is a flowchart of an example of a method for calculating the intersection area R ｃｒｏｓｓIn step S41, the intersection determination unit 27 detects an intersection object. In step S42, the intersection determination unit 27 extracts an intersection point P ｃｒｏｓｓ In step S43, the intersection time estimation unit 28 calculates the intersection speed V ｃｒｏｓｓ After calculating the crossing time T ｃｒｏｓｓ In step S44, the vehicle control unit 25 calculates the intersection time T ｃｒｏｓｓ At least one of the steering angle, driving force, and braking force of the vehicle 1 is controlled based on the above.

[0042] Effects of the Embodiment (1) The controller 18 detects the optical flow of feature points from captured images obtained by capturing images of the surroundings of the vehicle 1 with the camera 11a, measures the relative distance from the vehicle 1 to the feature points, measures the vehicle behavior of the vehicle 1, estimates a first feature point velocity, which is the velocity of the feature point in a spatial coordinate system, based on the relative distance, calculates a second feature point velocity, which is the lateral velocity component of the feature point perpendicular to the optical axis direction of the camera 11a, based on the optical flow, the relative distance, and the vehicle behavior, clusters the feature points into feature point groups based on the second feature point velocity, and detects objects from the feature point groups. This allows the second feature point velocity, which is independent of the accuracy of the measurement of the longitudinal velocity of the stereo camera, to be calculated from the captured images, and the feature points to be clustered based on the second feature point velocity, thereby improving object detection accuracy. Separating the feature point groups generated by clustering based on the first feature point velocity calculated from the relative distance measurement results allows objects with different velocities in the optical axis direction of the camera to be separated. As a result, object detection accuracy is further improved.

[0043] (2) When the relative distance is greater than a predetermined threshold, the controller 18 may cluster the feature points based on the second feature point velocity, and when the relative distance is equal to or less than the predetermined threshold, the controller 18 may cluster the feature points based on the first feature point velocity. The controller 18 may cluster the feature points based on a weighted sum of the first feature point velocity and the second feature point velocity, which are weighted according to the relative distance. In this way, feature points at close range for which the distance measurement accuracy by the stereo camera is high can be clustered by preferentially using the first feature point velocity estimated from the measurement results of the stereo camera, and feature points at long range for which the distance measurement accuracy by the stereo camera is low can be clustered by preferentially using the second feature point velocity that does not depend on the measurement accuracy of the vertical velocity by the stereo camera.

[0044] (3) The controller 18 may measure the relative distance from captured images captured by the stereo camera 11a, and if the relative distance is greater than a predetermined threshold, estimate the velocity of the feature point in the forward / backward direction based on the relative distance measured from multiple frames of captured images. This improves the accuracy of estimating the velocity of feature points at long distances for which the distance measurement accuracy of the stereo camera is low. (4) The controller 18 may label the feature points based on the second feature point velocity according to whether the feature points are moving rightward or leftward or stationary in the lateral direction, and cluster feature points with the same label. This prevents feature points moving rightward from being clustered in the same group as feature points moving leftward or stationary feature points. Similarly, it prevents feature points moving leftward from being clustered in the same group as feature points moving rightward or stationary feature points, and it prevents stationary feature points from being clustered in the same group as moving feature points.

[0045] (5) The controller 18 may detect an object from feature points located within a candidate area where an object expected to intersect with the path of the host vehicle 1 is present. This allows for efficient detection of objects that may approach the host vehicle 1. (6) The controller 18 may estimate an intersection direction, which is the direction of movement of an object detected by separating the feature point group, in which the object intersects with the path of the host vehicle, convert the second feature point velocity of the object into a velocity in the intersection direction, and estimate an intersection time until the object intersects with the path of the host vehicle based on the velocity of the object in the intersection direction. For example, the controller 18 may estimate the intersection time for a moving object that is located within a candidate area where an object that intersects with the path of the host vehicle is present and has a moving direction close to the intersection direction. This allows for rapid estimation of the intersection time until a moving object that may intersect with the path of the host vehicle 1 intersects with the path of the host vehicle 1.

[0046] All examples and conditional terms described herein are intended for educational purposes to aid the reader in understanding the present invention and the concepts provided by the inventor for the advancement of technology, and should be construed without limitation to the specifically described examples and conditions above, and the configuration of examples herein for illustrating the advantages and disadvantages of the present invention. Although the embodiments of the present invention have been described in detail, it should be understood that various changes, substitutions, and alterations can be made thereto without departing from the spirit and scope of the present invention.

[0047] 1... Vehicle, 10... Vehicle control device, 11... External sensor, 11a... Camera, 12... Vehicle sensor, 13... Positioning device, 14... Map database, 17... Actuator, 18... Controller, 18a... Processor, 18b... Storage device

Claims

1. An object detection method comprising: detecting an optical flow of a feature point from an image obtained by photographing an area around a host vehicle with a camera; measuring a relative distance from the host vehicle to the feature point; measuring vehicle behavior of the host vehicle; estimating a first feature point velocity, which is the velocity of the feature point in a spatial coordinate system, based on the relative distance; calculating a second feature point velocity, which is a lateral velocity component of the feature point perpendicular to the optical axis direction of the camera, based on the optical flow, the relative distance, and the vehicle behavior; clustering the feature points into feature point groups based on the second feature point velocity; and detecting an object from the feature point groups.

2. The object detection method of claim 1, further comprising the steps of: clustering the feature points into the feature point groups based on the second feature point velocity when the relative distance is greater than a predetermined threshold; clustering the feature points into the feature point groups based on the first feature point velocity when the relative distance is equal to or less than a predetermined threshold; separating the feature point groups based on the first feature point velocity; and detecting the separated feature point groups as objects.

3. The object detection method according to claim 1, further comprising clustering the feature points based on a weighted sum of the first feature point velocity and the second feature point velocity weighted according to the relative distance.

4. An object detection method according to any one of claims 1 to 3, characterized in that the relative distance is measured from images captured by a stereo camera, and if the relative distance is greater than a predetermined threshold, the speed of the feature point in the forward / backward direction, which is the optical axis direction of the camera, is estimated based on the relative distance measured from multiple frames of captured images.

5. An object detection method as described in any one of claims 1 to 4, characterized in that the feature points are labeled according to whether they are moving to the right or left or are stationary in the lateral direction based on the second feature point velocity, and feature points with the same label are clustered together.

6. An object detection method as described in any one of claims 1 to 5, characterized in that an object is detected from the feature points located within a candidate area in which an object that is expected to intersect with the vehicle's path is present.

7. The object detection method according to any one of claims 1 to 6, characterized in that it comprises estimating an intersection direction, which is a moving direction in which the object detected by separating the feature point groups intersects with the path of the host vehicle, converting the second feature point speed of the object into a speed in the intersection direction, and estimating an intersection time until the object intersects with the path of the host vehicle based on the speed of the object in the intersection direction.

8. The object detection method according to claim 7, characterized in that the intersection time is estimated for a moving object that is located within or near a candidate area in which an object intersects with the vehicle's path and has a moving direction close to the intersection direction.

9. An object detection device comprising: a camera that photographs an area around the host vehicle; a sensor that measures vehicle behavior of the host vehicle; and a controller that detects an optical flow of a feature point from an image captured by the camera, measures a relative distance from the host vehicle to the feature point, estimates a first feature point velocity that is the velocity of the feature point on a spatial coordinate system based on the relative distance, calculates a second feature point velocity that is a lateral velocity component of the feature point that is perpendicular to the optical axis direction of the camera based on the optical flow, the relative distance, and the vehicle behavior, clusters the feature points into feature point groups based on the second feature point velocity, and detects an object from the feature point groups.

Images