Object detection method and object detection device

Abstract

In this object detection method, the optical flow of a feature point of an object in the periphery of a host vehicle is detected from a captured image obtained by imaging the periphery of the host vehicle by a camera (S1), the position of the object on a spatial coordinate system is estimated (S2), a travel trajectory of the host vehicle in the spatial coordinate system is acquired (S3), a vicinity position existing in the vicinity of the object is identified from among positions on the travel trajectory (S4), at least one of a tangential direction of the travel trajectory at the vicinity position and an intersecting direction extending from the position of the object toward the vicinity position is calculated (S5), at least one of the tangential direction and the intersecting direction is projected onto the image coordinate system of the captured image to calculate a reference direction on the image coordinate system (S6), the optical flow according to the movement amount of the host vehicle is subtracted from the optical flow of the feature point to calculate a correction flow (S7), and the direction of the correction flow is compared with the reference direction to determine whether or not the object intersects the travel trajectory (S8).

Description

Object detection method and object detection device

[0001] The present invention relates to an object detection method and an object detection device.

[0002] The image recognition device described in Patent Document 1 below calculates a corrected flow by subtracting an optical flow corresponding to the amount of movement of the vehicle from the optical flow extracted from the captured image, and if it determines that there is no vanishing point on the extension line of the corrected flow, it determines a moving object that intersects with the direction of travel of the vehicle.

[0003] Japanese Patent Application Laid-Open No. 2020-34995

[0004] However, since the traveling direction of the host vehicle does not necessarily coincide with the direction of the vanishing point on the captured image, there is a risk that a moving object intersecting with the traveling direction of the host vehicle may not be properly determined depending on the traveling direction of the host vehicle. An object of the present invention is to improve the accuracy of determining a moving object intersecting with the traveling direction of the host vehicle based on the optical flow extracted from the captured image around the host vehicle.

[0005] In one aspect of the object detection method of the present invention, the optical flow of feature points of objects around the vehicle is detected from an image obtained by photographing the area around the vehicle with a camera, the position of the object in a spatial coordinate system is estimated, the vehicle's driving trajectory in the spatial coordinate system is obtained, nearby positions on the driving trajectory that are near the object are identified, at least one of the tangential direction of the driving trajectory at the nearby position or the intersecting direction from the object's position to the nearby position is calculated, at least one of the tangential direction or the intersecting direction is projected onto the image coordinate system of the captured image to calculate a reference direction in the image coordinate system, an optical flow corresponding to the amount of movement of the vehicle is subtracted from the optical flow of the feature points to calculate a corrected flow, and whether the object intersects with the driving trajectory is determined by comparing the direction of the corrected flow with the reference direction.

[0006] According to the present invention, it is possible to improve the accuracy of determining a moving object that intersects with the traveling direction of a vehicle based on an optical flow extracted from a captured image of the surroundings of the vehicle. The objects and advantages of the present invention are realized and achieved by using the elements and combinations thereof set forth in the claims. It should be understood that both the foregoing general description and the following detailed description are merely exemplary and explanatory and are not intended to limit the invention as defined by the claims.

[0007] 1 is a schematic configuration diagram of an example of a vehicle control device according to an embodiment. (a) is a schematic diagram of a captured image, (b) is a schematic diagram of a nearby point, a tangential direction, and an intersecting direction of a traveling trajectory, and (c) is an explanatory diagram of an example of intersection determination. An explanatory diagram of an example of an object detection method according to a first embodiment. A block diagram of an example of the functional configuration of a controller according to a second embodiment. (a) to (c) are explanatory diagrams of an example of intersection determination. An explanatory diagram of an example of an object detection method according to the second embodiment. (a) to (e) are schematic diagrams for explaining an example of an object detection method according to a third embodiment. An explanatory diagram of an example of an object detection method according to the third embodiment. An explanatory diagram of an example of classification of feature points. A block diagram of an example of the functional configuration of a controller according to a fourth embodiment. An explanatory diagram of an example of an object detection method according to the fourth embodiment.

[0008] (First embodiment) (Configuration) Fig. 1 is a schematic configuration diagram of an example of a vehicle control device according to an embodiment. A host vehicle 1 includes a vehicle control device 10 that controls the traveling of the host vehicle 1. For example, the vehicle control device 10 may execute autonomous driving control, which automatically drives the host vehicle 1 without the involvement of a driver, based on the traveling environment around the host vehicle 1, or driving assistance control, which assists the driver in driving the host vehicle 1 by controlling at least one of the steering mechanism, driving force, or braking force of the host vehicle 1. The driving assistance control may be, for example, automatic braking, preceding vehicle following control, constant speed traveling control, merging assistance control, etc.

[0009] The vehicle control device 10 includes an external sensor 11, a vehicle sensor 12, a positioning device 13, a map database (map DB) 14, an actuator 17, and a controller 18. The external sensor 11 includes a camera 11a mounted on the host vehicle 1 to capture images of the surroundings of the host vehicle 1. The camera 11a may be, for example, a stereo camera. The external sensor 11 may also include a plurality of different types of object detection sensors that detect objects around the host vehicle 1, such as laser radar, millimeter-wave radar, or LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging).

[0010] The vehicle sensors 12 are mounted on the host vehicle 1 and detect various information (vehicle signals) obtained from the host vehicle 1. The vehicle sensors 12 include, for example, a vehicle speed sensor that detects the vehicle speed of the host vehicle 1, a wheel speed sensor that detects the rotational speed of the tires of the host vehicle 1, an acceleration sensor that detects the acceleration and deceleration of the host vehicle 1, a steering angle sensor that detects the steering angle of the steering wheel, a turning angle sensor that detects the turning angle of the steered wheels, a gyro sensor that detects the angular velocity of the host vehicle 1, a yaw rate sensor that detects the yaw rate, an accelerator sensor that detects the accelerator opening of the host vehicle, and a brake sensor that detects the amount of brake operation.

[0011] The positioning device 13 includes a Global Navigation System (GNSS) receiver and receives radio waves from multiple navigation satellites to measure the current position of the vehicle 1. The GNSS receiver may be, for example, a Global Positioning System (GPS) receiver. The positioning device 13 may be, for example, an inertial navigation system. The map database 14 stores road map data. For example, the map database 14 may store high-precision map data (hereinafter simply referred to as "high-precision map") suitable as map information for autonomous driving.

[0012] The actuator 17 operates the steering wheel, accelerator opening, and brake device of the host vehicle in response to control signals from the controller 18 to generate vehicle behavior of the host vehicle. The actuator 17 includes a steering actuator, an accelerator opening actuator, and a brake control actuator. The steering actuator controls the steering direction and steering amount of the steering of the host vehicle. The accelerator opening actuator controls the accelerator opening of the host vehicle. The brake control actuator controls the braking operation of the brake device of the host vehicle.

[0013] The controller 18 is an electronic control unit that controls the driving of the vehicle 1. The controller 18 includes a processor 18a and peripheral components such as a storage device 18b. The processor 18a may be, for example, a CPU or an MPU. The storage device 18b may include a semiconductor storage device, a magnetic storage device, an optical storage device, etc. The storage device 18b may include memories such as a register, a cache memory, and a ROM and a RAM used as a main storage device. The functions of the controller 18 described below are realized, for example, by the processor 18a executing a computer program stored in the storage device 18b. Note that the controller 18 may be formed by dedicated hardware for executing each of the information processes described below.

[0014] When controlling the driving of the host vehicle 1, the controller 18 detects the optical flow on the image captured by the camera 11a, and detects moving objects that intersect with the planned driving path of the host vehicle 1 based on the optical flow detection results. In the following description, the planned driving path of the host vehicle 1 may be simply referred to as the "driving path". Specifically, the controller 18 detects feature points of objects around the host vehicle 1 from the image captured by the camera 11a, and calculates the apparent optical flow of the feature points. Figure 2(a) is a schematic diagram of the image IM captured by the camera 11a. The hatched area LN indicates the lane on which the host vehicle 1 is driving, the reference symbol OB indicates objects around the host vehicle 1, and the reference symbol P ｆ indicates the feature points of the object OB. Note that in FIG. 2(a), the object OB, the lane LN, and the feature points P ｆIn addition, the running trajectory TR of the vehicle 1 and the reference tangential direction DT (described later) ｒ and the reference cross direction DC ｒ , and the correction flow OF are shown for convenience of explanation, but in reality, the running trajectory TR, the reference tangential direction DT ｒ and the reference cross direction DC ｒ , Note that the correction flow OF does not appear in the captured image IM.

[0015] Next, the controller 18 calculates the three-dimensional coordinate system S of the object OB around the vehicle 1. ３Ｄ For example, the controller 18 may measure the relative distance based on the parallax between feature points on a pair of captured images taken by a stereo camera used as the camera 11a, and measure the azimuth angle from the coordinate position of the object OB. The controller 18 may calculate the azimuth angle from the coordinate position of the object OB in the three-dimensional coordinate system S based on the relative distance and the azimuth angle. ３Ｄ For example, the controller 18 estimates the position of the object OB on the object OB. ｆ The position of the object OB may be estimated as the position of the object OB. The controller 18 may measure the relative distance and azimuth angle of the object OB by a monocular stereo technique from an image captured by a monocular camera. The relative distance and azimuth angle may also be measured based on the laser radar, millimeter wave radar, LIDAR, etc. of the external sensor 11. In addition, in this specification, the three-dimensional coordinate system S ３Ｄ , a camera coordinate system with the viewpoint of the camera 11a (the center point of the image sensor) as the origin is shown as an example. However, a stationary coordinate system with a fixed point as the origin, such as a map coordinate system, can also be used as a three-dimensional coordinate system S ３Ｄ It may also be used as.

[0016] The controller 18 is configured in a three-dimensional coordinate system S ３Ｄ The position of the object OB above is expressed in a two-dimensional coordinate system S on a plane perpendicular to the up-down direction. ２Ｄ By projecting it onto the two-dimensional coordinate system S ２Ｄ In this specification, the position of the object OB on the two-dimensional coordinate system S ２Ｄ A camera coordinate system having the viewpoint of the camera 11a (the center point of the image sensor) as the origin is shown as an example, but a stationary coordinate system having a fixed point as the origin is also shown as a two-dimensional coordinate system S ２Ｄ In the following description, the three-dimensional coordinate system S３Ｄ and the two-dimensional coordinate system S ２Ｄ The two-dimensional coordinate system S of the traveling trajectory TR along which the host vehicle 1 is to travel may be collectively referred to as a "spatial coordinate system." ２Ｄ For example, the controller 18 may obtain the travel trajectory TR from the map database 14 or an image of the lane LN captured by the camera 11a. ２Ｄ The positions of the object OB and the travel trajectory TR are shown above.

[0017] The controller 18 is configured in a two-dimensional coordinate system S ２Ｄ A nearby position P on the travel trajectory TR that is located near the object OB ａｄｊ For example, the controller 18 may determine the point on the travel path TR that is closest to the object OB as the nearby position P ａｄｊ The position of the foot of a perpendicular line drawn from the object OB onto the traveling trajectory TR may be specified as the nearby position P ａｄｊ The neighboring position P ａｄｊ If there are multiple candidates for , the point closest to the vehicle 1 is designated as the nearby position P ａｄｊ The controller 18 may specify the nearby position P ａｄｊ The two-dimensional coordinate system S of the running trajectory TR in ２Ｄ The tangent direction DT above, or the position of the object OB to the nearby position P ａｄｊ Two-dimensional coordinate system S ２Ｄ At least one of the upper and lower cross directions DC is calculated.

[0018] Next, the controller 18 projects at least one of the tangential direction DT and the cross direction DC onto the image coordinate system SI of the captured image IM, thereby determining a reference tangential direction DT ｒ or the reference cross direction DC ｒ The reference tangential direction DT ｒ and the reference cross direction DC ｒ In the following description, the reference tangential direction DT ｒ and the reference cross direction DC ｒ are sometimes collectively referred to as the "reference direction." For example, the controller 18 may be configured to use a three-dimensional coordinate system S３Ｄ Upper characteristic point P ｆ The reference tangential direction DT is calculated by converting the direction (vector) parallel to the tangential direction DT and the cross direction DC at the position into a direction (vector) on the image coordinate system SI of the captured image IM. ｒ and the reference cross direction DC ｒ may be calculated respectively.

[0019] The controller 18 measures the vehicle behavior of the host vehicle 1 (the amount of movement and the amount of change in attitude of the host vehicle 1) based on the vehicle signals detected by the vehicle sensors 12. The controller 18 determines the characteristic point P ｆ The optical flow corresponding to the vehicle behavior is subtracted from the apparent optical flow of the characteristic point P ｆ The controller 18 compares the direction of the correction flow OF with the reference direction to determine whether the object OB intersects with the traveling trajectory TR. For example, the controller 18 calculates a correction flow OF based on the reference tangent direction DT. ｒ For example, it may be determined whether the object OB intersects with the traveling trajectory TR by comparing the corrected flow OF with the reference tangent direction DT. ｒ The difference between the reference tangential direction DT and the reference tangential direction DT is equal to or greater than the threshold value. ｒ On the other hand, when the direction of the correction flow OF and the reference tangential direction DT ｒ The difference between the reference tangential direction DT and the correction flow OF is less than the threshold value. ｒ It may be determined that the object OB will not intersect with the traveling trajectory TR if the direction of the object OB is moving away from the traveling trajectory TR.

[0020] Furthermore, for example, the controller 18 may be configured to calculate the correction flow OF and the reference cross direction DC ｒ For example, it may be determined whether the object OB crosses the traveling trajectory TR by comparing the correction flow OF with the reference cross direction DC ｒ If the difference between the reference tangential direction DT and the reference tangential direction DT is less than a threshold, it may be determined that the object OB intersects with the traveling trajectory TR, and if the difference is equal to or greater than the threshold, it may be determined that the object OB does not intersect with the traveling trajectory TR. ｒand the reference cross direction DC ｒ and the correction flow OF, the controller 18 may determine whether the object OB will intersect with the traveling trajectory TR by comparing both the correction flow OF and the reference tangential direction component ΔT and the reference transverse direction component ΔC of the correction flow OF. (See FIG. 2C.) For example, the controller 18 may calculate a reference tangential direction component ΔT and a reference transverse direction component ΔC of the correction flow OF. If the ratio of these components (ΔC / ΔT) is equal to or greater than a threshold, the controller 18 may determine that the object OB will intersect with the traveling trajectory TR, and if the ratio is less than the threshold, the controller 18 may determine that the object OB will not intersect with the traveling trajectory TR.

[0021] (Operation) Figure 3 is an explanatory diagram of an example of the object detection method of the first embodiment. In step S1, the controller 18 detects the apparent optical flow of feature points of objects around the host vehicle 1 from the captured image. In step S2, the controller 18 estimates the position of the object on a spatial coordinate system. In step S3, the controller 18 acquires a travel path along which the host vehicle 1 is scheduled to travel. In step S4, the controller 18 identifies nearby positions on the travel path that are located near the object OB.

[0022] In step S5, the controller 18 calculates at least one of the tangential direction and the intersecting direction. In step S6, the controller 18 calculates a reference direction on the image coordinate system SI by projecting at least one of the tangential direction and the intersecting direction onto the image coordinate system SI. In step S7, the controller 18 calculates a corrected flow by subtracting an optical flow according to the vehicle behavior from the apparent optical flow. In step S8, the controller 18 compares the direction of the corrected flow with the reference direction to determine whether the object intersects with the traveling trajectory.

[0023] 4 is a block diagram showing an example of the functional configuration of the controller 18 according to the second embodiment. The controller 18 includes an optical flow calculation unit 20, a spatial position estimation unit 21, a traveling trajectory acquisition unit 22, a direction calculation unit 23, an intersecting object determination unit 24, and a vehicle control unit 25. The optical flow calculation unit 20 calculates feature points P of an object OB around the vehicle 1 from the captured image of the camera 11a. ｆThe apparent optical flow of the object OB is calculated using a plurality of feature points P ｆ When the feature point P is detected, the difference in height between the optical axis of the camera 11a and the feature point P is large. ｆ The optical flow calculation unit 20 may select and use the optical flow of the characteristic point P by subtracting the optical flow corresponding to the vehicle behavior of the vehicle 1 measured based on the vehicle signal detected by the vehicle sensor 12 from the apparent optical flow. ｆ The correction flow OF is calculated.

[0024] The spatial position estimation unit 21 estimates the feature point P ｆ Three-dimensional coordinate system S ３Ｄ The position above and the two-dimensional coordinate system S ２Ｄ The traveling trajectory acquisition unit 22 estimates the position on the two-dimensional coordinate system S of the traveling trajectory TR. ２Ｄ The direction calculation unit 23 includes a nearby position identification unit 23a, a tangential direction calculation unit 23b, an intersecting direction calculation unit 23c, and a reference direction calculation unit 23d. The nearby position identification unit 23a calculates a position on the two-dimensional coordinate system S ２Ｄ A nearby position P on the travel trajectory TR that is located near the object OB ａｄｊ The tangential direction calculation unit 23b identifies the nearby position P ａｄｊ The two-dimensional coordinate system S of the running trajectory TR in ２Ｄ The intersecting direction calculation unit 23c calculates the tangent direction DT from the position of the object OB to the nearby position P ａｄｊ Two-dimensional coordinate system S ２Ｄ The upper cross direction DC is calculated.

[0025] The reference direction calculation unit 23d projects the tangential direction DT and the cross direction DC onto the image coordinate system SI of the captured image IM, thereby calculating the reference tangential direction DT ｒ and the reference cross direction DC ｒ For example, the reference direction calculation unit 23d calculates the three-dimensional coordinate system S ３Ｄ Upper characteristic point P ｆThe reference tangential direction DT is calculated by converting the direction (vector) parallel to the tangential direction DT and the cross direction DC at the position into a direction (vector) on the image coordinate system SI of the captured image IM. ｒ and the reference cross direction DC ｒ may be calculated respectively.

[0026] The intersecting object determination unit 24 determines the direction of the correction flow OF and the reference direction (reference tangential direction DT ｒ , reference cross direction DC ｒ ) to determine whether the object OB intersects with the traveling trajectory TR. For example, the intersecting object determination unit 24 compares the correction flow OF with the reference tangent direction DT ｒ For example, as shown in FIG. 5A, the correction flow OF and the reference tangential direction DT ｒ The difference between the reference tangential direction DT and the reference tangential direction DT is equal to or greater than the threshold value. ｒ On the other hand, when the correction flow OF is closer to the travel trajectory TR than the reference tangential direction DT, as shown in FIG. ｒ 5(c) , the direction of the correction flow OF and the reference tangential direction DT ｒ If the difference between the corrected flow OF and the reference crossing direction DC is less than the threshold value, it may be determined that the object OB does not cross the traveling trajectory TR. ｒ It may be determined whether the object OB intersects with the traveling trajectory TR by comparing the reference tangent direction DT ｒ and the reference cross direction DC ｒ It may be possible to determine whether the object OB intersects with the travel trajectory TR by comparing both of the above with the correction flow OF.

[0027] The vehicle control unit 25 executes vehicle control to control at least one of the steering mechanism, the drive device, and the braking device of the host vehicle 1 based on the determination result of the intersecting object determination unit 24. For example, when the vehicle control unit 25 determines that an object around the host vehicle 1 will intersect with the traveling trajectory TR, the vehicle control unit 25 may control at least one of the steering mechanism, the drive device, and the braking device of the host vehicle 1 so as to avoid the moving object.

[0028] 6 is an explanatory diagram of an example of an object detection method according to the second embodiment. The processes of steps S10 to S14 are the same as steps S1 to S5 shown in FIG. 3. In step S15, the reference direction calculation unit 23d calculates the reference tangential direction and the reference intersecting direction, assuming that the road surface gradient of the lane LN on which the host vehicle 1 is traveling is "0". The processes of steps S16 and S17 are the same as steps S7 and S8 shown in FIG. 3.

[0029] (Third embodiment) In the third embodiment, a plurality of feature points detected from a captured image IM are grouped (clustered) to group a plurality of feature points of the same object OB. The optical flows of the plurality of feature points of the same object OB are detected, and these corrected flows are compared with the reference direction, respectively, to determine whether the object OB intersects with the traveling trajectory TR. FIGS. 7(a) to 7(e) are schematic diagrams illustrating an example of an object detection method of the third embodiment. In the captured image IM of FIG. 7(c), a plurality of feature points are grouped to detect a plurality of feature points P of the same object OB. ｆ１ , P ｆ２ and P ｆ３ The optical flow calculation unit 20 calculates the feature points P ｆ１ , P ｆ２ and P ｆ３ The apparent optical flows of the two images are calculated, and the correction flow OF １ , OF ２ and OF ３ are calculated respectively.

[0030] The spatial position estimation unit 21 estimates these feature points P ｆ１ , P ｆ２ and P ｆ３ Three-dimensional coordinate system S ３Ｄ Upper position PＰｆ３１ , P Ｐｆ３２ and P Ｐｆ３３ (Fig. 7(b)) and the two-dimensional coordinate system S ２Ｄ Upper position P Ｐｆ２１ , P Ｐｆ２２ and P Ｐｆ２３ (FIG. 7(a)). The neighboring position specifying unit 23a estimates the two-dimensional coordinate system S ２Ｄ The position P of the object OB on the travel trajectory TR Ｐｆ２１ , P Ｐｆ２２ and P Ｐｆ２３ Nearby position P ａｄｊ１ , P ａｄｊ２ and P ａｄｊ３ The tangential direction calculation unit 23b identifies the nearby position P ａｄｊ１ , P ａｄｊ２ and P ａｄｊ３ The two-dimensional coordinate system S of the running trajectory TR in ２Ｄ Upper tangent direction DT １ , D.T. ２ and DT ３ The cross direction calculation unit 23c calculates the position P Ｐｆ２１ , P Ｐｆ２２ and P Ｐｆ２３ From the nearby position P ａｄｊ１ , P ａｄｊ２ and P ａｄｊ３ Two-dimensional coordinate system S ２Ｄ Upper cross direction DC １ , D.C. ２ and D.C. ３ Calculate.

[0031] FIG. 7(b) shows the tangential direction DT １ , D.T. ２ and DT ３ The direction (vector) parallel to and the cross direction DC １ , D.C. ２ and D.C. ３ The direction parallel to the three-dimensional coordinate system S ３Ｄ Upper characteristic point P ｆ１ , P ｆ２ and P ｆ３ Position P Ｐｆ３１ , P Ｐｆ３２ and P Ｐｆ３３ The reference direction calculation unit 23d calculates the position P Ｐｆ３１ , P Ｐｆ３２ and P Ｐｆ３３In the tangential direction DT １ , D.T. ２ and DT ３ The reference tangential direction DT is calculated by converting the direction parallel to the reference tangential direction DT into a direction on the image coordinate system SI of the captured image IM. ｒ１ , D.T. ｒ２ and DT ｒ３ Also, the position P Ｐｆ３１ , P Ｐｆ３２ and P Ｐｆ３３ In the cross direction DC １ , D.C. ２ and D.C. ３ The direction parallel to the reference cross direction DC is converted into a direction on the image coordinate system SI of the captured image IM. ｒ１ , D.C. ｒ２ and D.C. ｒ３ are calculated respectively.

[0032] The position P of the feature point Ｐｆ２１ , P Ｐｆ２２ and P Ｐｆ２３ For each of these, the neighboring positions P ａｄｊ１ , P ａｄｊ２ and P ａｄｊ３ , as shown in FIG. 7(d) and FIG. 7(e), a common neighboring position P ａｄｊ For example, the position P Ｐｆ２１ ~P Ｐｆ２３ The position with the smallest total distance from the common neighboring position P ａｄｊ These positions P Ｐｆ２１ ~P Ｐｆ２３ Any of the nearby positions is called the nearby position P ａｄｊ Alternatively, the neighboring position P ａｄｊ The tangential direction and the intersecting direction at the position P may be calculated as a common tangential direction DT and intersecting direction DC. Ｐｆ３１ , P Ｐｆ３２ and P Ｐｆ３３ The direction parallel to the common tangent direction DT in the image coordinate system SI of the captured image IM is converted into the reference tangent direction DT ｒ１ , D.T. ｒ２ and DT ｒ３ Similarly, the position P Ｐｆ３１ , PＰｆ３２ and P Ｐｆ３３ The direction parallel to the common cross direction DC in the image coordinate system SI of the captured image IM is converted into the reference cross direction DC ｒ１ , D.C. ｒ２ and D.C. ｒ３ As shown in FIG. 7(e), the position P Ｐｆ２１ , P Ｐｆ２２ and P Ｐｆ２３ from the common neighboring position P ａｄｊ Two-dimensional coordinate system S ２Ｄ Upper cross direction DC １ , D.C. ２ and D.C. ３ may be calculated separately.

[0033] The intersecting object determination unit 24 executes the correction flow OF １ ~OF ３ direction and reference direction DT ｒ１ ~DT ｒ３ , D.C. ｒ１ ~DC ｒ３ For example, the intersecting object determination unit 24 determines whether the object OB intersects with the traveling trajectory TR by comparing the correction flow OF １ ~OF ３ The crossover judgment may be performed for each time, and the final judgment result may be determined by majority vote. １ ~OF ３ For each object OB, the object OB may be separated into a reference tangential direction component ΔT and a reference cross direction component ΔC, and whether or not the object OB crosses the traveling trajectory TR may be determined based on the average ratio of these components.

[0034] 8 is an explanatory diagram of an example of an object detection method according to the third embodiment. In step S20, the optical flow calculation unit 20 detects the apparent optical flow of a plurality of feature points detected from the captured image. In step S21, the spatial position estimation unit 21 calculates a three-dimensional coordinate system S ３Ｄ The position above and the two-dimensional coordinate system S ２ＤIn step S22, the traveling trajectory acquisition unit 22 estimates the self-position of the vehicle 1 on the map. For example, the traveling trajectory acquisition unit 22 may estimate the self-position based on the positioning result of the positioning device 13, or may estimate the self-position by odometry based on the vehicle signal detected by the vehicle sensor 12. In step S23, the traveling trajectory acquisition unit 22 acquires position information of the lane on which the vehicle 1 is traveling from the map database 14 based on the estimated self-position and the traveling route plan, and acquires the center line of the traveling lane as the traveling trajectory.

[0035] In step S24, multiple feature points detected from the captured image are grouped to group multiple feature points of the same object. In step S25, the nearby position identification unit 23a calculates individual or common nearby positions for each of the multiple feature points of the same object OB. In step S26, the tangent direction calculation unit 23b calculates individual or common tangential directions for each of the multiple feature points of the same object OB. The intersecting direction calculation unit 23c calculates individual or common intersecting directions for each of the multiple feature points of the same object OB. In step S24, the reference direction calculation unit 23d calculates a reference direction in the image coordinate system SI for each feature point by projecting the tangential direction or intersecting direction onto the image coordinate system SI. In step S28, the optical flow calculation unit 20 calculates corrected flows of the apparent optical flows of the multiple feature points. In step S29, the intersecting object determination unit 24 compares the directions of the multiple corrected flows with the multiple reference directions to determine whether the object OB intersects with the traveling trajectory TR.

[0036] (Fourth embodiment) The intersecting object determination unit 24 of the fourth embodiment classifies a plurality of feature points of the same object OB detected from the captured image IM into first feature points having a relatively high distance measurement accuracy by the stereo camera used as the camera 11a and second feature points having a relatively low distance measurement accuracy. Fig. 9 is an explanatory diagram of an example of classification of feature points. For example, the intersecting object determination unit 24 classifies feature points Pf whose distance from the host vehicle 1 is closer than a threshold value. ２ , Pf ３ , Pf ６ and Pf ７For example, the intersecting object determination unit 24 may classify the feature point Pf whose absolute value of the height difference from the camera 11a or the absolute value of the height difference from the height HOA of the optical axis center of the camera 11a is within a threshold value. ４ The intersecting object determination unit 24 may classify other feature points Pf １ and Pf ５ may be classified into the second feature points.

[0037] The intersecting object determination unit 24 determines the first feature point based on the measurement results of the stereo camera in the two-dimensional coordinate system S ２Ｄ The intersecting object determination unit 24 measures the position of the first feature point on the two-dimensional coordinate system S ２Ｄ The intersecting object determination unit 24 determines whether the object OB intersects with the traveling trajectory TR based on a temporal change in the position of the first feature point above (hereinafter, this may be referred to as a "distance measurement flow"). The intersecting object determination unit 24 determines whether the object OB intersects with the traveling trajectory TR based on a comparison between the corrected flow on the captured image IM and the reference direction, as in the first to third embodiments. The intersecting object determination unit 24 determines whether the object OB intersects with the traveling trajectory TR based on a comparison between the corrected flow and the reference direction. The intersecting object determination unit 24 determines whether the object OB intersects with the traveling trajectory TR based on a comparison between the corrected flow and the reference direction.

[0038] FIG. 10 is a block diagram of an example of the functional configuration of the controller 18 of the fourth embodiment. In addition to the configuration shown in FIG. 4 , the controller 18 includes a road surface gradient estimator 27. The road surface gradient estimator 27 estimates the road surface gradient of the lane on which the host vehicle 1 is traveling (e.g., at least one of the pitch angle and roll angle of the road surface). The road surface gradient estimator 27 may estimate the road surface gradient of the lane on which the host vehicle 1 is traveling based on the estimation result of the host vehicle 1's own position and the map database 14. The reference direction calculator 23d calculates the reference tangent direction and the reference tangent direction by projecting the tangent direction calculated by the tangent direction calculator 23b and the intersecting direction calculated by the intersecting direction calculator 23c onto the image coordinate system SI, assuming the optical axis direction of the camera 11a when the vehicle horizontal plane, which is a plane perpendicular to the height direction of the host vehicle 1, is parallel to the road surface gradient. Note that, in the first to third embodiments described above, the reference tangent direction and the reference tangent direction may also be calculated based on the estimation result of the road surface gradient of the lane on which the host vehicle 1 is traveling.

[0039] The intersecting object determination unit 24 determines whether the object OB intersects with the traveling trajectory TR using the ranging flow for the first feature point. For example, the intersecting object determination unit 24 may determine whether the object OB intersects with the traveling trajectory TR by comparing the ranging flow with the tangential direction DT or the intersecting direction DC. For example, if the difference between the ranging flow and the tangential direction DT is equal to or greater than a threshold and the ranging flow is in a direction closer to the traveling trajectory TR than the tangential direction DT, the intersecting object determination unit 24 may determine that the object OB intersects with the traveling trajectory TR. On the other hand, if the difference between the ranging flow and the tangential direction DT is less than the threshold or the ranging flow is in a direction away from the traveling trajectory TR than the tangential direction DT, the intersecting object determination unit 24 may determine that the object OB does not intersect with the traveling trajectory TR.

[0040] For example, the intersecting object determination unit 24 may determine that the object OB intersects with the traveling trajectory TR when the difference between the ranging flow and the intersecting direction is less than a threshold, and may determine that the object OB does not intersect with the traveling trajectory TR when the difference is equal to or greater than the threshold. For example, the intersecting object determination unit 24 may determine whether the object OB intersects with the traveling trajectory TR based on the ratio between the tangential direction component and the intersecting direction component of the ranging flow.

[0041] As in the first to third embodiments, the intersecting object determination unit 24 determines whether the object OB intersects with the traveling trajectory TR at the second feature point based on a comparison between the corrected flow on the captured image IM and the reference direction. The intersecting object determination unit 24 determines whether the object OB intersects with the traveling trajectory TR based on the result of the intersection determination at the first feature point and the result of the intersection determination at the second feature point. For example, the intersecting object determination unit 24 may obtain a final determination result by a majority vote of these intersection determinations, or may determine whether the object OB intersects with the traveling trajectory TR based on the average of the angular difference between the two-dimensional ranging flow at the first feature point and the tangential direction or the intersecting direction and the angular difference between the corrected flow at the second feature point and the reference direction.

[0042] 11 is an explanatory diagram of an example of the object detection method of the fourth embodiment. In step S30, the optical flow calculation unit 20 detects the apparent optical flow of a plurality of feature points of the same object. In step S31, the spatial position estimation unit 21 calculates the three-dimensional coordinate system S of these feature points. ３Ｄ The position above and the two-dimensional coordinate system S ２Ｄ In step S32, the travel trajectory acquisition unit 22 detects the left and right lane boundary lines of the lane on which the host vehicle 1 is traveling. For example, the travel trajectory acquisition unit 22 may detect the positions of the lane boundary lines from the map database 14 or an image of the lane LN captured by the camera 11a. In step S33, the travel trajectory acquisition unit 22 acquires the positions of the lane boundary lines as the position of the travel trajectory TR. For example, when determining whether an object on the left front of the host vehicle 1 intersects with the travel trajectory TR of the host vehicle 1, the position of the left lane boundary line may be acquired as the position of the travel trajectory TR, and when determining whether an object on the right front of the host vehicle 1 intersects with the travel trajectory TR of the host vehicle 1, the position of the right lane boundary line may be acquired as the position of the travel trajectory TR.

[0043] In step S34, the nearby position specifying unit 23a specifies nearby positions on the traveling trajectory that are near the feature point. In step S35, the tangent direction calculation unit 23b and the intersecting direction calculation unit 23c calculate at least one of the tangential direction and the intersecting direction. In step S36, the road surface gradient estimation unit 27 estimates the road surface gradient of the lane on which the host vehicle 1 is traveling. In step S37, the intersecting object determination unit 24 classifies feature points whose distance from the host vehicle 1 is closer than a threshold into first feature points. In step S38, the intersecting object determination unit 24 classifies feature points whose absolute value of the difference in elevation from the camera 11a is small into first feature points. The intersecting object determination unit 24 classifies the remaining feature points into second feature points. In step S39, the intersecting object determination unit 24 determines whether the object OB intersects with the traveling trajectory TR based on the ranging flow of the first feature point.

[0044] In step S40, the reference direction calculation unit 23d calculates a reference direction for the second feature point, assuming that the vehicle horizontal plane is parallel to the road surface gradient estimated in step S36. In step S41, the intersecting object determination unit 24 determines whether the object OB intersects with the traveling trajectory TR based on a comparison between the correction flow of the first feature point and the reference direction. In step S42, the intersecting object determination unit 24 determines whether the object OB intersects with the traveling trajectory TR based on the results of the intersection determination at the first feature point and the results of the intersection determination at the second feature point.

[0045] (Effects of the Embodiments) (1) In the object detection method, the optical flow of feature points of an object around the vehicle is detected from an image captured by a camera around the vehicle, the position of the object in a spatial coordinate system is estimated, the vehicle's travel trajectory in the spatial coordinate system is acquired, a nearby position on the travel trajectory is identified, and at least one of a tangent direction of the travel trajectory at the nearby position and an intersecting direction from the object's position to the nearby position is calculated. A reference direction in the image coordinate system is calculated by projecting at least one of the tangent direction and the intersecting direction onto the image coordinate system of the captured image. An optical flow corresponding to the movement amount of the vehicle is subtracted from the optical flow of the feature points to calculate a corrected flow. The direction of the corrected flow is compared with the reference direction to determine whether the object intersects with the travel trajectory. This improves the accuracy of object intersection determination in the captured image when the accuracy is poor. Furthermore, the intersection determination can be performed taking into account the shape of the lane and the travel trajectory depending on whether the vehicle is turning left or right.

[0046] (2) The road surface gradient of the lane on which the host vehicle is traveling may be estimated, and a reference direction on the image coordinate system may be calculated by assuming that the vehicle horizontal plane, which is a plane perpendicular to the height direction of the host vehicle, is parallel to the road surface gradient. This allows the reference direction to be calculated taking into account the tilt of the optical axis direction of the camera due to the road surface gradient. (3) The center line of the lane on which the host vehicle is traveling may be acquired as the traveling trajectory. This allows intersection determination to be performed taking into account the lane shape. (4) The lane boundary line of the lane on which the host vehicle is traveling may be acquired as the traveling trajectory. This allows intersection determination to be performed by reflecting the difference in curvature between the left and right sides of the lane.

[0047] (5) The positions of multiple feature points detected from the same object in a spatial coordinate system may be estimated, and for each of the multiple feature points, at least one of the tangential direction and the intersecting direction at the feature point's position in the spatial coordinate system may be calculated as a reference direction projected onto the image coordinate system. The direction of the correction flow may be compared with the reference direction for each of the multiple feature points to determine whether the object intersects with the traveling trajectory. This determination based on multiple feature points can eliminate the influence of the height direction of the feature points. Furthermore, it can absorb variations in the detection accuracy of individual optical flows. (6) The determination of whether the object intersects with the traveling trajectory may be based on a first component of the correction flow in a first reference direction in which the tangential direction is projected onto the image coordinate system and a second component of the correction flow in a second reference direction in which the intersecting direction is projected onto the image coordinate system. This allows the behavior of the object to be determined to be accurately determined.

[0048] (7) A first determination method for determining whether an object will intersect with the traveling trajectory based on the results of measurement of the object's behavior by a stereo camera and a second determination method for determining whether an object will intersect with the traveling trajectory based on a comparison of the direction of the correction flow with a reference direction may be switched depending on the position of the object. For example, the first determination method and the second determination method may be switched depending on the distance from the host vehicle to the object, or the first determination method and the second determination method may be switched depending on the absolute value of the difference between the height of the stereo camera and the height of the object. This allows a more accurate determination method to be selected to determine whether an object will intersect with the traveling trajectory.

[0049] All examples and conditional terms described herein are intended for educational purposes to aid the reader in understanding the present invention and the concepts provided by the inventor for the advancement of technology, and should be construed without limitation to the specifically described examples and conditions above, and the configuration of examples herein for illustrating the advantages and disadvantages of the present invention. Although the embodiments of the present invention have been described in detail, it should be understood that various changes, substitutions, and alterations can be made thereto without departing from the spirit and scope of the present invention.

[0050] 1... Vehicle, 10... Vehicle control device, 11... External sensor, 11a... Camera, 12... Vehicle sensor, 13... Positioning device, 14... Map database, 17... Actuator, 18... Controller, 18a... Processor, 18b... Storage device

Claims

1. From the captured images obtained by photographing the area around the vehicle with a camera, the optical flow of feature points of objects around the vehicle is detected. The position of the object in the spatial coordinate system is estimated, The vehicle's trajectory in the aforementioned spatial coordinate system is obtained, Identify a nearby position on the aforementioned travel trajectory that is in the vicinity of the object, The tangential direction of the travel trajectory at the aforementioned nearby position or at least one of the intersecting directions from the position of the object toward the aforementioned nearby position is calculated. The reference direction on the image coordinate system is calculated by projecting at least one of the tangential direction or the intersecting direction onto the image coordinate system of the captured image. The optical flow corresponding to the amount of movement of the vehicle is subtracted from the optical flow of the feature point to calculate the corrected flow. By comparing the direction of the correction flow with the reference direction, it is determined whether or not the object intersects with the travel trajectory. A method for detecting an object characterized by the following features.

2. The road surface gradient of the lane in which the vehicle is traveling is estimated, The reference direction on the image coordinate system is calculated by assuming that the vehicle horizontal plane, which is a plane perpendicular to the height direction of the vehicle, is parallel to the road surface gradient. The object detection method according to feature 1.

3. The object detection method according to claim 1 or 2, characterized in that the center line of the lane in which the vehicle is traveling is acquired as the travel trajectory.

4. The object detection method according to claim 1 or 2, characterized in that the lane boundary line of the lane in which the vehicle is traveling is acquired as the travel trajectory.

5. The position of each of the multiple feature points detected from the same object in the spatial coordinate system is estimated. For each of the plurality of feature points, the direction obtained by projecting at least one of the tangential direction or the intersecting direction at the position of the feature point in the spatial coordinate system onto the image coordinate system is calculated as the reference direction. By comparing the direction of the correction flow with the reference direction for each of the plurality of feature points, it is determined whether or not the object intersects with the travel trajectory. The object detection method according to claim 1 or 2, characterized by the features described above.

6. The object detection method according to claim 1 or 2, characterized in that it determines whether or not the object intersects the travel trajectory based on a first component of the correction flow in a first reference direction obtained by projecting the tangential direction onto the image coordinate system, and a second component of the correction flow in a second reference direction obtained by projecting the intersection direction onto the image coordinate system.

7. The object detection method according to claim 1 or 2, characterized in that it switches between a first determination method, which determines whether or not the object intersects the travel trajectory based on the measurement results of the object's behavior by a stereo camera, and a second determination method, which determines whether or not the object intersects the travel trajectory based on a comparison of the direction of the correction flow with the reference direction, depending on the position of the object.

8. The object detection method according to claim 7, characterized in that the first determination method and the second determination method are switched according to the distance from the vehicle to the object.

9. The object detection method according to claim 7, characterized in that the first determination method and the second determination method are switched according to the absolute value of the difference between the height of the stereo camera and the height of the object.

10. A camera that takes pictures of the area around the vehicle, A sensor that measures the amount of movement of the vehicle, A controller that detects the optical flow of feature points of an object around the vehicle from an image captured by the camera, estimates the position of the object in a spatial coordinate system, obtains the vehicle's trajectory in the spatial coordinate system, identifies a nearby position on the trajectory that is close to the object, calculates at least one of the tangential direction of the trajectory at the nearby position or the intersecting direction from the object's position to the nearby position, calculates a reference direction on the image coordinate system by projecting at least one of the tangential direction or the intersecting direction onto the image coordinate system of the captured image, calculates a corrected flow by subtracting the optical flow corresponding to the vehicle's movement from the optical flow of the feature points, and determines whether the object intersects the trajectory by comparing the direction of the corrected flow with the reference direction. An object detection device characterized by comprising the following features.