Object detection method and object detection device

Abstract

In this object detection method, the optical flow of a feature point of an object in the periphery of a host vehicle is detected from a captured image obtained by imaging the periphery of the host vehicle by a camera (S1), the position of the object on a spatial coordinate system is estimated (S2), a travel trajectory of the host vehicle in the spatial coordinate system is acquired (S3), a vicinity position existing in the vicinity of the object is identified from among positions on the travel trajectory (S4), at least one of a tangential direction of the travel trajectory at the vicinity position and an intersecting direction extending from the position of the object toward the vicinity position is calculated (S5), at least one of the tangential direction and the intersecting direction is projected onto the image coordinate system of the captured image to calculate a reference direction on the image coordinate system (S6), the optical flow according to the movement amount of the host vehicle is subtracted from the optical flow of the feature point to calculate a correction flow (S7), and the direction of the correction flow is compared with the reference direction to determine whether or not the object intersects the travel trajectory (S8).

Description

[Technical Field]

[0001] This invention relates to an object detection method and an object detection device. [Background technology]

[0002] The image recognition device described in Patent Document 1 below calculates a corrected flow by subtracting the optical flow corresponding to the amount of movement of the vehicle from the optical flow extracted from the captured image, and determines a moving object that intersects the direction of travel of the vehicle when it is determined that there is no vanishing point on the extension of the corrected flow. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-34995 [Overview of the project] [Problems that the invention aims to solve]

[0004] However, since the direction of travel of the vehicle does not necessarily coincide with the direction of the vanishing point in the captured image, there is a risk that moving objects intersecting the direction of travel of the vehicle may not be properly identified depending on the direction of travel of the vehicle. The present invention aims to improve the accuracy of determining moving objects that intersect the direction of travel of a vehicle based on optical flow extracted from images captured from the area surrounding the vehicle. [Means for solving the problem]

[0005] In one embodiment of the present invention, an object detection method is used to detect the optical flow of feature points of objects around the vehicle from captured images obtained by photographing the area around the vehicle with a camera, estimate the position of the object in a spatial coordinate system, obtain the vehicle's trajectory in the spatial coordinate system, identify a nearby position on the trajectory that is close to the object, calculate at least one of the tangential direction of the trajectory at the nearby position or the intersecting direction from the object's position to the nearby position, calculate a reference direction on the image coordinate system by projecting at least one of the tangential direction or the intersecting direction onto the image coordinate system of the captured image, calculate a corrected flow by subtracting the optical flow corresponding to the amount of movement of the vehicle from the optical flow of the feature points, and determine whether or not the object intersects with the trajectory by comparing the direction of the corrected flow with the reference direction. [Effects of the Invention]

[0006] According to the present invention, the accuracy of determining moving objects that intersect the direction of travel of the vehicle can be improved based on optical flow extracted from captured images of the area around the vehicle. The objectives and advantages of the present invention are embodied and achieved using the elements and combinations thereof set forth in the claims. Both the general description above and the detailed description below are merely illustrative and descriptive, and should be understood not to limit the invention in any way that would be limited by the claims. [Brief explanation of the drawing]

[0007] [Figure 1] This is a schematic diagram of an example of a vehicle control device according to an embodiment. [Figure 2] (a) is a schematic diagram of the captured image, (b) is a schematic diagram of a point near the travel track, in the tangential direction and in the intersecting direction, and (c) is an explanatory diagram of an example of intersection detection. [Figure 3] This is an explanatory diagram of an example of an object detection method according to the first embodiment. [Figure 4] This is a block diagram of an example of the functional configuration of the controller in the second embodiment. [Figure 5] (a) to (c) are explanatory diagrams illustrating an example of intersection detection. [Figure 6]This is an explanatory diagram of an example of an object detection method according to the second embodiment. [Figure 7] Figures (a) to (e) are schematic diagrams illustrating an example of the object detection method of the third embodiment. [Figure 8] This is an explanatory diagram of an example of an object detection method according to the third embodiment. [Figure 9] This is an explanatory diagram illustrating an example of how feature points can be classified. [Figure 10] This is a block diagram of an example of the functional configuration of the controller in the fourth embodiment. [Figure 11] This is an explanatory diagram of an example of an object detection method according to the fourth embodiment. [Modes for carrying out the invention]

[0008] (First Embodiment) (composition) Figure 1 is a schematic diagram of an example of a vehicle control device according to an embodiment. The vehicle 1 is equipped with a vehicle control device 10 that controls the driving of the vehicle 1. For example, the vehicle control device 10 may perform autonomous driving control, which drives the vehicle 1 automatically without driver intervention, based on the driving environment around the vehicle 1, or driving assistance control, which assists the driver in driving the vehicle 1 by controlling at least one of the steering mechanism, driving force, or braking force of the vehicle 1. Driving assistance control may be, for example, automatic braking, preceding vehicle following control, constant speed driving control, merging assistance control, etc.

[0009] The vehicle control device 10 is object The system comprises a sensor 11, a vehicle sensor 12, a positioning device 13, a map database (map DB) 14, an actuator 17, and a controller 18. object The sensor 11 includes a camera 11a mounted on the vehicle 1 that captures images of the area around the vehicle 1. The camera 11a may be, for example, a stereo camera. Also, objectThe sensor 11 may include multiple different types of object detection sensors that detect objects around the vehicle 1, such as laser radar, millimeter-wave radar, and LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging).

[0010] The vehicle sensor 12 is mounted on the vehicle 1 and detects various information (vehicle signals) obtained from the vehicle 1. The vehicle sensor 12 includes, for example, a vehicle speed sensor that detects the vehicle speed of the vehicle 1, a wheel speed sensor that detects the rotational speed of the tires of the vehicle 1, an acceleration sensor that detects the acceleration and deceleration of the vehicle 1, a steering angle sensor that detects the steering angle of the steering wheel, a steering angle sensor that detects the turning angle of the steering wheels, a gyro sensor that detects the angular velocity of the vehicle 1, a yaw rate sensor that detects the yaw rate, an accelerator sensor that detects the accelerator opening of the vehicle, and a brake sensor that detects the amount of brake operation.

[0011] The positioning device 13 is equipped with a Global Navigation Satellite System (GNSS) receiver and measures the current position of the vehicle 1 by receiving radio waves from multiple navigation satellites. The GNSS receiver may be, for example, a GPS receiver. The positioning device 13 may also be, for example, an inertial navigation device. The map database 14 stores road map data. For example, the map database 14 may store high-precision map data (hereinafter simply referred to as "high-precision map") that is suitable as map information for autonomous driving.

[0012] The actuator 17 generates vehicle behavior by operating the vehicle's steering wheel, accelerator opening, and brake system in response to control signals from the controller 18. The actuator 17 comprises a steering actuator, an accelerator opening actuator, and a brake control actuator. The steering actuator controls the steering direction and amount of the vehicle's steering. The accelerator opening actuator controls the accelerator opening of the vehicle. The brake control actuator controls the braking operation of the vehicle's brake system.

[0013] The controller 18 is an electronic control unit that controls the driving of the vehicle 1. The controller 18 includes a processor 18a and peripheral components such as a storage device 18b. The processor 18a may be, for example, a CPU or an MPU. The storage device 18b may include a semiconductor storage device, a magnetic storage device, an optical storage device, etc. The storage device 18b may include registers, cache memory, and memory such as ROM and RAM used as main memory. The functions of the controller 18 described below are realized, for example, by the processor 18a executing a computer program stored in the memory device 18b. The controller 18 may also be formed by dedicated hardware for performing each of the information processing operations described below.

[0014] When controlling the movement of the vehicle 1, the controller 18 detects optical flow on the image captured by the camera 11a and detects moving objects that intersect with the planned trajectory of the vehicle 1 based on the optical flow detection results. In the following description, the planned trajectory of the vehicle 1 may be simply referred to as the "trajectory". Specifically, the controller 18 detects feature points of objects around the vehicle 1 from the image captured by the camera 11a and calculates the apparent optical flow of the feature points. Figure 2(a) is a schematic diagram of the image IM captured by the camera 11a. The hatched region LN indicates the lane in which the vehicle 1 is traveling, the reference code OB indicates objects around the vehicle 1, and the reference code P f The arrows indicate the characteristic points of object OB. Note that Figure 2(a) shows object OB, lane LN, and characteristic point P. f In addition, the running track TR of the vehicle 1, and the reference tangential direction DT described later. r and reference crossing direction DC r The correction flow OF is shown for the sake of explanation, but in reality, the running track TR and the reference tangential direction DT are shown. r and reference crossing direction DC r Note that the correction flow OF does not appear in the captured image IM.

[0015] Next, the controller 18 obtains the three-dimensional coordinate system S of the object OB surrounding the host vehicle 1 3D estimates the position on . For example, the controller 18 may measure the relative distance based on the parallax of feature points on a pair of captured images captured by a stereo camera used as the camera 11a, and measure the azimuth angle from the coordinate position of the object OB. The controller 18 calculates the three-dimensional coordinate system S based on the relative distance and the azimuth angle 3D estimates the position of the object OB on . For example, the controller 18 may use the feature point P of the object OB f as the position of the object OB for estimation. Note that the controller 18 may also measure the relative distance and azimuth angle of the object OB from a captured image captured by a monocular camera using a monocular stereo method. In addition, object the relative distance and azimuth angle may be measured based on a laser radar, millimeter-wave radar, LIDAR, or the like of the sensor 11. Furthermore, in the present specification, the three-dimensional coordinate system S 3D is exemplified by a camera coordinate system having the origin at the viewpoint of the camera 11a (the center point of the image sensor); however, a stationary coordinate system having the origin at a fixed point, such as a map coordinate system, may be used as the three-dimensional coordinate system S 3D instead.

[0016] The controller 18 projects the position of the object OB in the three-dimensional coordinate system S 3D onto the two-dimensional coordinate system S of a plane perpendicular to the vertical direction 2D , thereby estimating the position of the object OB in the two-dimensional coordinate system S 2D . In the present specification, the two-dimensional coordinate system S 2D is exemplified by a camera coordinate system having the origin at the viewpoint of the camera 11a (the center point of the image sensor); however, a stationary coordinate system having the origin at a fixed point may be used as the two-dimensional coordinate system S 2D instead. In the following description, the three-dimensional coordinate system S 3D and the two-dimensional coordinate system S 2D may be collectively referred to as a "spatial coordinate system". Furthermore, the controller 18 acquires the position of the travel track TR on which the host vehicle 1 is scheduled to travel in the two-dimensional coordinate system S 2DThe position above is obtained. For example, the controller 18 may obtain the driving trajectory TR from the map database 14 or from images of the lane LN captured by the camera 11a. Figure 2(b) shows the 2D coordinate system S. 2D The positions of the object OB and the travel track TR are shown.

[0017] Controller 18 controls the 2D coordinate system S 2D Among the positions on the upper travel track TR, the nearest position P is located near the object OB. adj To identify the object OB, the controller 18 identifies the nearest point P on the travel trajectory TR that is closest to the object OB. adj It may also be specified as the foot of the perpendicular line drawn from object OB onto the travel trajectory TR, and the nearby position P. adj It may also be specified as such. Furthermore, the nearby location P specified in this way adj If there are multiple candidates, the point closest to your vehicle 1 will be designated as the nearest location P. adj It may be specified as such. Controller 18 controls the nearby position P adj Two-dimensional coordinate system S of the travel trajectory TR in [location] 2D From the tangential direction DT above, or from the position of object OB to a nearby position P adj Two-dimensional coordinate system S heading towards 2D Calculate at least one of the crossing directions DC shown above.

[0018] Next, the controller 18 projects at least one of the tangential direction DT or the intersecting direction DC onto the image coordinate system SI of the captured image IM, thereby creating the reference tangential direction DT on the image coordinate system SI. r or reference crossing direction DC r Calculate each of these. Figure 2(a) shows the reference tangential direction DT. r and reference intersection direction DC r An example is shown below. Note that in the following explanation, the reference tangential direction DT is used. r and reference crossing direction DC r These are sometimes collectively referred to as the "reference direction." For example, the controller 18 controls the 3D coordinate system S 3D The feature point P shown above fBy converting the directions (vectors) parallel to the tangential direction DT and the intersecting direction DC at the position to directions (vectors) on the image coordinate system SI of the captured image IM, the reference tangential direction DT r and reference crossing direction DC r You may calculate each of them.

[0019] The controller 18 measures the vehicle behavior of its own vehicle 1 (amount of movement and change in attitude of its own vehicle 1) based on the vehicle signal detected by the vehicle sensor 12. The controller 18 measures the feature point P f Subtracting the optical flow corresponding to the vehicle behavior from the apparent optical flow, we obtain feature point P. f Calculate the corrected flow OF. The controller 18 determines whether object OB intersects with the travel trajectory TR by comparing the direction of the correction flow OF with the reference direction. For example, the controller 18 controls the correction flow OF and the reference tangential direction DT. r By comparing the following, it can be determined whether or not object OB intersects with the travel trajectory TR. For example, correction flow OF and reference tangential direction DT r The difference is greater than the threshold, and the correction flow OF is in the reference tangential direction DT. r If the direction is closer to the travel trajectory TR than the object OB, it can be determined that the object OB intersects with the travel trajectory TR. On the other hand, the correction flow OF and the reference tangential direction DT r The difference is less than the threshold, or the correction flow OF is in the reference tangential direction DT. r If the object OB is moving away from the travel path TR, it can be determined that the object OB does not intersect the travel path TR.

[0020] For example, the controller 18 controls the correction flow OF and the reference cross direction DC. r Whether or not object OB intersects with the travel trajectory TR can be determined by comparing the two. For example, the correction flow OF and the reference intersection direction DC. r If the difference between the object OB and the travel trajectory TR is less than the threshold, it can be determined that the object OB intersects with the travel trajectory TR. If the difference is greater than or equal to the threshold, it can be determined that the object OB does not intersect with the travel trajectory TR. For example, the controller 18 is in the reference tangential direction DT r and reference crossing direction DC r Whether or not object OB intersects with the travel trajectory TR can be determined by comparing both of these with the corrected flow OF. See Figure 2(c). For example, the controller 18 may calculate the reference tangential component ΔT and the reference intersecting component ΔC of the corrected flow OF. If the ratio of these components (ΔC / ΔT) is greater than or equal to a threshold, it can be determined that object OB intersects with the travel trajectory TR; if it is less than the threshold, it can be determined that object OB does not intersect with the travel trajectory TR.

[0021] (operation) Figure 3 is an explanatory diagram of an example of the object detection method of the first embodiment. In step S1, the controller 18 detects the apparent optical flow of feature points of objects around the vehicle 1 from the captured image. In step S2, the controller 18 estimates the position of the object in the spatial coordinate system. In step S3, the controller 18 obtains the travel trajectory that the vehicle 1 is scheduled to travel. In step S4, the controller 18 identifies a nearby position on the travel trajectory that is in the vicinity of object OB.

[0022] In step S5, the controller 18 calculates at least one of the tangential direction or the intersecting direction. In step S6, the controller 18 calculates the reference direction on the image coordinate system SI by projecting at least one of the tangential direction or the intersecting direction onto the image coordinate system SI. In step S7, the controller 18 calculates the corrected flow by subtracting the optical flow corresponding to the vehicle behavior from the apparent optical flow. In step S8, the controller 18 determines whether or not the object intersects the travel trajectory by comparing the direction of the corrected flow with the reference direction.

[0023] (Second Embodiment) Figure 4 is a block diagram of an example of the functional configuration of the controller 18 of the second embodiment. The controller 18 includes an optical flow calculation unit 20, a spatial position estimation unit 21, a travel trajectory acquisition unit 22, a direction calculation unit 23, an intersecting object determination unit 24, and a vehicle control unit 25. The optical flow calculation unit 20 calculates the characteristic points P of objects OB surrounding the vehicle 1 from the image captured by the camera 11a. f The apparent optical flow is calculated. Note that for the same object OB, multiple feature points P are considered. f If detected, feature point P is located at a position with a large difference in height from the optical axis of camera 11a. f The optical flow may be selected and adopted. The optical flow calculation unit 20 subtracts the optical flow corresponding to the vehicle behavior of the vehicle 1, measured based on the vehicle signal detected by the vehicle sensor 12, from the apparent optical flow to determine the feature point P f Calculate the corrected flow OF.

[0024] The spatial position estimation unit 21 identifies the feature point P of object OB. f 3D coordinate system S 3D The position above and the 2D coordinate system S 2D The position above is estimated. The track acquisition unit 22 uses the 2D coordinate system S of the track TR. 2D Get the position above. The direction calculation unit 23 includes a nearby position identification unit 23a, a tangential direction calculation unit 23b, an intersecting direction calculation unit 23c, and a reference direction calculation unit 23d. The nearby position identification unit 23a is a two-dimensional coordinate system S 2D Among the positions on the upper travel track TR, the nearest position P is located near the object OB. adj The tangential direction calculation unit 23b determines the nearby position P adj Two-dimensional coordinate system S of the travel trajectory TR in [location] 2D The tangential direction DT is calculated. The intersection direction calculation unit 23c calculates the distance from the position of object OB to the nearby position P. adj Two-dimensional coordinate system S heading towards 2D Calculate DC in the direction of the intersection shown above.

[0025] The reference direction calculation unit 23d calculates the reference tangential direction DT on the image coordinate system SI of the captured image IM by projecting the tangential direction DT and the intersecting direction DC onto the image coordinate system SI. r and reference crossing direction DC r The following are calculated. For example, the reference direction calculation unit 23d assumes that the optical axis direction of the camera 11a is when the road surface gradient of the lane LN on which the vehicle 1 is traveling is "0", and calculates the 3D coordinate system S 3D The feature point P shown above f By converting the directions (vectors) parallel to the tangential direction DT and the intersecting direction DC at the position to directions (vectors) on the image coordinate system SI of the captured image IM, the reference tangential direction DT r and reference crossing direction DC r You may calculate each of them.

[0026] The intersecting object determination unit 24 determines the direction of the correction flow OF and the reference direction (reference tangential direction DT). r , reference cross direction DC r By comparing the above, it is determined whether object OB intersects with the travel trajectory TR. For example, the intersecting object determination unit 24 compares the correction flow OF and the reference tangential direction DT. r By comparing the two, it can be determined whether or not object OB intersects with the travel trajectory TR. For example, as shown in Figure 5(a), the correction flow OF and the reference tangential direction DT can be used. r The difference is greater than the threshold, and the correction flow OF is in the reference tangential direction DT. r If the object OB is moving in a direction closer to the travel trajectory TR than the object OB, it can be determined that the object OB intersects with the travel trajectory TR. On the other hand, as shown in Figure 5(b), if the correction flow OF is in the reference tangential direction DT, r This is either in a direction away from the running track TR, or as shown in Figure 5(c), the correction flow OF and the reference tangential direction DT. r If the difference between the two is less than the threshold, it can be determined that object OB does not intersect with the travel trajectory TR. The intersecting object determination unit 24, as in the first embodiment, uses a correction flow OF and a reference intersection direction DC. r Whether or not object OB intersects the travel trajectory TR can be determined by comparing it with the reference tangential direction DT. r and reference crossing direction DC rWhether or not object OB intersects with the travel trajectory TR can be determined by comparing both of these with the correction flow OF.

[0027] The vehicle control unit 25 performs vehicle control by controlling at least one of the steering mechanism, drive unit, or braking unit of the vehicle 1 based on the determination result of the intersecting object determination unit 24. For example, if the vehicle control unit 25 determines that an object around the vehicle 1 intersects with the travel trajectory TR, it may control at least one of the steering mechanism, drive unit, or braking unit of the vehicle 1 to avoid the moving object.

[0028] Figure 6 is an explanatory diagram of an example of the object detection method of the second embodiment. The processing in steps S10 to S14 is the same as in steps S1 to S5 shown in Figure 3. In step S15, the reference direction calculation unit 23d calculates the reference tangential direction and the reference intersection direction, respectively, assuming that the road surface gradient of the lane LN on which the vehicle 1 is traveling is "0". The processing in steps S16 and S17 is the same as in steps S7 and S8 shown in Figure 3.

[0029] (Third embodiment) In the third embodiment, multiple feature points of the same object OB are grouped (clustered) by grouping (clustering) multiple feature points detected from the captured image IM. The optical flow of multiple feature points of the same object OB is detected, and by comparing these corrected flows with the reference direction, it is determined whether or not the object OB intersects with the travel trajectory TR. Figures 7(a) to 7(e) are schematic diagrams illustrating an example of the object detection method of the third embodiment. In the captured image IM in Figure 7(c), by grouping multiple feature points, the feature points P of the same object OB are identified. f1 , P f2 and P f3 The optical flow calculation unit 20 determines the characteristic points P of the same object OB. f1 , P f2 and P f3 The apparent optical flow is calculated, and the corrected flows OF1, OF2, and OF3 for these optical flows are calculated, respectively.

[0030] A spatial position estimation unit 21 estimates these feature points P f1 , P f2 and P f3 in a three-dimensional coordinate system S 3D positions P Pf31 , P Pf32 and P Pf33 (FIG. 7(b)) and positions P 2D in a two-dimensional coordinate system S Pf21 , P Pf22 and P Pf23 (FIG. 7(a)). A neighboring position specifying unit 23a is configured to, among positions on a travel trajectory TR in the two-dimensional coordinate system S 2D , specify neighboring positions P Pf21 , P Pf22 and P Pf23 that exist in the vicinity of positions P adj1 , P adj2 and P adj3 of an object OB, respectively. A tangential direction calculation unit 23b calculates tangential directions DT1, DT2 and DT3 adj1 , P adj2 and P adj3 of the travel trajectory TR at the neighboring positions P 2D in the two-dimensional coordinate system S. An intersection direction calculation unit 23c calculates intersection directions DC1, DC2 and DC3 Pf21 , P Pf22 and P Pf23 in the two-dimensional coordinate system S, which respectively direction from the positions P adj1 , P adj2 and P adj3 toward the neighboring positions P 2D .

[0031] FIG. 7(b) shows a state where directions (vectors) parallel to the tangential directions DT1, DT2 and DT3 and directions parallel to the intersection directions DC1, DC2 and DC3 are projected onto the positions P 3D , P f1 , P f2 and P f3 of the feature points in the three-dimensional coordinate system S Pf31 , P Pf32 and P Pf33 . A reference direction calculation unit 23d is configured to, for the positions P Pf31 , P Pf32and P Pf33 In this case, the directions parallel to the tangential directions DT1, DT2, and DT3 are transformed into directions on the image coordinate system SI of the captured image IM, thereby creating the reference tangential direction DT r1 , DT r2 and DT r3 Calculate each of these. Also, position P Pf31 , P Pf32 and P Pf33 In this case, the directions parallel to the intersecting directions DC1, DC2, and DC3 are transformed into directions on the image coordinate system SI of the captured image IM, thereby the reference intersecting direction DC r1 DC r2 and DC r3 Calculate each of them.

[0032] Note that the location P of the feature point Pf21 , P Pf22 and P Pf23 For each of these, the neighboring position P adj1 , P adj2 and P adj3 Instead of calculating the common neighboring position P, as shown in Figures 7(d) and 7(e), adj The following can also be calculated. For example, position P on the travel track TR. Pf21 ~P Pf23 The location with the smallest sum of distances from the common neighbor P is the common neighbor location. adj These positions P may also be calculated as follows, Pf21 ~P Pf23 One of the neighboring positions is neighboring position P adj It may also be selected as such. Furthermore, the neighboring position P adj The tangential direction and the intersecting direction can be calculated as a common tangential direction DT and intersecting direction DC. The reference direction calculation unit 23d calculates position P Pf31 , P Pf32 and P Pf33 In this case, the direction parallel to the common tangential direction DT is transformed into the direction on the image coordinate system SI of the captured image IM, thereby achieving the reference tangential direction DT r1 , DT r2 and DT r3 Calculate each of these. Similarly, position P Pf31 , P Pf32 and P Pf33In this case, the direction parallel to the common crossing direction DC is transformed into the direction on the image coordinate system SI of the captured image IM, thereby the reference crossing direction DC r1 DC r2 and DC r3 Calculate each of them. Furthermore, as shown in Figure 7(e), the position P of the feature point Pf21 , P Pf22 and P Pf23 From a common neighbor P adj Two-dimensional coordinate system S, each pointing towards 2D The crossing directions DC1, DC2, and DC3 above may be calculated separately.

[0033] The intersecting object detection unit 24 uses the directions of the correction flows OF1 to OF3 and the reference direction DT. r1 ~DT r3 DC r1 ~DC r3 By comparing these, it is determined whether or not object OB intersects with the travel trajectory TR. For example, the intersecting object determination unit 24 may perform an intersection determination for each correction flow OF1 to OF3 and obtain the final determination result by majority vote. Alternatively, for each correction flow OF1 to OF3, it may separate the object into a reference tangential direction component ΔT and a reference intersecting direction component ΔC, and determine whether or not object OB intersects with the travel trajectory TR based on the average ratio of these components.

[0034] Figure 8 is an explanatory diagram of an example of an object detection method according to the third embodiment. In step S20, the optical flow calculation unit 20 detects the apparent optical flow of a plurality of feature points detected from the captured image. In step S21, the spatial position estimation unit 21 calculates the three-dimensional coordinate system S of these feature points. 3D The position above and the 2D coordinate system S 2DThe above position is estimated. In step S22, the trajectory acquisition unit 22 estimates the position of the vehicle 1 on the map. For example, the trajectory acquisition unit 22 may estimate its own position based on the positioning result of the positioning device 13, or it may estimate its own position by odometry based on the vehicle signal detected by the vehicle sensor 12. In step S23, the trajectory acquisition unit 22 obtains position information of the lane in which the vehicle 1 is traveling from the map database 14 based on the estimated own position and the travel route plan, and obtains the center line of the journey as the trajectory.

[0035] In step S24, multiple feature points detected from the captured image are grouped together to group multiple feature points of the same object. In step S25, the proximity position identification unit 23a calculates individual or common proximity positions for each of the multiple feature points of the same object OB. In step S26, the tangential direction calculation unit 23b calculates individual or common tangential directions for each of the multiple feature points of the same object OB. The intersection direction calculation unit 23c calculates individual or common intersection directions for each of the multiple feature points of the same object OB. 27 In step S28, the reference direction calculation unit 23d calculates the reference direction on the image coordinate system SI for each feature point by projecting the tangential direction or intersection direction onto the image coordinate system SI. In step S28, the optical flow calculation unit 20 calculates the corrected flow of the apparent optical flow for each of the multiple feature points. In step S29, the intersecting object determination unit 24 determines whether or not object OB intersects with the travel trajectory TR by comparing the directions of the multiple corrected flows with the multiple reference directions.

[0036] (Fourth Embodiment) In the fourth embodiment, the intersecting object determination unit 24 classifies multiple feature points of the same object OB detected from the captured image IM into a first feature point for which the distance measurement accuracy by the stereo camera used as camera 11a is relatively high, and a second feature point for which the distance measurement accuracy is relatively low. Figure 9 is an explanatory diagram illustrating an example of feature point classification. For example, the intersecting object detection unit 24 may classify feature points Pf2, Pf3, Pf6, and Pf7, whose distance from the vehicle 1 is closer than a threshold, as first feature points. Alternatively, the intersecting object detection unit 24 may classify feature point Pf4 as a first feature point if the absolute value of the height difference with camera 11a or the absolute value of the height difference with the height HOA of the optical axis of camera 11a is within a threshold. The intersecting object detection unit 24 may classify the remaining feature points Pf1 and Pf5 as second feature points.

[0037] For the first feature point, the intersecting object determination unit 24 determines the two-dimensional coordinate system S based on the measurement results from the stereo camera. 2D The position of the first feature point above is measured. The intersecting object determination unit 24 is in a 2D coordinate system S 2D Based on the temporal change in the position of the first feature point shown above (which may be referred to as the "distance measurement flow" in the following description), it is determined whether or not object OB intersects with the travel trajectory TR. The intersection determination method based on the distance measurement flow is an example of the "first determination method" described in the claims. With respect to the second feature point, the intersecting object determination unit 24 determines whether or not object OB intersects with the travel trajectory TR based on a comparison of the correction flow on the captured image IM with the reference direction, similar to the first to third embodiments. The intersection determination method based on the comparison of the correction flow with the reference direction is an example of the "second determination method" described in the claims.

[0038] Figure 10 is a block diagram of an example of the functional configuration of the controller 18 of the fourth embodiment. In addition to the configuration shown in Figure 4, it is equipped with a road surface gradient estimation unit 27. The road surface gradient estimation unit 27 estimates the road surface gradient of the lane in which the vehicle 1 is traveling (for example, at least one of the road surface pitch angle or roll angle). The road surface gradient estimation unit 27 may estimate the road surface gradient of the lane in which the vehicle 1 is traveling based on the estimation result of the vehicle's own position and the map database 14. The reference direction calculation unit 23d assumes that the optical axis direction of the camera 11a is when the vehicle horizontal plane, which is a plane perpendicular to the height direction of the vehicle 1, is parallel to the road surface gradient, and projects the tangential direction calculated by the tangential direction calculation unit 23b and the intersecting direction calculated by the intersecting direction calculation unit 23c onto the image coordinate system SI, thereby determining the reference tangential direction and the reference Crossing Calculate the direction. Similarly, in the first to third embodiments described above, based on the estimated road surface gradient of the lane in which the vehicle 1 is traveling, the reference tangential direction and the reference Crossing You may calculate the direction.

[0039] The intersecting object determination unit 24 determines whether object OB intersects with the travel trajectory TR using the distance measurement flow for the first feature point. For example, the intersecting object determination unit 24 may determine whether object OB intersects with the travel trajectory TR by comparing the distance measurement flow with the tangential direction DT or the intersecting direction DC. For example, if the difference between the distance measurement flow and the tangential direction DT is greater than or equal to a threshold, and the distance measurement flow is in a direction closer to the travel trajectory TR than the tangential direction DT, it may be determined that object OB intersects with the travel trajectory TR. On the other hand, if the difference between the distance measurement flow and the tangential direction DT is less than a threshold, or if the distance measurement flow is in a direction further away from the travel trajectory TR than the tangential direction DT, it may be determined that object OB does not intersect with the travel trajectory TR.

[0040] For example, the intersecting object determination unit 24 may determine that object OB intersects with the travel trajectory TR if the difference between the distance measurement flow and the direction of intersection is less than a threshold, and determine that object OB does not intersect with the travel trajectory TR if the difference is greater than or equal to the threshold. Alternatively, for example, the intersecting object determination unit 24 may determine whether or not object OB intersects with the travel trajectory TR based on the ratio of the tangential component to the intersecting component of the distance measurement flow.

[0041] The intersecting object determination unit 24 determines, with respect to the second feature point, whether object OB intersects with the travel trajectory TR based on a comparison between the correction flow on the captured image IM and the reference direction, similar to the first to third embodiments. The object intersection determination unit 24 determines whether object OB intersects with the travel trajectory TR based on the result of the intersection determination at the first feature point and the result of the intersection determination at the second feature point. For example, the object intersection determination unit 24 may obtain the final determination result by majority vote of these intersection determinations, or it may determine whether object OB intersects with the travel trajectory TR based on the average of the angle difference between the two-dimensional distance measurement flow of the first feature point and the tangential or intersecting direction and the angle difference between the correction flow of the second feature point and the reference direction.

[0042] Figure 11 is an explanatory diagram of an example of an object detection method according to the fourth embodiment. In step S30, the optical flow calculation unit 20 detects the apparent optical flow of multiple feature points of the same object. In step S31, the spatial position estimation unit 21 calculates the three-dimensional coordinate system S of these feature points. 3D The position above and the 2D coordinate system S 2D The position above is estimated. In step S32, the trajectory acquisition unit 22 detects the left and right lane boundary lines of the lane in which the vehicle 1 is traveling. For example, the trajectory acquisition unit 22 may detect the position of the lane boundary lines from the map database 14 or from images of the lane LN taken by the camera 11a. In step S33, the trajectory acquisition unit 22 acquires the position of the lane boundary lines as the position of the trajectory TR. For example, when determining whether an object to the left front of the vehicle 1 intersects with the vehicle 1's trajectory TR, the position of the left lane boundary line may be acquired as the position of the trajectory TR, and when determining whether an object to the right front of the vehicle 1 intersects with the vehicle 1's trajectory TR, the position of the right lane boundary line may be acquired as the position of the trajectory TR.

[0043] In step S34, the proximity location identification unit 23a identifies a proximity location on the travel trajectory that is near the feature point. In step S35, the tangential direction calculation unit 23b and the intersection direction calculation unit 23c calculate at least one of the tangential direction or the intersection direction. In step S36, the road surface gradient estimation unit 27 estimates the road surface gradient of the lane on which the vehicle 1 is traveling. In step S37, the intersecting object detection unit 24 classifies feature points whose distance from the vehicle 1 is closer than a threshold as first feature points. In step S38, the intersecting object detection unit 24 classifies feature points with a small absolute difference in height from the camera 11a as first feature points. The intersecting object detection unit 24 classifies the remaining feature points as second feature points. In step S39, the intersecting object determination unit 24 determines whether object OB intersects with the travel trajectory TR based on the distance measurement flow of the first feature point.

[0044] In step S40, the reference direction calculation unit 23d calculates the reference direction for the second feature point, assuming that the vehicle horizontal plane is parallel to the road surface gradient estimated in step S36. In step S41, the intersecting object determination unit 24 calculates the second 2 The system determines whether object OB intersects with the travel trajectory TR based on a comparison of the correction flow of feature points with the reference direction. In step S42, the intersecting object determination unit 24 determines whether object OB intersects with the travel trajectory TR based on the result of the intersection determination at the first feature point and the result of the intersection determination at the second feature point.

[0045] (Effects of the embodiment) (1) In the object detection method, the optical flow of feature points of objects around the vehicle is detected from the captured image obtained by taking pictures of the area around the vehicle with a camera, the position of the object in the spatial coordinate system is estimated, the travel trajectory of the vehicle in the spatial coordinate system is obtained, a nearby position that is close to the object is identified from the position on the travel trajectory, at least one of the tangential direction of the travel trajectory at the nearby position or the intersecting direction from the position of the object toward the nearby position is calculated, at least one of the tangential direction or the intersecting direction is projected onto the image coordinate system of the captured image to calculate a reference direction in the image coordinate system, the optical flow corresponding to the amount of movement of the vehicle is subtracted from the optical flow of the feature points to calculate a corrected flow, and it is determined whether or not the object intersects with the travel trajectory by comparing the direction of the corrected flow with the reference direction. This allows for improved accuracy in captured images when the accuracy of object intersection detection is poor in the spatial coordinate system. Furthermore, intersection detection can be performed while considering lane shape and the vehicle's trajectory based on left and right turns.

[0046] (2) The road surface gradient of the lane in which the vehicle is traveling can be estimated, and the reference direction on the image coordinate system can be calculated by assuming that the vehicle horizontal plane, which is a plane perpendicular to the height direction of the vehicle, is parallel to the road surface gradient. This makes it possible to calculate the reference direction while taking into account the tilt of the camera in the direction of the optical axis due to the road surface gradient. (3) The center line of the lane in which the vehicle is traveling may be acquired as the travel path. This allows for intersection detection to be performed while taking the lane shape into consideration. (4) The lane boundary line of the lane in which the vehicle is traveling may be obtained as the travel trajectory. This allows intersection detection to be performed while taking into account the difference in curvature between the left and right sides of the lane.

[0047] (5) The position of each of the multiple feature points detected from the same object in the spatial coordinate system may be estimated, and for each of the multiple feature points, the direction obtained by projecting at least one of the tangential or intersecting directions at the position of the feature point in the spatial coordinate system onto the image coordinate system may be calculated as the reference direction, and by comparing the direction of the correction flow with the reference direction for each of the multiple feature points, it may be determined whether or not the object intersects with the travel trajectory. This makes it possible to eliminate the influence of the height direction of the feature points by making a determination based on multiple feature points. In addition, it is possible to absorb the variation in the detection accuracy of individual optical flows. (6) Whether or not an object intersects the travel trajectory may be determined based on the first component of the correction flow in the first reference direction obtained by projecting the tangential direction onto the image coordinate system, and the second component of the correction flow in the second reference direction obtained by projecting the intersection direction onto the image coordinate system. This allows for accurate determination of the behavior of the object being judged.

[0048] (7) A first determination method, which determines whether or not an object intersects with the travel trajectory based on the measurement results of the object's behavior by a stereo camera, and a second determination method, which determines whether or not an object intersects with the travel trajectory based on a comparison of the direction of the correction flow with the reference direction, may be switched depending on the position of the object. For example, the first determination method and the second determination method may be switched depending on the distance from the vehicle to the object, or the first determination method and the second determination method may be switched depending on the absolute value of the difference between the height of the stereo camera and the height of the object. This allows for the selection of a more accurate determination method to determine whether or not an object intersects with the travel path.

[0049] All examples and conditional terms set forth herein are intended for educational purposes to help the reader understand the concepts given by the inventors for the advancement of the invention and the art, and should be interpreted without limitation to the examples and conditions specifically described herein, as well as the configuration of examples relating to demonstrating the superiority and inferiority of the invention. Although embodiments of the invention are described in detail, it should be understood that various changes, substitutions, and modifications are possible without departing from the spirit and scope of the invention. [Explanation of Symbols]

[0050] 1...the vehicle itself, 10...vehicle control device, 11... object Sensors, 11a...Camera, 12...Vehicle sensors, 13...Positioning devices, 14...Map database, 17...Actuators, 18...Controllers, 18a...Processors, 18b...Storage devices

Claims

1. From the captured images obtained by photographing the area around the vehicle with a camera, the optical flow of feature points of objects around the vehicle is detected. The position of the object in the spatial coordinate system is estimated, The vehicle's trajectory in the aforementioned spatial coordinate system is obtained, Identify a nearby position on the aforementioned travel trajectory that is in the vicinity of the object, The tangential direction of the travel trajectory at the aforementioned nearby position or at least one of the intersecting directions from the position of the object toward the aforementioned nearby position is calculated. The reference direction on the image coordinate system is calculated by projecting at least one of the tangential direction or the intersecting direction onto the image coordinate system of the captured image. The optical flow corresponding to the amount of movement of the vehicle is subtracted from the optical flow of the feature point to calculate the corrected flow. By comparing the direction of the correction flow with the reference direction, it is determined whether or not the object intersects with the travel trajectory. A method for detecting an object characterized by the following features.

2. The road surface gradient of the lane in which the vehicle is traveling is estimated, The reference direction on the image coordinate system is calculated by assuming that the vehicle horizontal plane, which is a plane perpendicular to the height direction of the vehicle, is parallel to the road surface gradient. The object detection method according to feature 1.

3. The object detection method according to claim 1 or 2, characterized in that the center line of the lane in which the vehicle is traveling is acquired as the travel trajectory.

4. The object detection method according to claim 1 or 2, characterized in that the lane boundary line of the lane in which the vehicle is traveling is acquired as the travel trajectory.

5. The position of each of the multiple feature points detected from the same object in the spatial coordinate system is estimated. For each of the plurality of feature points, the direction obtained by projecting at least one of the tangential direction or the intersecting direction at the position of the feature point in the spatial coordinate system onto the image coordinate system is calculated as the reference direction. By comparing the direction of the correction flow with the reference direction for each of the plurality of feature points, it is determined whether or not the object intersects with the travel trajectory. The object detection method according to claim 1 or 2, characterized by the features described above.

6. The object detection method according to claim 1 or 2, characterized in that it determines whether or not the object intersects the travel trajectory based on a first component of the correction flow in a first reference direction obtained by projecting the tangential direction onto the image coordinate system, and a second component of the correction flow in a second reference direction obtained by projecting the intersection direction onto the image coordinate system.

7. The object detection method according to claim 1 or 2, characterized in that it switches between a first determination method, which determines whether or not the object intersects the travel trajectory based on the measurement results of the object's behavior by a stereo camera, and a second determination method, which determines whether or not the object intersects the travel trajectory based on a comparison of the direction of the correction flow with the reference direction, depending on the position of the object.

8. The object detection method according to claim 7, characterized in that the first determination method and the second determination method are switched according to the distance from the vehicle to the object.

9. The object detection method according to claim 7, characterized in that the first determination method and the second determination method are switched according to the absolute value of the difference between the height of the stereo camera and the height of the object.

10. A camera that takes pictures of the area around the vehicle, A sensor that measures the amount of movement of the vehicle, A controller that detects the optical flow of feature points of an object around the vehicle from an image captured by the camera, estimates the position of the object in a spatial coordinate system, obtains the vehicle's trajectory in the spatial coordinate system, identifies a nearby position on the trajectory that is close to the object, calculates at least one of the tangential direction of the trajectory at the nearby position or the intersecting direction from the object's position to the nearby position, calculates a reference direction on the image coordinate system by projecting at least one of the tangential direction or the intersecting direction onto the image coordinate system of the captured image, calculates a corrected flow by subtracting the optical flow corresponding to the vehicle's movement from the optical flow of the feature points, and determines whether the object intersects the trajectory by comparing the direction of the corrected flow with the reference direction. An object detection device characterized by comprising the following features.

Images