Object detection method and object detection device

Abstract

The present invention provides an object detection method and object detection apparatus that can smoothly detect objects present around a vehicle using a particle filter. [Solution] Optical flow of objects surrounding the vehicle is obtained from images captured by the vehicle's imaging device, and particles with initial velocities based on the velocity of the objects calculated from the optical flow are generated at positions corresponding to the distance measurement points of the objects.

Description

Technical Field

[0001] The present invention relates to an object detection method and an object detection device.

Background Art

[0002] There is known a device that tracks an object using a particle filter, which obtains a motion model of the object based on past tracking results, determines the positions of each particle based on the motion model, changes the amount and direction of movement of each particle in consideration of the assumed movement of the object, associates the positions of the particles with the tracking target object, obtains the association likelihood of each particle, and detects the position where the likelihood is highest as a whole as the position of the tracking target object (Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the above prior art, when the initial position of the particle is deviated from the position of the tracking target, it is necessary to repeat the process of updating the position of the particle according to the likelihood until the tracking target is accurately detected, and there is a problem that the tracking target cannot be detected smoothly.

[0005] The problem to be solved by the present invention is to provide an object detection method and an object detection device that can smoothly detect an object existing around a vehicle by a particle filter.

Means for Solving the Problems

[0006] The present invention solves the above problem by obtaining the optical flow of objects surrounding a vehicle from an image captured by the vehicle's imaging device, and generating particles with an initial velocity based on the velocity of the object calculated from the optical flow at positions corresponding to the distance measurement points of the objects. [Effects of the Invention]

[0007] According to the present invention, objects present around a vehicle can be smoothly detected by a particle filter. [Brief explanation of the drawing]

[0008] [Figure 1] A block diagram showing one embodiment of the object detection system according to the present invention. [Figure 2] This is a plan view showing an example of a driving scene in which an object is detected by the object detection system in Figure 1. [Figure 3] This is a plan view showing an example of an occupied grid map generated in the driving scene in Figure 2. [Figure 4A] This is a plan view (part 1) showing an example of object detection using a particle filter in the driving scene shown in Figure 2. [Figure 4B] This is a plan view (part 2) showing an example of object detection using a particle filter in the driving scene shown in Figure 2. [Figure 4C] This is a plan view (part 3) showing an example of object detection using a particle filter in the driving scene shown in Figure 2. [Figure 4D] This is a plan view (part 4) showing an example of object detection using a particle filter in the driving scene of Figure 2. [Figure 4E] This is a plan view (part 5) showing an example of object detection using a particle filter in the driving scene of Figure 2. [Figure 5] Figure 1 is a flowchart showing an example of the processing procedure in the object detection system. [Figure 6] Figure 1 is a flowchart showing another example of the processing procedure in the object detection system. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described below with reference to the drawings. In the following description, the front-rear direction of the vehicle will be simply referred to as the front-rear direction, the width direction of the vehicle will be simply referred to as the vehicle width direction, and the height direction of the vehicle will be simply referred to as the height direction.

[0010] [Object detection system configuration] Figure 1 is a block diagram showing one embodiment of the object detection system according to the present invention. The object detection system of this embodiment is a group of on-board devices for detecting objects around a vehicle using an occupied grid map. The occupied grid map, also called an occupied grid map, is a map that shows whether or not a section (hereinafter also called a cell or simply a section) defined by dividing the area in front of the vehicle in the direction of travel (hereinafter also called the forward area) into a grid is occupied (i.e., whether or not an object exists in the section).

[0011] The term "object" is not particularly limited to any object present around the vehicle (hereinafter also referred to as "object"). Examples of objects include roads and their surroundings, such as road lane markings, center lines, road markings, median strips, guardrails, curbs, traffic lights, and pedestrian crossings. Objects also include obstacles that may affect vehicle movement, such as other vehicles, motorcycles, bicycles, and pedestrians. Furthermore, objects include buildings such as houses, warehouses, office buildings, and bridges.

[0012] As shown in Figure 1, the object detection system 10 of this embodiment comprises a distance measuring device 11, an imaging device 12, and an object detection device 20. The devices constituting the object detection system 10 are connected to each other via CAN (Controller Area Network) or other in-vehicle LAN, and exchange information with each other.

[0013] The ranging device 11 detects the relative distance and relative speed between the vehicle and the object. The ranging device 11 includes a laser radar, a millimeter-wave radar, a LiDAR (light detection and ranging) unit, an ultrasonic radar, a sonar, etc. In order to suppress the occurrence of blind spots where the object cannot be detected, a plurality of ranging devices 11 are provided on one vehicle.

[0014] The imaging device 12 is a camera equipped with an imaging element such as a CCD, images the object, and generates an image including the object. The imaging device 12 includes a 3D camera, and may be a monocular camera or a stereo camera. Further, the imaging device 12 may be a ToF (Time of Flight) camera or a camera equipped with structured illumination. In order to suppress the occurrence of blind spots where the object cannot be imaged, a plurality of imaging devices 12 are provided on the front grille, side mirrors, rear bumper, etc. of the vehicle.

[0015] The ranging device 11 may scan electromagnetic waves in the vehicle width direction with respect to the object, and acquire information on the ranging point on the object from the received reflected waves. Similarly, the imaging device 12 may acquire information on the ranging point on the object by estimating the depth (that is, the distance from the imaging device 12 to the object on the image) at an arbitrary position in the image from one or more images. The electromagnetic waves scanned by the ranging device 11 include millimeter waves, infrared rays, lasers, etc.

[0016] The ranging point is a point on the object whose distance from at least one of the ranging device 11 and the imaging device 12 is measured. Examples of the information on the ranging point include the position information of the ranging point (information on the coordinates of the ranging point, information on the distance to the ranging point, etc.), information on the reflectivity of the electromagnetic wave at the ranging point, etc. Further, the ranging device 11 and the imaging device 12 may generate point cloud data (hereinafter, also simply referred to as point cloud data) in which the information is two-dimensionally arranged in the vehicle width direction and the height direction from the information on the ranging point of the object.

[0017] The object detection device 20 acquires the position information of the target object from the distance measurement device 11, acquires the image information from the imaging device 12, and recognizes the target object and the driving environment. The acquisition of information by the object detection device 20 is executed at a predetermined time interval (for example, every 0.1 to 1 millisecond). Further, the object detection device 20 may execute a process of generating point cloud data instead of the distance measurement device 11 and the imaging device 12.

[0018] The object detection device 2 controls the devices constituting the object detection system 10 to cooperate with each other, and detects the target object from the information acquired from the distance measurement device 11 and the imaging device 12. That is, the object detection device 20 integrates or synthesizes the information acquired from the distance measurement device 11 and the imaging device 12 to recognize the target object (or the driving environment).

[0019] The object detection device 20 is, for example, a computer, and includes a CPU (Central Processing Unit) which is a processor, a ROM (Read Only Memory) in which a program is stored, and a RAM (Random Access Memory) which functions as an accessible storage device. The CPU executes the program stored in the ROM and is an operation circuit for realizing the functions of the object detection device 20. Instead of or together with the CPU, an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), etc. may be used.

[0020] The object detection result by the object detection system 10 is output from the object detection device 20 to a driving support device (not shown) as information regarding the detection result, and is used for driving support by the driving support device. The driving support device drives the vehicle from the current position to the destination set by the vehicle occupant by autonomous driving control. Autonomous driving control is autonomous control of the driving operation of the vehicle, and the driving operation includes all driving operations such as acceleration, deceleration, starting, stopping, and steering. Further, as part of the driving support, the driving support device provides information regarding autonomous driving control to the vehicle occupant.

[0021] The driver assistance system uses devices mounted on the vehicle to perform autonomous control of the vehicle's driving motion, controlling the driving motion within a predetermined range. For driving motions not controlled by the driver assistance system, the driver performs manual operation. When the driver operates the vehicle manually, the driver assistance system does not perform autonomous control of the driving motion, and the vehicle's driving motion is controlled by the driver's operation. The driver assistance system may provide the vehicle occupants with information related to autonomous driving control (for example, information about the driving environment around the vehicle) while the driver is operating the vehicle manually.

[0022] [Object detection device functions] The ROM of the object detection device 20 stores a program for detecting objects by integrating or combining information acquired from the distance measuring device 11 and the imaging device 12. The CPU of the object detection device 20 executes this program to detect objects. Figure 1 shows, for convenience, the calculation unit 21, acquisition unit 22, generation unit 23, and detection unit 24 as functional blocks for detecting objects.

[0023] The processes performed by the calculation unit 21, acquisition unit 22, generation unit 23, and detection unit 24 will be explained below using Figures 2, 3, and 4A to 4E.

[0024] Figure 2 is a plan view showing an example of a driving scene in which an object is detected by the object detection system 10 of this embodiment. The X-axis, Y-axis, and Z-axis shown in Figure 2 correspond to the vehicle width direction, height direction, and longitudinal direction, respectively. In the following description, the coordinate system defined by the X-axis, Y-axis, and Z-axis shown in Figure 2 will also be referred to as the overhead coordinate system.

[0025] The road shown in Figure 2 is a two-lane road with lanes L1 and L2. Vehicles traveling in lane L1 travel from left to right in the figure (positive Z-axis direction), and vehicles traveling in lane L2 travel from right to left in the figure (negative Z-axis direction). Furthermore, the vehicle V shown in Figure 2 is assumed to be equipped with the object detection system 10 of this embodiment.

[0026] In the driving scene shown in Figure 2, vehicle V is traveling in the positive Z-axis direction at position P1 in lane L1. Building B is located to the left of vehicle V's direction of travel (negative X-axis direction), and a pedestrian crossing C is located in front of vehicle V's direction of travel. Furthermore, a pedestrian W is located in front of vehicle V's direction of travel (hereinafter also simply referred to as the direction of travel), approaching pedestrian crossing C to cross it. In the driving scene shown in Figure 2, vehicle V travels from position P1 to position P3 via position P2, and pedestrian W moves from position Q1 to position Q3 via position Q2.

[0027] In the driving scene shown in Figure 2, the object detection device 20 detects obstacles such as building B and pedestrian W from information acquired from the distance measuring device 11 and imaging device 12, and outputs information regarding the detection results to the driver assistance device (not shown). In order to detect obstacles present around the vehicle V, the object detection device 20 generates an occupied grid map using the functions of the calculation unit 21.

[0028] The calculation unit 21 divides (divides) the front area into a grid to generate sections. The front area may be a plane perpendicular to the height direction (for example, a plane corresponding to the road surface on which the vehicle travels), or it may be a space with length in the height direction. If the front area is a plane perpendicular to the height direction, the calculation unit 21 divides the front area along a grid that extends in the longitudinal direction and the vehicle width direction when the vehicle is viewed from above. On the other hand, if the front area is space, the calculation unit 21 sets a grid that extends in the vehicle width direction and the height direction in addition to the grid that extends in the longitudinal direction and the vehicle width direction, and divides the front area along the two grids.

[0029] The size of the forward area and the partition can be set to an appropriate size within a range that allows the object detection device 20 to repeatedly generate and update the occupied grid map at predetermined time intervals (for example, every 1 to 10 milliseconds). For example, if the forward area is a plane perpendicular to the height direction, the calculation unit 21 sets a forward area with a length of 50 to 100 m in the front-to-back direction and a length of 20 to 40 m in the vehicle width direction, and generates a rectangular partition with lengths of 10 to 30 cm in both the front-to-back and vehicle width directions.

[0030] In the driving scene shown in Figure 2, the calculation unit 21 divides the area R in front of the vehicle V into 12 equal parts in the vehicle width direction (X-axis direction) and 15 equal parts in the longitudinal direction (Z-axis direction) using the grid G, as shown in Figure 3. That is, the area R shown in Figure 3 is divided into 180 sections by the grid G. Each section shown in Figure 3 is assigned a number from 1 to 12 indicating its position in the vehicle width direction and an alphabet from a to o indicating its position in the longitudinal direction. In the following explanation, each section shown in Figure 3 will be represented by a combination of a number and an alphabet. For example, section Z enclosed by a dashed line in Figure 3 will be represented as section (9,m).

[0031] The calculation unit 21 calculates the probability (hereinafter also called the occupancy probability) that an object exists in a generated area based on the position information of the object's distance measurement point. The calculation unit 21 acquires the position information of the object's distance measurement point for a predetermined period (e.g., 1 to 10 milliseconds) according to the processing capacity of the object detection device 20, and calculates the probability that a distance measurement point existed in the area for each area during the predetermined period. For example, if the position information of the object's distance measurement point is acquired 10 times during the predetermined period, the calculation unit 21 calculates the occupancy probability of the area as the ratio of the number of times it was determined that a distance measurement point existed in the area out of the 10 measurements. The occupancy probability takes a value between 0 and 1.

[0032] The calculation unit 21 repeatedly calculates the occupancy probability at a certain time interval (for example, every 1 to 10 milliseconds) and updates the occupancy grid map. The time interval for updating the occupancy probability can be set to an appropriate time within a range that allows for avoidance of contact between the vehicle and obstacles. Furthermore, when calculating the occupancy probability, the calculation unit 21 may distinguish between obstacles (for example, other vehicles) and non-obstacle objects (for example, road markers), and to suppress unnecessary avoidance of non-obstacle objects, it may calculate the occupancy probability using only the position information of distance measurement points of objects that are located higher than the road surface on which the vehicle is traveling.

[0033] The calculation unit 21 determines, for example, that a section with an occupation probability of 1 or greater than a first predetermined value is occupied (i.e., an object exists in the section), and that a section with an occupation probability of less than a second predetermined value is not occupied (i.e., no object exists in the section). The first and second predetermined values ​​can be set to appropriate values ​​within a range that allows for proper determination of the occupation status of a section (i.e., "occupied" and "unoccupied"). For example, the first predetermined value can be set to 0.75 to 0.9, and the second predetermined value to 0.1 to 0.25.

[0034] Furthermore, if the occupancy probability is greater than or equal to the second predetermined value and less than the first predetermined value, the calculation unit 21 determines, for example, whether or not an object exists in the area based on information obtained from the image. Also, for areas where the position information of the object's distance measurement point cannot be obtained, the calculation unit 21 determines that the occupancy status is "unknown" because it cannot determine whether or not an object exists in the area.

[0035] In the driving scene shown in Figure 2, the calculation unit 21 acquires and temporarily stores positional information of distance measurement points of objects surrounding the vehicle V from the distance measuring device 11, and calculates the probability (occupancy probability) that a distance measurement point was present in each section shown in Figure 3 during a predetermined period. The calculation unit 21 uses only the positional information of distance measurement points of objects that are located higher than the road surface on which the vehicle V is traveling. Alternatively, the calculation unit 21 may acquire point cloud data from at least one of the distance measuring device 11 and the imaging device 12, and calculate the occupancy probability for each section shown in Figure 3 from the acquired point cloud data.

[0036] For example, if a distance measurement point corresponding to building B exists in section (1,b) for a predetermined period, the calculation unit 21 calculates the occupancy probability of section (1,b) as 1, and determines that section (1,b) is occupied (i.e., the occupancy status is "occupied") because the occupancy probability of section (1,b) is equal to or greater than the first predetermined value. In accordance with the determination result that section (1,b) is occupied, the calculation unit 21 applies dark hatching to section (1,b). The calculation unit 21 also performs the occupancy probability calculation process and the occupancy status determination process for other sections.

[0037] In the driving scene shown in Figure 2, if the position information of the distance measurement point of building B has been acquired by the distance measuring device 11 over a predetermined period, the calculation unit 21 applies dark hatching corresponding to the "occupied" status to sections (1,b) to (1,h), sections (2,b) to (2,h), sections (3,b) to (3,h), and sections (4,b) to (4,h), as shown in Figure 3. Similarly, if the position information of the distance measurement point of pedestrian W has been acquired over a predetermined period, the calculation unit 21 applies dark hatching to sections (4,i) and sections (5,i). In addition, in the driving scene shown in Figure 2, since objects in the area obscured by building B cannot be detected, the calculation unit 21 applies light hatching corresponding to the "unknown" occupancy status to sections (1,i) to (1,k), sections (2,i) to (2,j), and section (3,i).

[0038] The detection unit 24 detects objects (especially obstacles) from the occupied grid map. When the detection unit 24 tracks (detects) a moving object such as a pedestrian W, it uses, for example, a Bayesian filter, which is a method for determining the position of the object at each time as a probability distribution. Bayesian filters include Kalman filters, which are applicable to linear and Gaussian state-space models, and particle filters, which are applicable to nonlinear and / or non-Gaussian state-space models. A moving object is an object that is not stationary, and a stationary object is an object that remains stationary for a certain period of time (e.g., 30 seconds to 5 minutes) or longer.

[0039] In a particle filter, particles with arbitrary initial velocities are generated on a state-space model (e.g., on an occupied grid map). At predetermined time intervals (e.g., every 0.5 to 1 second), the particle positions are updated according to their velocities, and an observation is performed to determine whether or not an object exists at the corresponding particle positions. Based on the observation results, resampling is performed, in which particles located relatively far from the object disappear and new particles are generated in the vicinity of the object.

[0040] When the detection unit 24 uses a particle filter as a Bayesian filter, if the initial position of the particles in the particle filter is outside the position of the object being tracked, a relatively large number of resamplings will be required before the proportion of particles near the object being tracked becomes relatively high, which may prevent smooth tracking of the object. Therefore, in the object detection device 20 of this embodiment, when tracking (detecting) a moving object using a particle filter, optical flow is acquired from the image captured by the imaging device 12 (hereinafter also simply referred to as "image"), and the initial velocity of the particles in the particle filter is set based on the velocity of the object (especially the moving object) calculated from the optical flow, thereby enabling smooth detection of moving objects around the vehicle using the particle filter.

[0041] The acquisition unit 22 acquires the optical flow of an object from the image captured by the vehicle's imaging device 12. Optical flow is a two-dimensional vector that shows the movement (or change in position) of an object in an image when at least one of the vehicle on which the imaging device 12 is mounted or the object moves, and corresponds to the distribution of the object's apparent velocity in the image. Optical flow is also a pattern that shows the movement of an object between two temporally consecutive frames (images) in time-series image information.

[0042] The acquisition unit 22 acquires optical flow from the image using at least one of the block matching method and the gradient method. Examples of gradient methods include the Lucas-Kanade method and the Horn-Schunk method. The acquisition unit 22 may extract feature points from the image and acquire optical flow for each extracted feature point, or it may acquire optical flow for each pixel of the image.

[0043] Feature points are points that correspond to characteristic parts of an object and are used for object detection. Examples of feature points include points that correspond to the edges that capture the contour of the object, points that correspond to the corners of the object, and points that correspond to parts of the object where the brightness changes. The acquisition unit 22 extracts feature points from the image using known algorithms such as SIFT (Scale-Invariant Feature Transform).

[0044] The acquisition unit 22 acquires the optical flow of an object from an image at predetermined time intervals (for example, every 0.1 to 1 millisecond) according to the processing capacity of the object detection device 20. The acquisition unit 22 may set the frequency (time interval) for acquiring the optical flow to be shorter than the frequency (time interval) for acquiring the position information of the distance measurement point. This is because a shorter time between frames and smaller movement of the object on the image (or shorter distance traveled) allows for the acquisition of highly accurate optical flow, while a longer measurement time allows for the acquisition of highly accurate position information of the distance measurement point.

[0045] The acquisition unit 22 obtains, for example, a two-dimensional vector (i.e., optical flow) indicating the movement of an object between images from the latest image acquired from the imaging device 12 and an image acquired one frame before the latest image. Alternatively, the acquisition unit 22 may acquire images from the imaging device 12 at two consecutive time points, extract feature points from the image at the earlier time point, and obtain the optical flow corresponding to the feature points from the two acquired images at the two time points. The acquisition unit 22 obtains the optical flow using, for example, the Lucas-Kanade method.

[0046] In the driving scene shown in Figure 2, for example, the optical flow F1 shown in Figure 3 is acquired. In the driving scene shown in Figure 2, the vehicle V on which the imaging device 12 is mounted and the pedestrian W are moving, so the optical flow F1 includes components caused by the movement of the vehicle V and components caused by the movement of the pedestrian W. Of these, the component caused by the movement of the vehicle V is unnecessary for detecting obstacles present around the vehicle V, so the acquisition unit 22 subtracts the optical flow F2 caused by the movement of the vehicle V from the optical flow F1 to acquire the optical flow F3 caused by the movement of the pedestrian W. The optical flow F3 shown in Figure 3 is, for example, an optical flow corresponding to typical feature points of the pedestrian W.

[0047] The generation unit 23 generates particles having an initial velocity based on the velocity of the object calculated from the optical flow, at positions corresponding to the positions of the object's distance measurement points. The positions corresponding to the positions of the object's distance measurement points are, for example, positions corresponding to the positions of the distance measurement points, and are positions on the occupied grid map. The generation unit 23 associates the optical flow acquired from the image with sections, and calculates the velocity of the object from the optical flow for each section to which the optical flow is associated.

[0048] The generation unit 23 generates particles having the same initial velocity as the velocity of the object calculated from the optical flow. Alternatively, the generation unit 23 may calculate the speed and direction of movement of the object from the optical flow and set the initial velocity of the particles to have a speed obtained by multiplying the speed of the object by a predetermined coefficient, and to have a direction aligned with the direction of movement of the object. The predetermined coefficient can be set to an appropriate value within a range in which the detection unit 24 can smoothly detect the object, for example, 0.1 to 2. The angle (inferior angle) between the direction of the initial velocity and the direction of movement of the object may be less than a predetermined angle. The predetermined angle can be set to an appropriate value within a range in which the detection unit 24 can smoothly detect the object, for example, 0° to 10°.

[0049] The generation unit 23 may generate particles including a first particle having a first initial velocity that follows the velocity of the object calculated from the optical flow, and a second particle having a second initial velocity whose velocity distribution is a Gaussian distribution. The absolute value (speed) of the magnitude of the first initial velocity may be the speed obtained by multiplying the velocity of the object by a predetermined coefficient. The direction of the first initial velocity may be in line with the direction of movement of the object. The angle (sub-angle) between the direction of the first initial velocity and the direction of movement of the object may be less than a predetermined angle. On the other hand, the second particle having a second initial velocity whose velocity distribution and directional distribution are Gaussian distributions randomly walks on the occupied grid map.

[0050] The ratio of first particles to second particles is not particularly limited. For example, the generation unit 23 generates the same number of first particles and second particles. Alternatively, when generating first and second particles, the generation unit 23 may set the number of one type of particle to be greater (or less) than the number of the other type of particle.

[0051] If the state vector of the particles in the particle filter is represented by equation (1) below in the overhead coordinate system, then the state vectors of the first and second particles are represented by equations (2) and (3) below, respectively.

number

[0052] The definitions of the signs used in equations (1) to (3) above are as follows:

number

[0053] The optical flow F4 shown in Figure 4A is an example of the optical flow corresponding to a pedestrian W, acquired by the acquisition unit 22 in the driving scene shown in Figure 2. The optical flow F4 does not include components caused by the movement of the vehicle V. In the example shown in Figure 4A, the generation unit 23 associates the optical flow F4 with sections (4,i) and (5,i) and calculates the speed of the pedestrian W's movement from the optical flow F4. The generation unit 23 also generates first particles X1, X2, X3 having the same initial velocity as the pedestrian W's speed calculated from the optical flow F4, and second particles Y1, Y2, Y3 that perform a random walk, for sections (4,i) and (5,i) to which the optical flow F4 is associated.

[0054] In the example shown in Figure 4A, the number of first particles X1, X2, X3 and second particles Y1, Y2, Y3 are both the same, 3. The positions of the first particles X1, X2, X3 and second particles Y1, Y2, Y3 correspond to the positions of the distance measurement points of pedestrians W on the occupied grid map. If multiple optical flows are associated with a single section, the generation unit 23 sets the first initial velocity based, for example, on the average value (e.g., arithmetic mean) of the velocities of the multiple optical flows.

[0055] The detection unit 24 detects objects using a particle filter. Specifically, the detection unit 24 updates the position of particles on the occupied grid map based on information about the vehicle's behavior, information about the particle's characteristics, etc. For example, the detection unit 24 calculates the distance the particle has traveled relative to the vehicle from the elapsed time since the particle in the particle filter was generated until the present time. The detection unit 24 also calculates the distance the particle itself has traveled on the occupied grid map from the aforementioned elapsed time. Then, the detection unit 24 updates the position of the particle based on the relative distance the particle has traveled relative to the vehicle and the distance the particle itself has traveled on the occupied grid map. The detection unit 24 also causes the calculation unit 21 to acquire point cloud data at the time the particle's position is updated, and updates the occupancy probability of each section based on the newly acquired point cloud data.

[0056] The detection unit 24 increases or decreases the number of particles (first particles and / or second particles) present in the partition according to the updated occupancy probability. Alternatively, the detection unit 24 may maintain the number of particles present in the partition according to the updated occupancy probability. For example, if the updated occupancy probability is less than a second predetermined value, the detection unit 24 decreases the number of particles present in the partition; if the updated occupancy probability is greater than or equal to the second predetermined value and less than the first predetermined value, it increases or maintains the number of particles present in the partition; and if the updated occupancy probability is greater than or equal to the first predetermined value, it increases or maintains the number of particles present in the partition.

[0057] When the detection unit 24 decreases the number of particles in a section, it may preferentially decrease the first particle or the second particle, or it may decrease both the first and second particles equally. Similarly, when the detection unit 24 increases the number of particles in a section, it may preferentially increase the first particle or the second particle, or it may increase both the first and second particles equally. Furthermore, if the updated occupancy probability is greater than 0 and the number of particles present in the section is 0, the detection unit 24 may generate new first and second particles in the section, since a new object not tracked by the particle filter has appeared.

[0058] The detection unit 24 determines that the area is occupied if the number of particles present in the area is greater than or equal to a predetermined number. On the other hand, the detection unit 24 determines that the area is not occupied if the number of particles present in the area is less than the predetermined number. For areas where the position information of the object's distance measurement point cannot be obtained, the detection unit 24 determines that the area is occupied as "unknown," regardless of the number of particles present in the area. The predetermined number can be set to an appropriate value within a range that allows for proper determination of the area's occupation, for example, 3 to 6.

[0059] Furthermore, the detection unit 24 may determine that the occupancy status of a compartment is "occupied" if the number of particles present in the compartment is equal to or greater than a first predetermined number, determine that the occupancy status of a compartment is "unoccupied" if the number of particles present in the compartment is less than a second predetermined number, and determine that the occupancy status of a compartment is "unknown" if the number of particles present in the compartment is equal to or greater than a second predetermined number and less than a first predetermined number. The first predetermined number can be set to an appropriate value within a range that allows for proper determination of the occupancy status of a compartment, for example, 5 to 6. Similarly, the second predetermined number can be set to an appropriate value within a range that allows for proper determination of the occupancy status of a compartment, for example, 1 to 2.

[0060] In the driving scene shown in Figure 2, if vehicle V travels from position P1 to position P2 and pedestrian W moves from position Q1 to position Q2, the calculation unit 21 updates the occupancy probability of each section. Figure 4B is an occupancy grid map showing the occupancy probabilities updated by the calculation unit 21. In the occupancy grid map shown in Figure 4B, the occupancy probabilities of sections (1,a)~(1,f), (2,a)~(2,f), (3,a)~(3,f), and (4,a)~(4,f) included in range K1 are assumed to be 1, and the occupancy probability of sections (7,f)~(7,g) included in range K2 is assumed to be 0.6. Furthermore, in sections (1,g)~(1,i), (2,g)~(2,h), and (3,g) included in range K3, the position information of the distance measurement point cannot be obtained, and the occupancy probability of sections not included in ranges K1~K3 is assumed to be 0.

[0061] In this case, the detection unit 24 updates the positions of the first particles X1, X2, X3 and the second particles Y1, Y2, Y3 shown in Figure 4A, based on the travel time and distance traveled when the vehicle V shown in Figure 2 travels from position P1 to position P2, and the initial speeds set for the first particles X1, X2, X3 and the second particles Y1, Y2, Y3. The positions of each particle whose position has been updated by the detection unit 24 are shown on the occupied grid map in Figure 4B. Hereinafter, the sections included in range K1, the sections included in range K2, and the sections included in range K3 will also be referred to as the first section group, the second section group, and the third section group, respectively.

[0062] For example, in the example shown in Figure 4B, if the first predetermined value and the second predetermined value are set to 0.8 and 0.2, respectively, the occupancy probability of the first group of parcels is greater than or equal to the first predetermined value, and therefore the detection unit 24 determines that the occupancy status of the first group of parcels is "occupied". As shown in Figure 4C, the detection unit 24 applies dark hatching corresponding to the "occupied" occupancy status to parcels (1,a) to (1,f), (2,a) to (2,f), (3,a) to (3,f), and (4,a) to (4,f) of the occupancy grid map.

[0063] In a similar case, the occupancy probability of the second group of sections is greater than or equal to the second predetermined value and less than the first predetermined value, so the detection unit 24 increases the number of particles present in the second group of sections. Also, since the occupancy probabilities of section (6,g) where the second particle Y2 exists and section (5,i) where the second particle Y3 exists are less than the second predetermined value, the detection unit 24 eliminates at least one of the second particles Y2 and Y3. In the example shown in Figure 4C, the second particle Y3 that was present in section (5,i) disappears, and a new first particle X4 is generated in section (7,f). The first particle X4 has the same initial velocity as the pedestrian W's movement speed calculated from the optical flow F5 shown in Figure 4C.

[0064] The detection unit 24 calculates the number of particles contained in each section (6,g) and sections (7,f) to (7,g). If the predetermined number of particles used to determine the occupancy status of a section is set to 3, the detection unit 24 determines that the occupancy status of section (7,f), where the three first particles X1, X3, and X4 are located, is "occupied". As shown in Figure 4C, the detection unit 24 adds a dark hatch to section (7,f) on the occupancy grid map corresponding to the "occupied" occupancy status. On the other hand, the detection unit 24 determines that the occupancy status of section (6,g), where the second particle Y2 is located, and section (7,g), where the first particle X2 and the second particle Y1 are located, is "unoccupied". The detection unit 24 does not add hatching to sections (6,g) and (7,g).

[0065] Furthermore, since positional information of the distance measurement points cannot be obtained in the third group of parcels, the detection unit 24 determines that the occupation status of the third group of parcels is "unknown". As shown in Figure 4C, the detection unit 24 adds light hatching corresponding to the "unknown" occupation status to parcels (1,g) to (1,i), (2,g) to (2,h), and (3,g) of the occupation grid map. The detection unit 24 then recognizes that there is an obstacle at the location corresponding to the parcels with dark hatching in Figure 4C.

[0066] Furthermore, in the driving scene shown in Figure 2, if vehicle V travels from position P2 to position P3, the calculation unit 21 updates the occupancy probability of each section again. Figure 4D is an occupancy grid map showing the occupancy probabilities updated again by the calculation unit 21. In the occupancy grid map shown in Figure 4D, the occupancy probabilities of sections (1,a)~(1,d), (2,a)~(2,d), (3,a)~(3,d), and (4,a)~(4,d) included in range K4 are assumed to be 1, and the occupancy probabilities of sections (9,d) and (10,d) included in range K5 are assumed to be 0.6. Also, in sections (1,e)~(1,g), (2,e)~(2,f), and (3,e) included in range K6, the position information of the distance measurement point cannot be obtained, and the occupancy probability of sections not included in ranges K4~K6 is assumed to be 0.

[0067] In this case, the detection unit 24 updates the positions of the first particles X1, X2, X3, X4 and the second particles Y1, Y2 shown in Figure 4C, similar to the case shown in Figure 4B. The positions of each particle whose position has been updated by the detection unit 24 are shown on the occupied grid map in Figure 4D. Hereinafter, the sections included in range K4, the sections included in range K5, and the sections included in range K6 will also be referred to as the fourth section group, the fifth section group, and the sixth section group, respectively.

[0068] For example, in the example shown in Figure 4D, if the first predetermined value and the second predetermined value are set to 0.8 and 0.2, respectively, the probability of the fourth group of partitions being occupied is greater than or equal to the first predetermined value, and therefore the detection unit 24 determines that the occupation state of the fourth group of partitions is "occupied". As shown in Figure 4E, the detection unit 24 adds dark hatching corresponding to the "occupied" occupation state to partitions (1,a) to (1,d), partitions (2,a) to (2,d), partitions (3,a) to (3,d), and partitions (4,a) to (4,d) of the occupation grid map.

[0069] In a similar case, the occupancy probability of the fifth group of sections is greater than or equal to the second predetermined value and less than the first predetermined value, so the detection unit 24 increases the number of particles present in the fifth group of sections. Also, since the occupancy probability of section (8,f) where the second particle Y2 exists is less than the second predetermined value, the detection unit 24 eliminates the second particle Y2. In the example shown in Figure 4E, the second particle Y2 that was present in section (8,f) is eliminated, and a new first particle X5 is generated in section (10,d). The first particle X5 has the same initial velocity as the pedestrian W's movement velocity calculated from the optical flow F6 shown in Figure 4E.

[0070] The detection unit 24 calculates the number of particles contained in sections (9,d) and (10,d). If a predetermined number of particles used to determine the occupancy status of a section is set to 3, the detection unit 24 determines that the occupancy status of section (9,d), where the first particles X3, X4 and the second particle Y1 exist, and section (10,d), where the three first particles X1, X2, and X5 exist, is "occupied". As shown in Figure 4E, the detection unit 24 adds dark hatching to sections (9,d) and (10,d) of the occupancy grid map, corresponding to the "occupied" occupancy status.

[0071] Furthermore, since positional information of the distance measurement points cannot be obtained for the sixth group of parcels, the detection unit 24 determines that the occupation status of the sixth group of parcels is "unknown". As shown in Figure 4E, the detection unit 24 adds light hatching to parcels (1,e) to (1,g), parcels (2,e) to (2,f), and parcel (3,e) of the occupation grid map, corresponding to the "unknown" occupation status. The detection unit 24 recognizes that there is an obstacle at the location corresponding to the parcels with dark hatching in Figure 4E.

[0072] The generation unit 23 may generate first and second particles according to the number of optical flows contained in a section. That is, the generation unit 23 may set the number of first and second particles according to the number of optical flows contained in a section. For example, the generation unit 23 sets a larger number of first particles relative to the number of second particles as the number of optical flows contained in a section increases. Also, when updating the position of particles in the particle filter, the generation unit 23 may generate new first particles in sections where the number of particles increases, according to the number of optical flows.

[0073] The generation unit 23 may calculate the lateral velocity of the object in the width direction of the vehicle and the longitudinal velocity of the object in the direction of travel of the vehicle from the optical flow and the position information of the distance measurement point, and set the initial velocity of the particle (for example, a first initial velocity and a second initial velocity) based on the calculated lateral and longitudinal velocities. For example, the generation unit 23 generates a first particle in the overhead coordinate system shown in Figure 2, in which initial velocities are set that have a lateral velocity of the object in the X-axis direction and a longitudinal velocity of the object in the Z-axis direction.

[0074] Furthermore, if the generation unit 23 determines that the estimation accuracy of the object's velocity calculated from the optical flow is higher than a predetermined accuracy, it may set the initial velocity of the particles (e.g., a first initial velocity and a second initial velocity) based on the calculated transverse and longitudinal velocities. The estimation accuracy is, for example, the likelihood that the object's velocity calculated from the optical flow will actually be observed. The predetermined accuracy can be set to an appropriate value within the range in which the object can be properly tracked by the particle filter.

[0075] The generation unit 23 may generate particles with an initial velocity of 0 for an object determined to be stationary. For example, the generation unit 23 generates a first particle with an initial velocity set to 0 for an object determined to be stationary. The generation unit 23 may also generate a first particle with a first initial velocity of 0 and a second particle with a second initial velocity of 0 for an object determined to be stationary. Furthermore, if the generation unit 23 determines from the optical flow that the object is stationary, it may generate particles with an initial velocity of 0 for the object.

[0076] The detection unit 24 may generate a set of partitions from the partitions based on the number of particles present in each partition. For example, the detection unit 24 generates a set of partitions from partitions where the number of particles present in each partition is greater than or equal to a predetermined number. The detection unit 24 may also determine that a set of partitions is a moving object if its moving speed is greater than or equal to a predetermined threshold, and that a set of partitions is a stationary object if its moving speed is less than the predetermined threshold. The moving speed of a set of partitions is, for example, the average value (e.g., arithmetic mean) of the optical flow velocities associated with each partition constituting the set of partitions. The predetermined threshold can be set to an appropriate value within a range that allows for accurate distinction between moving objects and stationary objects, for example, 1 to 5 km / h.

[0077] If the generation unit 23 updates the position of a particle, it may assign a weight to the particles that were not erased by the update, according to the number of times they were not erased during the update. For example, in the examples shown in Figures 4A to 4E, the generation unit 23 assigns a weight of 3 to the first particles X1, X2, X3 and the second particle Y1, which were not erased twice during the update, and assigns a weight of 2 to the first particle X4, which was not erased once during the update. The detection unit 24 may also calculate the movement speed of the object using the particle weights. For example, in the example shown in Figure 4E, the detection unit 24 estimates the movement speed of the pedestrian W at position Q3 using the velocities of the first particles X1, X2, X3 and the second particle Y1, which are weighted 3.

[0078] In the above-described process, the object detection device 20 can, if necessary, convert the velocity and movement of the object on the image, calculated from the optical flow, into a top-down coordinate system or an absolute coordinate system using the camera parameters of the imaging device 12.

[0079] [Processing in object detection systems] Referring to Figures 5-6, the procedure for how the object detection device 20 processes information will be explained. The process described below is executed by the processor (CPU) of the object detection device 20 at predetermined time intervals (for example, every 0.1 to 1 millisecond).

[0080] Figure 5 is a flowchart showing an example of the processing steps performed in the object detection system 10 when tracking a moving object using a particle filter is initiated.

[0081] First, in step S1, the object detection device 20 sets up sections of the occupied grid map in the forward area. In the following step S2, the object detection device 20 acquires point cloud data from the distance measuring device 11. In the following step S3, the object detection device 20 associates the set sections with the points (distance measuring points) in the point cloud data. In the following step S4, the object detection device 20 calculates the occupancy probability for each section. In the following step S5, the object detection device 20 acquires images from the imaging device 12 at two consecutive time points.

[0082] In the following step S6, the object detection device 20 extracts feature points from the image at the earlier time of the two images. In the following step S7, the object detection device 20 obtains the optical flow from the two acquired images at the two time points. In the following step S8, the object detection device 20 associates the parcels with the optical flow. In the following step S9, the object detection device 20 calculates the velocity of the object from the optical flow. In the following step S10, the object detection device 20 generates first and second particles in the parcels to which the optical flow is associated.

[0083] Figure 6 is a flowchart showing an example of the processing procedure performed in the object detection system 10 when resampling of the particle filter is performed.

[0084] First, in step S11, the object detection device 20 updates the position of the particles in the occupied grid map based on information about the vehicle's behavior and the characteristics of the particles (for example, the initial velocity of the particles). In the following step S12, the object detection device 20 updates the occupancy probability of the area based on the point cloud data newly acquired from the distance measuring device 11. In the following step S13, the object detection device 20 determines for each area whether the updated occupancy probability is greater than 0 and whether the number of particles present in (or associated with) the area is 0.

[0085] If the updated occupancy probability is greater than 0 and it is determined that the number of particles present in the area is 0, the object detection device 20 proceeds to step S14 to generate a first particle and a second particle in the area, because a new object not tracked by the particle filter has appeared. On the other hand, if it is determined that at least one of the following conditions is met: the updated occupancy probability is 0 and the number of particles present in the area is greater than 0, the object detection device 20 proceeds to step S15 to increase or decrease the number of particles (first particle and / or second particle) present in the area according to the occupancy probability.

[0086] In step S16, the object detection device 20 determines whether the number of particles present in the area where the first and second particles were generated in step S14, and in the area where the number of particles was increased or decreased in step S15, is equal to or greater than a predetermined number. If it is determined that the number of particles present in the area is equal to or greater than a predetermined number, the object detection device 20 proceeds to step S17 and determines that the area is occupied. Conversely, if it is determined that the number of particles present in the area is less than a predetermined number, the object detection device 20 proceeds to step S18 and determines that the area is not occupied.

[0087] [Embodiments of the present invention] According to this embodiment, an object detection method is provided which involves acquiring the optical flow of objects present around a vehicle from an image captured by the vehicle's imaging device, generating particles having an initial velocity based on the velocity of the object calculated from the optical flow at positions corresponding to the distance measurement points of the objects, and detecting the objects using a particle filter with the particles, and an object detection device which executes the object detection method. This enables smooth detection of objects present around a vehicle using a particle filter.

[0088] In the object detection method and object detection device of this embodiment, particles are generated that include a first particle having a first initial velocity in line with the velocity of the object calculated from the optical flow, and a second particle having a second initial velocity whose velocity distribution is a Gaussian distribution, according to the number of optical flows included in a section defined by dividing the area in front of the vehicle in the direction of travel by a grid. This allows objects present around the vehicle to be smoothly detected by the particle filter.

[0089] In the object detection method and object detection device of this embodiment, the initial velocity is set based on the lateral velocity of the object in the width direction of the vehicle and the longitudinal velocity of the object in the direction of travel of the vehicle, which are calculated from the optical flow and the position information of the distance measurement point. This allows for the smooth detection of objects present around the vehicle by a particle filter in an overhead coordinate system.

[0090] In the object detection method and object detection apparatus of this embodiment, particles with an initial velocity of 0 are generated for objects that have been determined to be stationary. This allows for accurate identification of stationary objects.

[0091] In the object detection method and object detection device of this embodiment, at least one of the following is performed: setting the first initial velocity and the second initial velocity based on the lateral velocity in the width direction and the longitudinal velocity in the direction of travel of the vehicle, calculated from the optical flow and the position information of the distance measurement point; and generating the first particle with a first initial velocity of 0 and the second particle with a second initial velocity of 0 for the object determined to be stationary. This enables smooth detection of objects present around the vehicle by the particle filter.

[0092] In the object detection method and object detection device of this embodiment, if it is determined that the estimation accuracy of the object's velocity calculated from the optical flow is higher than a predetermined accuracy, the initial velocity is set based on the lateral velocity of the object in the width direction of the vehicle and the longitudinal velocity of the object in the direction of travel of the vehicle, which are calculated from the optical flow and the position information of the distance measurement point. This allows the particle filter to smoothly detect objects present around the vehicle.

[0093] In the object detection method and object detection device of this embodiment, the position of the particles is updated according to the behavior of the vehicle. This allows for a more accurate estimation of the particle's position.

[0094] In the object detection method and object detection apparatus of this embodiment, if it is determined from the optical flow that the object is stationary, particles with an initial velocity of 0 are generated relative to the object. This allows for accurate identification of stationary objects.

[0095] In the object detection method and object detection apparatus of this embodiment, when updating the position of the particles, the first particles are generated in the section where the number of particles increases, according to the number of optical flows. This allows for accurate identification of moving objects.

[0096] In the object detection method and object detection device of this embodiment, a set of partitions is generated from partitions defined by dividing the area in front of the vehicle's direction of travel into a grid, based on the number of particles present in each partition. If the movement speed of the set of partitions is greater than or equal to a predetermined threshold, it is determined that the set of partitions is a moving object. If the movement speed of the set of partitions is less than the predetermined threshold, it is determined that the set of partitions is a stationary object. This allows for more accurate detection of objects.

[0097] In the object detection method and object detection device of this embodiment, when the position of a particle is updated, a weight is assigned to the particles that were not erased by the update, according to the number of times they were not erased during the update, and the movement speed of the object is calculated using the weight. This allows the particle filter to detect objects present around the vehicle more smoothly. [Explanation of symbols]

[0098] 10...Object detection system, 11...Distance measuring device, 12...Imaging device, 20...Object detection device, 21...Calculation unit, 22...Acquisition unit, 23...Generation unit, 24...Detection unit, B...Building, C...Pedestrian crossing, F1, F2, F3, F4, F5, F6...Optical flow, G...Grid, K1, K2, K3, K4, K5, K6...Range, L1, L2...Lane, P1, P2, P3, Q1, Q2, Q3...Position, R...Area, V...Vehicle, W...Pedestrian, X1, X2, X3, X4, X5...First particle, Y1, Y2, Y3...Second particle, Z...District

Claims

1. In an object detection method performed by a vehicle object detection device, The object detection device is From the image captured by the vehicle's imaging device, the optical flow of objects surrounding the vehicle is obtained. A particle having an initial velocity based on the velocity of the object calculated from the optical flow is generated at a position corresponding to the position of the distance measurement point of the object. An object detection method comprising detecting the object using a particle filter with the aforementioned particles.

2. The object detection method according to claim 1, wherein the object detection device generates particles including a first particle having a first initial velocity in line with the velocity of the object calculated from the optical flow, and a second particle having a second initial velocity whose velocity distribution is a Gaussian distribution, in accordance with the number of optical flows included in a section defined by dividing the area in front of the direction of travel of the vehicle with a grid.

3. The object detection method according to claim 1, wherein the object detection device sets the initial speed based on the lateral velocity of the object in the width direction of the vehicle and the longitudinal velocity of the object in the direction of travel of the vehicle, which are calculated from the optical flow and the position information of the distance measurement point.

4. The object detection method according to claim 1 or 3, wherein the object detection device generates particles with an initial velocity of 0 for the object determined to be stationary.

5. The object detection device is The first initial speed and the second initial speed are set based on the lateral speed in the width direction of the vehicle and the longitudinal speed in the direction of travel of the vehicle, calculated from the optical flow and the position information of the distance measurement point. The object detection method according to claim 2, further comprising generating a first particle with a first initial velocity of 0 and a second particle with a second initial velocity of 0 for an object determined to be stationary.

6. The object detection method according to claim 1, wherein if the object detection device determines that the estimation accuracy of the velocity of the object calculated from the optical flow is higher than a predetermined accuracy, the initial velocity is set based on the lateral velocity of the object in the width direction of the vehicle and the longitudinal velocity of the object in the direction of travel of the vehicle, which are calculated from the optical flow and the position information of the distance measurement point.

7. The object detection method according to claim 2, wherein the object detection device updates the position of the particles in accordance with the behavior of the vehicle.

8. The object detection method according to any one of claims 1, 3, and 6, wherein the object detection device determines from the optical flow that the object is stationary, and generates the particle with an initial velocity of 0 relative to the object.

9. The object detection method according to claim 2, wherein when the object detection device updates the position of the particles, it generates the first particles in the section where the number of particles increases, in accordance with the number of optical flows.

10. The object detection device is From the area in front of the vehicle in the direction of travel, defined by a grid, a set of partitions is generated based on the number of particles present in each partition. If the movement speed of the aforementioned set of compartments is greater than or equal to a predetermined threshold, it is determined that the set of compartments is a moving object. The object detection method according to any one of claims 1 to 3, 5 to 6, and 9, wherein if the moving speed of the group of partitions is less than the predetermined threshold, the group of partitions is determined to be stationary.

11. The object detection device is If the position of the aforementioned particle is updated, the particles that were not erased by the update are weighted according to the number of times they were not erased during the update. An object detection method according to any one of claims 1 to 3, 5 to 6, and 9, wherein the movement speed of the object is calculated using the weighting described above.

12. An acquisition unit that acquires the optical flow of objects surrounding the vehicle from an image captured by the vehicle's imaging device, A generating unit that generates particles having an initial velocity based on the velocity of the object calculated from the optical flow at a position corresponding to the position of the distance measurement point of the object, An object detection device comprising a detection unit that detects the object using a particle filter with the aforementioned particles.

Images