Object detection method and object detection device

Abstract

The present invention provides an object detection method and an object detection device capable of distinguishing between moving objects and stationary objects. [Solution] The probability of an object being present in a defined section is calculated by dividing the area in front of the vehicle V1 in the direction of travel into a grid pattern, based on the positional information of the distance measurement points of objects present around the vehicle V1. Optical flow F of objects present around the vehicle V1 is obtained from the image captured by the vehicle V1's imaging device 12. A set of sections is generated by associating multiple sections based on the probability of occupancy and at least one of the speed and amount of movement of the object in the vehicle width direction in the image, calculated from the optical flow F.

Description

Technical Field

[0001] The present invention relates to an object detection method and an object detection device.

Background Art

[0002] There is known an object recognition device that integrates a point cloud in which the distance between each point among the point clouds of surrounding objects is less than or equal to a predetermined threshold value to generate a plurality of clusters, combines clusters whose feature amounts of the points constituting each cluster are approximated, and recognizes an object based on the combined clusters (Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the above prior art, when a moving object exists in the vicinity of a stationary object, one cluster is generated from the points corresponding to the moving object and the points corresponding to the stationary object, and there is a problem that the moving object and the stationary object cannot be discriminated.

[0005] The problem to be solved by the present invention is to provide an object detection method and an object detection device capable of discriminating a moving object and a stationary object.

Means for Solving the Problems

[0006] The present invention solves the above problem by calculating the probability of an object being present in a defined section, which is created by dividing the area in front of the vehicle in the direction of travel into a grid, based on the positional information of the distance measurement points of objects present around the vehicle; obtaining the optical flow of objects present around the vehicle from images captured by the vehicle's imaging device; and generating a set of sections by associating multiple sections based on the probability of occupancy and at least one of the velocity in the vehicle width direction and the amount of movement of the object in the image, which are calculated from the optical flow. [Effects of the Invention]

[0007] According to the present invention, it is possible to distinguish between moving objects and stationary objects. [Brief explanation of the drawing]

[0008] [Figure 1] A block diagram showing one embodiment of the object detection system according to the present invention. [Figure 2] This is a plan view showing an example of a driving scene in which an object is detected by the object detection system in Figure 1. [Figure 3] This is a plan view showing an example of an occupied grid map generated in the driving scene in Figure 2. [Figure 4] This is a plan view showing an example of a set of partitions generated in the driving scene in Figure 2. [Figure 5] This figure shows an example of the process for generating a set of partitions according to this embodiment. [Figure 6] Figure 1 is a flowchart showing an example of the processing procedure in the object detection system. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described below with reference to the drawings. In the following description, the front-rear direction of the vehicle will be simply referred to as the front-rear direction, the width direction of the vehicle will be simply referred to as the vehicle width direction, and the height direction of the vehicle will be simply referred to as the height direction.

[0010] [Object detection system configuration] Figure 1 is a block diagram showing one embodiment of the object detection system according to the present invention. The object detection system of this embodiment is a group of on-board devices for detecting objects around a vehicle using an occupied grid map. The occupied grid map, also called an occupied grid map, is a map that shows whether or not a section (hereinafter also called a cell or simply a section) defined by dividing the area in front of the vehicle in the direction of travel (hereinafter also called the forward area) into a grid is occupied (i.e., whether or not an object exists in the section).

[0011] The term "object" is not particularly limited to any object present around the vehicle (hereinafter also referred to as "object"). Examples of objects include roads and their surroundings, such as road lane markings, center lines, road markings, median strips, guardrails, curbs, traffic lights, and pedestrian crossings. Objects also include obstacles that may affect vehicle movement, such as other vehicles, motorcycles, bicycles, and pedestrians. Furthermore, objects include buildings such as houses, warehouses, office buildings, and bridges.

[0012] As shown in Figure 1, the object detection system 10 of this embodiment comprises a distance measuring device 11, an imaging device 12, and an object detection device 20. The devices constituting the object detection system 10 are connected to each other via CAN (Controller Area Network) or other in-vehicle LAN, and exchange information with each other.

[0013] The rangefinder 11 detects the relative distance and relative speed between the vehicle and the object. The rangefinder 11 includes laser radar, millimeter-wave radar, LiDAR (light detection and ranging) unit, ultrasonic radar, sonar, etc. To suppress the occurrence of blind spots where the object cannot be detected, multiple rangefinders 11 are installed on a single vehicle.

[0014] The imaging device 12 is a camera equipped with an image sensor such as a CCD, which images an object and generates an image including the object. The imaging device 12 may include a 3D camera, and may be a monocular camera or a stereo camera. Furthermore, the imaging device 12 may be a ToF (Time of Flight) camera or a camera equipped with structured illumination. To suppress the occurrence of blind spots where the object cannot be imaged, multiple imaging devices 12 are installed on the vehicle's front grille, side mirrors, rear bumper, etc.

[0015] The distance measuring device 11 may scan electromagnetic waves in the vehicle width direction around an object and acquire information about the distance measuring point on the object from the received reflected waves. Similarly, the imaging device 12 may acquire information about the distance measuring point on an object by estimating the depth at any position in one or more images (i.e., the distance from the imaging device 12 to the object in the image) from one or more images. The electromagnetic waves scanned by the distance measuring device 11 include millimeter waves, infrared rays, lasers, and the like.

[0016] A distance measurement point is a point on an object whose distance to at least one of the distance measuring device 11 and the imaging device 12 is measured. Information regarding the distance measurement point includes location information of the distance measurement point (coordinate information of the distance measurement point, distance information to the distance measurement point, etc.) and information on the reflectivity of electromagnetic waves at the distance measurement point. The distance measuring device 11 and the imaging device 12 may also generate point cloud data (hereinafter simply referred to as point cloud data) in which the information regarding the distance measurement point of the object is arranged in two dimensions in the width direction and height direction.

[0017] The object detection device 20 acquires positional information of an object from the distance measuring device 11 and image information from the imaging device 12 to recognize the object and the driving environment. The acquisition of information by the object detection device 20 is performed at predetermined time intervals (for example, every 0.1 to 1 millisecond). In addition, the object detection device 20 may perform at least one of the following processes: acquiring information about the distance measurement points of the object and generating point cloud data, instead of the distance measuring device 11 and the imaging device 12.

[0018] The object detection device 20 controls the devices constituting the object detection system 10 to cooperate with each other, and detects the target object from the information acquired from the distance measurement device 11 and the imaging device 12. That is, the object detection device 20 integrates or synthesizes the information acquired from the distance measurement device 11 and the imaging device 12 to recognize the target object (or the driving environment).

[0019] The object detection device 20 is, for example, a computer, and includes a CPU (Central Processing Unit) which is a processor, a ROM (Read Only Memory) in which programs are stored, and a RAM (Random Access Memory) which functions as an accessible storage device. The CPU executes the programs stored in the ROM and is an operation circuit for realizing the functions of the object detection device 20. Instead of or together with the CPU, an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), etc. may be used.

[0020] The object detection result by the object detection system 10 is output from the object detection device 20 to a driving support device (not shown) as information regarding the detection result, and is used for driving support by the driving support device. The driving support device drives the vehicle from the current position to the destination set by the vehicle occupant by autonomous driving control. The autonomous driving control is autonomous control of the driving operation of the vehicle, and the driving operation includes all driving operations such as acceleration, deceleration, starting, stopping, and steering. In addition, as part of the driving support, the driving support device provides information regarding the autonomous driving control to the vehicle occupant.

[0021] The driver assistance system uses devices mounted on the vehicle to perform autonomous control of the vehicle's driving motion, controlling the driving motion within a predetermined range. For driving motions not controlled by the driver assistance system, the driver performs manual operation. When the driver operates the vehicle manually, the driver assistance system does not perform autonomous control of the driving motion, and the vehicle's driving motion is controlled by the driver's operation. The driver assistance system may provide the vehicle occupants with information related to autonomous driving control (for example, information about the driving environment around the vehicle) while the driver is operating the vehicle manually.

[0022] [Object detection device functions] The ROM of the object detection device 20 stores a program for detecting objects by integrating or combining information acquired from the distance measuring device 11 and the imaging device 12. The CPU of the object detection device 20 executes this program to detect objects. Figure 1 shows, for convenience, the calculation unit 21, acquisition unit 22, generation unit 23, and detection unit 24 as functional blocks for detecting objects.

[0023] The processes performed by the calculation unit 21, acquisition unit 22, generation unit 23, and detection unit 24 will be explained below with reference to Figures 2 to 4.

[0024] Figure 2 is a plan view showing an example of a driving scene in which an object is detected by the object detection system 10 of this embodiment. The X-axis, Y-axis, and Z-axis shown in Figure 2 correspond to the vehicle width direction, height direction, and longitudinal direction, respectively. In the following description, the coordinate system defined by the X-axis, Y-axis, and Z-axis shown in Figure 2 will also be referred to as the overhead coordinate system.

[0025] The road shown in Figure 2 is a two-lane road with lanes L1 and L2. Vehicles traveling in lane L1 travel from left to right in the figure (positive Z-axis direction), and vehicles traveling in lane L2 travel from right to left in the figure (negative Z-axis direction). Furthermore, it is assumed that the vehicle V1 shown in Figure 2 is equipped with the object detection system 10 of this embodiment.

[0026] In the driving scene shown in Figure 2, vehicle V1 is traveling in the positive Z-axis direction at position P1 in lane L1. Building B is located to the left of vehicle V1's direction of travel (negative X-axis direction), and a pedestrian crossing C is located ahead of vehicle V1's direction of travel. Furthermore, another vehicle V2 is parked at position P2 in lane L2 to the right of vehicle V1's direction of travel (positive X-axis direction), and a pedestrian W is approaching pedestrian crossing C to cross it, located ahead of vehicle V1's direction of travel (hereinafter simply referred to as the direction of travel). In the driving scene shown in Figure 2, the other vehicle V2 does not move, and pedestrian W moves from position P3 towards pedestrian crossing C in the direction of arrow A.

[0027] In the driving scene shown in Figure 2, the object detection device 20 detects obstacles such as building B, other vehicles V2, and pedestrians W from information acquired from the distance measuring device 11 and imaging device 12, and outputs information regarding the detection results to the driver assistance device (not shown). In order to detect obstacles present around vehicle V1, the object detection device 20 generates an occupied grid map using the functions of the calculation unit 21.

[0028] The calculation unit 21 divides (divides) the front area into a grid to generate sections. The front area may be a plane perpendicular to the height direction (for example, a plane corresponding to the road surface on which the vehicle travels), or it may be a space with length in the height direction. If the front area is a plane perpendicular to the height direction, the calculation unit 21 divides the front area along a grid that extends in the longitudinal direction and the vehicle width direction when the vehicle is viewed from above. On the other hand, if the front area is space, the calculation unit 21 sets a grid that extends in the vehicle width direction and the height direction in addition to the grid that extends in the longitudinal direction and the vehicle width direction, and divides the front area along the two grids.

[0029] The size of the forward area and the partition can be set to an appropriate size within a range that allows the object detection device 20 to repeatedly generate and update the occupied grid map at predetermined time intervals (for example, every 1 to 10 milliseconds). For example, if the forward area is a plane perpendicular to the height direction, the calculation unit 21 sets a forward area with a length of 50 to 100 m in the front-to-back direction and a length of 20 to 40 m in the vehicle width direction, and generates a rectangular partition with lengths of 10 to 30 cm in both the front-to-back and vehicle width directions.

[0030] In the driving scene shown in Figure 2, the calculation unit 21 divides the area R in front of the vehicle V1 into 12 equal parts in the vehicle width direction (X-axis direction) and 15 equal parts in the longitudinal direction (Z-axis direction) using the grid G, as shown in Figure 3. That is, the area R shown in Figure 3 is divided into 180 sections by the grid G. Each section shown in Figure 3 is assigned a number from 1 to 12 indicating its position in the vehicle width direction and an alphabet from a to o indicating its position in the longitudinal direction. In the following explanation, each section shown in Figure 3 will be represented by a combination of a number and an alphabet. For example, section Z enclosed by a dashed line in Figure 3 will be represented as section (9,m).

[0031] The calculation unit 21 calculates the probability (hereinafter also called the occupancy probability) that an object exists in a generated area based on the position information of the object's distance measurement point. The calculation unit 21 acquires the position information of the object's distance measurement point for a predetermined period (e.g., 1 to 10 milliseconds) according to the processing capacity of the object detection device 20, and calculates the probability that a distance measurement point existed in the area for each area during the predetermined period. For example, if the position information of the object's distance measurement point is acquired 10 times during the predetermined period, the calculation unit 21 calculates the occupancy probability of the area as the ratio of the number of times it was determined that a distance measurement point existed in the area out of the 10 measurements. The occupancy probability takes a value between 0 and 1.

[0032] The calculation unit 21 repeatedly calculates the occupancy probability at a certain time interval (for example, every 1 to 10 milliseconds) and updates the occupancy grid map. The time interval for updating the occupancy probability can be set to an appropriate time within a range that allows for avoidance of contact between the vehicle and obstacles. Furthermore, when calculating the occupancy probability, the calculation unit 21 may distinguish between obstacles (for example, other vehicles) and non-obstacle objects (for example, road markers), and to suppress unnecessary avoidance of non-obstacle objects, it may calculate the occupancy probability using only the position information of distance measurement points of objects that are located higher than the road surface on which the vehicle is traveling.

[0033] The calculation unit 21 determines that a section is occupied (i.e., an object exists in the section) if the occupation probability is equal to or greater than a first predetermined value, and determines that a section is not occupied (i.e., an object does not exist in the section) if the occupation probability is less than a second predetermined value. The first and second predetermined values ​​can be set to appropriate values ​​within a range that allows for proper determination of the occupation status (i.e., occupied and unoccupied) of a section. For example, the first predetermined value can be set to 0.75 to 0.9 and the second predetermined value to 0.1 to 0.25.

[0034] Furthermore, if the occupancy probability is greater than or equal to the second predetermined value and less than the first predetermined value, the calculation unit 21 determines, for example, whether or not an object exists in the area based on information obtained from the image. In addition, for areas where the position information of the object's distance measurement point cannot be obtained, the calculation unit 21 determines that the occupancy status is unknown because it cannot determine whether or not an object exists in the area.

[0035] In the driving scene shown in Figure 2, the calculation unit 21 acquires and temporarily stores positional information of distance measurement points of objects surrounding the vehicle V1 from the distance measuring device 11, and calculates the probability (occupancy probability) that a distance measurement point was present in each section shown in Figure 3 during a predetermined period. The calculation unit 21 uses only the positional information of distance measurement points of objects that are located at a position higher than the road surface on which the vehicle V1 is traveling. Alternatively, the calculation unit 21 may acquire point cloud data from at least one of the distance measuring device 11 and the imaging device 12, and calculate the occupancy probability for each section shown in Figure 2 from the acquired point cloud data.

[0036] For example, if a distance measurement point corresponding to building B exists in section (1,b) for a predetermined period, the calculation unit 21 calculates the occupancy probability of section (1,b) as 1, and determines that section (1,b) is occupied because the occupancy probability of section (1,b) is equal to or greater than the first predetermined value. In accordance with the determination result that section (1,b) is occupied, the calculation unit 21 applies a dark hatch to section (1,b). The calculation unit 21 also performs the occupancy probability calculation process and the occupancy status determination process for other sections.

[0037] In the driving scene shown in Figure 2, if the position information of the distance measurement point of building B has been acquired by the distance measurement device 11 over a predetermined period, the calculation unit 21 applies dark hatching to sections (1,b) to (1,h), sections (2,b) to (2,h), sections (3,b) to (3,h), and sections (4,b) to (4,h) according to the determination that they are occupied, as shown in Figure 3. Similarly, if the position information of the distance measurement point of another vehicle V2 has been acquired over a predetermined period, the calculation unit 21 applies dark hatching to sections (8,c) to (8,e) and sections (9,c) to (9,e), and if the position information of the distance measurement point of pedestrian W has been acquired over a predetermined period, the calculation unit 21 applies dark hatching to sections (4,i) and sections (5,i).

[0038] Furthermore, in the driving scene shown in Figure 2, since objects in the area blocked by building B cannot be detected, the calculation unit 21 applies a light hatch to sections (1,i) to (1,k), sections (2,i) to (2,j), and section (3,i) according to the determination that the occupation status is unknown. Similarly, since objects in the area blocked by other vehicle V2 cannot be detected, the calculation unit 21 applies a light hatch to sections (10,f) to (10,k), sections (11,g) to (11,o), and sections (12,g) to (12,o).

[0039] The detection unit 24 detects objects (especially obstacles) from the occupied grid map. When the detection unit 24 detects obstacles around the vehicle V1 using the occupied grid map shown in Figure 3, since sections (4,h) and (4,i) are adjacent, the detection unit 24 may detect sections (1,b) to (1,h), sections (2,b) to (2,h), sections (3,b) to (3,h), sections (4,b) to (4,i), and section (5,i) as a single obstacle, and may not be able to distinguish between building B and pedestrian W. Therefore, the object detection device 20 of this embodiment combines the occupied grid map with optical flow acquired from images captured by the vehicle's imaging device 12 (hereinafter also simply referred to as images) to accurately distinguish between a moving object (e.g., pedestrian W) moving in the vicinity of a stationary object (e.g., building B).

[0040] The acquisition unit 22 acquires the optical flow of an object from the image captured by the vehicle's imaging device 12. Optical flow is a two-dimensional vector that shows the movement (or change in position) of an object in an image when at least one of the vehicle on which the imaging device 12 is mounted or the object moves, and corresponds to the distribution of the object's apparent velocity in the image. Optical flow is also a pattern that shows the movement of an object between two temporally consecutive frames (images) in time-series image information.

[0041] The acquisition unit 22 acquires optical flow from the image using at least one of the block matching method and the gradient method. Examples of gradient methods include the Lucas-Kanade method and the Horn-Schunk method. The acquisition unit 22 may extract feature points from the image and acquire optical flow for each extracted feature point, or it may acquire optical flow for each pixel of the image.

[0042] Feature points are points that correspond to characteristic parts of an object and are used for object detection. Examples of feature points include points that correspond to the edges that capture the contour of the object, points that correspond to the corners of the object, and points that correspond to parts of the object where the brightness changes. The acquisition unit 22 extracts feature points from the image using known algorithms such as SIFT (Scale-Invariant Feature Transform).

[0043] The acquisition unit 22 acquires the optical flow of an object from an image at predetermined time intervals (for example, every 0.1 to 1 millisecond) according to the processing capacity of the object detection device 20. The acquisition unit 22 may set the frequency (time interval) for acquiring the optical flow to be shorter than the frequency (time interval) for acquiring the position information of the distance measurement point. This is because a shorter time between frames and smaller movement of the object on the image (or shorter distance traveled) allows for the acquisition of highly accurate optical flow, while a longer measurement time allows for the acquisition of highly accurate position information of the distance measurement point.

[0044] In the driving scene shown in Figure 2, the acquisition unit 22 obtains a two-dimensional vector (i.e., optical flow) indicating the movement of the object between images from the latest image acquired from the imaging device 12 and the image acquired one frame before the latest image. The acquisition unit 22 obtains the optical flow, for example, using the Lucas-Kanade method. The Lucas-Kanade method is based on the assumption that the movement of the object is smooth in the vicinity of a certain pixel (or feature point), and that a similar optical flow is obtained for the surrounding pixels (or surrounding feature points).

[0045] In the driving scene shown in Figure 2, the vehicle V1 equipped with the imaging device 12 and the pedestrian W are moving, so the acquired optical flow includes components caused by the movement of the vehicle V1 and components caused by the movement of the pedestrian W. Of these, the component caused by the movement of the vehicle V1 is unnecessary for detecting obstacles present around the vehicle V1, so the acquisition unit 22 removes the component caused by the movement of the vehicle V1 from the optical flow acquired from the two images to obtain the optical flows F1 and F2 shown in Figure 3. Optical flows F1 and F2 are optical flows calculated for representative feature points of the pedestrian W, respectively.

[0046] The generation unit 23 calculates at least one of the velocity and displacement of the object in the image (in particular, at least one of the velocity and displacement in the vehicle width direction) from the optical flow acquired by the acquisition unit 22. For example, the generation unit 23 extracts the vehicle width direction component from the optical flow, which is a vector, and calculates the displacement of the object in the vehicle width direction from the magnitude of this component. In this case, the generation unit 23 may calculate the velocity of the object in the vehicle width direction from the displacement of the object in the vehicle width direction using the time difference between when the two images used to acquire the optical flow were captured.

[0047] The generation unit 23 generates a set of associated sections based on at least one of the speed and amount of movement of the object (in particular, at least one of the speed and amount of movement in the vehicle width direction) and the occupancy probability calculated by the calculation unit 21. For example, the generation unit 23 associates the amount of movement of the object in the vehicle width direction with each section in which an object is determined to exist based on the occupancy probability, and determines whether the difference in the amount of movement associated with each section is less than a third predetermined value for two adjacent sections. The third predetermined value can be set to an appropriate value within a range in which objects (especially obstacles) around the vehicle can be accurately detected, for example, between 0 and 0.2.

[0048] If the difference in the amount of movement between two adjacent sections is less than a third predetermined value, the generation unit 23 associates the two adjacent sections and generates them as a single section set. On the other hand, if the difference in the amount of movement between two adjacent sections is equal to or greater than the third predetermined value, the generation unit 23 determines that the two adjacent sections belong to different section sets. The generation unit 23 repeats this determination process for two adjacent sections and generates a single section set corresponding to each obstacle present around the vehicle.

[0049] The generation unit 23 may associate a section with at least one of the velocity and movement of an object calculated from optical flow (in particular, at least one of the velocity and movement in the vehicle width direction), and label the section to which at least one of the velocity and movement is associated according to the associated velocity and movement. In this case, the generation unit 23 generates a set of sections corresponding to the labels from the labeled sections based on the occupancy probability and the section labels. The generation unit 23 may also perform at least one of generating a set of sections from adjacent sections and generating a set of sections from sections that are within a predetermined distance from one section. The predetermined distance can be set to an appropriate value within a range in which objects (especially obstacles) can be properly detected, for example, 0.1 to 0.5 m (1 to 100 pixels in the case of distance on an image).

[0050] In the example shown in Figure 3, the generation unit 23 calculates the amount of movement of the object (pedestrian W) in the vehicle width direction for each of the optical flows F1 and F2 acquired by the acquisition unit 22, and associates it with sections (4,i) and (5,i). If there are multiple optical flows in a given section, the generation unit 23 associates the average value (arithmetic mean) of the amount of movement (or velocity) of each optical flow in that section. For example, if the length of optical flow F1 in the vehicle width direction is 1.3m and the length of optical flow F2 in the vehicle width direction is 1.1m, the generation unit 23 associates a movement of (1.3 + 1.1) / 2 = 1.2m with sections (4,i) and (5,i).

[0051] The length of the optical flow in the vehicle width direction may be calculated using the actual length (in meters) in the overhead coordinate system or absolute coordinate system, or it may be calculated using the length corresponding to the number of pixels in the image (in pixels).

[0052] Figure 4 is an occupied grid map in which the amount of movement in the vehicle width direction calculated from optical flow is associated with the areas where an object is determined to exist. In addition to the optical flows F1 and F2 shown in Figure 3, the occupied grid map in Figure 4 reflects the amount of movement in the vehicle width direction of optical flow (not shown) caused by the behavior of the vehicle V1 and measurement errors of the distance measuring device 11.

[0053] The generation unit 23 assigns a label to each section associated with a vehicle width direction, according to the amount of movement. For example, the generation unit 23 assigns labels for moving objects to sections (4,i) and (5,i) where the amount of movement is greater than or equal to a fourth predetermined value (e.g., 0.3m), and labels for stationary objects to sections (1,b) to (1,h), sections (2,b) to (2,h), sections (3,b) to (3,h), sections (4,b) to (4,h), sections (8,c) to (8,e), and sections (9,c) to (9,e).

[0054] Furthermore, as shown in Figure 4, the generation unit 23 generates a set of sections X1 consisting of sections (1,b) to (1,h), sections (2,b) to (2,h), sections (3,b) to (3,h), and sections (4,b) to (4,h), which are labeled as stationary objects, and generates a set of sections X2 consisting of sections (8,c) to (8,e) and sections (9,c) to (9,e), which are labeled as stationary objects. In addition, the generation unit 23 generates a set of sections X3 consisting of sections (4,i) and (5,i), which are labeled as moving objects.

[0055] The detection unit 24 detects objects corresponding to the set of partitions generated by the generation unit 23. For example, the detection unit 24 detects objects (especially obstacles) corresponding to the set of partitions based on the labels attached to the partitions by the generation unit 23. In the example shown in Figure 4, the detection unit 24 detects stationary objects located at positions corresponding to the set of partitions X1 and X2, and detects moving objects located at positions corresponding to the set of partitions X3.

[0056] Furthermore, the first process in which the calculation unit 21 generates an occupied grid map and calculates the occupancy probability, and the second process in which the acquisition unit 22 acquires the optical flow, may be executed independently (separately). That is, the first process may be executed before the second process, the second process may be executed before the first process, or the first and second processes may be executed in parallel.

[0057] The generation unit 23 may determine that a section where the absolute value of the associated vehicle width direction velocity is equal to or greater than a predetermined absolute value (hereinafter also referred to as the first section) and a section where the associated vehicle width direction movement amount is equal to or greater than a predetermined movement amount (hereinafter also referred to as the second section) are sections corresponding to moving objects. For example, the generation unit 23 labels the first section and the second section as moving objects. The generation unit 23 may also determine that a section where the absolute value of the associated vehicle width direction velocity is less than a predetermined absolute value (hereinafter also referred to as the third section) and a section where the associated vehicle width direction movement amount is less than a predetermined movement amount (hereinafter also referred to as the fourth section) are sections corresponding to stationary objects. For example, the generation unit 23 labels the third section and the fourth section as stationary objects.

[0058] A stationary object is an object that remains stationary for a certain period of time (e.g., 30 seconds to 5 minutes). A moving object is an object that is in motion and is not a stationary object. The predetermined absolute value can be set to an appropriate value within the range in which moving objects can be properly detected, for example, 1 to 5 km / h. The predetermined amount of movement can be set to an appropriate value within the range in which moving objects can be properly detected, for example, 0.5 to 1 m. The absolute value (speed) of the optical flow in the vehicle width direction may be calculated using the actual speed (unit: km / h) in the overhead coordinate system or absolute coordinate system, or it may be calculated using the speed corresponding to the number of pixels on the image (unit: pixels / s).

[0059] The generation unit 23 may determine whether a section corresponds to a moving object moving to the right relative to the vehicle, a moving object moving to the left relative to the vehicle, or a stationary object. For example, in the occupied grid map shown in Figure 4, the generation unit 23 sets a positive (positive X-axis direction) first determination speed for determining a moving object moving to the right relative to the vehicle, and a negative (negative X-axis direction) second determination speed for determining a moving object moving to the left relative to the vehicle. The generation unit 23 then labels sections associated with a speed equal to or greater than the first determination speed as moving objects moving to the right relative to the vehicle, sections associated with a speed less than the second determination speed as moving objects moving to the left relative to the vehicle, and sections associated with a speed equal to or greater than the second determination speed but less than the first determination speed as stationary objects.

[0060] The generation unit 23 may generate a histogram of at least one of the vehicle width direction velocity and vehicle width direction displacement, set thresholds corresponding to the inflection points of the histogram's approximation curve, and generate a set of sections using the set thresholds. For example, if the vehicle width direction velocity (hereinafter also referred to as lateral velocity) associated with each section is plotted in order of velocity magnitude and the histogram shown in Figure 5 is generated, the generation unit 23 generates an approximation curve K and sets thresholds Kc and Kd corresponding to the inflection points Ka and Kb of the approximation curve K, respectively. The generation unit 23 then generates a set of sections consisting of sections where the lateral velocity is less than threshold Kc, a set of sections consisting of sections where the lateral velocity is greater than or equal to threshold Kc and less than threshold Kd, and a set of sections consisting of sections where the lateral velocity is greater than or equal to threshold Kd.

[0061] The generation unit 23 may generate a set of partitions from unlabeled partitions based on the occupancy probability and at least one of the vehicle width direction velocity and distance traveled, and may label the generated set of partitions based on at least one of the vehicle width direction velocity and distance traveled. When generating a set of partitions from unlabeled partitions, the generation unit 23 may use a clustering method that does not specify the number of partition sets to generate, such as a Gaussian mixture model, hierarchical clustering, or X-means, or it may use a clustering method that specifies the number of partition sets to generate in advance, such as k-means. For example, the generation unit 23 may set the number of partition sets to generate to a predetermined number (e.g., 3 to 5), generate a set of partitions from unlabeled partitions, and then label the generated set of partitions based on at least one of the vehicle width direction velocity and distance traveled.

[0062] The calculation unit 21 may calculate the occupancy probability using a particle filter. For example, in the occupancy grid map shown in Figure 3, the calculation unit 21 scatters discrete samples of particles according to a predetermined probability distribution, observes whether or not an object (building B, pedestrian W, etc.) exists at the location where the particles are present, estimates the location of the object based on the observation results, and generates new particles (resampling) based on the estimated location. By repeating this process, the occupancy probability is calculated. The occupancy probability is calculated, for example, as the ratio of particles present in a section to the total number of particles. The predetermined probability distribution is, for example, a Gaussian distribution.

[0063] If the lateral velocity corresponding to one section cannot be calculated from the optical flow, the generation unit 23 may use the lateral velocities corresponding to the surrounding sections to interpolate the lateral velocity corresponding to one section. Also, if the amount of movement in the vehicle width direction corresponding to one section cannot be calculated from the optical flow, the generation unit 23 may use the amount of movement in the vehicle width direction corresponding to the surrounding sections to interpolate the amount of movement in the vehicle width direction corresponding to one section.

[0064] When supplementing the vehicle width direction velocity and displacement, the generation unit 23 may use the optical flow with the closest Euclidean distance from the compartment to supplement the vehicle width direction velocity and displacement, or it may use the vehicle width direction velocity and displacement associated with surrounding compartments to supplement the vehicle width direction velocity and displacement. For example, if the lateral velocity of compartment (9,d) shown in Figure 4 cannot be calculated from the optical flow, the generation unit 23 will use the average value of the lateral velocity of compartment (9,c) and the lateral velocity of compartment (9,e) as the lateral velocity of compartment (9,d).

[0065] The generation unit 23 may use behavioral information regarding the vehicle's behavior to convert the relative velocity of the object with respect to the imaging device 12, calculated from optical flow, into an absolute velocity in a fixed coordinate system. For example, the generation unit 23 may add the vehicle's travel speed to the relative velocity of the object with respect to the vehicle, calculated from optical flow, based on the vehicle's travel speed obtained from a vehicle speed sensor (not shown), to determine the velocity of the object in a fixed coordinate system with respect to an arbitrary fixed point.

[0066] In the above-described process, the object detection device 20 can, if necessary, convert the velocity and movement of the object on the image, calculated from the optical flow, into a top-down coordinate system or an absolute coordinate system using the camera parameters of the imaging device 12.

[0067] [Processing in object detection systems] Referring to Figure 6, the procedure for processing information by the object detection device 20 will be explained. Figure 6 is a flowchart showing an example of a processing procedure performed in the object detection system 10. The processing described below is performed by the processor (CPU) of the object detection device 20 at predetermined time intervals (for example, every 0.1 to 1 millisecond).

[0068] First, in step S1, the object detection device 20 acquires point cloud data from the distance measuring device 11, and in the following step S2, it associates the points in the point cloud data with the plots of the occupied grid map based on the positional information of the point cloud data. In step S3, the object detection device 20 calculates the occupancy probability for each plot based on the positional information of the point cloud data. In step S4, the object detection device 20 acquires an image from the imaging device 12, and in the following step S5, it acquires the optical flow from the image.

[0069] In step S6, the object detection device 20 calculates the lateral velocity from the acquired optical flow, and in the following step S7, it associates the lateral velocity calculated from the optical flow with the compartments based on the position information of the optical flow. In step S8, the object detection device 20 labels each compartment according to its lateral velocity, and in the following step S9, it generates a set of compartments from multiple compartments based on the labels. In step S10, the object detection device 20 detects the object corresponding to the set of compartments.

[0070] [Embodiments of the present invention] According to this embodiment, an object detection method is provided which calculates the probability of an object being present in a section defined by dividing the area in front of the vehicle in the direction of travel into a grid pattern from the positional information of the distance measurement points of objects present around the vehicle, obtains the optical flow of the object from an image captured by the vehicle's imaging device 12, generates a set of sections associating a plurality of the sections based on the probability of occupancy and at least one of the speed in the vehicle width direction and the amount of movement of the object in the image calculated from the optical flow, and detects the object corresponding to the set of sections. An object detection device 20 that executes this object detection method is also provided. This makes it possible to distinguish between moving objects and stationary objects. Furthermore, it is possible to smoothly detect moving objects moving toward the direction of the vehicle from the shadow of a stationary object.

[0071] In the object detection method and object detection device 20 of this embodiment, a section in which the absolute value of the velocity is equal to or greater than a predetermined absolute value is determined to be the section corresponding to a moving object, and a section in which the absolute value of the velocity is less than the predetermined absolute value is determined to be the section corresponding to a stationary object. This allows for smooth differentiation between moving objects and stationary objects.

[0072] In the object detection method and object detection device 20 of this embodiment, a section in which the amount of movement is equal to or greater than a predetermined amount of movement is determined to be the section corresponding to a moving object, and a section in which the amount of movement is less than the predetermined amount of movement is determined to be the section corresponding to a stationary object. This allows for smooth differentiation between moving objects and stationary objects.

[0073] In the object detection method and object detection device 20 of this embodiment, it is determined whether the area corresponds to a moving object moving to the right relative to the vehicle, a moving object moving to the left relative to the vehicle, or a stationary object. This allows for smooth differentiation between moving objects and stationary objects.

[0074] In the object detection method and object detection device 20 of this embodiment, a histogram of at least one of the velocity and the amount of movement is generated, a threshold corresponding to the inflection point of the approximation curve of the histogram is set, and the partition set is generated using the threshold. This simplifies the partition set generation process.

[0075] In the object detection method and object detection device 20 of this embodiment, a set of unlabeled sections is generated from the unlabeled sections based on the occupancy probability and at least one of the velocity and the amount of movement, and the generated set of sections is labeled based on at least one of the velocity and the amount of movement. This simplifies the process of generating the set of sections.

[0076] The object detection method and object detection device 20 of this embodiment, in which the number of partition sets to be generated is set to a predetermined number when generating the partition set from partitions that are not labeled, is the object detection method according to claim 5. This simplifies the partition set generation process.

[0077] In the object detection method and object detection device 20 of this embodiment, the occupancy probability is calculated using a particle filter. This allows for a more accurate calculation of the occupancy probability.

[0078] In the object detection method and object detection device 20 of this embodiment, if the velocity corresponding to a section defined by dividing the region into a grid cannot be calculated from the optical flow, the velocity corresponding to the section is supplemented using the velocity corresponding to the section surrounding the section. Similarly, if the amount of movement corresponding to a section defined by dividing the region into a grid cannot be calculated from the optical flow, the amount of movement corresponding to the section is supplemented using the amount of movement corresponding to the section surrounding the section. This makes it possible to generate a set of sections even if there are sections for which the corresponding vehicle width velocity and amount of movement cannot be calculated.

[0079] In the object detection method and object detection device 20 of this embodiment, the relative velocity to the imaging device, calculated from the optical flow, is converted to an absolute velocity in a fixed coordinate system using behavioral information relating to the vehicle's behavior. This makes it possible to suppress the influence of the vehicle's behavior when calculating the velocity of the target object.

[0080] In the object detection method and object detection device 20 of this embodiment, at least one of the following is performed: generating a set of partitions from adjacent partitions, and generating a set of partitions from partitions that are within a predetermined distance from a partition defined by dividing the region into a grid. This simplifies the process of generating a set of partitions.

[0081] In the object detection method and object detection device 20 of this embodiment, point cloud data is acquired in which the position information of the distance measurement points is arranged in two dimensions in the vertical direction and the vehicle width direction of the vehicle. From the point cloud data, the occupancy probability is calculated for each section defined by a grid extending in the longitudinal direction and the vehicle width direction of the vehicle when the vehicle is viewed from above. The section is associated with at least one of the speed and the amount of movement calculated from the optical flow. The section to which at least one of the speed and the amount of movement is associated is labeled according to at least one of the speed and the amount of movement. A set of sections is generated from the labeled sections based on the occupancy probability, and the object corresponding to the set of sections is detected based on the label attached to the section. This makes it possible to distinguish between moving objects and stationary objects. [Explanation of Symbols]

[0082] 10...Object detection system, 11...Distance measuring device, 12...Imaging device, 20...Object detection device, 21...Calculation unit, 22...Acquisition unit, 23...Generation unit, 24...Detection unit, A...Arrow, B...Building, C...Pedestrian crossing, F1, F2...Optical flow, G...Grid, K...Approximate curve, Ka, Kb...Inflection point, Kc, Kd...Threshold, L1, L2...Lane, P1, P2, P3...Position, R...Area, V1...Vehicle, V2...Other vehicle, W...Pedestrian, X1, X2, X3...Plot set, Z...Plot

Claims

1. In an object detection method performed by a vehicle object detection device, The object detection device is Based on the positional information of the distance measurement points of objects surrounding the vehicle, the probability of the object being located in a defined area by dividing the area in front of the vehicle in the direction of travel into a grid is calculated. From the image captured by the vehicle's imaging device, the optical flow of the object is obtained. Based on the occupancy probability and at least one of the vehicle width direction velocity and movement amount of the object in the image calculated from the optical flow, a set of associated sections is generated. An object detection method for detecting the object corresponding to the set of partitions.

2. The object detection device is If the absolute value of the speed is greater than or equal to a predetermined absolute value, it is determined that the section corresponds to the moving body. The object detection method according to claim 1, wherein a section in which the absolute value of the speed is less than the predetermined absolute value is determined to be the section corresponding to a stationary object.

3. The object detection device is The section in which the amount of movement is equal to or greater than a predetermined amount of movement is determined to be the section corresponding to the moving body. The object detection method according to claim 1, wherein the section in which the amount of movement is less than the predetermined amount of movement is determined to be the section corresponding to a stationary object.

4. The object detection method according to any one of claims 1 to 3, wherein the object detection device determines whether the compartment corresponds to a moving body moving to the right relative to the vehicle, a moving body moving to the left relative to the vehicle, or a stationary object.

5. The object detection device is A histogram is generated of at least one of the velocity and the amount of displacement. A threshold is set corresponding to the inflection point of the approximation curve of the histogram. The object detection method according to any one of claims 1 to 3, wherein the set of partitions is generated using the threshold.

6. The object detection device is Based on the occupancy probability and at least one of the velocity and the amount of movement, the set of partitions is generated from the unlabeled partitions. The object detection method according to any one of claims 1 to 3, wherein the generated set of partitions is labeled based on at least one of the velocity and the amount of movement.

7. The object detection method according to claim 6, wherein when the object detection device generates the set of sections from sections that are not labeled, the number of section sets to be generated is set to a predetermined number.

8. The object detection method according to any one of claims 1 to 3, wherein the object detection device calculates the occupancy probability using a particle filter.

9. The object detection device is If the velocity corresponding to a section defined by dividing the region into a grid cannot be calculated from the optical flow, the velocity corresponding to the section surrounding the first section is used to supplement the velocity corresponding to the first section. If the amount of movement corresponding to one section defined by dividing the region into a grid cannot be calculated from the optical flow, the object detection method according to any one of claims 1 to 3, wherein the amount of movement corresponding to one section is supplemented using the amount of movement corresponding to the surrounding sections of the one section.

10. The object detection method according to any one of claims 1 to 3, wherein the object detection device uses behavioral information relating to the behavior of the vehicle to convert the relative velocity to the imaging device, calculated from the optical flow, into an absolute velocity in a fixed coordinate system.

11. The object detection method according to any one of claims 1 to 3, wherein the object detection device performs at least one of generating a set of sections from adjacent sections and generating a set of sections from sections located within a predetermined distance from a section defined by dividing the region into a grid.

12. The object detection device is The positional information of the distance measurement points is arranged in a two-dimensional array in the vertical and horizontal directions of the vehicle to obtain point cloud data. From the point cloud data, the occupancy probability is calculated for each section defined by a grid extending in the longitudinal direction and the vehicle width direction when the vehicle is viewed from above. The aforementioned section is associated with at least one of the velocity and the amount of movement calculated from the optical flow. The section to which at least one of the speed and the amount of movement is associated is labeled according to at least one of the speed and the amount of movement. Based on the occupancy probability, the set of partitions is generated from the labeled partitions. An object detection method according to any one of claims 1 to 3, wherein an object corresponding to the set of partitions is detected based on a label attached to the partition.

13. A calculation unit calculates the probability that an object is located in a section defined by dividing the area in front of the vehicle in the direction of travel into a grid, based on the positional information of the distance measurement points of the object surrounding the vehicle. An acquisition unit that acquires the optical flow of the object from an image captured by the vehicle's imaging device, A generation unit generates a set of associated sections based on the occupancy probability calculated by the calculation unit and at least one of the vehicle width direction velocity and movement amount of the object in the image calculated from the optical flow acquired by the acquisition unit, An object detection device comprising: a generation unit and a detection unit that detects the object corresponding to the set of partitions generated by the generation unit.

Images