Object detection method and object detection device

Abstract

The present invention provides an object detection method and an object detection device that can detect objects with high accuracy even when only a portion of the object is visible in the image. [Solution] The system obtains the distance from the position of a sensor mounted on the vehicle to objects around the vehicle, generates a distance array by arranging the distances for each azimuth angle based on the sensor's position, calculates the optical flow from images taken around the vehicle, updates the distance array based on the optical flow and the vehicle's speed, and detects objects based on the difference between the updated distance array (which was updated from a previously generated distance array) and the newly generated distance array.

Description

Technical Field

[0001] The present invention relates to an object detection method and an object detection device.

Background Art

[0002] An imaging image in front of the traveling direction of a vehicle is acquired from a first imaging unit and a second imaging unit. On the condition that an object is detected in an image captured by one of the imaging units and no object is detected in an image captured by the other imaging unit, it is determined that there is a blind spot as seen from the other imaging unit in front of the traveling direction and that an object exists in the blind spot (Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, since the technique described in Patent Document 1 is a technique for detecting an object in each image by performing template matching on the images captured by each imaging unit, there is a problem that the presence of an object cannot be recognized if the form of the object that matches the template does not appear in the image. [[ID=3�]]

[0005] The problem to be solved by the present invention is to provide an object detection method and an object detection device that can detect an object with high accuracy even when the object is only partially shown in the image.

Means for Solving the Problems

[0006] The present invention solves the above problem by obtaining the distance from the position of a sensor mounted on the vehicle to an object in the vicinity of the vehicle, generating a distance array in which the distances are arranged according to the azimuth angle based on the position of the sensor, calculating the optical flow from an image captured around the vehicle, updating the distance array based on the optical flow and the vehicle speed, and detecting an object based on the difference between the updated distance array, which is an updated version of the previously generated distance array, and the newly generated distance array. [Effects of the Invention]

[0007] According to the present invention, objects can be detected with high accuracy even when they are only partially visible in the image. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is a block diagram showing an example of an embodiment of the object detection device according to the present invention. [Figure 2] Figure 2 is a diagram illustrating an example of object detection using a distance array according to this embodiment. [Figure 3] Figure 3 is an example of a flowchart showing the object detection method performed in the object detection device of this embodiment. [Figure 4] Figure 4 is a diagram illustrating an example of a method for setting a height threshold according to this embodiment. [Figure 5] Figure 5 is a diagram illustrating an example of a height threshold smoothing method according to this embodiment. [Figure 6] Figure 6 is a diagram illustrating an example of object detection using the distance array and height threshold array according to this embodiment. [Figure 7] Figure 7 illustrates an example of a method for changing the angular range resolution according to this embodiment. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described below with reference to the drawings. Figure 1 is a block diagram showing an object detection device 1 according to one embodiment of the present invention. The object detection device 1 is a device that detects objects present around the vehicle. An object is the object to be detected by the object detection device 1 and includes boundary lines, center lines, road markings, guardrails, curbs, road signs, traffic lights, pedestrian crossings, buildings, road fixtures (such as mailboxes and vending machines). Objects also include moving objects such as other vehicles, motorcycles, bicycles, and pedestrians. The object detection device 1 is a device mounted on the vehicle. As shown in Figure 1, the object detection device 1 comprises a distance measuring device 2, an imaging device 3, a vehicle information acquisition device 4, and a control device 10. These devices are connected by CAN (Controller Area Network) or other in-vehicle LAN and can exchange information with each other.

[0010] The rangefinder 2 detects the relative distance and relative speed between the vehicle and an object. The rangefinder 2 includes a laser radar, millimeter-wave radar, LiDAR (Light Detection and Ranging) unit, etc. Multiple rangefinders 2 are installed on a single vehicle. The rangefinder 2 is an example of a device having the "rangefinder means" as defined in the claims. The "rangefinder means" may be a camera capable of measuring distance, such as a monocular camera or a stereo camera, and in this case, the imaging device 3 may have the "rangefinder means" rather than being limited to the rangefinder 2. If a stereo camera is used, the camera used to calculate optical flow is used as the reference camera to calculate the distance. The rangefinder 2 may also remove noise using a static-based outlier filter algorithm or the like.

[0011] The distance measuring device 2 may generate point cloud data in which information about the distance measuring points of surrounding objects is arranged in two dimensions in the left-right and up-down directions of the vehicle. The distance measuring points of an object are points on the object whose distance to the distance measuring device 2 has been measured. The information about the distance measuring points includes positional information of the distance measuring points (coordinate information of the distance measuring points, distance information from the distance measuring device 2 to the distance measuring points, height information from the road surface to the distance measuring points, etc.). The distance measuring device 2 generates point cloud data at regular time intervals and outputs it to the control device 10.

[0012] For example, the distance measuring device 2 irradiates electromagnetic waves (millimeter waves, infrared rays, lasers, etc.) in the left-right direction (horizontal direction) of its own vehicle and detects the electromagnetic waves (reflected waves) reflected by objects around the vehicle to detect the distance and direction (angle) of the distance measuring point. The distance measuring device 2 generates point cloud data in which the detected position information of the distance measuring point is arranged in the left-right and up-down directions (two-dimensional array). The irradiation range of the electromagnetic waves is not particularly limited. Examples of electromagnetic waves include millimeter waves (wavelength: 1-10 mm), infrared rays (wavelength: 700-1000 nm), and lasers (wavelength: 500-800 nm). The position of the distance measuring point is determined, for example, as coordinates in a three-dimensional Cartesian coordinate system with the position of the distance measuring device 2 as the origin. Point cloud data is generated in which points indicating the position of the distance measuring point are arranged in the left-right and up-down directions.

[0013] The imaging device 3 is a camera equipped with an image sensor such as a CCD, which images the area around the vehicle, including surrounding objects, and generates an image that includes the area around the vehicle. Multiple imaging devices 3 are installed on the vehicle's front grille, side mirrors, rear bumper, etc. The imaging device 3 generates images at regular time intervals and outputs them to the control device 10. The imaging device 3 is an example of a "sensor having imaging means" as described in the claims. The imaging device 3 may also be equipped with a "distance measuring means" as described in the claims.

[0014] The vehicle information acquisition device 4 acquires vehicle information, including the status of the vehicle. This vehicle information includes steering information (including the amount of steering input), drive information (including brake input and accelerator input), speed information (including the vehicle's speed), direction of travel information, attitude information, and behavior information. The vehicle information acquisition device 4 can exchange information with one or more of the vehicle's steering sensor, speed sensor, acceleration sensor, braking sensor, direction of travel sensor, attitude sensor, and behavior sensor, and acquires the necessary information over time and outputs it to the control device 10. For example, the vehicle information acquisition device 4 acquires the vehicle's speed at regular time intervals and outputs it to the control device 10.

[0015] The control device 10 is a device for detecting objects around the vehicle by controlling and coordinating the devices that constitute the object detection device 1. The control device 10 is, for example, a computer and comprises a CPU, which is a processor, a ROM, which stores programs, and RAM, which functions as an accessible storage device. The CPU is an operating circuit that executes the programs stored in the ROM and realizes the functions of the control device 10. The control device 10 has a detection function for detecting objects. The ROM stores a program for realizing the detection function, and the detection function is realized when the CPU executes the program stored in the ROM. In Figure 1, the distance acquisition unit 11, the distance array generation unit 12, the optical flow calculation unit 13, the distance array update unit 14, and the object detection unit 15 are conveniently extracted and shown as functional blocks that realize the detection function. The control device 10 is an example of the "controller" described in the claims.

[0016] The distance acquisition unit 11 performs a distance acquisition process using the rangefinder 2 to acquire the distance from the imaging device 3 mounted on the vehicle to objects around the vehicle. The distance from the imaging device 3 to the object is the distance along the direction of travel of the vehicle from the coordinate position of the imaging device 3 to the coordinate position of the object. At regular time intervals, the distance acquisition unit 11 acquires point cloud data from the rangefinder 2 and calculates the distance from the imaging device 3 to the object along the direction of travel of the vehicle based on the point cloud data and the coordinate position of the imaging device 3. An example of the distance acquisition process will be described below.

[0017] The distance acquisition unit 11 estimates the point cloud data corresponding to the road surface portion from the acquired point cloud data and removes the point cloud data corresponding to the road surface portion. The distance acquisition unit 11 sets a processing range for each azimuth angle based on the imaging device 3, and aggregates the point cloud data included in the processing range for each azimuth angle. The azimuth angle is an angle indicating a predetermined direction from a certain reference point (for example, taking the traveling direction of the host vehicle as 0 degrees). The processing range for each azimuth angle is an angular range obtained by dividing the image at a predetermined angle θ in the horizontal direction. Alternatively, the processing range for each azimuth angle may be a range obtained by dividing the image with a horizontal width of tan θ for each predetermined angle θ. As will be described later, in the present embodiment, the distance to an object existing in the azimuth direction is acquired for each azimuth angle. When the predetermined angle θ becomes smaller, the horizontal length of the processing range becomes shorter and it is divided more finely, so the resolution becomes higher.

[0018] Based on the point cloud data included in the processing range for each azimuth angle, the distance acquisition unit 11 calculates the distance to an object existing in the azimuth direction for each azimuth angle. The distance calculated for each azimuth angle is also referred to as the target distance. Specifically, the distance acquisition unit 11 removes outliers using the standard deviation or the like from the distances of each of the plurality of ranging points included in the processing range of the azimuth angle, or calculates the average value of the distances of the plurality of ranging points to calculate the target distance for each azimuth angle. Further, the distance acquisition unit 11 may extract the distances of the number of data corresponding to a predetermined ratio of the whole in ascending order of distance from the distances of the plurality of ranging points, and calculate the average value of the extracted distances as the target distance.

[0019] In addition, there may be cases where a plurality of objects are included in the processing range for each azimuth angle. The distance acquisition unit 11 may create a histogram with the distances of a plurality of distance measurement points included in the processing range on the horizontal axis, group the distances of the distance measurement points for each object, and calculate the average value of the distances of the distance measurement points within the grouped group. Further, when there is a processing range in which the number of point cloud data excluding the road surface portion is less than or equal to a predetermined number, the distance acquisition unit 11 may set the azimuth angle as an azimuth angle where no object exists. In this case, the distance acquisition unit 11 sets the target distance corresponding to the azimuth angle to an infinite point. Also, when there is a processing range in which the average value of the distances of a plurality of distance measurement points is greater than or equal to a predetermined distance upper limit value, the distance acquisition unit 11 may set the azimuth angle as an azimuth angle where no object exists and set the target distance corresponding to the azimuth angle to an infinite point.

[0020] Here, an example of distance acquisition in the present embodiment will be described using FIG. 2. FIG. 2 is a diagram for explaining an example of object detection using the distance array according to the present embodiment. FIG. 2 shows an example of the data structure of the distance array and an overhead view representing the situation around the host vehicle. In the example of FIG. 2, the object detection around the left side of the host vehicle will be described as an example, but the description of the object detection on the left side is also appropriately applied when performing object detection around the right side of the host vehicle. The diagram of scene A represents the situation around the host vehicle at time t-1. In scene A, the host vehicle V1 is traveling in the traveling direction D1. The object T1 is moving in the lateral direction (the right side direction in the figure). The object T1 is, for example, a pedestrian. There is an object T2 that serves as a shielding object such as a building around the host vehicle V1. In scene A, the distance acquisition unit 11 acquires the distance (target distance) to the object existing at each azimuth angle on the left side with respect to the traveling direction of the host vehicle V1. In the example of FIG. 2, for example, the target distances at azimuth angle a1 and azimuth angle a2 are 40 m and 25 m, respectively.

[0021] The distance array generation unit 12 executes a distance array generation process to generate a distance array in which the distance to an object is arranged for each azimuth angle based on the position of the imaging device 3. The distance array generation process is executed at regular time intervals. The distance array is a data structure in which the value of the target distance corresponding to each azimuth angle is stored. That is, the distance value is stored in each element of the distance array that corresponds to the azimuth angle. In the distance array, the distance values ​​corresponding to each azimuth angle are arranged in order. In addition, each element of the distance array is assigned an index that indicates its position in the array (array position). The index is a number used to uniquely identify each element. The index is represented by a unique integer value. Each index corresponds to an azimuth angle. Specifically, the index is represented by a mapping that converts from an azimuth angle to an index. For example, the projection transformation formula that converts from an azimuth angle θ to an index i is expressed by the following equation (1).

number

[0022] The distance array generation unit 12 may smooth the distance array using a median filter or a Gaussian filter. In smoothing the distance array, the distance array generation unit 12 may exclude the element containing the point at infinity. If there is an element containing the point at infinity, the distance array generation unit 12 may calculate the average value of the elements adjacent to the left and right of that element in the array (target distance) and store the calculated average value in the element where the point at infinity was stored.

[0023] In the example shown in Figure 2, the distance array generation unit 12 generates the distance array A1 at time t-1. That is, distance array A1 is a distance array in which the distances (target distances) to objects at each azimuth angle in scene A are arranged. Distance array A1 is arranged with the direction of travel of the vehicle (0-degree direction) as the center, and the values ​​of the target distance corresponding to each azimuth angle are arranged sequentially from left to right toward the center. As shown in Figure 2, the elements of distance array A1 include 10m, 15m, 20m, 25m, 40m, 40m, 45m, ∞, ∞, ... ∞ represents the element in which the point at infinity is stored. The target distance (40m) corresponding to azimuth angle a1 in scene A is stored in the 5th element from the left of distance array A1. The target distance (25m) corresponding to azimuth angle a2 in scene A is stored in the 4th element from the left of distance array A1.

[0024] The optical flow calculation unit 13 calculates the optical flow from the image of the area around the vehicle captured by the imaging device 3. One method for calculating the optical flow is, for example, a method using block matching. In block matching, the optical flow calculation unit 13 may recognize occluding objects in front of the object based on point cloud information indicating the position of the object acquired by the distance measuring device 2, and generate an object template with the area of ​​the occluding object in the image masked. The optical flow calculation unit 13 calculates the optical flow based on the generated template. Alternatively, a more dense method may be used for calculating the optical flow. For example, a method using a neural network such as RAFT can be used for calculating the optical flow. The target of the optical flow calculation may be feature points on the image, or all pixels in the image. If the point cloud data is grouped, the optical flow calculation unit 13 may calculate the optical flow for each group. The optical flow calculation unit 13 may also estimate the vehicle's velocity in the direction of travel (longitudinal velocity) by matching it with point cloud data, calculating the distance in the direction of travel of the vehicle, and tracking it. If the estimation accuracy is high, the vehicle's travel distance may be updated based on the longitudinal velocity calculated by the optical flow calculation unit 13. To improve the estimation accuracy, if a Kalman filter is used to estimate the longitudinal velocity, error covariance and the like may be used in the calculation.

[0025] The distance array update unit 14 updates the distance array based on the optical flow and the vehicle speed. First, the method for updating the distance array based on the vehicle speed will be explained. The distance array update unit 14 calculates the vehicle's mileage based on the vehicle speed and the elapsed time from the time the previous distance array was generated to the time a new distance array is generated (elapsed time from time t-1 to time t). The elapsed time is, for example, a fixed time interval for generating the distance array. The distance array update unit 14 updates the target distances stored in each element of the distance array by subtracting the vehicle's mileage from each target distance stored in each element of the previous distance array at time t-1. In the example in Figure 2, the distance array update unit 14 updates distance array A1 to distance array A2 by subtracting the mileage (for example, 5m) from each target distance included in each element of distance array A1. As shown in Figure 2, each element of distance array A2 stores, from left to right, 5m, 10m, 15m, 20m, 35m, 35m, 40m, ∞, ∞, ... While the target distances stored in distance array A1 are observed values, the target distances stored in distance array A2 are estimated values.

[0026] JPEG2026109237000003.jpg31165

number

[0027] JPEG2026109237000005.jpg24167

number

number

number

[0028] Next, a method for updating the distance array based on optical flow will be described. The distance array update unit 14 calculates the lateral velocity (azimuth angular velocity) for each processing range of the azimuth angle based on the optical flow. In this embodiment, by superimposing point cloud data onto an image, the optical flow can be calculated while excluding backgrounds that are farther than the processing range, such as mountains. Alternatively, outliers may be removed from the velocity of each optical flow calculated for each azimuth angle, and the average value of the velocities of multiple optical flows may be calculated. Note that the elements for calculating the optical flow velocity may be limited to elements that store distance values ​​excluding points at infinity, or they may be limited to elements corresponding to objects that are above a certain height from the road surface.

[0029] JPEG2026109237000010.jpg39167

number

[0030] The distance array update unit 14 may, if there are groups of distance measurement points with different azimuth velocities within the azimuth angle processing range, calculate the updated exponent for each group's distance and store the distance for each group in the element corresponding to the different exponent. Furthermore, if the updated exponents for multiple different target distances become the same single exponent, considering that distant objects may be obscured by closer objects, the distance array update unit 14 may compare the target distances and store the shorter distance in the element corresponding to the updated exponent. Also, if there is an element in the updated distance array that does not contain a target distance, the distance array update unit 14 may set that element to the infinity point. Additionally, the projection transformation formula may be a transformation formula that projects from the azimuth angle to an exponent including a fractional part. In this case, the transformed exponent may not be uniquely determined. The distance array update unit 14 may store the target distance in the element corresponding to the exponent with the closest value, and then convert the exponent including a fractional part back to an azimuth angle for the next azimuth angle processing range, updating the processing range to the transformed azimuth angle.

[0031] The object detection unit 15 performs object detection processing to detect objects present around the vehicle based on the position information of the distance measurement points (coordinates of the distance measurement points, distance to the object, etc.). For example, the object detection unit 15 creates an image plotting the distance measurement points on a 3D map based on the position information of the distance measurement points, and detects objects from this image. In this embodiment, the object detection unit 15 also detects objects present around the vehicle based on the difference between an updated distance array, which is updated from a previously generated distance array, and a newly generated distance array. Newly detected objects are also referred to as new objects. The updated distance array is a distance array updated by the distance array update unit 14 based on the optical flow and the vehicle speed of the vehicle. In the example in Figure 2, the previously generated distance array is the distance array A1 from the previous time (t-1), and the updated distance array is the distance array A3 from the previous time (time t-1). The newly generated distance array is a distance array generated by the distance array generation unit 12 based on the distance newly acquired by the distance acquisition unit 11. In the example in Figure 2, the newly generated distance array is the distance array B1 at the present time (time t).

[0032] Figure 2 shows Scene B at time t. Scene B is a scene where time has progressed from Scene A, and the vehicle V1 has moved 5m forward from its position at time t-1. In Scene B, the distance array generation unit 12 generates a distance array B1 by arranging the distances acquired by the distance acquisition unit 11 for each azimuth angle. That is, distance array B1 is a distance array in which the distances to objects (target distances) at each azimuth angle in Scene B are arranged. As shown in Figure 2, the elements of distance array B1 include 10m, 15m, 20m, 30m, 35m, 40m, ∞, ∞, ∞, ... The target distance (35m) corresponding to azimuth angle a1 in Scene B is stored in the 5th element from the left of distance array B1. The target distance (30m) corresponding to azimuth angle a2 in Scene B is stored in the 4th element from the left of distance array B1.

[0033] The object detection unit 15 compares each element of the updated distance array with each element of the newly generated distance array, index by index. If there is an index in which the distance value stored in the elements of the distance array becomes smaller than or equal to a predetermined threshold, the object detection unit 15 detects a new object at the azimuth angle corresponding to that index. In the example in Figure 2, each element of distance array A3 and each element of distance array B1 are compared index by index. For the element corresponding to the fourth index from the left, the distance is 35m in distance array A3, while the distance is 30m in distance array B1. In this way, if there is an index in which the distance value has become smaller, the object detection unit 15 detects a new object at the azimuth angle corresponding to that index. In Scene A, object T1 (pedestrian) is obscured by object T2 (building) in the foreground, and the distance at azimuth angle a1 is measured as the distance to object T2 (building) in the background (40m). Therefore, based on the estimation from scene A, at time t, the distance of azimuth angle a2 is estimated to be the distance to the object T2 (building) in the background (35m). On the other hand, in the actual scene B, since object T1 (pedestrian) appears at azimuth angle a2, the target distance of azimuth angle a2 is measured to be the distance to object T1 (pedestrian). The target distance of azimuth angle a2 corresponds to the fourth index from the left in the distance array. Since the distance of azimuth angle a2 has changed from the estimated value of 35m to 30m at time t, the object detection unit 15 detects object T1 (pedestrian) at azimuth angle a2. Alternatively, the object detection unit 15 may detect an object at the azimuth angle corresponding to an element whose distance has changed to less than a predetermined upper limit of distance from the point at infinity.

[0034] Furthermore, the object detection unit 15 may also track the movement of detected objects. First, the object detection unit 15 determines whether the newly detected object is a moving object. For example, the object detection unit 15 calculates the relative velocity of the new object with respect to the vehicle when the new object is stationary. The object detection unit 15 also obtains a measured value of the amount of movement of the new object by calculating the optical flow. The object detection unit 15 determines whether the new object is a moving object based on the difference between the apparent amount of movement of the new object calculated from the relative velocity and the measured amount of movement. When the imaging device 3 tracks the movement of an object based on the image captured by the imaging device 3, it may capture images at a faster cycle than the cycle in which the distance measuring device 2 acquires the distance to the object.

[0035] The object detection unit 15 may obtain a measurement of the movement amount of a new object by calculating optical flow based on past images stored in memory. Past images stored in memory include images acquired before the new object was detected, for example, from the previous time (time t-1) to the current time (time t). The object detection unit 15 also recognizes occluding objects in front of the new object based on point cloud information indicating the position of the object acquired by the distance measuring device 2, and generates a template of the new object with the area of ​​the occluding object in the image masked. The object detection unit 15 may then obtain a measurement of the movement amount by calculating optical flow based on the generated template. Alternatively, the object detection unit 15 may obtain a measurement of the movement amount by calculating an optical flow with a higher density than the optical flow calculated to update the distance array.

[0036] Next, with reference to Figure 3, the procedure for the control device 10 to detect an object will be described. Figure 3 is an example of a flowchart showing the object detection method performed in the object detection device of this embodiment. The control flow shown in Figure 3 is executed repeatedly at regular time intervals. First, in step S11, the control device 10 obtains the target distance from the imaging device 3 to the object around the vehicle at time t-1. In step S12, the control device 10 generates a distance array in which the target distances are arranged for each azimuth angle centered on the coordinate position of the imaging device 3. In step S13, the control device 10 calculates the optical flow from the image of the vehicle's surroundings captured by the imaging device 3. In step S14, the control device 10 calculates the lateral velocity (azimuth angular velocity) for each azimuth angle based on the optical flow. In step S15, the control device 10 calculates the vehicle's travel distance. The vehicle's travel distance is the distance the vehicle travels between time t-1 and time t.

[0037] In step S16, the control device 10 subtracts the travel distance from each target distance included in the distance array at time t-1. In step S17, the control device 10 updates the distance array based on the azimuth angular velocity. Specifically, the control device 10 calculates an exponent at time t for each target distance stored in each element of the distance array at time t-1 based on the azimuth angular velocity, and updates the distance array by storing each target distance in the element corresponding to the exponent at time t. In step S18, the control device 10 compares the updated distance array at time t-1 with the distance array containing the distances newly acquired at time t for each azimuth angle. In step S19, if there is an azimuth angle in which the distances included in the newly generated distance array are smaller than the distances included in the updated distance array by a predetermined threshold, the control device 10 detects a new object in that azimuth angle. As described above, even when surrounding objects are only partially visible in the image, the appearance of new objects can be detected on an azimuth angle basis, thus enabling more accurate object detection.

[0038] In real-world environments, situations may occur where objects are partially obscured by guardrails, bushes, etc. Therefore, in this embodiment, the control device 10 may generate distance arrays for multiple predetermined heights. The control device 10 then detects objects at each predetermined height based on the difference between the updated distance array and the newly generated distance array. This makes it possible to detect new objects at each height. The specific methods for updating the distance array and detecting objects are the same as those described above. An example of generating distance arrays for each height will be described below.

[0039] The distance array generation unit 12 may divide the point cloud data into multiple groups according to the height from the road surface and generate a distance array for each height. A predetermined height threshold is used for dividing the point cloud data. For each distance measurement point included in the point cloud data, the distance array generation unit 12 determines whether the height of the distance measurement point is greater than or equal to a predetermined height threshold, and divides each distance measurement point into a group whose height is greater than or equal to the predetermined height threshold and a group whose height is less than the predetermined height threshold. A distance array is generated for each divided group. The height threshold may be a value set in advance, or it may be set dynamically as described later. In addition, when the distance histogram of distance measurement points included in the azimuth angle processing range is grouped, the distance array generation unit 12 may generate a distance array for each height as a trigger when the distance of the distance measurement points is divided into multiple groups.

[0040] Here, an example of a specific method for setting the height threshold will be explained using Figure 4. Figure 4 is a diagram illustrating an example of a method for setting the height threshold according to this embodiment. The left side of Figure 4 shows an overhead view representing the surroundings including the vehicle V1. The right side of Figure 4 shows a histogram of the distance of the distance measurement point and a histogram of the height of the distance measurement point. The distance measurement point is a distance measurement point included in the processing range of azimuth angle a3 shown in the left side of Figure 4. In the left side of Figure 4, objects T2 and T3 exist in the processing range of azimuth angle a3. Object T2 is an object that is taller than object T3, such as a building. Object T3 is an obstruction such as vegetation. In the processing range of azimuth angle a3, object T3 is located at a distance of approximately 10m from the vehicle V1 in the direction of travel D1 of the vehicle V1. Object T2 is located at a distance of approximately 20m from the vehicle V1 in the direction of travel D1 of the vehicle V1.

[0041] The distance array generation unit 12 creates a histogram of the distances of the measurement points included in the processing range for each azimuth angle, and groups the distances of the measurement points in the histogram. In the example in Figure 4, the distances of the measurement points are divided into two groups on the histogram: Group A for distances around 10m and Group B for distances around 20m. Group A corresponds to the group of distances to object T3, and Group B corresponds to the group of distances to object T2. Distances of measurement points outside of each group may be removed as noise. When the distances of the measurement points are divided into multiple groups, the distance array generation unit 12 treats these groups as height groups, acquires height information for multiple measurement points within each group, and creates a histogram. The distance array generation unit 12 sets the value between the heights of two adjacent groups on the histogram as the height threshold. In the example in Figure 4, a height histogram is created from the heights of the measurement points included in Groups A and B. In addition, there may be objects with heights exceeding the observation range of the point cloud data, such as buildings, within the processing range of the azimuth angle. In other words, one of the height groups may have a height that exceeds the observation range of the point cloud data. In such cases, the distance array generation unit 12 may offset the height of the lower group by a certain amount and set the height after the offset as the height threshold.

[0042] As an example of how to calculate the height for each group, the distance array generation unit 12 may calculate the group height based on the top few percent of the heights of the distance measurement points within the group, or it may perform noise reduction to remove at least outliers from the heights of the distance measurement points within the group, and calculate the maximum value of the distance measurement point heights within the group after noise reduction as the group height. As described above, the distance array generation unit 12 calculates a height threshold for each azimuth angle, divides the multiple distance measurement points into groups according to the height threshold, and generates a distance array for each height based on the distance of the distance measurement points within each group.

[0043] Furthermore, when generating distance arrays according to height, the distance array generation unit 12 may divide the processing range of all azimuth angles into two according to height and generate a distance array for each height. If there are azimuth angles that divide the distance measurement points into groups according to height, the distance array generation unit 12 may also divide the processing range of nearby azimuth angles located near those azimuth angles into two according to height and generate a distance array for each height. In this case, the distance array generation unit 12 may generate a single distance array without dividing the processing range for azimuth angles located further away than nearby azimuth angles.

[0044] Furthermore, the distance array generation unit 12 may calculate the height threshold for each element and smooth it. Below, an example of a specific method for smoothing the height threshold will be described using Figure 5. Figure 5 is a diagram illustrating an example of a height threshold smoothing method according to this embodiment. Figure 5 shows an example of the data structure of the height threshold array. The distance array generation unit 12 generates a height threshold array with the same number of elements as the distance array and stores the height thresholds as elements. That is, the distance array generation unit 12 generates a height threshold array in which the height thresholds are arranged for each azimuth angle. Each element of the height threshold array and each element of the distance array are associated with each azimuth angle. For each azimuth angle, the distance array generation unit 12 creates a histogram of the distances of the distance measurement points included in the processing range and groups the distances of the distance measurement points in the histogram. If the distances of the distance measurement points can be divided into three or more groups, the distance array generation unit 12 sets a height threshold to distinguish between each group. As a result, if there are three or more groups, two or more height thresholds are set. The distance array generation unit 12 may add a height threshold array in the vertical direction to store the height thresholds. In Figure 5, the distance array generation unit 12 generates a height threshold array C1. The height threshold array C1 has two levels in the vertical direction because the distances of the distance measurement points are divided into three groups and two height thresholds are set in the processing range of the second and ninth azimuth angles from the left.

[0045] The distance array generation unit 12 compares the height threshold stored in each element of the height threshold array with the height threshold stored in neighboring elements and groups elements with similar height thresholds into the same group. In the example in Figure 5, the height threshold array C1 is grouped into the second (height threshold 0.5) and third (height threshold 0.4) groups from the left, the fifth (height threshold 0.6) and sixth (height threshold 0.5) groups from the left, and the ninth (height threshold 1.0) and tenth (height threshold 1.2) groups from the left. The distance array generation unit 12 may remove an element containing a height threshold as noise if there are no other elements containing height thresholds in its vicinity. The distance array generation unit 12 may also perform smoothing by, for example, calculating the average value of the height thresholds for the same group and storing the average value in each element within that group. In the example in Figure 5, the height threshold array C2 is generated by noise reduction and smoothing. Furthermore, if there is an element for which a height threshold is not stored, the distance array generation unit 12 performs an interpolation process to store a height threshold in that element based on the height threshold stored in a nearby element. In the example in Figure 5, the height threshold array C3 is generated by the interpolation process. This makes it possible to generate a distance array for each height based on a constant threshold. The height threshold may be constant for each azimuth angle, or it may be variable based on the result of smoothing, or there may be azimuth angles for which no height threshold is set.

[0046] The distance array generation unit 12 generates a distance array according to the height for azimuth angles for which a height threshold is set. The distance array generation unit 12 may also generate a distance array according to the height for all azimuth angles by setting a height threshold for azimuth angles for which a height threshold is not set, in the same manner as the interpolation process described above. The interpolation process may use the nearest neighbor method or a linear interpolation method based on the height thresholds stored in neighboring elements. If multiple height threshold arrays are arranged vertically, the distance array generation unit 12 may perform the interpolation process on the lower upper limit of the height threshold arrays all at once. If there is an element without a height threshold set near an element with a height threshold set, it is preferable for the distance array generation unit 12 to perform an interpolation process at least once, considering the possibility that a new object may appear in the element without a height threshold, and to interpolate based on the height threshold stored in the nearby element.

[0047] Next, an example of object detection when distance arrays are generated for each height will be explained using Figure 6. Figure 6 is a diagram illustrating an embodiment of object detection using distance arrays and height threshold arrays according to this embodiment. Figure 6 shows an example of the data structure of the distance array and height threshold array, and an overhead view representing the situation around the vehicle. In this embodiment, by generating distance arrays for each height, new objects that are taller than existing objects such as plants and guardrails can be detected. For example, consider a scenario like Scene C, where object T3 (e.g., plants) is located to the left of the direction of travel of the vehicle V1, in front of the vehicle V1, and object T2 (e.g., a building) is located behind it. In such a scenario, the distance array generation unit 12 generates a distance array corresponding to the height up to the height of object T3 (plant), and a distance array corresponding to heights greater than or equal to the height of object T3. Specifically, first, as shown in Figure 6, the distance array generation unit 12 generates height threshold array D1 and distance array D2 based on the observation data at time t-1. In each array, grouping according to height is performed by the azimuth angle corresponding to the 5th element from the left. Therefore, in the height threshold array D1, the height threshold is stored in the first five elements from the left. In the distance array D2, the array is divided into two rows up to the fifth element from the left, and the target distance is stored in each element. Furthermore, the distance array generation unit 12 can generate distance arrays by dividing them according to height, even for blind spots in buildings, through interpolation processing, so that the distance to the plantings and the point at infinity can be stored in the distance array in two rows.

[0048] Then, by updating the distance array and performing object detection on the distance array for each height using the method described above, the object detection unit 15 can detect a new object if it appears behind object T3. In the example in Figure 6, the distance array update unit 14 updates the distance array D2 based on the vehicle's mileage and the optical flow to generate the updated distance array D3. Furthermore, the distance array generation unit 12 generates the distance array E1 at time t based on newly acquired observation data. Scene D represents the situation around the vehicle at time t. In Scene D, object T1 (e.g., a pedestrian) is located at azimuth angle a3. The distance to object T1 along the direction of travel is 35m. If the element corresponding to azimuth angle a3 is located in the fourth position from the left in the distance array E1, the distance 35m is stored in the upper fourth position from the left in the distance array E1. Then, the object detection unit 15 detects an object by comparing the updated distance array D3, which was updated based on the distance array at time t-1, with the newly generated distance array E1. Comparing the fourth upper element from the left in each distance array (azimuth angle a3), the corresponding element in distance array D3 stores the point at infinity (∞), while the corresponding element in distance array E1 stores a distance of 35m. Therefore, the object detection unit 15 detects that a new object exists at azimuth angle a3.

[0049] Furthermore, in this embodiment, the control device 10 may change the resolution of the angular range, which is the unit for acquiring distance within the range of the azimuth angle, for each azimuth angle. The resolution of the angular range is determined, for example, by the horizontal length of the angular range. The resolution of the angular range before the change corresponds to the horizontal length of the processing range of the azimuth angle. The smaller the horizontal length of the angular range, the finer the processing range of the azimuth angle can be divided, and therefore the higher the resolution. The distance acquisition unit 11 increases the resolution and acquires the target distance in an angular range smaller than the processing range of the azimuth angle. For example, if the distance acquisition unit 11 groups point cloud data included in the processing range of the azimuth angle, and there is a difference in the horizontal distance of the grouped point cloud data, it changes the resolution so that the angular range becomes finer and acquires the distance with the changed resolution. The distance array generation unit 12 generates a distance array by arranging the distances acquired with the changed resolution. The specific method for generating the distance array is the same as the method described above.

[0050] The distance acquisition unit 11 may increase the resolution of azimuth angles that include elements with large distance changes, based on the change in the target distance included in the distance array. Since the distance changes are large at edges such as corners of buildings around the vehicle, it is necessary to increase the resolution of azimuth angles with large distance changes in order to detect the situation of building corners and the like with higher accuracy. For example, the distance acquisition unit 11 calculates the second derivative of the target distance for each azimuth angle. The distance acquisition unit 11 then sets the resolution in the processing range of azimuth angles where the second derivative is greater than or equal to a predetermined value to be higher than the resolution in the processing range of azimuth angles where the second derivative is less than the predetermined value. At this time, the distance acquisition unit 11 updates the conversion formula for projecting from azimuth angles to exponents.

[0051] Here, an example of a method for changing the angular range resolution according to this embodiment will be explained using Figure 7. Figure 7 is a diagram illustrating an example of a method for changing the angular range resolution according to this embodiment. Figure 7 shows an example of the data structure of a distance array and an overhead view representing the situation around the vehicle. The diagram in Scene E represents the situation around the vehicle V1 at time t-1. Around the vehicle V1, there are objects T2 in the foreground (e.g., a building) and objects T2 in the background (e.g., a building). Distance array F1 is a distance array in which the distances to the surrounding objects in Scene E are arranged by azimuth angle. Distance array F1 is the distance array before changing the resolution. For example, the fifth element from the left of distance array F1 stores the target distance of 40m corresponding to azimuth angle a4. An example of detecting the corner portion P of the foreground object T2 with higher accuracy in Scene E will be explained.

[0052] When calculating the second derivative of each distance stored in the distance array F1, for example, the second derivative of the fifth element from the left corresponding to azimuth angle a4 (distance 40m) is -15m, indicating a large change in distance. Therefore, the distance acquisition unit 11 increases the resolution in the processing range of azimuth angle a4, which corresponds to the fifth element from the left in the distance array F1, compared to the resolution before the change. If the processing range of the azimuth angle (angle range before the change) is an angle range of a predetermined angle θ, the distance acquisition unit 11 changes the resolution of the processing range of the azimuth angle to an angle range of 1 / 5 of the processing range of the azimuth angle (θ / 5). That is, the processing range of the azimuth angle is divided into 5 parts. The distance acquisition unit 11 acquires a new target distance in each of the divided angle ranges. The distance array generation unit 12 updates the distance array by arranging the newly acquired target distances in the elements corresponding to each angle range. In the example shown in Figure 7, the distance array generation unit 12 arranges each target distance acquired with the modified resolution for the azimuth angle a4 corresponding to the fifth element from the left in the distance array F1. For example, in the distance array F2 within the processing range of azimuth angle a1, the target distances corresponding to each angular range are, from left to right, 25m, 40m, 40m, 40m, and 40m.

[0053] Furthermore, if a distance array is generated for each height, or if a distance array is generated with distances arranged based on the changed resolution, the distance array generation unit 12 generates a new distance array with the same data structure in the next cycle. The object detection unit 15 then detects a new object by comparing the updated distance array, which is an update from the previously generated distance array, with the newly generated distance array. In addition, the distance array generation unit 12 may generate a new distance array for comparison with the next distance array (time t+1) using the conventional method described above, separate from the distance array with the same data structure used for comparison with the previous distance array (time t-1).

[0054] As described above, in the object detection method and object detection device according to this embodiment, the controller obtains the distance from a sensor having an imaging means mounted on the vehicle to an object in the vicinity of the vehicle using a distance measuring means, generates a distance array in which the distances are arranged according to the azimuth angle based on the position of the sensor, calculates the optical flow from the image captured by the imaging means around the vehicle, updates the distance array based on the optical flow and the vehicle speed of the vehicle, and detects the object based on the difference between the updated distance array, which is updated from the distance array generated in the past, and the newly generated distance array. As a result, objects can be detected with high accuracy even when only a portion of the object is visible in the image.

[0055] Furthermore, in the object detection method and object detection device according to this embodiment, the controller treats a newly detected object as a new object, calculates the relative velocity of the new object with respect to the vehicle when the new object is stationary, obtains a measured value of the amount of movement of the new object, and determines whether the new object is a moving object based on the difference between the apparent amount of movement of the new object calculated from the relative velocity and the measured amount of movement. This makes it possible to determine whether a newly detected object is a moving object or not.

[0056] Furthermore, in the object detection method and object detection apparatus according to this embodiment, the imaging means captures an image at a faster frequency than the distance measuring means acquires the distance, and stores the captured image in memory. This allows for earlier detection of a moving object than the next distance measurement.

[0057] Furthermore, in the object detection method and object detection apparatus according to this embodiment, the controller obtains measured values ​​of movement by calculating optical flow based on past images stored in memory. This makes it possible to track the area around a new object retrospectively.

[0058] Furthermore, in the object detection method and object detection apparatus according to this embodiment, the controller recognizes an occluding object present in front of the new object based on point cloud information indicating the position of the object acquired by the distance measuring means, generates a template of the new object with the area of ​​the occluding object in the image masked, and obtains a measured value of the amount of movement by calculating the optical flow based on the generated template. This makes it possible to track the movement of the new object more accurately.

[0059] Furthermore, in the object detection method and object detection apparatus according to this embodiment, the controller obtains a measurement of the amount of movement by calculating an optical flow with a higher density than the optical flow calculated to update the distance array. This makes it possible to track the movement of new objects more accurately.

[0060] Furthermore, in the object detection method and object detection apparatus according to this embodiment, the controller generates distance arrays for each of several different predetermined heights, and detects objects at each predetermined height based on the difference between the updated distance array and the newly generated distance array. This makes it possible to detect objects with high accuracy even when only a portion of the object is visible in the image at different heights.

[0061] Furthermore, in the object detection method and object detection device according to this embodiment, the controller changes the resolution of the angular range, which is the unit for acquiring distance within the range of the azimuth angle, for each azimuth angle, and acquires the distance with the changed resolution to generate a distance array. This makes it possible to generate a distance array in which distances acquired with different resolutions for each azimuth angle are arranged.

[0062] Furthermore, in the object detection method and object detection device according to this embodiment, the controller calculates the second derivative of the distance included in the corresponding distance array for each azimuth angle, and sets the resolution in the range of azimuth angles where the second derivative is greater than or equal to a predetermined value to be higher than the resolution in the range of azimuth angles where the second derivative is less than the predetermined value. This makes it possible to acquire distance with higher resolution in angular ranges that include locations where the amount of distance change is large.

[0063] While embodiments of the present invention have been described above, these embodiments are provided to facilitate understanding of the present invention and are not intended to limit it. Therefore, each element disclosed in the above embodiments is intended to include all design modifications and equivalents that fall within the technical scope of the present invention. [Explanation of symbols]

[0064] 1... Object detection device 2…Distance measuring device 3…Imaging device 4. Vehicle information acquisition device 10...Control device 11...Distance acquisition section 12... Distance array generation unit 13…Optical Flow Calculation Unit 14... Distance array update section 15...Object detection unit

Claims

1. An object detection method performed by a controller, The aforementioned controller, The distance measuring means obtains the distance from a sensor having an imaging means mounted on the vehicle to an object in the vicinity of the vehicle. A distance array is generated by arranging the distances for each azimuth angle based on the position of the sensor. The optical flow is calculated from the image captured by the imaging means around the vehicle. Based on the optical flow and the vehicle speed of the vehicle, the distance array is updated. An object detection method for detecting an object based on the difference between an updated distance array, which is obtained by updating the distance array from one previously generated, and a newly generated distance array.

2. The object detection method according to claim 1, The aforementioned controller, The newly detected object is treated as a new object, and the relative speed of the new object with respect to the vehicle is calculated when the new object is stationary. The measured amount of movement of the aforementioned new object is obtained. An object detection method for determining whether a new object is a moving object based on the difference between the apparent amount of movement of the new object calculated from the relative velocity and the measured amount of movement.

3. The object detection method according to claim 2, The object detection method comprises an imaging means that captures an image at a faster frequency than the period during which the distance measuring means acquires the distance, and stores the captured image in a memory.

4. The object detection method according to claim 3, The aforementioned controller, An object detection method that obtains a measured value of the amount of movement by calculating optical flow based on past images stored in the memory.

5. The object detection method according to claim 2, The aforementioned controller, Based on the point cloud information indicating the position of the object obtained by the distance measuring means, an obstructing object present in front of the new object is recognized. A template of the new object is generated by masking the area of ​​the occluding object in the aforementioned image. An object detection method that obtains a measured value of the displacement by calculating the optical flow based on the generated template.

6. The object detection method according to claim 2, The aforementioned controller, An object detection method for obtaining a measurement of the amount of displacement by calculating an optical flow with a density higher than the optical flow calculated to update the distance array.

7. The object detection method according to claim 1, The aforementioned controller, The distance array is generated for each of several different predetermined heights, An object detection method for detecting an object at each predetermined height based on the difference between the updated distance array and the newly generated distance array.

8. The object detection method according to claim 1, The aforementioned controller, An object detection method that generates a distance array by changing the resolution of the angular range, which is the unit for acquiring the distance within the range of the azimuth angle, for each of the aforementioned azimuth angles, and acquiring the distance with the changed resolution.

9. The object detection method according to claim 8, The aforementioned controller, For each azimuth angle, calculate the second derivative of the distance included in the corresponding distance array. An object detection method comprising setting the resolution in the range of the azimuth angle where the second derivative value is greater than or equal to a predetermined value to be higher than the resolution in the range of the azimuth angle where the second derivative value is less than the predetermined value.

10. An object detection device equipped with a controller, The aforementioned controller, The distance measuring means obtains the distance from the position of a sensor having an imaging means mounted on the vehicle to objects around the vehicle, A distance array is generated by arranging the distances for each azimuth angle based on the position of the sensor. The optical flow is calculated from the image captured by the imaging means around the vehicle. Based on the optical flow and the vehicle speed of the vehicle, the distance array is updated. An object detection device that detects an object based on the difference between an updated distance array, which is obtained by updating the distance array from one previously generated, and a newly generated distance array.

Images