Object detection by high-resolution clustering

Abstract

The invention relates to a method for detecting a portion of surroundings of a vehicle. The method involves repeatedly scanning the portion by means of an active sensor, and in the process generating a plurality of point data sets (PS(tk)) in respect of the portion at different times (tk). Furthermore, a fused point data set (PSF) is generated on the basis of the plurality of point data sets (PS(tk)) by applying a filter (F). Finally, an algorithm for object detection is applied to the fused point data set (PSF). The invention also relates to a detection device (50). Furthermore, the invention relates to a vehicle (60).

Description

[0001] Description

[0002] Object detection through high-resolution clustering

[0003] The invention relates to a method for detecting a partial area of ​​a vehicle's environment. The invention also relates to a detection device. Furthermore, the invention relates to a vehicle equipped with such a detection device.

[0004] When using automated and driverless vehicles, especially rail vehicles, obstacle detection systems are needed to identify objects along a track. Active sensor systems can be used for obstacle detection; these systems actively emit electromagnetic waves and detect electromagnetic waves reflected or scattered by objects. Lidar sensor systems are particularly common as active sensor systems for environmental monitoring. A lidar sensor system emits laser pulses and receives sensor signals from light beams generated by the reflection and scattering of the laser pulses by objects. The laser periodically changes its orientation during monitoring, so that a monitoring area is scanned periodically. The detection range of lidar sensors is limited primarily by the resolution of the lidar sensor, in addition to the signal strength of the laser.However, lasers from lidar systems can only have a limited signal strength, otherwise they would no longer meet the safety regulations, especially regarding the protection of the eyes of people who are in the monitored area.

[0005] To reliably detect an obstacle, the number of reflection points on the obstacle should be as high as possible. A high number of reflection points is associated with a high degree of confidence that a measurement is indeed capturing a real object and not a false detection. Such false detections can be caused, for example, by noise, ground echoes, or other environmental influences. Related lidar points, which may originate from the same object, are grouped together using a process called clustering. Clustering, or cluster analysis, is a method for discovering similarity patterns in datasets, in this case, lidar points. Groups of lidar points with similar properties are assigned to a cluster. These clusters are then tracked using temporal filtering, for example, implemented with a tracking filter such as a caiman filter.Clusters with a minimum number of lidar points and a minimum spatial extent are classified as objects. Clusters with a small number of lidar points or insufficient spatial extent are discarded, as they are assumed to have been erroneously generated by interference or are not sufficiently relevant due to their small size.

[0006] For illustrative purposes, FIG 1 shows a lidar point cloud which, in addition to vegetation, also depicts a person at a distance of 30 m.

[0007] Figure 2 illustrates the same scenario as Figure 1, but at a distance of 100 m. Both figures show that, due to the polar measurement characteristics of lidar, fewer lidar reflection points are measured for the same object at greater distances. Figures 1 and 2 also demonstrate that smaller objects are measured with fewer lidar points than larger objects at the same distance. As mentioned earlier, an insufficient number of detected lidar points leads to the points being classified as caused by interference and discarded, resulting in the failure to detect a potentially present object.

[0008] Previously, lidar sensors with higher resolution were used to improve object detection. However, this approach involves increased resource consumption and therefore high costs.

[0009] Alternatively, several conventional lidar sensors were used, with overlapping detection areas, thus also increasing the density of detected lidar points. However, this approach also significantly increases the complexity compared to a lidar system with only one sensor.

[0010] Furthermore, the measurement period for generating a point cloud was also increased. This resulted in a lower measurement rate and thus an increased tendency towards motion blur.

[0011] The task is therefore to enable reliable detection of objects that, with conventional scanning by active sensors with only a few sampling points, cannot be reliably detected or are even not recognized as objects. This task is solved by a method for detecting a partial area of ​​a vehicle's environment according to claim 1, a detection device according to claim 12, and a vehicle according to claim 13.

[0012] In the inventive method for detecting a partial area of ​​a vehicle's environment, the partial area is scanned several times by an active sensor, preferably a lidar sensor, but not limited to that. The vehicle's environment includes, in particular, the areas that lie in the direction of travel within the vehicle's light path profile and around it, and are therefore of particular interest for the vehicle's operation.

[0013] In this process, multiple point data sets are generated from the sub-area at different times. Each point data set contains multiple sampling points, which are generated by actively scanning the sub-area and result from the scanning direction of the active sensor and the propagation time of its signal. For example, the active sensor fully scans the sub-area in a first time interval and generates a first point data set after completing the scan. The active sensor then scans the sub-area again and generates a second point data set after the second scan at a second time, and so on.

[0014] Furthermore, a fused point dataset is generated based on a combination of multiple point datasets. Finally, an object detection algorithm is applied to the fused point dataset, and if successful, an object is detected.

[0015] By fusing multiple point datasets, it becomes possible to recognize even small objects and, in particular, to detect objects located at a greater distance from the active sensor, since the fused point dataset contains more data points than individual point datasets. Therefore, at the same distance to an object, the number of sensors required to detect it can be reduced. Furthermore, the higher information density increases the reliability of object detection at the same distance.By dividing the measurement time into measurement intervals for multiple point data sets, the option is created to determine the position and dimensions of a detected object based on only one of the point data sets. This avoids motion blur in determining the object's position and dimensions, despite the longer overall measurement time for the merged point data set. The approach according to the invention thus eliminates the disadvantages of conventional solutions, which consist of increased resource consumption and / or motion blur.

[0016] The detection device according to the invention comprises a scanning unit for repeatedly scanning the partial area by an active sensor. During the scanning process, a plurality of point data sets of the partial area are generated at different times.

[0017] Part of the detection device according to the invention is also a fusion unit for generating a fused point data set based on the plurality of point data sets by combining the individual point data sets with each other.

[0018] The detection device according to the invention also includes a detection unit for applying an object detection algorithm to the fused point data set. The detection device according to the invention shares the advantages of the inventive method for detecting a partial area of ​​a vehicle's environment.

[0019] The vehicle according to the invention, preferably a rail vehicle, comprises the detection device according to the invention and a control unit for controlling the vehicle's movement based on object detection by the detection device. The vehicle according to the invention shares the advantages of the detection device according to the invention.

[0020] A large proportion of the aforementioned components of the detection device can be implemented wholly or partially as software modules in a processor of a suitable computer system, e.g., a vehicle control unit. A largely software-based implementation has the advantage that even existing computer systems can be easily retrofitted via a software update to operate according to the invention. In this respect, the problem is also solved by a corresponding computer program product with a computer program that can be directly loaded into a computer system, containing program sections to execute the steps of the method according to the invention, at least those steps that can be performed by a computer, i.e.,The steps for generating a fused point dataset based on a plurality of point datasets by combining the point datasets and for applying an object detection algorithm to the fused point dataset are executed when the program is run on the computer system. Such a computer program product may, in addition to the computer program itself, include additional components such as documentation and / or additional components, including hardware components such as hardware keys (dongles, etc.) for using the software.

[0021] For transport to and / or storage on or in the computer system or control unit, a computer-readable medium, such as a memory stick, a hard drive, or other portable or permanently installed data carrier, can be used. This medium stores the program sections of the computer program that can be read and executed by a computer system. The computer system may, for example, include one or more cooperating microprocessors or similar components.

[0022] The dependent claims and the subsequent description each contain particularly advantageous embodiments and further developments of the invention. In particular, the claims of one claim category may also be further developed analogously to the dependent claims of another claim category and their descriptive parts. Furthermore, within the scope of the invention, the various features of different embodiments and claims may also be combined to form new embodiments.

[0023] In one embodiment of the inventive method, the position and dimensions of the detected object are measured based on a current point data set, which is preferably stored in a buffer memory. The measurement data are then arranged in a measurement vector. The measurement vector is preferably filtered using a Kalman filter method, and a filtered measurement vector is preferably output as the result. A buffer memory is understood to be a data storage device in which data is temporarily stored. The buffer memory stores the data temporarily and can be designed as volatile or non-volatile memory. In one embodiment of the inventive method for capturing a partial area of ​​a vehicle's environment, the most recently acquired point data set is used for one of the following measurements:

[0024] - a localization of a detected object,

[0025] - Determining the dimensions of a detected object.

[0026] The measurement vector required for the aforementioned measurements is therefore not determined based on the merged point dataset, but only on the current, most recent point dataset. This avoids any inaccuracies caused by object movement, which would occur when using the merged point dataset, since only point data recorded at approximately the same time or at least within the same time interval are used.

[0027] In a particularly preferred embodiment of the inventive method for capturing a partial area of ​​a vehicle's environment, the majority of point data sets are first stored in a buffer memory. A predetermined number of the most recently stored point data sets are then read from the buffer memory to generate the fused point data set. The number of point data sets to be fused can depend on the dynamics of the event. This is because it should be prevented that the object changes too much in terms of its pose and shape between the times or time intervals in which the individual point data sets are captured, as this could otherwise lead to a kind of blurring or motion blur effect. To compensate for this effect, the movement of the sensor can be factored out.If the movement of the object is determined based on the position changes of the points and, in particular, the object's center of gravity in the individual point data sets, this information can also be used to correct the object's pose in the individual point data sets in order to avoid the aforementioned blurring effect and to increase the number of fused point data sets.

[0028] Preferably, during the storage process in the buffer, older point data records are overwritten by newer point data records. In this way, the point data records stored in the buffer are constantly updated. A ring buffer is particularly preferred. Advantageously, in a ring buffer, the memory location adjacent to the current memory location can be automatically used to be overwritten for the next point data record. In a ring buffer, the oldest stored data is overwritten with the newest data to be stored when the ring buffer is completely full. To visualize this, the ring buffer is a ring-shaped memory in which each data storage location is adjacent to exactly two other data storage locations, thus forming a closed ring. The data is stored in a circular fashion until all data storage locations are occupied.The data storage space used first is then overwritten with new data, and so on. In this way, the oldest data is always overwritten, and the stored data retains the maximum possible up-to-dateness.

[0029] The length of the buffer, particularly the ring buffer, is preferably determined based on the measurement time for a single point data set and the expected velocity of an object to be detected in the sub-area. The measurement time is thus chosen as a compromise between a short and a long measurement time, such that the longest possible measurement time achieves the densest possible measurement of the sub-area with the highest possible point data density, while the shortest possible measurement time achieves the lowest possible motion blur of a detected object.

[0030] In one variant of the inventive method for capturing a partial area of ​​a vehicle's surroundings, an irregular scan pattern is selectively used during each subsequent scan of the partial area in order to target different points for each point data set. The fused point data set is, figuratively speaking, created by superimposing the point clouds of the individual combined point data sets. To prevent the same locations on an object from being scanned repeatedly, an irregular scanning method, particularly in the case of lidar sensors, ensures a scan pattern with different scanning positions for each pass. An irregular scan pattern comprises a pattern that does not follow a recurring scheme. Such an irregular scan pattern is preferably generated randomly.

[0031] Preferably, a regular, uniform scan pattern is applied for each subsequent scan of the sub-area, and different points are scanned for each point data set through vibrations of the scanning sensor and movements of the vehicle. In this case, different measurement locations are achieved through the movement of the vehicle and the resulting vibrations of the active sensor.

[0032] Preferably, in the inventive method, the fused dataset is first subjected to clustering before object detection is performed based on the cluster. Clustering is used to assign a plurality of points from the point cloud of the fused point data to one and the same object. If a sufficient number of points are assigned to a cluster, an object comprising these points is detected. In contrast to classification, clustering can be used for the detection of any object, even unknown ones.

[0033] Tracking filtering is also a preferred method for determining the dimensions of a detected object. Tracking filtering stabilizes the measurement vector generated from a single point data set. The high measurement rate prevents latencies that could occur when combining multiple measurement points. In contrast, with an extended measurement interval, the sensor's first measurement points would only be acquired after the extended interval has ended – thus introducing a latency.

[0034] In one embodiment of the method according to the invention, it is checked whether all point data sets to be merged have similar characteristics, in particular the same geometric dimensions, and whether any differing point data sets are discarded. This is intended to prevent measurements of different objects from being considered in a single cluster. If objects come close to each other in reality (e.g., pedestrians crossing at a traffic light from both sides), they should not be unintentionally grouped together in the overall point cloud. Characteristics include features that identify or classify an individual object or object type.

[0035] Preferably, the position of the detected object is determined based on each individual point data set, and the object's velocity is calculated based on any change in position over time. In this approach, point data sets recorded at different times, each showing a different position of the detected object, are used to locate the object, and a trajectory is determined as a function of the recording time. A velocity vector can then be derived from this trajectory.

[0036] Preferably, the dimensions of the detected object are determined based on the fused point data set. During fusion, the individual point data sets are transformed based on the known velocity of the object to be detected. If the object's dimensions do not change, it suffices to "shift" or "superimpose" the individual point data sets based on the known velocity of the object to be detected. The point data thus fused are preferably used to determine the dimensions of the detected object. This avoids distortion of the dimensions due to motion effects between the individual point data sets. This results in an improved estimation of the dimensions of the detected, moving object.

[0037] For a static object, the overall point cloud, due to its higher resolution, provides a better basis for determining the dimensions of the observed object. Since this is not known a priori, it must first be determined whether the observed object is moving.

[0038] Alternatively, the movement information can also be used for the fusion of the point data sets determined by the downstream tracking filter.

[0039] Preferably, the active sensor unit comprises one of the following sensor types:

[0040] - a lidar sensor unit,

[0041] - a high-resolution radar sensor unit.

[0042] Lidar sensor units are particularly precise, but can be disrupted by weather conditions.

[0043] Radar typically has a slightly lower resolution, but is not affected by rain or snow.

[0044] In a preferred embodiment of the detection device according to the invention, a measuring unit is also part of the detection device. This measuring unit is configured to determine the position and dimensions of the detected object, arranged in a measurement vector, based on a current point data set, which is preferably stored in a buffer memory. The measuring unit preferably receives information from the detection unit regarding detected objects in order to determine their position and dimensions based on the current point data set. The measurement vector is transmitted to a filter unit, which is also part of the detection device. This filter unit processes the measurement vector using a Kalman filter method and outputs a filtered measurement vector.

[0045] The invention is explained in more detail below with reference to the accompanying figures and exemplary embodiments. The figures show:

[0046] FIG 1 shows a lidar point cloud which illustrates vegetation as well as a person at a distance of 30 m.

[0047] FIG 2 shows the same scenario as FIG 1 at a distance of 100 m.

[0048] FIG 3 shows a flowchart illustrating a method for capturing a sub-area of ​​a vehicle's environment.

[0049] FIG 4 shows a schematic representation which illustrates in detail a measurement process and a filtering process of the method illustrated in FIG 3,

[0050] FIG 5 shows a schematic representation of a detection device according to an embodiment of the invention,

[0051] FIG 6 shows a schematic representation of a rail vehicle according to an embodiment of the invention.

[0052] Figure 1 shows a lidar point cloud 10, which illustrates vegetation as well as a person P at a distance of 30 m. The person P is represented by a multitude of lidar points.

[0053] FIG. 2 shows the same scenario as FIG. 1 at a distance of 100 m. The person P, already shown in FIG. 1 and represented in FIG. 2 by points within a rectangular frame, exhibits only a small number of lidar points when scanned from the greater distance. FIG. 3 shows a flowchart 300 illustrating a method for sensing a partial area of ​​a vehicle's surroundings according to an embodiment of the invention.

[0054] In step 3.I, the sub-area is repeatedly scanned by an active sensor, generating a plurality of point data sets PS(tk) of the sub-area at different times tk.

[0055] In step 3.II, a merged point data set PSF is generated based on the plurality of point data sets PS(tk) through a fusion process F.

[0056] In step 3.1, an algorithm for detecting an object O is applied to the fused point dataset PSF.

[0057] Figure 4 shows a schematic representation 40, which depicts in detail a measurement process and a filtering process of the method illustrated in Figure 3. Figure 4 initially shows an ordinal number OZ-LM of lidar measurements, which are taken at different times depending on the time t. The sensor data from several lidar measurements, in Figure 4 N = 4 lidar measurements, are to be superimposed. For this purpose, the last 4 measurements are written to a ring buffer RP. With each new lidar measurement, the oldest entry in the ring buffer is overwritten. Thus, at the fifth time ts, the ring buffer RP(ts) contains the second to fifth lidar measurements, with the fifth lidar measurement having overwritten the first lidar measurement. To prevent motion blur due to the movement of the vehicle, the following steps are performed:To avoid interference from the lidar sensor, the sensor's or vehicle's own movement is factored out of the sensor data.

[0058] In the ring buffer RP, a dense point cloud is created from the superposition of all LiDAR point clouds stored in the buffer. For object detection, clustering is then performed based on this dense point cloud.

[0059] To locate the detected object and determine its dimensions or pose, a measurement vector z(4), assigned to the fourth time point t4, and a measurement vector z(5), assigned to the fifth time point ts, are not derived from the entire cluster or the entire densified point cloud. Instead, the measurement vectors z(4) and z(5) are determined from the lidar points of the last measurement, i.e., in the case shown in FIG. 4, from the lidar points acquired at the fourth time point t4 and the fifth time point ts, respectively. For this purpose, the position and geometric dimensions of the detected object O are determined from the individual lidar points, for example, by center-of-gravity analysis. Separate clustering is not necessary for this.The determined values ​​of the measurement vector z, in particular the vectors z(4), z(5), are stabilized via a temporal filter FL, so that “outliers” due to measurement errors are prevented.

[0060] FIG 5 shows a schematic representation of a detection device 50 according to an embodiment of the invention.

[0061] The detection device 50 comprises a scanning unit 51 for repeatedly scanning the sub-area with an active sensor. During the scanning process, sensor data SD is acquired, generating a plurality of point data records PS(tk) of the sub-area at different times tk. For this purpose, the active sensor moves periodically, and the scanning area of ​​the active sensor repeatedly sweeps over the sub-area.

[0062] The detection device 50 also has a buffer memory 51a in which the recorded point data sets PS(tk) are stored as a group.

[0063] Part of the detection device 50 is also a fusion unit 52 for generating a fused point data set PSF based on the majority of point data sets PS(tk) through a clustering process.

[0064] Furthermore, the detection device 50 includes a detection unit 53, which is set up to perform object detection based on the fused point data set PSF.

[0065] Part of the detection device 50 is a measuring unit 54, which is configured to perform a measurement of the position and dimensions of the detected object O based on a current point data set PS(ti) stored in the buffer memory 51a. These measurements are arranged in a measurement vector z. The measuring unit 54 receives information from the detection unit 53 regarding detected objects O in order to determine their position and dimensions based on the current point data set PS(ti). The measurement vector z is transmitted to a filter unit 55, which filters the measurement vector using a Kalman filter method and outputs a filtered measurement vector Zf.

[0066] Figure 6 shows a schematic representation of a rail vehicle 60 according to an embodiment of the invention. The rail vehicle 60 includes a detection device 50, as illustrated in Figure 5. The detection device scans the area around the rail vehicle 60 for objects O that might potentially pose an obstacle. Based on the detected objects O, a control unit 61 is supplied with information about the area around the rail vehicle 60.

[0067] Finally, it should be noted once again that the methods and devices described above are merely preferred embodiments of the invention and that the invention can be varied by a person skilled in the art without departing from the scope of the invention, insofar as it is defined by the claims. For the sake of completeness, it should also be noted that the use of the indefinite articles "a" or "an" does not preclude the possibility that the features in question may be present multiple times. Likewise, the term "unit" does not preclude the possibility that it consists of several components, which may also be spatially distributed. Regardless of the grammatical gender of a particular term, persons of male, female, or other gender identities are included.

Claims

Patent claims 1. Method for detecting a sub-area of ​​a vehicle's environment (60), comprising the steps: - repeated scanning of the sub-area by an active sensor, whereby a plurality of point data sets (PS(tk)) of the sub-area are generated at different times (tk), - Generating a fused point dataset (PSF) based on the plurality of point datasets (PS(tk)) by combining the point datasets (PS(tk)), - Applying an algorithm to detect an object (O) on the fused point dataset (PSF).

2. The method of claim 1, wherein the last recorded point data set (PS(ti)) is used for one of the following measurements: - a localization of a detected object (O), - a determination of the dimensions of a detected object (O).

3. Method according to one of the preceding claims, wherein the plurality of point data sets (PS(tk)) are first stored in a buffer memory (PS) and a predetermined number of recently stored point data sets (PS(tk)) are read from the buffer memory (PS) to generate the fused point data set (PSF).

4. Method according to claim 3, wherein older point data sets (PS(tk)) are overwritten by newer point data sets (PS(tk)) during the storage process.

5. Method according to claim 3 or 4, wherein the buffer storage (PS) comprises a ring buffer storage.

6. Method according to one of claims 3 to 5, wherein the length of the buffer memory is determined as a function of the measurement time for a point data set (PS(tk)) and of the expected speed of an object (O) to be detected in the sub-area.

7. Method according to one of the preceding claims, wherein an irregular scan pattern is used in each subsequent scan of the sub-area in order to target different points for each point data set (PS(tk)).

8. Method according to one of claims 1 to 6, wherein a regular scan pattern is applied during each re-scanning of the sub-area and scanning of different points per point data set (PS(tk)) is achieved by vibrations of the active sensor and movements of the vehicle (60).

9. Method according to one of the preceding claims, wherein the fused point data set (PSF) is first subjected to clustering before object detection is performed on the basis of the cluster.

10. Method according to one of the preceding claims, wherein tracking filtering is performed to determine the dimensions of a detected object (O) and thereby prevents latencies that could occur from combining multiple measurement times.

11. Method according to any of the preceding claims, wherein at least one of the following measures is carried out: - It is checked whether all point data sets (PS(tk)) to be merged have similar characteristics and, if necessary, discard any differing point data sets (PS(tk)). - Based on the individual point data sets (PS(tk)), a position of the detected object (O) is determined, and based on a change in position over time, a speed of the object (O) is determined. - The determination of the dimensions of the detected object (O) is based on the fused point data set (PSF), and during fusion, the individual point data sets (PS(tk)) are transformed based on the knowledge of the velocity of the detected object (O).

12. Detection device (50), comprising: - a scanning unit (51) comprising an active sensor for repeatedly scanning a sub-area of ​​a vehicle's environment (60), wherein a plurality of point data sets (PS(tk)) are generated from the sub-area at different times (tk), - a fusion unit (52) for generating a fused point data set (PSF) based on the plurality of point data sets (PS(tk)) by combining the point data sets (PS(tk)), - a detection unit (53) for applying an object detection algorithm to the fused point data set (PSF).

13. Vehicle (60), preferably a rail vehicle, comprising - a detection device (50) according to claim 12 - a control unit (61) for controlling a journey of the rail vehicle (60) based on object detection by the detection device (50).

14. Computer program product comprising instructions which, when the program is executed by a computer, cause the computer to perform the steps for generating a fused point data set (PSF) based on the plurality of point data sets (PS(tk)) by combining the point data sets (PS(tk)) and for applying an object detection algorithm to the fused point data set (PSF) of the method according to any one of claims 1 to 11.

15. Computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform the steps for generating a fused point data set based on the plurality of point data sets (PS(tk)) by combining the point data sets (PS(tk)) and for applying an object detection algorithm to the fused point data set (PSF) of the method according to claims 1 to 11.

Images