Method and apparatus for assigning a bounding box to an object

Through sensor data and algorithm processing, the selection of merging or separation of bounding boxes solves the problem of inaccurate bounding boxes assignment in the vehicle environment in the prior art, and achieves the accuracy of the accurate identification and perception system of large vehicles and improves the reliability of the system.

CN115115824BActive Publication Date: 2024-11-19APTIV TECHNOLOGIES AG
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210275377.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2021-03-23
Filing Date
2022-03-21
Publication Date
2024-11-19
Estimated Expiration
2042-03-21

AI Technical Summary

Technical Problem

Existing automotive perception systems are difficult to provide accurate bounding boxes in vehicle environments, especially for large vehicles, which are affected by bias and size differences in vehicle size distribution, resulting in multiple small bounding boxes being assigned to large vehicles.

Method used

The vehicle environment data is obtained through sensors, based on the speed and position of the preliminary bounding box, select the merge or separate bounding box, generate the final bounding box, and use the maximum connected sub-graph algorithm and machine learning algorithm to determine the spatial position and size of the bounding box to ensure the bounding box. Overwrite the object without being affected by the dimensions.

Benefits of technology

Accurate bounding box assignments to objects in the vehicle environment are realized, the reliability and accuracy of the perception system are improved, multiple bounding box assignments to large vehicles are avoided, and the recognition ability of trucks and buses is enhanced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115115824B_ABST
    Figure CN115115824B_ABST
Patent Text Reader

Abstract

The present disclosure relates to a method and apparatus for assigning a bounding box to an object. A method for assigning a bounding box to an object in an environment of a vehicle is provided. Data related to an object located in the environment of the vehicle is acquired via a sensor. Based on the data, a corresponding spatial position and a corresponding size of a plurality of preliminary bounding boxes are determined, so that each preliminary bounding box at least partially covers one of the objects. A corresponding speed of each preliminary bounding box is estimated based on the data. A subset of the plurality of preliminary bounding boxes that is related to a corresponding one of the objects is selected, wherein the selection of the subset depends on the corresponding speed of each preliminary bounding box in the preliminary bounding boxes. A final bounding box is assigned to the corresponding one of the objects by merging the preliminary bounding boxes of the corresponding subsets.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to methods and apparatus for assigning bounding boxes to objects in a vehicle environment. Background Art

[0002] Reliable perception of the vehicle's environment is essential for autonomous driving and driver assistance systems. Radar and LiDAR sensors are now commonly used in automotive perception systems because they provide the shape and distance of objects in the vehicle's environment and are less affected by weather than, for example, camera systems.

[0003] The data captured by radar and lidar sensors are usually provided in the form of a so-called bird's eye view (BEV), i.e., as if looking from above at the plane on which the vehicle is currently moving. In the bird's eye view, the size of objects is independent of their distance to the corresponding radar or lidar sensor installed in the vehicle. However, the size of objects belonging to different types may vary greatly. For example, a pedestrian is much smaller than a car, which is usually much smaller than a truck or a bus.

[0004] When considering only vehicles, the number of cars in a vehicular environment is usually higher than the number of trucks or buses. Therefore, the size distribution of vehicles is mostly biased towards small vehicles. As a result, perception systems that rely on machine learning or neural networks and are trained using this biased vehicle size distribution as ground truth tend to prefer small vehicles, for example by assigning small bounding boxes to any type of vehicle. As a result, multiple small bounding boxes may be assigned to a large vehicle. The biased vehicle size distribution and the large variance in size may prevent the perception system from providing accurate bounding boxes for large vehicles.

[0005] Data balancing has been proposed to address this bias problem with respect to vehicle size. That is, the object class for small vehicles can be downsampled, while the object class for large vehicles can be oversampled. Although some improvements can be achieved through such data balancing, in many cases multiple small bounding boxes are still assigned to large vehicles.

[0006] Therefore, it is desirable to provide a method and apparatus that can assign an accurate bounding box to an object regardless of the size of the object. Summary of the invention

[0007] The present disclosure provides a computer-implemented method, a computer system, and a non-transitory computer-readable medium.

[0008] In one aspect, the present disclosure relates to a computer-implemented method for assigning bounding boxes to objects in an environment of a vehicle. According to the method, data related to objects located in the environment of the vehicle are acquired via a sensor. Via a processing unit, respective spatial positions and respective sizes of a plurality of preliminary bounding boxes are determined based on the sensor data, such that each preliminary bounding box at least partially covers one of the objects. Also via the processing unit, respective velocities of each preliminary bounding box are estimated based on the sensor data. Via the processing unit, a subset of the plurality of preliminary bounding boxes that is associated with a respective one of the objects is selected, wherein the step of selecting a subset depends on the respective velocities of each of the preliminary bounding boxes. Via the processing unit, a final bounding box is assigned to the respective one of the objects by merging the preliminary bounding boxes of the corresponding subset.

[0009] The method relies on the general idea that fast-moving objects are unlikely to be close together in their direction of movement. Therefore, based on the speed of the corresponding objects, the method is able to select the correct subset of preliminary bounding boxes that actually belong to an object. That is, for high speeds, different preliminary bounding boxes can be merged because they are more likely to belong to the same object rather than to two objects that are close in the direction of movement. On the other hand, for low speeds (i.e., the corresponding low speeds of the preliminary bounding boxes assigned to these objects), preliminary bounding boxes belonging to two different objects can be separated.

[0010] The sensor may be a radar sensor and / or a lidar sensor installed in the vehicle and capable of monitoring, for example, the environment in front of the vehicle. For these sensors, a bounding box may be defined as a rectangle in a bird's-eye view, i.e., a two-dimensional object in a plane parallel to the ground on which the vehicle is currently moving and viewed from a height. In this case, the bounding box may be defined via two vertices located at diagonally opposite corners of the rectangle representing the bounding box. These two vertices may be positioned to set the size and orientation of the bounding box in the bird's-eye view, i.e., the spatial position. However, in general, the bounding box may also be defined by a three-dimensional cuboid via a corresponding set of three vertices.

[0011] To determine the velocity of the respective preliminary bounding box, a data sequence from a sensor, such as a sequence of radar and / or lidar scans, may be used. The velocity of the respective preliminary bounding box may be almost the same as the velocity of the respective object at least partially covered by the preliminary bounding box.

[0012] The spatial position and size of each preliminary bounding box can be determined by a machine learning algorithm (e.g., a neural network). However, if such a machine learning algorithm is trained, the ground truth may be biased to include more small objects, such as cars. Therefore, the machine learning algorithm may output multiple small bounding boxes for a larger object (e.g., a truck or a bus).

[0013] Based on the speed of the preliminary bounding boxes and their spatial positions, the method therefore checks whether more than one of the preliminary bounding boxes is associated with an object, so that the preliminary bounding boxes belonging to a certain subset of the bounding boxes associated with the object need to be merged in order to generate a final bounding box that correctly covers, for example, a larger object. On the contrary, the subset of the preliminary bounding boxes can also consist of only one box. In this case, the method will simply keep this box as the final bounding box. Therefore, the method according to the present disclosure is able to provide correct bounding boxes for objects regardless of their size. That is, a truck or a bus can be correctly covered by a single final bounding box generated by the method according to the present invention.

[0014] The method may include one or more of the following features:

[0015] The direction of the speed may be determined for each preliminary bounding box, and a corresponding resized box may be generated for each preliminary bounding box by the steps of shrinking the preliminary bounding box in the direction of the speed and in a lateral direction perpendicular to the direction of the speed, and expanding the preliminary bounding box in the direction of the speed, wherein the step of expanding the preliminary bounding box depends on an absolute value of the speed estimated for the preliminary bounding box. Preliminary bounding boxes having overlapping resized boxes may be selected for the respective subsets.

[0016] To shrink the preliminary bounding box, the length and width of the corresponding preliminary bounding box may be reduced by the same factor. To expand the preliminary bounding box, the length of the corresponding preliminary bounding box in the direction of the velocity may be multiplied by the absolute value of the velocity and by the expansion factor.

[0017] To select a subset of the plurality of preliminary bounding boxes, a maximum connected subgraph algorithm may be applied to all preliminary bounding boxes. If the intersection-over-union ratio of two resized boxes is greater than a predefined threshold, the two resized boxes may be considered to overlap.

[0018] A centroid may be estimated for a preliminary bounding box of the corresponding subset, and a farthest vertex having a maximum distance from the centroid may be determined for the preliminary bounding box of the subset. The final bounding box may be determined based on the farthest vertex.

[0019] The speed of the final bounding box may be determined by averaging the estimated speeds over the corresponding subset of preliminary bounding boxes. A property may be determined for each of the preliminary bounding boxes, and it may be determined whether the selected preliminary bounding box may be excluded from the corresponding subset based on the property. A probability distribution may be assigned to the values ​​of the property of the preliminary bounding boxes belonging to the corresponding subset, and if the probability of the value of the property assigned to the selected preliminary bounding box is less than a predetermined threshold, the selected preliminary bounding box may be excluded from the corresponding subset. A machine learning algorithm may be applied to the sensor data to determine the spatial position and size of each preliminary bounding box.

[0020] According to an embodiment, the direction of the speed may be determined for each preliminary bounding box. A corresponding modified-size box may be generated for each preliminary bounding box by the following steps: contracting the preliminary bounding box in the direction of the speed and in a lateral direction perpendicular to the direction of the speed; and extending the preliminary bounding box in the direction of the speed, wherein the step of extending the preliminary bounding box may depend on the absolute value of the speed estimated for the preliminary bounding box. A preliminary bounding box having an overlapping modified-size box may be selected for the respective subset to generate a final bounding box for the respective object.

[0021] For this embodiment, a shrinking step and an expanding step may be applied to each preliminary bounding box to cluster the preliminary bounding boxes via overlapping resized boxes belonging to respective objects. Since the shrinking step is independent of speed and affects the length and width of the preliminary bounding boxes, the shrinking step may separate the resized boxes and thus separate the preliminary bounding boxes belonging to different objects. Furthermore, the shrinking step may separate the preliminary bounding boxes in the speed direction as well as in the lateral direction.

[0022] In contrast, the expansion step can only affect the preliminary bounding boxes in the direction of speed. Therefore, different subsets of preliminary bounding boxes can be merged only in the direction of speed, but not vertically in the lateral direction. Since the expansion step depends on the speed, whether the corresponding preliminary bounding boxes are shrunk or expanded in the direction of speed also depends on the speed in order to generate the corresponding modified size boxes. For low speeds, the contraction step may be dominant, so that the method may tend to separate the modified size boxes at low speeds, thereby separating, for example, two different sets of preliminary bounding boxes. On the other hand, for high speeds, the expansion step may be dominant, so that the preliminary bounding boxes can be merged in the direction of speed of fast-moving objects, such as in highway scenes. For such fast-moving objects, it is unlikely that they move close to each other in the direction of speed. On the contrary, different preliminary bounding boxes located close to each other in the direction of speed are likely to actually belong to the same object and can therefore be merged.

[0023] Due to the shrinking step and the expansion step of the preliminary bounding box, the reliability of the final bounding box can be improved, because the preliminary bounding boxes belonging to two different objects can be separated via the shrinking step, and the preliminary bounding boxes belonging to the same object can be merged into one subset due to the expansion step. Since the expansion step depends only on the speed, the decision to merge a set of preliminary bounding boxes into one subset or divide them into two subsets can also depend on the speed.

[0024] To shrink the preliminary bounding boxes, the length and width of the corresponding preliminary bounding boxes may be reduced by the same factor. By this means, when the preliminary bounding boxes may be divided into different subsets via the shrinking step, biases with respect to directions may be avoided.

[0025] In order to expand the preliminary bounding box, the length of the corresponding preliminary bounding box in the velocity direction can be multiplied by the absolute value of the velocity and multiplied by the expansion factor. This multiplication of different factors can be a direct means to implement the expansion step. In addition, the weighting between the expansion step and the contraction step can be controlled by defining a suitable expansion factor.

[0026] To select a subset of the multiple preliminary bounding boxes, a maximum connected subgraph algorithm can be applied to all preliminary bounding boxes. If the intersection-over-union ratio of two resized boxes is greater than a predefined threshold, they can be considered to overlap. These means for selecting subsets and checking the overlap of resized boxes can again improve the reliability of the final bounding boxes while requiring a reasonable amount of computation.

[0027] According to another embodiment, a centroid may be estimated for a preliminary bounding box of a corresponding subset, and a farthest vertex having a maximum distance from the centroid may be determined for the preliminary bounding box of the subset. A final bounding box associated with the subset may be determined based on the farthest vertices. If the bounding box is represented by a rectangle in a bird's eye view, two farthest vertices may be sufficient to determine the final bounding box of the corresponding subset. Therefore, determining the farthest vertices may be a direct process of merging the preliminary bounding boxes of the corresponding subset to generate the corresponding final bounding box. Such a merging process may require a lower amount of computation.

[0028] The velocity of the final bounding box, and thus the velocity of the object assigned to the final bounding box, can be determined by averaging the estimated velocities over a corresponding subset of the preliminary bounding boxes. Thus, the property of the final bounding box (i.e., other than its size and spatial position) can be determined by simply averaging the properties of the final bounding box over all relevant preliminary bounding boxes belonging to the corresponding object.

[0029] According to another embodiment, a property may be determined for each preliminary bounding box, and it may be determined based on the property whether the selected preliminary bounding box can be excluded from the corresponding subset. In other words, by considering the properties of the preliminary bounding boxes, the preliminary bounding boxes that have been selected for the corresponding subset may be considered as outliers, and such outliers may be removed from the relevant subset of preliminary bounding boxes. The property may be the size, orientation or speed of the corresponding preliminary bounding box. For example, it may be determined whether the size or orientation of a certain preliminary bounding box differs too much from the size or orientation of all other preliminary bounding boxes belonging to the same subset. By excluding preliminary bounding boxes that have been identified as outliers, the reliability of the final bounding box may be further improved, thereby improving the reliability of the entire method.

[0030] In detail, a probability distribution may be assigned to the values ​​of the attributes of the preliminary bounding boxes belonging to the corresponding subset, and if the probability of the value of the attribute assigned to the selected preliminary bounding box is less than a predetermined threshold, the selected preliminary bounding box may be excluded from the corresponding subset. Via the probability distribution and the predetermined threshold, a clear criterion may be defined to determine whether a certain preliminary bounding box is an outlier. Therefore, the reliability of the method may be further improved.

[0031] Furthermore, a machine learning algorithm may be applied to the sensor data to determine the spatial position and size of each preliminary bounding box. Such a machine learning algorithm may provide a solid basis for the further method step of determining the final bounding box.

[0032] In another aspect, the present disclosure relates to an apparatus for assigning a bounding box to an object in an environment of a vehicle. The apparatus comprises a sensor and a processing unit. The sensor is configured to acquire data related to an object located in the environment of the vehicle. The processing unit is configured to: determine a respective spatial position and a respective size of a plurality of preliminary bounding boxes based on the sensor data, such that each preliminary bounding box at least partially covers one of the objects, estimate a respective velocity of each preliminary bounding box based on the sensor data, select a subset of the plurality of preliminary bounding boxes related to a respective one of the objects, wherein the selection of the subset depends on the respective velocity of each of the preliminary bounding boxes, and assign a final bounding box to the respective one of the objects by merging the preliminary bounding boxes of the corresponding subsets.

[0033] As used herein, the terms "processing device", "processing unit" and "module" may refer to, be part of, or include an application specific integrated circuit (ASIC); an electronic circuit; a combinational logic circuit; a field programmable gate array (FPGA); a processor (shared, dedicated, or group) that executes code; other suitable components that provide the functionality; or a combination of some or all of the above, such as in a system on a chip. The term "module" may include a memory (shared, dedicated, or group) that stores code executed by a processor.

[0034] In summary, the device according to the present disclosure includes a sensor and a processing unit configured to perform the steps of the above corresponding method. Therefore, the above benefits, advantages and disclosures for the method are also valid for the device according to the present disclosure.

[0035] According to an embodiment, the sensor may include a radar sensor and / or a lidar sensor. In addition to the size and spatial position of the preliminary bounding box, the data provided by these sensors may also be suitable for determining the velocity of the preliminary bounding box.

[0036] In another aspect, the present disclosure relates to a computer system configured to perform some or all of the steps of the computer-implemented method described herein.

[0037] The computer system may include a processing unit, at least one memory unit and at least one non-transitory data storage. The non-transitory data storage and / or the memory unit may include a computer program for instructing a computer to perform several or all steps or aspects of the computer-implemented method described herein.

[0038] In another aspect, the present disclosure relates to a non-transitory computer-readable medium comprising instructions for performing some or all steps or aspects of the computer-implemented methods described herein. The computer-readable medium can be configured as: an optical medium, such as a compact disk (CD) or a digital versatile disk (DVD); a magnetic medium, such as a hard disk drive (HDD); a solid-state drive (SSD); a read-only memory (ROM); a flash memory; etc. In addition, the computer-readable medium can be configured as a data storage unit accessible via a data connection such as an Internet connection. For example, the computer-readable medium can be an online data repository or cloud storage.

[0039] The present disclosure also relates to a computer program for instructing a computer to perform several or all steps or aspects of the computer-implemented method described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Herein, exemplary embodiments and functions of the present disclosure are described in conjunction with the following drawings, which schematically illustrate:

[0041] Figure 1 depicts an overview of an apparatus according to the present disclosure,

[0042] Figure 2 depicts an example of predicting a bounding box according to background technology,

[0043] Figure 3 depicts an example of clustering bounding boxes according to the present disclosure,

[0044] FIG. 4A to FIG. 4C depicts the challenging scenario of clustering bounding boxes,

[0045] FIG. 5A to FIG. 5C depicts a scheme for separating and merging bounding boxes according to the present disclosure,

[0046] Fig. 6A and Figure 6B depicts an example of removing outliers from a set of bounding boxes, and

[0047] Figure 7 Depicted is a scheme for merging bounding boxes via the farthest vertices.

[0048] Reference numerals list

[0049] 10 Vehicles

[0050] 11. Installation

[0051] 13 Sensors

[0052] 14 Instrument Field of View

[0053] 15 Processing Units

[0054] 17 Objects

[0055] 19 Bounding Box

[0056] 21 Vertex

[0057] 23 Ground Truth

[0058] 25 Predicting Bounding Boxes

[0059] 27 Preliminary Bounding Box

[0060] 29 Final Bounding Box

[0061] 31 Velocity vector

[0062] 33 Frame after resizing

[0063] 37 Outlier Box

[0064] 39 Farthest Vertex DETAILED DESCRIPTION

[0065] Figure 1 A schematic overview of an apparatus 11 according to the present disclosure is depicted, mounted in a vehicle 10. The apparatus 11 comprises a sensor 13 with an instrument field of view 14 and a processing unit 15.

[0066] The sensor 13 is a radar sensor and / or a lidar sensor and is configured to monitor the environment of the vehicle 10. That is, the sensor 13 is configured to provide data related to objects 17 located in the environment of the vehicle 10. The data provided by the sensor 13 is transmitted to the processing unit 15, which is configured to generate corresponding bounding boxes 19 related to each object 17.

[0067] It should be noted that for illustration purposes only, object 17 is Figure 1 In practice, the data provided by the radar sensor and / or lidar sensor 13 is presented in the form of a bird's eye view, i.e., a perspective from above when the vehicle 10 and the objects 17 (including their bounding boxes 19) are viewed. In the bird's eye view, the bounding boxes 19 are rectangles, for each of which the spatial position is uniquely defined by a pair of vertices 21 diagonally opposite each other.

[0068] Figure 2 An example of predicting a bounding box of a large vehicle via a perception system according to the background art is depicted. The ground truth of the corresponding bounding box is denoted by 23 and illustrated by a dashed line. Figure 2 On the left side of , the dashed rectangle 23 represents the ground truth of the truck, while Figure 2 On the right side of , two dashed rectangles 23 represent the ground truth of a double-section bus, ie, the rectangles to be predicted.

[0069] Typically, the perception system according to the background art is based on a machine learning algorithm that is trained with ground truth containing mostly small vehicles (i.e., cars), because in a typical environment of the vehicle 10, the number of cars is usually much higher than the number of trucks and buses. As a result, the perception system according to the background art predicts a plurality of small bounding boxes 25, each of which only partially covers the ground truth of a bus. Figure 2 As can be seen in FIG. 1 , each of the multiple predicted bounding boxes 25 has poor quality compared to the ground truth 23, where Figure 2 The two-section bus on the right is even more difficult to predict. The orientations of some predicted bounding boxes 25 do not fit the orientation of the ground truth rectangle 23 at all.

[0070] Although each single predicted bounding box 25 has poor quality compared to the ground truth 23, the entire set of predicted bounding boxes 25 can better cover the ground truth 23. This is the motivation for one of the principles according to the present disclosure, i.e., merging a certain number of small bounding boxes 25 to generate a large bounding box.

[0071] This is Figure 3 The same ground truth 23 for a truck and a bus is shown twice with dashed lines. On the respective left side, the corresponding Figure 2 The bounding box 25 shown represents a preliminary bounding box 27 of the output of a perception system according to the background art. That is, the preliminary bounding box 27 is the output of a machine learning algorithm that has been trained using ground truth that mainly includes smaller vehicles (such as cars).

[0072] Based on the set of preliminary bounding boxes 27, a final bounding box 29 is generated, which is Figure 3 To generate the final bounding box 29, the so-called maximum connected subgraph algorithm is applied to the instrument field of view 14 of the sensor 13 (see Figure 1 ) within the entire preliminary bounding box 27 identified. Through this algorithm, a cluster or subset of preliminary bounding boxes 27 assumed to belong to a certain object 17 is identified. Figure 2 and Figure 3 If the truck and the bus in the ground truth 23 are located in the same field of view 14 of the sensor 13, the maximum connected subgraph algorithm will be used for each of these objects 17 (i.e., for Figure 2 and Figure 3 The example of a truck and a bus (in this example) generates two subsets or clusters of preliminary bounding boxes 27.

[0073] In the maximum connected subgraph algorithm, an important step is the calculation of the so-called adjacency matrix, which represents the similarity between each pair of preliminary bounding boxes 27. The intersection over union (IoU) known in the art is used as a measure of the similarity of two preliminary bounding boxes 27. Usually, a threshold is defined for the intersection over union in order to decide whether two objects (i.e., two preliminary bounding boxes 27) belong together. For the current case, i.e., for multiple preliminary bounding boxes 27, it is not easy to define a universal threshold, which will be explained in detail below. One reason is that one larger object must be distinguished from two or more smaller objects that are positioned close to each other (i.e., they have only a small gap between them so that the preliminary bounding boxes 27 of the two objects overlap). In defining a certain object 17 in the environment belonging to the vehicle 10 (see Figure 1 ) of the preliminary bounding box 27, by determining the farthest vertex 39 (see Figure 7 ) to generate the final bounding box 29. This will also be explained in detail below.

[0074] exist FIG. 4A to FIG. 4C The challenging scenario of generating a correct final bounding box29 is depicted in Figure 4A and Figure 4B In the example, the ground truth 23 (illustrated by dashed lines) includes the following: Figure 4A ) and in the transverse direction ( Figure 4B ) are bounding boxes of two different smaller vehicles located close to each other. In both cases, the preliminary bounding boxes 27 overlap for the two objects of the ground truth 23. As a result, the entire subset of preliminary bounding boxes 27 will be seen as one large cluster, and thus a single large final bounding box 29 will be generated. That is, the perception system will not be able to distinguish between the two objects represented by the ground truth 23 (e.g. Figure 4A and Figure 4B in ).

[0075] exist Figure 4C In the example of FIG. 1 , the predicted bounding boxes or preliminary bounding boxes 27 are not connected to each other in the middle of the sub-image. This may be caused by the sensor 13 (i.e., the radar sensor and / or the lidar sensor (see FIG. 1 ). Figure 1 As a result, although the ground truth 23 represents a large vehicle, such as a truck, Figure 4C The preliminary bounding box 27 shown will be divided into two clusters or subsets. Therefore, although the ground truth 23 represents a larger object, Figure 4C In the example of , two final bounding boxes 29 are generated, indicating that there are two objects.

[0076] In order to overcome such concerns, the apparatus 11 and method according to the present disclosure additionally consider the corresponding speed of the assumed object, which is reflected in the corresponding speed of the preliminary bounding box 27. In general, the object 17 (see also FIG. 1 ) can also be determined based on the data of the sensor 13 (i.e., based on a sequence of radar and / or lidar scans). Figure 1 Therefore, the respective preliminary bounding boxes 27 located in the region of the respective object 17 are assigned the respective velocities.

[0077] Along the moving direction of the object 17, i.e., along the direction of its velocity vector, the probability of two fast-moving vehicles being close to each other is low. In other words, even if the preliminary bounding boxes 27 do not overlap, these preliminary bounding boxes that are close to each other in the moving direction of the object are more likely to belong to a larger object (e.g., Figure 4C and Figure 5C In contrast, in the lateral direction perpendicular to the direction of movement of one or more vehicles, although the preliminary bounding boxes 27 may overlap, they are more likely to belong to different objects (e.g. Figure 4B and Figure 5B shown in the corresponding left subfigure).

[0078] In order to take into account the speed of the object when generating the bounding box, a shrinking step and an expanding step are applied to the preliminary bounding box 27. The shrinking step is performed in the direction of movement and in the lateral direction, while the expanding step is performed only in the direction of movement.

[0079] Before executing the contraction step and the expansion step, based on the data from the sensor 13 (see Figure 1 ) data for each preliminary bounding box 27 to estimate a corresponding velocity vector 31. Examples of velocity vectors 31 are as follows: FIG. 5A to FIG. 5C As shown. Figure 5A and Figure 5B , assuming that the velocity vector 31 has a smaller absolute value, and for Figure 5C Assume that the absolute value of the velocity vector 31 is much greater than Figure 5A and Figure 5B The absolute value of the velocity vector 31. The corresponding contraction step and expansion step of the preliminary bounding box 27 are defined as follows:

[0080] Shrinkage step: L = L * Shrinkage factor, W = W * Shrinkage factor

[0081] Expansion step: L = L * speed * expansion factor

[0082] Wherein, L and W represent the length (i.e., in the longitudinal or moving direction) and width (i.e., in the lateral direction) of the corresponding preliminary bounding box 27, respectively. The shrink factor is a number between 0 and 1. In the formula of the expansion step, the term "speed" represents the absolute value of the corresponding speed vector 31. The product of the speed and the expansion factor is a number greater than 1.

[0083] The shrinking factor and the expanding factor may be appropriately defined during a training process for the entire apparatus 11. That is, both factors are empirically determined by applying the apparatus 11 and the method to real-life scenarios in which the apparatus 11 and the method are intended to be used.

[0084] for Figure 5A and Figure 5B In the example shown, the preliminary bounding box 27 shown in the left sub-figure is first shrunk in the direction of movement (i.e., along the velocity vector 31) and then shrunk by the same shrinkage factor in the lateral direction (i.e., perpendicular to the velocity vector 31). Thereafter, the preliminary bounding box 27 is expanded only in the direction of movement along the velocity vector 31, wherein the expansion factor is multiplied by the absolute value of the velocity vector 31. As a result of the shrinking step and the expanding step, a modified size box 33 is generated, which is Figure 5A , Figure 5B and Figure 5C Based on the resized boxes 33, it is determined which of the preliminary bounding boxes 27 form a corresponding subset associated with a certain assumed object.

[0085] exist Figure 5A and Figure 5B In the example shown, the plurality of preliminary bounding boxes 27 as shown in the corresponding left sub-figure are divided into two different subsets of resized boxes 33, as can be seen in the middle sub-figure. Figure 5A and Figure 5B In the example of , due to the low absolute value of the velocity vector 31, the shrinking step is dominant over the expanding step, because the expanding step depends on the absolute value of the velocity vector 31, while the shrinking step is independent of the velocity. Due to the dominance of the shrinking step, two different objects represented by the ground truth 23 (illustrated with dashed lines) can be distinguished by the device 11 and the method, and the final bounding box 29 can be assigned to the object it correctly covers.

[0086] In contrast, for Figure 5C In the example shown, it is assumed that the absolute value of the velocity vector 31 is much larger, so that the expansion step prevails over the contraction step. Figure 5C As shown on the left side of FIG, the preliminary bounding boxes 27 do not overlap in the middle of the sub-image, but the advantage of the expansion step results in overlapping resized boxes 33 for the entire subset. In other words, the separation that existed for the preliminary bounding boxes 27 no longer exists for the resized boxes 33. As a result, for example, Figure 5C The example shown generates only one final bounding box 29 that correctly covers the ground truth 23 (illustrated with a dashed line), ie, a larger object like a truck.

[0087] Notice, FIG. 5A to FIG. 5C The resized boxes 33 in the intermediate sub-image of are only used to identify those preliminary bounding boxes 27 that must be grouped in a certain subset. If the corresponding resized box 33 overlaps with another resized box 33 belonging to the same subset, the preliminary bounding box 27 is selected for the subset assumed to belong to a certain object. In other words, if the resized boxes 33 of the preliminary bounding boxes 27 overlap at least partially, these preliminary bounding boxes 27 are grouped in the same subset for generating one of the final bounding boxes 29. Therefore, the resized boxes 33 are only used to determine which preliminary bounding boxes 27 belong to a certain subset, and the resized boxes 33 are not used to generate the final bounding box 29. Instead, the preliminary bounding boxes 27 of the identified subset are still used to generate the corresponding final bounding box 29, as shown below in Figure 7 explained in the context of.

[0088] The scheme described so far for determining the correct final bounding box 29 may be degraded by outliers in the prediction of the preliminary bounding box 27. Fig. 6A and Figure 6B In FIG. 3 , two examples of such outliers are illustrated as outlier boxes 37. Fig. 6A In the example of , the outlier box 37 incorrectly connects two sets of preliminary bounding boxes 27, which actually belong to two different objects represented by the corresponding ground truth 23 (illustrated by dashed lines). Figure 6B In the example of , the outlier box 37 has an abnormal orientation compared to other preliminary bounding boxes 27 associated with the same object.

[0089] Therefore, an additional outlier removal process is applied to the preliminary bounding box 27 as the original prediction result. For outlier removal, the corresponding distribution is modeled for the attributes of the preliminary bounding box, wherein these attributes include the size, orientation and / or speed of the preliminary bounding box 27. That is, based on the values ​​of these attributes, a probability distribution is modeled. Thereafter, probability values ​​can be assigned to the respective attributes of the respective preliminary bounding boxes 27. If the corresponding probability value of at least one attribute of a certain preliminary bounding box 27 is lower than a predetermined threshold, the corresponding preliminary bounding box 27 is considered to be an outlier box 37. The outlier box 37 is excluded from any subset selection and is therefore not considered for determining the final bounding box 29.

[0090] When all clusters or subsets of preliminary bounding boxes 27 are identified, preliminary bounding boxes 27 belonging to the corresponding clusters or subsets are merged, as shown in FIG. Figure 7 To merge a subset of the preliminary bounding box 27, consider the vertices 21 of the preliminary bounding box 27 (see also Figure 1 ). Since the bounding box is represented as a rectangle, each preliminary bounding box 27 includes four vertices 21.

[0091] In order to merge the subsets of preliminary bounding boxes 27, a so-called farthest vertex 39 is identified for each subset. The farthest vertex 39 has the largest distance relative to the centroid of the entire subset of preliminary bounding boxes 27, i.e., a greater distance than the other vertices 21. Figure 7 In the example of , the two furthest vertices 39 are sufficient to identify the vertices of the corresponding final bounding box 29 .

[0092] Furthermore, further properties of the final bounding box 29 may be determined as an average of all preliminary bounding boxes 27 belonging to the subset that have been used to generate the final bounding box 29. For example, the velocity vector of the final bounding box is determined as an average of all velocity vectors 31 determined for the preliminary bounding boxes 27 that have been used to generate the final bounding box 29.

Claims

1. A computer-implemented method for assigning a bounding box to an object (17) in an environment of a vehicle (10), the computer-implemented method comprising the following steps: acquiring data related to an object (17) located in the environment of the vehicle (10) via a sensor (13); Through processing unit (15): determining respective spatial positions and respective sizes of a plurality of preliminary bounding boxes (27) based on the sensor data, such that each preliminary bounding box (27) at least partially covers one of the objects (17), estimating a corresponding velocity (31) of each preliminary bounding box (27) based on the sensor data, selecting a subset of the plurality of preliminary bounding boxes (27) associated with a corresponding one of the objects (17), wherein the selection of the subset is dependent on the corresponding speed (31) of each of the preliminary bounding boxes (27); and A final bounding box (29) is assigned to a corresponding one of the objects (17) by merging the preliminary bounding boxes (27) of the corresponding subsets, wherein: determining the direction of the velocity (31) for each preliminary bounding box (27), The corresponding resized boxes (33) are generated for each preliminary bounding box (27) by the following steps: shrinking the preliminary bounding box (27) in the direction of the velocity (31) and in a lateral direction perpendicular to the direction of the velocity (31), and expanding the preliminary bounding box (27) in the direction of the speed (31), wherein the step of expanding the preliminary bounding box (27) depends on the absolute value of the speed (31) estimated for the preliminary bounding box (27), A preliminary bounding box (27) having overlapping resized boxes (33) is selected for the respective subset.

2. The computer-implemented method of claim 1 , wherein: In order to shrink the preliminary bounding box (27), the length and width of the corresponding preliminary bounding box (27) are reduced by the same factor.

3. The computer-implemented method of claim 1 or 2, wherein: In order to expand the preliminary bounding box (27), the length of the corresponding preliminary bounding box (27) in the direction of the speed (31) is multiplied by the absolute value of the speed (31) and by the expansion factor.

4. The computer-implemented method of claim 1 , wherein: In order to select a subset of the plurality of preliminary bounding boxes (27), a maximum connected subgraph algorithm is applied to all preliminary bounding boxes (27).

5. The computer-implemented method of claim 1 , wherein: If the intersection-over-union ratio of two resized boxes (33) is greater than a predefined threshold, the two resized boxes (33) are considered to be overlapping.

6. The computer-implemented method of claim 1 , wherein: estimating the centroid of said preliminary bounding box (27) for the corresponding subset, for the preliminary bounding box (27) of the subset, determining the farthest vertex (39) having the maximum distance to the centroid, The final bounding box (29) is determined based on the farthest vertex (39).

7. The computer-implemented method of claim 1 , wherein: The velocity of the final bounding box is determined by averaging the estimated velocities (31) over a corresponding subset of the preliminary bounding boxes (27).

8. The computer-implemented method of claim 1 , wherein: determining attributes for each of the preliminary bounding boxes (27), Based on the attributes, it is determined whether to exclude the selected preliminary bounding box from the corresponding subset.

9. The computer-implemented method of claim 8, wherein: assigning a probability distribution to the values ​​of said attributes of said preliminary bounding boxes (27) belonging to the respective subset, If the probability of the value of the attribute assigned to the selected preliminary bounding box is less than a predetermined threshold, the selected preliminary bounding box is excluded from the corresponding subset.

10. The computer-implemented method of claim 1, wherein: A machine learning algorithm is applied to the sensor data to determine the spatial location and size of each preliminary bounding box (27).

11. An apparatus (11) for assigning a bounding box to an object (17) in an environment of a vehicle (10), the apparatus (11) comprising: a sensor (13) configured to acquire data relating to an object (17) located in the environment of the vehicle (10), and A processing unit (15), wherein the processing unit (15) is configured to: determining respective spatial positions and respective sizes of a plurality of preliminary bounding boxes (27) based on the sensor data, such that each preliminary bounding box (27) at least partially covers one of the objects (17), estimating a corresponding velocity (31) of each preliminary bounding box (27) based on the sensor data, selecting a subset of the plurality of preliminary bounding boxes (27) associated with a corresponding one of the objects (17), wherein the selection of the subset is dependent on the corresponding speed (31) of each of the preliminary bounding boxes (27), and A final bounding box (29) is assigned to a corresponding one of the objects (17) by merging the preliminary bounding boxes (27) of the corresponding subsets, wherein: The processing unit (15) is further configured to: determining the direction of the velocity (31) for each preliminary bounding box (27), The corresponding resized boxes (33) are generated for each preliminary bounding box (27) by the following steps: shrinking the preliminary bounding box (27) in the direction of the velocity (31) and in a lateral direction perpendicular to the direction of the velocity (31), and expanding the preliminary bounding box (27) in the direction of the speed (31), wherein the step of expanding the preliminary bounding box (27) depends on the absolute value of the speed (31) estimated for the preliminary bounding box (27), and A preliminary bounding box (27) having overlapping resized boxes (33) is selected for the respective subset.

12. The device (11) according to claim 11, wherein The sensor (13) comprises a radar sensor and / or a lidar sensor.

13. A computer system comprising a processing unit, at least one memory unit and at least one non-transitory data storage unit, wherein: The at least one non-transitory data storage and / or the at least one memory unit comprises a computer program for instructing a computer to perform the computer-implemented method according to any one of claims 1 to 10.

14. A non-transitory computer-readable medium comprising instructions for executing the computer-implemented method according to any one of claims 1 to 10.