Tracking objects in the vicinity of a motor vehicle

Abstract

To track objects in the vicinity of a motor vehicle (1), a sequence of data records, each containing a point cloud (5) representing the environment, is received from a lidar sensor system (3). A first tracking instance (6), corresponding to an apparent first object, and a second tracking instance (7), corresponding to an apparent second object, are tracked, with the apparent first object being closer to the motor vehicle (1) than the apparent second object. A set of object properties, including the size of the apparent first object, is determined based on the sequence of data records. A unified tracking instance (8) is created by combining the first tracking instance (6) and the second tracking instance (7), and the unified tracking instance (8) is tracked as long as the apparent first object and the apparent second object belong to the same single object.

Description

[0001] The present invention is directed to a computer-implemented method for tracking objects in the environment of a motor vehicle and to a method for at least partially automatically guiding a motor vehicle, wherein such a method is carried out. The invention is further directed to a data processing device for carrying out such a computer-implemented method, to an electronic vehicle guidance system for a motor vehicle comprising such a data processing device, and to a corresponding computer program product.

[0002] Various driver assistance functions or other functions for at least partially automatically driving a motor vehicle require the detection and tracking of objects in the vicinity of the motor vehicle, in particular dynamic objects such as other motor vehicles, pedestrians, etc.

[0003] For this purpose, known object detection and tracking algorithms can be applied to point cloud data obtained from a lidar sensor system mounted on the vehicle. A disadvantage of such approaches is that, due to the limited resolution of the point cloud data, especially at large distances from the lidar system, it is not always clear whether different point clusters in the point cloud belong to the same object and should be tracked as a single tracking instance or to different independent objects that should be tracked individually. Furthermore, the amount of light reflected from objects in the vehicle's surroundings depends strongly on the angle of incidence. Therefore, the reliability of scan points with a shallow angle of incidence, i.e., when the incident light is relatively far from normal incidence, can be limited.The amount of reflected light also depends on the reflectivity of the reflective surface. Surfaces with relatively low reflectivity can lead to reduced reliability of the detected points or a smaller number of reliable points. Surfaces with varying reflectivity, such as different parts of a vehicle, can also cause similar problems. This can also make it difficult to distinguish between two or more independent objects being tracked and objects that belong together, such as a truck's tractor unit and a truck's trailer.

[0004] One possible way to overcome these disadvantages is to use multi-fusion approaches, using multiple sensors, such as cameras and / or radar systems, in addition to the lidar sensor system to identify the completeness of the detected objects. By calibrating the sensors relative to each other, missing scan points of the lidar sensor systems and resulting gaps between related objects can be compensated. However, this increases the complexity of the system and the data analysis, for example, with regard to temporal synchronization, calibration, and ambiguities between the data from different sensors. Due to this increased complexity, the risk of system failure also increases.Furthermore, it may be necessary, particularly for safety-relevant applications, that the reliability of certain tracking and object detection functions is sufficiently high when only a single sensor system or a single type of sensor system is used.

[0005] The publication J. Fang et al.: “3D Bounding Box Estimation for Autonomous Vehicles by Cascaded Geometric Constraints and Depurated 2D Detections Using 3D Results” (arXiv:1909.01867v1) describes how 3D bounding boxes can be generated for objects in the environment of autonomous vehicles and is also dedicated to lidar-based 3D object detection.

[0006] An object of the present invention is to improve the reliability of distinguishing multiple independent objects from objects that belong together in object tracking based on data from a lidar sensor system, in particular without the use of additional types of sensor systems.

[0007] This object is achieved by the subject matter of the independent claim. Further embodiments and preferred embodiments are the subject matter of the dependent claims.

[0008] The invention is based on the idea of ​​determining the size of an apparent first object that is closer to the lidar sensor system than an apparent second object, and deciding, depending on the size of the apparent first object, whether the apparent first object and the apparent second object should be combined into a single tracking instance. In further embodiments, various additional properties of the apparent first object and / or the apparent second object are used to improve the reliability and robustness of the method.

[0009] According to one aspect of the invention, a computer-implemented method for tracking objects in an environment of a motor vehicle is provided. A sequence of data sets is obtained from a lidar sensor system of the motor vehicle, each data set of the sequence of data sets containing a corresponding point cloud representing an environment of the motor vehicle. A first tracking instance corresponding to an apparent first object and a second tracking instance corresponding to an apparent second object are tracked, in particular detected and tracked, based on the sequence of data sets. A distance of the apparent first object from the motor vehicle, in particular from the lidar sensor system, is less than a distance of the apparent second object from the motor vehicle, in particular from the lidar sensor system, in particular for each data set of the entire sequence of data sets.

[0010] A set of object properties, in particular of the apparent first object and / or the apparent second object and / or a combination of the apparent first object and the apparent second object, is determined based on the sequence of data records. The set of object properties includes a size of the apparent first object. Depending on the set of properties, in particular depending, among other things, on the size of the apparent first object, it is determined whether the apparent first object and the apparent second object belong to a single object.If the apparent first object and the apparent second object belong to the single object, in particular if it has been determined that the apparent first object and the apparent second object belong to the single object, a unified tracking instance is created by unifying the first tracking instance and the second tracking instance, and the unified tracking instance is tracked.

[0011] Unless otherwise stated, all steps of the computer-implemented method can be performed by a data processing device having at least one computing unit, in particular a data processing device of the motor vehicle. In particular, the at least one computing unit is configured or adapted to perform the steps of the computer-implemented method. For this purpose, the at least one computing unit can, for example, store a computer program containing instructions that, when executed by the at least one computing unit, cause the at least one computing unit to perform the computer-implemented method.

[0012] All computing units of the at least one computing unit can be contained in the motor vehicle. However, it is also possible for all computing units of the at least one computing unit to be part of an external computing system external to the motor vehicle, for example, a backend server or a cloud computing system. It is also possible for the at least one computing unit to have at least one vehicle computing unit of the motor vehicle and at least one external computing unit contained in the external computing system. The at least one vehicle computing unit can, for example, be contained in one or more electronic control units (ECUs) and / or one or more zone control units (ZCUs) and / or one or more domain control units (DCUs) of the motor vehicle and / or in the lidar sensor system.

[0013] The sequence of data sets can be understood, for example, as a sequence of frames or a sequence of data sets corresponding to a sequence of frames. Each frame can, for example, correspond to a complete scan of the lidar sensor system if the lidar sensor system is implemented as a laser scanner, or to a predefined part of a complete scan or a single image if the lidar sensor system is implemented as a flash lidar system.

[0014] A known design of lidar sensor systems are so-called laser scanners, in which a laser beam is deflected by means of a light deflection arrangement, so that different deflection angles of the laser beam can be realized. The light deflection arrangement can, for example, contain one or more rotatably mounted mirrors. Alternatively, the light deflection arrangement can include a mirror element with a tiltable and / or pivotable surface. The mirror element can, for example, be configured as a microelectromechanical system (MEMS). The emitted laser beams can be partially reflected in the environment, and the reflected portions can in turn strike the laser scanner, in particular the light deflection arrangement, which can direct them onto a detector unit of the laser scanner, which contains the at least one optical detector.In particular, each optical detector of the detector unit generates an associated detector signal based on the components detected by the respective optical detector. Based on the spatial arrangement of the respective optical detector, together with the current position of the light deflection arrangement, in particular its rotational position or its tilt and / or pivoting position, it is thus possible to infer the direction of incidence of the detected reflected light components. A processing unit or an evaluation unit of the laser scanner can, for example, perform a time-of-flight measurement to determine a radial distance of the reflecting object. Alternatively or additionally, a method according to which a phase difference between the emitted and detected light is evaluated can be used to determine the distance.

[0015] Other designs of lidar systems are flash lidar systems. These are non-scanning systems that do not require the aforementioned light deflection arrangement. The laser light generated by the light source is dispersed by an optical element to spread over a wide angle in a single flash of light.

[0016] The lidar sensor system can also include two or more laser scanners or two or more flash lidars, which can also be mounted at different positions on the vehicle. The point clouds can then be obtained, for example, based on the data generated by the two or more laser scanners or two or more flash lidars.

[0017] A point cloud can be understood as a set of points, where each point is characterized by respective coordinates in a two-dimensional or three-dimensional coordinate system. In the case of a three-dimensional point cloud, the three-dimensional coordinates can be determined, for example, by the direction of incidence of the reflected light components and the corresponding time-of-flight or the measured radial distance for that particular point. In other words, the three-dimensional coordinate system can be a three-dimensional polar coordinate system. However, the information can also be preprocessed to provide three-dimensional Cartesian coordinates for each of the points. In general, the points of a point cloud, unlike, for example, a camera image, can be in a disordered or unsorted manner.In addition to the spatial information, namely the two-dimensional or three-dimensional coordinates, the point cloud can also store additional information or measured values ​​for the individual points, such as an echo pulse width (EPW) of the respective sensor signal.

[0018] The fact that the apparent first object and the apparent second object must belong to the single object for the unified tracking instance to be created and tracked can be understood as a necessary condition, but not necessarily a sufficient condition. In other words, in some implementations, the creation and tracking of the unified tracking instance may depend on further conditions, in particular on further properties of the set of object properties.

[0019] Known algorithms for lidar-based object detection and tracking can be used to detect and track the apparent first object and the apparent second object. As a result of such algorithms, the apparent first object and the apparent second object are treated as separate and independent objects, and therefore, the first tracking instance and the second tracking instance are treated separately.After the above-mentioned steps of the computer-implemented method according to the invention have been carried out, either the combined tracking instance is generated and tracked instead of the first tracking instance and the second tracking instance, or, if it is determined, depending on the set of object properties, that the apparent first object and the apparent second object do not belong to the single object, the first tracking instance and the second tracking instance can continue to be tracked in a conventional manner as independent tracking instances.

[0020] In other words, the apparent first object and the apparent second object are both real objects, but they are originally apparently independent before deciding whether they belong to the single object. Note that if the apparent first object and the apparent second object belong to the single object, they may still correspond to different parts of the single object, but they do not move independently of each other, such as a tractor unit of a truck and a trailer of the truck, or such as a car and a trailer pulled by the car.

[0021] A tracking instance can be understood as a representation of the corresponding object to be tracked within the framework of the tracking algorithm used, which can be, for example, a well-known tracking algorithm based on a Kalman filter or similar. The tracking instance is defined, for example, by corresponding state variables, such as the position of certain predefined points, their velocity, etc. The tracking instance can be characterized, for example, by a bounding box, for example, a rectangular bounding box in a two-dimensional approach, or a three-dimensional bounding box, i.e., a cuboid-shaped bounding box, or a three-dimensional bounding box of a different predefined shape.Alternatively, the tracking instance may be characterized by one or more lines or other geometric figures that approximate the shape of the object, for example, a contour of the object, or a portion thereof. For example, L-shaped or I-shaped lines may be used to approximate a portion of the object's contour and define the tracking instance. In other words, the tracking instance may be understood as a mathematical representation of the corresponding object, determined based on the sequence of data sets.

[0022] The distance of the apparent first object from the motor vehicle and the distance of the apparent second object from the motor vehicle are given in particular by respective distances of the first tracking instance and the second tracking instance from the motor vehicle, in particular in a corresponding reference coordinate system. The distance can be a radial distance from an origin point of the reference coordinate system or can be a distance along a predefined direction, for example, along a longitudinal axis of the motor vehicle. Since the distance of the apparent first object is less than the distance from the apparent second object, the apparent first object is closer to the motor vehicle than the apparent second object.

[0023] Here and below, the size of an object, which is determined based on the sequence of data sets, can be understood as at least one geometric size measurement of the corresponding tracking instance, in particular the bounding box or an alternative geometric representation. In particular, the size can comprise a width and / or height and / or length of the respective object. If the tracking instance is defined by means of a cuboid-shaped three-dimensional bounding box, the length can be understood, for example, as a length of the side of the bounding box along an estimated longitudinal direction or direction of movement of the corresponding object, and a height of the object can be understood as a height along a predefined height axis of the motor vehicle, in particular along a normal direction to the road surface. The width can correspond to the width in the remaining direction.An analogous interpretation for width and length can also be used in the case of a two-dimensional rectangular bounding box or, for example, when an L-shaped geometric figure is used to approximate the object for the tracking instance. The width and / or height and / or length of an object can also be calculated as respective dimensions along the transverse, longitudinal, and elevation axes of the vehicle.

[0024] Because the apparent first object is closer to the motor vehicle than the apparent second object, the reliability of the data from the respective point clouds describing the apparent first object is greater than the data describing the apparent second object, particularly in the case of a laser scanner lidar system, whose angular resolution may be fixed, which can lead to a higher point density closer to the lidar sensor system. Consequently, determining whether the apparent first object and the apparent second object belong to a single object based on the size of the apparent first object is also more reliable.

[0025] Furthermore, the size of the apparent first object can provide a clear indication of the type or class of object to which the apparent first object belongs. For example, if the size is relatively large, the probability that the object is a large vehicle, such as a trailer or a truck, is higher than for smaller objects. On the other hand, the probability that two or more apparent objects belong to a single object is higher for such large vehicles or objects. For example, the probability that a trailer and a tractor vehicle or a tractor unit, etc., are detected as independent apparent objects first may be higher than for a pedestrian or a motorcycle or other small vehicle or object.

[0026] Thus, in some embodiments, the invention uses the size of the apparent first object, along with other object properties, to estimate whether or not the apparent first object and the apparent second object belong to the single object. It is noted that the unified tracking instance may be tracked based on the sequence of data records and / or depending on another sequence of data records following the sequence of data records.

[0027] It is further noted that if the merged tracking instance is generated and tracked, the method steps may be performed again, wherein the single object may be considered a new apparent object, and it may be determined whether the single object should be merged with an apparent third object in a corresponding manner. On the other hand, if it is determined, depending on the set of object properties, that the apparent first object and the apparent second object do not belong to the single object, the analogous steps may be performed for further apparent objects in the environment of the motor vehicle that are detected and tracked based on the sequence of data sets, and so on.

[0028] For applications and application situations that may arise in a computer-implemented method according to the invention and which are not explicitly described here, it may be provided that an error message is output and / or a standard setting and / or a predetermined initial state is set in accordance with the computer-implemented method.

[0029] According to some implementations, the size of the apparent first object includes a width and / or a height of the apparent first object.

[0030] The width of the apparent first object can be a spatial extension along the transverse axis of the motor vehicle or a respective side length of the bounding box of the geometric figure representing the apparent first object. The height of the apparent first object can be a spatial extension along the height axis of the motor vehicle or a respective side length of the bounding box of the geometric figure representing the apparent first object.

[0031] The height and width provide a reliable indicator of the type of apparent first object and thus improve the reliability of the method.

[0032] According to some embodiments, it is determined that the apparent first object is a dynamic object.

[0033] In other words, all objects in the vehicle's surroundings detected based on the sequence of data sets can be classified as either dynamic objects or static objects. Only dynamic objects, for example, are considered as potential candidates for the first object. This reduces the computational effort without losing significantly relevant information.

[0034] According to some implementations, a first object class is determined for the apparent first object depending on a first subset of the set of object properties, wherein the first subset includes the size of the apparent first object. In particular, the first object class is determined depending at least, among other things, on the size of the apparent first object. A union region in a neighborhood of the apparent first object is determined depending on the first object class. The unioned tracking instance is created and tracked only if the apparent second object lies at least partially within the union region.

[0035] In other words, it can be considered a necessary condition for the generation and tracking of the unified tracking instance that the apparent second object lies at least partially within the unification domain.

[0036] The first object class is one of a number of predefined classes. These predefined classes can be, for example, size classes such as "large," "medium," and "small," or they can be more specific, such as "passenger car," "trailer," "truck," "pedestrian," "motorcycle," etc. Known object tracking algorithms can be used to determine the first object class.

[0037] The merging region may extend, according to a predefined geometric shape, from the apparent first object into an area surrounding the apparent first object, for example, at least partially in front of the apparent first object. The merging region corresponds, for example, to a closed two-dimensional region or three-dimensional region defined in the reference coordinate system.

[0038] The merging region, in particular a size and / or shape of the merging region, depends on the first object class, in particular on the size of the apparent first object. In this way, the merging region can be adapted to the first apparent object in such a way that it can be taken into account that for different object types, the probability that the apparent object belongs to the same single object as the apparent first object can be high in different regions around the apparent first object. For example, the merging region may tend to be larger for larger sizes of the apparent first object, although in some embodiments the actual size and / or shape of the merging region also depends on other object properties of the first subset of the set of object properties, for example the distance of the apparent first object from the motor vehicle.

[0039] According to some implementations, a confidence value that the apparent first object and the apparent second object belong to the single object is calculated depending on a second subset of the set of object properties when the apparent second object lies at least partially within the union region. The unioned tracking instance is generated depending on the confidence value.

[0040] In other words, whether the unified trace instance is generated and tracked depends on the confidence value. In particular, the unified trace instance can only be generated and tracked if the confidence value is greater than a predefined minimum confidence.

[0041] For example, the first and second subclauses are distinct from each other, but may overlap in some implementations. In other words, an object property contained in the first subclause may also be contained in the second subclause.

[0042] In particular, if it is determined that the apparent second object is not at least partially within the union region, the confidence value does not need to be calculated for the particular combination of the apparent first object and the apparent second object, which reduces the computational effort of the method.

[0043] According to some embodiments, the second subset includes a speed of the apparent first object and / or a speed of the apparent second object. In other words, the confidence value is calculated depending on the speed of the apparent first object and / or the speed of the apparent second object.

[0044] The speeds can be determined, for example, as speeds relative to the motor vehicle or within the reference coordinate system. In particular, the confidence value can be calculated depending on a difference between the speed of the apparent first object and the speed of the apparent second object. For example, the confidence value can be larger the smaller the difference between the stated speeds.

[0045] This exploits the fact that the probability that the apparent first object and the apparent second object belong to the single object is higher when their speeds are similar. In other words, it exploits the fact that the components of the single object are expected to move at the same or approximately the same speed. The current traffic situation can also influence typical speeds.

[0046] According to some embodiments, the merging region, in particular the size and / or shape of the merging region, is determined depending on the speed of the apparent first object.

[0047] For example, the length of the merging region, which is its spatial extent along the direction of movement of the apparent first object, can be larger for higher speeds of the apparent first object and smaller for lower speeds. In this way, it can be taken into account that independent objects, for example, motor vehicles, tend to be closer together when their speed is lower. The merging region can then be smaller or shorter in this case to avoid the erroneous assumption that at low speeds, for example in a stop-and-go scenario or in heavy traffic, the apparent first object and the apparent second object belong to the same single object.

[0048] According to some embodiments, the second subset includes the size of the apparent first object and / or a size of the apparent second object and / or a size of the single object, in particular a length of the single object.

[0049] The above explanations regarding the size of the apparent first object can be applied accordingly to the size of the apparent second object. The size of the single object corresponds, assuming that the apparent first object and apparent second object belong to the single object, in particular, to a size given by the unified tracking instance.

[0050] For example, the confidence value can be reduced if the size of the single object is larger than a predefined threshold size, especially if the length of the single object is larger than a predefined threshold length. This allows for the fact that the probability of extremely long objects is usually low.

[0051] According to some embodiments, the second subset includes an orientation of the apparent first object and / or an orientation of the apparent second object. In other words, the confidence value is calculated depending on the orientation of the apparent first object and / or the orientation of the apparent second object.

[0052] The orientation of an object can be understood, for example, as the orientation of a longitudinal direction of the respective object or of the corresponding bounding box or geometric figure representing the respective tracking instance, in particular in the reference coordinate system.

[0053] In particular, the confidence value can be calculated depending on an orientation difference between the orientation of the apparent first object and the orientation of the apparent second object. For example, the confidence value can be larger the smaller the orientation difference. This exploits the fact that the probability that the apparent first object and the apparent second object belong to the single object is higher when their orientations are similar. In other words, it exploits the fact that the components of the single object are expected to move with the same or approximately the same orientation.

[0054] According to some embodiments, the second subset includes a direction of movement of the apparent first object and / or a direction of movement of the apparent second object. In other words, the confidence value is calculated depending on the direction of movement of the apparent first object and / or the direction of movement of the apparent second object.

[0055] The direction of movement of an object can be understood, for example, as the direction of movement of a longitudinal axis of the respective object or of the corresponding bounding box or geometric figure representing the respective tracking instance, in particular in the reference coordinate system.

[0056] In particular, the confidence value can be calculated based on a difference between the direction of motion of the apparent first object and the direction of motion of the apparent second object. For example, the confidence value can be higher the smaller the difference between the directions of motion. In this way, it can be exploited that the probability that the apparent first object and the apparent second object belong to the single object is higher when their directions of motion are similar. In other words, it is exploited that the components of the single object are expected to move in the same or approximately the same direction of motion.

[0057] According to some embodiments, the merging region, in particular the size and / or shape and / or an orientation of the merging region, is determined depending on the orientation of the apparent first object and / or depending on the direction of movement of the apparent first object.

[0058] According to some embodiments, the second subset includes the distance of the apparent first object from the motor vehicle. In other words, the confidence value is calculated depending on the distance of the first object from the motor vehicle.

[0059] The closer the apparent first object is to the motor vehicle, the more reliable the corresponding point cloud data of the sequence of data sets can be. Thus, other object properties of the set of object properties, especially the second subset, can also be determined with greater reliability. This can be taken into account when calculating the confidence value, which in turn increases the reliability of the method.

[0060] According to some embodiments, the merging region, in particular the size and / or shape and / or the orientation of the merging region, is determined depending on the distance of the apparent first object from the motor vehicle.

[0061] According to some embodiments, the second subset includes a size of a gap between the apparent first object and the apparent second object. In other words, the confidence value is calculated depending on the size of the gap between the apparent first object and the apparent second object.

[0062] The gap is, for example, a distance between the apparent first object and the apparent second object, in particular between the respective bounding boxes or other geometric figures representing the respective tracking instances. The distance can, for example, be a distance along a longitudinal axis of the apparent first axis or along the longitudinal axis of the motor vehicle.

[0063] For example, the confidence value can be smaller the larger the gap. This takes advantage of the fact that the probability that the apparent first object and the apparent second object belong to the single object is higher when the apparent first object and the apparent second object are closer together.

[0064] According to some implementations, the second subset includes a tracking status of the first tracking instance and / or a tracking status of the second tracking instance.

[0065] The tracking status can, for example, be determined based on the confidence or certainty according to the object tracking algorithm used that the state of the respective tracking instance is correct. For example, the tracking status can indicate how certain the estimate is and / or for how long, in particular how many frames, the respective tracking instance has already been tracked. The tracking status can, for example, include an "initial detected" status and / or an "unstable tracking" status and / or a "stable tracking" status and / or a "predicted" status, and so on, or other suitable categories. The tracking status can also be understood as a validation status of the respective tracking instance. The tracking status can therefore also indicate how reliable the point cloud data of the sequence of datasets is.

[0066] According to some versions, the confidence value is calculated depending on the first object class.

[0067] For example, the confidence value may be higher for certain object types, such as relatively large objects, trucks, trailers, etc., than for other object types, such as relatively small objects, pedestrians, motorcycles, etc.

[0068] According to some implementations, a second object class is determined for the apparent second object based on a third subset of the set of object properties, wherein the third subset includes the size of the apparent second object.

[0069] The explanations regarding the first object class can be applied to the second object class. In particular, the confidence value can be calculated depending on the second object class.

[0070] According to some embodiments, the merging region is determined depending on a position of the apparent first object, in particular in the reference coordinate system.

[0071] According to some implementations, an object tracking algorithm includes a Kalman filter algorithm or an extended Kalman filter algorithm or an unscented Kalman filter algorithm is used to track the federated tracking instance.

[0072] The same object tracking algorithm used to track the merged tracking instance can also be used to track the first tracking instance and / or to track the second tracking instance.

[0073] According to some embodiments, a plurality of apparent objects, including the apparent first object and the apparent second object, are detected depending on the sequence of data sets. The apparent first object is determined as an apparent object of the plurality of apparent objects having a smallest distance from the motor vehicle.

[0074] In other words, the apparent first object can be determined as the dynamic object closest to the motor vehicle. The closer the apparent first object is to the motor vehicle, the more reliable the corresponding point cloud data of the sequence of data sets can be.

[0075] According to a further aspect of the invention, a method for at least partially automatically driving a motor vehicle is provided. A computer-implemented method according to one of the preceding claims is carried out by the at least one computing unit. At least one control signal for at least partially automatically driving the motor vehicle is generated by the at least one computing unit as a function of the tracked unified tracking instance, and / or assistance information for assisting a driver of the motor vehicle when driving the motor vehicle is generated by the at least one computing unit as a function of the tracked unified tracking instance.

[0076] The at least one control signal can be provided, for example, to one or more actuators of the motor vehicle, including, for example, one or more brake actuators and / or one or more steering actuators and / or one or more drive motors of the motor vehicle. The one or more actuators can influence a longitudinal and / or lateral control of the motor vehicle in order to guide the motor vehicle at least partially automatically.

[0077] The assistance information can be output by means of an output device of the motor vehicle, for example a display and / or an audio output system and / or a haptic output system.

[0078] According to a further aspect of the invention, a data processing device is provided which has at least one computing unit. The at least one computing unit is adapted to carry out a computer-implemented method according to the invention.

[0079] A computing unit can be understood, in particular, as a data processing device that contains data processing circuits. A computing unit can thus perform computing operations for processing data. This can also include indexed access to a data structure, for example, a look-up table (LUT).

[0080] In particular, the computing unit may contain one or more computers, one or more microcontrollers, and / or one or more integrated circuits, for example one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), and / or one or more single-chip systems (SoCs). The computing unit may also contain one or more processors, for example one or more microprocessors, one or more central processing units (CPUs), one or more graphics processing units (GPUs), and / or one or more signal processors, in particular one or more digital signal processors (DSPs). The computing unit may also include a physical or virtual network of computers or other of the aforementioned units.

[0081] In various embodiments, the computing unit includes one or more hardware and / or software interfaces and / or one or more memory units.

[0082] A memory unit can be a volatile data memory, for example a dynamic random access memory (DRAM) or a static random access memory (SRAM), or a non-volatile data memory, for example a read-only memory (ROM), a programmable read-only memory (PROM), an erasable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), a flash memory or flash EEPROM, a ferroelectric random access memory (FRAM), a magnetoresistive random access memory (MRAM), or a phase-change random access memory.PCRAM (phase-change random access memory) is implemented.

[0083] According to a further aspect of the invention, an electronic vehicle guidance system for a motor vehicle is provided, which comprises a data processing device according to claim 1. The at least one computing unit is configured to generate at least one control signal for at least partially automatically guiding the motor vehicle depending on the tracked unified tracking instance and / or the at least one computing unit is configured to generate assistance information for assisting a driver of a motor vehicle when driving the motor vehicle depending on the tracked unified tracking instance.

[0084] In other words, the electronic vehicle guidance system can carry out a method according to the invention for at least partially automatically guiding a motor vehicle.

[0085] An electronic vehicle guidance system can be understood as an electronic system designed to guide a vehicle fully automatically or fully autonomously, in particular without requiring manual intervention or control by a driver or user of the vehicle. The vehicle automatically performs all required functions, such as steering maneuvers, braking maneuvers, and / or acceleration maneuvers, as well as monitoring and recording road traffic and corresponding reactions. In particular, the electronic vehicle guidance system can implement a fully automatic or fully autonomous driving mode according to Level 5 of the SAE J3016 classification. An electronic vehicle guidance system can also be implemented as a driver assistance system (ADAS), which assists a driver in partially automatic or partially autonomous driving.In particular, the electronic vehicle guidance system can implement a semi-automatic or semi-autonomous driving mode according to levels 1 to 4 of the SAE J3016 classification. Here and below, SAE J3016 refers to the corresponding standard dated April 2021.

[0086] The at least partially automatic driving of the vehicle can therefore include driving the vehicle according to a fully automatic or fully autonomous driving mode according to Level 5 of the SAE J3016 classification. The at least partially automatic driving of the vehicle can also include driving the vehicle according to a partially automatic or partially autonomous driving mode according to Levels 1 to 4 of the SAE J3016 classification.

[0087] According to some embodiments, the electronic vehicle guidance system includes the lidar sensor system.

[0088] Further embodiments of the electronic vehicle guidance system according to the invention result directly from the various embodiments of the method according to the invention for at least partially automatically driving a motor vehicle, and vice versa. In particular, individual features and corresponding explanations as well as advantages relating to the various embodiments of the method can be transferred accordingly to corresponding embodiments of the electronic vehicle guidance system according to the invention.

[0089] According to a further aspect of the invention, a computer program containing instructions is provided. When the instructions are executed by a data processing device, in particular by a data processing device according to the invention, the instructions cause the at least one computing unit to execute a computer-implemented method according to the invention.

[0090] The instructions can be provided, for example, as program code. The program code can be provided, for example, as binary code or assembly code and / or as source code of a programming language, for example, C, and / or as a program script, for example, Python.

[0091] According to a further aspect of the invention, a further computer program containing further instructions is provided. When the further instructions are executed by an electronic vehicle guidance system according to the invention, in particular by the data processing device of the electronic vehicle guidance system, the further instructions cause the electronic vehicle guidance system to execute a method according to the invention for at least partially automatically guiding a motor vehicle.

[0092] The additional instructions can be provided, for example, as program code. The program code can be provided, for example, as binary code or assembly code and / or as source code of a programming language, for example, C, and / or as a program script, for example, Python.

[0093] According to a further aspect of the invention, a computer-readable storage medium is provided which stores a computer program according to the invention and / or another computer program.

[0094] The computer program, the further computer program and the computer-readable storage medium are respective computer program products with the instructions and / or the further instructions.

[0095] Further features of the invention emerge from the claims, the figures and the description of the figures. The features and combinations of features mentioned above in the description as well as the features and combinations of features mentioned below in the description of the figures and / or shown in the figures can be encompassed by the invention not only in the respectively specified combination, but also in other combinations. In particular, embodiments and combinations of features can also be encompassed by the invention that do not have all of the features of an originally formulated claim. Furthermore, embodiments and combinations of features can be encompassed by the invention that go beyond or deviate from the combinations of features set out in the backreferences of the claims.

[0096] The invention is explained in detail below using specific exemplary embodiments and corresponding schematic drawings. In the drawings, identical or functionally equivalent elements may be designated by the same reference numerals. The description of identical or functionally equivalent elements is not necessarily repeated with reference to different figures.

[0097] The figures show Fig. 1 schematically shows a motor vehicle with an exemplary embodiment of an electronic vehicle guidance system according to the invention; Fig. 2 a schematic representation of a point cloud and tracking instances according to an exemplary embodiment of a computer-implemented method according to the invention; Fig. 3 shows a schematic representation of tracking instances and a merging region according to a further exemplary embodiment of a computer-implemented method according to the invention; and Fig. 4 a schematic representation of a point cloud and tracking instances according to another exemplary embodiment of a computer-implemented method according to the invention.

[0098] In Fig. 1, a motor vehicle 1 with an exemplary embodiment of an electronic vehicle guidance system 2 according to the invention is shown schematically. The motor vehicle 1, for example the electronic vehicle guidance system 2, has a lidar sensor system, and the electronic vehicle guidance system 2 has a data processing device with at least one computing unit 4. The at least one computing unit 4 is adapted to carry out a computer-implemented method according to the invention for tracking objects in an environment of the motor vehicle 1. Different aspects of exemplary embodiments of such a computer-implemented method are shown schematically in Fig. 2 to Fig. 4 shown.

[0099] In the course of the computer-implemented method or as a result of the computer-implemented method, the at least one computing unit 4 may track different tracking instances, in some cases including a unified tracking instance 8.

[0100] The at least one computing unit 4 can generate at least one control signal for at least partially automatically driving the motor vehicle 1 depending on the tracked tracking instances, in particular the tracked combined tracking instance 8. Alternatively or additionally, the at least one computing unit 4 can generate assistance information for assisting a driver of the motor vehicle 1 in driving the motor vehicle depending on the tracked tracking instances, in particular the tracked combined tracking instance 8.

[0101] The lidar sensor system 3 is configured to generate a sequence of data sets, each data set containing a respective point cloud 5 representing the surroundings of the motor vehicle 1. In Fig. 2 to Fig. 4, the point cloud 5 is shown in a two-dimensional bird's-eye view, i.e., in a projection plane perpendicular to a height axis of the motor vehicle 1. However, the point cloud 5 can also be a 3D point cloud or a so-called 2.5D point cloud.

[0102] The at least one computing unit 4 receives the sequence of data sets from the lidar sensor system 3 and detects an apparent first object and an apparent second object based on the sequence of data sets. Based on the sequence of data sets, the at least one computing unit 4 tracks a first tracking instance 6 corresponding to the apparent first object and a second tracking instance 7 corresponding to the apparent second object. A distance of the apparent first object from the motor vehicle 1 is smaller than a distance of the apparent second object from the motor vehicle 1.

[0103] The first tracking instance 6 and the second tracking instance 7 are defined as rectangular bounding boxes in Fig. 2 to Fig. 4. However, the first tracking instance 6 and the second tracking instance 7 can also be given by three-dimensional cuboids as bounding boxes or by L-shaped or I-shaped figures, etc.

[0104] As in Fig. 2 to Fig. 4, there may be a gap 9 between the apparent first object and the apparent second object. The first tracking instance 6 and the second tracking instance 7 may therefore initially be treated as individual objects and tracked independently of each other. However, it is possible that the apparent first object and the apparent second object belong to a single object, such as a truck. In the example of a truck, the apparent first object may, for example, be a trailer pulled by a tractor unit that corresponds to the apparent second object. However, the gap 9 is not necessarily required. For example, the sides of a long vehicle, such as a trailer of a truck, may result in unreliable measurements due to a shallow angle of incidence of the light emitted by the lidar sensor system 3.On the other hand, a side mirror or a tractor unit, as well as the rear end of the truck, provide more reliable measurement points of the point cloud 5. Even in such a situation, the various components of the truck can be detected as individual apparent objects. Fig. Figure 4 shows an example in which there are three apparent objects and respective tracking instances 6, 7, 11, which may correspond, for example, to a rear end, a side area and, for example, a side mirror of a truck.

[0105] The at least one computing unit 4 determines a set of object properties of the apparent first object and / or the apparent second object, which can also be understood as semantic properties, wherein the set of object properties includes a size of the apparent first object, for example, a width and / or height of the apparent first object, based on the sequence of data records. The set of object properties can also include various other properties, such as, for example, the size of the gap 9, a type of the apparent first object and / or a type of the apparent second object, a length of the apparent first object, a size of the apparent second object, for example, a width and / or height and / or length of the apparent second object, a total length of the combination of the apparent first object and the apparent second object,an orientation of the apparent first object and / or an orientation of the apparent second object, a direction of movement of the apparent first object and / or a direction of movement of the apparent second object, a speed of the apparent first object and / or a speed of the apparent second object, a tracking validation state of the first tracking instance 6 and / or a tracking validation state of the second tracking instance 7, a distance of the apparent first object from the lidar sensor system, and so on.

[0106] The at least one computing unit 4 determines, depending on the set of object properties, whether the apparent first object and the apparent second object belong to a single object. If the apparent first object and the apparent second object belong to the single object, the at least one computing unit 4 generates a unified tracking instance 8 by unifying the first tracking instance 6 and the second tracking instance 7 and tracks the unified tracking instance 8 instead of the first tracking instance 6 and the second tracking instance 7.

[0107] For example, the at least one computing unit 4 can determine a first object class for the apparent first object depending on a first subset of the set of object properties, wherein the first subset includes the size of the apparent first object, in particular the width and height of the apparent first object. In some embodiments, the first subset also includes the speed of the apparent first object, the position of the apparent first object relative to the motor vehicle 1, in particular the distance of the apparent first object from the motor vehicle 1, and the orientation of the apparent first object.

[0108] The at least one computing unit 4 determines a union area 10 in the environment of the apparent first object depending on the first object class. The combined tracking instance 8 is only generated and tracked if the apparent second object lies at least partially within the union area 10. As shown in Fig.As shown in Figure 3, the merging region 10 can, for example, have a strip-like shape or the shape of a truncated cone, particularly in the projection plane, extending from the apparent first object along a length direction of the apparent first object to a defined maximum length. The shape, size, and / or orientation of the merging region 10 can, for example, be determined based on the first object class. Further properties of the set of object properties relating to the apparent first object can also be used to define the shape, size, and / or orientation of the merging region 10.

[0109] In some embodiments, a confidence value that the apparent first object and the apparent second object belong to the single object is calculated depending on a second subset of the set of object properties, and the unified tracking instance 8 is generated depending on the confidence value if the apparent second object lies at least partially within the union region 10. In particular, the unified tracking instance 8 is generated and tracked only if the confidence value is greater than a predefined minimum confidence.The second subset may include, for example, the speed of the apparent first object and / or the size of the apparent second object and / or a size of the single object after potential merging, the orientation of the apparent first object and / or the orientation of the apparent second object, the direction of movement of the apparent second object and / or the direction of movement of the apparent second object, the distance of the apparent first object from the motor vehicle 1, the size of the gap 9, a tracking status of the first tracking instance 6 and / or a tracking status of the second tracking instance 7.

[0110] As described particularly with reference to the figures, the invention improves the reliability of distinguishing multiple independent objects from objects that belong together in object tracking based on data from a lidar sensor system, without using additional types of sensor systems.

[0111] In some implementations, a method for evaluating and / or processing point clouds is used to overcome missing scan data due to physical or hardware limitations of the lidar sensor system. Dynamic and static objects are tracked, and the representations of dynamic objects can be merged into a single representation, for example, with a single bounding box, based on semantic properties and environmental analysis.

[0112] In some implementations, different object properties can be combined and evaluated to decide whether or not to merge apparent objects. It should be understood that fixed thresholds for individual object properties may not necessarily be feasible, since objects, such as vehicles, can have different sizes and shapes, and their speed and distance between them can also vary. While motorcycles are relatively small, cars are wider and slightly longer, and if towing a trailer, their length can be similar to that of a small truck. Additionally, vehicles can be larger at higher speeds, while in traffic congestion scenarios, vehicles travel very closely behind one another.

[0113] The risk that different components of the same object will be tracked individually increases, for example, the further away the object is, the larger its size, and the flatter its sides. For vehicles with trailers, the risk that a connection between the vehicle and the trailer is obscured also increases. In contrast, in a traffic jam scenario, two motor vehicles of the same length might appear, such as a large truck or a motor vehicle pulling a trailer. While the first situation should result in one object being tracked, in the second situation two individual objects should be tracked. This challenge can be at least partially overcome by the invention. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited non-patent literature

[0000] J. Fang et al.: “3D Bounding Box Estimation for Autonomous Vehicles by Cascaded Geometric Constraints and Depurated 2D Detections Using 3D Results” (arXiv:1909.01867v1

[0005]

Claims

[1] Computer-implemented method for tracking objects in an environment of a motor vehicle (1), wherein - a sequence of data sets is obtained from a lidar sensor system (3) of the motor vehicle (1), each data set containing a respective point cloud (5) representing the surroundings of the motor vehicle (1); - a first tracking instance (6) corresponding to an apparent first object and a second tracking instance (7) corresponding to an apparent second object are tracked based on the sequence of data sets, wherein a distance of the apparent first object from the motor vehicle (1) is less than a distance of the apparent second object from the motor vehicle (1); - a set of object properties including a size of the apparent first object is determined based on the sequence of data records; - depending on the set of object properties, it is determined whether the apparent first object and the apparent second object belong to a single object; and - a combined tracking instance (8) is generated by combining the first tracking instance (6) and the second tracking instance (7), and the combined tracking instance (8) is tracked if the apparent first object and the apparent second object belong to the single object. [2] A computer-implemented method according to claim 1, wherein - a first object class is determined for the apparent first object depending on a first subset of the set of object properties, wherein the first subset includes the size of the apparent first object; - a union area (10) in a surrounding of the apparent first object is determined depending on the first object class; and - the combined tracking instance (8) is only generated and tracked if the apparent second object lies at least partially within the combined area (10). [3] A computer-implemented method according to claim 2, wherein, if the apparent second object lies at least partially within the merging region (10), a confidence value that the apparent first object and the apparent second object belong to the single object is calculated depending on a second subset of the set of object properties, and the merged tracking instance (8) is generated depending on the confidence value. [4] The computer-implemented method of claim 3, wherein the second subset includes a speed of the apparent first object and / or a speed of the apparent second object. [5] A computer-implemented method according to any one of claims 3 or 4, wherein the second subset includes the size of the apparent first object and / or a size of the apparent second object and / or a size of the single object. [6] Computer-implemented method according to one of claims 3 to 5, wherein the second subset - includes an orientation of the apparent first object and / or an orientation of the apparent second object; and / or - a direction of movement of the apparent second object and / or a direction of movement of the apparent second object. [7] Computer-implemented method according to one of claims 3 to 6, wherein the second subset - the distance of the apparent first object from the motor vehicle (1); and / or - a size of a gap (9) between the apparent first object and the apparent second object. [8] Computer-implemented method according to one of claims 3 to 7, wherein the second subset includes a tracking status of the first tracking instance (6) and / or a tracking status of the second tracking instance (7). [9] A computer-implemented method according to any one of claims 3 to 8, wherein the confidence value is calculated depending on the first object class and / or a second object class for the apparent second object determined based on a third subset of the set of object properties, the third subset including a size of the apparent second object. [10] Computer-implemented method according to one of claims 2 to 9, wherein the merging region (10) is determined depending on a speed of the apparent first object and / or a position of the apparent first object. [11] A computer-implemented method according to any preceding claim, wherein the apparent first object is determined to be a dynamic object. [12] A computer-implemented method according to any one of the preceding claims, wherein an object tracking algorithm including a Kalman filter algorithm or an extended Kalman filter algorithm or an unscented Kalman filter algorithm is used to track the unified tracking instance (8). [13] Computer-implemented method according to one of the preceding claims, wherein - a plurality of apparent objects including the apparent first object and the apparent second object are detected depending on the sequence of data sets; and - the apparent first object is determined as an apparent object of the plurality of apparent objects which has a smallest distance from the motor vehicle (1). [14] Method for at least partially automatically driving a motor vehicle (1), wherein a computer-implemented method according to one of the preceding claims is carried out by at least one computing unit (4) and - at least one control signal for at least partially automatically driving the motor vehicle (1) is generated by the at least one computing unit (4) depending on the tracked combined tracking instance (8); and / or - Assistance information for assisting a driver of the motor vehicle (1) when driving the motor vehicle (1) is generated by the at least one computing unit (4) depending on the tracked combined tracking instance (8). [15] Data processing device comprising at least one computing unit (4) adapted to carry out a computer-implemented method according to one of claims 1 to 13. [16] Electronic vehicle guidance system (2) for a motor vehicle (1), which has a data processing device according to claim 15, wherein the at least one computing unit (4) is designed to - to generate at least one control signal for at least partially automatically driving the motor vehicle (1) depending on the tracked combined tracking entity (8); and / or - to generate assistance information for assisting a driver of the motor vehicle (1) when driving the motor vehicle (1) depending on the tracked combined tracking instance (8). [17] Electronic vehicle guidance system (2) according to claim 16, which comprises the lidar sensor system (3) for the motor vehicle (1). [18] A computer program product comprising instructions which, when executed by a data processing device, cause the data processing device to perform a computer-implemented method according to any one of claims 1 to 13 or a method according to claim 14.

Images