Method and control unit for preprocessing sensor data

Abstract

The present invention relates to a method for preprocessing sensor data (100) of a sensor, wherein dv values ​​of the sensor data (100) acquired in an earlier measurement (104) are predicted and stacked with dv values ​​acquired in at least one later measurement (104) to form a stack (102), wherein the dv values ​​comprise a d value and a v value, wherein the d value represents a distance between the sensor and a point (106) and the v value represents a relative speed between the sensor and the point (106), wherein predicted dv values ​​are generated by shearing the d values ​​acquired in the earlier measurement (104) using their v values ​​and a time step (110) between the earlier measurement (104) and the later measurement (104) in order to obtain predicted distances between the points (106) and the sensor at a time of the later measurement (104) to be determined.

Description

Field of the invention

[0001] The invention relates to a method for preprocessing sensor data of a sensor, in particular a radar or lidar sensor, a corresponding control unit, such as can be used in a vehicle or a driver assistance system, and a corresponding computer program product. State of the art

[0002] Radar data can be processed using filters and physical equations to model imaged objects and their movements. This involves estimating the state of an object from measurement time to measurement time. The underlying algorithms require extensive optimization to achieve the best possible results. Disclosure of the invention

[0003] Against this background, the approach presented here presents a method for preprocessing sensor data from a sensor, a corresponding control unit, and a corresponding computer program product according to the independent claims. Advantageous further developments and improvements of the approach presented here emerge from the description and are described in the dependent claims. Advantages of the invention

[0004] In the approach presented here, measurements from a sensor are distorted before further processing in order to combine a sequence of measurements into a stack. It is assumed that a detected object moves at a constant relative velocity to the sensor. Older measurements are distorted more severely than more recent measurements. Due to the distortion, points of the object captured in different measurements are mapped in the stack approximately to where the object is located in the most recent measurement. The object is not imaged or is only slightly blurred due to the distortion of the measurements.

[0005] The approach presented here can result in clusters of points recorded at different times at object positions that clearly distinguish themselves from randomly recorded points. For example, outlines of objects can be identified that would not be visible in individual measurements. These clusters can be easily evaluated in subsequent processing.

[0006] A method for preprocessing sensor data of a sensor is presented, wherein dv values ​​of the sensor data acquired in a previous measurement are predicted and stacked with dv values ​​acquired in at least one later measurement, wherein the dv values ​​have a d value and a v value, wherein the d value represents a distance between the sensor and a point and the v value represents a relative velocity between the sensor and the point, wherein predicted dv values ​​are generated by shearing the d values ​​acquired in the previous measurement using their v values ​​and a time duration between the previous measurement and the later measurement to determine predicted distances between the points and the sensor at a time of the later measurement.

[0007] Ideas for embodiments of the present invention can be considered, among other things, to be based on the thoughts and findings described below.

[0008] A sensor can, in particular, be a radar sensor. The sensor can also be a lidar sensor. The sensor can provide points as measurement data. The points can form a point cloud. Different measured quantities can be mapped to a point. For the approach presented here, a quantity is required that represents a distance between the sensor and the point. This quantity is referred to here as the d-value. Another required quantity represents a relative velocity between the point and the sensor. This quantity is referred to here as the v-value. The relative velocity can be positive or negative, depending on whether the point is approaching or moving away from the sensor.

[0009] The sensor can perform numerous measurements consecutively. Between measurements, a moving object will move away from or closer to the sensor by a distance determined by its relative speed to the sensor. The greater the time interval between measurements, the greater the distance traveled.

[0010] In the approach presented here, the d-values ​​of past measurements are each changed by an individual distance, which is calculated from their v-value and a period of time that has passed since their measurement.

[0011] Since the resulting effect in a graphical representation of the d-values ​​and the v-values ​​leads to a distortion in the direction of the d-values, the effect is referred to here as shear.

[0012] After a predefined number of measurements, the oldest predicted dv values ​​in the stack can be discarded. The older the measurement, the greater the uncertainty of the constant velocity assumption. The stack can therefore be updated continuously. The most recent measurement thus displaces the oldest measurement contained in the stack. For example, five measurements can be stacked on top of each other in the stack.

[0013] The dv values ​​can span a dv space, whereby the dv space is further sheared from measurement to measurement, whereby dv spaces with varying degrees of shear are stacked together. At the time of each measurement, the dv space can have orthogonal axes. Due to the shear, the axes of the dv space can become oblique to each other. The shear can be repeated from measurement to measurement, and the axes or the dv spaces can be further distorted.

[0014] The stack can be made available for object detection. The current stack can be made available. Since the information from previous measurements is updated through shearing, previous measurements can be made available for object detection multiple times. Object detection can use a larger number of points. Object detection can require less computation because all historical measurements of an object are consolidated in close proximity, making it easier and / or more unambiguous to process.

[0015] A cluster of predicted and unpredicted dv values ​​recorded from different measurements can be assigned a common association information if the differences in the dv values ​​within the cluster are smaller than an association tolerance. If dv values ​​are close to each other based on predicting the d values ​​across multiple measurements, the points likely represent the same object. Association information can express that the points likely represent the same object. This association information can simplify object detection and tracking. When using machine learning methods to further process the measurement data stacks, an explicit association step is eliminated. A neural network can solve the association task without any external observation.

[0016] Different association information can be assigned to predicted and unpredicted dv values ​​acquired from different measurements if the differences in the dv values ​​are greater than the association tolerance. If the dv values ​​differ widely from each other due to predicting the d values ​​across multiple measurements, the points likely represent different objects.

[0017] The sensor data can be preprocessed as raw data. At least one of the dv values ​​can be read in as frequency information. Raw data can be presented as a frequency spectrum. The raw data can be stacked to highlight clusters around probable objects. The shearing can create a so-called heatmap.

[0018] The method is preferably computer-implemented and can be implemented, for example, in software or hardware or in a mixed form of software and hardware, for example in a driver assistance system.

[0019] The approach presented here further creates a control unit, wherein the control unit is designed to carry out, control or implement the steps of a variant of the method presented here in corresponding devices.

[0020] The control unit can be an electrical device with at least one computing unit for processing signals or data, at least one memory unit for storing signals or data, and at least one interface and / or a communication interface for reading in or outputting data embedded in a communication protocol. The computing unit can be, for example, a signal processor, a so-called system ASIC, or a microcontroller for processing sensor signals and outputting data signals depending on the sensor signals. The memory unit can be, for example, a flash memory, an EPROM, or a magnetic storage unit. The interface can be designed as a sensor interface for reading in the sensor signals from a sensor and / or as an actuator interface for outputting the data signals and / or control signals to an actuator.The communication interface can be configured to read or output data wirelessly and / or via a wired connection. The interfaces can also be software modules, which are present, for example, on a microcontroller alongside other software modules.

[0021] Also advantageous is a computer program product or computer program with program code that can be stored on a machine-readable carrier or storage medium such as a semiconductor memory, a hard disk memory or an optical memory and is used to carry out, implement and / or control the steps of the method according to one of the embodiments described above, in particular when the program product or program is executed on a computer, in a control unit or a device.

[0022] It should be noted that some of the possible features and advantages of the invention are described herein with reference to different embodiments. A person skilled in the art will recognize that the features of the control device and the method can be combined, adapted, or exchanged as appropriate to achieve further embodiments of the invention. Short description of the drawings

[0023] Embodiments of the invention are described below with reference to the accompanying drawings, wherein neither the drawings nor the description are to be construed as limiting the invention. Fig. 1 shows a representation of preprocessed sensor data according to an embodiment; and Fig. 2 shows a representation of a preprocessing sequence according to an embodiment.

[0024] The figures are merely schematic and not to scale. Like reference numerals denote like or equivalent features. Embodiments of the invention

[0025] Fig. 1 shows a representation of preprocessed sensor data 100 from a sensor. The sensor here is a radar sensor. The sensor data 100 is preprocessed into a stack 102 consisting of multiple measurements 104. Here, a temporal sequence of four measurements 104 from the sensor is stacked into the stack 102. A measurement 104 consists of multiple reflections recorded by the sensor at the same measurement time, which are represented as points 106 in the respective measurement 104. The points 106 each have at least one d-value and one v-value. The d-value represents a distance between the recorded point 106 and the sensor. The v-value represents a relative speed between the point 106 and the sensor. The points 106 can also have other values.

[0026] Here, the points 106 of a measurement 104 are mapped in a two-dimensional dv-space 108. The dv-space 108 is orthogonal to the measurement time and has the distance d on its abscissa and the relative velocity v on its ordinate. The points 106 are arranged in the dv-space 108 according to their d-values ​​and v-values. Other values ​​of the points 106 are not shown here.

[0027] In the approach presented here, the dv space 108 is sheared over time, i.e., distorted, to stack the measurements 104, so that a statement about the distances of the reflections to the sensor can be made even after the measurement time. The further back the measurement time is, the more strongly the dv space 108 is sheared. Due to the shearing, the v-axis of the dv space 108 is no longer perpendicular to the d-axis, and the dv space 108 is distorted in a parallelogram.

[0028] In the stack 102, an unsheared measurement 104 acquired at the current measurement time t0 and three measurements 104 sheared to varying degrees are stacked on top of each other. The oldest sheared measurement 104 is sheared by three time steps 110 between two measurement times, the second oldest sheared measurement 104 is sheared by two time steps 110 between two measurement times, and the most recent sheared measurement 104 is sheared by only one time step 110 between two measurement times.

[0029] Shearing results in a change in the d-values ​​of all points 106 of a measurement 104, while their v-value remains the same. Since points 106 with large v-values ​​are farther from the d-axis than points 106 with small v-values, the d-values ​​of points 106 with large v-values ​​are changed more than the d-values ​​of points 106 with small v-values. The d-values ​​change proportionally to the v-values ​​and according to the time step 110 used.

[0030] Shearing results in a prediction of the relative positions of the reflections to the sensor based on their relative velocities recorded at the time of measurement. Measurements 104 recorded at later measurement times can thus be stacked with sheared older measurements 104 to form stack 102. Stacking results in clusters of points 106 at the positions of objects moving at a constant relative velocity to the sensor.

[0031] The stack 102 is provided for further processing, for example for object recognition.

[0032] In one embodiment, stack 102 is updated with each measurement time. At each measurement time, the current measurement 104 is stacked onto stack 102, and the oldest measurement 104 is removed from stack 102. All measurements 104 already stacked before the current measurement time t0 are sheared, or further sheared, by one time step 110. The d values ​​change with each shearing.

[0033] Fig. Figure 2 shows a diagram of a preprocessing sequence 200 according to an embodiment. The preprocessing 200 essentially corresponds to the preprocessing in Fig.1. The current measurement 104 is fed to preprocessing 200. The measurement 104 is represented here as a point cloud of reflections 202. In preprocessing 200, the points 106 of the reflections 202 are transferred to the dv space 108 and stacked onto the stack 102 of older measurements 104. During stacking, the older measurements 104 are each sheared by one time step, and the oldest measurement 104 is removed.

[0034] The stack 102 is then processed by a detection network 204. The detection network 204 is, in particular, a neural network in which the features contained in the stack 102 are compressed and evaluated to detect objects 206 imaged in the stack 102.

[0035] In one embodiment, clusters 208 of reflections 202 arising in stack 102 are assigned association information 210, since the clusters 208 most likely represent objects 206. A uniquely identifiable piece of association information 210 is added as a value to each of the points 106 of a cluster 208. The association information 210 is thus retained even during subsequent processing steps. For example, paths 212 of objects 206 can be tracked more easily using the association information 210.

[0036] In the following, possible embodiments of the invention are summarized again or presented with slightly different wording.

[0037] A radar object detection and tracking based on a distance-speed diagram is presented.

[0038] Driver assistance functions and automated driving require detailed information about objects in the ego vehicle's environment. These include other road users, obstacles, or the course of the road. Environmental sensors such as radar, video, or lidar scan the environment and provide the necessary measurement data about objects in the vehicle's vicinity. In the classic approach, the measurement data is temporally aggregated or filtered with the help of a tracking and fusion system (e.g., Kalman filter) and supplemented with additional attributes. These include, for example, derived speed, acceleration, or yaw rate. An environmental model is built from all tracked objects, upon which the driver assistance function or automated driving function can be based.

[0039] When it comes to complex technical problems, traditional model-based approaches reach their limits, especially when good physical-mathematical models are unavailable. Modern methods from the field of machine learning can be advantageous here. Prominent examples include speech and image recognition.

[0040] As available computing power continues to increase, solutions based on machine learning methods have become increasingly popular in recent years. These methods use, for example, deep neural networks that are trained with large amounts of data without the need for a physical model to solve problems. The results achieved in this way are often superior in performance to traditional approaches because they can also learn complex relationships that are unknown or difficult to model mathematically. It is expected that these machine learning methods will become more widespread in the future and partially replace traditional analytical solutions. It is becoming apparent that this will also be the case for traditional object tracking and sensor data fusion systems. This is already a technical standard for object detection using video. It is expected to be the case for radar object detection within a few years. Initial results in the literature are promising.

[0041] Radar-based environmental detection currently primarily uses classical methods such as Bayesian filters (e.g., Kalman filters). In these, both the measurement model and the object's motion are modeled using physical equations. When certain boundary conditions are met, such as white Gaussian measurement noise, these filters perform optimally. This means that, mathematically speaking, the filter produces the best possible estimate of an object's state.

[0042] In addition to the Kalman filter, other components are required to implement a tracking system. These include, for example, the association of measurement data, the generation of new object tracks, and the deletion of improbable or no longer relevant object tracks.

[0043] For all sub-algorithms, there are numerous alternatives that vary in complexity and performance. Certain algorithms have become established in recent years, but the number of possible solutions remains very diverse.

[0044] Unfortunately, the mathematical constraints of tracking algorithms are often violated in reality, which is why mathematical optimality cannot be achieved.

[0045] Furthermore, not all object tracking problems can be solved using a mathematically correct approach. Many sub-algorithms, such as generating new object tracks, associating measurement data, or deleting implausible tracks, often have to be solved using heuristics developed through experience.

[0046] Even if these algorithms currently meet the system's requirements, they are usually suboptimal solutions. The selection and optimization of these algorithms is time-consuming. Many parameters must be manually adjusted and tested. Automating this task is difficult because there are no generally accepted evaluation criteria. Furthermore, side effects often manifest themselves in undesirable system behavior that occurs only rarely and is only discovered after lengthy testing. Ensuring error-free interaction between all sub-algorithms at all times is almost impossible. Finding an optimum in such a complex system is difficult and very time-consuming.

[0047] It is expected that some of the problems mentioned can be solved or mitigated using machine learning methods. Since the system behavior is learned directly from the measured data, the development of numerous specialized algorithms is eliminated. Furthermore, no (or only significantly fewer) parameters need to be adjusted manually. The optimum is determined automatically using the training objective and the learning procedure. Thus, a large part of the human development work is taken over by the computer. Only the network architecture suitable for the task must be developed or selected by hand. Furthermore, a large amount of training data must be provided.

[0048] Currently, there is no established network architecture for solving the tracking task with machine learning. In particular, the best method for temporally linking the measurement data from consecutive time steps is unclear. In classical tracking methods, this is achieved, for example, using physical equations (coordinate transformation, Newton's equations of motion). However, this cannot be easily integrated into a neural network.

[0049] For temporal linking, the most complete object state possible may be required to perform the conversion from one time step to the next. In particular, the most accurate position and velocity estimate possible may be required. This is usually only the case at the network output. While internal states can also be temporally linked in a neural network, they have no physical meaning and therefore cannot be processed using known mathematical formulas.

[0050] To provide a complete description of the object's condition, the measurement data must be highly compressed, which typically results in a loss of information. Therefore, it seems sensible to perform the temporal linking as early as possible in the processing chain, when all the information from the measurement data is still available.

[0051] This paper presents a neural network architecture that, in addition to object detection, also performs tracking (tracking over time). The key point is that the temporal coupling from one time step to the next occurs at the input of the network—not within the network or after the output.

[0052] The advantage of this method is that the measurement data is still in a minimally processed form at the input, so little or no information loss can be expected. This provides the network with unfiltered information for object estimation and tracking.

[0053] Temporal preprocessing involves aggregating the measurement data from multiple measurement cycles over time so that related measurement data from an object remain close together. This can greatly simplify interpretation and further processing. Smearing of the measurement data due to object movements is largely avoided or greatly reduced. All measurement data from an object remains in a compact cluster, even across multiple time steps, and does not need to be linked across long distances. This implicitly solves the tracking problem because the measurement data from the same object is linked across multiple time steps. Short-term measurement interruptions can be bridged, and object attributes can be estimated more accurately because the measurement data from multiple time steps can be combined.

[0054] One of the biggest problems in tracking, namely the association of measurement data, is significantly simplified by temporal preprocessing. Only additional logic for generating a time-unique ID is required.

[0055] Convolutional neural networks (CNNs) are based on the principle of searching for patterns in a limited neighborhood at different levels of abstraction. CNNs are considered particularly efficient and powerful for problems that essentially have local relationships (such as images). However, when related data are widely separated (such as human language), alternative network architectures must be used, which are significantly more computationally intensive.

[0056] The radar measurement data is processed in blocks.

[0057] In a first step, the measurement data is plotted in a distance-velocity diagram (dv diagram), referred to below as a feature map. Either the radar locations are plotted according to their distance and velocity, or a 2-dimensional radar spectrum is projected directly onto the feature map. Each point in the diagram then consists of the coordinates d and v in the feature map, as well as other attached attributes, such as azimuth angle, elevation angle, radar cross-section (RCS), and quality measures (e.g., variances).

[0058] In a second block, the oldest diagram is removed if there are already more than n diagrams (sliding window method).

[0059] In a third block, a temporal prediction of the existing, older dv diagrams to the current time is carried out by geometric “shearing” of the diagram.

[0060] In a fourth block, the current diagram and the predicted diagrams of the n previous cycles (e.g., n=5) are stacked on top of each other.

[0061] In a fifth block, the diagram stack is further processed using a detection network.

[0062] In a sixth block, the box estimates are output with attributes such as a position in x and y, optionally z, an object velocity vx and vy, optionally vz, a box extent length | and width b, optionally height h, an orientation φ, an object acceleration ax and ay and / or an object yaw rate ω.

[0063] In a seventh block, ID generation takes place with the assignment of a unique object ID that remains constant over time.

[0064] Preprocessing, or preprocessing with temporal prediction, of the old (not current) feature maps is performed by shearing the dv diagrams. The measurement data from the current time step are entered directly into a dv diagram.

[0065] All dv diagrams of the previous time steps are sheared according to the delta time to the last time step. This corresponds to a geometric prediction of the measured data with a constant velocity. The larger the velocity v, the greater the shift in the distance d: This results in a stack of feature maps, each of which is converted to the current time step by means of the sometimes multiple prediction.

[0066] These stacks of dv diagrams form the input of the neural network.

[0067] A unique object ID that remains constant over time is assigned so that a subsequent function (driver assistance function or automated driving) can assign and evaluate object-specific information over a longer period of time.

[0068] With the proposed method, the output object detections have good accuracy and reliability. If the time between two processing cycles is short (e.g., less than 100 ms), the association of related object detections from one cycle to the next is trivial. Since the velocity of the objects is also estimated, the object can be accurately predicted from one cycle to the next, and the capture window can be chosen accordingly small. The probability of false associations is low, and thus the use of standard association methods is sufficient.

[0069] For example, the algorithm works by predicting all objects from the last cycle to the current time step using a constant-velocity model (Newton's equations of motion). Each time a current object cannot be assigned to an object from the last cycle, the object is assigned a new, unique ID. If a current object can be assigned to an object from the last cycle, its ID is adopted.

[0070] The proposed method can aggregate measurement data from very fast-moving objects very effectively. There is no smearing of the measurement data space. Therefore, the proposed method can be expected to achieve particularly good detection performance even for very fast objects.

[0071] The proposed method uses a constant-velocity model to predict feature maps. This makes it less effective to detect objects that move with high acceleration in the measurement data space, i.e., objects that do not conform to this model. This applies, for example, to all nearby objects that move rapidly tangentially to the sensor (e.g., when passing a pole quickly).

[0072] The method can be used with all radar systems equipped with neural networks for object detection. Furthermore, further processing of the aggregated measurement data using conventional methods (clustering, Kalman filtering, etc.) is also possible. This allows any driver assistance functions up to and including autonomous driving to be implemented or improved.

[0073] Finally, it should be noted that terms such as "comprising," "having," etc., do not exclude other elements or steps, and terms such as "a" or "an" do not exclude a plurality. Reference signs in the claims are not to be considered limiting.

Claims

[1] A method for preprocessing sensor data (100) of a sensor, wherein dv values ​​of the sensor data (100) acquired in an earlier measurement (104) are predicted and stacked with dv values ​​acquired in at least one later measurement (104) to form a stack (102), wherein the dv values ​​comprise a d value and a v value, wherein the d value represents a distance between the sensor and a point (106) and the v value represents a relative speed between the sensor and the point (106), wherein predicted dv values ​​are generated by shearing the d values ​​acquired in the earlier measurement (104) using their v values ​​and a time step (110) between the earlier measurement (104) and the later measurement (104) in order to obtain predicted distances between the points (106) and the sensor at a time of the later measurement (104). to determine. [2] Method according to claim 1, wherein after a predefined number of measurements (104) the oldest predicted dv values ​​of the stack (102) are discarded. [3] Method according to one of the preceding claims, in which the dv values ​​span a dv space (108), wherein the dv space (108) is further sheared from measurement (104) to measurement (104), wherein during stacking, dv spaces (108) sheared to different degrees are stacked to form the stack (102). [4] Method according to one of the preceding claims, wherein the stack (102) is provided for object recognition. [5] Method according to one of the preceding claims, in which a cluster (208) of predicted and unpredicted dv values ​​acquired in different measurements (104) is assigned a common association information (210) if differences in the dv values ​​within the cluster (208) are smaller than a predetermined association tolerance. [6] Method according to claim 5, wherein different association information (210) is assigned to the predicted and unpredicted dv values ​​acquired in different measurements (104) if the differences in the dv values ​​are greater than the predetermined association tolerance. [7] Method according to one of the preceding claims, in which the sensor data (100) are preprocessed as raw data, wherein at least one value of the dv values ​​is read in as frequency information. [8] Control device, wherein the control device is designed to carry out, implement and / or control the method according to one of the preceding claims in corresponding devices. [9] Computer program product which is designed to instruct a processor, when the computer program product is executed, to carry out, implement and / or control the method according to one of claims 1 to 7. [10] A machine-readable storage medium on which the computer program product according to claim 9 is stored.

Images