Radar data fusion assessment for classification of objects

Abstract

Radar data fusion evaluation for classifying objects. Method (100) for classifying an object (4) from measurement data (2) recorded with at least one radar sensor (1), with the steps: • providing a spectrum (5) of time-dependent measurement data (2) of the radar sensor (1) (110); • determining positions (6) from the spectrum (5) (120), from which positions radar radiation reflected reaches the radar sensor (1); • determining at least one group (7) of such positions (6*) belonging to one and the same object (4) (130); • determining for each position (6*) in the group (7) a component (5*) of the spectrum (5) (140), which corresponds to the radar radiation reflected by the position (6*); • aggregating (150) all these components (5*) for the object (4) and delivering to a classifier (8) (160); • assigning the object (4) with the classifier (8) to one or more classes (3a-3c) of a predefined classification (3) (170).

Description

TECHNICAL FIELD

[0001] The present application relates to radar data evaluation for classifying objects, which radar data indicate the presence of the objects. BACKGROUND

[0002] For vehicles to be able to move at least partially automatically in road traffic, it is necessary to detect the environment of the vehicle and, if a collision with an object in the environment of the vehicle is to occur, to introduce countermeasures. Creating an environment representation and localization is also necessary for safe autonomous driving.

[0003] With the aid of radar, objects are detected independently of the lighting conditions and, for example, even at large distances at night without dazzling oncoming traffic by means of high beams. Furthermore, the distance and the speed of the objects are directly known from the radar data. These information are important for evaluating whether a collision with an object is possible. However, it is not possible to directly recognize from the radar signals which type of object is involved. This recognition is currently solved by calculating properties from the digital signal processing.

[0004] From DE 10 2018 222 195 A1, a method for locating or classifying objects on the basis of radar data is known. The method uses a convolutional neural network in order to classify the detected objects from a frequency representation of the radar data. SUMMARY

[0005] Within the scope of the present application, a method for classifying objects from measurement data recorded with at least one radar sensor is developed.

[0006] In the case of the method, a frequency spectrum of the time-dependent measurement data of the radar sensor is first provided. The frequency spectrum can be determined from the time-dependent measurement data, for example, by means of a Fourier transform. However, the radar sensor used can have outputted the frequency spectrum, for example.

[0007] From the frequency spectrum, positions are determined from which radar radiation reflected by the positions reaches the radar sensor. For this purpose, for example, a detector with a constant false alarm rate (CFAR) can be used, and the angle of incidence of the reflected radar radiation can be estimated for the events outputted by the detector, respectively.

[0008] At least one group of such positions belonging to the same object is determined. This can be achieved, in particular, by clustering under the assumption of at least one specific object, for example.

[0009] For each position in the set, a component of the spectrum is determined, which corresponds to the radar radiation reflected from this position. All these components for the object are aggregated and fed to the classifier. With the classifier, the object is assigned to one or more classes of a predefined classification. This means that, inter alia, the type of an object such as a tree, a house, a pedestrian, a passenger car (PKW), a lorry (LKW) or a lane boundary can be determined.

[0010] It has been recognized that in this way the classifier can be designed with significantly lower resource requirements and requires much less computing time. The reason for this is that not the entire spectrum is fed to the classifier at once, but only a small part of the spectrum. Inter alia, it is possible to classify a scene in which several objects are located gradually. For this purpose, the position groups with radar reflections, which indicate the presence of different objects, are processed in turn. Thereby, for each of these groups, a class assignment of an object is obtained, wherein the group indicates the presence of the object.

[0011] The classifier can also be trained correspondingly more simply, since the classifier only has to learn to recognize individual objects, but not interactions that can occur when trying to recognize several objects at the same time.

[0012] In a particularly advantageous configuration, further positions are selected from the positions from which radar radiation reflected according to the set reaches the radar sensor. The selection depends on how reasonable it is that the object characterized by the set also reflects radar radiation from the respective additional position. Components of the spectrum are also determined for these additional positions and aggregated together. The purpose of the division of the total set of positions into clusters of groups is to find reasonable dividing lines between the groups on average. This leads to the fact that in individual cases a position whose assignment to an object is still meaningful is no longer assigned to this object when clustering. By selecting additional positions taking into account the clustering, this effect is at least partially compensated. The selection of too many positions and thus the possible aggregation of unnecessary components of the spectrum is no longer critical for the result of the classification.

[0013] In another particularly advantageous configuration, the positions at which components of the spectrum are concentrated and / or other characteristics of the radar reflections received from these positions are delivered to the classifier together with the components of the spectrum. The other characteristics can for example include the effective cross section (radar cross section, RCS) and / or at least one angle (azimuth angle and / or elevation angle relative to the measurement position). The spatial information or information from the other characteristics and the information from the spectrum can then particularly well complement one another so that the classifier can resolve the respective ambiguities of the two types of information and identify the object as unambiguously as possible and resolve contradictions as far as possible. Thus, for example, the spectrum also provides information about whether radar radiation from a particular position is more likely to be reflected at an edge or more likely to be reflected at a face. Furthermore, the spectrum also depends on whether radar radiation is more likely to be reflected at a soft material or more likely to be reflected at a hard material. Some traffic-related objects, for example bicycles with a rider wearing a safety helmet, can also be a composite of soft and hard materials.

[0014] The spectrum is advantageously provided as a range-velocity representation. This is particularly important for evaluation in terms of traffic-related objects. As soon as the radar sensor does not provide the spectrum directly as a range-velocity representation, the range-velocity representation can be transformed into these coordinates. In particular, a representation in the form of the coordinates range and velocity can constitute a common denominator for the measurement data recorded with different sensors. The classifier then only has to be trained in the coordinates range and velocity in view of the processing of the spectrum.

[0015] In one particularly advantageous configuration, measurement data recorded with modulated type joint sampling frequency modulated continuous wave (JSFMCW) radar radiation is selected. In consideration of any additional information, ambiguities with respect to the velocity can be resolved. This has the advantage compared to modulated type frequency modulated continuous wave (FMCW) that less hardware-intensive hardware is required, which generates "chirps" with a frequency that increases or decreases linearly during transmission. The measurement data recorded with JSFMCW is ambiguous in terms of velocity for this purpose. However, by resolving this ambiguity, the advantage of lower hardware intensity can be achieved and a range-velocity representation of the measurement data that can be used for classification can still be obtained.

[0016] In another particularly advantageous configuration, the at least one component of the spectrum that is coherently aggregated is a segment of the spectrum that is rectangular in the coordinates of the spectrum. This segment can in particular be centered on coordinates that belong to a reflection position at the object. As previously set out, analysis can be performed in such a limited area according to variations in the radar reflections, at which the radar radiation is specifically reflected at this position. The segments ("patches") selected for all positions that belong to the object to be classified then together form a spectrum with significantly reduced data compared to the original spectrum, which only has components relevant to the analysis of just this object. This reduced spectrum can then be pre-processed for the classifier in any way, for example by truncating to a certain number of bins in each coordinate or by transforming the units of measure and / or the dynamic range.

[0017] In another particularly advantageous configuration, the aggregated components of the spectrum are transformed into at least one range-velocity representation before being fed to the classifier. Optionally, the ambiguity with respect to the velocity in the at least one range-velocity representation can then also be resolved taking into account additional information. Transforming only the aggregated components and not the entire spectrum saves computation time. Furthermore, the representation of the aggregated components in range-velocity coordinates can be simpler and / or more easily understandable for further processing.

[0018] The joint processing of the positions at which the particular object reflected the radar radiation and for which the components of the spectrum were therefore aggregated and / or on the one hand other features of the radar reflections and on the other hand the components of the frequency in the classifier can be implemented in different ways. Three exemplary possibilities are explained below.

[0019] In one particularly advantageous configuration, the components of the spectrum are fed to a first neural network in the classifier. The positions or other features of the radar reflections are fed to a second neural network in the classifier. The first neural network and the second neural network are fused into a common neural network in the classifier.

[0020] The parts of the two neural networks that have not yet been fused with one another can then, for example, identify specific features and patterns in the distribution of the positions and / or other features or in the spectrum. In the previously mentioned example of a cyclist as an object, for example, the first neural network can identify that the object is a complex composed of a soft component (the cyclist) and a hard component (the bicycle and the safety helmet). The second neural network can identify the contours of the bicycle. The common fused part of the networks can then, for example, resolve ambiguities and contradictions.

[0021] In particular, the not yet fused parts can, for example, place the recognized features and patterns on a common level of abstraction, so that the features and patterns can be weighed against each other and cancel each other out in the fused part of the network. It is no longer significant in the fused part, therefore, that the recognized features and patterns were obtained from strongly different data types.

[0022] By the two networks being fused with each other only later, the not yet fused parts can, in particular, for example, have different architectures, which are set up for processing different data types.

[0023] For example, in order to process components of a spectrum, a convolutional neural network can be selected as the first neural network. Such a network is particularly suitable for reducing high-dimensional data in its dimensions step by step and mapping it to a very low-dimensional result, for example an assignment to one or more classes, wherein the spectrum is the high-dimensional data.

[0024] In order to process positions from which objects reflected radar radiation or other features of radar reflections, for example, a neural network with a PointNet architecture can be selected.

[0025] In another advantageous configuration, components of a spectrum on the one hand and positions or other features of radar reflections on the other hand are fed to the classifier at the same time in different channels from the same input. If, for example, the classifier is configured for processing image data, an input image for the classifier can consist of a first color channel with components of a spectrum and a second color channel with positions or other features of radar reflections to which the components relate. In the same way, an input for the classifier that can be divided into multiple channels can also consist of components of a spectrum on the one hand and positions or other features of radar reflections on the other hand, different from an image.

[0026] In another advantageous configuration, components of a spectrum are fed to a first layer of a neural network in the classifier. Positions or other features of radar reflections to which the components are aggregated are fed to a deeper layer of the same neural network. In this way, for example, high-dimensional components of a spectrum can be placed on a level of abstraction, wherein position coordinates of reflection positions or other features of radar reflections are already at this level of abstraction from the outset. Both data types can then be aggregated in the neural network.

[0027] The assignment to one or more classes determined by the classifier can, in particular, for example, constitute a maneuvering signal and can be used to maneuver the vehicle with this maneuvering signal. Thereby, the following objectives can be pursued, in particular, for example: adapting the trajectory of the vehicle so that it does not intersect the trajectory of an object recognized in the traffic situation.

[0028] The method can be implemented in particular completely or partially by computer. The application therefore also relates to a computer program having machine-readable instructions which, when executed on one or more computers, cause the one or more computers to carry out the described method. In this sense, a control device for a vehicle and an embedded system for a technical device which are likewise able to execute machine-readable instructions can also be regarded as computers.

[0029] The application likewise relates to a machine-readable data carrier and / or to a download product having a computer program. A download product is a digital product which can be transmitted via a data network, i.e. can be downloaded by a user of a data network, which can be sold for immediate download, for example in an online shop.

[0030] Furthermore, a computer can be equipped with a computer program, a machine-readable data carrier or a download product. BRIEF DESCRIPTION OF DRAWINGS

[0031] Further measures for improving the application are shown in more detail below together with the description of preferred embodiments of the application.

[0032] Figure 1 An embodiment of a method 100 for classifying objects 4 is shown.

[0033] Figure 2 A diagram showing the association of locations 6* belonging to the same object 4 with components 5* of the spectrum 5. DETAILED DESCRIPTION

[0034] Figure 1 is a schematic flow chart of an embodiment of a method 100 for classifying measurement data 2 recorded with at least one radar sensor 1.

[0035] In step 110, a spectrum 5 of time-dependent measurement data 2 of a radar sensor 1 is provided. According to block 111, the spectrum 5 can be provided in particular, for example, as a range-velocity representation. Measurement data 2 recorded with radar radiation of the modulation type JSFMCW can be selected in particular, for example, according to block 111a. Ambiguities of the measurement data 2 with respect to the velocity can then be resolved according to block 111b, taking into account any additional information.

[0036] In step 120, positions 6 are determined from the spectrum 5 from which radar radiation reflected to the radar sensor 1. In step 130, at least one group 7 of such positions 6* belonging to the same object 4 is determined. In step 140, for each position 6* in this group 7, a component 5* of the spectrum 5 is determined, which corresponds to the radar radiation reflected from this position 6*. In step 150, all these components 5* for the object 4 are aggregated and in step 160 are delivered to a classifier 8. In step 170, the object 4 is assigned to one or more classes 3a-3c of the predefined classification 3 using the classifier 8.

[0037] According to block 131, the group 7 of positions 6* can be determined, for example, by clustering according to at least one hypothesis for the presence of an object.

[0038] According to block 132, further positions 6** can be selected from the group 7 for which it is plausible that radar radiation was reflected there from the object 4 characterized by the group 7. According to block 141, components 5** of the spectrum 5 can then also be determined for these positions 6** and can be aggregated together according to block 151.

[0039] According to block 161, the positions 6*, 6** for which components 5*, 5** of the spectrum 5 have been aggregated and / or further features of the radar reflections received from these positions (6*, 6**) are delivered to the classifier 8 together with these components 5*, 5** of the spectrum 5.

[0040] Some examples are explained within block 170 of how the two different information types can be processed together in the classifier 8.

[0041] According to block 171, the components 5*, 5** of the spectrum 5 can be delivered to a first neural network in the classifier 8. According to block 171a, this first neural network can be, inter alia, a convolutional neural network, for example.

[0042] According to block 172, the positions 6*, 6** or further features of the radar reflections can then be delivered to a second neural network in the classifier 8. According to block 172a, this second neural network can be, inter alia, a neural network with a PointNet architecture.

[0043] According to block 173, the first neural network and the second neural network can be fused with one another in the classifier 8. As previously set out, the information can be, inter alia, brought to a similar level of abstraction in the two networks separately from one another before the information is processed together in the fusion section.

[0044] According to block 174, the components 5*, 5** of the spectrum 5 can be fed to a first layer of a neural network in the classifier 8. According to block 175, the positions 6*, 6** or other features of the radar reflections can be fed to a deeper layer of the same neural network. As set forth previously, here too the level of abstraction of the two data types can be adapted to each other before the two data types are processed together.

[0045] According to block 176, the components 5*, 5** of the spectrum 5 on the one hand and the positions 6*, 6** or other features of the radar reflections on the other hand can be fed to the classifier 8 in different channels at the same input. The entire processing of the two data types then runs synchronously in the classifier 8.

[0046] In a step 180, the assignment of the objects 4 to one or more classes 3a-3c can be further processed into a maneuver signal 180a. In a step 190, the vehicle 50 can be maneuvered using the maneuver signal 180a.

[0047] Figure 2 The relationship between the positions 6* at the objects 4 (here: the vehicle 50) and the components 5* of the spectrum 5, which are aggregated and fed to the classifier 8, is illustrated. For each position 6*, there is a corresponding point 6' in the coordinates of the spectrum 5. Around each of these corresponding points 6', a rectangular (here: square) segment is cut out as a component 5*.

[0048] Additionally, the positions 6* themselves or other features of the radar reflections can also be fed to the classifier 8 and processed together with the components 5*.

Claims

1. A method (100) for classifying an object (4) from measurement data (2) recorded with at least one radar sensor (1), the method having the steps of: • providing a spectrum (5) of time-dependent measurement data (2) of the radar sensor (1), • determining positions (6) from which radar radiation reflected to the radar sensor (1) from the spectrum (5), • determining at least one group (7) of such positions (6*) belonging to the same object (4), • determining, for each position (6*) in the group (7), a component (5*) of the spectrum (5) corresponding to radar radiation reflected from the position (6*), • aggregating all these components (5*) for the object (4), • the positions (6*) to which the components (5*) of the spectrum (5) are aggregated and / or other features of the radar reflections received from these positions (6*) being delivered to a classifier together with the components (5*) of the spectrum (5); and • assigning the object (4) to one or more classes (3a-3c) of a pre-given classification (3) with the classifier (8).

2. The method (100) of claim 1, wherein the group (7) of positions (6*) is determined by clustering according to at least one hypothesis of the existence of an object (4).

3. The method (100) of any one of claims 1 to 2, wherein further positions (6**) of the determined positions (6) are selected from the group (7), for which it is plausible that radar radiation reflected there from an object (4) characterized by the group (7), and wherein components (5**) of the spectrum (5) are also determined for these positions (6**) and aggregated therewith.

4. The method (100) of any one of claims 1 to 2, wherein the spectrum (5) is provided as a range-velocity representation.

5. The method (100) of claim 4, wherein measurement data (2) are selected which are recorded with modulated-type joint sampling frequency continuous wave, JSFMCW, radar radiation, and wherein ambiguities with respect to velocity are resolved taking into account additional information.

6. The method (100) of any one of claims 1 to 2, wherein the at least one component (5*, 5**) aggregated therewith of the spectrum (5) is a rectangular piece of the coordinates of the spectrum (5).

7. The method (100) of any one of claims 1 to 2, wherein • the components (5*, 5**) of the spectrum (5) are delivered to a first neural network in the classifier, • the positions (6*, 6**) or other features are delivered to a second neural network in the classifier, and • the first neural network and the second neural network are fused into a common network in the classifier (8).

8. The method (100) of claim 7, wherein • a convolutional neural network is selected as the first neural network, and / or • the second neural network is selected as a recurrent neural network. • a neural network with a PointNet architecture is selected as the second neural network.

9. The method (100) according to any one of claims 1 to 2, wherein components (5*, 5**) of the spectrum (5) and the position (6*, 6**) or other features of the radar reflections are fed to the classifier (8) simultaneously in different channels at the same input.

10. The method (100) according to any one of claims 1 to 2, wherein components (5*, 5**) of the spectrum (5) are fed to a first layer of a neural network in the classifier (8), and wherein the position (6*, 6**) or other features of the radar reflections are fed to a deeper layer of the same neural network.

11. The method (100) according to any one of claims 1 to 2, wherein components (5*, 5**) of the spectrum (5) are transformed into at least one range-velocity representation before being fed to the classifier (8).

12. The method (100) according to claim 11, wherein the ambiguity with respect to the velocity is resolved in the at least one range-velocity representation taking into account additional information.

13. The method (100) according to any one of claims 1 to 2, wherein a steering signal (180a) is constituted by an assignment to one or more classes (3a-3c) determined by the classifier and wherein a vehicle (50) is steered with the steering signal (180a).

14. A computer program product having a computer program, the computer program containing machine readable instructions which, when executed on one or more computers, cause the one or more computers to perform the method (100) according to any one of claims 1 to 13.

15. A machine readable data carrier having a computer program, the computer program containing machine readable instructions which, when executed on one or more computers, cause the one or more computers to perform the method (100) according to any one of claims 1 to 13.

16. A computer equipped with a computer program and / or with a machine readable data carrier having a computer program, the computer program containing machine readable instructions which, when executed on one or more computers, cause the one or more computers to perform the method (100) according to any one of claims 1 to 13.

Images