Feature extractor, method for using and training same, and driver assistance system

Abstract

The present invention relates to a feature extractor (100) which is configured to read in one subregion (102) of a spectrum (106), captured by a sensor (104), for each of the points (110) of a point cloud (112) provided by the sensor (104), to extract one feature vector (108) from each of the subregions (102) and to combine the feature vector (108) with the respective point (110) of the point cloud (112).

Description

[0001] Description

[0002] Feature extractor and method for using and training the same, and driver assistance system

[0003] Field of the invention

[0004] The invention relates to a feature extractor, a method for using a corresponding feature extractor, and a method for training a corresponding feature extractor. Furthermore, the invention relates to a driver assistance system for a vehicle equipped with at least one radar sensor, using a feature extractor.

[0005] State of the art

[0006] A radar sensor typically provides a point cloud of detected reflections as its output signal. Internally, however, the radar sensor receives a spectrum of electromagnetic radar waves from which the point cloud is derived. For example, it searches for maxima in the spectrum that most likely represent a reflection from an actual object.

[0007] By providing the preprocessed point cloud, the amount of data to be transmitted can be kept to a minimum. However, additional information contained in the spectrum is lost.

[0008] DE 10 2016 002 208 A1 describes a method for evaluating radar data from at least one radar sensor in a motor vehicle. DE 10 2019 219 144 A1 describes a method and system for image analysis for object detection in the surroundings of vehicles.

[0009] Disclosure of the invention

[0010] Against this background, the approach presented here presents a feature extractor, a method for using a corresponding feature extractor, and a method for training a corresponding feature extractor according to the independent claims. Advantageous further developments and improvements of the approach presented here emerge from the description and are described in the dependent claims.

[0011] Advantages of the invention

[0012] A point in a point cloud provided by a radar sensor is described by a vector of multiple numerical values. Specifically, the values ​​are coordinates of the point, such as polar coordinates of a (two-dimensional or three-dimensional) direction and a distance to the point. Additionally, a radial relative velocity between the radar sensor and an object represented by the point can be mapped.

[0013] Information about a region of an electromagnetic spectrum around the point is not yet included in the point cloud or the vectors.

[0014] In the approach presented here, such information is extracted from the spectrum by the feature extractor, specifically from a defined section of the spectrum in the region of the point. The feature extractor does not search for predefined features or patterns. Rather, the feature extractor learns independently and unsupervised which information it can and cannot extract from the spectrum of the specific sensor.

[0015] Machine learning traditionally requires training data. The training data is usually pre-labeled, meaning attributes are assigned to it, so that an "artificial intelligence" knows what is contained in the training data.

[0016] However, labeling determines the direction of learning, because anything not included as a label is not learned and is therefore ignored. The success of learning therefore depends crucially on the labeled training data. Furthermore, providing labeled training data is time-consuming and therefore expensive.

[0017] The approach presented here does not specify what should be learned during training of the feature extractor. In other words, the training goal is left largely open, but rather it is specified that something new, not precisely specified in advance, should be learned based on the provided training data. Training is based on the principle that like should be classified the same and different should be classified differently.

[0018] Training is performed with unlabeled training data. Extracts are taken from a spectrum and provided to the feature extractor. The feature extractor extracts a feature or feature vector from each of these extracts. During training, a self-running validation process checks whether the feature has been extracted according to the underlying condition that likes should be classified the same and differents should be classified differently. If the condition is not met, the feature extractor's parameters are changed until the condition is met.

[0019] The trained feature extractor is then applied to sections of a spectrum from which a point cloud has already been derived. For each point, a section of the spectrum is provided to the feature extractor, or the feature extractor reads such sections from the spectrum. The feature extractor then provides a feature vector for each point. The feature vector has a defined length or a defined number of values. The feature vector only insignificantly increases the file size of the point cloud. This allows the enriched point cloud to continue to be transmitted over data lines with a limited bandwidth or transmission rate. The feature vector is then used as additional information for object detection, route planning, or another downstream process.

[0020] The approach presented here can cost-effectively provide additional information about points in a point cloud. The self-contained unsupervised training allows information to be represented in the feature vector that is not labeled in the labeled training data.

[0021] According to a first aspect of the invention, a feature extractor is proposed which is configured to read in a sub-area of ​​a spectrum detected by the sensor for each point of a point cloud provided by a sensor, to extract a feature vector from each of the sub-areas and to merge the feature vector with the respective point of the point cloud.

[0022] According to a second aspect of the invention, a method for using a feature extractor according to the first aspect is proposed, wherein a point cloud of a sensor is read in and a sub-area of ​​an underlying spectrum of the sensor is read in for each point of the point cloud, wherein a feature vector is extracted from each of the sub-areas and merged with the respective point of the point cloud, wherein the point cloud is provided with the feature vectors.

[0023] According to a third aspect of the invention, a method for training a feature extractor according to the first aspect is proposed, wherein unlabeled training data consisting of sub-regions of a spectrum of a sensor are read in, a feature vector is extracted from each of the sub-regions, and an evaluation of the extracted feature vectors is carried out using a self-monitored cost function, wherein parameters of the feature extractor are optimized using a result of the evaluation.

[0024] According to a fourth aspect of the invention, a driver assistance system for a vehicle equipped with at least one radar sensor is proposed, comprising a feature extractor according to an embodiment of the first aspect of the invention, which is configured to read in a sub-region of a spectrum detected by the radar sensor for each point of a point cloud provided by the radar sensor, to extract a feature vector from each of the sub-regions, and to merge the feature vector with the respective point of the point cloud; and a control unit configured to control functionalities of the vehicle taking into account the points of the point cloud provided by the feature extractor and merged with the feature vectors.

[0025] Ideas for embodiments of the present invention can be considered, among other things, to be based on the thoughts and findings described below.

[0026] A feature extractor can be a software module for execution on a sensor's hardware. The feature extractor can be configured specifically for pattern recognition. The patterns the feature extractor recognizes can be determined through unsupervised training of the feature extractor.

[0027] A spectrum can contain raw data from a sensor. The sensor can be a radar sensor, so the raw data is therefore radar sensor raw data. However, it can also contain other sensors or

[0028] Spectra can be analyzed in the form of raw data from other sensor types. The spectrum can be different for different sensors or sensor types. The feature extractor can be trained specifically for a sensor or sensor type. The feature extractor can be trained specifically for a specific spectrum. This allows the imaging or detection characteristics of the respective sensor or sensor type to be taken into account.

[0029] The structure of a feature vector can be specified. This allows feature extractors trained for different sensors to provide jointly usable feature vectors.

[0030] The feature extractor can be configured to compress spectral features of the sub-area into the feature vector. More information may be present in the spectrum than in the point cloud. Spectral features may be underrepresented when searching for points. Similar to background information, the spectral features can enable different interpretations of otherwise similar point vectors.

[0031] The feature extractor can be configured to read in sub-regions centered on the points. A point in the point cloud can be located in the center of a sub-region. The sub-regions can overlap.

[0032] The feature extractor can be configured to read a uniform sub-area for each point. Uniform sub-areas allow the feature extractor to evaluate each sub-area equally. Uniform sub-areas make the feature vectors comparable.

[0033] The feature extractor can be or include a self-trained, unsupervised neural network. A neural network can, in particular, be a convolutional neural network (CNN). This neural network is particularly good at compressing or abstracting information.

[0034] The neural network can be multi-layered. A final layer can be a summarization layer, which is designed to summarize all remaining information from the previous layers into the feature vector. A summarization layer can ensure similar feature vectors.

[0035] The point cloud with the feature vectors can be provided via a medium-bandwidth interface. The interface can be used to connect the sensor to a central control unit, for example. The interface can have a lower bandwidth than would be required to transmit the entire spectrum.

[0036] The feature extractor can be executed on the sensor's signal processing hardware. The feature extractor may require limited resources, eliminating the need for specially accelerated hardware to process the entire spectrum.

[0037] During training, the parameters can be changed until the evaluation result reaches an optimum. The parameters can be changed iteratively. If the result deteriorates, the direction of the parameter change can be reversed and / or the step size of the change can be reduced.

[0038] The method is preferably computer-implemented and can be implemented, for example, in software or hardware or in a mixed form of software and hardware, for example in a control unit.

[0039] Also advantageous is a computer program product or computer program with program code that can be stored on a machine-readable carrier or storage medium such as a semiconductor memory, a hard disk memory or an optical memory and is used to carry out, implement and / or control the steps of the method according to one of the embodiments described above, in particular when the program product or program is executed on a computer or a device.

[0040] It should be noted that some of the possible features and advantages of the invention are described herein with reference to different embodiments. A person skilled in the art will recognize that the features of the control device and the method can be combined, adapted, or exchanged as appropriate to achieve further embodiments of the invention.

[0041] Short description of the drawings

[0042] Embodiments of the invention are described below with reference to the accompanying drawings, wherein neither the drawings nor the description are to be construed as limiting the invention.

[0043] Fig. 1 shows a representation of a feature extractor according to an embodiment; and

[0044] Fig. 2 shows a representation of the training of a feature extractor according to one embodiment. The figures are merely schematic and not to scale. Like reference numerals denote like or equivalent features.

[0045] Embodiments of the invention

[0046] Fig. 1 shows a representation of a feature extractor 100 according to an exemplary embodiment. The feature extractor 100 is configured to extract a feature vector 108 from a sub-region 102 of a spectrum 106 acquired by a sensor 104. Each of the sub-regions 102 is assigned to a point 110 of a point cloud 112 derived from the spectrum 106. The point 110 is contained in the sub-region 102. The feature vectors 108 are assigned to the points 110 of the point cloud 112 and made available for further processing.

[0047] In one embodiment, sensor 104 is a radar sensor and detects an electromagnetic spectrum 106. Spectrum 106 is a frequency spectrum (range Doppler spectrum). A coordinate in spectrum 106 corresponds, for example, to a specific distance and Doppler (relative) velocity. Spectrum 106 represents, for example, a relative velocity of detected objects to sensor 104 via a frequency shift. Likewise, a phase shift can represent, for example, a distance of the objects from sensor 104.

[0048] The feature extractor 100 compares patterns in the sub-areas 102 with unsupervised learned patterns of features and maps a recognized pattern into the respective feature vector 108.

[0049] Fig. 2 shows a representation of a training of a feature extractor 100 according to an embodiment. The feature extractor 100 essentially corresponds to the feature extractor in Fig. 1 . Unlabeled subregions 102 of spectra of a specific sensor or sensor type are provided to the feature extractor 100 as training data 200. The feature extractor 100 extracts a feature vector 108 for each. The feature vector is evaluated by a self-monitored cost function 202, wherein parameters 204 of the feature extractor 100 are changed until the costs of the cost function 202 are minimal. The trained feature extractor 100 is then used in the sensor 104, as shown in Fig. 1, to assign corresponding feature vectors 108 to the points 110 of point clouds 112 of the sensor 104.The points 110 enriched with the feature vectors 108 are then transmitted from the sensor to a control unit 206, on which a target task 208, such as object detection, is then executed. The control unit 206 may, for example, be located in a vehicle and possibly be part of a driver assistance system. Accordingly, the target task 208 may ultimately be controlling functions of the vehicle.

[0050] For transmission to the control unit 206, only a line with average bandwidth is required, since only the enriched point cloud 112 and not the entire spectrum is transmitted.

[0051] In one embodiment, feature extractors 100 trained for the different sensors 104 are executed on different sensors 104. The enriched point clouds 112 are then transmitted to a common control unit 206 and evaluated to fulfill the target task 208.

[0052] In the following, possible embodiments of the invention are summarized again or presented with slightly different wording.

[0053] A method for unsupervised and data-driven extraction of spectral-based feature vectors for radar sensors is presented.

[0054] Advanced driver assistance systems (ADAS) and autonomous driving (AD) require a precise representation of the vehicle's surroundings. For this purpose, radar sensors are used alongside cameras. These sensors provide measurements in the form of a point cloud (radar reflections, locations). Each point is characterized by a polar coordinate (distance, azimuth angle) and other properties, referred to below as manually designed features, such as signal strength, radar cross-section RCS, elevation angle, and many more. From these point clouds, the position, pose, class, and possibly other properties of relevant objects (e.g., cars, trucks, or pedestrians) are determined using perception algorithms. With the advent of deep learning, traditional perception algorithms are increasingly being replaced by data-driven neural networks, e.g., object detection networks.While perception using object detection networks is already very common with lidar sensors, radar object detection networks are still less widespread due to the smaller number of points. Other deep learning-based applications, such as semantic segmentation (including the classification of drivable areas or visibility testing) or localization, are also becoming increasingly technically feasible. Radar measurements in the form of a point cloud are often used for these purposes, as point clouds provide a good abstraction while simultaneously requiring a small amount of data.

[0055] In the literature, however, approaches that utilize information from the radar spectrum sometimes achieve better results. Spectra are a more raw data format than reflections, which may contain additional information that is discarded during point cloud extraction.

[0056] Direct spectra processing is very computationally and memory-intensive and will therefore not be easy to implement in practice. In particular, transmitting spectra from sensors to a central control unit would require very high data rates.

[0057] Currently, radar sensors mostly rely on manually designed features. Here, an approach is considered that uses feature extraction from spectra.

[0058] To date, a target task, such as object detection, is trained using supervised learning (with so-called labels), using both the point cloud and a patch of the spectrum (for each reflection) as input. Ultimately, the intermediate representation, i.e., the patch of the spectrum, can be extracted and used as a feature vector.

[0059] Manually designed features, such as the Radar Cross Section (RCS), have several advantages and disadvantages. They allow for ease of use and good interpretation. However, designing additional features and adapting them to new sensors and algorithms requires a lot of manual work. The informative value of the features and their associated application potential are limited, as manually created features can lead to a loss of information.

[0060] Data-driven and supervised feature extraction from spectral data also has advantages and disadvantages. The features are highly informative and thus lead to an improvement in the target task. Since feature extraction is learned end-to-end with the target task in a supervised manner, labels are required. This results in considerable additional effort. Learning in conjunction with the target task may make the features suitable only for the target task and not generally for a broad spectrum of tasks.

[0061] Current approaches to feature extraction are essentially based on human decisions such as the design of a feature or the selection of a target task for a data-driven approach.

[0062] In contrast to manual feature design, the approach presented here learns feature extraction automatically. Furthermore, feature extraction is performed in a general way (without prior definition of the target task) and unsupervised (without labels).

[0063] As a result, feature extraction from spectral data is learned automatically and unsupervised, allowing the features to be used in various algorithms and target tasks. Through unsupervised learning, the approach presented here enables cost-effective and scalable improvement and expansion of future sensors. This is a significant advantage over a supervised data-driven approach, which requires both time- and cost-intensive labels and is dependent on the target task, e.g., object detection.

[0064] At its core is a module for extracting features from the spectrum, but specific to each reflection point. In contrast to previously used manually created features, such as signal strengths, variances in distance / speed estimates, or the extent of peaks in the spectrum, the module described is a neural network. This is self-supervised and trained independently of any subsequent target task. The network is trained to extract a compressed feature vector for the region of interest from the spectrum for each reflection. Training is self-supervised, e.g., using a contrastive cost function (contrastive loss), thus eliminating the need for manual labeling.

[0065] In practice, the feature extraction module can then run on the individual radar sensors, while the subsequent algorithm for the target task, e.g., an object detection network, runs on another sensor or on a central control unit, possibly processing data from multiple radar sensors. The interface (data transmitted between sensors and the central unit) then consists of a point cloud (as before), with each point additionally containing the learned (trained) features from the spectrum.

[0066] The approach presented here has significant advantages. The interface still requires a low data rate for transmitting the point cloud from the sensor to the central control unit and also contains spectrum information. For the (self-supervised) learning of the extraction network, only the recorded data from the sensor (spectrum and reflections) are required. Neither the target task nor its labels are necessary. Thanks to self-supervised learning, the learned features are independent of their subsequent use, thus enabling their use in a wide range of applications and algorithms. Since no (manual) labels are required for training, generating the training data is cost- and time-efficient.

[0067] Fig. 1 shows a simple example. A 2-dimensional patch is extracted from the range Doppler or kl power spectrum. This happens for each location, so that the range Doppler bin of the location lies in the center of the respective patch. The patch has a size of, for example, 7x7 bins. The feature extractor consists of a neural network, e.g., several 2D convolutional layers and pooling layers. The last layer of the feature extractor is a pooling layer over all remaining cells of the local grid, resulting in a single feature vector for each location. The feature vector can, for example, contain 32 learned features, each encoded with 16 bits.

[0068] The point cloud with the learned feature vectors can then be further processed by a common network, e.g., for object detection. During training, the feature extractor is trained using a self-supervised cost function (loss) of a so-called pretext task, as shown in the upper part of Fig. 2. No manual labeling is necessary, as self-supervised cost functions do not require them. A widely used method for self-supervised learning is the use of a so-called contrastive loss. This or other self-supervised cost functions can be used for training. The self-supervised loss is then used to adapt all trainable weights of the network using gradient descent (backpropagation).

[0069] At inference time (in the product), the feature extractor in the sensor is used to solve the desired target task, e.g., object detection. The feature extractor and the target task algorithm can run on different devices, as shown in the lower half of Fig. 2. For example, the feature extractor can run on the radar sensor, and an object detector can run on a central control unit.

[0070] In a system with several different radar sensors, a separate feature extractor can be used for each type of sensor and an object detector can aggregate the point clouds with the learned features of all sensors.

[0071] In one embodiment, the method described above is applied in a simplified manner. The region of the spectrum can be an image other than a two-dimensional range-Doppler image. Instead of a power spectrum, amplitudes and phases or complex-valued spectra can also be used. The region in the spectrum can be other than two-dimensional, i.e. 1-dimensional (e.g., only range or only Doppler), 3-dimensional (e.g., range, Doppler and azimuth) or multi-dimensional. The region in the spectrum can be other than a rectangular patch. The region can use any cells in the spectrum. The feature extractor can consist of any layers of a neural network. The network for the final target task is not limited to one task, e.g., object detection, but can also include semantic segmentation, localization, other tasks, or a combination of several tasks.

[0072] Further extensions are possible. Multiple networks can be trained for different tasks using the same feature extractor. The feature extractor can then be used for multiple tasks. The tasks of the network following the feature extractor are not limited to perception, but can also include planning, functional, etc. Furthermore, modulation parameters, antenna calibration data, and RF settings (transmit power, RF filter, etc.) can be used to account for or compensate for sensor and modulation dependencies.

[0073] Instead of feature extraction for each reflection, a (highly compressed) feature representation can also be learned across the entire spectrum. The goal would be to enable a targeting task on radar spectra, with the feature extraction of the neural network running on the sensor and the actual task on the central control unit. Furthermore, the application is not limited to radar data. The method can be applied to any sensor that has a point cloud as its output representation, such as a lidar, RGBD camera, etc. Instead of the spectrum, a different raw data representation is then selected. For example, the received time signal of each beam of a time-of-flight lidar sensor could be used.

[0074] Figure 1 shows an example of feature extraction on a radar sensor based on a section of the spectrum for all reflections. For each reflection, a patch (partial area) from the corresponding spectrum is used to extract the features.

[0075] Fig. 2 illustrates the training process and subsequent use of the trained feature extractor. During training, the extractor is trained using patches from the spectrum. A self-supervised cost function is used to solve a so-called pretext task. This task requires no labels. After training, the feature extractor is deployed on one or more sensors to extract features from the corresponding spectra in addition to the reflections. These point clouds are then used for any target task, e.g., object detection.

[0076] The proposed architecture can be used in future assistance systems and automated driving functions. Further applications include all radar sensors that use machine learning, for example, stationary radar sensors for traffic monitoring or radar sensors for bicycles. The approach can also be used in robotics (e.g., obstacle detection for autonomous lawnmowers).

[0077] Finally, it should be noted that terms such as “comprising”, “comprehensive”, etc. do not exclude other elements or steps and

[0078] Terms such as "a" or "an" do not exclude a plurality. Reference signs in the claims are not to be considered as limiting.

Claims

Claims 1 . Feature extractor (100) which is configured to read in a sub-area (102) of a spectrum (106) detected by the sensor (104) for each point (110) of a point cloud (112) provided by a sensor (104), to extract a feature vector (108) from each of the sub-areas (102), and to merge the feature vector (108) with the respective point (110) of the point cloud (112).

2. Feature extractor (100) according to claim 1, which is configured to compress spectral features of the sub-area (102) into the feature vector (108).

3. Feature extractor (100) according to one of the preceding claims, which is configured to read in partial areas (102) centered on the points (110).

4. Feature extractor (100) according to one of the preceding claims, which is configured to read in a uniform sub-area (102) for each point (110).

5. Feature extractor (100) according to one of the preceding claims, wherein the feature extractor (100) is an independently and unsupervised trained neural network.

6. The feature extractor (100) according to claim 5, wherein the neural network is multi-layered, wherein a last layer is a summary layer configured to summarize all remaining information of the previous layers into the feature vector (108).

7. A method for using a feature extractor (100) according to one of claims 1 to 6, wherein a point cloud (112) of a sensor (104) is read in and for each point (110) of the point cloud (112) a sub-area (102) of an underlying spectrum (106) of the sensor (104) is read in, wherein a feature vector (108) is extracted from each of the sub-areas (102) and merged with the respective point (110) of the point cloud (112), wherein the point cloud (112) is provided with the feature vectors (108).

8. The method according to claim 7, wherein the point cloud (112) with the feature vectors (108) is provided via a medium bandwidth interface.

9. Method according to one of the preceding claims, wherein the feature extractor (100) is executed on a signal processing hardware of the sensor (104).

10. A method for training a feature extractor (100) according to one of claims 1 to 6, in which unlabeled training data (200) consisting of partial areas (102) of a spectrum (106) of a sensor (104) are read in, a feature vector (108) is extracted from each of the partial areas (102), and an evaluation of the extracted feature vectors (108) is carried out using a self-monitored cost function (202), wherein parameters (204) of the feature extractor (100) are optimized using a result of the evaluation.

11. Method according to claim 10, wherein the parameters (204) are changed until the result of the evaluation reaches an optimum.

12. A driver assistance system for a vehicle equipped with at least one radar sensor, comprising: a feature extractor (100) according to one of claims 1 to 6, which is configured to read in a sub-region (102) of a spectrum (106) detected by the radar sensor for each point (110) of a point cloud (112) provided by the radar sensor, to extract a feature vector (108) from each of the sub-regions (102), and to merge the feature vector (108) with the respective point (110) of the point cloud (112); and a control unit (206) configured to control functionalities of the vehicle taking into account the points (110) of the point cloud (112) provided by the feature extractor and merged with the feature vectors (108).

Images