Identification of objects by a sensor system for a vehicle using at least one deep neural network of a radar sensor

Abstract

The invention relates to a sensor system (100) for a vehicle, having at least one radar sensor (10) which comprises a first evaluation unit (20) which is designed to generate distance-relative speed spectra (32) of received radar signals (320) and, on the basis of the distance-relative speed spectra (32), to generate respective detections (34); the first evaluation unit (20) comprising a first deep neural network (30.4) which is designed to generate first processing results (36), assigned to the respective detections (34), on the basis of sections (32.2), assigned to the respective detections (34), of the distance-relative speed spectra (32); the sensor system (100) comprising an object identification unit (24) which is designed to generate object data (52) of identified objects (300) from input data of the object identification unit (24), wherein the input data of the object identification unit (24) are based on and / or comprise the detections (34) and the first processing results (36).

Description

[0001] Description

[0002] title

[0003] Detection of objects by a sensor system for a vehicle using at least one deep neural network of a radar sensor

[0004] The invention relates to a sensor system for a vehicle, wherein the sensor system comprises at least one radar sensor.

[0005] background

[0006] Sensor systems are ubiquitous in modern vehicles and play an important role in ensuring safety and comfort.

[0007] Radar sensors are widely used in vehicles and are specifically designed to measure the distance between a vehicle and other objects in its surroundings, as well as the relative speed of the objects with respect to the vehicle.

[0008] Radar systems use radar signals (electromagnetic waves) to

[0009] To locate and detect objects in the vehicle's surroundings. The radar signals are emitted by the vehicle's radar sensor and

[0010] objects in the surrounding area. The reflected radar signals are received by the sensor and can be analyzed by the sensor or by a vehicle control unit. Radar sensors use the Doppler effect to

[0011] To measure the relative speed between the vehicle and other objects in the surrounding area. The Doppler effect occurs when the electromagnetic wave is reflected by an object approaching or moving away from the radar sensor, thereby changing the frequency of the reflected wave. When a radar sensor is emitted by a moving vehicle, the electromagnetic wave hits a moving object in the surrounding area, such as another vehicle. As the object moves towards the sensor, the frequency of the reflected wave increases, indicating a shortening of the wavelength. Conversely, as the object moves away from the sensor, the frequency of the reflected wave decreases, thus lengthening the wavelength.

[0012] By measuring these frequency changes, the radar sensor can calculate the relative speed between the host vehicle and the surrounding object. This information can be used to adjust the host vehicle's speed to maintain a safe distance or avoid a collision.

[0013] DE 102021207093 A1 describes a device and a method for providing classified digital images, in particular radar images, lidar images, or camera images, for a system for automatic machine learning and for updating a machine-readable program code therewith. At a first point in time, a first digital image is acquired using a recording device, in particular a radar system, a lidar system, or a camera system, which comprises an object that is located at a first distance from the recording device at the first point in time. A first classification of the object is determined using the data from the first digital image. At a second point in time, a second classification of the object is determined using data from a second digital image.In one example, a recording device comprises at least part of a data connection for communication of a processor with a first memory, a second memory, and the radar system. The processor is configured to store, among other things, a current segment in the first memory. In the example, a segment is a part of a recording. A recording captured with a radar sensor comprises a spectrum. The segment of such a recording is a part of the spectrum. In the example, the processor is configured to perform object type recognition for a segment that includes a detected object.

[0014] Disclosure of the invention

[0015] One of the challenges in developing radar systems is the amount of data to be evaluated. Greater measurement accuracy, greater range, and higher temporal resolution each require increasing amounts of data that must be generated and processed. In order to process the growing volumes of data without overloading the sensor system or impairing the reaction time when detecting obstacles, larger and more powerful hardware is required, which in turn increases the costs and energy requirements of a sensor system. The larger the hardware used, the more expensive the overall radar environment sensor and signal processing system becomes. The object of the invention is to provide a novel sensor system that allows sensor information from a vehicle's radar sensor to be used efficiently in such a way that computing resources are saved and the associated production costs are reduced.It is particularly desirable to enable improved recognition of object types.

[0016] At the same time, there is also a strong trend in radar sensor technology towards more machine learning, especially machine learning / deep learning, which requires more computing resources.

[0017] One or more of the objects are achieved according to the invention by a sensor system having the features specified in independent claim 1. Advantageous embodiments are specified below and in the subclaims.

[0018] According to one aspect of the invention, a sensor system for a vehicle comprises: at least one radar sensor designed to transmit radar signals and to receive radar signals reflected from a radar target, wherein the radar sensor comprises a first evaluation unit designed to generate range-relative speed spectra of the received radar signals, wherein the first evaluation unit is designed to generate respective detections based on the range-relative speed spectra; characterized in that the first evaluation unit comprises a first deep neural network designed to generate first processing results associated with the respective detections based on sub-regions of the range-relative speed spectra associated with the respective detections, wherein the sensor system comprises an object recognition unit designedTo generate object data of detected objects from input data of the object recognition unit, wherein the input data of the object recognition unit is based on and / or includes the detections and the initial processing results. uO A deep neural network is also referred to as a DNN. The term neural network is used here and below for an artificial neural network. The radar sensor can, in particular, be a radar sensor for detecting the surroundings of the vehicle. A sensor for detecting the surroundings of the vehicle can also be referred to as an environment sensor.

[0019] The core of the invention is that the data is evaluated where it is generated, thus eliminating the need for a complex transmission of "raw data" to the next processing step. By generating initial processing results associated with the respective detections in the first evaluation unit based on the sub-ranges of the range-relative velocity spectra assigned to the respective detections, these sub-ranges of the spectra are evaluated directly where the spectra are available or calculated. This enables small and cost-effective hardware for radar sensors and central control units of such sensor systems, even when more machine learning / deep learning is used, for example, in the form of the first deep neural network.In order to do justice to both trends and find a good compromise between them, embodiments are also presented below which, for example, comprise a second deep neural network and / or a third deep neural network in addition to the first deep neural network. The deep neural networks can, for example, be implemented in a distributed manner on different hardware, for example a radar sensor and a (e.g. central) control device / unit. This makes it possible, for example, to implement distributed machine learning on different hardware. This makes it possible to use deep learning in radar on several or even all levels of signal processing. At the same time, as few computing resources as possible are used. This makes it possible to achieve maximum functionality at minimal cost.In particular, distributed machine learning can be implemented across different devices, rather than brute-force deep learning of raw signals on a central control unit. The first processing results, the second processing results mentioned below, and / or the portions of object data generated by a third deep neural network mentioned below are appended as additional information to a signal that is already transmitted as standard (such as a detection list, a list of radar target locations, etc.). The raw data for this can therefore be inexpensively deleted from memory. In particular, the results of distributed machine learning can thus be appended as additional information to a signal that is already transmitted as standard, for example.

[0020] A sub-area of ​​a spectrum assigned to a respective detection is also referred to as a region of interest (ROI). The initial processing results can be referred to as ROI features or sub-area features.

[0021] The generated detections can, for example, be output by the first evaluation unit as a list of detections (also referred to as a detection list). The first evaluation unit can, for example, be configured to output the detections expanded by the first processing results as an expanded list of detections.

[0022] For further evaluation, the initial processing results can be transmitted to the object recognition unit without requiring transmission of the partial areas or the entire spectra. The input data of the object recognition unit is based on and / or includes the initial processing results. The object recognition unit thus differs from an object recognition unit that does not have such initial processing results available and that itself accesses the partial areas of the spectra or the entire spectra. The data to be stored or transmitted

[0023] The amount of data can thus be significantly reduced, while the DNN still evaluates the information contained in the relevant sub-range of a spectrum. This allows the amount of data to be further processed to be reduced to such an extent that less hardware is required, thus saving production costs.

[0024] The input data of the object detection unit may, for example, be based on and / or include the extended list of detections.

[0025] The first evaluation unit can be designed, in particular, to generate the range-relative velocity spectra of the received radar signals by Fourier transformation (in particular two-dimensional Fourier transformation) of baseband signals of the received radar signals. This can be done for each evaluation channel of the radar sensor. The spectra can, for example, be complex spectra, i.e., include amplitudes and phases.

[0026] The first evaluation unit can, for example, comprise a first memory. The first evaluation unit can, for example, comprise a processor. The processor can, for example, be designed to store the range-relative velocity spectra of the received radar signals, the generated detections, and / or the first processing results in the first memory. At least part of the first evaluation unit can, for example, be implemented as a system-on-chip (SoC).

[0027] The object recognition unit may, for example, comprise a third memory. The object recognition unit may, for example, comprise a third processor. The third processor may, for example, be designed to store the input data of the object recognition unit and / or the object data of recognized objects in the third memory.

[0028] In embodiments, the sensor system comprises a control unit, also referred to as an electronic control unit (ECU). The radar sensor comprises the first evaluation unit. The radar sensor may comprise the object detection unit, or the control unit may comprise the object detection unit. The control unit may be a central control unit of the sensor system.

[0029] The sensor system can, for example, comprise three (hardware) modules or hardware components: a DSP (Digital Signal Processing) front-end on the respective radar sensor itself (the DSP front-end is thus a component of the radar sensor); a DSP back-end (depending on the architecture, either on the respective radar sensor or on a central control unit), which can include the second evaluation unit mentioned below; and a perception-with-fusion unit (which can be or include the object detection unit), which, depending on the architecture, can also be implemented on the sensors themselves or on a central control unit. The results can be transmitted between these hardware elements via a data connection or a respective data connection, for example in the form of a respective bus system.For example, the results of distributed machine learning can be appended to the conventional results, which can be determined individually on each hardware element. These can be additional machine-learned features.

[0030] The first deep neural network can be configured to receive and process, from the range-relative velocity spectra or spectral data obtained therefrom, only the subregions (region of interest) of the range-relative velocity spectra associated with the respective detections as input. In particular, the first deep neural network can be configured to receive and process, from the range-relative velocity spectra or spectral data obtained therefrom, only one subregion (region of interest) of a range-relative velocity spectrum associated with a respective detection as input (i.e., for a respective processing run).

[0031] The sensor system or in particular the first evaluation unit can be designed, after generating the first processing results, to delete from the first memory at least those areas of the range-relative speed spectra that exceed the range-relative speed spectral data assigned to the respective radar targets, or to use the corresponding memory areas for overwriting

[0032] The sensor system or in particular the first evaluation unit can be designed to delete the distance-relative speed spectra from the first memory after generating the first processing results or to release the corresponding memory areas for overwriting (or for re-assignment).

[0033] occupancy).

[0034] In embodiments, the radar sensor comprises an antenna arrangement with a plurality of antenna elements arranged in different positions in a direction in which the radar sensor is angle-resolving, wherein the sensor system further comprises: a second evaluation unit designed to estimate an angle of the associated radar target based on range-relative speed spectral data of a plurality of evaluation channels of the radar sensor assigned to a radar target, characterized in that the second evaluation unit comprises a second deep neural network designed to generate a second processing result assigned to the respective radar target based on range-relative speed spectral data of the plurality of evaluation channels of the radar sensor assigned to the radar target, wherein the input data of the object detection unit is based on the detections, the estimated angles,are based on and / or include the first processing results and the second processing results.

[0035] The spectral data of the multiple evaluation channels assigned to a radar target can, for example, comprise at least one spectral value for each evaluation channel from a sub-range of the range-relative velocity spectrum of the evaluation channel, wherein the sub-range is assigned to a detection corresponding to the radar target. In particular, the spectral data of a respective evaluation channel can have a smaller data volume than the relevant sub-range of the range-relative velocity spectrum of the evaluation channel.

[0036] The range-relative speed spectral data of several evaluation channels of the radar sensor assigned to a radar target can, for example, be transmitted from the first evaluation unit to the second evaluation unit.

[0037] By generating the second processing result assigned to the respective radar target in the second evaluation unit based on the range-relative velocity spectral data of the radar sensor's multiple evaluation channels assigned to the radar target, the range-relative velocity spectral data of the multiple evaluation channels assigned to the radar target are evaluated directly where the spectral data of the multiple evaluation channels are available or are calculated. In particular, for example, the second evaluation unit can generate the second processing result based on the same spectral data of the multiple evaluation channels assigned to a radar target on which the second evaluation unit's estimation of the angle of the radar target is also based.

[0038] The first evaluation unit can be designed, in particular, to generate the range-relative velocity spectra of the received radar signals of a respective evaluation channel of the multiple evaluation channels by Fourier transforming baseband signals of the received radar signals. The multiple antenna elements can, for example, comprise multiple transmit antenna elements and / or multiple receive antenna elements. A respective evaluation channel can, for example, be assigned to a respective receive antenna element of the radar sensor, or in particular, to a respective combination of a transmit antenna element and a receive antenna element of the radar sensor.

[0039] In order to evaluate the range-relative velocity spectral data of the multiple evaluation channels assigned to a radar target, these can be transmitted to the second evaluation unit without the need to transmit the partial ranges or the entire spectra.

[0040] In addition, for further evaluation of the second processing results, these can be transmitted to the object recognition unit without the need to transmit the spectral data of the multiple evaluation channels, the sub-areas or the entire spectra.

[0041] The input data of the object recognition unit can be based on and / or include the second processing results. The object recognition unit thus differs from an object recognition unit that does not have such second processing results available and that itself accesses the spectral values ​​of the evaluation channels, the partial ranges of the spectra, or the entire spectra. The amount of data to be stored or transmitted can thus be significantly reduced.

[0042] The range and relative velocity spectral data from the multiple analysis channels assigned to the respective radar target are also referred to below as the direction-of-arrival spectrum or DOA spectrum. The term "spectrum" here refers to the spectral values ​​originating from the multiple analysis channels. The spectral values ​​can, for example, be complex spectral values, i.e., include amplitude and phase. The secondary processing results can be referred to, in particular, as DOA features (direction-of-arrival features) or angular features.

[0043] The second evaluation unit may, for example, comprise a second memory. The second evaluation unit may, for example, comprise a processor. The processor may, for example, be designed to store in the second memory the range-relative velocity spectral data of the plurality of evaluation channels assigned to a respective radar target, the angle of the respective radar target, the first processing results, and / or the second processing results. The radar sensor may comprise the second evaluation unit, or the control unit may comprise the second evaluation unit. The second evaluation unit may comprise an angle estimator designed to estimate an angle of the associated radar target based on the range-relative velocity spectral data of the plurality of evaluation channels of the radar sensor assigned to a radar target.

[0044] The second deep neural network can be configured to receive and process, from the range-relative velocity spectra or spectral data obtained therefrom, only the range-relative velocity spectral data from multiple evaluation channels assigned to the respective radar target as input. The second deep neural network can be configured to receive and process the input data of the angle estimator as input. In particular, the second deep neural network can be configured to receive and process, from the range-relative velocity spectra or spectral data obtained therefrom, only the input data of the angle estimator and, optionally, i.e., depending on the embodiment, the first processing results as input.

[0045] In embodiments, the second evaluation unit is configured to determine locations of radar targets in Cartesian coordinates based on the detections and the estimated angles, wherein the input data of the object detection unit is based on and / or includes the locations of radar targets, the first processing results, and the second processing results.

[0046] The determined locations of radar targets can be output by the second evaluation unit, for example, as a list of locations (also referred to as a location list). The second evaluation unit can, for example, be configured to output the locations of radar targets expanded by the first and second processing results as an expanded list of radar target locations.

[0047] In embodiments, the sensor system comprises a data connection, wherein the object detection unit is configured to receive the input data via the data connection.

[0048] The data connection may in particular comprise a data bus or at least one data line.

[0049] The second evaluation unit can, for example, be designed to receive the range-relative speed spectral data associated with a radar target from a plurality of evaluation channels of the radar sensor via the data connection, in particular to receive them from the first evaluation unit via the data connection.

[0050] The object recognition unit can be designed to receive the input data via the data connection from the first evaluation unit and / or from the second

[0051] to receive the evaluation unit.

[0052] In embodiments, the object recognition unit comprises a third deep neural network configured to generate at least a portion of the object data of recognized objects from the input data of the object recognition unit.

[0053] In embodiments, the first deep neural network, the second deep neural network and the third deep neural network can be designed for distributed processing, in which, in a first processing thread, the first deep neural network generates the first processing results based on partial regions of the range-relative velocity spectra assigned to the respective detections, and in a second processing thread, the second deep neural network generates the second processing results based on the range-relative velocity spectral data of a plurality of evaluation channels assigned to the respective radar target, wherein the third deep neural network receives and processes at least the first processing results and the second processing results as input.The third deep neural network can, for example, be designed to generate at least part of the object data of detected objects based on the locations of radar targets, the first processing results, and the second processing results. The second processing thread can be arranged in parallel to the first processing thread.

[0054] In embodiments, the input data of the object detection unit does not include the range-relative velocity spectra. In particular, the input data of the third deep neural network does not include the range-relative velocity spectra. In particular, the input data of the object detection unit does not include the range-relative velocity spectra or the range-relative velocity spectral data of multiple evaluation channels associated with a radar target. In particular, the input data of the third deep neural network does not include the range-relative velocity spectra or the range-relative velocity spectral data of multiple evaluation channels associated with a radar target.The third deep neural network may be designed to receive and process only the generated detections, angles and / or locations of radar targets as input, in addition to the first processing results and the second processing results from the range-relative velocity spectra or spectral data obtained therefrom or detections, angles or locations of radar targets.

[0055] For example, the range-relative velocity spectral data of the multiple evaluation channels assigned to a radar target can be limited to the respective sub-range of the spectrum that is fed to the first deep neural network in the respective evaluation channel. This sub-range is, in particular, a true sub-range, i.e., a range whose scope is smaller than the entire spectrum. In particular, the second evaluation unit can be configured to receive and / or process exclusively the range-relative velocity spectral data of the multiple evaluation channels assigned to the respective radar target from the range-relative velocity spectra of the multiple evaluation channels.In particular, the object detection unit can be configured to obtain the angles, the first processing results, and / or the second processing results from the range-relative velocity spectral data of the multiple evaluation channels, excluding the detections, or to obtain and process the locations of radar targets based thereon and the first processing results and / or second processing results. Compared to brute force processing of the complete spectra on a central control unit, this achieves a significant reduction in the amount of data to be stored and the amount of data to be transmitted. In embodiments, the third deep neural network is configured to generate an object type classification as part of the object data of detected objects.

[0056] In embodiments, the third deep neural network comprises a backbone (or preprocessing part) and a plurality of heads, wherein the heads are each configured to generate at least one of an object type classification, an object box, an object orientation, an object speed, and an object yaw rate as part of the object data of detected objects. The object box can, in particular, be or comprise a (box-shaped) geometric boundary of a detected object.

[0057] The deep neural networks mentioned herein are specifically artificial deep neural networks.

[0058] For example, the first deep neural network, the second deep neural network, and / or the third deep neural network may each be a trained deep neural network based on machine learning.

[0059] In embodiments, the sensor system comprises at least one further sensor for detecting the surroundings of the vehicle, wherein the input data of the object detection unit comprises further input data that is based on or includes data from the at least one further sensor. The at least one further sensor for detecting the surroundings of the vehicle can be referred to as an environment sensor. The data from the at least one further sensor can be, for example, environment data. The environment data can be or include location data of objects detected by the at least one further sensor.

[0060] In particular, the object recognition unit can be designed to merge the input data based on the detections and, if applicable, the angles as well as on the first processing results of the first evaluation unit and, if applicable, the second processing results of the second evaluation unit and / or comprising these with the further input data.

[0061] For example, the at least one further sensor comprises a lidar sensor or a camera system and / or a further radar sensor.

[0062] The lidar sensor can, for example, be designed to provide location data of detected objects based on lidar images. The camera system can be designed to provide location data of detected objects based on camera images. The respective location data can be fed to the object detection unit as further input data.

[0063] In embodiments, the sensor system comprises an object tracking unit configured to track objects based on the object data of detected objects. Tracking objects is also referred to as object tracking.

[0064] The object data of detected objects can, for example, be transferred from the object detection unit to the object tracking unit.

[0065] The control unit may comprise the object tracking unit. The object tracking unit may, for example, be designed to receive the object data of recognized objects via the data connection, in particular from the object recognition unit via the data connection. The object tracking unit may, for example, comprise a fourth memory. The object tracking unit may, for example, comprise a processor. The processor may, for example, be designed to store object data of recognized objects from a current

[0066] 5 evaluation cycle of the object recognition unit and / or from at least one previous evaluation cycle of the object recognition unit.

[0067] The object tracking unit can, for example, be designed to track objects based on the object data of detected objects from a current evaluation cycle and from at least one previous evaluation cycle.

[0068] In embodiments, the sensor system comprises a (or the) data connection and a control unit, wherein the control unit comprises the object tracking unit, and wherein the control unit (in particular a processor of the control unit) is designed to receive the object data of detected objects via the data connection.

[0069] In the following, examples of implementation are explained in more detail with the aid of drawings.

[0070] 20 explains. They show:

[0071] Fig. 1 is a schematic representation of a sensor system with a radar sensor, another sensor and a central control unit;

[0072] 25

[0073] Fig. 2 is a schematic representation of an embodiment of the sensor system with a first evaluation unit, a second evaluation unit, an object detection unit, and an object tracking unit; Fig. 3 is a schematic representation of signal processing in the radar sensor, including the first evaluation unit;

[0074] Fig. 4 is a schematic representation of signal processing by the second evaluation unit; and

[0075] Fig. 5 is a schematic representation of signal processing by the object detection unit and the object tracking unit.

[0076] The sensor system 100 for a vehicle shown in Fig. 1 comprises a radar sensor 10 and a central control unit 12 and optionally at least one further sensor 14.

[0077] The radar sensor 10 includes a control and evaluation unit 16. The central control unit 12 is separate from the control and evaluation unit 16 of the radar sensor 10 and is connected to it, for example, via a data connection 18. The radar sensor 10 and at most one additional sensor 14 are configured to detect an object 300 in the surroundings of the vehicle. The radar sensor 10 is designed to transmit FMCW radar signals 310 and to receive radar signals 320 reflected from the object 300. The extended object 300 can, for example, comprise multiple radar targets from which individual reflections are received.

[0078] The radar sensor 10 comprises an antenna arrangement 10.1 with several

[0079] Antenna elements 10.2, which are arranged, for example, in different positions in the azimuthal direction. The radar sensor 10 is configured, for example, to determine the angle of a radar target using MIMO (multiple input-multiple output) beamforming.

[0080] The at least one further sensor 14 can, for example, be configured to transmit a signal 330 to the object 300 and to receive at least one signal 340 reflected by the object 300. The further sensor 14 can, for example, be a lidar sensor. The at least one further sensor 14 can alternatively or additionally comprise a camera system configured to capture an image of the surroundings of the vehicle, wherein the image contains the object 300.

[0081] The at least one further sensor 14 is connected to the central control unit 12 or the radar sensor 10 via the or a further data connection 18. The data connection 18 can be implemented, for example, as a data bus.

[0082] An embodiment of the sensor system 100 is shown in more detail in Fig. 2. The control and evaluation unit 16 comprises a transmitting and receiving unit 16.1 and a first evaluation unit 20, a DSP front-end (digital signal processing front-end). The sensor system 100 further comprises a second evaluation unit 22, a DSP back-end (digital signal processing back-end). The second evaluation unit 22 can be a component of the radar sensor 10 or the central control unit 12. The sensor system 100 further comprises an object detection unit 24, which is a component of the

[0083] radar sensor 10 or the central control unit 12. Finally, the central control unit 12 includes an object tracking unit 26.

[0084] The transmitting and receiving unit 16.1 generates the radar sensor's transmission signals using FMCW modulation as an RF oscillator signal, performs a baseband transformation of the received radar signals 320, and performs an analog-to-digital conversion of the received radar signals 320. The converted radar signals are processed and evaluated by the first evaluation unit 16. For example, a preprocessing unit 30.1 can preprocess the signals and / or eliminate interference from other radar sensors and / or decode the radar signals.

[0085] A Fourier transformation unit 30.2 calculates distance-relative speed spectra of the received radar signals 320. In Fig. 1, a first memory 30.3 with a calculated distance-relative speed spectrum 32 of an evaluation channel of the radar sensor 10 is shown by way of example and schematically.

[0086] The first evaluation unit 20 determines detections 34 of radar targets of the object 300 from the respective spectrum 32 by evaluating the peaks 32.1 of the respective spectrum 32.

[0087] The first evaluation unit 20 comprises a first deep neural network 30.4, DNN (deep neural network), which evaluates an associated sub-area 32.2 of the spectrum 32 for a respective detection 34 and therefrom first

[0088] Processing results 36 are generated. The subarea 32.2 is also referred to as the region of interest (ROI). The detections 34 and associated first processing results 36 are also stored as an extended detection list 38 in the first memory 30.3 and transmitted from the first evaluation unit 20 to the second evaluation unit 22 via the data connection 18.

[0089] The second evaluation unit 22 receives from the first evaluation unit 20 range and relative velocity spectral data 42 of the multiple evaluation channels, particularly associated with a respective radar target. These data include, in particular, the respective peak 32.1 of the evaluation channel for the radar target.

[0090] Based on the spectral data 42, an angle estimator 40.3 of the second evaluation unit 22 estimates angles of the radar targets and determines locations 46 of radar targets in Cartesian coordinates from the detections 34 and the estimated angles.

[0091] The second evaluation unit 22 comprises a second deep neural network 40.1, which processes the spectral data 42 and generates second processing results 48 therefrom. The locations 46 and the associated second processing results, as well as, for example, the associated first processing results 36, are stored in a second memory 40.2 of the second evaluation unit 22 and can be transmitted to the object recognition unit 24.

[0092] The object recognition unit 24 receives as input data a location list stored in the second memory 40.2, which includes the locations 46, the first processing results 36, and the second processing results 48. The input data is thus based on the detections 34, the estimated angles, and the first and second processing results 36, 48. The third deep neural network 50.1 receives as input data the same input data as the object recognition unit 24. Furthermore, the object recognition unit 24 and / or the third deep neural network 50.1 can receive data from the at least one further sensor 14 as additional input data.

[0093] The object recognition unit 24 and the third deep neural network 50.1 determine data 52 of recognized objects 300 from their input data object. The object data 52 can be stored in a corresponding memory 50.2 of the object recognition unit 24 and transmitted to the object tracking unit 26.

[0094] The object tracking unit 26 performs object tracking of detected objects based on current object data 52 and previous object data stored in a memory 60.1.

[0095] As indicated in Fig. 2, the estimation of the angles and the processing by the second deep neural network 40.1 can be done in parallel.

[0096] Fig. 3 schematically shows an operating method of the control and evaluation unit 16 according to one embodiment. In a step 80, the transmission signals (radar signals 310) are modulated and transmitted. In a step 82, a radar signal 310 strikes an object 300 in the surroundings of the vehicle and is reflected. In a step 84, the reflected radar signal 320 is received, transformed into the baseband, and passed through the analog-to-digital converter (ADC). The remaining steps are performed by the transmitting and receiving unit 16.1. The subsequent steps are performed by the first evaluation unit 20.

[0097] In a step 86, the preprocessing, the Fourier transformation and, if necessary, the elimination of interference takes place.

[0098] In a step 88, the range-Doppler spectrum 32 is calculated. This can also be referred to as a dv spectrum. Depending on the modulation method, the coordinates of the spectrum are partially or predominantly separated according to distance d and relative velocity or Doppler velocity v, or the spectrum is structured according to distances d. Peak detection occurs in a step 90. In a step 92, the first deep neural network 30.4 is applied to a respective sub-area of ​​a spectrum (region of interest, ROI). The extended detection list 38 is obtained, in which the detections 34 are extended by the first processing results 36.

[0099] Fig. 4 schematically shows an operating method of the second evaluation unit 22 according to one embodiment. In a step 94, any velocity ambiguities in the spectral values ​​42 are resolved.

[0100] In step 95, an angle estimation follows, in which the direction of arrival of the received signals is estimated. Thus, an angle is determined.

[0101] In a step 96, the second deep neural network 40.1 is applied to the spectral values ​​42. In the process, the second processing results 48 are generated.

[0102] In step 97, radar cross sections (RCS) are estimated. Combined with range information from a detection, this can provide information about the size of an object.

[0103] From the information obtained, a location list 49 is generated, which includes, for example, locations 46 and associated first processing results 36 and second processing results 48. The aforementioned steps are performed by the second evaluation unit 22. In particular, the steps of angle estimation 95 and the application of the second deep neural network 96 can be performed in parallel. Fig. 5 schematically shows an operating method of the object recognition unit 24 and the object tracking unit 26 according to one embodiment. In a step 98, the third deep neural network 50.1 is applied to the location list 49 or to a respective location of the location list 49 with the associated first and second processing results 36, 48. In particular, the third deep neural network 50.1 can comprise a DNN backbone 50.3 and several DNN heads 50.4 (Fig. 2). The DN heads 50.4 are head pieces based on a uniform pre-processing part (backbone) 50.3 of the DNN. In a step 99, the DNN heads, also referred to as deep learning heads or DL ​​heads, perform, for example, an object type classification (OTC), a determination of an object box, a determination of an object orientation, a determination of an object speed, and / or a determination of an object yaw rate.

[0104] In a step 101, the object tracking unit 26 performs object tracking. This is also referred to as object tracking. During object detection, several locations are combined to form an object, and the object is classified. The object is thus assigned to a class, such as a class comprising cars, trucks, cyclists, pedestrians, etc. The detection of the objects thus represents a detection of the vehicle's surroundings.

[0105] During object detection, a fusion with the data from at least one further sensor 14 can also take place, for example data from another radar sensor and / or a fusion with data from other sensors 14 of other sensor types, for example a lidar sensor or a camera system. In the described embodiments of the sensor system, additional information can be obtained in the respective system part by processing the extensive data or raw data available only there by the respective deep neural network, in particular the first processing results 36, the second processing results 48 or parts of the object data 52, which are then available for further processing without the evaluated raw data having to be further stored or transmitted.

[0106] When determining an object box, for example, a box can be determined from a cloud of points or locations that corresponds to the known object.

[0107] The object classification or the additional information obtained from the first processing results and second processing results can be incorporated into a situation analysis and the tracking of the objects by the object tracking unit. In the described embodiments, the first, second, and / or third deep neural networks can be implemented as distributed deep learning modules, i.e., they can be distributed across the respective hardware units, in particular across the radar sensor 10 or the central control unit 12. The deep neural networks 30.4, 40.1, and 50.1 used can, in particular, be artificial neural networks trained by deep learning.

[0108] By evaluating the regions of interest (ROI) of the spectra 32 by the first deep neural network 30.4, the fine information contained in the spectra 32 is evaluated, which cannot be captured by a conventional evaluation of only the peaks. For example, micro-Doppler velocities can be evaluated in the respective region of interest, for example, due to the arm movements of a pedestrian. In contrast, radar targets on the outer edge of a motor vehicle are rigid, so that a motor vehicle may include several reflection points, each of which has a nearly identical relative velocity.

Claims

Patent claims 1. A sensor system (100) for a vehicle, the sensor system (100) comprising: at least one radar sensor (10) configured to emit radar signals (310) and to receive radar signals (320) reflected from a radar target, the radar sensor (10) comprising a first evaluation unit (20) configured to generate range-relative speed spectra (32) of the received radar signals (320), the first evaluation unit (20) being configured to generate respective detections (34) based on the range-relative speed spectra (32); characterized in that the first evaluation unit (20) comprises a first deep neural network (30.4) configured to generate first processing results (36) assigned to the respective detections (34) based on subregions (32) assigned to the respective detections (34).2) to generate the range-relative speed spectra (32), wherein the sensor system (100) comprises an object recognition unit (24) which is designed to generate object data (52) of detected objects (300) from input data of the object recognition unit (24), wherein the input data of the object recognition unit (24) are based on and / or comprise the detections (34) and the first processing results (36).

2. Sensor system according to claim 1, wherein the radar sensor (10) comprises an antenna arrangement (10.1) with a plurality of antenna elements (10.2) which are arranged in different positions in a direction in which the radar sensor (10) is angle-resolving, wherein the sensor system (100) further comprises: a second evaluation unit (22) which is designed to calculate, based on distance-relative speed- Spectral data (42) of a plurality of evaluation channels of the radar sensor (10) to estimate an angle of the associated radar target, characterized in that the second evaluation unit (22) comprises a second deep neural network (40.1) which is designed to generate a second processing result (48) associated with the respective radar target based on range-relative speed spectral data (42) of the plurality of evaluation channels of the radar sensor (10) associated with the radar target, wherein the input data of the object recognition unit (24) are based on and / or comprise the detections (34), the estimated angles, the first processing results (36) and the second processing results (48).

3. Sensor system according to claim 2, wherein the second evaluation unit (22) is designed to determine locations (46) of radar targets in Cartesian coordinates based on the detections (34) and the estimated angles, wherein the input data of the object recognition unit (24) are based on and / or comprise the locations (46) of radar targets, the first processing results (36) and the second processing results (48).

4. Sensor system according to one of the preceding claims, wherein the sensor system (100) comprises a data connection (18), wherein the object recognition unit (24) is designed to receive the input data via the data connection (18). 25 5. Sensor system according to one of the preceding claims, wherein the object recognition unit (24) comprises a third deep neural network (50.1) designed to generate at least part of the object data (52) of recognized objects (300) from the input data of the object recognition unit (24). uO 6. Sensor system according to claim 5, wherein the third deep neural network (50.1) is designed to generate an object type classification as part of the object data (52) of detected objects (300).

7. Sensor system according to claim 5 or 6, wherein the third deep neural Network (50.1) comprises a backbone (50.3) and a plurality of heads (50.4), wherein the heads (50.4) are designed to each generate at least one of an object type classification, an object box, an object orientation, an object speed and an object yaw rate as part of the object data of detected objects.

8. Sensor system according to one of the preceding claims, wherein the sensor system (100) comprises at least one further sensor (14) for detecting the surroundings of the vehicle, wherein the input data of the object recognition unit (24) comprise further input data which are based on or comprise data from the at least one further sensor (14).

9. Sensor system according to claim 8, wherein the at least one further sensor (14) comprises a lidar sensor or a camera system.

10. Sensor system according to one of the preceding claims, wherein the sensor system (100) comprises an object tracking unit (26) designed to track objects (300) based on the object data (52) of detected objects (300).

11. Sensor system according to claim 10, wherein the sensor system (100) comprises a (or the) data connection (18) and a control unit (12), wherein the control unit (12) comprises the object tracking unit (26), and wherein (a processor of / ) the control unit (12) is designed to receive the object data (52) of detected objects (300) via the data connection (18).

Images