Method and system for detecting an environment of a device using sparse spectra

Abstract

Disclosed are a device, a system, and a method for detecting an environment of a device using sparse spectra. The method comprises generating, with a sensor of the device, a time signal with information about features of one or more objects in the environment of the device; determining dense spectra with N dimensions based on the time signal, wherein the time signal comprises information for generating dense spectra with K dimensions, where K is greater than or equal to N; determining sparse spectra with N dimensions based on the dense spectra with N dimensions, wherein the amount of data for representing the sparse spectra is smaller than the amount of data for representing the dense spectra with N dimensions; and determining first features of the one object or the multiple objects for one point or for multiple points in the sparse spectra.

Description

[0001] The present invention relates to a method and a system for detecting an environment of a device using sparse spectra, in particular for devices in the automotive, aviation, marine, aerospace and / or manufacturing industries. State of the art

[0002] Assistance systems for controlling or supporting the control of a device, such as driver assistance systems or systems that enable autonomous control of, for example, a vehicle, an aircraft or a ship, require precise information about the environment of the device in order to enable reliable and safe control of the device.

[0003] Electromagnetic radiation in different frequency ranges can be used to collect information about the environment, using different technologies such as analog or digital photography, LIDAR (light detection and ranging) technology, or RADAR (radio detection and ranging) technology. Alternatively or additionally, acoustic waves can also be used to collect information about the environment, using, for example, ultrasound technology and / or SONAR (sound detection and ranging) technology. Other technologies suitable for scanning an environment are also possible.

[0004] The sensors of the respective technology typically provide measurements in the form of spectra or as a dense point cloud. Radar sensors, for example, can provide a point cloud consisting of the detected radar reflections. In a point cloud, each point can be characterized by one or more different dimensions, such as a distance, an azimuth angle, an elevation angle, a Doppler velocity, a radar cross-section, etc., or a selection thereof. Radar spectra can contain measured radar signals and have, for example, the dimensions of distance, Doppler velocity, azimuth, and elevation, or a selection thereof.

[0005] To provide a precise representation of a device's surroundings, environment detection algorithms process the measurement data as it is. These algorithms can, for example, work with radar point clouds and / or radar spectra.

[0006] A typical task of the algorithms can be, for example, the detection of one or more objects and / or their classification. Objects can be, for example, cars or traffic control systems, pedestrians or animals, etc. The algorithm can, for example, provide a position, a pose, a speed, a class, and possibly other properties of one or more of the detected objects in the environment. A class can, for example, be a genus of an object, such as a vehicle or living being.

[0007] Another typical task can be, for example, estimating a trajectory that is available in the environment of the device, such as a drivable area in the vicinity of a car or possible flight paths of an aircraft in a mountain range.

[0008] These tasks can be solved using deep learning methods, i.e., deep neural networks. An approach that uses object detection based on RADAR point clouds is presented by Ulrich, M., Braun, S., Köhler, D., Niederlöhner, D., Faion, F., Gläser, C., and Blume, H. in Improved Orientation Estimation and Detection with Hybrid Object Detection Networks for Automotive RADAR-, 2022 IEEE 25 th International Conference on Intelligent Transportation Systems (ITSC), arXiv:2205.02111. An approach for detection and classification on RADAR spectra is described by Patel, K., Rambach, K., Visentin, T., Rusev, D., Pfeiffer, M. and Yang, B. in Deep Learning-based Object Classification on Automotive RADAR-Spectra, 2019 IEEE RADAR-Conference (RADAR-Conf), Boston, MA, USA, 2019, pp. 1 to 6.

[0009] If an approach is based on point clouds, it uses, for example, radar point clouds as input data. This can result in information acquired by the radar sensor being lost when the measured data is mapped onto the point cloud. The algorithm may have less information available to solve a task, since the mapping of radar spectra onto radar point clouds is usually not invertible.

[0010] Algorithms that use spectra, such as radar spectra, as input data can access more information. Using spectra can be more computationally intensive than, for example, point clouds, because larger amounts of data must be processed. Furthermore, recording spectra can be more complex, because recording spectra in a short time during a measurement requires storing large amounts of data. Compared to, for example, point clouds, saving spectra during a measurement may require higher bandwidth, and more storage space may be required to store the measurement data.

[0011] The above applies analogously to measurements with other sensors that can provide spectra and / or point clouds, such as LIDAR or SONAR sensors.

[0012] It would therefore be desirable to provide a method and system that enables rapid, precise, and reliable detection of an environment without losing information that would be significant for the respective application. Furthermore, a reduction in technical effort would be desirable to enable simpler and more cost-effective recording and / or processing of the measurement data. Disclosure of the invention

[0013] The invention provides a method and a system for detecting an environment of a device using sparse spectra having the features of the independent patent claims.

[0014] Preferred embodiments are the subject of the respective subclaims.

[0015] The disclosed methods, systems, and devices are particularly directed toward enabling rapid, precise, and reliable detection of an environment without loss of information that would be relevant for the respective applications in, for example, the automotive, aviation, maritime, aerospace, and / or manufacturing industries. The disclosed methods, systems, and devices can exhibit reduced technical complexity and thus enable simpler and more cost-effective acquisition and / or processing of the measurement data.

[0016] According to a first aspect, the invention relates to a method for detecting an environment of a device using sparse spectra. The method comprises: generating, with a sensor of the device, a time signal with information about features of one or more objects in the environment of the device; determining dense spectra with N dimensions based on the time signal, wherein the time signal comprises information for generating dense spectra with K dimensions, where K is greater than or equal to N; determining sparse spectra with N dimensions based on the dense spectra with N dimensions, wherein the amount of data for representing the sparse spectra is smaller than the amount of data for representing the dense spectra with N dimensions; and determining first features of the one or more objects for one point or for several points in the sparse spectra.

[0017] According to a further development, the method further comprises selecting the one or more points in the sparse spectra.

[0018] According to a further development, the method comprises determining first features of the one object or the plurality of objects for one or more points in the sparse spectra and further determining second features of the object from the dense spectra with K dimensions without the dense spectra with N dimensions according to the one point or the plurality of points.

[0019] According to a further development, the method further comprises: detecting the one object or the multiple objects in the environment of the device based on the first features; and / or detecting the one object or the multiple objects in the environment of the device based on the second features; and / or classifying the one object or the multiple objects in the environment of the device; and / or semantically segmenting the first features and / or the second features and / or the dense spectra and / or the sparse spectra; and / or estimating a free space in the environment of the device

[0020] According to a further development, K is greater than N.

[0021] According to a further development, determining the sparse spectra with N dimensions based on the dense spectra comprises neglecting data that are smaller than a threshold value.

[0022] According to a further development, determining the sparse spectra with N dimensions based on the dense spectra with N dimensions comprises neglecting data outside a respective region around a point, in particular a local maximum.

[0023] According to a further development, a respective area around the point or the local maximum is an N-dimensional rectangle, an N-dimensional sphere or an N-dimensional ellipsoid.

[0024] According to a further development, the determination of sparse spectra with N dimensions is carried out using a neural network.

[0025] According to a further development, the sensor of the device is a RADAR sensor, a LIDAR sensor, a SONAR sensor or an ultrasonic sensor and / or N is equal to 2 with the dimensions distance and speed.

[0026] According to a second aspect, the invention relates to a system for detecting the environment of a device using sparse spectra. The system comprises: a processor; and a non-transitory computer-readable storage medium comprising instructions that, when executed by the processor, cause the system to perform the method as described above.

[0027] According to a third aspect, the invention relates to a device comprising the system described above; and one or more sensors coupled to the system. Short description of the drawings

[0028] It shows: Fig. 1 shows an exemplary method for detecting an environment of a device using sparse spectra according to one embodiment; and Fig. 2 is a schematic representation of an exemplary spectrum and a sparse spectrum generated therefrom according to an embodiment using RADAR technology.

[0029] In all figures, identical or functionally equivalent elements and devices are provided with the same reference numerals. The numbering of process steps serves the purpose of clarity and is generally not intended to imply a specific chronological order. In particular, several process steps can be performed simultaneously. Description of the embodiments

[0030] The present invention proposes the use of sparse spectra to solve environmental sensing tasks such as object detection. Sparse spectra can be created, for example, by considering only data that exceeds a certain signal threshold. This could, for example, be all points that exceed an estimated noise level. The data thus obtained can be interpreted either as sparse spectra or as very dense point clouds.

[0031] Methods that can efficiently process this data are disclosed. In contrast to the methods cited above by Ulrich et al. (2022) and Patel et al. (2019), these methods use neither full spectra nor point clouds, but rather an intermediate representation that requires less storage space than full spectra but has a higher information content than point clouds.

[0032] Compared to methods that operate on full spectra, the proposed methods can have the advantages described below.

[0033] When recording measurement data, the sparse spectra can be generated during the measurement. This reduces the bandwidth requirements for data storage compared to the requirements for using full spectra. Recording measurement data can therefore be simpler and / or more cost-effective, or the measurement data can be recorded at a higher frequency. This can be particularly advantageous for detecting objects in the environment and / or for tracking objects in real time.

[0034] Sparse spectra can contain less data than full spectra. This reduces the storage requirements for data storage compared to using full spectra. When applying deep learning algorithms, a large data set is generally required. The proposed invention makes it easier to provide large data sets with many measurements, as less storage space is required.

[0035] Conventional storage of full spectra during a measurement may require compromises that result in information contained in the full spectra not being saved. One or more dimensions may not be saved during storage. For example, for a spectrum with the dimensions range, Doppler velocity, and azimuth angle, only two dimensions can be saved, e.g., range and Doppler velocity. Since the disclosed methods work with sparse spectra, which require less storage space, it may be possible to save more or all dimensions compared to conventional storage, thereby avoiding or reducing information loss.

[0036] It may also be possible that sparse spectra contain less data than conventional spectra, thus placing lower demands on computational capacity. The disclosed methods can lead to lower computational effort for detecting and / or tracking one or more objects in the vicinity of a device, for example, because less data needs to be processed. This can be advantageous in the development of new algorithms, as it can reduce the training effort for neural networks.

[0037] The methods may also be advantageous for devices configured for autonomous operation, such as a controller in a vehicle, which regularly requires cheaper hardware with less computing power to survive on the market, particularly with regard to production costs.

[0038] Computational effort can also be reduced compared to using full spectra, for example, by determining a sparse two-dimensional spectrum (cf. e.g. Fig. 2) Information in one or more additional dimensions, such as the azimuth angles, is determined only for the sparse spectrum. Such a method can reduce computational and / or hardware costs both during development when recording measurements for training the device using neural networks and in the finished product.

[0039] Compared to conventional methods that work with point clouds, the disclosed methods can provide more information, which enables, for example, improved detection of objects and / or improved classification of objects with possibly higher accuracy.

[0040] The methods can be used, for example, in devices in the automotive, aviation, maritime, aerospace, and / or manufacturing industries, particularly in conjunction with radar sensors. The methods are particularly suitable for use in object detection, such as semantic segmentation, or for estimating free space for environmental sensors, e.g., in the automotive sector. Use with other sensors and / or sensor systems is also possible, as described above.

[0041] Fig. Figure 1 shows an exemplary method 1000 for detecting the environment of a device using sparse spectra according to one embodiment. The method 1000 essentially illustrates a signal processing chain of a sensor that measures a time signal and determines spectra from it, from which points are extracted to detect objects.

[0042] The method comprises measuring an analog signal using a sensor or a sensor system. The method may further comprise converting the analog signal into a digital signal. In summary, the method may comprise generating 1100 a time signal based on information about the environment of a sensor or a sensor system of a device. The time signal may comprise one or more signals, e.g., one signal for each measured channel.

[0043] From the time signal, a spectrum can be determined by digital signal processing, e.g. Fourier transformations, e.g. a RADAR spectrum 2100 as in the Fig. 2. These are dense spectra, also referred to here as full spectra. In other words, the method 1000 may include determining 1200 a dense spectrum based on the time signal.

[0044] For example, noise can be filtered out from the dense spectrum. This can be done, for example, with a constant false alarm rate detector (CFAR), which estimates the strength of noise and the strength of an echo signal and retains only the data that lies above a threshold. Data that lies below the threshold is filtered out to create a sparse spectrum, such as the sparse spectrum 2200 in Fig. 2. In other words, the method 1000 may include determining 1300 a sparse spectrum based on the dense spectrum.

[0045] From the sparse spectrum, features can be determined for a selection of points, e.g., for local maxima, called RADAR detections or RADAR reflections in the case of RADAR sensors, or for each point of the sparse spectrum. In other words, the method 1000 can comprise selecting 1400 points based on the sparse spectrum. The method can comprise determining 1500 features according to a point, such as an azimuth angle, an elevation angle, and / or a RADAR cross-section. Each feature can correspond to a dimension of the dense and / or sparse spectrum.

[0046] As a result of the determination 1500, a list can be generated with one or more features corresponding to the respective point, also referred to as a point cloud. Conventional methods for detecting objects use either the point cloud based on the dense spectra or the dense spectra as input data for a deep learning algorithm. The method can include determining 1500 first features according to a point in the sparse spectrum and second features according to a point in the dense spectrum.

[0047] Finally, the method 1000 may include recognizing 1600 objects based on the determined first features and / or second features.

[0048] In the methods according to the present disclosure, sparse spectra are used as input data for an algorithm for detecting the environment of a device. The following explains by way of example how sparse spectra are determined from dense spectra and which algorithms can be used to process the sparse spectra.

[0049] Sparse spectra are determined from dense spectra. The dense spectra can have different formats. In one example, a sparse spectrum can be determined for only a portion of the input data (dimensions). The unprocessed dimensions can remain in their original format. In the case of a radar spectrum, the processed dimensions can be the range and the Doppler velocity. The dense spectra can include these two dimensions for all (virtual) antenna channels. The azimuth and elevation dimensions can be left undetermined. In other words, the generation of the azimuth angle and elevation angle dimensions for the full spectra in step 1100 can be omitted, while the dimensions remain fundamentally available.

[0050] In one example, the processed dimensions may be range, Doppler velocity, and azimuth angle. The dense spectra may include these three dimensions for all (virtual) antenna channels. In other words, generating the elevation angles for the full spectra in step 1100 can be omitted, while the dimension remains fundamentally available.

[0051] In an example, the processed dimensions may be range, Doppler velocity, azimuth angle, and elevation angle.

[0052] If not all dimensions of the dense spectra are determined, the points in the sparse spectra can be determined with reduced dimensions, and the features in the remaining dimensions can be determined based on the points. This can lead to reduced computational effort, both when creating the datasets and when using the algorithm to detect objects.

[0053] Sparse spectra can be calculated in various ways. In one example, noise can be filtered out from the dense spectrum. This can be done, for example, with a constant false alarm rate detector (CFAR), which estimates the strength of noise and the strength of an echo signal and retains only the data that lies above a threshold. Data that lies below the threshold is filtered out to create a sparse spectrum, such as the sparse spectrum 2200 in Fig. 2.

[0054] An offset can be added to the threshold. The offset can also be zero or negative. By choosing the size of the offset, you can adjust how sparse or dense the specific sparse spectrum is. All data points that lie above the threshold plus the offset are retained; the remaining points are discarded.

[0055] In an example, points in the sparse spectrum are selected as described above. A point can be a local maximum, for example. A region can be selected around each point. If the full spectrum has two dimensions, the region can be a rectangle, a circle, or an ellipse around a point. For higher dimensions, the procedure is analogous: with n input dimensions, an n-dimensional rectangle (hyperrectangle), an n-dimensional sphere, or an n-dimensional ellipsoid can be used around the point. All points that lie within these regions can be reused. Points that lie outside the regions can be filtered out and therefore not reused.

[0056] In one example, a neural network can be used to determine the sparse spectra from the dense spectra. The neural network can consist of a sequence of convolutional layers or fully connected layers, for example. However, other layers can also be used. This neural network can be trained together with the environment detection algorithm. This can have the advantage of making the determination of the sparse spectra less computationally intensive.

[0057] The sparse spectra can be represented in various formats. In one example, the sparse spectra can be represented as a sparse matrix, where each point in the sparse matrix has multiple features. For example, this can be a three-dimensional sparse matrix, where the dimensions are a range, a Doppler velocity, and a number of features. The features of the points, range and Doppler velocity, can be, for example, azimuth, elevation, and radar cross-section.

[0058] In one example, azimuth spectra can also be determined. In this example, the sparse spectrum can be represented by a four-dimensional matrix with the dimensions range, Doppler velocity, azimuth, and number of features. The features can be, for example, range and radar cross-section.

[0059] In one example, the sparse spectra can be represented as a point cloud consisting of N points. Each point can contain K features. The features can be, for example, a range, a Doppler velocity, an azimuth angle, an elevation angle, a radar cross-section, or other features. The position of the respective point in the sparse spectrum can be used to determine some of the features. In a range-Doppler velocity spectrum (cf. Fig. 2) For example, a position of the point in the spectrum can be used to determine the distance and the Doppler velocity.

[0060] For processing the sparse spectra, the algorithms described below can be used.

[0061] In one example, sparse spectra are represented as a point cloud. This allows conventional point cloud processing algorithms to be used, as described, for example, by Ulrich et al. (2022) and the works cited therein.

[0062] To process sparse spectra represented as a sparse matrix, the sparse elements can be filled with zeros to create a fully populated matrix. This allows well-known algorithms and deep learning architectures to be used to process the spectra, see Patel et al. (2019) and the work cited therein.

[0063] In one example, the sparse spectra can be processed directly. This can have the advantage of saving computing time, since only data that actually contains information is processed.

[0064] Supervised, semi-supervised, or unsupervised approaches can be used to train neural networks. If labeled data is required, automatic labels can be created using well-known methods. Measurements are recorded using additional sensors, such as a camera or LIDAR sensors in addition to radar sensors. Using this additional measurement data, labels can be created automatically. Alternatively, labels can also be created manually. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited non-patent literature

[0000] Ulrich, M., Braun, S., Köhler, D., Niederlöhner, D., Faion, F., Gläser, C. und Blume, H. in Improved Orientation Estimation and Detection with Hybrid Object Detection Networks for Automotive RADAR-, 2022 IEEE 25 th International Conference on Intelligent Transportation Systems (ITSC), arXiv:2205.02111

[0008] Patel, K., Rambach, K., Visentin, T., Rusev, D., Pfeiffer, M. und Yang, B. in Deep Learning-based Object Classification on Automotive RADAR-Spectra, 2019 IEEE RADAR-Conference (RADAR-Conf), Boston, MA, USA, 2019, pp. 1 bis 6

[0008] Ulrich et al. (2022

[0031] Patel et al. (2019

[0031]

Claims

[1] Method (1000) for detecting an environment of a device using sparse spectra, the method (1000) comprising: Generating (1100), with a sensor of the device, a time signal with information about features of an object or several objects in the environment of the device; Determining (1200) dense spectra with N dimensions based on the time signal, wherein the time signal comprises information for generating dense spectra with K dimensions, where K is greater than or equal to N; Determining (1300) sparse spectra with N dimensions based on the dense spectra with N dimensions, wherein the amount of data for representing the sparse spectra is smaller than the amount of data for representing the dense spectra with N dimensions; and Determining (1500) first features of the one or more objects for one or more points in the sparse spectra. [2] The method (1000) of claim 1, wherein the method (1000) further comprises: Selecting (1400) the one or more points in the sparse spectra. [3] The method (1000) of any one of claims 1 to 2, wherein determining (1500) first features of the one or more objects for one or more points in the sparse spectra further comprises determining second features of the object from the dense spectra of K dimensions without the dense spectra of N dimensions according to the one or more points. [4] The method (1000) of any one of claims 1 to 3, wherein the method (1000) further comprises: Recognizing (1600) the one or more objects in the environment of the device based on the first features; and / or Recognizing (1600) the one or more objects in the environment of the device based on the second features; and / or Classifying the one or more objects in the vicinity of the device; and / or semantic segmentation of the first features and / or the second features and / or the dense spectra and / or the sparse spectra; and / or; Estimating free space around the device. [5] The method (1000) of any one of claims 2 to 4, wherein K is greater than N. [6] The method (1000) of any one of claims 1 to 5, wherein determining (1300) the sparse spectra of N dimensions based on the dense spectra of N dimensions comprises neglecting data smaller than a threshold. [7] The method (1000) of any one of claims 1 to 6, wherein determining (1300) the sparse spectra of N dimensions based on the dense spectra of N dimensions comprises neglecting data outside a respective region around a point, in particular a local maximum. [8] The method (1000) of claim 7, wherein a respective region around the point or the local maximum is an N-dimensional rectangle, an N-dimensional sphere, or an N-dimensional ellipsoid. [9] Method (1000) according to any one of claims 1 to 8, wherein determining (1300) sparse spectra with N dimensions is performed using a neural network. [10] Method (1000) according to any one of claims 1 to 9, wherein the sensor of the device is a RADAR sensor, a LIDAR sensor, a SONAR sensor or an ultrasonic sensor and / or N is equal to 2 with the dimensions of distance and speed. [11] A system for detecting an environment of a device using sparse spectra, the system comprising: a processor; and a non-transitory computer-readable storage medium comprising instructions that, when executed by the processor, cause the system to perform the method (1000) of any one of claims 1 to 10. [12] Device comprising: the system according to claim 11; and one or more sensors coupled to the system.

Images