Method and system for detecting environment of device by means of sparse spectrum

Abstract

The invention discloses a device, a system and a method for detecting an environment of a device by means of sparse spectrum. The method comprises the following steps: generating a time signal by means of a sensor of the device, the time signal having information about a characteristic of an object or a plurality of objects in an environment of the device; determining a dense spectrum having N dimensions based on the time signal, where the time signal includes information for generating a dense spectrum having K dimensions, where K is greater than or equal to N; determining a sparse spectrum having N dimensions based on the dense spectrum having N dimensions, where an amount of data for representing the sparse spectrum is less than an amount of data for representing the dense spectrum having N dimensions; a first feature of an object or objects is determined for a point or points in the sparse spectrum.

Description

Technical Field

[0001] The present invention relates to a method and a system for detecting the environment of a device using sparse spectra, in particular for devices in the automotive, aviation, marine, aerospace and / or manufacturing industries. Background Art

[0002] Assistance systems for controlling or supporting control devices of an installation, such as driver assistance systems or systems enabling autonomous control of vehicles, aircraft or ships, require precise information about the installation's surroundings in order to enable reliable and safe control of the installation.

[0003] To detect information about the environment, electromagnetic radiation in different frequency ranges can be used with the aid of various information-detecting technologies, such as analog or digital photography, light detection and ranging (LIDAR) or radar technology. Alternatively or in addition, to detect information about the environment, sound waves can also be used, for example using ultrasound and / or sonar (SONAR) technology. Other technologies suitable for sampling the environment are also possible.

[0004] Sensors of the corresponding technology typically provide measurements in the form of a spectrum or a dense point cloud. For example, a radar sensor can provide a point cloud consisting of detected radar reflections. In the case of a point cloud, each point can be characterized by one or more different dimensions, such as range, azimuth, elevation, Doppler velocity, radar cross section, or a selection thereof. A radar spectrum can contain the measured radar signal and can have dimensions such as range, Doppler velocity, azimuth, and elevation, or a selection thereof.

[0005] In order to provide a precise representation of the surroundings of the device, an algorithm for detecting the surroundings processes the measurement data in a current form. The algorithm can, for example, work with radar point clouds and / or with radar spectra.

[0006] For example, a typical task of the algorithm may be to detect one or more objects and / or classify the one or more objects. The objects may be, for example, cars or traffic guidance systems, or pedestrians or animals. For example, the algorithm may provide the position, posture, speed, category, and possibly other characteristics of one or more of the detected objects in the environment. For example, the category may be the type of object, such as a vehicle or a living being.

[0007] For example, another typical task may be to estimate a trajectory for use in the environment of the device, such as the drivable area in the environment of a car or the possible flight path of an airplane in a mountain range.

[0008] These tasks can be solved with the help of deep learning methods, namely deep neural networks. D.、 D., Faion,F., C. and Blume, H., “Improved Orientation Estimation and Detectionwith Hybrid Object Detection Networks for Automotive RADAR-,” 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC), arXiv:2205.02111. A scheme for object detection based on radar point clouds is described by Patel, K., Rambach, K., Visentin, T., Rusev, D., Pfeiffer, M., and Yang, B., “Deep Learning-based Object Classification on Automotive RADAR-Spectra,” 2019 IEEE Radar Conference, Boston, MA, USA, 2019, pp. 1–6.

[0009] If a point cloud-based approach uses, for example, a radar point cloud as input data, information already detected by the radar sensor may be lost when the measured data is mapped onto the point cloud. To solve this problem, the algorithm can be provided with less information, as the mapping of the radar spectrum to the radar point cloud is generally irreversible.

[0010] Algorithms that use spectra, such as radar spectra, as input data can extract more information. Using spectra can be computationally more expensive than using point clouds, for example, because a larger amount of data must be processed. Furthermore, recording spectra can be more expensive because a large amount of data must be stored in a short period of time during the corresponding measurement. Storing spectra during a measurement may require, for example, a higher bandwidth than, for example, a point cloud, and more storage space to store the measurement data.

[0011] The above applies analogously to measurements performed with the aid of other sensors which can provide, for example, spectra and / or point clouds, such as lidar sensors or sonar sensors.

[0012] Therefore, it is desirable to provide a method and a system that can detect the environment quickly, accurately, and reliably without losing information that is relevant to the respective application. In addition, it is desirable to reduce the technical expenditure so that the measurement data can be recorded and / or processed more simply and cost-effectively. Summary of the Invention

[0013] The present invention provides a method and a system having the features according to the application for detecting an environment of a device by means of a sparse spectrum.

[0014] Preferred embodiments are corresponding alternative configurations.

[0015] The disclosed methods, systems, and devices are particularly intended to enable rapid, precise, and reliable detection of the environment without losing information that is crucial for corresponding applications, for example, in the automotive, aviation, marine, aerospace, and / or manufacturing industries. The disclosed methods, systems, and devices can have reduced technical complexity and thus enable simpler and more cost-effective recording and / or processing of measurement data.

[0016] According to a first aspect, the present invention relates to a method for detecting an environment of a device using a sparse spectrum. The method comprises: generating a time signal with a sensor of the device, the time signal containing information about a characteristic of one or more objects in the environment of the device; determining a dense spectrum with N dimensions based on the time signal, wherein the time signal contains information for generating a dense spectrum with K dimensions, wherein K is greater than or equal to N; determining a sparse spectrum with N dimensions based on the dense spectrum with N dimensions, wherein the amount of data used to represent the sparse spectrum is less than the amount of data used to represent the dense spectrum with N dimensions; and determining a first characteristic of one or more objects for one or more points in the sparse spectrum.

[0017] According to one embodiment, the method further comprises selecting one or more points in the sparse spectrum.

[0018] According to an extension, the method includes determining a first feature of an object or multiple objects for one or more points in a sparse spectrum, and further includes determining a second feature of the object based on the one or more points from a dense spectrum with K dimensions, rather than from a dense spectrum with N dimensions.

[0019] According to an extension scheme, the method also includes: identifying one or more objects in the environment of the device based on the first feature; and / or identifying one or more objects in the environment of the device based on the second feature; and / or classifying one or more objects in the environment of the device; and / or performing semantic segmentation on the first feature and / or the second feature and / or the dense spectrum and / or the sparse spectrum; and / or estimating the free space in the environment of the device.

[0020] According to one embodiment, K is greater than N.

[0021] According to an extended solution, determining a sparse spectrum with N dimensions based on a dense spectrum includes: ignoring data smaller than a threshold.

[0022] According to one embodiment, determining the sparse spectrum with N dimensions based on the dense spectrum with N dimensions includes ignoring data outside a corresponding region around the point, in particular ignoring data outside a corresponding region around a local maximum.

[0023] According to one embodiment, the corresponding region around the point or the local maximum is an N-dimensional rectangle, an N-dimensional sphere or an N-dimensional ellipsoid.

[0024] According to one embodiment, a sparse spectrum having N dimensions is determined using a neural network.

[0025] According to one embodiment, the sensor of the device is a radar sensor, a lidar sensor, a sonar sensor or an ultrasonic sensor, and / or N is equal to 2, wherein the dimensions are distance and speed.

[0026] According to a second aspect, the present invention relates to a system for detecting an environment with a sparse spectrum device. The system comprises: a processor; and a computer-readable non-volatile storage medium comprising instructions that, when executed by the processor, cause the system to perform the method described above.

[0027] According to a third aspect, the invention relates to a device comprising the above-described system and one or more sensors coupled to the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings show:

[0029] Figure 1 An exemplary method for detecting an environment of a device using a sparse spectrum according to one embodiment is shown; and

[0030] Figure 2 A schematic diagram of an exemplary spectrum and the resulting sparse spectrum using radar technology according to one specific embodiment is shown. DETAILED DESCRIPTION

[0031] In all figures, identical or functionally identical elements and devices are provided with the same reference numerals. The numbering of the method steps is for the sake of clarity and should generally not imply a specific chronological order. In particular, multiple method steps can also be performed simultaneously.

[0032] The present invention proposes using sparse spectra to solve environmental monitoring tasks, such as object detection. For example, a sparse spectra can be created by only considering data above a certain signal threshold. For example, this data could be all points above an estimated noise level. The resulting data can be interpreted as either a sparse spectra or a very dense point cloud.

[0033] A method for efficiently processing this data is disclosed. Unlike the methods cited above by Ulrich et al. (2022) and Patel et al. (2019), this method uses neither the full spectrum nor the point cloud, but rather a representation (Darstellung) that is intermediate between the full spectrum and the point cloud, which requires less storage space than the full spectrum but has a higher information content than the point cloud.

[0034] Compared with methods based on full spectrum work, the proposed method may have advantages as described below.

[0035] When recording measurement data, a sparse spectrum can be generated during the measurement. This reduces the bandwidth required for storing the data compared to when using a full spectrum. This makes recording measurement data simpler and / or more cost-effective, or allows for higher frequency recording. This can be particularly advantageous when detecting objects in an environment and / or tracking objects in real time.

[0036] Compared to full spectra, sparse spectra can contain less data. Therefore, the memory required to store the data can be reduced compared to when using full spectra. Deep learning algorithms generally require large datasets. The proposed invention makes it easier to provide large datasets using multiple measurements because less memory is required.

[0037] When storing the full spectrum during the measurement in a conventional manner, compromises may be necessary, resulting in the information contained in the full spectrum not being stored. During storage, one or more dimensions cannot be stored. For example, in a spectrum having the dimensions distance, Doppler velocity, and azimuth, only two dimensions, such as distance and Doppler velocity, can be stored. Since the disclosed method operates with sparse spectra that require less storage space, more dimensions, or all dimensions, can be stored compared to conventional storage, thereby eliminating or reducing information losses.

[0038] It is also possible that sparse spectra have a smaller amount of data than traditional spectra and therefore place lower demands on computing power. The disclosed method can result in a reduction in the computational overhead for identifying and / or tracking one or more objects in the environment of the device, since, for example, less data needs to be processed. This can be advantageous when developing new algorithms, as it can reduce the training overhead of neural networks.

[0039] Furthermore, the method is advantageous for devices configured for autonomous operation, such as control units in vehicles which generally require more cost-effective hardware with lower computing power in order to survive on the market, particularly with regard to production costs.

[0040] The computational effort can also be reduced compared to using the full spectrum by, for example, determining a two-dimensional sparse spectrum (see Figure 2 ) After that, information in one or more additional dimensions, such as the azimuth angle, is determined only for this sparse spectrum. This method can reduce the computational and / or hardware expenditure not only during development when recording measurements for teaching the device using a neural network, but also during product manufacturing.

[0041] Compared to conventional methods working with point clouds, the disclosed method can provide more information, thereby enabling, for example, improved object detection and / or improved object classification with possibly higher accuracy.

[0042] The method can be used, for example, in devices in the automotive, aviation, marine, aerospace, and / or manufacturing industries, in particular in conjunction with radar sensors. The method is particularly suitable for object recognition, such as semantic segmentation, or for estimating free space for environmental sensors, such as those in the automotive sector. As mentioned above, it can also be used with other sensors and / or sensor systems.

[0043] Figure 1An exemplary method 1000 for detecting an environment using a sparse spectrum device according to one embodiment is shown. Method 1000 essentially describes a signal processing chain of a sensor that measures a time signal, determines a spectrum from the time signal, and extracts points from the spectrum for detecting an object.

[0044] The method includes measuring an analog signal via a sensor or sensor system. Furthermore, the method may include converting the analog signal into a digital signal. In summary, the method may include generating a time signal 1100 based on information about the environment of the sensor or sensor system of the device. The time signal may include one or more signals, such as a signal for each measured channel.

[0045] From this time signal, a spectrum, for example a radar spectrum 2100, can be determined by signal processing, for example a Fourier transform. Figure 2 This relates to a dense spectrum, also referred to herein as a full spectrum. In other words, the method 1000 may include determining a dense spectrum 1200 based on the time signal.

[0046] For example, noise can be filtered out from the dense spectrum. This can be achieved, for example, with the help of a constant false alarm rate detector (CFAR), which estimates the strength of the noise and the strength of the echo signal and only retains data that is greater than a threshold. Data that is less than a threshold is filtered out to produce a sparse spectrum, such as Figure 2 In other words, the method 1000 may include determining a sparse spectrum 1300 based on the dense spectrum.

[0047] A feature can be determined from the sparse spectrum for a selected point, such as a local maximum (called a radar detection or radar reflection in the case of a radar sensor), or for each point in the sparse spectrum. In other words, method 1000 can include selecting a point based on the sparse spectrum 1400. The method can also include determining a feature from a point 1500, such as an azimuth, elevation, and / or radar cross section. Each feature can correspond to a dimension of the dense spectrum and / or a dimension of the sparse spectrum.

[0048] As a result of determination 1500, a list may be generated, the list having one or more features corresponding to corresponding points, also referred to as a point cloud. Conventional methods for identifying objects use either a dense spectrum-based point cloud or a dense spectrum as input data for a deep learning algorithm. The method may include determining a first feature based on points in a sparse spectrum and determining a second feature based on points in a dense spectrum 1500.

[0049] Finally, method 1000 may include identifying 1600 the object based on the determined first feature and / or second feature.

[0050] In the method according to the present disclosure, a sparse spectrum is used as input data for an algorithm for detecting the environment of a device. The following is an exemplary explanation and description of how to determine a sparse spectrum from a dense spectrum and which algorithms can be used to process the sparse spectrum.

[0051] The sparse spectrum is determined from the dense spectrum. The dense spectrum can have different formats. In one example, the sparse spectrum can be determined for only a portion (dimension) of the input data. The dimensions that have not yet been processed can remain in their original format. In the case of a radar spectrum, the processed dimensions can be range and Doppler velocity. For all (virtual) antenna channels, the dense spectrum can include these two dimensions. The dimensions azimuth and elevation cannot be determined. In other words, the generation of the dimensions azimuth and elevation for the full spectrum in step 1100 can be omitted, and the dimensions remain available for use in principle.

[0052] In one example, the processed dimensions may be range, Doppler velocity, and azimuth. For all (virtual) antennas, the dense spectrum may include these three dimensions. In other words, the dimension elevation generated for the full spectrum in step 1100 may be omitted, while this dimension remains available for use in principle.

[0053] In one example, the processed dimensions may be range, Doppler velocity, azimuth, and elevation.

[0054] If it is not possible to determine all dimensions of the dense spectrum, it is possible to determine points in the sparse spectrum with reduced dimensions and, based on these points, determine features in the remaining dimensions. This can lead to a reduction in the computational overhead not only when creating the dataset, but also when using the algorithm to identify objects.

[0055] The sparse spectrum can be calculated in different ways. In one example, noise can be filtered out from the dense spectrum. This can be achieved, for example, by using a detector with a constant false alarm rate, which estimates the strength of the noise and the strength of the echo signal and only retains data that is greater than a threshold. Data that is less than a threshold is filtered out to produce a sparse spectrum, for example Figure 2 Sparse spectrum 2200 in.

[0056] An offset can be added to the threshold. The offset can also be zero or negative. By selecting the size of the offset, you can adjust how sparse or dense the determined sparse spectrum is. All data points greater than the threshold plus the offset are used, and the remaining points are not used.

[0057] In one example, points as described above are selected in the sparse spectrum. For example, the points can be local maxima. A region can be selected around each point. If the full spectrum has, for example, two dimensions, the region can be, for example, a rectangle, a circle or an ellipse around a point. In the case of higher dimensions, a similar approach is used: in the case of n input dimensions, an n-dimensional rectangle (hyperrectangle), an n-dimensional sphere or an n-dimensional ellipsoid around the point can be used. All points within these regions can continue to be used. Points outside the region can be filtered out, i.e., cannot continue to be used.

[0058] In one example, a neural network can be used to determine a sparse spectrum from a dense spectrum. For example, the neural network can consist of a sequence of convolutional or fully connected layers. However, other layers can also be used. The neural network can be trained together with the algorithm used for environmental detection. This can have the advantage that the computational overhead of determining the sparse spectrum is low.

[0059] Sparse spectra can be represented in various formats. In one example, a sparse spectrum can be represented as a sparse matrix, where each point in the sparse matrix has multiple features. The sparse matrix can be, for example, a three-dimensional sparse matrix, where the dimensions are range, Doppler velocity, and number of features. The features of a point—range and Doppler velocity—can be, for example, azimuth, elevation, and radar cross section.

[0060] In one example, an azimuth spectrum can additionally be determined. In this example, the sparse spectrum can be represented by a four-dimensional matrix having the dimensions range, Doppler velocity, azimuth, and number of features. These features can be, for example, range and radar cross section.

[0061] In one example, the sparse spectrum can be represented in the form of a point cloud consisting of N points. Here, each point can include K features. The features can be, for example, range, Doppler velocity, azimuth, elevation, radar cross section, or other features. The position of the corresponding point in the sparse spectrum can be used to determine a portion of the features. For example, in the range-Doppler velocity spectrum (see Figure 2 ), the position of a point in the spectrum can be used to determine the range and Doppler velocity.

[0062] For example, the algorithm described below can be used to process sparse spectra.

[0063] In one example, the sparse spectrum can be represented in the form of a point cloud. Therefore, conventional algorithms for processing point clouds can be used, such as those described by Ulrich et al. (2022) and the papers cited therein.

[0064] To process a sparse spectrum represented in the form of a sparse matrix, the sparse spectrum can be padded with zeros so that a full matrix is generated. This allows the use of known algorithms and deep learning architectures for processing the spectrum, see Patel et al. (2019) and the papers cited therein.

[0065] In one example, the sparse spectrum can be processed directly. This can have the following advantages: computation time can be saved because only data containing information is processed.

[0066] To train a neural network, supervised, semi-supervised, or unsupervised approaches can be used. If labeled data is required, automatic labeling can be performed using known methods. Measurements are recorded using additional sensors, such as cameras or lidar sensors in addition to radar sensors. These additional measurement data can be used to automatically generate the labels. Alternatively, manual labeling is also possible.

Claims

1. A method (1000) for detecting an environment of a device by means of a sparse spectrum, wherein: The method (1000) comprises: generating (1100) a time signal with the aid of a sensor of the device, the time signal containing information about a characteristic of one or more objects in the environment of the device; determining (1200) a dense spectrum having N dimensions based on the time signal, wherein the time signal includes information for generating a dense spectrum having K dimensions, wherein K is greater than or equal to N; determining (1300) a sparse spectrum having N dimensions based on a dense spectrum having N dimensions, wherein an amount of data used to represent the sparse spectrum is less than an amount of data used to represent the dense spectrum having N dimensions; and A first feature of the object or the objects is determined (1500) for a point or points in the sparse spectrum.

2. The method (1000) according to claim 1, wherein: The method (1000) further comprises: A point or points in the sparse spectrum are selected (1400).

3. The method (1000) according to any one of claims 1 to 2, wherein: Determining (1500) a first feature of the object or the objects for one or more points in the sparse spectrum also includes: determining a second feature of the object based on the one or more points by a dense spectrum with K dimensions instead of a dense spectrum with N dimensions.

4. The method (1000) according to any one of claims 1 to 3, wherein: The method (1000) further comprises: Identifying (1600) an object or objects in the environment of the device based on the first feature; and / or Identifying (1600) an object or objects in the environment of the device based on the second feature; and / or classifying an object or objects in the environment of the device; and / or performing semantic segmentation on the first feature and / or the second feature and / or the dense spectrum and / or the sparse spectrum; and / or Estimate the free space in the environment of the device.

5. The method (1000) according to any one of claims 2 to 4, wherein: K is greater than N.

6. The method (1000) according to any one of claims 1 to 5, wherein: Determining (1300) a sparse spectrum having N dimensions based on a dense spectrum having N dimensions includes ignoring data that is smaller than a threshold value.

7. The method (1000) according to any one of claims 1 to 6, wherein: Determining (1300) a sparse spectrum having N dimensions based on a dense spectrum having N dimensions includes ignoring data outside a corresponding region around a point, in particular ignoring data outside a corresponding region around a local maximum.

8. The method (1000) according to claim 7, wherein: The corresponding region around the point or the local maximum is an N-dimensional rectangle, an N-dimensional sphere or an N-dimensional ellipsoid.

9. The method (1000) according to any one of claims 1 to 8, wherein: A sparse spectrum having N dimensions is determined (1300) using a neural network.

10. The method (1000) according to any one of claims 1 to 9, wherein: The sensor of the device is a radar sensor, a lidar sensor, a sonar sensor or an ultrasonic sensor, and / or N is equal to 2, where the dimensions are distance and speed.

11. A system for detecting an environment of a device by means of a sparse spectrum, wherein: The system comprises: processor; and A computer-readable non-volatile storage medium comprising instructions which, when executed by the processor, cause the system to perform the method (1000) according to any one of claims 1 to 10.

12. A device comprising: The system according to claim 11; and One or more sensors coupled to the system.

Images