Ultrasound and audio-based sensor system

Abstract

The present invention relates to a sensor system for classifying environmental objects for driver assistance systems, comprising an ultrasonic sensor unit configured to emit ultrasonic signals and to receive ultrasonic signals reflected from environmental objects, an audio transmitter unit configured to emit audio signals, an audio receiver unit configured to receive audio signals reflected from environmental objects, and a computing unit, wherein the computing unit is configured to classify environmental objects based on the reflected ultrasonic signal (12) and the reflected audio signal.

Description

State of the art

[0001] The present invention relates to a sensor system for classifying environmental objects for driver assistance systems and an associated method. A further aspect of the invention relates to a vehicle equipped with the sensor system.

[0002] Ultrasonic sensors are used in automotive and industrial applications for distance measurement and environmental sensing, measuring the travel time of sound from emission to return. Using the known speed of sound propagation, the distance to the backscattering object can be calculated. Although backscattered sound is inherently a time-dependent quantity, the full time signal is not utilized, primarily for cost reasons and to simplify data acquisition, storage, and transmission. Typically, only echoes from objects with amplitudes above a defined threshold are considered relevant and further analyzed. Distance estimation can be performed relatively robustly with such ultrasonic systems. Other tasks, such as...The sensory differentiation of objects or the determination of object dimensions can only be implemented in a rudimentary way with current technology. This information is particularly relevant and applicable for complex or safety-critical driving functions, such as highly or fully automated driving. It would be desirable to have a sensor system for a vehicle that, especially with regard to object classification of obstacles around the vehicle, enables an increase in the performance of environmental sensing. Disclosure of the invention

[0003] The sensor system according to the invention for classifying environmental objects for driver assistance systems, comprising the features of claims 1 and 13, and the method for classifying environmental objects, comprising the features of claim 6, have the advantage that the inclusion of audio signals enables a powerful and robust classification of objects in the immediate vicinity of the vehicle. According to the invention, this is achieved by the sensor system comprising an ultrasonic sensor unit configured to emit ultrasonic signals and to receive ultrasonic signals reflected from environmental objects. Furthermore, the sensor system comprises an audio transmitter unit and an audio receiver unit. The audio transmitter unit is configured to emit audio signals, and the audio receiver unit is configured to receive audio signals reflected from environmental objects.The sensor system also includes a processing unit configured to classify environmental objects based on the reflected ultrasound signal and the reflected audio signal. The received audio signal and / or the received ultrasound signal is preferably a sound-time signal. The additional acquisition and processing of reflected audio signals offers advantages over the use of ultrasound signals alone. Due to the different wavelengths in the audio and ultrasound signals, different object properties are focused in the backscatter of an object. On the one hand, in the ultrasound range with very short wavelengths (approx. 5 to 10 mm), even very small backscattered areas can produce effective reflection, allowing even slight differences in object geometry to be resolved in the reflected ultrasound signal. In the reflected audio signal (wavelength approx.In contrast, the larger object geometries (17 to 340 mm) are the focus. Furthermore, the use of audio signals can enable higher signal-to-noise ratios, which naturally result in lower airborne sound attenuation at lower frequencies. This allows, for example, more distant objects to be detected sufficiently well for classification purposes. Evaluating a separate frequency range also allows for the derivation of more information about the object's material and texture due to the frequency-dependent backscattering properties of many objects. Detecting object-specific resonances in the audio range can also expand the information content of the reflected audio signal. Classification can then be performed, for example, into obstacle classes.

[0004] The dependent claims describe preferred embodiments of the invention.

[0005] Preferably, the ultrasound signal has a frequency greater than or equal to 40 kHz. In particular, the frequency of the ultrasound signal is between 40 kHz and 70 kHz. The audio signal preferably has a frequency between 1 kHz and 20 kHz, particularly between 5 kHz and 20 kHz. The claimed frequency range can enable a high level of information acquisition from environmental objects by the sensor system and contribute to a reliable and accurate classification of these objects.

[0006] Preferred options for use in the audio transmitter unit include an Acoustic Vehicle Alert System (AVAS), a reversing alarm, a task-specific loudspeaker, a horn, and / or an interior loudspeaker. These or other audio-emitting systems can be used individually or in any combination as an audio transmitter unit. An Acoustic Vehicle Alert System (AVAS) is an acoustic warning system for low-noise vehicles, especially electric cars. The aforementioned audio transmitter units are suitable for integrating transient or chirp-like signal components, which can be reliably extracted and evaluated from the reflected audio signal using appropriate signal processing. The horn can be adapted to generate quieter audio signals for environmental sensing. Interior loudspeakers can be used for the sensor system if significant external noise is generated for environmental sensing.

[0007] The processing unit preferably incorporates a neural network, in particular a convolutional neural network, to classify environmental objects based on the reflected ultrasound signal and the reflected audio signal. Neural networks can efficiently detect complex features in the received audio and ultrasound signals and contribute to a fast and reliable classification of environmental objects.

[0008] The audio receiver and ultrasonic sensor units are preferably arranged together in a common housing. This allows for a compact design of the sensor system. Preferably, the audio receiver and ultrasonic sensor units are aligned in the same direction within the housing, which improves the detection of spatial information.

[0009] Furthermore, the invention relates to a method for classifying environmental objects for driver assistance systems. The method comprises the steps of emitting an ultrasonic signal and an audio signal. Subsequently, the ultrasonic signal and the audio signal reflected from an environmental object are received. The received ultrasonic signal is then classified in an ultrasonic classification output. The received audio signal is also classified in an audio classification output. Finally, the ultrasonic classification output and the audio classification output are combined and evaluated to produce a classification result. The described steps can enable efficient object classification, for example, into obstacle classes.The transmission, reception, and classification of the audio signal is preferably used in conjunction with the ultrasound signal and does not need to be continuous. Separate processing and classification of the audio and ultrasound signals take place. If an audio classification output and an ultrasound classification output are available, a robust, combined classification result can be derived. The ultrasound signal and the audio signal can be taken as analog signals directly from the output of an electrical amplification circuit after a piezoelectric element. Using an analog-to-digital converter, preferably high-resolution digital sound-time signals can be generated from the received ultrasound signals and / or audio signals.

[0010] Preferably, the received ultrasound signal and / or the received audio signal is filtered, in particular by means of a bandpass filter and / or a matched filter. This can improve the signal-to-noise ratio or suppress extraneous noise. The filtering can take place before the analog-to-digital conversion or, particularly advantageously, after the digitization of the time signals.

[0011] Preferably, the received ultrasound signal and / or the received audio signal is trimmed, in particular using a sliding window technique. This can enable data reduction. During the trimming process, the regions in the time signal are preferably separated by presuming the presence of objects.

[0012] The received ultrasound signal and / or audio signal are preferably normalized. Normalization is the process of increasing or decreasing the amplitudes of the analog or digital ultrasound and / or audio signal so that they lie within a predetermined range. Preferably, the ultrasound and / or audio signals are normalized with defined values ​​for standard deviation and mean. This allows the amplitude ratios between the signals to be preserved and the values ​​to be shifted into advantageous ranges for neural networks. Preferably, the normalization is combined with feature extraction, in particular a time-frequency transformation, such as a short-time Fourier transform or a wavelet transform. This can contribute to increased robustness in the classification of environmental objects.

[0013] Preferably, the classification of the ultrasound signal and / or the audio signal is performed using a machine learning model, in particular a convolutional neural network (CNN). A 1D CNN is particularly suitable for directly processing the time signals, while 2D CNNs are suitable for processing time-frequency representations. The ultrasound signal is preferably classified using an ultrasound network, and the audio signal is classified using an audio network. The audio classification output and / or the ultrasound classification output of the machine learning model can be class-specific softmax probabilities or classification embeddings. Machine learning models can enable efficient classification of environmental objects.

[0014] The ultrasound signal and / or the audio signal are preferably enhanced with an additional feature prior to classification, in particular a distance to the environmental object, a signal-to-noise ratio, a sound energy, and / or a spectral feature. These additional features can contribute to increased accuracy in the classification of environmental objects.

[0015] Preferably, the transmitted audio signal has a transient and / or chirp-like signal component. This can enable improved extraction of features from the received audio signal, particularly using a matched filter.

[0016] The invention also relates to a vehicle comprising a driver assistance system and a previously described sensor system, wherein the sensor system is configured to transmit classification results of environmental objects to the driver assistance system, for example, regarding obstacle classes of the environmental objects. This allows the assignment of a specific obstacle to an obstacle class to be determined with a high degree of certainty. Possible obstacle classes include: traversable objects, non-traversable objects, and people. It is also possible to make further subdivisions into more classes. This allows, for example, the addition of classes such as curbs, poles, small objects on the road, or speed bumps. These classification results can improve the driver assistance system of a vehicle and contribute to greater safety. Brief description of the drawings

[0017] A preferred embodiment of the invention is described in detail below with reference to the accompanying drawings. The drawing shows: Fig. 1 a schematic representation of a vehicle with a sensor system according to a preferred embodiment of the invention and Fig. 2 a flowchart of a method for classifying environmental objects using a sensor system according to the preferred embodiment of the invention. Embodiments of the invention

[0018] Preferably, all identical components, elements and / or units in all figures are provided with the same reference numerals.

[0019] The following refers to the Fig. 1 and Fig. 2 a vehicle 100 with a sensor system 1 for classifying environmental objects 2 and a method for classifying environmental objects 2 according to a preferred embodiment of the invention is described in detail.

[0020] Fig. Figure 1 shows the vehicle 100 with a driver assistance system 40 and the sensor system 1, wherein the sensor system 1 is configured to transmit classification results 4 of environmental objects 2 to the driver assistance system 40.

[0021] The sensor system 1 comprises an ultrasonic sensor unit 10, an audio transmitter unit 20, an audio receiver unit 23, and a processing unit 30. The audio receiver unit 23, the ultrasonic sensor unit 10, and the processing unit 30 are arranged on a common housing 3. The sensor system is preferably arranged on a bumper of the vehicle 100.

[0022] The ultrasonic sensor unit 10 is configured to emit ultrasonic signals 11. The ultrasonic signals 11 are reflected by surrounding objects 2 and can also be received and detected by the ultrasonic sensor unit 10.

[0023] The audio transmitter 20 is configured to transmit audio signals. When an audio signal encounters an environmental object 2, it is reflected by the object and can be received by the audio receiver 23. The received ultrasound signal 12 and the received audio signal 22 are preferably in the form of digitized time signals.

[0024] The processing unit 30 can classify environmental objects 2 based on the received reflected ultrasound signals 12 and reflected audio signals 22. The environmental object 2 is in Fig. Figure 1 shows a schematic representation and arranges the object on a roadway. The surrounding object 2 can be, for example, a curb, a pole, a small object on the road, or a speed bump. The surrounding object 2 can be classified into different obstacle classes, such as traversable objects, non-traversable objects, and people.

[0025] The emitted ultrasound signal 11 preferably has a frequency between 40 kHz and 70 kHz. The emitted audio signal 21 preferably has a frequency between 5 kHz and 20 kHz.

[0026] The audio transmitter 20 can function as an acoustic vehicle alert system, reversing alarm, task-specific loudspeaker, horn, and / or interior loudspeaker. The audio transmitter 20 can preferably integrate transient or chirp-like signal components into the transmitted audio signal, which can later be extracted and evaluated from the received audio signal 22 using suitable signal processing.

[0027] The computing unit 30 preferably comprises a neural network with which environmental objects can be classified based on the reflected ultrasound signal 12 and the reflected audio signal 22. The classification, for example an obstacle class of the environmental objects 2, can be forwarded to the driver assistance system 40. The driver assistance system 40 can make a decision based on the classification result 4 to ensure the safety of the vehicle 100.

[0028] Fig. Figure 2 schematically represents a method for classifying environmental objects. The received ultrasound signals 12 and the received audio signals 22 are first processed separately and then combined to produce a classification result 4.

[0029] First, in step S1.1, an ultrasonic sensor unit 10 emits an ultrasonic signal 11. Parallel to or slightly offset from this, in step S1.2, an audio transmitter unit 20 emits an audio signal 21. The ultrasonic signal 11 and the audio signal 21 propagate away from the vehicle 100 until they encounter the surrounding object 2 and are reflected by it.

[0030] The ultrasound signal 12 reflected from the surrounding object 2 is received by the ultrasound sensor unit 10 in step S2.1. The ultrasound signal is preferably received by a piezoelectric element in the ultrasound sensor unit 10 in the form of an analog time signal, which is then preferably converted into a digital time signal by means of an analog-to-digital converter.

[0031] In step S2.2, the reflected audio signal 21 is received by an audio receiver 23, either simultaneously or slightly offset in time. The received audio signal 22 is preferably also converted into a digital time signal.

[0032] The received ultrasound signal 12 is then filtered in step S3.1. A bandpass filter is preferably used to suppress extraneous noise. Alternatively or additionally, suitable high-pass or low-pass filters can be used.

[0033] Furthermore, in step S3.2, the received audio signal 22 is filtered. Here too, the filter can include high-pass, low-pass, or, particularly advantageously, band-pass filters.

[0034] To further improve the signal-to-noise ratio of the received ultrasound signal 12 and / or the received audio signal 22, matched filters can be used. These are particularly advantageous for the received audio signals 22 for further processing in a classifier, since the transmitted audio signals 21 can differ significantly from the received audio signals 22.

[0035] In step S4.1, the filtered ultrasound signal 12 is trimmed, and in a parallel step S4.2, the received audio signal 22 is trimmed. This allows relevant and time-limited time segments to be extracted from the audio signal 22 and the ultrasound signal 12. This serves, among other things, to reduce data. The extraction can be performed based on the available echo propagation time data (the time signal is only further evaluated at points with detected echoes) or via a sliding window method (the complete time signal is evaluated segment by segment). The sliding window method is particularly preferable if no temporal limitation of the object segment can be determined. This is especially the case for the received audio signal 22.

[0036] The tailored ultrasound signal 12 is then normalized in step S5.1. Furthermore, the tailored audio signal 22 is normalized in parallel in step S5.2. Specifically, the ultrasound and audio signals are normalized using defined values ​​for standard deviation and mean. This is intended to preserve the amplitude ratios between the ultrasound and audio signals and to ensure that the values ​​are transmitted within a range advantageous for neural networks.

[0037] Preferably, a time-frequency transformation is additionally applied to the ultrasound signal 12 and / or the received audio signal 22 for feature extraction. This can contribute to increasing the robustness of the method.

[0038] The processing of the ultrasound signal 22 takes place in step S6.1, and the processing of the received audio signals 22 in step S6.2, using a machine learning model. In step S6.1, an ultrasound network provides an ultrasound classification output, and in step S6.2, an audio network provides an audio classification output. The classification outputs can be class-specific softmax probabilities or classification embeddings.

[0039] The machine learning model is preferably a Convolutional Neural Network (CNN), in particular a 1D-CNN for processing time signals and a 2D-CNN for processing time frequency representations.

[0040] The ultrasound signal 12 and the audio signal 22 are enriched with an additional feature 5 before classification in steps S6.1 and S6.2. The additional feature 5 is based on the received raw signal of the ultrasound signal 12 and the audio signal 22 and includes, for example, a distance to the environmental object 2, a signal-to-noise ratio, a sound energy and / or a spectral feature.

[0041] In step S7, the ultrasound classification output and the audio classification output are combined and evaluated to produce a classification result 4. Preferably, the audio classification output and the ultrasound classification output are synchronized by assigning distance values. The distances can be calculated using the speed of sound and sound travel times.

[0042] Based on calculated signal-to-noise ratios and / or correlations with the emitted ultrasound signal 11 and emitted audio signal 21, the audio classification output and the ultrasound classification output can be weighted. The evaluation of the ultrasound classification output and the audio classification output is preferably performed via an additional neural network that can process the input values ​​to determine a classification result 4.

[0043] This allows the classification of an environmental object 2, for example into an obstacle class, to be determined with a high degree of certainty. This can increase the performance of environmental sensing in the immediate vicinity of the vehicle 100 and improve driver assistance systems 40.

Claims

[1] Sensor system for classifying environmental objects (2) for driver assistance systems (40), comprising - an ultrasonic sensor unit (10) configured to emit ultrasonic signals (11) and to receive ultrasonic signals (12) reflected from surrounding objects (2), - an audio transmitter unit (20) configured to transmit audio signals (21), - an audio receiving unit (23) configured to receive audio signals (22) reflected from surrounding objects (2), and - one computing unit (30), - wherein the computing unit (30) is set up to classify environmental objects (2) based on the reflected ultrasound signal (12) and the reflected audio signal (22). [2] Sensor system according to claim 1, wherein the ultrasound signal (11) has a frequency greater than or equal to 40 kHz, in particular between 40 kHz and 70 kHz, and wherein the audio signal (21) has a frequency between 1 kHz and 20 kHz, in particular between 5 kHz and 20 kHz. [3] Sensor system according to one of the preceding claims, wherein the audio transmitter unit (20) is an Acoustic Vehicle Alert System, a reversing warning system, a task-specific loudspeaker, a horn and / or an interior loudspeaker. [4] Sensor system according to one of the preceding claims, wherein the computing unit (30) comprises a neural network, in particular a Convolutional Neural Network, to classify environmental objects (2) based on the reflected ultrasound signal (12) and the reflected audio signal (22). [5] Sensor system according to one of the preceding claims, wherein the audio receiver unit (23) and the ultrasonic sensor unit (10) are arranged in a common housing (3). [6] Method for classifying environmental objects (2) for driver assistance systems, comprising the steps: - Emitting (S1.1) an ultrasound signal (11), - Emitting (S1.2) an audio signal (21), - Receiving (S2.1) the ultrasound signal (12) reflected from the surrounding object (2), - Receiving (S2.2) the audio signal (22) reflected from the surrounding object, - Classification (S6.1) of the received ultrasound signal (12) into an ultrasound classification output, - Classification (S6.2) of the received audio signal (22) into an audio classification output and - Combining (S7) the ultrasound classification output and the audio classification output to produce a classification result (4). [7] Method according to claim 6, wherein the received ultrasound signal (12) and / or the received audio signal (22) is filtered (S3.1, S3.2), in particular by means of a bandpass filter and / or a matched filter. [8] Method according to one of claims 6 or 7, wherein the received ultrasound signal (12) and / or the received audio signal (22) is cut (S4.1, S4.2), in particular by means of a sliding window method. [9] Method according to any one of claims 6 to 8, wherein the received ultrasound signal (12) and / or the received audio signal (22) is normalized (S5.1, S5.2). [10] Method according to any one of claims 6 to 9, wherein the classification of the ultrasound signal (S6.1) and / or the audio signal (S6.2) is carried out using a machine learning model, in particular by means of a Convolutional Neural Network. [11] Method according to one of claims 6 to 10, wherein the ultrasound signal (12) and / or the audio signal (22) is enhanced with an additional feature (5) prior to classification (S6.1, S6.2), in particular a distance to the surrounding object (2), a signal-to-noise ratio, a sound energy and / or a spectral feature. [12] Method according to any one of claims 6 to 11, wherein the emitted audio signal (21) has a transient and / or chirp-like signal component. [13] Vehicle comprising a driver assistance system (40) and a sensor system (1) according to any one of claims 1 to 5, wherein the sensor system (1) is configured to transmit classification results (4) of environmental objects (2) to the driver assistance system (40).

Images