Method for classifying an object to be detected, using at least one ultrasonic sensor taking into account phase information of the acquired signal

Abstract

The invention relates to a method (100) for classifying at least one object (10) to be detected, using at least one ultrasonic sensor (50). This involves determining (120) object class information (12) for the at least one object (10) using a neural network (9), wherein the neural network (9), on the basis of at least one first input variable (11), generates an output variable (15) indicating a probability value for at least one defined object class for the one object (10).

Description

[0001] Description

[0002] Title:

[0003] Method for classifying an object to be detected with at least one ultrasonic sensor taking into account phase information of the detected signal

[0004] The present invention relates to a method for classifying an object to be detected using at least one ultrasonic sensor, taking into account phase information of the detected signal.

[0005] State of the art

[0006] Ultrasonic sensors are used in automotive and industrial applications to determine distances, for example in the context of parking assistance systems in a vehicle, or more generally to detect objects in an environment

[0007] Ultrasonic sensors operate according to the well-known pulse-echo principle. An electrical signal excites a transducer on its membrane to vibrate, which is emitted as sound. The sound propagates through the air until it encounters an object in the surrounding area. The surface of the object reflects the sound, causing it to be backscattered toward the ultrasonic sensor. When the backscattered sound hits the membrane, it causes it to vibrate, generating an electrical signal at the piezo element.

[0008] State-of-the-art ultrasonic sensors today measure the travel time of sound from its emission to its return. Using the known speed of sound propagation, the distance to the backscattering object can be determined. Although the physical nature of backscattered sound pressure is a quantity with a time characteristic, the time signal has not been fully utilized to date, particularly for reasons of cost and ease of data acquisition, storage, and transmission. After several signal preprocessing steps, such as filtering, individual amplitude values ​​and associated correlation values ​​are generated using thresholding methods using simple or more complex and adaptive methods. These correlation values ​​represent the large values ​​in the sound pressure-time signal. An echo from an object is therefore detected by comparing the received amplitude of the sound with a threshold value.

[0009] Typically, only those echoes whose amplitude is above the threshold are considered relevant and further evaluated.

[0010] State-of-the-art ultrasonic sensors transmit only these few amplitude or correlation values, often referred to as echo values. This compromises the accuracy of sensory discrimination of objects or the determination of object dimensions.

[0011] Furthermore, it should be noted that current approaches for distinguishing object classes have so far only considered amplitude-based signal features. An example of such a state-of-the-art approach is the publication by Jona Eisele, Andre Gerlach, Marcus Maeder, and Steffen Marburg (2023): "Convolutional neural network with data augmentation for object classification in automotive ultrasonic sensing." J.

[0012] Acoust. Soc. On. April 1, 2023; 153 (4): 2447, https : / / doi. org / 10. 1121 / 10. called 0017922.

[0013] It is therefore the object of the present invention to provide a solution by means of which the performance of an ultrasonic sensor and thus a detection of objects in an environment of a vehicle with an ultrasonic sensor can be improved in an efficient and reliable manner.

[0014] Disclosure of the invention This object is achieved by a method for classifying an object to be detected with at least one ultrasonic sensor having the features of the independent claim.

[0015] According to a first aspect, the disclosure relates to a method for classifying at least one object to be detected using at least one ultrasonic sensor, comprising the following steps:

[0016] In a first step, a first signal is sent by the at least one ultrasonic sensor to the object.

[0017] In a second step, a second signal is received by the at least one ultrasonic sensor, wherein the second signal represents a signal scattered back from the object.

[0018] In a third step, the second signal is processed into a first digital signal.

[0019] In a fourth step, the first digital signal is converted into a complex-valued signal representation based on a time-frequency transformation or a modulation method in order to obtain a second digital signal.

[0020] In a fifth step, the second digital signal is separated into a first feature vector and / or at least one second feature vector, wherein the first feature vector contains a number of amplitude values ​​of the second digital signal and the at least second feature vector contains a number of phase values ​​of the second digital signal.

[0021] In a sixth step, a discontinuous range of values ​​of the phase of the second feature vector is converted into a continuous curve (= phase unwrapping). In a seventh step, a gradient is determined over the phase curve in the time or frequency direction.

[0022] In an eighth step, a third feature vector is created based on the determined gradient.

[0023] In a ninth step, at least the third feature vector is fed into a neural network as the first input variable.

[0024] In a tenth step, object class information for the at least one object is determined using the neural network, wherein the neural network generates an output variable on the basis of the at least first input variable, which output variable indicates a probability value for at least one defined object class for the one object.

[0025] A fundamental concept of the present invention is that an ultrasonic sensor not only determines distances between the sensor and objects, but also uses the acquired sensor data to classify objects. Classification involves determining the type or characteristic properties of objects from backscattered ultrasonic signals.

[0026] Based on the classification, groups of objects, so-called object classes, can be identified by sensors, rather than just a single individual or representative of a detected object. Typical object classes can include, for example, people, poles, trees, curbs, small objects, manhole covers, etc. Aggregated classes such as drivable / non-driftable can also be classified.

[0027] Knowing the vehicle's class affiliation advantageously opens up many possibilities for more efficient driving functions. For example, it is very helpful to distinguish between static and moving objects, or between drivable and non-drivable objects. From this knowledge, additional information can be derived directly or indirectly, or in combination with other information or sensors, to support the driving function. This information is particularly relevant and applicable for complex or safety-relevant driving functions such as higher or fully automated driving.

[0028] The present invention enables the performance of the ultrasonic sensor to be increased. This is achieved in particular by not only using amplitude-based features in the processing of the sensor data, as previously known in the prior art, but also by taking phase information into account for classification or to increase the robustness of the classification. This efficiently generates more robust object classification models.

[0029] Of key importance for the present invention is that the phase contains precise timing information, which allows for more accurate temporal analysis of the object-specific individual reflections overlapping within a backscatter pattern. The corresponding spacing of the individual reflections provides information about the geometric properties of an object. Using phase information leads to improved detection rates, especially in noisy signals.

[0030] For example, in speech recognition, phase is relevant for increased robustness at low signal-to-noise ratios. The phase of an echo also contains material-dependent absorption properties. Furthermore, circumferential waves (= creeping waves) in object backscattering lead to interference, depending on the shape and size of an object.

[0031] Such circumferential waves are important for numerous typical objects in the application area of ​​vehicle environment sensing discussed here, e.g., poles (traffic signs), trees, parking posts, and bicycle racks. The object-specific phase shifts resulting from interference are particularly clearly visible in the phase derivative.

[0032] One embodiment of the method provides that the step of processing the second signal into a first digital signal includes filtering the first digital signal. This improves the S / N ratio of the first digital signal.

[0033] One embodiment of the method provides for the modulation method to be implemented as an in-phase and quadrature (I&Q) modulation. This achieves an efficient and application-dependent conversion of the first digital signal into the complex-valued signal representation, which is mapped by at least one one-dimensional feature vector.

[0034] One embodiment of the method provides for the complex-valued signal representation to be represented as a time-frequency representation. This enables, in particular, frequency-related feature extraction in a two-dimensional feature vector.

[0035] One refinement of the method involves feeding the first feature vector into the neural network as a second input. This further improves the object classification result.

[0036] According to a second aspect, the disclosure relates to a detection system for classifying an object to be detected, comprising at least one ultrasonic sensor which is designed to carry out the method according to the invention, wherein the at least one ultrasonic sensor can be used in a vehicle.

[0037] According to a third aspect, the disclosure relates to a computer program containing machine-readable instructions which, when executed on one or more computers and / or computer instances, cause the computer or computer instances to carry out the method according to the invention.

[0038] According to a fourth aspect, the disclosure relates to a machine-readable data carrier and / or downloadable product with the computer program. According to a fifth aspect, the disclosure relates to one or more computers and / or computer instances with the computer program, and / or with the machine-readable data carrier and / or the downloadable product.

[0039] Further measures improving the invention are described in more detail below together with the description of the preferred embodiments of the invention with reference to figures

[0040] Examples of implementation

[0041] It shows:

[0042] Figure 1 Schematic flow diagram of the method 100 for classifying an object 10 to be detected with at least one ultrasonic sensor 50;

[0043] Figure 2 Schematic representation of a detection system 200 for carrying out the method 100 according to an embodiment of the present invention, and

[0044] Figure 3 Schematic representation of object classification by means of a neural network 9 using the method 100 according to an embodiment of the present invention.

[0045] Figure 1 shows a schematic flow diagram of the method 100 for classifying at least one object 10 to be detected with at least one ultrasonic sensor 50.

[0046] In step 102, a first signal 1 is sent by the at least one ultrasonic sensor 50 to the object 10.

[0047] In step 104, a second signal 2 is received by the at least one ultrasonic sensor 50, wherein the second signal 2 represents a signal backscattered by the object 10. In step 106, the second signal 2 is processed into a first digital signal 3. Optionally, the step of processing the second signal 2 into a first digital signal 3 can include filtering the first digital signal 3 to improve an S / N ratio.

[0048] In step 108, the first digital signal 3 is converted 108 into a complex-valued signal representation 20 based on a time-frequency transformation or a modulation method in order to obtain a second digital signal 4. Optionally, the complex-valued signal representation is depicted as a time-frequency representation.

[0049] Optionally, the modulation method is designed as an in-phase & quadrature (I&Q) modulation.

[0050] In step 110, the second digital signal 4 is separated into a first feature vector 5 and / or into at least one second feature vector 6, wherein the first feature vector 5 contains a number of amplitude values ​​21 of the second digital signal 4 and the at least second feature vector 6 contains a number of phase values ​​23 of the second digital signal 4. It is therefore also conceivable according to the invention that only the at least second feature vector 6 is separated or extracted from the second digital signal 4.

[0051] In step 112, a transfer 112 of a discontinuous value range of the phase 23 of the second feature vector 6 into a continuous curve 24 takes place.

[0052] In step 114, a gradient 8 is determined over the course of the phase 24, preferably in the time or frequency direction.

[0053] In step 116, a third feature vector 7 is created based on the determined gradient 8.

[0054] In step 118, the at least third feature vector 7 is fed 118 into a neural network 9 as a first input variable 11. In step 120, an object class information 12 for the at least one object 10 is determined 120 using the neural network 9, wherein the neural network 9 generates an output variable 15 on the basis of the at least first input variable 11, which output variable indicates a probability value for at least one defined object class for the one object 10.

[0055] Optionally, in addition to feeding the first input variable 11 into the neural network 9, the first feature vector 5 is also fed into the neural network 9 as a second input variable 12. This can improve the accuracy of the probability value to be determined for at least one defined object class for the one object.

[0056] Figure 2 shows a schematic representation of a detection system 200 for implementing method 100 according to an embodiment of the present invention. Detection system 200 includes an ultrasonic sensor 50 that transmits a first signal 1 to the object 10 to be detected and classified. This first signal 1 is reflected by the object 10 and reaches the ultrasonic sensor 50 in the form of the second signal 2, which is processed in the manner described above according to the inventive method 100 to classify the object 10.

[0057] In general, it should be said in this context that the ultrasonic time signal (= raw signal) backscattered by the object 10 and detected by the sensor is used.

[0058] Figure 3 shows a schematic representation of object classification using a neural network 9 using method 100 according to an exemplary embodiment of the present invention. A first input variable 11 and / or (at least) a second input variable 12 are fed to the neural network 9.

[0059] These two input variables 11, 12 are processed by the neural network 9 according to the inventive method 100 as explained above, so that an output variable 15 is output by the neural network 9, which represents object class information 12 for the detected object 10.

[0060] It is therefore conceivable that only one of the two input variables 11, 12 is fed to the neural network 9. For example, only the input variable containing at least the second feature vector 6 with the corresponding phase information of the second digital signal 4 could be fed to the neural network 9.

[0061] The neural network 9 can be a component of the ultrasonic sensor 50, but it can also be located outside the ultrasonic sensor 50, for example as part of a cloud- or network-based connection or an external connection to a data processing device, such as a control unit in a vehicle.

[0062] Further aspects and embodiments of the present invention are explained in detail below.

[0063] In summary, and as already explained at the beginning, according to the invention, the ultrasonic time signals (raw signals) backscattered by at least one object and detected by sensors are processed, including their phase information.

[0064] This enables a particularly robust classification of object classes through the combination of the following steps: a. Acquisition of the raw time signal at the transducer element of the ultrasonic sensor b. Transformation into a complex-valued signal representation using a modulation method or time-frequency representation c. Separate extraction of the amplitude and phase values ​​d. Unwrapping (= transfer as shown in step 112 of the method) and derivation of the phase values ​​in the time and / or frequency direction e. Input of the processed amplitude and phase feature vector sequences for classification into a neural network (NN)

[0065] At the end of the procedure, the probabilities of assignment to one of the object classes are available as the output variables of the NN. This parameter achieves a higher statistical certainty in the assignment to an object class than previous methods.

[0066] To a) Acquisition of the raw time signal at the transducer element of the ultrasonic sensor

[0067] The prerequisite for the presented method is the acquisition and evaluation of the raw time signal without conventional processing steps such as the generation of amplitude envelopes or thresholds. The proposed processing of the acquired time signals can be performed using known methods as described above.

[0068] To b) Transformation into a complex-valued signal representation

[0069] The acquired time-signal segments are converted into a complex-valued time-frequency representation using a time-frequency transformation such as the short-time Fourier transform (STFT) or, particularly advantageously, the continuous wavelet transform (CWT). With CWT, scaling in the time direction is particularly suitable for data reduction using linear interpolation. A comparable resolution in the time and frequency directions is desirable. Alternatively, the conversion to a complex-valued signal representation can be achieved using a modulation method, particularly advantageously the I&Q modulation method.

[0070] To c) Separate extraction of the amplitude and phase values

[0071] Real-valued feature vectors or feature vector sequences are formed from the complex-valued signal representation, on the one hand with respect to the amplitude via magnitude formation and with respect to the phase via the argument (arctan2). (More likely: at least the phase vector is formed, optionally also the amplitude vector.) The value range of the amplitude representation is logarithmically compressed to emphasize low-energy signal components.

[0072] Regarding d) Unwrapping (= transferring as shown in step 112 of the method) and deriving the phase values ​​in the time and frequency directions. The phase representation is difficult to interpret due to its value range in the interval (-TT,TT). According to the invention, unwrapping in the time direction is therefore used. 2 TT are added to each phase jump. In the phase signal processed in this way, changes in the phase curve are considered as the relevant feature. For this purpose, the phase derivative is formed in the feature vector of the continuous phase in two different ways (only one option can be used! Gradient determination in the time or frequency direction). This is done in the time direction to extract the so-called local instantaneous frequencies and in the frequency direction to extract the local group delay. This results in a total of three real-valued feature vector sequences: one with respect to the amplitude and two with respect to the phase derivative in the time and frequency directions.Alternatively, only one of the phase feature vector sequences can be calculated and used.

[0073] To e) Input of the processed feature vector sequences for classification into a neural network (NN)

[0074] The at least one extracted feature vector sequence is arranged channel by channel and fed into a neural network as a tensor for classification. A convolutional neural network (CNN) is particularly suitable for this purpose.

Claims

Claims 1 . Method (100) for classifying at least one object (10) to be detected with at least one ultrasonic sensor (50) with the following steps: Sending (102) a first signal (1) by the at least one ultrasonic sensor (50) to the object (10); Receiving (104) a second signal (2) by the at least one ultrasonic sensor (50), wherein the second signal (2) represents a signal scattered back from the object (10), Processing (106) the second signal (2) into a first digital signal (3); Transferring (108) the first digital signal (3) on the basis of a time-frequency transformation or a modulation method into a complex-valued signal representation (20) in order to generate a second digital signal (4) to receive, Separating (110) the second digital signal (4) into a first feature vector (5) and / or at least one second feature vector (6), wherein the first feature vector (5) contains a number of amplitude values (21) of the second digital signal (4) and the at least second feature vector (6) contains a number of phase values (23) of the second digital signal (4), Converting (112) a discontinuous value range of the phase (23) of the second feature vector (6) into a continuous curve (24); determining (114) a gradient (8) over the curve of the phase (24) in the time or frequency direction; Creating (116) a third feature vector (7) based on the determined gradient (8); feeding (118) the at least third feature vector (7) as a first input variable (11) into a neural network (9); Determining (120) an object class information (12) for the at least one object (10) using the neural network (9), wherein the neural network (9) generates an output variable (15) on the basis of the at least first input variable (11), which output variable indicates a probability value for at least one defined object class for the one object (10).

2. The method (100) of claim 1, wherein the step of processing the second signal (2) into a first digital signal (3) includes filtering the first digital signal (3) to improve an S / N ratio.

3. Method (100) according to one of the preceding claims, wherein the modulation method is designed as an in-phase & quadrature (l&Q) modulation.

4. Method (100) according to one of the preceding claims, wherein the complex-valued signal representation is mapped as a time-frequency representation.

5. Method (100) according to one of the preceding claims, wherein the first feature vector (5) is fed into the neural network (9) as a second input variable (12).

6. Detection system (200) for classifying an object (10) to be detected, comprising at least one ultrasonic sensor (50) configured to carry out the method (100) according to any one of the preceding claims 1 to 5, wherein the at least one ultrasonic sensor (50) is usable in a vehicle.

7. A computer program comprising machine-readable instructions which, when executed on one or more computers and / or compute instances, cause the computer or computers or compute instances to carry out the method according to any one of claims 1 to 5.

8. Machine-readable data carrier and / or download product with the computer program according to claim 7.

9. One or more computers and / or compute instances with the computer program according to claim 7, and / or with the machine-readable Data carrier and / or download product according to claim 8.

Images