Method for ultrasound-based object classification and apparatus for implementing ultrasound-based object classification

Abstract

The invention implements a method for ultrasound-based object classification, the method comprising: generating (S1) a time reflected signal on a first ultrasound sensor; receiving (S2) and / or generating a secondary signal of at least one or more ultrasonic sensors adjacent to said first ultrasonic sensor and extracting a predetermined feature from said secondary signal, said predetermined feature being generated when said time reflected signal is received on an adjacent ultrasonic sensor; transmitting (S3) the time reflected signal and a predetermined feature from the secondary signal to a classifier device; fusing (S4) the temporal reflection signal and the predetermined feature by means of the classifier means, taking into account a training data set for a current sensor arrangement; an object classification is output (S5) by the classifier means.

Description

Technical Field

[0001] The present invention relates to a method for ultrasound-based object classification and a device for implementing ultrasound-based object classification. Background Art

[0002] Common ultrasonic sensors can be based on the pulse-echo principle, where an electrical signal can excite a transducer on its diaphragm to vibrate, and the vibration can be emitted as sound. The surface of the object reflects the sound, which can cause backscattering in the direction of the ultrasonic sensor. If the backscattered sound hits the diaphragm, the diaphragm can be put into vibration and an electrical signal can be generated on the piezoelectric element. Such an ultrasonic sensor measures the flight time of the sound from emission to return, and the distance to the object that performs the backscattering can be obtained therefrom with the aid of the known speed of sound propagation. After some signal preprocessing steps (such as filtering), single amplitude values and assigned correlation values can be formed by means of threshold formation methods, either with simple or also more complex and adapted methods, and the single amplitude values and assigned correlation values can represent large values in the sound pressure time signal. The echo of the object can be detected by comparing the received amplitude of the sound with a threshold, and typically, only the following echoes of the object are considered relevant and further analyzed: the amplitude of the echo is greater than the threshold. In the case of common ultrasonic sensors, it is possible that these ultrasonic sensors only transmit a small number of amplitude values or correlation values, which are usually referred to as echo values. For one measurement cycle, typically, up to 20 echo values can be obtained in this way and the echo values are transmitted from the sensor output to the controller.

[0003] In the automotive field, especially in the field of parking assistance systems, ultrasonic systems can already be used for distance estimation, where distance estimation can be performed relatively robustly with such ultrasonic systems, however, the sensing discrimination of objects or the determination of object sizes may have relatively high inaccuracies according to currently commonly used methods / systems. Improving the working ability of ultrasonic sensors may be particularly relevant for complex driving functions (such as a higher degree of automated driving or fully automated driving).

[0004] An ultrasonic sensor is described in DE 10 2015 120 659 A1. Summary of the Invention

[0005] The present invention realizes a method for ultrasound-based object classification according to the present invention and a device for implementing ultrasound-based object classification according to the present invention.

[0006] Preferred extensions are described below.

[0007] Advantages of the present invention

[0008] The concept underlying the present invention is to provide a method for ultrasound-based object classification and a device for implementing ultrasound-based object classification, the device having a plurality of ultrasound sensors, wherein object classification and domain adaptation in the case of different sensor configurations can be improved.

[0009] According to the present invention, in the method for ultrasound-based object classification, a time reflection signal is generated on a first ultrasound sensor; secondary signals of at least one or more ultrasound sensors adjacent to the first ultrasound sensor are received and / or generated, and predetermined features are extracted from the secondary signals, the predetermined features being generated when a time reflection signal is received on the adjacent ultrasound sensors; the time reflection signal (and features are obtained from the time reflection signal) and the predetermined features from the secondary signals (and these predetermined features are obtained from the signals) are transmitted to a classifier device; the time reflection signal and the predetermined features are fused by the classifier device, wherein a training data set for the current sensor arrangement (e.g., the original position and / or a position changed according to a preset) is considered; and object classification is output by the classifier device.

[0010] Object classification can be performed on a device having an ultrasound sensing device that can be used for distance measurement or environmental sensing in automotive applications and industrial applications, wherein such an ultrasound sensor can be composed of multiple components, such as a sensor, a sensor housing, a seal, an electronic device, and a plug.

[0011] According to the present invention, in addition to the feature fusion of adjacent sensors and the corresponding network architecture, there can also be a method for domain adaptation that can allow efficient classification in a system complex in the case of different sensor arrangements without the workload of additional measurement activities or with a reduced workload by means of additional measurement activities. Here, an increased robustness of the classification model and a lower application workload can be achieved with reduced network complexity and hardware requirements.

[0012] For example, the predetermined features can relate to signal variation curves (certain variations, extreme values, etc.). The time reflection signal can be the transmitted and already reflected signal. The secondary signal can be the received signal from an adjacent sensor.

[0013] According to a preferred embodiment of the method, the classifier device includes a neural network or particularly a convolutional neural network (CNN), the neural network including feature maps of the first ultrasound sensor and / or adjacent ultrasound sensors.

[0014] According to a preferred embodiment of the method, in addition to the time signal of the first ultrasonic sensor, the compressed feature maps from the CNN convolutional layers of adjacent ultrasonic sensors are also transmitted and considered.

[0015] According to a preferred embodiment of the method, in an initial step, features are extracted on the first ultrasonic sensor and on adjacent ultrasonic sensors, and a convolutional layer is applied here. And in order to fuse the features, the feature maps are obtained (hinzugezogen) and concatenated (konkateniert) to the features of the first ultrasonic sensor. Then, when extracting the fused features, additional convolutional layers are used for the concatenated feature maps. Then, the feature data is smoothed And object classification is performed by means of a fully connected layer.

[0016] According to a preferred embodiment of the method, the training dataset includes measurements of known objects at known positions of the objects and a specified sensor arrangement (the original sensor arrangement and / or a sensor arrangement changed afterwards according to a pre-given (Vorgabe)).

[0017] According to a preferred embodiment of the method, the training dataset is processed separately for a pre-determined vehicle type.

[0018] According to a preferred embodiment of the method, the training dataset takes into account the adjustment angle (Anstellwinkel) of the first ultrasonic sensor and / or adjacent ultrasonic sensors. The adjustment angle can refer to the vertical inclination of the sensor, which can vary depending on the vehicle type and the sensor position.

[0019] According to a preferred embodiment of the method, the training dataset takes into account the sensor position in terms of the time-of-flight modification and / or amplitude modification in the shift of the horizontal and / or vertical sensor positions and / or object positions.

[0020] According to a preferred embodiment of the method, the training dataset takes into account the geometry of the installation environment of the first ultrasonic sensor and / or adjacent ultrasonic sensors.

[0021] According to a preferred embodiment of the method, the classifier device and the model applied to the classifier device are fine-tuned, wherein the predetermined weights in the neural network are adapted and thereby the adaptation to the target domain of the sensor arrangement is achieved.

[0022] According to the present invention, a device for implementing ultrasound-based object classification includes a control device and / or a computer device, which can be connected to a first ultrasound sensor and to an adjacent ultrasound sensor and is configured to implement the method according to the present invention.

[0023] By means of the method and / or the device, it is possible to increase the performance of ultrasonic sensors in a system complex, wherein the use of machine learning methods or artificial neural networks is particularly suitable for classifying obstacles. Obstacles can be classified here based on the time signal, the envelope curve or only on the echo points as input to the neural network. The corresponding time signal can provide a large amount of information content as input, and the classification can be output, for example, into aggregated object classes or can be output in terms of object height.

[0024] In order to improve the classification performance of a single sensor, multiple sensors in the system complex can be used for classification, and it can be advantageously aimed at taking into account the features of the detected signals of adjacent sensors for the classification decision of each single sensor. In addition, a spatial scan of obstacles at different angles can further illustrate the object geometry and can thus have a positive influence on the classification.

[0025] However, the sensor position or sensor arrangement may vary from vehicle to vehicle. In particular, the changed sensor arrangement produces changes in the amplitude and the sound flight time. If a neural network learns to process the characteristics of a specific sensor arrangement during training, this can often result in a significantly reduced performance when processing other sensor arrangements that are unknown to the network. As a result, the distribution of data points in the target domain may differ from the distribution of data points in the domain of the training data, which in the field of machine learning can also be generally referred to as domain shift or distribution shift.

[0026] Conventional classification models can usually handle different sensor arrangements with similar performance only if there is a training data set with measurements made with all potential sensor arrangements and the model is trained with a subset of the data set for each vehicle type using a corresponding subset of the data set. On the other hand, according to conventional methods, a model can be trained with the aid of training data for all possible sensor arrangements, which can be applied to all vehicle types in the same way and can implicitly recognize and take into account the sensor arrangements. However, with the latter approach, a significantly higher complexity of the model may be required, which may result in increased hardware requirements. In so-called applications, the ultrasound system can be adapted to new vehicle types, but this may require additional simulations and vehicle measurements.

[0027] The approach to addressing the problem of domain shift is defined as domain adaptation, where a model that has been trained for a certain data domain can be adapted to the target domain.

[0028] It is intended to achieve and implement such a method for domain adaptation within the scope of the method and the device, by which an additional measurement effort can be avoided or reduced.

[0029] In common domain adaptation methods from the field of machine learning, in addition to the initial training of the model with the aid of labeled data, fine-tuning can also be performed with the aid of additional, specifically selected measurement data from the target domain.

[0030] This enables a reduced-scale acquisition of measured data for each vehicle type.

[0031] Advantageously, classification can be implemented in the sensor complex, and for this purpose, the feature maps of the adjacent sensors preprocessed for the classification decision of the single sensor can be obtained, and the feature fusion method can be implemented as a classification model by a CNN with multiple input interfaces. Here, although there are still classification results (obtained) for each single sensor, these classification results each benefit from the detected information of the adjacent sensors. The individual classification results can be transmitted to the control unit, for example, in the form of a Softmax function or generalized embeddings (generalisierten Embeddings) and further processed.

[0032] The method can be used for domain adaptation for ultrasound-based object classification in the presence of different sensor arrangements or vehicle types, and is achieved by modifying the measured time signals in an existing training dataset to simulate the modified sensor arrangement, whereby the workload for detecting new training datasets can be omitted or reduced.

[0033] In order to simulate a modified sensor arrangement, an amplitude correction as well as a flight time correction of the time signal may be required, and the following influences can be taken into account by the sensor arrangement (during training and / or adaptively during execution): the adjustment angle (-25° to +25°) and the resulting changed amplitude based on the directional characteristics of the sensor; the vertical or horizontal position of the sensor in the bumper and the resulting changed flight time and amplitude based on the changed distance to obstacles and the ground; and / or the geometry of the installation environment (smooth bumper / grille / mounting funnel) and the resulting changed directional characteristics.

[0034] The ultrasonic time signal of at least one sensor can be used and can be extracted as an analog signal directly at the output of an amplifier circuit downstream of the piezoelectric element. Then, in addition, a high-resolution digital ultrasonic time signal can be generated by means of an analog-digital converter according to a predetermined sampling theorem (for example according to Shannon's theorem), wherein typically the sampling can be performed at ≥100 kHz, preferably 200 kHz.

[0035] The ultrasonic time signal can be preprocessed by means of filtering, for example, in order to improve the signal-to-noise ratio or to suppress external sounds. Such filtering can be performed before the analog-to-digital conversion or preferably after the digitization of the time signal, wherein suitable high-pass filters, low-pass filters or advantageously bandpass filters or decimation filters can be used for the corresponding filtering.

[0036] In a further processing step, relevant and temporally limited time segments can be cut out of the overall signal according to a specification, which can be used, for example, for data reduction.

[0037] This type of shearing can be realized automatically using known time and geometric connections or can be realized based on conventional existing threshold-based echo flight time data. In addition, it is also possible to perform an analysis of the time signal by means of shearing using a sliding window method. The use of correlation functions, such as the use of cross-correlations with known (synthetically generated or measured) transmitted signals, can prove to be particularly advantageous. Automatic shearing can be realized particularly advantageously with the aid of spacing-related variations of the transmitted signal used for the association. As an option, the signal can be subjected to a credibility check by means of a cross-correlation with the transmitted signal during or directly after shearing, whereby the reliability of the classification can be increased and a high robustness of the algorithm can be achieved as a whole.

[0038] According to the invention, a highly accurate and efficient ultrasound-based object classification can advantageously be achieved in a sensor complex.

[0039] The device may also be distinguished by the features mentioned in combination with the method and the advantages of the method, and vice versa.

[0040] Further features and advantages of embodiments of the invention emerge from the following description with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In the following, the invention is explained in more detail with reference to exemplary embodiments illustrated in the schematic drawings.

[0042] The accompanying drawings show:

[0043] Figure 1A block diagram showing method steps of a method for ultrasound-based object classification according to one embodiment of the present invention;

[0044] Figure 2 A schematic arrangement showing an ultrasonic sensor and the relative effect of the sensor position on the sound flight time;

[0045] Figure 3 shows a schematic flow of signal detection up to classification in a sensor complex according to one embodiment of the present invention; and

[0046] Figure 4 A schematic sequence of training from measured value detection to adaptation of the classifier model to a changed sensor arrangement according to one exemplary embodiment of the present invention is shown. DETAILED DESCRIPTION

[0047] In the figures, the same reference signs denote identical or functionally identical elements.

[0048] Figure 1 A block diagram shows method steps of a method for ultrasound-based object classification according to one embodiment of the present invention.

[0049] In the method, a time reflection signal is generated S1 on a first ultrasonic sensor; a secondary signal of at least one or more ultrasonic sensors adjacent to the first ultrasonic sensor is received S2 and / or generated and predetermined features are extracted from the secondary signal, wherein the predetermined features are generated when the time reflection signal is received on the adjacent ultrasonic sensor; the time reflection signal and the predetermined features from the secondary signal are transmitted S3 to a classifier device; the time reflection signal and the predetermined features are fused S4 by the classifier device, wherein a training data set for the current sensor arrangement is taken into account; and an object classification is output S5 by the classifier device.

[0050] Figure 2 Schematic arrangement showing ultrasound sensors and the relevant effect of sensor position on sound flight time.

[0051] according to Figure 2, showing a first sensor position Sn1 and a second sensor position Sn2 displaced relative to the first sensor position, the first sensor position and the second sensor position can be displaced vertically relative to each other. For this purpose, the signal variation curves from the first object O1 to the two sensors and from the second object O2 (or the change of the object position) to the two sensors (or the sensor position) simplified via the beam model are shown. In order to be able to process the signal under such influence in the case of a changed sensor position (for example from Sn1 to Sn2), it may be necessary to know the exact sensor arrangement and the position of one or more objects (obstacles or backscatter points). The position of the sensors Sn1 and Sn2 (or, if only the position of the sensor is displaced, the sensor) can be known at any time for this purpose, for example for different vehicle types. However, usually for each sensor, only the distance can be derived from the echo on the obstacle, but not the exact object position or the corresponding coordinates of the backscatter point in the sound field. As a result, a more accurate positioning via a common sensor arrangement through multi-lateral measurement can usually only be insufficient.

[0052] On the test bench, the exact object positions and object geometries for the test bench measurements can be present in the training data set, and since the object positions and obstacle positions are present in this case, any sensor arrangement can be simulated for the training data set. Figure 2 , the influence of the changed sensor arrangement on the flight time according to the obstacle position can be estimated. The changed flight time also means the changed amplitude based on the geometric propagation attenuation and the air sound attenuation, respectively. The change of the sound flight time can be related to the sensor position and the object position. In particular, for example, at the first object position O1, the change of the sensor position from Sn1 to Sn2 can result in a flight time difference Δt1=5ms. And in the case of the same change in the sensor position, correspondingly, for the object position O2, a flight time difference Δt2=2ms can be obtained.

[0053] Figure 3 A schematic sequence of signal detection up to classification in a sensor complex according to one exemplary embodiment of the present invention is shown.

[0054] Three sensors are shown by way of example, wherein the middle sensor (first ultrasonic sensor) is set up for receiving E and transmitting S, and the two adjacent sensors are each set up only for receiving E, in particular for receiving reflections on the object OBJ. After reception, the signal can be preprocessed by each sensor, and in a subsequent step, the features of the single sensor can be extracted and applied in the corresponding convolutional layer. Then, with the help of the extracted feature map, the features of multiple sensors can be fused and applied in another convolutional layer. Via the feature map extracted therefrom, a classification probability or a classification embedding can be derived by applying a fully connected layer. Then, on this basis, a prediction can be made with the help of a classifier (classification) and the prediction can be output. A convolutional neural network (CNN) can be used, which can provide a classification output for the detected signal of the ultrasonic sensor, but in addition to the time signal detected by the first ultrasonic sensor or the corresponding time-frequency transformation, the features extracted from the detected time signal of the adjacent sensor can also be obtained. Inputting multiple complete time signals into the classifier will mean a large number of trainable parameters and increased requirements for hardware and data transmission. Therefore, according to the present invention, in addition to the time signal of a single sensor, the compressed feature map of the CNN convolution layer from the adjacent sensor can also be transmitted.

[0055] That is, after the first ultrasonic sensor has emitted a transmit signal, backscatter can also be detected by an adjacent sensor. In other words, the preprocessed feature map (single sensor feature extraction) can be transmitted by the adjacent sensor to the active sensor (first ultrasonic sensor) and can be learned for classification decisions (multi-sensor feature extraction + classifier). Alternatively, the extracted feature maps can be processed together on a central controller. In the case of an input size of, for example, 64x32 (time-frequency representation), a more accurate CNN architecture for the sensor can be defined as follows:

[0056]

[0057] The above table shows an exemplary CNN architecture when clustering over seven object categories.

[0058] The input image (transformed signal) can be generated in preprocessing via a time-frequency transform (e.g., short-time Fourier transform or wavelet transform). After a series of convolutional and pooling layers, the preprocessed feature maps from adjacent sensors can be acquired and concatenated, for example, in layer 8. Further convolutional layers process the concatenated feature maps (multi-sensor feature extraction), which can then be input into a fully connected layer for classification after smoothing.

[0059] Figure 4A schematic sequence of training from measured value detection to adaptation of the classifier model to a changed sensor arrangement according to one exemplary embodiment of the present invention is shown.

[0060] This sequence is shown for training from the acquisition of measured values ​​with a certain basic sensor arrangement to the adaptation of the classifier model to any desired vehicle-specific sensor arrangement.

[0061] An exemplary training data set may consist of measurements of known objects at known measurement locations using a defined sensor arrangement.

[0062] According to one embodiment of the method, it can be provided that the training data can be processed separately for each vehicle type (existing in the application or in general), so that the corresponding sensor arrangement can be simulated in the training data set, thereby making it possible to efficiently apply the classifier to different vehicle types without the need for additional measurement activity.

[0063] Therefore, a domain adaptation approach can be used in the training dataset from the base sensor arrangement to the vehicle-specific sensor arrangement.

[0064] Advantageously, for each of the sensors, the signal processing on the object-specific time signal segment can provide the following steps:

[0065] Advantageously, the adjustment angle can be considered, wherein it can be considered that, by means of the direction factor Γ, the angle Sound pressure related to θ With the angle and the reference sound pressure at θ0 ratio, making amplitude modifications in the horizontal and vertical directional characteristics of the sensor.

[0066] Furthermore, the sensor position can be taken into account, wherein a flight time modification and an amplitude modification can be taken into account with respect to a shift of the sensor position and the object position in the horizontal and vertical directions by the speed of sound c, in particular in the case of a change in the distance Δd between the sensor and the object backscatter point, resulting in a changed flight time of Δt=Δd·c and an amplitude change of p′(t)=d0d1p(t) via geometric propagation attenuation on the time signal p(t), wherein the distance between the sensor and the object in the basic sensor arrangement is d0 and the distance in the sensor arrangement to be simulated is d1. In addition, a frequency-dependent atmospheric sound attenuation can be taken into account.

[0067] In addition, the geometry of the installation environment can be taken into account. In this case, the directional factor Γ can be taken into account, taking into account the acoustic effects on elements of the installation environment (such as diffraction, reflection and scattering). The process described so far involves a physics-based simulation of a changed sensor arrangement for a training process.

[0068] Then, we describe the fine-tuning of the pre-trained classifier to adapt it to the target domain.

[0069] In order to make it possible to dispense with completely new training with the aid of processed data sets, fine-tuning of a model pre-trained with the aid of data sets of the basic sensor arrangement can be provided. The pre-trained model may already be able to extract relevant features from the current signal. With fine-tuning, only a slight adaptation of the weights in the neural network is then performed, whereby the model can be adapted to the target domain. This results in a significantly reduced training duration and thus a lower application workload. With a small number of training epochs, the model can be adapted to new vehicle types.

[0070] according to Figure 4 After the measured value acquisition in step 1 (assuming the existence of a basic sensor arrangement), the sampled measured data can be filtered and preprocessed into a training data set in a preprocessing step 2. With the help of this training data set, on the one hand, a pretraining of the CNN can be carried out in step 3.a, and on the other hand, taking into account the parameters of the sensor arrangement, a new training data set can be created in step 3.b with the help of analog signal processing (for domain adaptation), and then, with the help of this new training data set (adapted training data), a fine-tuning of the CNN can be performed in step 4 (also taking into account the pretraining in step 3a). Finally, in step 5, the adapted classification model can be integrated in the vehicle.

[0071] Although the invention has been completely described above on the basis of preferred exemplary embodiments, it is not restricted thereto but can be modified in an advantageous manner.

Claims

1. A method for ultrasound-based object classification, the method comprising the following steps: generating (S1) a time reflection signal on a first ultrasonic sensor; receiving (S2) and / or generating a secondary signal of at least one or more ultrasonic sensors adjacent to the first ultrasonic sensor and extracting a predetermined feature from the secondary signal, the predetermined feature being generated when the time reflection signal is received at the adjacent ultrasonic sensor; transmitting (S3) said time reflection signal and predetermined features from said secondary signal to a classifier device; fusing the time reflection signal and the predetermined feature by the classifier device (S4); The object classification is output (S5) by the classifier means.

2. The method according to claim 1, wherein: The classifier device comprises a neural network or, in particular, a convolutional neural network (CNN), wherein the neural network comprises a feature map of the first ultrasonic sensor and / or of adjacent ultrasonic sensors.

3. The method according to claim 1 or 2, wherein: The fusion ( S4 ) is achieved in that the training data set for the current sensor arrangement is taken into account.

4. The method according to claim 1, wherein: In addition to the time signal of the first ultrasonic sensor, the compressed feature map from the convolutional layer of the CNN of the adjacent ultrasonic sensor is also transmitted and taken into account.

5. The method according to claim 4, wherein: In an initial step, features are extracted at the first ultrasonic sensor and at the adjacent ultrasonic sensors, and convolutional layers are applied in this process, and in order to fuse the features, a feature map is obtained and connected to the features of the first ultrasonic sensor, and then, when extracting the fused features, further convolutional layers are used for the connected feature maps, and then the feature data are smoothed and the object classification is performed with the help of fully connected layers.

6. The method according to any one of claims 1 to 5, wherein: The training data set includes measurements of known objects at known locations of the objects and uses a prescribed arrangement of sensors.

7. The method according to any one of claims 1 to 6, wherein: The training data set is processed individually for predetermined vehicle types.

8. The method according to any one of claims 1 to 7, wherein: The training data set takes into account an adjustment angle of the first ultrasonic sensor and / or of the adjacent ultrasonic sensors.

9. The method according to any one of claims 1 to 7, wherein: The training data set takes into account the sensor position with regard to a flight time modification and / or an amplitude modification with regard to a shift of the horizontal and / or vertical sensor position and / or the object position.

10. The method according to any one of claims 1 to 9, wherein: The training data set takes into account the geometry of the installation environment of the first ultrasonic sensor and / or of the adjacent ultrasonic sensors.

11. The method according to any one of claims 1 to 10, wherein: The classifier device and the model used by the classifier device are fine-tuned, wherein predetermined weights in the neural network are adapted and thereby an adaptation to a target range of the sensor arrangement is achieved.

12. A device for implementing ultrasound-based object classification, the device comprising a control device and / or a computer device, which can be connected to a first ultrasound sensor and to an adjacent ultrasound sensor, and the control device and / or the computer device is configured to implement the method according to any one of claims 1 to 11.

Images