Method for ultrasound-based object classification and device for performing ultrasound-based object classification

Abstract

The present invention provides a method for ultrasound-based object classification, comprising generating (S1) a time-reflection signal at a first ultrasonic sensor; receiving (S2) and / or generating secondary signals from at least one or more ultrasonic sensors adjacent to the first ultrasonic sensor and extracting predetermined features from the secondary signals, which are generated upon receiving the time-reflection signal at the adjacent ultrasonic sensors; transmitting (S3) the time-reflection signal and the predetermined features from the secondary signals to a classifier device; fusing (S4) the time-reflection signal and the predetermined features by the classifier device, wherein a training data set for a given sensor arrangement is taken into account; and outputting (S5) an object classification by the classifier device

Description

The present invention relates to a method for an ultrasonic object classification and to an apparatus for carrying out an ultrasonic object classification.Prior ArtConventional ultrasonic sensors can be based on the pulse-echo principle, wherein an electrical signal can excite the transducer at its membrane to oscillations, which can be radiated as sound. The surface of an object reflects the sound, wherein a back scattering in the direction of the ultrasonic sensor can occur. If the backscattered sound impinges on the diaphragm, it can be set into oscillations and an electrical signal can arise at the piezoelectric element. Such ultrasonic sensors measure the transit time of the sound from transmission to return and the distance to the back-scattering object can be determined from this by means of the known sound propagation speed. After some signal preprocessing steps, such as filtering, for example, individual amplitude values and associated correlation values can be formed by means of threshold value formation methods using simple or more complex and adaptive methods, which can represent the large values in the sound pressure time signal. An echo of an object can be detected by comparing the reception amplitude of the sound with a threshold value and typically only those echoes of objects whose amplitude lies above the threshold value are considered relevant and further evaluated. In conventional ultrasonic sensors, it may be that these transmit only such few amplitude or correlation values that are frequently referred to as echo values. In this way, typically up to 20 echo values can be determined for a measurement cycle and transmitted from the sensor output to a control device.Ultrasonic systems can already be used in the automotive sector for distance estimation, in particular also in the sector of parking assistance systems, wherein distance estimation can be carried out with such ultrasonic systems in a relatively robust manner, but a sensory distinction of objects or the determination of object dimensions according to currently common methods / systems can be subject to a higher degree of imprecision. Particularly relevant may be the increase in performance of ultrasonic sensors for complex driving functions, such as higher or fully automated driving.DE 10 2015 120 659 A1 describes an ultrasonic sensor.Disclosure of the InventionThe present invention provides a method for an ultrasonic object classification according to claim 1 and an apparatus for carrying out an ultrasonic object classification according to claim 12.Preferred refinements are the subject matter of the dependent claims.Advantages of the InventionThe idea underlying the present invention is to specify a method for an ultrasonic-based object classification and a device for carrying out an ultrasonic-based object classification having a plurality of ultrasonic sensors, it being possible to improve object classification and domain adaptation at different sensor constellations.According to the invention, in the method for an ultrasonic-based object classification, a time reflection signal is generated at a first ultrasonic sensor; a reception and / or generation of secondary signals of at least one or more ultrasonic sensors adjacent to the first ultrasonic sensor and extraction of predetermined features from the secondary signals, which are generated when the time reflection signal is received at the adjacent ultrasonic sensors; a transmission of the time reflection signal (and the features are determined from it) and the predetermined features (and these are determined from the signals) from the secondary signals to a classifier device; a fusion of the time reflection signal and the predetermined features by the classifier device, wherein a training data set for a present sensor arrangement (for example an original position and / or changed according to default) can be taken into account; and an output of an object classification by the classifier device.The object classification can be carried out on a device with ultrasonic sensors, as can be used in automotive and industrial applications for distance determination or environment sensing, wherein such ultrasonic sensors can consist of a plurality of components, for example a transducer, a sensor housing, a seal, an electronics unit and a plug.According to the invention, in addition to a feature fusion of adjacent sensors and corresponding network architecture, a method for domain adaptation can be present, which can allow an efficient classification in the system cluster in the case of different sensor arrangements without the outlay or with reduced outlay of additional measurement campases. This can result in increased robustness of the classification models with reduced network complexity and hardware requirement and a low application outlay.The predetermined features may relate, for example, to the signal profile (certain variations, extremes, etc.). The time reflection signal can be a transmitted and already reflected signal. The secondary signals may be received signals of the adjacent sensors.According to a preferred embodiment of the method, the classifier device comprises a neural network or, in particular, a convolutional neural network (CNN) with feature maps of the first ultrasonic sensor and / or of the adjacent ultrasonic sensors.According to a preferred embodiment of the method, compressed feature maps from CNN convolutional layers of the adjacent ultrasonic sensors are transmitted and taken into account in addition to time signals of the first ultrasonic sensor.According to a preferred embodiment of the method, features are extracted at the first ultrasonic sensor and at the adjacent ultrasonic sensors in an initial step and convolutional layers are applied and used to merge the features feature maps into the features of the first ultrasonic sensor and concatenated, and then a further convolutional layer is used for the concatenated feature maps when extracting the merged features, and then the feature data is smoothed and an object classification is carried out using a fully connected layer.According to a preferred embodiment of the method, the training data set comprises measurements of known objects at known positions of the objects and a defined sensor arrangement (originally and / or subsequently modified according to a specification).According to a preferred embodiment of the method, the training data set is / is individually processed for a predetermined vehicle type.According to a preferred embodiment of the method, the training data set takes into account an angle of attack of the first ultrasonic sensor and / or of the adjacent ultrasonic sensors. The angle of attack may be the vertical inclination of a sensor, which may vary for each vehicle type and sensor position.According to a preferred embodiment of the method, the training data set takes into account a sensor position with respect to a runtime and / or amplitude modification with respect to a displacement of the horizontal and / or vertical sensor position and / or object position.According to a preferred embodiment of the method, the training data set takes into account a geometry of the installation environment of the first ultrasonic sensor and / or of the adjacent ultrasonic sensors.According to a preferred embodiment of the method, the classifier device and a model which the classifier device uses are fine-tuned, wherein predetermined weights are adapted in the neural network and an adaptation to a target domain of the sensor arrangements is effected.According to the invention, the apparatus for carrying out an ultrasonic-based object classification comprises a control and / or computer device which can be connected to the first ultrasonic sensor and to adjacent ultrasonic sensors and the control and / or computer device is configured to carry out a method according to the invention.The method and / or the device can achieve an increase in the performance of the ultrasonic sensors in the system group, wherein, in particular for classifying obstacles, use of methods of machine learning or artificial neural networks is conceivable. In this case, a classification of obstacles can take place on the basis of time signals, envelopes or merely echo points as input into a neural network. A corresponding time signal can provide a large information content as input and the classification output can take place, for example, in aggregated object classes or with respect to the object height.In order to improve the classification performance of an individual sensor, a plurality of sensors in the system group can be used for classification and it can advantageously be sought to include features of the detected signals of adjacent sensors for the classification decision of each individual sensor. Furthermore, a spatial scanning of obstacles over different angles can provide further information about the object geometry and thus have a positive effect on the classification.However, a sensor position or the sensor arrangement may vary per vehicle type. In particular, changes in the amplitudes and sound transit times result from the changed sensor arrangement. When a neural network has learned the handling of features of a specific sensor arrangement in a training process, a significantly reduced performance may usually be present in the handling of other sensor arrangements unknown to the network. A different distribution of the data points than in the domain of the training data can therefore be present in the target domain, wherein domain shift or distributed shift can generally also be referred to in the field of machine learning.The usual classification model can usually bypass different sensor arrangements with comparable performance only if training data sets with measurements are present among all potential sensor arrangements and a separate model is trained with a corresponding partial data set for each vehicle type. On the other hand, according to conventional approaches, a model can be trained with training data of all possible sensor arrangements, which model can be applied equally to all vehicle types and can implicitly recognize and calculate the sensor arrangements. However, due to the latter approach, it may be necessary to use a significantly higher complexity of the model, which may result in increased hardware requirements. In the case of the so-called application, an ultrasound system can be adapted to a new vehicle type, for which purpose, however, increased simulations and vehicle measurements can occur.Methods that solve the problem of domain shift are defined as domain adaptation, wherein a model that has already been trained on a specific data domain can be adapted to a target domain.Such methods for domain adaptation, whereby the additional measurement effort can be prevented or reduced, are desirable and are achieved within the scope of the method and the device.In conventional methods of domain adaptation from the field of machine learning, in addition to the initial training of the model with labeled data, fine tuning with additional, specifically selected measurement data from the target domain can be used.This makes it possible to acquire measurement data for each vehicle type in a reduced amount.A classification in the sensor group can advantageously be implemented and feature maps of the adjacent sensors preprocessed for this purpose can be used for the classification decision of an individual sensor and this method of feature fusion can be implemented by a CNN having a plurality of input interfaces as a classification model. Although classification results for each individual sensor may still be present (ascertained), these may benefit from the detected information of the adjacent sensors. Individual classification results can be transmitted to a control unit, for example, in the form of softmax probabilities or generalized embedding, and can be further processed.The method can be used for domain adaptation for the ultrasound-based object classification in different sensor arrangements or vehicle types and can be carried out by means of modifications of the measured time signals in an existing training data set for simulating a changed sensor arrangement, as a result of which it is possible to dispense with or reduce a outlay for recording new training data sets.To simulate a modified sensor arrangement, amplitude and time of flight corrections on the time signals may be necessary and the following influences may be taken into account by the sensor arrangement (during training and / or adaptively during execution): the angle of attack (-25° to +25°) and thereby modified amplitude on the basis of the directional characteristic of the sensors; vertical or horizontal position of the sensors in the bumper and thereby modified time of flight and amplitude on the basis of modified distance from the obstacle and from the ground; and / or geometry of the installation environment (smooth bumper / grid / installation funnel) and thereby modified directional characteristic.Ultrasonic time signals from at least one sensor can be used and taken as an analog signal directly at the output of the electrical amplification circuit downstream of the piezoelectric element. Furthermore, a high-resolution digital ultrasound time signal can then be generated by means of an analog-to-digital converter according to a predetermined scanning theory, for example according to Shannon, wherein the scanning can typically take place with >=100 kHz, preferably 200 kHz.With filtering, the ultrasonic time signals can be preprocessed, for example, in order to improve a signal-to-noise ratio or to suppress extraneous sounds. Such filtering can already take place before the analog-to-digital conversion, or preferably after the digitalization of the time signals, it being possible to use high-pass, low-pass or advantageously band-pass filters or decimation filters which are suitable for corresponding filtering.In a further processing step, relevant and time-limited time sections can be cut out of the overall signal according to a specification, which can serve, for example, for data reduction.Such a cutting out can take place automatically using known temporal and geometric relationships or on the basis of the conventionally present threshold-value-based echo runtime data. Furthermore, it is also possible to perform an evaluation of the time signal by means of sections using a sliding window approach. A use of a correlation function, e.g. cross-correlation with the known (synthetically generated or measured) transmission signal can result as particularly advantageous. The automatic cutting out can be carried out particularly advantageously with the aid of distance-dependent variants of the transmission signal for the correlation. As an option, a plausibility check of the signal can be carried out during or immediately after the cutting out by means of cross-correlation with the transmission signal, as a result of which reliability of the classification can be increased and overall a high robustness of the algorithm can be achieved.According to the invention, a highly precise and efficient ultrasound-based object classification in the sensor system can be advantageously achieved.The device can also be distinguished by the features mentioned in connection with the method and the advantages thereof, and vice versa.Further features and advantages of embodiments of the invention will become apparent from the following description with reference to the attached drawings.Brief Description of the DrawingsThe present invention is explained in more detail below with reference to the exemplary embodiments indicated in the schematic figures of the drawing.The following are shown: FIG. 1 shows a block diagram of method steps of the method for an ultrasonic-based object classification according to an exemplary embodiment of the present invention; FIG. 2 shows a schematic arrangement of ultrasonic sensors and the associated influence of the sensor position on the sound transit time; FIG. 3 shows a schematic sequence of a signal detection up to classification in the sensor network according to an exemplary embodiment of the present invention; and FIG. 4 shows a schematic sequence of training from a measured value acquisition until the adaptation of the classifier model to a modified sensor arrangement according to an exemplary embodiment of the present invention.In the figures, identical reference numerals designate identical or functionally identical elements.FIG. 1 shows a block diagram of method steps of the method for an ultrasonic-based object classification according to an exemplary embodiment of the present invention.In the method, a time reflection signal is generated S 1 at a first ultrasonic sensor; a reception S 2 and / or generation of secondary signals of at least one or more ultrasonic sensors adjacent to the first ultrasonic sensor and extraction of predetermined features from the secondary signals which are generated when the time reflection signal is received at the adjacent ultrasonic sensors; a transmission S 3 of the time reflection signal and the predetermined features from the secondary signals to a classifier device; a fusion S 4 of the time reflection signal and the predetermined features by the classifier device, wherein a training data set for a present sensor arrangement is taken into account; and an output S 5 of an object classification by the classifier device.FIG. 2 shows a schematic arrangement of ultrasonic sensors and the associated influence of the sensor position on the sound transit time.According to FIG. 2, a first sensor position Sn 1 and a second sensor position Sn 2 displaced relative thereto are shown, which can be displaced vertically with respect to one another. For this purpose, simplified signal curves from a first object O 1 to the two sensors and from a second object O 2 (or change in the object position) to the two sensors (or sensor positions) are shown via the beam model. In order to be able to process the signals under such influences in the case of a changed sensor position, for example from Sn1 to Sn2, it may be necessary to know the exact sensor arrangement and the position of the objects / the object (obstacles or backscatter points). The positions of the sensors Sn 1 and Sn 2 (or the sensor if only the position of a sensor is displaced) may be known at any time, for example for different vehicle types. From the echoes at obstacles, however, usually only the distances per sensor and no exact object positions or corresponding coordinates of the backscatter points in the sound field can result. As a result, more accurate localization via multilateration via conventional sensor arrangements can usually only be inadequate.In a test stand, the exact object positions and geometries for the test stand measurements can be given in the training data set, and since object and obstacle positions are given in such a case, the simulation of arbitrary sensor arrangements for the training data set can take place. An influence of a changed sensor arrangement depending on the obstacle position on the runtime can thus be estimated according to the illustration of FIG. 2. A changed transit time also means a changed amplitude due to the geometric propagation attenuation and the airborne sound attenuation. A change in the sound transit time can be dependent on the sensor position and object position. In particular, for example, at the first object position O 1, a propagation time difference of Δ t 1=5 ms can result from the change in the sensor position from Sn 1 to Sn 2. With the same change of the sensor position, on the other hand, a delay time difference of Δ t2=2 ms can correspondingly result for object position O2.FIG. 3 shows a schematic sequence of a signal detection up to classification in the sensor network according to an exemplary embodiment of the present invention.Three sensors are shown by way of example, wherein the central sensor (first ultrasonic sensor) is configured for receiving E and transmitting S and the two adjacent sensors are each configured only for receiving E, in particular for reflection at the object OBJ. After the reception, the signal from each sensor can be preprocessed and, in a subsequent step, features of the individual sensor can be extracted in each case and applied in a respective convolutional layer. With the aid of extracted feature maps, the features of a plurality of sensors can then be merged and applied in a further convolutional layer. Feature maps extracted therefrom can be derived by using fully connected layer classification probabilities or classification embedding. Based on this, a prediction can then be made with the classifier (classification) and output. A convolutional neural network (CNN) can be used, which can provide a classification output for detected signals of an ultrasonic sensor, wherein, however, features extracted from the detected time signals of adjacent sensors can also be used in addition to the time signal detected by the first ultrasonic sensor or corresponding time-frequency transform. The input of a plurality of complete time signals into the classifier, on the other hand, would mean a high number of trainable parameters and increased requirements for the hardware and data transmission, whereby according to the invention compressed feature maps from the CNN convolutional layers of the adjacent sensors can be transmitted in addition to the time signal of the individual sensor.After a first ultrasonic sensor has emitted the transmission signal, the backscatters can therefore additionally be detected by the adjacent sensors. In other words, preprocessed feature maps (single sensor feature extraction) can be transmitted from the adjacent sensors to the active sensor (first ultrasonic sensor) and used for the classification decision (multisensor feature extraction+classifier). Alternatively, the joint processing of the extracted feature maps can take place on a central control device. A more accurate CNN architecture of a sensor could be defined at an input of, for example, 64x32 (time-frequency representation) as follows:11 × 64 × 32Convolution 2D5 × 716 × 64 × 32Zero Padding, Batch Normalization, ReLu216 × 64 × 32Average Pooling2 × 216 × 32 × 16316 × 32 × 16Convolution 2D1 × 532 × 32 × 16Zero Padding, Batch Normalization, ReLu432 × 32 × 16Convolution 2D5 × 132 × 32 × 16Zero Padding, Batch Normalization, ReLu532 × 32 × 16Average Pooling2 × 232 × 16 × 8632 × 16 × 8Convolution 2D3 × 364 × 16 × 8Zero Padding, Batch Normalization, ReLu764 × 16 × 8Average Pooling2 × 264 × 8 × 483 × 64 × 8 × 4Concatenates192 × 8 × 4Interface Neighbor Sensors9192 × 8 × 4Convolution 2D3 × 3192 × 8 × 4Zero Padding, Batch Normalization, ReLu10192 × 8 × 4Average Pooling2 × 2192 × 4 × 211192 × 4 × 2Flattening1536121536Fully-connected256Batch Normalization, ReLu13256Fully-connected7SoftmaxmaxThe above table shows an example CNN architecture when aggregated into seven object classes.The input images (transformed signals) can be generated in a preprocessing via a time-frequency transformation, such as a short-time Fourier transformation or a wavelet transformation. After a series of convolutional layer and pooling layer, for example in layer 8, the preprocessed feature maps from the adjacent sensors can be added and concatenated. A further convolutional layer processes the concatenated feature maps (multisensor feature extraction) before these can be given for classification into fully connected layers after flattening.FIG. 4 shows a schematic sequence of training from a measured value acquisition until the adaptation of the classifier model to a modified sensor arrangement according to an exemplary embodiment of the present invention.The sequence is shown for training from the measured value acquisition under a specific basic sensor arrangement until the classification model is adapted to any desired, vehicle-specific sensor arrangement.An exemplary training data set may consist of measurements of known objects at known measurement positions with a fixed sensor arrangement.According to one specific embodiment of the method, it may be provided that the training data is individually processed for each vehicle type (present or generally present for the application), so that the corresponding sensor arrangement in the training data set may be simulated, as a result of which efficient application of the classifier for different vehicle types may be made possible without the effort of additional measurement cameras.Therefore, a methodology for domain adaptation in the training dataset from the basic sensor arrangement to the vehicle-specific sensor arrangement can be used.Advantageously, signal processing on the object-specific time signal sections can provide the following steps for each of the sensors:Advantageously, an angle of attack can be taken into account, wherein an amplitude modification with respect to the horizontal and vertical directional characteristics of the sensor via the directional factor Γ can be taken into account with the ratio of the sound pressure p ~ as a function of the angles φ and θ to a reference sound pressure p ~ max at the angles φ0and θ0.Furthermore, the sensor position can be taken into account, wherein a propagation time and amplitude modification with respect to the displacement of the horizontal and vertical sensor position and the object position over the speed of sound c can be taken into account, in particular in the case of a change in the distance Δd between sensor and object back scatter point, the changed propagation time results in Δt=Δd·c and the amplitude change on the time signal p(t) over the geometric propagation attenuation results in p' (t)=d0d1p(t) with the distance d0between sensor and object in the basic sensor arrangement and the distance d1in the sensor arrangement to be simulated. In addition, the frequency-dependent atmospheric sound attenuation can be taken into account.Furthermore, a geometry of the installation environment can be taken into account. In this case, the directivity Γ can be calculated taking into account sound effects such as diffraction, reflection and scattering at elements of the installation environment. The process described so far relates to a physically based simulation of a modified sensor arrangement for the training process.A fine tuning of a pre-trained classifier for adaptation to the target domain is explained below.In order that completely new training with the processed data set can be dispensed with, fine tuning of a model pre-trained with the data set of the basic sensor arrangement can be provided. The pre-trained model may already be able to extract relevant features from present signals. As a result of the fine tuning, only slight weight adaptations are then carried out in the neural network, as a result of which the model can be adapted to the target domain. This results in a significantly reduced training duration and thus a low application effort. With few training epochs, the model can be adapted to a new vehicle type.According to FIG. 4, after the measurement value acquisition in step 1 (assumption of a basic sensor arrangement), the sampled measurement data can be filtered in a preprocessing step 2 and processed to form the training data set. With this, on the one hand, the pre-training of the CNN can take place in step 3.a and, on the other hand, a new training data set can be created by means of the simulative signal processing in step 3.b (for domain adaptation) taking into account the parameters of the sensor arrangement with which (adapted training data) the fine tuning of the CNN can then be carried out in step 4 (also taking into account the pre-training of step 3 a). Finally, in step 5, the adapted classification model can be integrated in the vehicle.Although the present invention has been fully described above with reference to the preferred exemplary embodiment, it is not limited thereto, but can be modified in a variety of ways.References included in the specificationThis list of documents cited by the applicant has been produced in an automated manner and is only included for the better information of the reader. The list is not part of the German patent application or utility model application. The DPMA does not take any adhesion for any faults or omissions.Patent Literature citedDE 10 2015 120 659 A1

[0004]

Claims

Method for an ultrasonic-based object classification, comprising the steps of: - generating (S1) a time reflection signal at a first ultrasonic sensor; - receiving (S2) and / or generating secondary signals of at least one or more ultrasonic sensors adjacent to the first ultrasonic sensor and extracting predetermined features from the secondary signals which are generated when the time reflection signal is received at the adjacent ultrasonic sensors; - transmitting (S3) the time reflection signal and the predetermined features from the secondary signals to a classifier device; - merging (S4) the time reflection signal and the predetermined features by the classifier device; - outputting (S5) an object classification by the classifier device.Method according to Claim 1, in which the classifier device comprises a neural network, or in particular a convolutional neural network (CNN), with feature maps of the first ultrasonic sensor and / or of the adjacent ultrasonic sensors.Method according to Claim 1 or 2, in which the fusion (S4) takes place in such a way that a training data set for a present sensor arrangement is taken into account.Method according to Claim 1, in which compressed feature maps from CNN convolutional layers of the adjacent ultrasonic sensors are transmitted and taken into account in addition to time signals from the first ultrasonic sensor.Method according to Claim 4, in which, in an initial step, features are extracted at the first ultrasonic sensor and at the adjacent ultrasonic sensors, and convolutional layers are used in the process and are used to merge the features feature maps into the features of the first ultrasonic sensor and are concatenated, and a further convolutional layer then uses the concatenated feature maps in the extraction of the merged features, and then smoothing of the feature data takes place and object classification takes place using a fully connected layer.Method according to one of Claims 1 to 5, in which the training data set comprises measurements of known objects and at known positions of the objects and with a fixed sensor arrangement.Method according to one of Claims 1 to 6, in which the training data set is / is individually processed for a predetermined vehicle type.Method according to one of Claims 1 to 7, in which the training dataset takes into account an angle of attack of the first ultrasonic sensor and / or of the adjacent ultrasonic sensors.Method according to one of Claims 1 to 7, in which the training dataset takes into account a sensor position with respect to a runtime and / or amplitude modification with respect to a displacement of the horizontal and / or vertical sensor position and / or object position.Method according to one of Claims 1 to 9, in which the training data set takes into account a geometry of the installation environment of the first ultrasonic sensor and / or of the adjacent ultrasonic sensors.Method according to one of Claims 1 to 10, in which the classifier device and a model which the classifier device uses are fine-tuned, predetermined weights being adapted in the neural network and an adaptation to a target domain of the sensor arrangements being effected in the process.Apparatus for carrying out an ultrasonic-based object classification, comprising a control and / or computer device which can be connected to the first ultrasonic sensor and to adjacent ultrasonic sensors, and the control and / or computer device is configured to carry out a method according to one of Claims 1 to 11.

Images