Method for training an artificial intelligence (AI) module for object classification in ultrasound data

Abstract

The present invention relates to a method for training an artificial intelligence (AI) module for object classification in ultrasound data, comprising the steps of: providing a measurement value data set on a data carrier, the measurement value data set having at least one data entry relating to an air-based ultrasound measurement value which was generated by means of beam forming of an individual ultrasound array; providing, on a data carrier, an object type data set having at least one entry relating to a type of the object; generating a training data set on the basis of the measurement value data set and the object type data set, wherein the generating of the training data set comprises the steps of: forming an input data set on the basis of the air-based ultrasound measurement values, and forming an output data set on the basis of the object type data set; training the AI module on the basis of the training data set.

Description

[0001] R.413830

[0002] - 1 -

[0003] Description

[0004] title

[0005] Method for training an artificial intelligence (AI) module for object classification in ultrasound data

[0006] State of the art

[0007] The present invention relates to a method for training an artificial intelligence (AI) module for object classification in ultrasound data, an ultrasound sensor unit and a vehicle.

[0008] Currently, numerous solutions exist for evaluating ultrasound signals in the automotive sector. Due to the increasing number of sensors on vehicles and the rising demands for quality and performance, the need for innovative and robust methods for evaluating ultrasound data is constantly growing.

[0009] The continuous reduction of weight in the vehicle sector to reduce fuel consumption, as well as increasing competition, is causing cost pressure, leading to a greater demand for cheaper and more efficient vehicle components.

[0010] Disclosure of the invention

[0011] The inventive method for training an artificial intelligence (AI) module for object classification in ultrasound data with the features of claim 1 has the advantage over the known methods that, even with low sound intensity coverage, it is possible to detect difficult, low and / or weakly reflective and / or absorbing objects and to classify them accordingly.

[0012] By using a beam-steering capability of a MEMS ultrasonic R.413830

[0013] - 2 -

[0014] Transducer arrays can direct ultrasonic waves precisely at objects to be classified and / or cover a desired area with high sound intensity to improve detection accuracy. This increases the robustness of object detection against static influences on hardware and / or environmental conditions, greater robustness against the nature of the substrate, greater robustness against different object sizes, surfaces, materials, positions, and identical or similar objects to be classified, as well as redundancy in classification through multiple signal channels.

[0015] According to the invention, this is achieved by the method for training a computer module for object classification in ultrasound data comprising the following steps:

[0016] - Providing a measurement data set on a data carrier, wherein the measurement data set includes at least one data entry about an air-based ultrasound measurement value generated by beam-forming a single ultrasound array,

[0017] - Provision on a data carrier of an object type record which contains at least one entry about a type of object,

[0018] - Generating a training dataset based on the measurement dataset and the object type dataset, wherein the generation of the training dataset comprises the following steps:

[0019] - Creating an input data set based on the air-based ultrasound measurements,

[0020] - Creating an output data set based on the object type data set,

[0021] - Training the AI ​​module based on the training dataset.

[0022] In other words, a computer intelligence (KL) module can be trained to improve object detection using ultrasonic measurements generated by beamforming. For example, the ultrasonic array can emit one or more signals at one or more different angles away from the vehicle. The reflection of these signals from the environment is converted by the array into as many electrical signals as there are array elements. The signals are pre-processed, for example, by resampling and / or filtering. The echo signals are then used for a specific area where a detected object is expected, such as up to R.413830.

[0023] - 3 - to 1 m, for the defined area in which the echo occurs. The extracted signals are transformed so that they can be used as input in a suitable representation, such as a time-frequency representation, for a trained classification module. Preferably, the representation of the array signal can be fed into the forward path of the classification model, and the probability for each class can be determined as output. Preferably, the class with the highest probability is processed further as the classification result. Preferably, several classifications are performed iteratively in close temporal sequences with identical transmission configuration to validate the probabilities of the respective classes. The determined probability values ​​can be made available to the ECU for processing within the system.

[0024] A single model can be used to process the signals together and derive the classification, particularly the object types. Different and / or parallel models can also be used for each individual sensor element of an array. Fusion of the individual object classification predictions can lead to improved accuracy and / or redundancy through rule-based evaluation. The method can be partially distributed across the system. For example, the raw signals can be preprocessed at the sensor, and the classification can run on the ECU (e.g., an autoencoder approach). Various representations of the ultrasonic signals can be used for input into classification models, e.g.,...Time-frequency representation of the echo signal created using any method; raw echo signal data at any sampling frequency; demodulated data as the baseband of the echo signal at any (lower than raw) sampling frequency; demodulated data as the complex envelope of the echo signal at any (lower than raw) sampling frequency. Various machine learning models can be used to process the input data, e.g., a CNN consisting of convolutional and linear layers for any input data; a linear neural network with demodulated data input; a transformer or LSTM; or an RNN of any structure with linear layers for classification of any input data. Any transmitted signal at any frequency can be used for classification, e.g., bursts or chirps.A detected object can also be subjected to multiple sonication scans with different wavefront angles to classify it more accurately. This can also serve as validation for R.413830.

[0025] - 4 - detected object in the sense of "avoiding false positive detection". Different models can be implemented, e.g., for classifying the object surface, object size, object height, object type, or a binary classification of "driveable" vs. "not driveable". The described models and data can also be used to classify the ground condition. The described models and data can also be used for regression of the object position to establish redundancy. The described models and data can also be used to cyclically or on demand test and verify the correct function of the sensors during operation. Targeted US-based classification can be used for gesture recognition. E.g.The trunk should open via a foot gesture (wavefront downwards) or a hand gesture (wavefront upwards). For this purpose, regular scanning is advantageous, targeting a defined area to classify gestures. The invention also relates to approaches that utilize multiple sensors in the system to illuminate an object to be classified. For example, sensor array A can be aimed at the object at an angle so that sensor array B receives the reflected signal for evaluation. Camera signals can also be used via sensor fusion approaches for improved classification.

[0026] The dependent claims describe preferred embodiments of the invention.

[0027] The procedure further preferably includes the following steps:

[0028] - Receiving a conjecture signal,

[0029] - Selecting a subset from the measurement data set and / or the input data set based on the guess signal.

[0030] One advantage of this embodiment is that if an object is already suspected to be in a certain area or at a certain distance due to a previous detection or similar method, only measurement values ​​relevant to this suspicion are used in the measurement data set or the input data set. This improves both the detection accuracy and the resource consumption.

[0031] Preferably, the air-based ultrasound measurements are generated using an ultrasound array, in particular a 2x2 array. R.413830

[0032] - 5 -

[0033] One advantage of this embodiment is that beamforming can be performed using the ultrasound array, in particular a 2x2 array, in order to improve the spatial resolution.

[0034] Preferably the method further includes the step:

[0035] - Fusion of a multitude of air-based ultrasound measurements from different individual ultrasound arrays, where the multitude of air-based ultrasound measurements were generated by beamforming, to form the input data set.

[0036] One advantage of this embodiment is that, in addition to the multitude of ultrasonic array elements on each ultrasonic sensor unit, a multitude of ultrasonic sensor units can also be used in order to further improve the spatial context.

[0037] The procedure further preferably includes the following step:

[0038] - Converting the measurement data set into a time frequency diagram to form the input data set.

[0039] One advantage of this embodiment is that by converting the measurement data set, the comparability of different entries in the measurement data set can be improved.

[0040] Another aspect of the invention relates to an ultrasonic sensor unit comprising at least one ultrasonic array configured to generate at least one measurement value of an object by means of beamforming, wherein the ultrasonic sensor unit was trained by means of the method as described above and below, wherein the ultrasonic sensor unit is configured to determine a type of object based on the measurement value.

[0041] One advantage of this embodiment is that the trained ultrasonic sensor unit can significantly improve the classification accuracy of objects in its vicinity. The ultrasonic sensor unit can transmit and receive ultrasonic measurements using beamforming to determine the type of object. R.413830

[0042] - 6 -

[0043] Preferably, the ultrasonic sensor unit is configured to determine a first probability value of the type of object, wherein the ultrasonic sensor unit is configured to determine the type of object when the first probability reaches a limit value.

[0044] One advantage of this embodiment is that the process for determining the type of object can be simplified, as the probability of the object can be determined linearly based on the threshold. For example, the ultrasonic sensor unit can detect a variety of object types, and ambiguity may arise based on the respective ultrasonic measurement. Resolving this ambiguity can be predicted for each type of probability; for instance, based on the ultrasonic measurements, it can be predicted with 80% probability that the object is a bollard, and the threshold is set at 75% probability to define the object type.

[0045] Preferably, the ultrasonic sensor unit is configured to determine a second probability value of the type of object, wherein the ultrasonic sensor unit is configured to determine the type of object based on a comparison of the first probability value with the second probability value.

[0046] One advantage of this embodiment is that when the difference between the two probability values ​​reaches a certain amount, a decision can be made regarding the type of object. For example, there might be a 70% probability that the detected object is a bollard and a 30% probability that it is a park bench. Since the probability values ​​differ by 50%, it can be assumed that the object is a bollard.

[0047] Preferably, the ultrasonic sensor unit is configured to receive and / or determine an image of the object, wherein the ultrasonic sensor unit is configured to determine the type of object based on the measured value and the image.

[0048] One advantage of this embodiment is that ambiguity in object classification can be resolved or improved with the help of the image. R.413830

[0049] - 7 -

[0050] For example, the object's contour or similar features can be used to rule out one type of object.

[0051] Another aspect of the invention relates to a vehicle which has a component which has been trained by means of the method as described above and below and / or has an ultrasonic sensor unit as described above and below.

[0052] Brief description of the drawings

[0053] Exemplary embodiments of the invention are described in detail below with reference to the accompanying drawings. The drawings show:

[0054] Figure 1 shows an ultrasonic sensor unit according to one embodiment,

[0055] Figures 2 to 4 are circuit diagrams illustrating the operation of the ultrasonic sensor unit according to one embodiment.

[0056] Figures 5 and 6 are flowcharts illustrating the functioning of the method according to one embodiment.

[0057] Figure 7 shows a vehicle according to one embodiment.

[0058] Embodiments of the invention

[0059] Preferably, all identical elements, units and / or steps in all figures are labelled with the same reference symbols.

[0060] Figure 1 shows an ultrasonic sensor unit according to one embodiment. Figure 1 shows an ultrasonic sensor unit comprising an ultrasonic array 102 configured to generate at least one measurement value of an object 104 by beamforming, wherein the ultrasonic sensor unit 100 has been trained by the method 10 as described above and below, and wherein the ultrasonic sensor unit 100 is configured to determine a type of object 104 based on the measurement value. R.413830

[0061] - 8 -

[0062] Figure 2 shows a circuit diagram 200 described as having been trained, wherein the ultrasonic sensor unit 100 is set up to determine a type of object 104 based on the measured value.

[0063] Figure 2 shows a circuit diagram 200 to illustrate the operation of the Kl module according to one embodiment. Layers 202, 206, 210, and 212 can be transformed into frequency-time diagrams 214, 216, and 218. Preferably, sub-areas 204, 208, and 210 between the respective layers can be used.

[0064] Figure 3 shows an ultrasonic sensor unit 100 according to one embodiment. The ultrasonic sensor unit 100 has an ultrasonic array 102 which is configured to emit ultrasonic signals 304, 306 towards an object 302. Preferably, the distance as well as the direction of movement 310 can be determined using a magazine 208.

[0065] Figure 4 shows a vehicle 200 according to one embodiment. The vehicle 200 is represented as an environment 400, in which an object 104 can be classified by the ultrasonic sensor unit 100. For example, the ultrasonic sensor unit 100 can generate a first measured value 402 and a second measured value 404 by means of beamforming in order to determine the type of object 104.

[0066] Figure 5 shows a flowchart illustrating the operation of method 10 according to one embodiment. Method 10 for training an artificial intelligence (AI) module for object classification in ultrasound data comprises the following steps: 1. Provide S1 on a data carrier a measurement data set, wherein the measurement data set contains at least one data entry about an air-based ultrasound measurement value generated by beamforming a single ultrasound array 102; 2. Provide S2 on a data carrier an object type data set, which contains at least one entry about a type of object 104; 3. Generate S3 a training data set based on the measurement data set and the object type data set, wherein the generation of the training data set comprises the following steps: 4. Create S4 an input data set based on the air-based ultrasound measurements; 5. Create S5 an output data set.

[0067] - 9 - based on the object type data set, training S6 of the Kl module based on the training data set.

[0068] Figure 6 shows a flowchart illustrating the operation of method 10 according to one embodiment. Method 10 has the same steps S1 to S6 as already described with reference to Figure 5. Furthermore, method 10 preferably includes the steps S7 receiving a guess signal and S8 selecting a subset from the measurement data set and / or the input data set based on the guess signal. More preferably, method 10 includes the step S9 fusing a plurality of air-based ultrasonic measurements from different individual ultrasonic arrays 102, wherein the plurality of air-based ultrasonic measurements were generated by beamforming, to form the input data set. Preferably, method 10 includes the step S10 converting the measurement data set into a time-frequency diagram to form the input data set.

[0069] Figure 7 shows a vehicle 200 according to one embodiment. The vehicle 200 preferably has a component 202 which has been trained using the method 10 as described above and below, and / or an ultrasonic sensor unit 100 as described above and below.

Claims

R.413830 - 10 - Claims 1. Method (10) for training an artificial intelligence module, or AI module, for object classification in ultrasound data, comprising the steps: - Providing (S1) on a data carrier a measurement data set, wherein the measurement data set includes at least one data entry about an air-based ultrasound measurement value, which was generated by beam-forming of a single ultrasound array (102), - Provision (S2) on a data carrier of an object type data record which has at least one entry about a type of object (104), - Generating (S3) a training dataset based on the measurement dataset and the object type dataset, wherein the generation of the training dataset comprises the following steps: - Forming (S4) an input data set based on the air-based ultrasound measurements, - Create (S5) an output data set based on the object type data set, - Training (S6) the KL module based on the training dataset.

2. Method (10) according to claim 1, further comprising the steps: - Receiving (S7) a conjecture signal, - Selecting (S8) a subset from the measurement data set and / or the input data set based on the guess signal.

3. Method (10) according to one of the preceding claims, wherein the air-based ultrasound measurements were generated by means of an ultrasound array (102), in particular a 2x2 array.

4. Method (10) according to any of the preceding claims, further comprising the step: - Fusion (S9) of a multitude of air-based ultrasound measurements from different individual ultrasound arrays (102), wherein the multitude R.413830 - 11 - were generated from air-based ultrasound measurements using beamforming to form the input data set.

5. Method (10) according to any of the preceding claims, further comprising the step: - Converting (S10) the measurement data set into a time frequency diagram to form the input data set.

6. Ultrasonic sensor unit (100) comprising at least one ultrasonic array (102) which is configured to generate at least one measurement value of an object (104) by means of beamforming, wherein the ultrasonic sensor unit (100) was trained by means of the method (10) according to one of the preceding claims, wherein the ultrasonic sensor unit (100) is configured to determine a type of object (104) based on the measurement value.

7. Ultrasonic sensor unit (100) according to claim 6, wherein the ultrasonic sensor unit (100) is configured to determine a first probability value of the type of object (104), wherein the ultrasonic sensor unit (100) is configured to determine the type of object (104) when the first probability reaches a limit value.

8. Ultrasonic sensor unit (100) according to one of claims 6 to 7, wherein the ultrasonic sensor unit (100) is configured to determine a second probability value of the type of object (104), wherein the ultrasonic sensor unit (100) is configured to determine the type of object (104) based on a comparison of the first probability value with the second probability value.

9. Ultrasonic sensor unit (100) according to one of claims 6 to 8, wherein the ultrasonic sensor unit (100) is configured to receive and / or capture at least one image of the object (104), wherein the ultrasonic sensor unit (100) is configured to determine the type of object (104) based on the measured value and the image.

10. Vehicle (200) comprising a component (202) which has been trained by the method (10) according to any one of claims 1 to 5 R.413830 - 12 - and / or an ultrasonic sensor unit (100) according to any one of claims 6 to 9.

Images