Method for training neural network

Abstract

A method for training a neural network for determining a position of an object surface (8) around a vehicle (1) on the basis of ultrasonic measurement data (10), has the following steps: a) providing a training measurement data set comprising ultrasonic measurement data (10) relating to the object surface (8) around the vehicle (1); b) providing a reference data set describing information of the object surface (8) relative to its position; and c) training the neural network based on the training measurement data set and the reference data set.

Description

Technical Field

[0001] The invention relates to a method for training a neural network, a method for determining the position of a surface of an object in the surroundings of a vehicle, a computer program product, a control unit for a vehicle and a vehicle. Background Art

[0002] Many new cars today are equipped with ultrasonic sensors and driver assistance systems. Driver assistance systems with ultrasonic sensors support the driver, in particular when parking and when the vehicle is moving at low speeds. Ultrasonic sensors determine the distance between the vehicle and objects or surfaces located in the vehicle's surroundings. Ultrasonic sensors are designed to detect multiple objects. The data from the ultrasonic sensors are evaluated and transmitted to the driver assistance system. This supports the driver by, for example, indicating the distance of the vehicle to surrounding objects acoustically and / or visually. Semi-autonomous or fully autonomous driver assistance systems can use the ultrasonic data to perform, for example, parking procedures semi-autonomously or fully autonomously.

[0003] The range of applications for ultrasonic sensors and the complexity of driver assistance systems are expanding day by day. The evaluation of ultrasonic measurement data is based on mathematical methods and therefore requires a relatively large amount of time and computing power. This means that the evaluation of the ultrasonic data is only available with a time delay. However, in order to reliably use ultrasonic sensors at higher speeds than those used during parking procedures, for example, an almost instantaneous evaluation of the ultrasonic data is required. Summary of the invention

[0004] Against this background, it is an object of the present invention to provide an improved method, in particular for faster evaluation of ultrasound data.

[0005] According to a first aspect, a method for training a neural network for determining the position of a surface of an object around a vehicle based on ultrasonic measurement data is provided. The method comprises the following steps:

[0006] a) providing a training measurement data set, the training measurement data set comprising ultrasonic measurement data related to the surface of an object around the vehicle;

[0007] b) providing a reference data set describing information about the surface of the object relative to its position; and

[0008] c) Training a neural network based on the training measurement dataset and the reference dataset.

[0009] The advantage of this method is that by training the neural network, the ultrasonic measurement data of the vehicle's ultrasonic sensor can be evaluated using the trained neural network. This makes the evaluation of the ultrasonic measurement data faster than conventionally used mathematical methods (such as the Gauss-Newton method).

[0010] The vehicle is, for example, a motor vehicle, such as a passenger car or a heavy goods vehicle.

[0011] In step a), ultrasonic measurement data are recorded, for example, using an ultrasonic sensor arranged on the vehicle.

[0012] The reference data set describes the position of the surface of the object with high precision. The reference data set is usually recorded before step a). For example, it is used together with a measurement system such as a LiDAR (Light Detection and Ranging) system, which is able to clearly and accurately detect the position of the surface of the object in space. The reference data set may contain measurement data or be based on such data already collected by sensors, in particular sensors on board a (test) vehicle or sensors integrated into a (test) vehicle. However, the reference data set or measurement data may also be collected by sensors temporarily mounted on the (test) vehicle. For example, a sensor temporarily or permanently mounted on a vehicle may be a LiDAR sensor. Therefore, the reference data set may include LiDAR measurement data.

[0013] In steps a) and b) of the method, the on-board and / or temporary sensors are installed on the same vehicle.

[0014] In steps a) and b) of the method, the surface of the objects surrounding the vehicle is identical.

[0015] The training in step c) of the method is based on a comparison of the evaluation of the training measurement data set by the untrained neural network with a reference data set (corresponding to the target output data of the trained neural network) and a subsequent adjustment of the weights within the neural network (so-called supervised learning). The training may comprise a plurality of iterative steps, each comprising: determining the positions of the surfaces of objects around the vehicle from the ultrasonic measurement data by means of the untrained neural network, followed by a comparison with the reference data set and an adjustment of the weights within the neural network.

[0016] Once the output data of the neural network is sufficiently close to the reference data set, the training in step c) of the method is completed.

[0017] For example, the neural network is a feed-forward neural network, specifically a multilayer perceptron.

[0018] According to a variant, instead of "providing a training measurement data set comprising ultrasonic measurement data about surfaces of objects in the vehicle environment", step a) comprises the following steps:

[0019] a1) Provide ultrasonic measurement data related to the surface of objects around the vehicle;

[0020] a2) identifying the intersection points of circles associated with the ultrasonic measurement data, and / or tangents to these circles;

[0021] a3) providing a training measurement data set based on the identified intersection points and / or tangent lines;

[0022] This is followed by steps b) and c). In other words, a so-called "heat map" is created, which represents the input data for training the neural network. For example, identifying intersection points or tangent lines as described in step a2) can be performed using the Gauss-Newton method.

[0023] According to one embodiment, the training measurement data set comprises ultrasonic measurement data acquired at different times, wherein tracking data is generated by a tracking process based on the ultrasonic measurement data acquired at different times, wherein the training of the neural network in step c) is also performed based on the tracking data.

[0024] Tracking data generated by applying the tracking process includes, for example, the number of associations or the number of missed associations.The tracking process can be used to check the credibility of ultrasound data based on its temporal classification in physical and / or statistical models, and ignore or correct untrustworthy data points.

[0025] Therefore, the training in step c) may be performed based on the training measurement data set, the reference data set and the tracking data.

[0026] According to another embodiment, the tracking process includes a Kalman filter.

[0027] A Kalman filter is a mathematical method for iteratively estimating parameters to describe the state of a system. Alternatively or additionally, a Bernoulli filter or a multi-Bernoulli filter may be used.

[0028] According to another embodiment, the training of the neural network in step c) is also performed based on the tracking speed.

[0029] Tracking speed refers to the cycle time of the (iterative) tracking process. The shorter the cycle time, the more precise the tracking. This in turn affects the quality of the neural network output data and is advantageously taken into account in its training.

[0030] According to a further embodiment, the vehicle speed is determined during the acquisition of the ultrasonic measurement data, wherein the training of the neural network in step c) is also performed based on the vehicle speed.

[0031] This enables the trained neural network to be used over a wider range of vehicle speeds.

[0032] According to a further embodiment, the signal-to-noise ratio is determined during the acquisition of the ultrasound measurement data, wherein the training of the neural network in step c) is also performed based on the signal-to-noise ratio.

[0033] This takes into account the (in)accuracies of the ultrasonic measurement data when applying the trained neural network in the vehicle.

[0034] According to a second aspect, a method for determining the position of a surface of an object around a vehicle based on ultrasonic measurement data is provided. The method comprises the following steps:

[0035] a) collecting ultrasonic measurement data related to the surfaces of objects around the vehicle; and

[0036] b) Determining the position of the surface of the object using the trained first neural network and the acquired ultrasonic measurement data.

[0037] The advantage of the method is that the positions of object surfaces in the vehicle's surroundings are determined using a trained neural network based on ultrasonic measurement data acquired by means of an ultrasonic sensor of the vehicle. This allows for a faster evaluation of the ultrasonic measurement data.

[0038] The ultrasonic measurement data acquired in step a) of the method according to the second aspect is acquired using an ultrasonic sensor that is the same, identical or different from the training measurement data in the method according to the first aspect.

[0039] The neural network used in step b) has been trained according to the method of the first aspect.

[0040] In an embodiment of the method according to the second aspect, in step b), a second trained neural network may be applied to output data of the first trained neural network to generate features for defining the position of the surface of the object from said output data.

[0041] For example, the generated features may be geometric shapes. Alternatively, the features may be standardized geometric shapes.

[0042] According to another embodiment, the feature comprises a plurality of ellipses having specified positions and / or orientations, uncertainties and / or types.

[0043] Thus, the object surface can be described by a plurality of ellipses. At least one ellipse is assigned to the object surface around the vehicle. The position, orientation, uncertainty and / or type of the ellipse describes properties of the object surface around the vehicle.

[0044] According to a third aspect, there is provided a computer program product comprising instructions, which, when executed by a computer, cause the computer to perform the above method according to the first aspect or the above method according to the second aspect.

[0045] The computer program product according to the third aspect may be provided, for example, on a computer-readable storage medium, such as a memory card, a USB flash drive, a CD-ROM or a DVD. Alternatively, the computer program product may also be provided as a file downloadable from a server on a network. The transmission of the computer program product may be performed, for example, by transmitting a file corresponding to the computer program product in a wireless communication network.

[0046] According to a fourth aspect, a control unit for a vehicle is provided, comprising an acquisition unit for acquiring ultrasonic measurement data related to a surface of an object around the vehicle, and a determination unit for determining a position of the surface of the object using the trained first neural network as described above and the acquired ultrasonic measurement data.

[0047] The control unit, for example in the form of a central vehicle control unit or an electronic control unit ("ECU"), is particularly designed for processing the above-mentioned computer program product for determining the position of the surface of objects in the vehicle's surroundings, for example on a processor unit of the control unit. Furthermore, the control unit comprises a collection unit for receiving ultrasonic sensor data, in particular collected by on-board sensors. The collection unit can receive the ultrasonic sensor data in a wired and / or wireless manner. Furthermore, the collection unit can be designed to receive, in addition to the ultrasonic sensor data, further sensor data from sensors on the vehicle.

[0048] The units described herein, such as the acquisition unit, may be implemented as hardware and / or software.

[0049] According to a fifth aspect, a vehicle is provided having one or more ultrasonic sensors and a control unit according to the fourth aspect.

[0050] The present invention will be explained in more detail below based on preferred embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 shows a schematic plan view of a vehicle having an ultrasonic sensor and a control unit according to an exemplary embodiment;

[0052] Figure 2 A schematic diagram of a vehicle control unit according to an exemplary embodiment is shown;

[0053] Figure 3 A flow chart illustrating a method for training an untrained neural network according to an exemplary embodiment is shown;

[0054] Figure 4 A flow chart showing a method for determining the position of a surface of an object around a vehicle based on ultrasonic measurement data;

[0055] Figure 5 a schematic diagram showing an ellipse used to describe features defining the location of a surface of an object; and

[0056] Figure 6 A schematic diagram showing ultrasonic measurements by an ultrasonic sensor of a vehicle while driving at a surface of an object.

[0057] Unless stated otherwise, identical or functionally identical elements are provided with the same reference symbols in the figures. DETAILED DESCRIPTION

[0058] Figure 1 1 shows a schematic plan view of a vehicle 1 according to an embodiment. Figure 1 In the example shown, the vehicle 1 is a motor vehicle, in particular a passenger car.

[0059] The vehicle 1 may comprise an ultrasonic sensor 2 which may be mounted at different locations. The ultrasonic sensor 2 may be implemented as part of a driver assistance system.

[0060] The driver assistance system is designed, for example, to determine the distance from the vehicle 1 to the object surface 8. Furthermore, the driver assistance system can be designed to communicate the determined distance to the driver, for example by means of an acoustic and / or visual display.

[0061] The ultrasonic measurement data 10 ( Figure 2 ) is generated by transmitting ultrasonic waves 4 (here only the sensors shown in the upper figure) by means of ultrasonic sensors 2 and receiving ultrasonic waves 4 reflected at the surface 8 of the object by means of ultrasonic sensors 2. The ultrasonic measurement data 10 thus generated are transmitted from the ultrasonic sensors 2 to the control unit 6 in the vehicle 1 wirelessly and / or by wire.

[0062] Figure 2 The control unit 6 shown comprises a processor unit 14 (also a “determination unit” in this case) and a memory unit 16. The control unit 6 is designed to carry out the following method, preferably by means of software 18 stored on the memory unit 16 and executed by means of the processor unit 14, for determining the position of an object surface 8 in the vicinity of the vehicle 1 based on ultrasonic measurement data 10. The control unit 6 also comprises a capture unit 12 for capturing ultrasonic measurement data 10 of an ultrasonic sensor 2, which is in particular on the vehicle or permanently integrated in the vehicle. The control unit 6 can output features 22 for determining the position of an object surface 8 in the vicinity of the vehicle, in particular via an output unit 20, to a driver assistance system or other system of the vehicle 1.

[0063] Figure 3A flow chart of a first exemplary embodiment of a method for training a neural network to determine the position of surfaces of objects around a vehicle based on ultrasonic measurement data is shown.

[0064] Typically, training is performed before manufacturing the vehicle 1. For example, a test vehicle 1' can be used for this purpose. The test vehicle 1' can be the same, identical or a different vehicle than the vehicle 1. The test vehicle 1' can also include at least one, but preferably a plurality of ultrasonic sensors 2, and for example at least one lidar sensor (not shown).

[0065] The neural network trained with the test vehicle 1' can be introduced into the control units 6 of the vehicles 1 produced on the production line, for example at the end of the production line. The training can be carried out on a vehicle model-specific basis.

[0066] In a first step S1 of the method, a training measurement data set containing ultrasonic measurement data of object surfaces 8 of objects in the environment of the test vehicle 1 ′ is provided by means of the ultrasonic sensors 2 of the test vehicle 1 ′. The training measurement data set may include, for example, signal flight times of the respective ultrasonic signals, the current position of the transmitting and / or receiving ultrasonic sensors 2, time stamps of the respective ultrasonic signals and / or information about which of the plurality of ultrasonic sensors 2 has transmitted or received the respective ultrasonic signal.

[0067] In a second step S2 of the method, a reference data set is provided, which describes information about the surface of the object relative to its position. For example, the reference data set can also be acquired using a test vehicle 1 '. For example, the reference data set can be acquired using a LiDAR sensor or other high-resolution sensors. For this purpose, sensors installed on the test vehicle 1 ' only for recording the reference data set can be used. However, it is also conceivable to use on-board sensors to create the reference data set.

[0068] The reference data set provided in S2 describes the object surfaces 8 of the same objects around the vehicle 1 as the training measurement data set provided in S1, but with a higher precision. Ultimately, for example, the reference data set corresponds to an exact mapping of the object surfaces. Information about the object surfaces 8 must first be extracted in a suitable manner from the training data set, for which the neural network is trained accordingly, as explained in conjunction with step S3.

[0069] In an optional step S3, a tracking process is applied to the training measurement data set recorded in step S1. By applying the tracking process in step S3, a plurality of ultrasonic measurement values ​​acquired at different times can be correlated with each other. This takes the form of so-called associations or missed associations within the ultrasonic measurements. For example, the tracking process is based on a Kalman filter, in particular with a defined cycle time (tracking speed). Thus, data points within the training data set can be improved, added and / or eliminated.

[0070] In step S4 of the method, an untrained first neural network is trained. The training of the untrained neural network occurs in at least one iterative step. In this process, training data (which may have been improved using the tracking process in step S3) is input into the neural network and the results output by the neural network are compared with a reference data set. Based on the comparison and deviation between the output and the reference data set, the weights within the neural network are adjusted. The output variable of the neural network is the position of the object surface 8 around the vehicle.

[0071] In a variation of the method, other input variables, such as vehicle speed during acquisition of the ultrasonic measurement data 10 , tracking speed during acquisition of the ultrasonic measurement data 10 and / or signal-to-noise ratio, may be provided to the first untrained or partially trained neural network.

[0072] The training of the neural network ends when the deviation of the output of the neural network from the reference data set is below a predetermined threshold and / or when the object surface 8 is approximated sufficiently accurately.

[0073] Figure 4 The method for determining the position of an object surface 8 around a vehicle 1 by means of two trained neural networks 24, 26 is shown. The trained neural networks 24, 26 are implemented as software 18 on a memory unit 16 of a control unit 6, for example.

[0074] The ultrasonic measurement data 10 collected using the ultrasonic sensor 2 of the vehicle 1 are analyzed by a tracking process S3 using a tracking algorithm 28. The data from the tracking process and optionally additional ultrasonic measurement data 10 (depending on the output of the tracking algorithm 28) are input into a first trained neural network 24. The output of the first trained neural network 24 is the position of the object surfaces 8 of the objects around the vehicle 1.

[0075] The output of the first neural network 24 is the input to a second trained neural network 26. Using the input data, the second trained neural network 26 determines features 22 defining the position of the surface 8 of the object.

[0076] Figure 5 Features 22 are shown for defining the position of the surface 8 of the object. Figure 5 In FIG. 2 , the features for defining the position of the object surface 22 are shown as ellipses as an example. However, the features for defining the position of the object surface 22 may also be different geometric shapes.

[0077] Figure 5 The ellipse shown in φ describes, by its position and angle, the position of an object surface 8 of an object in the vicinity of the vehicle 1. Its axis describes the uncertainty of the position of the object surface.

[0078] Figure 6 A schematic diagram of an ultrasonic measurement of a vehicle 1 at an object surface 8 around the vehicle 1 is shown.

[0079] The vehicle 1 is traveling in the direction of the arrow at an object surface 8 . The ultrasonic sensor 2 emits ultrasonic waves 4 . These are reflected at the object surface 8 . The resulting ultrasonic measurement data 10 are evaluated by the ultrasonic sensor 2 and transmitted to the control unit 6 .

[0080] The memory unit 16 of the control unit 6 stores software 18, which is based on Figure 4 The diagram shown evaluates received ultrasound measurement data 10. Two trained neural networks 24, 26 are used.

[0081] according to Figure 4 The output variable of the evaluation of the graph in is a feature 22 (here an ellipse) for defining the position of the object surface 8.

[0082] Figure 6 It is shown how two examples of ellipses 22 , 22 ′ describe the shape of the object surface 8 .

[0083] Although the present invention has been described based on exemplary embodiments, it can be modified in many ways.

[0084] Reference numerals list

[0085] 1 Vehicle

[0086] 1' test vehicle

[0087] 2 Ultrasonic Sensor

[0088] 4 Ultrasound

[0089] 6 Control unit

[0090] 8 Surface

[0091] 10 Ultrasonic measurement data

[0092] 12 Collection Unit

[0093] 14 Processor Units

[0094] 16 Storage Units

[0095] 18 Software

[0096] 20 Output Units

[0097] 22 Features

[0098] 24 First trained neural network

[0099] 26 Second trained neural network

[0100] 28 Tracking Algorithm

[0101] S1-S4 Method Steps

Claims

1. A method for training a neural network to determine the position of an object surface (8) around a vehicle (1) based on ultrasonic measurement data (10), the method comprising the following steps: a) providing a training measurement data set, the training measurement data set comprising ultrasonic measurement data (10) associated with an object surface (8) around the vehicle (1); b) providing a reference data set, the reference data set describing information about the surface of the object (8) relative to its position; as well as c) Training a neural network based on the training measurement data set and the reference data set.

2. The method of claim 1, wherein: The training measurement data set comprises ultrasonic measurement data (10) acquired at different times, wherein tracking data are generated based on the ultrasonic measurement data (10) acquired at different times by means of a tracking process, wherein the training of the neural network in step c) is also performed based on the tracking data.

3. The method of claim 2, wherein: The tracking process includes a Kalman filter.

4. The method according to claim 2 or 3, wherein: The training of the neural network in step c) is also performed based on the tracking speed.

5. A method as claimed in any one of the preceding claims, wherein: The vehicle speed is determined during the acquisition of the ultrasonic measurement data (10), wherein the training of the neural network in step c) is also performed based on the vehicle speed.

6. A method as claimed in any one of the preceding claims, wherein: During the acquisition of the ultrasound measurement data (10), a signal-to-noise ratio is determined, wherein the training of the neural network in step c) is also performed based on the signal-to-noise ratio.

7. A method for determining the position of an object surface (8) around a vehicle (1) based on ultrasonic measurement data (10), comprising the following steps: a) collecting ultrasonic measurement data (10) related to the surface of an object (8) around the vehicle (1); and b) Determining the position of the object surface (8) using a first neural network (24) trained according to any of the preceding claims and the acquired ultrasonic measurement data (10).

8. The method of claim 7, wherein: In step b), a second trained neural network (26) is applied to the output data of the first trained neural network (24) to generate features for defining the position of the object surface (22) from the output data.

9. The method of claim 8, wherein: The features include a plurality of ellipses having specified positions and / or orientations, uncertainties, and / or types.

10. A computer program product comprising instructions for causing a computer to carry out the method as claimed in any one of the preceding claims during execution of the program by the computer.

11. A control unit (6) for a vehicle (1), comprising: A collection unit (12) for collecting ultrasonic measurement data (10) related to the surface of an object (8) around the vehicle; as well as A determination unit (14) for determining a position of a surface (8) of an object using a first neural network (24) trained according to any one of claims 1 to 6 and the acquired ultrasonic measurement data (10).

12. A vehicle (1), comprising: one or more ultrasonic sensors (2); as well as A control unit (6) as claimed in claim 11.

Images