Method for learning a first neural network by controlling a device for generating training data for at least one second neural network
By designing a first neural network for generating multiple traffic scenarios, the problem of limited training data of autonomous driving vehicles is solved, the recognition and navigation capabilities of neural networks are improved, and efficient and safe autonomous driving in multiple environments is achieved .
Patent Information
- Application Number
- CN202080089587.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-12-23
- Filing Date
- 2020-12-15
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2040-12-15
AI Technical Summary
Due to the limited training data of existing autonomous driving vehicles, the neural networks of existing autonomous driving vehicles are insufficient in identifying objects and navigating, making it difficult to operate efficiently and accurately in various traffic scenarios.
By designing a first neural network, receiving environmental sensor data or artificially generated initial traffic scene data, learning to generate multiple different or ideal traffic scene data, and using it to train other neural networks to improve its recognition and navigation capabilities. The method includes inputting data into the first neural network, generating artificial ideal road images and traffic conditions for training data to improve the performance of the neural network.
By generating multiple possible traffic and driving scenarios, the performance of the neural network is improved, allowing it to identify objects and navigation with high accuracy in a wider environment, reducing test driving time and improving the safety of automated operations.
Smart Images

Figure CN114830204B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a method for learning a first neural network, in particular for generating training data for at least one second neural network, a method for learning at least one second neural network, a control device, a computer program and a machine-readable storage medium. Background Art
[0002] Vehicles with automated driving functions can be operated autonomously and without a driver from a certain degree of automation. For example, the vehicle can automatically follow the course of the road, independently identify other traffic members or obstacles and calculate corresponding control commands in the vehicle and forward these control commands to actuators in the vehicle, thereby influencing the driving process of the vehicle. In the case of fully autonomous vehicles, the driver does not participate in driving.
[0003] Currently used vehicles that can be operated automatically have a plurality of environmental sensors. These environmental sensors are used, for example, for the vehicle's environmental recognition. Based on algorithms, the sensor data of these environmental sensors can be evaluated and, for example, objects around the vehicle can be recognized. These sensor data influence the trajectory planning of the vehicle and thereby also influence the subsequent control of the vehicle actuators, thereby realizing autonomous driving functions.
[0004] In order to recognize objects from the measurement data of these environmental sensors and for trajectory planning, neural networks or artificial intelligence are used. These neural networks are essentially trained with the measurement data of the environmental sensors from as many use cases or traffic scenarios as possible in order to be able to accurately recognize objects with a high probability and in order to navigate the vehicle safely through the environment based on this. The limited number of traffic scenarios for training the neural networks can hinder the performance of these neural networks. Summary of the invention
[0005] The object of the present invention may be to specify a method for improving the performance of a neural network.
[0006] This object is achieved by means of the corresponding subject matter according to the invention. Advantageous embodiments of the invention are the subject matter of different embodiments.
[0007] According to one aspect of the present invention, a method for learning or training a first neural network is provided. The first neural network is preferably used to generate training data for at least one second neural network. The method can be performed by a control device.
[0008] In one step, measurement data determined by at least one environment sensor or data artificially generated from an initial traffic scene are received. These data or measurement data can be called up from a database or directly received from the vehicle via a communication connection.
[0009] In a next step, the received measurement data are supplied as input data to the first neural network in order to learn the first neural network.
[0010] Then, the first neural network learned based on these input data is used to generate data of traffic scenes different from the initial traffic scene. Therefore, multiple new traffic scenes can be generated, which exceed the initial traffic scene or are different from this initial traffic scene.
[0011] The first neural network may be designed as artificial intelligence. The first neural network is preferably used to generate ideal traffic scenarios, which are used as training data for other neural networks. Preferably, the ideal traffic scenario may be a traffic scenario without accidents.
[0012] The first neural network preferably has a classifier by means of which it can recognize other traffic elements and objects in the environmental sensor data and associate these with the current high-precision vehicle position of the at least one vehicle providing the measurement data. The high-precision position of the vehicle can be determined based on GNSS or based on Car-to-X or based on a feature map.
[0013] The goal of the learning process of the first neural network is to provide a neural network or artificial intelligence that can artificially generate ideal road traffic or traffic situations. Therefore, based on the measured data of the traffic scene, the first neural network is provided with an ideal road image and an ideal road traffic without accidents in the form of input data for the learning step.
[0014] After learning or training the first neural network, in a subsequent step as many artificial traffic scenarios as possible are generated with the aid of the newly learned first neural network. As a result, the learned first neural network is able to artificially generate the ideal traffic scenarios learned in the first step also for environments that do not exist in the world. In this way, the first neural network is able to generate as many driving scenarios and traffic scenarios as possible that may occur in the daily operation of the vehicle and which were not explicitly present in the original learning data. In this regard, preferably different local traffic regulations are observed and the location of structures and road courses is also taken into account. In addition, so-called 3D feature maps can also be included in the data generation for a specific vehicle position.
[0015] By means of the method, a first neural network can be trained with the aid of training data and / or real measurement data of environmental sensors for as many use cases as possible, so that objects can be accurately recognized with a high probability and in order to navigate the vehicle safely through the environment based thereon. In particular, the generation of a plurality of possible traffic scenarios can be used as a basis for a training method for the neural network in order to improve the performance of the neural network.
[0016] According to another aspect of the invention, a control device is provided, wherein the control device is configured to implement the method. The control device can be designed, for example, as cloud technology, a high performance computer (HPC) or a so-called hardware in the loop (HIL).
[0017] According to one aspect of the present invention, there is also provided a computer program, the computer program comprising the following instructions, when the computer program is executed by a computer or a control device, the instructions cause the computer or the control device to implement the method according to the present invention. According to another aspect of the present invention, there is provided a machine-readable storage medium, on which the computer program according to the present invention is stored.
[0018] The measurement data collected from real traffic scenes by environmental sensors can preferably be provided by different vehicles. The corresponding vehicles can be operated in an assisted, partially automated, highly automated and / or fully automated or driverless manner according to the BASt standard. In addition, the neural network to be learned by the first neural network can be used in such vehicles.
[0019] These vehicles can be designed as passenger cars, robots, drones, ships, rail vehicles, robot taxis, industrial robots, commercial vehicles, buses, airplanes, helicopters, and more.
[0020] The method according to the present invention provides the following advantages:
[0021] - Artificial environmental sensor data or training data can be generated by the taught first neural network, which are used as the basis for learning new algorithms or neural networks in environmental sensors or vehicle control units. The first neural network can create artificial road courses based on real road courses and traffic conditions and based on the physical properties of traffic members.
[0022] By training the first neural network or the training network on a high-performance computer and by learning new neural networks based on such training networks, a plurality of possible use cases can be considered. In this case, use cases that rarely occur in reality can also be considered.
[0023] The measurement data determined by the environmental sensors are fully taken into account. For this purpose, these measurement data are transformed into a unified world coordinate system and generated into a target coordinate system of the corresponding environmental sensor at the specified planned insertion position or installation position of the environmental sensor in the vehicle, so that the second neural network to be learned can be optimally adapted to the specific insertion position of the environmental sensor in the vehicle.
[0024] Furthermore, safety during automated operation of the vehicle can be improved by extensive training of the second neural network with the aid of artificial output data of the first neural network. The second neural network can thus recognize objects and / or traffic situations more efficiently. Subsequent or included trajectory planning in the second neural network can be performed with greater accuracy by means of the method.
[0025] The driving time actually completed by test drives in order to learn as many traffic scenarios as possible can be reduced to a minimum in a targeted manner by means of this method.
[0026] According to one embodiment, measurement data of ideal traffic scenarios determined by at least one environmental sensor of different vehicles are received and used to learn the first neural network. The first neural network is thereby trained using real environmental sensor data from daily road activities of as many vehicles as possible that can be operated automatically.
[0027] According to another embodiment, the received measurement data or the data artificially generated from the initial traffic scene are fed to the first neural network for learning the first neural network as a function of at least one sensor type of the environmental sensor, as a function of at least one installation position of the environmental sensor on the vehicle and / or as a function of at least one vehicle position. As a result, different road scenarios, different environmental sensors and different installation positions of these sensors in the vehicle can be taken into account when training the first neural network.
[0028] These environmental sensors may be, for example, radar sensors, LIDAR sensors, camera sensors, ultrasonic sensors, position sensors, and the like.
[0029] Furthermore, the method can also take into account the orientation of sensors in the vehicle. In this way, the corresponding measurement data of the surrounding sensors and the objects detected based on these measurement data are transformed into the world coordinate system and based on this, the first neural network is trained for an ideal traffic scenario.
[0030] According to another aspect of the present invention, a method for learning at least one second neural network is provided. Preferably, such a method can be performed by a control device.
[0031] The data of a plurality of traffic scenes are generated by the first neural network, and then the generated data are transmitted as input data to the second neural network to learn the second neural network.
[0032] In this way, the artificial intelligence or the first neural network can be used to train at least one further artificial intelligence or at least one second neural network for at least one object recognition, at least one autonomous driving function, at least one environmental sensor or at least one vehicle control device for an automatically operable vehicle.
[0033] The first neural network used for training purposes can be used to create a plurality of artificial driving scenarios for learning at least one second neural network. This has the advantage that it is avoided to use test vehicles that can be operated automatically to complete millions of kilometers in order to adjust the neural network used even with respect to critical traffic situations.
[0034] The method also enables the vehicle's surrounding sensors and the neural network of the control unit to learn traffic scenarios that occur rarely or not at all in practice.
[0035] In one embodiment, the output data generated by the first neural network are configured to learn an object classifier of the at least one environmental sensor based on a second neural network as a function of the sensor type and the installation location of the environmental sensor.
[0036] In particular, the output data are created by the first neural network in such a way that they are suitable for learning an object classifier of the at least one environmental sensor based on the second neural network as a function of the sensor type and the installation location of the environmental sensor.
[0037] The newly created artificial intelligence or first neural network can, for example, generate training data for the environmental sensors of the vehicle that can be operated automatically and their object classifiers, depending on the installation position of the environmental sensors in the future vehicle. For this purpose, a transformation of the input data generated from the traffic scene into the environmental sensor coordinate system can be performed. In this case, the direction in which the environmental sensors are installed or oriented with respect to the vehicle coordinates or the world coordinates is taken into account.
[0038] Furthermore, the visibility and recognizability of different sensor types can be taken into account when implementing the method. For example, lidar sensors can be used to detect different objects and object properties than radar sensors. Furthermore, the scanning area representing at least a part of the traffic scene can influence the training of the second neural network depending on the sensor type.
[0039] By means of the method, artificially generated measurement data of, for example, an environmental sensor are generated based on a transformation of the measurement data of the environmental sensor into a world coordinate system in order to train an object classifier of the environmental sensor. As a result, only objects visible to the corresponding environmental sensor are artificially generated, and the subsequent second neural network to be learned for object recognition of the environmental sensor is learned for exactly the installation position of the environmental sensor in the subsequent autonomous vehicle. This approach can reduce the required amount of input data and enables an effective and targeted training process of the second neural network. By generating as much artificial data as possible by the first neural network, the environmental sensor mentioned by way of example can be learned for as many situations and traffic scenarios as possible that rarely or not occur in road traffic. Therefore, the classifier of the environmental sensor for object recognition is also trained for as many special situations as possible in road traffic.
[0040] According to a further embodiment, the second neural network is trained by input data generated by the first neural network and by measurement data determined by the environmental sensors. This measure makes it possible to train the second neural network of the environmental sensors or the vehicle control unit with the aid of artificially generated input data of the first neural network using a combination of real sensor data. In this way, it can be ensured that the newly learned second neural network learns not only on the basis of artificial training data, but on the basis of a mixture of artificially generated training data and measured training data.
[0041] According to another embodiment, the measurement data determined by the environmental sensors are used to check the reaction of the second neural network to the traffic situation. Thus, the test of the second neural network can be carried out with the help of real environmental sensor data. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] In the following, a preferred embodiment of the present invention is explained in detail based on a very simplified schematic diagram. In this case:
[0043] Figure 1 A schematic diagram showing a traffic scene with vehicles; and
[0044] Figure 2 A schematic flow chart is shown to illustrate a method according to one specific embodiment. DETAILED DESCRIPTION
[0045] Figure 1 The schematic diagram shows a traffic scene 1 with a vehicle 2. For the sake of clarity, the traffic scene 1 has a vehicle 2, but it can preferably have a plurality of vehicles 2 which can determine measurement data and use these measurement data to train the neural network.
[0046] Preferably, the measured data of the traffic situation 1 can be determined by the environmental sensors 4, 6. The determined measured data can be preprocessed or at least temporarily stored by the vehicle-side control unit 8. In a subsequent step, the measured data of the traffic situation 1 determined by the environmental sensors 4, 6 can be transmitted to the control device 12 via the communication connection 10.
[0047] The surroundings sensors 4 , 6 may be designed, for example, as radar sensors, lidar sensors, camera sensors, ultrasonic sensors, position sensors or the like.
[0048] In the exemplary embodiment shown, the vehicle 2 has a camera sensor 4 and a lidar sensor 6. The vehicle 2 thus has two different sensor types, which are fastened to the vehicle 2 at different mounting locations 5, 7. For example, the camera sensor 4 can be fastened to the vehicle 2 in the region of the windshield 5 and the lidar sensor 6 can be fastened to the roof 7 of the vehicle 2.
[0049] The measurement data determined by the environmental sensors 4, 6 are transmitted by the vehicle-side control unit 8 to the central control device 12 and used there as input data for training the neural network. For this purpose, the control device 12 can be designed as a high-performance computer, for example.
[0050] Preferably, the vehicle 2 can be designed as a vehicle that can be operated automatically according to the BASt standard.
[0051] exist Figure 2 Detailed description of the invention A schematic flow chart is shown in FIG. 2 for explaining a first method 14 and a second method 16 according to one specific embodiment.
[0052] The first method 14 is to train a first neural network. The second method 16 is to use the first neural network to train other neural networks. These methods 14 , 16 are basically implemented by the control device 12 .
[0053] In step 18 of the first method 14, measurement data determined by at least one surroundings sensor 4, 6 or artificially generated data from an initial traffic scenario 1 are received. The traffic scenario 1 is preferably a fault-free or ideal traffic scenario 1.
[0054] The received measurement data are then fed as input data to the first neural network in order to learn the first neural network 20 .
[0055] The first neural network learned based on these input data is then used to generate 22 data of a traffic scene 1 ′ that is different from the initial traffic scene 1 .
[0056] In the second method 16 , data 22 of a traffic scene 1 ′ which is different from the initial traffic scene 1 and which is generated by the first neural network is supplied as input data to the second neural network in order to learn the second neural network 24 .
[0057] In a next step 26, the learned second neural network is implemented, for example, in a vehicle-side control unit 8 or in environmental sensors 4, 6 of vehicle 1. The second neural network is then used in at least one object recognition, at least one autonomous driving function, at least one environmental sensor or in at least one vehicle control unit of an automatically operable vehicle.
Claims
1. A method (14) for learning a first neural network by means of a control device (12) for generating training data for at least one second neural network, wherein - measurement data determined by at least one environmental sensor (4, 6) or artificially generated data from an initial traffic scene (1) are received; The received measurement data are fed as input data to the first neural network in order to learn the first neural network; - A first neural network learned based on the input data is used to generate data of a traffic scene (1') different from the initial traffic scene (1), wherein the data generated by the first neural network serves as training data for a second neural network, wherein the first neural network is able to generate as many driving scenes and traffic scenes as possible, which can occur in the daily operation of the vehicle and which are not explicitly present in the original learning data.
2. The method according to claim 1, wherein measurement data of an ideal traffic scenario determined by at least one environmental sensor (4, 6) of different vehicles (2) are received and used to learn the first neural network.
3. A method according to claim 1 or 2, wherein the received measurement data or the data artificially generated from the initial traffic scene (1) are supplied to the first neural network for learning the first neural network according to at least one sensor type of the environmental sensor (4, 6), according to at least one installation position (5, 7) of the environmental sensor (4, 6) on the vehicle (2) and / or according to the position of at least one vehicle (2).
4. A method for learning at least one second neural network by means of a control device (12), wherein a plurality of traffic scenarios (1') are generated by means of a first neural network and the generated data are supplied as input data to the second neural network for learning the second neural network, wherein the first neural network is able to generate as many driving scenarios and traffic scenarios as possible which can occur in the daily operation of the vehicle and which are not explicitly present in the original learning data.
5. The method according to claim 4, wherein the input data created by the first neural network is set up to learn an object classifier of the second neural network based on the environmental sensor (4, 6) according to the sensor type and installation position (5, 7) of at least one environmental sensor (4, 6).
6. The method according to claim 4 or 5, wherein the second neural network is trained by output data generated by the first neural network and by measurement data determined by environmental sensors (4, 6).
7. The method according to claim 6, wherein measurement data determined by environmental sensors (4, 6) are used to check the reaction of the second neural network to traffic situations.
8. A control device (12), wherein the control device (12) is configured to carry out the method (14) according to any one of claims 1 to 3 or the method (16) according to any one of claims 4 to 7.
9. A computer program product comprising instructions which, when executed by a computer or a control device (12), cause the computer or the control device to implement the method (14) according to any one of claims 1 to 3 or the method (16) according to any one of claims 4 to 7.
10. A machine-readable storage medium having a computer program stored thereon, wherein the computer program comprises instructions which, when the computer program is executed by a computer or a control device (12), cause the computer or the control device to implement the method (14) according to any one of claims 1 to 3 or the method (16) according to any one of claims 4 to 7.
