A virtual training method for neural networks to operate technical devices.

Abstract

To provide a method for training a neural network for actuating a technical device in a virtual training environment.SOLUTION: In a computing system 2, a method includes establishing a first data link between a neural network 8 and a first simulation 14a of a technical device and training the neural network by actuating the first simulation. Once a first training goal has been achieved, the first data link is broken, and a second data link is established between the neural network and a second simulation of the technical device to train the neural network by actuating the second simulation 14b. The second simulation is configured to be more realistic than the first simulation and requires more mathematical operations than the first simulation. The second simulation trains the neural network.SELECTED DRAWING: Figure 1

Description

[Background technology]

[0001] In recent years, neural networks have brought about significant advances in the development of autonomous systems capable of independently performing complex motion patterns or monitoring tasks in changing environments. However, a problem in this regard is the sheer volume of training data required to adequately train neural networks, particularly when they are intended for safety-critical tasks where misrecognition could have catastrophic consequences. Examples include self-driving cars intended for road traffic, smart surveillance cameras that recognize people drowning in lakes, or rescue robots that search for injured persons in a region or building. These types of systems require not only a large amount of diverse training data to optimally eliminate malfunctions caused by insufficient or one-sided training, but the training data must also contain a large amount of data depicting catastrophic or critical scenarios that the neural network should be trained to reliably recognize and process. Since such scenarios occur rarely by their nature, corresponding data collection is also rare, which means that collecting sufficient training data represents an enormous amount of work.

[0002] One known solution to this problem from prior art is to generate synthetic training data using computer simulations. In this case, scenarios for training a neural network are reconstructed in a virtual environment, and training data is generated in a format that the neural network can understand based on these reconstructed scenarios. In this way, the virtual scenarios can be easily modified as desired. However, since synthetic data generally does not perfectly replicate real-world data, there is a fundamental risk that the neural network will become accustomed to the synthetic training data and will no longer be adequately trained for use in the real world.

[0003] The scientific paper "Synthetic Data for Deep Learning" by Sergey I. Nikolenko (2019) includes a comprehensive overview of the relevant prior art. The paper describes various different examples of neural networks successfully trained in virtual environments. However, these academic studies only express the fact that it is possible in principle. The fact that the more realistic the synthetic training data becomes, the higher the reliability of the neural networks trained using those synthetic data does not change thereby. **Summary of the Invention** **Problems to be Solved by the Invention**

[0004] The present invention relates to training a neural network for operating a technical device using synthetic training data. In this case, when the technical device is a device subject to strict safety requirements, for example, a highly automated vehicle, in order to be able to reliably transfer the capabilities of the neural network learned in a virtual environment to the real world, synthetic training data with a high degree of realism must be used for training. Therefore, it is necessary to provide a virtual simulation of the technical device that gives the neural network almost a complete and outstanding illusion that it is operating the technical device in the real world. However, the realism of the simulation model is inversely proportional to its execution speed. This is because the more technical components of the device as well as physical phenomena and interactions are considered in the simulation, the more realistically the simulation model of the technical device behaves. Therefore, the requirement to seek the highest possible realism conflicts with the requirement to seek the fastest possible execution speed of the simulation in order to complete the training within a reasonable time frame.

[0005] Against this backdrop, the objective of the present invention is to speed up the training of a neural network for operating a technical device in a virtual training environment.

[0006] German Patent Application Publication No. 102018220865 proposes a training method that begins in a purely virtual training environment and gradually replaces the virtual training environment with a real-world training environment in order to gradually acclimate the neural network to a more complex real world in subsequent training phases. While this method allows for the use of relatively simple simulations, the subsequent training phases depend on real-world training data, which has the aforementioned drawbacks. [Means for solving the problem]

[0007] The present invention relates to a method for training a neural network for operating a technical device and a virtual training environment for carrying out the method. According to the present invention, a first data link is established between the neural network to be trained and a first simulation of the technical device. For this purpose, the first simulation is equipped with a first data interface, through which the neural network can read state data from the first simulation and transfer control data to the first simulation.

[0008] Preferably, the first data interface replicates a real-world data interface of the technical device, and the neural network is intended to use the aforementioned data interface to operate the real-world technical device once it has been trained and validated. State data provides neural network information about the current state of the technical device and should be interpreted as data required by the neural network to operate the technical device. Control data should be interpreted as data suitable for changing the current state of the technical device toward a target state of the technical device specified by the neural network. If the technical device is a land-based vehicle, for example, possible state data may be the distance from a line marking the edge of the road, the vehicle's speed, the distance from the vehicle ahead, the gear meshing, the vehicle's roll angle, or a description of the vehicle's surroundings, and possible control data may be the steering angle, the throttle valve opening, or the braking force.

[0009] In addition, a first training objective is set to activate the first simulation, the neural network is trained by the first simulation which is being operated by the neural network, and the training progress of the neural network against the first training objective is checked periodically.

[0010] According to the present invention, training of a neural network using a first simulation continues until the neural network achieves a first training objective. Once the first training objective is achieved, the first data link is disconnected, and a second data link is established between the neural network and a virtual second simulation. The second simulation is configured to be more realistic than the first simulation, and due to its higher realism, it requires more mathematical operations per simulation cycle than the first simulation. The second data link is established by a second data interface of the second simulation for reading state data from the second simulation and for transferring control data to the second simulation.

[0011] A simulation cycle refers to the complete execution of one simulation step, specifically one time step, of a simulation.

[0012] Therefore, according to the present invention, in the initial training phase, a relatively simple but fast simulation model of the technical device is used. As soon as the neural network has sufficiently mastered the operation of the simple simulation model, as demonstrated by the achievement of the training objective, the simple simulation model is replaced by a more complex and slower but more realistic simulation model, and training continues with the second simulation operated by the neural network.

[0013] A fundamental assumption of this invention is that the simulation model does not need to be highly realistic in the initial training phase, because the neural network is still learning the fundamentals at this point. To illustrate this, it is assumed that the neural network should be trained to steer an autonomous road vehicle based on the method of the present invention. An untrained neural network would first need to understand what its task actually is (keeping the vehicle between the right-hand edge marking and the center line) and how to learn this task in basic terms (a negative steering angle makes the vehicle move to the left, and a positive steering angle makes the vehicle move to the right).

[0014] To learn these fundamentals, a very simple simulation of a vehicle is sufficient. As soon as the neural network has sufficiently learned these fundamentals, the first simulation is replaced by a second simulation of the vehicle, which takes into account more complex phenomena, such as roll angle, instability in curves, driving comfort, skidding conditions, or crosswinds. The concept of the present invention is a special application of so-called "transfer learning," in which the previously learned capabilities of the neural network are used as an advantage for learning another capability (in this case, operating a more realistic simulation). In training scenarios where very complex and realistic simulations must be used for training for safety reasons, the present invention provides significant time savings.

[0015] Advantageously, a second training objective is set to activate a second simulation, and the neural network is trained by activating the second simulation until the second training objective is achieved. For this purpose, the training progress of the neural network against the second training objective is checked while the second simulation is running. The second training objective may be the same as the first training objective, or it may be different from the first training objective. Once the second training objective is achieved, the training can be considered complete, and a physical data link can be established between the neural network and the real-world technological device to verify the training of the neural network in the real world. However, it goes without saying that training in a virtual training environment can be continued before real-world verification by activating a third simulation and any number of further simulations of the technological device, in which case the realism of each simulation of the technological device is higher than that of the preceding simulations in each case. In both cases, terminating the simulation once the relevant training objectives are achieved and moving on to the next relevant simulation will be carried out in the same manner as described for the examples of the first and second simulations.

[0016] The first and second data interfaces are preferably configured identically, so that the first data link can be replaced immediately with the second data link without modifying the data exchange that operates on the neural network.

[0017] Simulating a technical device advantageously involves generating simulated sensor data that contains information about the technical device's virtual environment. Naturally, the nature of this sensor data depends on the technical device and the task to which the neural network is trained to accomplish. For example, the sensor data may be an object list generated based on simulated moving or stationary objects in the virtual environment, lane boundaries, polygon-based reproductions of surfaces in the environment, or raw data from image-generating sensors, particularly cameras, lidars, radars, or echolocation sensors. The virtual sensor data is transferred to a neural network, which is trained to evaluate the sensor data and to take the sensor data into account when operating the associated simulation and ultimately the technical device.

[0018] Needless to say, the simulated sensor data must be generated in a format that neural networks can understand, and in particular, in the same format in which real-world sensor data is also available for use in technical devices. Conventional technologies offer solutions for appropriately simulating sensor data based on a virtual environment; see, for example, the scientific paper “Development of Full Speed ​​Range ACC with SiVIC, a virtual platform for ADAS Prototyping, test and evaluation” Dominique Gruyer et al., IEEE Intelligent Vehicle Symposium (2013).

[0019] In an extended stage of the present invention, it is also possible to simultaneously modify the virtual environment in order to increase the complexity of the simulation of the technical device. These modifications may be purely geometric, ensuring that the neural network learns to generalize rather than simply memorize behavioral patterns in a particular virtual environment. However, these modifications may also be extensions to the virtual environment that have a fundamental impact on the behavior of the simulation of the technical device, thereby making the simulation behavior more realistic and, in some cases, more demanding on the neural network.

[0020] The technical device is preferably a robot. A robot should be interpreted as any device that is capable of performing independent and complex motion patterns to carry out a predetermined action, even if it is operated by a computer, particularly by a neural network. In particular, the technical device may be at least a partially automated aircraft, ship, or aircraft, robotic arm, robot for positioning or mounting materials and / or objects, robot for cleaning surfaces, robot for inspecting spaces or surfaces, robot for applying chemicals, such as cleaning agents, disinfectants, paints, varnishes, or coatings, robot for performing medical procedures, or any combination of the robot categories described above.

[0021] The training objectives may be configured to vary depending on the type of technical device and the task the neural network is to learn. For example, the training objectives may be a predetermined section along a virtual training course or virtual test route, a predetermined number of movement patterns performed without collisions or within a predetermined range of movement, a predetermined period of time during which the neural network successfully operates the first simulation without any undesirable events occurring, or a predetermined threshold reached for the underlying reward function during the training of the neural network. The reward function is used periodically to train the neural network for the purpose of evaluating during training whether the output of the neural network is improving compared to the previous training phase. The reward function is defined such that the result of this reward function can indicate how well the neural network has mastered its particular task. The neuronal and synaptic configurations that have resulted in improvement are retained as a basis for further training steps, while configurations that have resulted in impairment are rejected.

[0022] There are various options for configuring the second simulation to be more realistic than the first simulation. For example, the second simulation can consider more mechanical and / or electrical components of the technical device than the first simulation, can consider more mechanical degrees of freedom, can simulate physical phenomena and / or laws more realistically or with greater accuracy, can consider a larger number of physical forces and / or interactions, or can have a smaller simulation step size.

[0023] The drawings and the following description of the drawings will illustrate the present invention in more detail based on specific examples. [Brief explanation of the drawing]

[0024] [Figure 1] This is a schematic diagram of a virtual training environment. [Figure 2]It is a flowchart of a training operation. [Figure 3] It is a diagram showing a virtual environment for the simulation of a highly automated vehicle. [Figure 4] It is a diagram showing a sensor simulation for simulating lane recognition in a highly automated vehicle. [Figure 5] It is a diagram showing a first simulation of a highly automated vehicle. [Figure 6] It is a diagram showing a second simulation of a highly automated vehicle. [Figure 7] It is a diagram showing a third simulation of a highly automated vehicle. **[Embodiment for Carrying Out the Invention]**

[0025] The drawing of FIG. 1 shows a computer system 2, on which a virtual training environment 4 and a development environment 6 for a neural network 8 are set up. The neural network 8, which is intended to operate a technical device and is to be trained for that task, is embedded in the development environment 6. The development environment 6 is logically separated from the training environment 4 and includes, in addition to the neural network 8, a first development routine 10 for executing a reward function T and a second development routine 12. The second development routine 12 is configured to monitor the function value of the reward function T and to change the configuration of the neural network 8 based on this function value, i.e., to change the values stored in the neurons and synapses of the neural network 8 in order to optimize the function value of T by a predetermined method. The function value of T depends on a matrix x, and the entries in the matrix x are supplied by the virtual training environment 4.

[0026] A development environment such as the illustrated development environment 6 is known in the prior art and is commercially available. Examples include PyTorch and TensorFlow.

[0027] For example, the virtual training environment 4 includes three simulations of the technical device, namely a first simulation 14a, a second simulation 14b, and a third simulation 14c. The first simulation 14a includes a first data interface 16a for the neural network 8 to operate the first simulation 14a. While being trained on the first simulation 14a, the neural network 8 reads state data from the first simulation 14a and transfers control data to the first simulation 14a. Similarly, the second simulation 14b includes a second data interface 16b, and the third simulation 14c includes a third data interface 16c. Both the second data interface 16b and the third data interface 16c are functionally identical to the first data interface 16a. As a result, the three simulations 14a, 14b, and 14c can be instantly swapped with each other, and the neural network 8 cannot directly identify which of the three simulations 14a, 14b, and 14c it is currently running.

[0028] The degree of realism of these simulations of the technical device gradually increases, namely the second simulation 14b is configured to be more realistic than the first simulation 14a, and the third simulation 14c is configured to be more realistic than the second simulation 14b. As a result of the increasing degree of realism of each of these simulations, the simulation cycle of the second simulation 14b requires more mathematical calculations to be performed by the computer system 2 than the simulation cycle of the first simulation 14a, making the second simulation 14b slower than the first simulation 14a, and the simulation cycle of the third simulation 14c requires more mathematical calculations to be performed by the computer system 2 than the simulation cycle of the second simulation 14b, making the third simulation 14c even slower than the second simulation 14b.

[0029] The virtual training environment 4 includes one of three simulations 14a, 14b, and 14c, and a first virtual multiplexer 20 for establishing a data link between the virtual training environment 4 and the programming interface 18. The development environment 6 is connected to the training environment 4 by the programming interface 18, which allows for the establishment of a first data link between the neural network 8 and the first simulation 14a, or a second data link between the neural network 8 and the second simulation 14b, or a third data link between the neural network 8 and the third simulation 14c, as a result of the positioning of the first virtual multiplexer 20.

[0030] Similar to the simulations 14a, 14b, and 14c of the technical device, the virtual training environment 4 also includes a virtual environment 22 of the technical device, and a second virtual multiplexer 21 is configured to establish a data link between this virtual environment 22 and one of the simulations 14a, 14b, and 14c. The virtual environment 22 is a reproduction of a typical environment of the technical device and simulates typical interactions between the technical device and the real-world environment. One of the three simulations 14a, 14b, and 14c is incorporated into the virtual environment 22 based on the data link established by the second virtual multiplexer 21, so that the interaction between the relevant simulations 14a, 14b, and 14c and the virtual environment 22 can be considered by the training environment 4. The first simulation 14a, the second simulation 14b, and the third simulation 14c each also include at least one sensor simulation 30, which simulates a real-world sensor of a technical device, generates simulated sensor data 34, and transmits this simulated sensor data to the neural network 8, from which the neural network 8 can read information about a virtual environment 22.

[0031] The training function 24 is configured to operate the first virtual multiplexer 20 and the second virtual multiplexer 22 so that both of these multiplexers are connected to the same simulations 14a, 14b, and 14c of the technical device at any given time, thereby ensuring that a continuous and indirect data link exists between the virtual environment 22 and the neural network 8 via the first simulation 14a, the second simulation 14b, or the third simulation 14c at any given time. In addition, the training function 24 is configured to read the first training objective for the first simulation 14a, the second training objective for the second simulation 14b, and the third training objective for the third simulation 14c from the training objective memory 26. An operator computer 28 containing configuration software is connected to the computer system 2, allowing the user to set the first, second, and third training objectives and store them in the training objective memory 26 via the operator interface of the configuration software.

[0032] Figure 2 illustrates the steps taken by the training function 24 to train the neural network 8. Initially, the training function 24 connects the first simulation 14a to the neural network 8 by activating the first virtual multiplexer 20, thereby establishing a first data link. Next, the training function 24 connects the first simulation 14a to the virtual environment 22 by activating the second virtual multiplexer 21. Thus, the training function establishes a continuous data link between the neural network 8 and the virtual environment 22, mediated by the first simulation 14a, allowing the neural network 8 to operate the first simulation 14a in the virtual environment 22. At this point, the training function 24 begins training the neural network 8 with the first simulation 14a being operated by the neural network 8, and during training, continuously checks the training progress of the neural network 8 against the first training objective, continuing training until the first training objective is achieved.

[0033] Since the execution speed of the first simulation 14a is fast, the first training objective is achieved in a relatively short time. Advantageously, the configuration of the first simulation 14a and the first training objective are selected so that, once the first training objective is achieved, the neural network 8 learns the foundation for operating the technical device.

[0034] Once the first training objective is achieved, the training function 24 connects the neural network 8 to the second simulation 14b by activating the first virtual multiplexer 20. In this way, the training function 24 disconnects the first data link and replaces it with the second data link between the neural network 8 and the second simulation 14b. Next, the training function 24 connects the second simulation 14b to the virtual environment by activating the second virtual multiplexer 21. Here, the second simulation can be incorporated into the virtual environment 22 in the same way and operated by the neural network 8, just as the first simulation 14a was operated previously. Similarly, the training function 24 starts and monitors the training and continues training until the second training objective is achieved. Similarly, a third training phase, including the third simulation 14c, is performed until the third training objective is achieved. Once the third training objective is achieved, the training function 24 terminates the training.

[0035] The final third simulation 14c is preferably a very complex and realistic simulation of the technical device, and the configuration of the third training objective and the third simulation is preferably selected so that, once the final third training objective is achieved, the neural network 8 learns how to operate the technical device safely even under rare and exceptional circumstances.

[0036] Once training is completed in the virtual training environment 4, a data link can be established between the neural network 8 and the technical device in order to operate the technical device in the real-world environment, for the purpose of verifying the training conducted in the virtual training environment 4 or for the purpose of continuing the training in the real-world environment.

[0037] The diagram in Figure 3 shows a virtual training environment for training a neural network to operate a highly automated road vehicle. The virtual environment 22 replicates a typical environment for a highly automated road vehicle and for this purpose includes numerous stationary and moving 3D objects O1,...,O8 such as vehicles, vegetation, traffic signs, and buildings, as well as a virtual test route 32 that replicates a road. In the illustrated example, the technical device is a road vehicle, and a first simulation 14a of the road vehicle is incorporated into the virtual environment 22 as a virtual road user, thereby allowing the first simulation 14a to move in the virtual environment 22 in the same way as a real-world road vehicle in a real-world road traffic scenario.

[0038] The first simulation 14a (and the second simulation 14b and the third simulation 14c as well) includes a sensor simulation 30 to which a sensor position is assigned in a virtual environment 22 (in the illustrated case, in front of the simulated road-driving vehicle 14a). As an example, the sensor simulation 30 simulates a smart camera for lane recognition. In the virtual environment 22, the sensor simulation 30 is assigned a field of view (FV). Lanes are visible to the sensor simulation 30 unless they are within the field of view (FV) and are not hidden from the viewpoint of the sensor simulation 30 by 3D objects O1,...,O8 or the topography of the virtual environment 22.

[0039] The neural network 8 should be trained to steer the road vehicle 14a laterally by manipulating the steering angle in a virtual environment. Other aspects of operating the road vehicle 14a, such as acceleration, braking, and distance control, are performed by routines in the first simulation 14a, the second simulation 14b, or the third simulation 14c, and are not affected by the neural network 8.

[0040] Figure 4 illustrates the functionality of the sensor simulation 30 as an example. As input, the sensor simulation 30 receives a description of the virtual environment 22 from the virtual training environment 4, for example in the form of an object list, generates simulated sensor data 34 from this description, and transmits this simulated sensor data 34 to the neural network 8 via the programming interface 18. In the illustrated example, the sensor data 34 consists of rows of spots, each row of spots representing a lane segment visible to the smart camera, and each spot is represented by three spatial coordinates. The rows of spots specify the range of movement for the simulated road-driving vehicle 14a in the virtual environment 22. The first training objective of the neural network 8 for the first simulation 14a is to operate the first simulation 14a so that the simulated road vehicle proceeds along a predetermined section, for example, a 5-kilometer section, on a virtual test route 32 without colliding with any spot from any row of spots in the simulated sensor data 34 while it is on this section. To this end, the training function 24 is configured to monitor the route taken by the simulated road vehicle on the virtual test route 32 and to recognize collisions between the road vehicle and the road boundary line.

[0041] The spot rows in the sensor data 34 are an example of so-called ground truth sensor data. The spot rows in the sensor data 34 simulate sensor data that becomes available after a smart evaluation of raw data from an image-generating sensor, and it is particularly easy to simulate based on a virtual environment 22. However, the simulated sensor data 34 can basically consist of any form depending on the type of sensor data 34 to be trained to process by the neural network 8, for example, as an object list, or as raw data from a camera, thermal imaging camera, radar, lidar, or echolocation. The simulated sensor data 34 may originate from a real-world camera pointed at a screen on which images of the virtual environment 22 are being rendered in real time from the viewpoint of the sensor simulation 30. Those skilled in the art can find an overview of the various options for sensor simulation and the breakpoints that should be implemented for them in the scientific paper “Full spectrum camera simulation for reliable virtual development and validation of ADAS and automated driving applications” Rene Molenaar et al., 2015 IEEE Intelligent Vehicles Symposium (IV), which uses a camera simulation as an example.

[0042] Figure 5 shows a possible configuration of the first simulation 14a of the road vehicle. As input from the neural network 8, the first simulation receives the steering angle θ and assigns the change in direction Δφ of the road vehicle on the two-dimensional plane of the virtual test route 32 to the steering angle θ based on a simple linear function by multiplying it by a constant steering sensitivity s and the simulation step size Δt. Based on the change in direction Δφ, the first simulation 14b calculates the new direction of travel in the subsequent simulation step. Thus, the first simulation 14a is a highly simplified model of the road vehicle, in which the road vehicle is a solid body with no rotational degrees of freedom and moves parallel to the ground on a perfect arc at any given time, the curvature of this perfect arc being linearly predetermined by the steering angle θ. Based on the first simulation 14a, the neural network 8 learns the basics of steering the road vehicle.

[0043] Once the first training objective is achieved, the first simulation 14a is replaced by the second simulation 14b, as shown in Figure 6, as described above based on Figure 1. Exactly like the first simulation 14a, the second simulation receives the steering angle θ as input from the neural network 8 and confirms the change in direction Δφ, which is visible to the neural network 8 thanks to the sensor data 34 transmitted to the neural network 8, based on this steering angle θ, while the composition of the sensor data 34 remains unchanged compared to the first simulation 14a. Therefore, the second data interface 16b that the neural network 8 uses to read state data (in this case, sensor data 34) from the second simulation 14b and to transfer control data (in this case, steering angle θ) to the second simulation 14b is identical to the first data interface 16a of the first simulation. Consequently, the neural network cannot directly identify that the first simulation 14a has been replaced by the second simulation 14b. However, the second simulation 14b, while still considerably simplified compared to reality, is configured to be more realistic than the first simulation 14b. The second simulation 14b is a so-called kinematic bicycle model, a basic model known to those skilled in the art for simple simulations of four-wheeled land-based vehicles. The kinematic bicycle model combines two wheels, each located on a single level, into a single wheel such that these two wheels lie on a common longitudinal axis, with the front wheel assigned an inclination angle corresponding to the steering angle θ, from which the change in direction Δφ is calculated in conjunction with the fixed rear wheel. The kinematic bicycle model takes into account more degrees of freedom than the first simulation 14a, namely the ability of the front wheel to rotate, and more physical phenomena, such as the range of the vehicle in space or the existence of the vehicle's center of mass.Therefore, the simulation cycle of the second simulation 14b requires processing more mathematical operations than the simulation cycle of the first simulation 14a. Consequently, overall, training the neural network 8 is slower compared to training on the first simulation 14a. However, the second simulation 14b still ignores many phenomena, such as the loss of tire adhesion and the degrees of freedom of the vehicle. Therefore, the second simulation 14b is fast enough to achieve rapid training success even in the second training phase.

[0044] The second simulation 14b is assigned a second training objective, which may be identical to the first training objective. Once the training objective is achieved, the second simulation 14b is replaced by the third simulation 14c, as shown in Figure 7, as described above. The third simulation 14c is a complex simulation that reproduces the technical and mechanical operations of a real-world road vehicle with high detail. The third simulation 14c includes a complete simulation of the vehicle's drivetrain and all suspensions, a simulation of the engine with realistic acceleration and braking behavior, a realistic tire model, the yaw and roll motion of the vehicle when the vehicle body is on the road, the loss of contact between some or all of the tires and the road, and the aerodynamic forces acting on the vehicle. The third data interface 16c of the third simulation 14c is also identical to the second data interface 16b and the first data interface 16a.

[0045] Due to the higher level of detail in the third simulation 14c, training using the third simulation 14c takes significantly longer than training using the second simulation 14b and the first simulation 14a. However, the performance of the third simulation 14c closely resembles the actual performance of a road-going vehicle. As a result, once the third training objective is achieved, the neural network reaches a high level of training, making it possible to experimentally operate a real-world road-going vehicle on a real-world test route.

Claims

1. A computer-implemented method for training a neural network to operate a technical device, the method comprising the following steps: The steps include establishing a first data link between the neural network and a virtual first simulation of the technical device by a first data interface of the first simulation for reading state data from the first simulation and transferring control data to the first simulation, The steps include setting a first training objective for operating the first simulation, The steps include training the neural network by the first simulation operated by the neural network, and checking the training progress of the neural network against the first training objective, Steps include: disconnecting the first data link once the first training objective is achieved; then establishing a second data link between the neural network and a virtual second simulation of the technical device by a second data interface of the second simulation for reading state data from the second simulation and transferring control data to the second simulation, wherein the second simulation is configured to be more realistic than the first simulation and requires more mathematical operations per simulation cycle than the first simulation due to the higher realism of the second simulation; The steps include training the neural network by the second simulation which is operated by the neural network, A method that includes this.

2. The first data interface and the second data interface are configured identically. The method according to claim 1.

3. The above method involves the following steps, namely The steps include generating simulated sensor data that includes information about the virtual environment of the aforementioned technical device, The steps include transferring the simulated sensor data to the neural network, The steps include training the neural network to evaluate the sensor data and to take the sensor data into consideration when operating the first simulation and / or the second simulation, including, The method according to claim 1.

4. The aforementioned technical device is a robot, particularly in the following categories, namely vehicle, Robot arm, Robots for positioning or mounting materials and / or objects, A robot for cleaning surfaces. Robots for inspecting spaces or surfaces A robot for applying chemical substances, such as cleaning agents, disinfectants, paints, varnishes, or coatings. Robots for performing medical procedures, It is a robot belonging to at least one of the following categories. The method according to claim 1.

5. The first training objective mentioned above falls under the following categories of training objectives, namely Sections that proceed along a virtual training course or a virtual test route, Multiple motion patterns performed without any collision or within a predetermined range of motion, During the period in which the neural network operates the first simulation appropriately without any unwanted events occurring, The threshold reached for the reward function, Belonging to at least one of the following: The method according to claim 4.

6. The second simulation described above is: This simulation considers more mechanical and / or electrical components of the technical device than the first simulation described above, and / or Considering more mechanical degrees of freedom than the first simulation, and / or To simulate physical phenomena and / or laws more realistically and / or with greater accuracy than the first simulation, and / or Taking into account a greater number of physical forces and / or interactions than the first simulation, and / or Having a smaller simulation step size than the first simulation described above, The method according to claim 1.

7. The above method involves the following steps, namely The steps include setting a second training objective for operating the second simulation, During the operation of the second simulation, the step of checking the training progress of the neural network with respect to the second training objective, After the second training objective is achieved, the procedure involves establishing a physical data link between the neural network and the technical device. The steps include operating the technical device using the neural network, including, The method according to claim 1.

8. A virtual training environment for training a neural network to operate a technical device, wherein the virtual training environment is A first simulation of the technical device having a first data interface for reading state data from the first simulation and for transferring control data to the first simulation, A programming interface for establishing a first data link between the neural network and the first simulation, A training function configured to set a training objective for the first simulation to be operated by the neural network, and to check the training progress of the neural network against the training objective, In a virtual training environment including, The virtual training environment includes a second simulation of the technical device, The second simulation is configured to be more realistic than the first simulation, and due to the higher realism of the second simulation, it requires more mathematical operations per simulation cycle than the first simulation. The second simulation includes a second data interface for reading state data from the second simulation and for transferring control data to the second simulation. The aforementioned training function is When the aforementioned training objective is achieved, the first data link is shut off. The first data link is replaced by a second data link between the neural network and the second simulation. The neural network (8) is trained by the second simulation (14b) which is operated by the neural network (8). It is characterized by being configured in such a way, A virtual training environment.

9. The first data interface and the second data interface are configured identically. The virtual training environment described in claim 8.

10. The virtual training environment includes a virtual environment for the technical device and a sensor simulation. The aforementioned virtual training environment is The sensor simulation generates simulated sensor data that includes information about the virtual environment. The simulated sensor data is transmitted to the neural network via the programming interface. It is structured in such a way. The virtual training environment described in claim 8.

11. The aforementioned technical device is a robot, particularly in the following categories, namely vehicle, Robot arm, Robots for positioning or mounting materials and / or objects, A robot for cleaning surfaces. Robots for inspecting spaces or surfaces A robot for applying chemical substances, such as cleaning agents, disinfectants, paints, varnishes, or coatings. Robots for performing medical procedures, It is a robot belonging to at least one of the following categories. The virtual training environment described in claim 8.

12. The aforementioned training objectives fall into the following categories of training objectives, namely Sections that proceed along a virtual training course or a virtual test route, Multiple motion patterns performed without any collision or within a predetermined range of motion, During the period in which the neural network operates the first simulation appropriately without any unwanted events occurring, The threshold reached for the reward function, Belonging to at least one of the following: The virtual training environment according to claim 11.

13. The second simulation described above is: This simulation considers more mechanical and / or electrical components of the technical device than the first simulation described above, and / or Considering more mechanical degrees of freedom than the first simulation, and / or To simulate physical phenomena and / or laws more realistically and / or with greater accuracy than the first simulation, and / or Taking into account a greater number of physical forces and / or interactions than the first simulation, and / or The virtual training environment according to claim 8, having a smaller simulation step size than the first simulation.

Images