Device and computer-implemented method for providing training data to agent for reinforcement learning for generating strategy to control vehicle, device and method for generating strategy to control vehicle, and vehicle

Abstract

To provide a device, a method and a program which provide training data to an agent for reinforcement learning which generates strategy to control a vehicle.SOLUTION: In a vehicle 100, an initialization unit 110 of a device 105 creates an environmental model 115 using a selected physical feature value IN of granular material and work environment. The selected physical feature value IN is input from an external data source of the device 105 by the initialization unit. An association unit 120 receives or loads the environmental model including a relative height map based on a probability density function from the initialization unit. An agent 130 for reinforcement learning receives training data 125 from the association unit 120. A control unit 140 of an optional choice receives strategy 135 for controlling the vehicle from the agent for reinforcement learning, and outputs a control signal 145 based on the strategy to at least one actuator 102.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an apparatus or method described in an independent claim. Computer programs are also covered by the present invention. [Background technology]

[0002] In relation to autonomous vehicles, granular simulations or simulations of granular materials can generally be classified into three types of simulations: for example, solid mechanics and empirical formulas, discrete element methods, and height-map approaches. In particular, the above methods use iterative algorithms to identify the interaction between the granular material and the vehicle. Furthermore, these simulations can be computed offline and then run and observed, as with other numerical methods such as computational fluid dynamics (CFD), but such methods are generally insufficient for reinforcement learning (BL) because the simulations must be run multiple times to train the BL agent. U.S. Patent No. 10481603 (US10481603B2) discloses a trajectory planning algorithm for off-road vehicles.

[0003] From German Patent Application Publication No. 19859169 (DE19859169A1), a system comprising a machine having multiple mechanical links connected by joints is known. The processing system executes at least one script, each script having at least one variable parameter that defines the movement of the machine. A learning algorithm modifies the value of at least one variable parameter based on the desired outcome and measured conditions that affect the outcome.

[0004] From German Patent Application Publication No. 102014203312 (DE102014203312A1), it is known that road gradients can be modeled across regions. A sequence of gradient values ​​is generated at a predetermined rate during vehicle operation, and each gradient value identifies a gradient region that is subsequently reached. In response to the sequence of gradient values, a Markov chain road gradient model is updated, in which each stochastic element is represented in the form of a matrix of transition events from each predetermined gradient region to each subsequent gradient region.

[0005] From the German Patent Application Publication No. 112009001554 (DE112009001554T5), a method for operating a vehicle over a pile of material at a construction site is known, wherein the material is discharged at the construction site through an opening, and the method includes detecting the surface of the pile, identifying fault zones on the surface of the pile resulting from the discharge of material based on the detected surface and a known location of the opening, and transmitting a signal to the vehicle indicating the fault zones. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] U.S. Patent No. 10481603 (US10481603B2) [Patent Document 2] German Patent Application Publication No. 19859169 (DE19859169A1) [Patent Document 3] German Patent Application Publication No. 102014203312 (DE102014203312A1) [Patent Document 4] German Patent Application Publication No. 112009001554 Specification Translation (DE112009001554T5) [Overview of the project] [Problems that the invention aims to solve]

[0007] Disclosure of the invention Against this backdrop, the approaches presented herein provide, respectively, a method of the main claim, an apparatus for using the method, and a corresponding computer program. Advantageous embodiments and improvements of the subject matter defined in the independent claims are made possible by the features described in the dependent claims. [Means for solving the problem]

[0008] According to the embodiment, for example, a simplified granular simulation for reinforcement learning training can be provided. The simulation can be based on, for example, a Gaussian distribution, in which case it can be assumed that the granular material behaves like a continuous multivariate Gaussian probability density function (PDF). For the simulation, a usable relationship was found between the physics of the granular material and mathematical Gaussian coefficients. Furthermore, the interaction between the granular material and the vehicle can be formalized as a change in the Gaussian PDF. In particular, the environment can be represented by a height map, where H(x,y)=h means that the ground height at position (x,y) is the same as h. Furthermore, the vehicle's position can be assumed as a combination of the vehicle's current location and the azimuth angle with respect to the Earth's axis. This simulation may include, for example, two main algorithms: one is an algorithm for initializing a region or environment, in which case an initial environment height map can be generated based on a list of the locations of granular material piles; and the other is a further algorithm for calculating an updated region, in which case a new environment can be identified based on the vehicle's movement or action.

[0009] According to the embodiments, a simplified simulation can be provided that can be used, for example, for a mobile trajectory planning algorithm, particularly for off-road use. In particular, the approach presented herein makes it possible to create such a type of simulation that detects complex physics while maintaining a simple, automated simulator that takes only a short time. The simulation or model can be used to train a reinforcement learning algorithm (BL algorithm). Therefore, it is proposed to improve the Gym-API, which requires the simulation to include the following: reset, step, is_done, make, reward_calculate. In particular, for BLs where multiple simulations must be performed to train the BL agent, conventional methods may be insufficient. For this reason, a simplified simulator can be provided that can be used in training and that can simulate, for example, a day's work at a construction site in the shortest possible time.

[0010] The simulation approach presented herein is advantageous over the previously mentioned approaches because it eliminates the need for multiple calculations for each vehicle movement. Furthermore, the complexity of the simulation can increase with simpler distributions. Therefore, it is possible to add spillage to vehicle movements, for example, by adding a probability function that causes spillage with probability P. The approach presented herein can simulate the most important interaction features between the vehicle and the environment without sacrificing simplicity. Moreover, this type of simulation is advantageously usable for training BL agents. This is because not all interactions between the vehicle and the ground are important for the success of the BL agent, but this type of simulation detects the most important interactions necessary for BL training to select the optimal action. It is also advantageously possible to convert the features of granular materials into mathematical variables that can be easily used for creating Gaussian PDFs.

[0011] A computer-implemented method for providing training data to a reinforcement learning agent for generating strategies configured to control a vehicle, particularly an off-road vehicle, wherein the vehicle is configured to interact with granular material in a working environment. This method is A step of initializing an environmental model of a work environment containing granular material, wherein selected physical feature values ​​of the granular material and the work environment are used as variables of at least one probability density function, and a relative height map between the actual height state and the target height state of the work environment containing the granular material is generated using the probability density function, To achieve a target state, the process involves associating the change in the probability density function of the environmental model and the reward value with each action from a set of multiple vehicle actions, wherein the relative height map of the environmental model and the vehicle position are updated for each action in order to provide training data. A method is proposed that includes

[0012] The vehicle or off-road vehicle may be a bulldozer, a land grader, a roller, a dump truck, an excavator, a construction vehicle, a heavy machine, a vehicle for heavy loads, or other types of vehicles having a plurality of tasks including interaction with the environment, particularly interaction with granular materials. Examples of such types of tasks or actions are leveling, unloading granular materials, compacting the ground, removing granular materials, etc. The granular materials may include soil, sand, snow, rubble, stones, and / or similar particulate materials. The strategy may include at least one planned movement trajectory for the vehicle. The environmental model may include a map of relative heights based on a probability density function. The reward value may be positive or negative or zero. The change in the probability density function may have an absolute value different from zero or may be equal to zero.

[0013] According to one embodiment, in the initializing step, the probability density function may be a continuous multivariate Gaussian probability density function. Additionally or alternatively, in the initializing step, the selected physical characteristic values may include the specific mass of the granular material, the spread parameter of the granular material, the volume of each hill of the granular material, and the position of the center point of each hill with respect to the environment. Such an embodiment provides the advantage of being able to detect the complex physics of granular materials while providing a simple automatic simulator that takes only a short time.

[0014] According to one embodiment, in the associating step, the working environment can be divided into a plurality of boundary frames, and the change in the probability density function can be calculated individually for each boundary frame. Such an embodiment provides the advantage of being able to quickly identify changes in the environment caused by the actions of the vehicle at a relatively low computational cost.

[0015] According to one embodiment, in the mapping step, depending on whether the action is forward translation, backward translation, or rotation, and depending on whether the vehicle position is within or outside the acceptable range of the work environment, different variations of the action and, additionally or alternatively, different reward values ​​can be associated with each action. Such embodiments offer the advantage that the vehicle's actions are easily evaluateable and / or performable for the training agent.

[0016] This specification provides for a method for generating a strategy configured to control a vehicle, particularly an off-road vehicle, wherein the vehicle is configured to interact with granular material in a working environment, and the method is The steps include providing training data by the method for providing it as presented herein, The steps include: using training data to generate a strategy using a reinforcement learning agent, Methods including this are also proposed.

[0017] In the generation step, the training data is available to a reinforcement learning agent to identify at least one optimal action for the vehicle.

[0018] In one embodiment, the method also includes the step of controlling at least one actuator of the vehicle using a control signal determined by the strategy. Such embodiments offer the advantage that the vehicle can be controlled so that it efficiently performs actions and tasks.

[0019] Each of the methods presented herein can be implemented, for example, in the form of software or hardware, or in a hybrid form of software and hardware, for example, in a device or controller.

[0020] The approach presented herein also envisions an apparatus configured to perform, control, or carry out steps of modified versions of the method presented herein using corresponding means. The fundamental objectives of the present invention can also be achieved quickly and efficiently by this embodiment of the invention, which is in the form of an apparatus.

[0021] For this purpose, the device may include at least one computing unit for processing signals or data, at least one memory unit for storing signals or data, at least one interface to a sensor or actuator for reading sensor signals from a sensor or outputting data or control signals to an actuator, and / or at least one communication interface for reading or outputting data embedded in a communication protocol. The computing unit may be, for example, a signal processor, a microcontroller, etc., and the memory unit may be flash memory, EEPROM, or a magnetic memory unit. The communication interface may be configured to read or output data wirelessly and / or via a wired connection, in which case the communication interface capable of reading or outputting wired data may, for example, read data electrically or optically from or output to a corresponding data transmission line.

[0022] The device can be understood as an electrical device that processes sensor signals and outputs control signals and / or data signals in response to these sensor signals. The device may include interfaces implemented in hardware and / or software. If implemented in hardware, the interface may be, for example, part of a so-called system ASIC that includes several different functions of the device. However, the interface may be an individual integrated circuit or consist of at least partially discrete components. If implemented in software, the interface may be a software module implemented together with other software modules, for example, on a microcontroller.

[0023] The approach presented herein also provides a vehicle, particularly an off-road vehicle, configured to interact with granular materials in a working environment, the vehicle including the apparatus presented herein.

[0024] Computer program products or computer programs having program code that can be stored on a machine-readable carrier or storage medium such as semiconductor memory, hard disk, or optical memory, which are particularly advantageous when implemented on a computer or device, are used to implement, execute and / or control the steps of the method according to one of the embodiments described above.

[0025] One embodiment can be used as a prerequisite in a toolchain for machine learning. This embodiment can be used as a method for generating training data to train a machine learning system that can be used for the following applications. After the machine learning system has been trained in this way, it can be used for subsequent purposes as described below. The machine learning system can be used to analyze data from off-road vehicles and for tasks that interact with the ground. The machine learning system can be used to determine one or more continuous values, i.e., to perform regression on changes in the ground caused by vehicle actions. In this case, these embodiments are connected to a specific physical system, i.e., an off-road vehicle. Using the simulation obtained by one embodiment, a reinforcement learning agent can be trained to create an optimal strategy for an off-road vehicle. Thus, control signals can be computed or generated to control off-road vehicles, computer-controlled machines such as robots, vehicles, or other physical systems that interact with the ground.

[0026] Embodiments of the approach presented herein are shown in the drawings and described in more detail below. [Brief explanation of the drawing]

[0027] [Figure 1] This is a schematic diagram of one embodiment of a device inside a vehicle. [Figure 2] This is a flowchart of one embodiment of a method for providing training data to a reinforcement learning agent for generating strategies configured to control a vehicle. [Figure 3] This is a flowchart of one embodiment of a method for generating a strategy configured to control a vehicle. [Figure 4] This is a schematic diagram of a simulation of a series of vehicle actions. [Figure 5] This is a flowchart of the simulation process according to one embodiment. [Figure 6] This is a flowchart of the simulation process according to one embodiment. [Modes for carrying out the invention]

[0028] In the following description of advantageous embodiments of the present invention, elements that function similarly as shown in multiple different drawings are denoted by the same or similar reference numerals, and the description of these elements is omitted.

[0029] Figure 1 shows a schematic diagram of one embodiment of a device 105 within a vehicle 100. The vehicle 100 is configured to interact with granular material in the working environment. In particular, the vehicle 100 is an off-road vehicle, such as a bulldozer, earthmoving machine, compactor, dump truck, excavator, construction vehicle, heavy machinery, heavy lifting vehicle, or other type of vehicle having multiple tasks that include interaction with the environment, particularly interaction with granular material. The vehicle 100 includes a device 105. The vehicle 100 further includes at least one actuator 102 that triggers or performs actions of the vehicle 100. The device 105 and the actuator 102 are connected to each other for signal communication or data communication.

[0030] The device 105 is configured to provide training data 125 to a reinforcement learning agent 130 for generating a strategy 135 configured to control a vehicle such as vehicle 100, and / or the device 105 is configured to generate a strategy 135 for controlling vehicle 100. In other words, the device 105 is configured to perform and / or control the steps of the method according to Figure 2 and / or the method according to Figure 3 in the corresponding units. The device 105 also includes an initialization unit 110 and a mapping unit 120, and optionally a reinforcement learning agent 130 or a generation unit. According to one embodiment, the device 105 further includes a control unit 140 or a controller.

[0031] The providing portion of device 105 includes an initialization unit 110 and a mapping unit 120. The generating portion of device 105 includes a reinforcement learning agent 130 and an arbitrary choice control unit 140. As shown in Figure 1, the initialization unit 110, the mapping unit 120, the reinforcement learning agent 130, and the arbitrary choice control unit 140 are all implemented as parts of device 105 located inside the vehicle 100. In other embodiments, the initialization unit 110 and the mapping unit 120, i.e., the providing portion, may be implemented as parts of device 105 located physically outside the vehicle 100.

[0032] The initialization unit 110 is configured to initialize an environmental model 115 of the vehicle 100's work environment, which includes granular material. The initialization unit 110 is configured to create the environmental model 115 using selected physical feature values ​​IN of the granular material and the work environment. The selected physical feature values ​​IN are input or loaded by the initialization unit 110 from an external data source of the device 105. The selected physical feature values ​​IN are used by the initialization unit 110 as variables relating to at least one probability density function. A relative height map between the actual height state and the target height state of the work environment, which includes granular material, is generated by the initialization unit 110 using the probability density function. The initialization unit 110 is configured to output the environmental model 115 to the mapping unit 120.

[0033] According to one embodiment, the initialization unit 110 uses a continuous multivariate Gaussian probability density function as the probability density function. The selected physical feature values ​​IN may include, in particular, the specific mass of the granular material, the distribution parameter of the granular material, the volume of each pile of the granular material, and the position of the center point of each pile relative to the environment.

[0034] The mapping unit 120 is configured to receive or load an environment model 115, which includes a relative height map based on a probability density function, from the initialization unit 110. The mapping unit 120 is configured to associate changes in the probability density function of the environment model 115 with reward values ​​in order to achieve the target state of each action in a set of multiple vehicle actions. The relative height map of the environment model 115 and the vehicle position are updated by the mapping unit 120 for each action in order to provide training data 125. In other words, the mapping unit 120 is configured to provide training data 125 using the environment model 115.

[0035] According to one embodiment, the mapping unit 120 is configured to divide the work environment into a plurality of boundary frames and to calculate the change in the probability density function individually for each boundary frame. Additionally or alternatively, according to one embodiment, the mapping unit 120 is configured to associate different changes and / or different reward values ​​with actions depending on whether the action is forward translation, backward translation or rotation, and / or whether the vehicle position is within or outside the allowable area of ​​the work environment.

[0036] The reinforcement learning agent 130 is configured to receive or load training data 125 from the mapping unit 120, i.e., from the providing portion of the device 105. The reinforcement learning agent 130 is configured to use the training data 125 to generate a strategy 135 through reinforcement learning. The agent 130 is further configured to output the strategy 135 for reinforcement learning.

[0037] An optional control unit 140 is configured to receive or load a strategy 135 from the reinforcement learning agent 130. The control unit 140 is further configured to determine a control signal 145 based on the strategy 135. The control unit 140 is further configured to control at least one actuator 102 of the vehicle 100 using the control signal 145. For this purpose, the control unit 140 is configured to output the control signal 145 to at least one actuator 102. According to other embodiments, the control unit 140 may be a unit separated from the device 105.

[0038] Figure 2 shows a flowchart of one embodiment of Method 200 for providing training data to a reinforcement learning agent for generating strategies configured to control a vehicle, particularly an off-road vehicle. The vehicle is configured to interact with granular material in the working environment. Method 200 for providing can be carried out using or in connection with the apparatus shown in Figure 1 or a similar apparatus. Method 200 for providing is a computer-implemented method. Method 200 for providing includes an initialization step 210 and a mapping step 220.

[0039] In initialization step 210, an environmental model of the work environment containing granular material is initialized. For initialization, selected physical feature values ​​of the granular material and the work environment are used as variables in at least one probability density function. Furthermore, a relative height map between the actual height state and the target height state of the work environment containing granular material is created using the probability density function. Subsequently, in mapping step 220, the changes in the probability density function of the environmental model and the reward values ​​are mapped to each action in a set of multiple vehicle actions in order to achieve the target state. To provide training data, the relative height map of the environmental model and the vehicle position are updated for each action.

[0040] Figure 3 shows a flowchart of one embodiment of Method 300 for generating a strategy configured to control a vehicle, particularly an off-road vehicle. The vehicle is configured to interact with granular material in the working environment. Method 300 for generating can be carried out using or in connection with the apparatus shown in Figure 1 or a similar apparatus. Method 300 for generating includes step 325 of providing and step 330 of generating.

[0041] In step 325, the providing step, training data is provided by performing the steps of the providing method or a similar method shown in Figure 2. Subsequently, in step 330, a strategy is generated using the training data with a reinforcement learning agent. According to one embodiment, the generating method 300 also includes a controlling step 340. In the controlling step 340, at least one actuator of the vehicle is controlled using a control signal determined using the strategy generated in step 330.

[0042] Figure 4 shows a schematic diagram of a simulation of a series of vehicle actions. This simulation is created by the apparatus shown in Figure 1 or a similar apparatus, and / or by performing the method shown in Figures 2 and / or Figure 3 or a similar method. Specifically, Figure 4 shows eight schematic diagrams of a simulation example for vehicle 100, in this case a simulation example for a bulldozer leveling a sand pile by a set of seven actions. Figure 4 shows eight diagrams or sub-diagrams T=0, T=1, T=2, T=3, T=4, T=5, T=6 and T=7, each representing a relative height map of the environmental model 115, where T=0 represents the initial situation as an initialized state, and the subsequent diagrams T=1, T=2, T=3, T=4, T=5, T=6 and T=7 sequentially represent one action of vehicle 100 each.

[0043] This simulation illustrates the interaction between a bulldozer, represented as vehicle 100, and the ground, in which case the bulldozer is required to level a sand pile. The sand pile shown here is formed from six adjacent sub-sand piles, and the final sand pile is the sum of all the Gaussian PDFs of these sand piles. Each diagram T=1, T=2, T=3, T=4, T=5, T=6, and T=7 represents one different point in time as vehicle 100 moves and levels the sand pile. It is assumed that each forward movement of vehicle 100 is performed with the bulldozer blade placed on the ground, leveling the ground. As can be seen, with each forward movement, a further portion of the ground is leveled, and as soon as vehicle 100 arrives at the final position for the current movement, the remaining ground within the blade is newly distributed as a Gaussian PDF (see, for example, T=2). In other embodiments, more complex representations, such as Gaussian mixture models, may be used, but it should be noted that using Gaussian PDFs can simplify mathematical calculations, both to maintain a rapid approach and to achieve a good representation of the interaction with the ground.

[0044] In the following, with reference to the aforementioned drawings, the embodiments and their advantages over the context of the embodiments will be summarized and / or described in more detail, with some partial paraphrasing.

[0045] The training data 125 and / or strategy 135 can be used for any vehicle 100 that interacts with the ground, for example, but the emphasis of this specification is on specific vehicle types that are required to raise sand piles to a predetermined height. In this description, it is assumed that the height map of the environmental model 115 in the simulation is a relative map between the actual sand conditions and the desired final sand height. Furthermore, a grid is created to convert between grid units and meters, as follows: (x,y) grid =(x m *[pixel / meter],y mThe task is simplified by defining it as *[pixel / meter]).

[0046] To better understand the embodiments, the multivariate Gaussian PDF should be understood. In this case, assuming that two variables (x, y), which are Cartesian coordinates, are both taken from a normal distribution, the probability density function in this case, under the assumption that these two variables follow an independent and identical distribution, [Number] becomes like this.

[0047] This function is used to specify the height of the ground at each (x, y) point. Note that the integral is performed on this function V, and this integral represents the volume of the mountain of granular material. The units for these variables are V [m 3 ; σ x [m]; σ y [m]. Since the sand is homogeneous, it is further assumed that the mountain is uniformly distributed in all directions, and thus the following relationship: σ x = σ y = σ is assumed. In this case, the height distribution is reduced to [Number] up to

[0048] That is, the maximum height of the mountain is H max = v / 2πσ 2 . Further, by changing (x, y) at (r, θ) using (x - μ x ) / σ = r * cosθ and (y - μ y ) / σ = r * sinθ, the following relationship: [Number] <000023​​​​​Assuming a simple bulldozer mechanism for vehicle 100, there are three possible speed levels, and there is a linear relationship between the bulldozer speed and the load on the blade. The simple relationship between vehicle speed and load affects the total time taken for the action and is calculated based on the volume of sand moved during the selected action. Furthermore, it is assumed that the blade has a known volume, which is the maximum volume that this blade can spread within a given time.

[0050] Step 210 is performed to initialize the method 200 provided for initializing the simulation. The simulation can be initialized using configuration data, in which the following information is provided: namely, the specific mass of the sand,

number

number

[0051] As soon as these parameters are determined, the initial environment or environment model 115 can be created in the following substeps, namely, • Create a blank height map using the actual area size and resolution: δh [meter / pixel] step, • For each mountain in the list of mountains, Using the equations listed above, the height of the sand pile

number

number

[0052] Next, in order to provide training data 125, it is necessary to calculate the following steps. In order to calculate the steps in the simulation, in step 220 of the method 200 for providing the steps, changes in the ground are identified based on the movement of vehicle 100. If vehicle 100 is a bulldozer and the task is leveling, the ground changes as the bulldozer moves, because the blade pushes the sand piles forward, making the surface as flat as possible. This change is modeled by dividing the area in which vehicle 100 is moving into several relatively small bounding boxes, and then separately calculating the interaction effect on each box. In Figures 5 and 6 described below, the simulation logic of the movement of vehicle 100 can be seen in the partial steps of vehicle 100's action.

[0053] In the simulation, the action space for the actions of vehicle 100, particularly the bulldozer, includes two possible actions: (a) forward / backward and (b) rotation. The neural network (NN) outputs a (2×1) vector. A positive value in the first entry signifies forward movement, while a negative value signifies backward movement. A positive value in the second entry of the action vector represents clockwise rotation, while a negative value represents counterclockwise rotation. If a situation and a corresponding preferred action exist, the reinforcement learning agent 130 performs this action. To understand that sand movement is a result of these actions, translational and rotational movement must be distinguished.

[0054] In the case of translation, a new vehicle position is calculated. If the new position is not valid, i.e., located outside the acceptable area, a large negative reward is obtained and a new scenario is initiated. If the new position is valid, spreading is performed. Here, the area inside this simulation iteration field of view in the inertial coordinate system is calculated as an indication of this step boundary. Subsequently, a height map within this area is calculated. The sand within this area is spread. The simulator then calculates the total volume of this incremental unit area, and the bulldozer can distribute the total volume to the average value of this unit area. If the translational motion is completed and the blade is covered with sand, the simulator, i.e., the corresponding unit of device 105, generates an additional Gaussian sand pile with the volume remaining on the blade. Subsequently, the simulator mows this Gaussian sand pile to simulate a suboptimal spreading pattern. Finally, the simulator calculates the time required for this action and obtains a reward. Reverse driving or turning does not change the sand, i.e., the height map, and only results in a negative reward.

[0055] In the case of rotation, the angle of rotation is calculated. The vehicle's DCM (direction cosine matrix) is updated because it represents the direction of travel. Since rotational motion does not change the distribution of sand, a small negative reward is added. If there is no interaction between vehicle 100 and the sand, the bulldozer rotates, and therefore no positive reward is obtained during rotation.

[0056] Figure 5 shows a flowchart of the simulation process 500 according to one embodiment. The simulation process 500 can be carried out by the apparatus shown in Figure 1 and / or in connection with the method for providing and / or the method for generating shown in Figure 2 and / or Figure 3. In other words, Figure 5 shows the action steps of the simulation.

[0057] Block 502 is the input for the vehicle posture, in this case the bulldozer posture, the height map, and the blade posture. Process 500 proceeds from block 502 to block 504, which represents the strategy network, then to block 506, which represents the action, and then to block 508, where it is determined whether the action is translational. If the action is translational, process 500 proceeds from block 508 to block 510, where it is determined whether the vehicle is moving forward. If the vehicle is moving forward, process 500 continues to block 512. Block 512 contains the following instructions: Between (current location - final location > eps), 1. Create a height map for the area element of the integral. 2. Calculate the new height of this area element. 3. Updated position = position + delta_pos 4. Go back to step 1. Includes.

[0058] If the action is not translational (see block 508) or the vehicle is not moving forward (see block 510), process 500 proceeds to block 514. Block 514 contains the following instructions: 1. Step to calculate the total time for each step. 2. Steps to update the bulldozer posture and blade posture. 3. Steps to update the height map 4. Step to calculate reward = -time This includes terminating the process.

[0059] Process 500 proceeds from block 514 to block 516, where it is checked whether the simulated action is performed. This check is based on the following criteria: 1. Max (Local thermal image map) <eps 2. N_steps > N_steps_max It depends on.

[0060] If an action is performed, process 500 proceeds to block 518, which represents a reset to generate a new scenario. If no action is performed, process 500 returns to block 502.

[0061] Figure 6 shows a flowchart of the simulation process 600 according to one embodiment. The simulation process 600 can be implemented by the apparatus shown in Figure 1 and / or in relation to the method for providing and / or generating as shown in Figure 2 and / or Figure 3. In other words, Figure 6 shows the algorithm for the simulation steps described above, or the logic for each relatively small region or boundary frame.

[0062] In block 602, the vehicle path, in this case the bulldozer path, is divided into small rectangles along the vehicle axis. Process 600 proceeds from block 602 to block 604, where for each rectangle in the list of rectangles, the volume of the rectangle to be removed is calculated. If the volume is greater than zero, process 600 proceeds to block 606, where the new volume in the blade and the new ground height are calculated. V ブレード =V ブレード + volume height マップ (x,y) = target height It is calculated as follows.

[0063] If the volume is less than zero, process 600 proceeds further to block 608. Block 608 is, V ブレード >In the case of volume: height マップ (x,y) = Volume V ブレード =V ブレード -Target height Otherwise: height マップ (x,y) = Average volume within the blade V ブレード =0 Includes.

[0064] Next, process 600 proceeds from blocks 606 and 608 to block 610, which represents the transition to the next rectangle.

[0065] Whenever an embodiment includes the conjunction "and / or" between a first feature and a second feature, this means that in the first variation, the embodiment includes both the first and second features, and in further variations, it includes either the first or the second feature.

Claims

1. A computer-implemented method (200) for providing training data (125) to a reinforcement learning agent (130) for generating a strategy (135) suitable for controlling a vehicle (100), particularly an off-road vehicle (100), wherein the vehicle (100) is configured to interact with granular material in a working environment, The method (200) comprises: Initializing (210) an environmental model (115) of the working environment including the granular material, wherein selected physical characteristic values (IN) of the granular material and the working environment are used as variables in at least one probability density function, and a relative height map between an actual height state and a target height state of the working environment including the granular material is generated using the probability density function, step (210); Associating (220), for each action in a set of a plurality of actions of the vehicle (100), a change in the probability density function of the environmental model (115) and a reward value with achieving a target state, wherein, for providing the training data (125), the relative height map of the environmental model (115) and the vehicle position are updated for each action, step (220); The method (200) including the above.

2. In the initializing step (210), The probability density function is a continuous multivariate Gaussian probability density function, and / or The selected physical characteristic values include the specific mass of the granular material, the spread parameter of the granular material, the volume of each hill of the granular material, and the position of the center point of each hill with respect to the environment, The method (200) according to claim 1.

3. In the associating step (220), the working environment is divided into a plurality of boundary frames, The change in the probability density function is calculated individually for each boundary frame, The method (200) according to claim 1.

4. In the associating step (220), different changes and / or different reward values are associated with the action according to whether the action is a forward translation, a backward translation, or a rotation, and / or according to whether the vehicle position is within an allowable region or outside the allowable region of the working environment, The method (200) according to claim 1.

5. A method (300) for generating a strategy (135) configured to control a vehicle (100), in particular an off-road vehicle (100), wherein the vehicle (100) is configured to interact with granular material in a work environment, The method (300) comprises: providing (325) training data (125) by the method (200) according to claim 1; generating (330) the strategy (135) by a reinforcement learning agent (130) using the training data (125); Method (300).

6. The method (300) includes controlling (340) at least one actuator (102) of the vehicle (100) using a control signal (145) determined by the strategy (135). The method (300) according to claim 5.

7. An apparatus (105) configured to perform and / or control the steps of the method (200, 300) according to any one of claims 1 to 6 in corresponding units (110, 120, 130, 140).

8. A vehicle (100), in particular an off-road vehicle (100), configured to interact with granular material in a work environment, The vehicle (100) includes the apparatus (105) according to claim 7. Vehicle (100).

9. A computer program configured to perform and / or control the steps of the method (200, 300) according to any one of claims 1 to 6.

10. A machine-readable storage medium storing the computer program according to claim 9.

Images