METHOD FOR TRAINING A CONTROL UNIT FOR A RAIL VEHICLE, CONTROL UNIT AND RAIL VEHICLE

DE502021005773D1Active Publication Date: 2024-11-28SIEMENS MOBILITY GMBH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
DE502021005773
Authority / Receiving Office
DE · DE
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-01-29
Publication Date
2024-11-28
Estimated Expiration
2041-01-29

AI Technical Summary

Technical Problem

Existing control systems for rail vehicles, such as PID and MPC controllers, require significant tuning efforts and accurate system models, which can be time-consuming and impractical, especially for complex systems, and do not effectively handle delays in response to control inputs.

Method used

A method using machine learning techniques to train a surrogate model of the rail vehicle based on sensor data, generating an action selection rule through reinforcement learning to control the vehicle from one speed state to another, thereby reducing the need for complex model adjustments and improving control precision.

Benefits of technology

This approach enables efficient and precise training of the control system, allowing for reliable and variable control of rail vehicles without the need for extensive tuning or accurate system modeling, effectively addressing the limitations of existing control methods.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader
Need to check novelty before this filing date? Find Prior Art

Description

[0001] The invention relates to a method for training a control system for a rail vehicle. The invention further relates to a control system and a rail vehicle with a control system.

[0002] Automatic train operation requires a controller that can translate a desired trajectory from a higher-level planning unit into the acceleration and braking commands required to execute the trajectory. Typically, the controllers used require considerable tuning effort to adapt to the specific characteristics of each vehicle.

[0003] Typical control approaches are based on proportional-integral-derivative (PID) controllers and model predictive control (MPC) controllers. PID controllers are widely used because they are very easy to implement. However, they require significant tuning effort when used in a new vehicle. Furthermore, they do not consider more than the current time step and can therefore perform poorly in systems where there is a delay in the system's response to control inputs. In these cases, MPC controllers are typically used. However, MPC control requires an accurate model of the system to be controlled. For complex systems, this can be difficult or impossible to achieve with sufficient precision.

[0004] From EP 3 552 921 A1, an AUTONOMOUS VELOCITY PLANNING OF A MOVABLE ACTOR RESTRICTED TO A PREDEFINED PATH is known.

[0005] Furthermore, DE 10 2017 215 802 A1 describes a driver assistance system for rail vehicles.

[0006] Furthermore, the design of high-speed trains is reported in SHUANBAO YAO ET AL: "Optimization design for aerodynamic elements of high-speed trains", COMPUTERS AND FLUIDS, PERGAMON PRESS, NEW YORK, NY, UK, Vol. 95, March 3, 2014 (2014-03-03), pages 56-73.

[0007] The invention is based on the object of providing an improved method for training a control system for a rail vehicle, an improved control system and a rail vehicle with a control system.

[0008] This object is achieved according to the invention by a method for training a control system of a rail vehicle, a correspondingly trained control system, and a rail vehicle with a trained control system according to the independent claims. Advantageous embodiments are specified in the subclaims.

[0009] According to one aspect of the invention, a method for training a control of a rail vehicle is provided, the method comprising: Providing training data based on sensor data of a rail vehicle; training a surrogate model of the rail vehicle based on the training data regarding a relationship between a control of a drive of the rail vehicle and a resulting speed of the rail vehicle; training an action selection rule based on the training data and the surrogate model using machine learning and taking into account at least one objective control objective, wherein the action selection rule comprises control instructions for controlling the drive of the rail vehicle, which are configured to accelerate the rail vehicle from a first speed state to a second speed state;and generating a trained action selection rule, wherein the trained action selection rule includes control instructions configured to accelerate the rail vehicle and achieve the control objective.

[0010] This approach offers the technical advantage of providing an improved method for training a rail vehicle's control system. The control system is trained using machine learning techniques. Training data based on sensor data from the rail vehicle is provided. Based on this training data, a surrogate model of the rail vehicle is then trained to understand the relationship between the activation of a drive system and the resulting speed. Subsequently, an action selection rule is developed based on the training data and the trained surrogate model using reinforcement learning techniques and considering at least one control objective. This results in a trained action selection rule.Based on the trained action selection rule, the rail vehicle can be controlled and the predetermined control goal can be achieved.

[0011] A surrogate model of a rail vehicle, as defined in the application, is a virtual model of a real rail vehicle that represents all relevant features of the real rail vehicle. A surrogate model can be designed analogously to a virtual twin of a real machine and virtually simulate the operation of a real machine or rail vehicle. The surrogate model is thus configured to simulate a movement of a rail vehicle through appropriate control. The surrogate model can, for example, be designed as an appropriately trained neural network and can be trained to simulate or represent the driving behavior or control-relevant properties of the rail vehicle.

[0012] An action selection rule, as defined in the application, is a set of control instructions designed to accelerate a rail vehicle from a first speed state to a second speed state. The first and second speed states can be an actual state, in which the rail vehicle is located at a specific time, and a target state, into which the rail vehicle is to be brought by activating the drive system. Alternatively, the first and second speed states can be two temporally successive states of the rail vehicle, into which the rail vehicle is brought when activated by the control system. By executing the control instructions of the action selection rule, the respective rail vehicle can thus be controlled in accordance with the control objective to be achieved.

[0013] Control instructions, as defined in the application, are instructions or commands for controlling the rail vehicle. Control instructions can, for example, include accelerating or decelerating the rail vehicle, including detailed control of the drive system. Furthermore, control instructions can include shifting into different gears or into an energy-saving mode of the drive system. In addition, various control-related aspects can be addressed in the control instructions in the form of corresponding instructions.

[0014] Control objectives, as defined in the application, are goals to be achieved by controlling the system. For example, control objectives can be a speed to which the rail vehicle should accelerate. Alternatively or additionally, control objectives can be defined for various future points in time, for example, via speed trajectories. Alternatively, control objectives can define an energy consumption that must be achieved or not exceeded when controlling the rail vehicle. As defined in the application, the control objectives are based on training data, which in turn is generated from sensor data of a rail vehicle. Control objectives can thus be generated from the sensor data of the rail vehicle.The control objectives can be derived from the sensor data and, for example, describe target speeds or energy consumption levels according to which the rail vehicle would be controlled while the sensor data is being acquired. Control objectives can also be defined or modified during the execution of the trained action selection rule for controlling the rail vehicle by higher-level systems, such as corresponding planning modules for the rail vehicle.

[0015] According to one embodiment, the machine learning is designed as reinforcement learning.

[0016] This can achieve the technical effect of enabling efficient training of the action selection rule.

[0017] Reinforcement learning, as defined in this context, is a branch of machine learning that focuses on training operational participants to perform desired actions, thereby guiding them from a current state to a desired target state. This training is conducted with a control goal in mind, which is achieved by the participant performing the action. The training may include maximizing a reward function that represents the action to be performed and the goal to be achieved to the participant.

[0018] According to one embodiment, training comprises: randomly modifying the at least one objective control objective based on the training data and defining modified control objectives; and training the action selection rule with respect to fulfillment of the modified control objectives.

[0019] This offers the technical advantage of providing more precise training for the rail vehicle's control system. By randomly modifying at least one control target and defining modified control targets, the training data can be used more effectively for training the control system. Modifying the control targets and generating or defining modified control targets allows for the definition of control targets that are supported by, but not limited to, the training data.

[0020] The modified control objectives thus result in control objectives that are not based on training data, but rather represent control objectives that go beyond those according to which the rail vehicle was controlled during the acquisition of the sensor data. By training the action selection rule based on these modified control objectives, an improved trained action selection rule can be generated that defines control instructions for an increased number of different control objectives. This allows for improved control.

[0021] According to one embodiment, training the action selection rule includes maximizing a reward function, wherein the reward function is maximal for an action selection rule that satisfies the objective control goal and / or the modified control goals.

[0022] This allows for the technical advantage of achieving the most precise possible training of the action selection rule according to the techniques of reinforcement learning.

[0023] According to one embodiment, the reward function takes into account a difference between a velocity state achieved by executing a control action of the action selection rule and the objective control target and / or the modified control targets.

[0024] This allows for the technical advantage of providing efficient training of the action selection rule and a precisely trained action selection rule that accurately fulfills the desired control objective.

[0025] In one embodiment, the maximization of the reward function is performed by an artificial neural network.

[0026] This allows for the technical advantage of providing efficient training of the action selection rule or the control of the rail vehicle.

[0027] According to one embodiment, the control objective includes a target speed of the rail vehicle and / or a target energy consumption and / or a target acceleration and / or low-wear acceleration and / or braking behavior.

[0028] This allows for the technical advantage of providing an efficiently trained action selection rule and, consequently, an efficiently trained control system for a rail vehicle. The action selection rule trained in this way comprises control instructions suitable for controlling the rail vehicle while taking the aforementioned control objectives into account.

[0029] According to one embodiment, the training data is recorded during a procedure of the rail vehicle and includes sensor data of state variables, control actions and speed trajectories, wherein the state variables include speed data, acceleration data, position data, specification data of the drive and / or the rail vehicle, wherein the control actions include drive and / or brake actuations, and wherein the speed trajectories describe corresponding temporal speed developments of the rail vehicle.

[0030] This provides the technical advantage of enabling precise training of the control system through comprehensive training data.

[0031] According to a second aspect of the invention, a control system for a rail vehicle is provided, wherein the control system is trained according to a method for training a control system of a rail vehicle according to one of the preceding embodiments, and wherein the control system is configured to control the rail vehicle by executing the trained action selection rule.

[0032] This allows for the provision of an improved control system for a rail vehicle, trained using machine learning methods, in particular reinforcement learning. The control system comprises a trained action selection rule, which is trained according to the inventive method for training a control system for a rail vehicle according to the embodiments described above. Based on the trained action selection rule, which contains a plurality of control instructions for controlling the rail vehicle taking into account various control objectives, the rail vehicle can be controlled to achieve the respective control objectives.By training the action selection rule based on the surrogate model of the rail vehicle, the controller can be trained for any rail vehicle corresponding to the surrogate model. This eliminates the need for complex adjustments to the controller model used for a new rail vehicle, as is necessary with a Model Predictive Control (MPC) controller. This enables precise, reliable, and variable control of rail vehicles.

[0033] According to a third aspect, a rail vehicle is provided with a control system according to one of the preceding embodiments.

[0034] This makes it possible to provide a rail vehicle with improved control and the advantages mentioned above.

[0035] According to a fourth aspect, a computer program product is comprehensively provided with instructions which, when the program is executed by a data processing unit, cause it to execute the procedure for training a control system for a rail vehicle according to one of the preceding embodiments.

[0036] The features and advantages of this invention described above, as well as the manner in which they are achieved, will become clearer and more easily understood through the explanations of the following highly simplified, schematic representations of preferred embodiments. These show: FIG 1 a flowchart of a method for training a control system of a railway vehicle according to one embodiment; FIG 2 a schematic representation of a railway vehicle with a control system according to one embodiment; and FIG 3 a schematic representation of a computer program product.

[0037] FIG 1Figure 1 shows a flowchart of a method 100 for training a control system 200 of a rail vehicle 201 according to an embodiment.

[0038] To train the control system 200 of the rail vehicle 201, training data 101 are first provided in a first process step. The training data is based on sensor data from a rail vehicle 201 and, in the illustrated embodiment, includes state variables 217, control actions 219, and velocity trajectories 221. The training data 203, in particular the sensor data of the rail vehicle 201, can be acquired, for example, during operation of the rail vehicle 201 or a comparable rail vehicle by means of appropriate sensors.

[0039] In the sense of the application, state variables 217 describe points within a state space that describes various states of the rail vehicle 201. In particular, the state variables 217 can include speed data, acceleration data, and / or location data of the rail vehicle 201 that were recorded during the movement of the rail vehicle 201 or the comparable rail vehicle. The comparable rail vehicle can, for example, be a rail vehicle of the identical type. Furthermore, the state variables 217 can include specification data of the drive and / or the rail vehicle 201, which include, for example, the type of drive, maximum power or speed and other parameters of the drive, or a size or weight of the rail vehicle 201.

[0040] Control actions 219, as defined in the application, are actions performed during the procedure of the rail vehicle 201 for controlling the rail vehicle 201 by the control unit 200. Control actions 219 may, for example, include the actuation of the accelerator pedal or the control lever of a rail vehicle, or the actuation of the brake of the rail vehicle 201.

[0041] In the sense of the application, speed trajectories 221 are traces of temporally successive speed values ​​that run through the state space and were reached during the movement of the rail vehicle 201.

[0042] After providing the training data 203, a surrogate model 205 of the rail vehicle 201 is generated or trained in a further method step 103, and the surrogate model 205 is taught a relationship between controls of a drive 207 of the rail vehicle 201 and a resulting speed of the rail vehicle 201.

[0043] Based on the training data 203, which were recorded during the operation of the rail vehicle 201, the surrogate model 205, which represents a virtual copy of the rail vehicle 201, is trained using machine learning methods to determine which control actions 219, such as actuation of the accelerator pedal or application of the brake, lead to which final speeds of the rail vehicle 201, which are determined based on the corresponding state variables 217. The surrogate model 205 trained in this way thus enables a simulation of the control system 200 of the rail vehicle 201, in which corresponding speed trajectories 221 can be achieved by executing appropriate control functions 219.

[0044] In a further process step 105, an action selection rule 209 is trained based on the training data 203 and the trained surrogate model 205 using methods of reinforcement learning and taking into account at least one objective control goal 211. The action selection rule 209 comprises control instructions for controlling the drive 207 of the rail vehicle 201. These control instructions are configured to accelerate the rail vehicle 201 from a first speed state to a second speed state.

[0045] The control instructions can, for example, correspond to the control actions 219, which were recorded as training data 203 during the operation of the rail vehicle 201, and include accelerating or decelerating the rail vehicle. The action selection rule 209 can have a plurality of control instructions by means of which the drive 207 of the rail vehicle 201 can be controlled. The control instructions are designed such that the objective control goal 211 is achieved during the actuation of the drive 207 of the rail vehicle 201. The objective control goal 211 can, for example, be a final speed to be achieved, a desired energy consumption, or a maximum acceleration of the rail vehicle 201, each of which must be achieved or maintained during the actuation of the rail vehicle 201. The objective control goal 211 can also be provided by the training data 203.For example, the control target 211 may be represented by the recorded velocity trajectories 221 of the training data 203.

[0046] The first and second speed states can each be an actual state and a target state, respectively, of the rail vehicle 201, whereby the rail vehicle 201 is to be moved from the actual state to the target state by control according to the control instructions of the action selection rule 209. Alternatively, the first and second speed states can be two temporally successive states of the state space, into which the rail vehicle 201 is to be moved by control according to the control instructions of the action selection rule.

[0047] To train the action selection rule 209 by reinforcement learning, an arbitrarily selected action selection rule with arbitrary control instructions is trained or optimized based on the training data 203, including the state variables 217 of the state space of the rail vehicle 201 and taking into account the respective selected control objective 211, so that the action selection rule 209 is configured to achieve the selected control objective 211. In the embodiment shown, the training of the arbitrarily selected action selection rule 209 is carried out in a further process step 111 by maximizing a correspondingly configured reward function. The reward function can, for example, be defined as a difference between a velocity state achieved by executing a control action 219 of the action selection rule 209 and the objective control objective 211.The training of the action selection rule 209 is thus carried out by modifying the control instructions or the action selection rule 209 such that the correspondingly defined reward function reaches a maximum value. An action selection rule 209 with a maximum reward function is therefore capable of achieving the selected control goal 211. By training the action selection rule 209 in this way, the state space of the rail vehicle 201, defined by the training data 203 and containing various states of the rail vehicle 201, can be traversed to determine the optimal action selection rule 209, which is configured to transfer the rail vehicle 201 in an optimized trajectory into states that ensure optimal control 200 and the achievement of the predetermined objective control goal 211 by executing the corresponding control instructions.

[0048] According to the embodiment shown, to train the action selection rule 209, the objective control target 211 is randomly changed in a process step 109, and modified control targets 215 are generated. By randomly changing the control target 211, modified control targets 215 can be generated that may differ from the training data 203.

[0049] For example, the objective control target 211 can be formed by a speed trajectory 221 of the training data 203. The respective speed trajectory 221 can be supported by the sensor data of the rail vehicle 201 that was recorded during the movement of the rail vehicle 201. By changing individual values ​​of the speed trajectory 221, modified speed trajectories 221 can thus be generated as modified control targets 215, wherein the modified speed trajectories 221 are not fully supported by the training data 203 and have speed values ​​that differ during the movement of the rail vehicle 201 and the recording of the respective speed values ​​of the rail vehicle 201. By changing the speed trajectory 221, points in the state space of the rail vehicle 201 can thus be reached for which no explicit training data 203 was generated.

[0050] Alternatively, several control objectives 211, 215 can be considered during the training of the action selection rule 209, so that the action selection rule 209 is configured to fulfill a plurality of control objectives 211, 215. In addition to the final speed to be achieved by the rail vehicle 201, the control objectives can, for example, include the energy consumption of the rail vehicle 201 or a maximum permissible acceleration of the rail vehicle 201, which must be fulfilled or considered during the control 200 of the rail vehicle 201.

[0051] After training the action selection rule 209 in process step 105, a correspondingly trained action selection rule 213 is generated based on the initial action selection rule 209, which includes control instructions that are set up to accelerate the rail vehicle 201 and to fulfill the control objective 211 or the modified control objectives 215.

[0052] Training the action selection rule 209, or maximizing the reward function, can be performed by a trained artificial intelligence, for example, by a suitably trained neural network. By maximizing the appropriately configured reward function, an action selection rule 213 can be generated that fulfills the corresponding control objectives 211 and 215, thus ensuring optimized control 200 of the rail vehicle 201.

[0053] FIG 2Figure 1 shows a schematic representation of a rail vehicle 201 with a control system 200 according to one embodiment.

[0054] Fig. 2Figure 1 shows a rail vehicle 201 with a controller 200, wherein the controller 200 comprises an action selection rule 213 trained according to the inventive method 100 for training a controller 200 of a rail vehicle 201. The rail vehicle 201 further comprises a drive 207 and a sensor 223. Sensor values ​​from the drive 207 can be recorded via the sensor 223 in order to monitor the control of the drive 207 by means of the controller 200. The rail vehicle 201 further comprises a planning unit 225, which is connected to the controller 200. For controlling the rail vehicle 201, the controller 200 can thus receive a corresponding speed trajectory 221 from the planning unit 225, which describes a planned speed profile of the rail vehicle 201.By executing the control instructions defined by the trained action selection rule 213, the controller 200 can thus execute the corresponding control instructions suitable for accelerating the rail vehicle 201 according to the planned speed trajectory 221 of the planning unit 225. The controller 200 of the rail vehicle 201 can take various control objectives 211, 215 into account. For example, the rail vehicle 201 can be controlled with a predetermined maximum energy consumption. Alternatively or additionally, the rail vehicle 201 can be controlled taking into account a maximum permissible acceleration.Since the appropriately trained action selection rule 213 comprises corresponding control instructions for a plurality of different control objectives 211, 215, which are suitable for controlling the rail vehicle 201 taking into account the respective control objectives 211, 215, the corresponding control objectives 211, 215 that are to be achieved during the control of the rail vehicle 201 can be modified during the operation of the rail vehicle 201, and in particular when the controller 200 is already installed. A change to the controller 200 and, in particular, an adaptation of the respective defined control instructions is not necessary due to the training of the trained action selection rule 213.

[0055] FIG 3 shows a schematic representation of a computer program product 300.

[0056] Figure 3Figure 3 shows a computer program product 300 comprising instructions that, when executed by a processing unit, cause the unit to execute the method 100 according to one of the embodiments described above. In the embodiment shown, the computer program product 300 is stored on a storage medium 301. The storage medium 301 can be any storage medium known from the prior art.

[0057] Although the invention has been illustrated and described in detail by the preferred embodiment, the invention is not limited by the disclosed examples and other variations can be derived by the person skilled in the art without leaving the scope of protection of the invention as defined by the claims.

Claims

1. Method (100) for training a controller (200) of a rail vehicle (201) including: - providing (101) training data (203) based on sensor data for a rail vehicle (201); - training (103) a surrogate model (205) of the rail vehicle (201) based on the training data (203) with respect to a relationship between an actuation of a drive (207) of the rail vehicle (201) and a resulting speed of the rail vehicle (201); - training (105) an action selection rule (209) based on the training data (203) and the surrogate model (205) using machine learning and taking into account at least one objective control target (211), wherein the action selection rule (209) includes control instructions for actuating the drive (207) of the rail vehicle (201) configured to accelerate the rail vehicle (201) from a first speed state into a second speed state; and - generating (107) a trained action selection rule (213), wherein the trained action selection rule (213) includes control instructions configured to accelerate the rail vehicle (201) and to meet the control target (211).

2. Method (100) according to claim 1, wherein the machine learning is embodied as reinforcement learning.

3. Method (100) according to claim 1 or 2, wherein the training (105) of the action selection rule (209) includes: randomised modification (109) of the at least one objective control target (211) based on the training data (203) and the definition of modified control targets (215); and training the action selection rule (209) in terms of meeting the modified control targets (215).

4. Method (100) according to claim 3, wherein the training (105) of the action selection rule (209) includes: maximising (111) a reward function, wherein the reward function is maximum for an action selection rule (209) that meets the objective control target (211) and / or the modified control targets (215).

5. Method (100) according to claim 4, wherein the reward function takes account of a difference between a speed state achieved by executing a control instruction of the action selection rule (209) and the objective control target (211) and / or the modified control targets (215).

6. Method (100) according to claim 1 or 2, wherein the training (105) of the action selection rules (209) includes: maximising (111) a reward function, wherein the reward function is maximum for an action selection rule (209) that meets the objective control target (211).

7. Method (100) according to claim 6, wherein the reward function takes into account a difference between a speed state achieved by executing a control instruction of the action selection rule (209) and the objective control target (211).

8. Method (100) according to one of claims 4 to 7, wherein the maximisation (111) of the reward function is executed by an artificial neural network.

9. Method (100) according to one of the preceding claims 3, 4 or 5, wherein the control target (211) and / or the modified control targets (215) include a target speed of the rail vehicle (201) and / or a target energy consumption and / or a target acceleration and / or low-wear acceleration behaviour and / or braking behaviour.

10. Method (100) according to one of the preceding claims 6, 7 or 8, wherein the control target (211) includes a target speed of the rail vehicle (201) and / or a target energy consumption and / or a target acceleration and / or low-wear acceleration behaviour and / or braking behaviour.

11. Method (100) according to one of the preceding claims, wherein the training data (203) is recorded while the rail vehicle (201) is moving and includes sensor data for state variables (217), control actions (219) and speed trajectories (221), wherein the state variables (217) include speed data, acceleration data, location data, specification data for the drive and / or the rail vehicle (201), wherein the control actions (219) include drive and / or brake actuations, and wherein the speed trajectories (221) describe corresponding temporal speed developments of the rail vehicle (201).

12. Controller (200) for a rail vehicle (201), wherein the controller (200) is trained in accordance with a method (100) for training a controller (200) of a rail vehicle (201) according to one of the preceding claims 1 to 11, and wherein the controller (200) is configured to control the rail vehicle (201) by executing the trained action selection rule (213).

13. Rail vehicle (201) with a controller (200) according to claim 12.

14. Computer program product (300) including commands which, when the program is executed by a data processing unit, cause it to execute the method (100) according to one of the preceding claims 1 to 11.