Integrated intelligent optimization method and system for autonomous train speed curve

Abstract

The present invention relates to an integrated intelligent optimization method and system for autonomous train speed curve, comprising: constructing an autonomous train speed curve optimization model under virtual marshaling based on discrete distance; converting the autonomous train speed curve optimization model into a Markov decision process; in the Markov decision process, using a deep reinforcement learning algorithm TD3 to train a neural network and an intelligent agent to obtain a trained neural network and intelligent agent; deploying the trained neural network and intelligent agent to an autonomous train to make an optimization decision on the autonomous train speed curve, thereby enabling autonomous safe, efficient and comfortable operation of the train.

Description

Technical Field

[0001] The present invention relates to the technical field of train operation optimization, and in particular to an integrated train autonomous operation speed curve intelligent optimization method and system. Background Art

[0002] In recent years, the global railway transportation system has developed rapidly due to its characteristics of high speed, large transportation capacity and high comfort. Due to the increase in passenger and cargo transportation demand and the more frequent occurrence of emergencies, the realization of integrated train dispatching and control is a key technology to solve the contradiction between transportation capacity and demand at this stage. Since some railway lines have reached the upper limit of transportation capacity, the current train control system cannot increase transportation capacity by increasing the number of train operations. Compared with building new lines, improving the capacity of line railway lines is a more economical and effective solution. Therefore, under the premise of ensuring the safe operation of trains, shortening the train running interval is a necessary means to realize the integration of train dispatching and control and improve line capacity and operation efficiency.

[0003] With the development of train-to-train communication and automatic control technology, the emergence of virtual marshaling enables trains to operate in a relative braking protection mode to improve train operation efficiency and line capacity. Under virtual marshaling, the rear train always takes the rear position of the front train after emergency braking as the braking end point and dynamically adjusts the speed according to the speed and position information of the front train. In addition, railway transportation needs will cause trains to be dynamically marshaled or unmarshaled, especially for high-speed or intercity railways. Two trains will be marshaled after leaving the turnout and running from two different tracks to the same track, or when running from the same track to two different tracks, they will be separated. This requires trains to not only operate safely in the relative braking protection mode, but also make the best decision on occupying key line resources, such as turnout routes, based on the operating status of the front train while considering energy saving and comfortable operation.

[0004] The application of the automatic train operation (ATO) system solves the problems of poor uniformity and lack of accuracy caused by traditional manual driving, and provides conditions for shortening the tracking interval between trains. However, the current ATO system cannot enable communication between trains to complete information exchange, and the interlocking system needs to lock and clear the route, which will seriously reduce the flexibility of virtual marshaling. Autonomous trains with decision-making capabilities have the ability to communicate with adjacent trains. They can also make autonomous decisions and apply for line resources based on the train operation and line resource utilization status on the line. Therefore, autonomous train operation is an effective solution to achieve integrated virtual marshaling.

[0005] In summary, in order to achieve autonomous, safe, efficient and comfortable operation of trains, it is necessary to design an integrated intelligent optimization method for autonomous train operation speed curve. Summary of the invention

[0006] 1. Technical issues to be resolved

[0007] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides an integrated intelligent optimization method and system for autonomous train operation speed curve, which can realize autonomous, safe, efficient and comfortable operation of trains.

[0008] (II) Technical solution

[0009] In order to achieve the above object, the main technical solutions adopted by the present invention include:

[0010] In a first aspect, an embodiment of the present invention provides an integrated intelligent optimization method for autonomous train speed curve, including: constructing an autonomous train speed curve optimization model under virtual marshaling based on discrete distance; converting the autonomous train speed curve optimization model into a Markov decision process; in the Markov decision process, using a deep reinforcement learning algorithm TD3 to train a neural network and an intelligent agent to obtain a trained neural network and intelligent agent; deploying the trained neural network and intelligent agent to an autonomous train to make an optimization decision on the autonomous train speed curve;

[0011] The autonomous train speed curve optimization model includes multiple constraints; the first constraint among the multiple constraints is:

[0012] d(s)+ebd(v a (s))+Δ l +Δ sm ≤P l (t(s))+ebd(V l (t(s)));

[0013]

[0014] Where d(s) represents the position of the end of segment s; ebd(v a (s)) represents the emergency braking distance of autonomous train a when the final speed of segment s is v(s); Δ l Indicates the length of the preceding vehicle; Δ sm Indicates the minimum safety interval; P l (t(s)) represents the position of the preceding vehicle at time t(s); t(s) represents the time it takes for the autonomous train to reach the end of section s; V l (t(s)) represents the speed of the preceding vehicle at time t(s); s' represents the segment connected to and preceding segment s; S represents the segment index set; Indicates the location of the outgoing turnout connection point; Indicates the location of the entry turnout connection point.

[0015] In a possible embodiment, the second constraint among the multiple constraints is:

[0016]

[0017] In the formula, Indicates the section index where the outgoing turnout is located; It indicates the time when autonomous train a arrives at the outgoing switch; Indicates the time when the preceding vehicle leaves the outgoing switch; Δ sw Indicates the duration of the turnout switching action; Indicates the section index where the entry turnout is located; It indicates the time when autonomous train a arrives at the station turnout; Indicates the time when the preceding vehicle leaves the station switch.

[0018] In a possible embodiment, the autonomous train speed curve optimization model further includes an objective function; the objective function is:

[0019]

[0020] In the formula, J represents the minimum objective function value; α represents the first weight; rt a (s) represents the total running time of autonomous train a in section s; β represents the second weight; ec a (s) represents the total operating energy consumption of autonomous train a in section s; γ represents the third weight; It represents the accumulated value of the acceleration change of autonomous train a in section s.

[0021] In one possible embodiment, in a Markov decision process, a deep reinforcement learning algorithm TD3 is used to train a neural network and an intelligent agent, including: generating multiple training scenarios for the reinforcement learning intelligent agent in the Markov decision process; wherein each of the multiple training scenarios includes but is not limited to line length, line topology, and line speed limit information; a training completion step: determining whether the training of the neural network and the intelligent agent is completed; if the training of the neural network and the intelligent agent is not completed, randomly selecting a target training scenario from the multiple training scenarios, and based on the target training scenario, resetting the reinforcement learning environment, and determining the maximum speed curve of the autonomous train and the planned speed of the preceding vehicle curve; execute the storage step: the agent selects an action according to the state information of the current environment and outputs it to the environment. After executing the action, the environment updates the state information and calculates the reward value, and stores the action, the state information of the previous time step of the current environment, the state information of the current time step of the current environment and the reward value as a group of storage data into the memory bank, and selects a random target group storage data from the memory bank, and uses the target group storage data to update the neural network parameters; judge whether the autonomous train has reached the end point; if the autonomous train has not reached the end point, return to execute the storage step; if the autonomous train has reached the end point, return to the step of judging the completion of training until the training is completed, and then stop the loop.

[0022] In one possible embodiment, the state of the Markov decision process is:

[0023]

[0024] In the formula, φ s represents the state of the Markov decision process; v a (s') represents the final speed of autonomous train a in section s'; d(s') represents the position of the end of section s'; l(s) represents the length of section s; V a (s) represents the maximum speed of autonomous train a in section s; rrt a (s') represents the remaining running time of autonomous train a to the next switch when it is at the end of section s'; rd a (s') represents the distance from the autonomous train a to the next switch when it is at the end of section s'; acc a (s') represents the acceleration of autonomous train a at the end of segment s'; represents the maximum safe action of autonomous train a in section s; It represents the minimum safe action of autonomous train a in section s.

[0025] In one possible embodiment, the reward function of the Markov decision process is:

[0026]

[0027] In the formula, r s Represents the reward value; case1 indicates that the action selected by the agent does not violate the constraints of the first constraint or the second constraint; μ represents a preset negative number; case2 indicates that the action selected by the agent violates the constraints of the first constraint or the second constraint.

[0028] In a possible embodiment, the environment of the Markov decision process includes a virtual train dynamics simulation environment and a train operation environment.

[0029] In a second aspect, the embodiment of the present application provides an integrated train autonomous operation speed curve intelligent optimization system, including: a construction module for constructing an autonomous train speed curve optimization model under virtual marshaling based on discrete distance;

[0030] A conversion module is used to convert the autonomous train speed curve optimization model into a Markov decision process; a training module is used to train the neural network and the intelligent agent using the deep reinforcement learning algorithm TD3 in the Markov decision process to obtain the trained neural network and the intelligent agent; a deployment module is used to deploy the trained neural network and the intelligent agent to the autonomous train to make an optimization decision on the autonomous train speed curve;

[0031] The autonomous train speed curve optimization model includes multiple constraints; the first constraint among the multiple constraints is:

[0032] d(s)+ebd(v a (s))+Δ l +Δ sm ≤P l (t(s))+ebd(V l (t(s)));

[0033]

[0034] Where d(s) represents the position of the end of segment s; ebd(v a (s)) represents the emergency braking distance of autonomous train a when the final speed of segment s is v(s); Δ l Indicates the length of the preceding vehicle; Δ sm Indicates the minimum safety interval; P l (t(s)) represents the position of the preceding vehicle at time t(s); t(s) represents the time when the autonomous train runs to the end of the segment s; V l (t(s)) represents the speed of the preceding vehicle at time t(s); s' represents the segment connected to and preceding the segment s; S represents the segment index set; Indicates the location of the outgoing turnout connection point; Indicates the location of the entry turnout connection point.

[0035] In a third aspect, an embodiment of the present application provides a storage medium having a computer program stored thereon, and the computer program is executed by a processor to execute the method described in the first aspect or any optional implementation of the first aspect.

[0036] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the method described in the first aspect or any optional implementation method of the first aspect is performed.

[0037] In a fifth aspect, the present application provides a computer program product, which, when running on a computer, enables the computer to execute the method in the first aspect or any possible implementation of the first aspect.

[0038] (III) Beneficial effects

[0039] The beneficial effects of the present invention are:

[0040] The embodiment of the present application provides an integrated intelligent optimization method and system for the speed curve of autonomous train operation, which constructs an optimization model of the autonomous train speed curve under virtual formation based on discrete distance, and converts the optimization model of the autonomous train speed curve into a Markov decision process; in the Markov decision process, the deep reinforcement learning algorithm TD3 is used to train the neural network and the intelligent agent, and the trained neural network and the intelligent agent are deployed in the autonomous train to make optimization decisions on the autonomous train speed curve, thereby realizing autonomous safe, efficient and comfortable operation of the train.

[0041] In order to make the above-mentioned objectives, features and advantages to be achieved by the embodiments of the present application more obvious and understandable, the following specifically cites preferred embodiments and describes them in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 A flow chart of an integrated train autonomous running speed curve intelligent optimization method provided in an embodiment of the present application is shown;

[0044] Figure 2 A schematic diagram showing a restriction of a turnout on a virtual marshaling provided in an embodiment of the present application is shown;

[0045] Figure 3 A schematic diagram of a reinforcement learning structure provided in an embodiment of the present application is shown;

[0046] Figure 4 A schematic diagram of a neural network and an intelligent agent training method provided in an embodiment of the present application is shown;

[0047] Figures 5A to 5D A schematic diagram of an optimized autonomous train speed curve under four scenarios provided in an embodiment of the present application is shown;

[0048] Figure 6 A schematic diagram of an integrated train autonomous operation speed curve intelligent optimization system provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0049] In order to better explain the present invention and facilitate understanding, the present invention is described in detail below through specific implementation modes in conjunction with the accompanying drawings.

[0050] In view of the problems of changeable virtual marshaling scenarios, difficult dynamic marshaling decisions and low line resource utilization caused by railway lines with complex line structures such as high-speed railways and intercity railways, the embodiments of the present application provide an integrated intelligent optimization method and system for the autonomous operation speed curve of trains, by constructing an autonomous train speed curve optimization model under virtual marshaling based on discrete distance, and converting the autonomous train speed curve optimization model into a Markov decision process, and in the Markov decision process, using the deep reinforcement learning algorithm TD3 to train neural networks and intelligent agents, and deploying the trained neural networks and intelligent agents to autonomous trains to make optimization decisions on the autonomous train speed curve, so as to achieve autonomous, safe, efficient and comfortable operation of trains.

[0051] In order to better understand the above technical solution, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a clearer and more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0052] See also Figure 1 , Figure 1The flowchart of an integrated train autonomous running speed curve intelligent optimization method provided by an embodiment of the present application is shown. It should be understood that the method can be executed by a device for optimizing the autonomous train speed curve, and the specific device of the device can be set according to actual needs, and the embodiment of the present application is not limited thereto. For example, the device can be a computer or a server. Specifically, the method includes:

[0053] Step S110, constructing an autonomous train speed curve optimization model under virtual marshaling based on discrete distance;

[0054] Step S120, converting the autonomous train speed curve optimization model into a Markov decision process;

[0055] Step S130, in the Markov decision process, using the deep reinforcement learning algorithm TD3 to train the neural network and the intelligent agent to obtain a trained neural network and intelligent agent;

[0056] Step S140, deploying the trained neural network and intelligent agent into the autonomous train to make an optimization decision on the speed curve of the autonomous train.

[0057] For step S110 , the autonomous train speed curve optimization model includes a plurality of constraints and an objective function.

[0058] It should be understood that the specific conditions of the constraint conditions and the specific functions of the objective function can be set according to actual needs, and the embodiments of the present application are not limited to this.

[0059] Alternatively, the autonomous train can be regarded as a mass point, and the forces it is subjected to during operation include the actual traction or braking force output by it, the Davis force and the additional resistance. And the basic dynamic model of the autonomous train is shown in formula (1), specifically:

[0060]

[0061] In the formula, acc a (s) represents the acceleration of autonomous train a in section s; a (s) represents the actual force output by autonomous train a in section s; s' represents the section connected to and before section s; v a (s') represents the final speed of autonomous train a in section s'; df(v a (s')) is used to indicate the Davis force on the autonomous train a in section s calculated using the final speed of the autonomous train a in section s'; af(s) indicates the additional resistance of section s to the train; ρ indicates the rolling coefficient; m a represents the train quality of autonomous train a; S represents the segment index set.

[0062] And, the actual force output by the autonomous train a in section s can be calculated according to formula (2), specifically:

[0063]

[0064] In the formula, of a (s) represents the actual force output by autonomous train a in section s; It represents the ratio of the actual traction force output by autonomous train a in section s to the maximum traction force that can be output; represents the ratio of the actual braking force output by the autonomous train a in section s to the maximum braking force that can be output; the maximum traction and braking force that can be output in section s are calculated using the final speed of the autonomous train in section s', tf a (v a (s')) indicates that the maximum traction force that the autonomous train a can output in section s is calculated using the final speed of the autonomous train a in section s'; bf a (v a (s')) indicates that the maximum braking force that can be output by the autonomous train a in section s is calculated using the final speed of the autonomous train a in section s'.

[0065] And, the relationship between the terminal velocities of the autonomous train in two consecutive sections is described by formula (3), specifically:

[0066]

[0067] In the formula, v a (s) represents the final speed of autonomous train a in section s; v a (s') represents the final speed of autonomous train a in section s'; acc a (s) represents the acceleration of autonomous train a in section s; l(s) represents the length of section s.

[0068] And, the change in acceleration of the autonomous train in section s is calculated by formula (4), specifically:

[0069]

[0070] In the formula, represents the acceleration change of autonomous train a in section s; acc a (s') represents the acceleration of autonomous train a in section s'.

[0071] And, the actual running time of the autonomous train in section s can be calculated according to formula (5), specifically:

[0072]

[0073] In the formula, rt a(s) represents the actual running time of autonomous train a in section s; l(s) represents the length of section s; v a (s) represents the final speed of autonomous train a in section s; v a (s') represents the final speed of autonomous train a in section s'.

[0074] Furthermore, the autonomous train can generate a maximum speed curve according to the static information of the line (such as slope, static speed limit, etc.) and the received temporary speed limit command, and formula (6) limits the speed of the autonomous train at the end of section s to not exceed the maximum speed. Specifically:

[0075]

[0076] In the formula, represents the maximum speed of autonomous train a at the end of section s.

[0077] And, the energy consumption of the autonomous train in section s is calculated by formula (7), specifically:

[0078]

[0079] Where, ec a (s) represents the operating energy consumption of autonomous train a in section s; It represents the ratio of the actual traction force output by autonomous train a in section s to the maximum traction force that can be output; tf a (v a (s)) represents the maximum traction that the autonomous train a can output in section s; l(s) represents the length of section s.

[0080] And, the requirement that the autonomous train safely tracks its preceding train is guaranteed by formula (8), specifically:

[0081] d(s)+ebd(v a (s))+Δ l +Δ sm ≤P l (t(s))+ebd(V l (t(s))) (8);

[0082]

[0083] Where s represents the segment index; d(s) represents the position of the end of segment s; s * represents any segment from segment 0 to s, l(s * ) indicates segment s * The length of ebd(v a (s)) represents the emergency braking distance of autonomous train a when the final speed in section s is v(s); Δl Indicates the length of the preceding vehicle; Δ sm Indicates the minimum safety interval; due to the speed curve V of the front vehicle l is known, so P can be used l (t(s)) represents the position of the preceding train at time t(s); t(s) represents the time when the autonomous train reaches the end of section s, and s * Indicates any segment from segment 0 to s, rt a (s * ) indicates that the autonomous train is in section s * The running time of the vehicle ahead is V l is known, so V l (t(s)) represents the speed of the preceding vehicle at time t(s), then ebd(V l (t(s))) represents the speed V l (t(s)) is the emergency braking distance; s' represents the segment connected to and before segment s; S represents the segment index set; Indicates the location of the outgoing turnout connection point; Indicates the location of the entry turnout connection point.

[0084] It should be noted here that in the virtual marshaling, the front and rear vehicles operate in a relative braking mode, that is, after both vehicles perform emergency braking at the same time, the front position of the rear vehicle cannot exceed the rear position of the front vehicle minus the safety interval Δ sm Since the autonomous train may be on a different route from the preceding train when approaching the outgoing turnout, that is, the two trains are running in parallel, only when the end point of the autonomous train's emergency braking passes the connection point of the turnout do the two trains need to comply with the safety rule of relative braking. Similarly, when the end point of the autonomous train's emergency braking passes the connection point of the turnout of the approach route to the station, even if the two trains are still running on the same track, the two trains do not need to comply with the safety rule of relative braking.

[0085] And, the safe operation of the autonomous train in the turnout section can be guaranteed by formula (9). The influence of turnout on the autonomous train in the virtual marshaling operation mode is as follows: Figure 2 As shown, the solid line and dash-dot line in the upper part of the figure represent the paths of the leading train and the autonomous train, respectively, and the two rectangular shadow parts represent the turnouts. The thick solid line and dash-dot line in the lower part represent the position-time trajectory of the two trains, respectively. The leading train passes through the main line at the first station while the autonomous train runs on the siding, so the outbound turnout needs to be switched from the positioning state to the reverse state, which may cause the two trains to conflict at the turnout. The autonomous train can only reach the starting point of this turnout when this outbound turnout is switched to the reverse state.

[0086] like Figure 2As shown, the time when the front vehicle leaves this switch is Therefore, the turnout The state transition can be completed when sw is the duration of the turnout switching action, so the earliest time for the autonomous train to arrive at this turnout is Right now Similarly, the earliest time that the autonomous train arrives at the next station’s turnout is At least in, Indicates the section index where the outgoing turnout is located; Indicates the section index where the entry turnout is located.

[0087] Therefore, the specific formula of formula (9) is as follows:

[0088]

[0089] In the formula, Indicates the section index where the outgoing turnout is located; It indicates the time when autonomous train a arrives at the outgoing switch; Indicates the time when the preceding vehicle leaves the outgoing switch; Δ sw Indicates the duration of the turnout switching action; Indicates the section index where the entry turnout is located; It indicates the time when autonomous train a arrives at the station turnout; Indicates the time when the preceding vehicle leaves the station switch.

[0090] Optionally, the objective function of the autonomous train speed curve optimization model is as shown in formula (10), which is to minimize the weighted sum of the total running time, total energy consumption and the accumulated value of acceleration change of the autonomous train, specifically:

[0091]

[0092] In the formula, J represents the minimum objective function value; α represents the first weight; rt a (s) represents the total running time of autonomous train a in section s; β represents the second weight; ec a (s) represents the total operating energy consumption of autonomous train a in section s; γ represents the third weight; It represents the accumulated value of the acceleration change of autonomous train a in section s.

[0093] For step S130, in the Markov decision process, the deep reinforcement learning algorithm TD3 is used to train the neural network and the intelligent agent, and the trained neural network and the intelligent agent can be set according to actual needs, and the embodiments of the present application are not limited to this.

[0094] It should be noted here that the neural network includes LSTM network, Actor network and Critic network.

[0095] Optionally, the deep reinforcement learning framework constructed by the present invention is as follows Figure 3 See Figure 3 The environment consists of two parts. The first part is the train dynamics simulation under virtual marshaling, and the other part is the train operation scenario. After the environment executes the action selected by the agent, the environment state transitions to the next state. The reward function is related to the train operation efficiency, energy consumption and comfort. The above elements will be stored in the agent's memory bank during the training process to update the neural network. Since the agent's neural network introduces LSTM, it has a more powerful ability to handle sequence problems. The following is the specific content of the agent, action, environment, state and reward of the Markov decision process.

[0096] Among them, for the agent and action:

[0097] The agent acts as the controller of the autonomous train and selects action a in each section s. s Control the actual force output by the train. In order to achieve smoother train control, a s The value range of a is [-1,1], and s Represents the ratio of the actual force output by the train to the maximum force that can be output. s <0, then the train output size is a s ×bf a (v a (s')) braking force. If a s > 0, the train output size is a s ×tf a (v a It should be noted that s' represents the previous interval, and the maximum braking force and traction force that can be output in the current interval s are calculated based on the final speed of the previous interval s'.

[0098] And, for the environment and state,

[0099] like Figure 2 As shown in Figure 1, the environment consists of two parts. The first part is the train dynamics simulation of virtual marshaling based on formulas (1) to (9). The second part is the train operation scenario, which generates the autonomous train maximum speed curve and the preceding train planned speed curve according to the line topology, stop mode and speed limit information.

[0100] And, the state of the autonomous train consists of 9 elements, as shown in formula (11):

[0101]

[0102] In the formula, φ s represents the state of the Markov decision process; v a (s') represents the final speed of autonomous train a in section s'; d(s') represents the position of the end of section s'; l(s) represents the length of section s; V a (s) represents the maximum speed of autonomous train a in section s; rrt a (s') represents the remaining running time of autonomous train a to the next switch when it is at the end of section s'; rd a (s') represents the distance from the autonomous train a to the next switch when it is at the end of section s'; acc a (s') represents the acceleration of autonomous train a at the end of segment s'; represents the maximum safe action of autonomous train a in section s; It represents the minimum safe action of autonomous train a in section s.

[0103] And, for rewards:

[0104] The definition of the reward function is shown in formula (12):

[0105]

[0106] In the formula, r s represents the reward value; case 1 represents that the action selected by the agent does not violate the constraint expressed by formula (8) or formula (9); μ represents a preset negative number, and the specific value of μ can be set according to actual needs. For example, μ is a negative number with a large absolute value; case 2 represents that the action selected by the agent violates the constraint expressed by formula (8) or formula (9).

[0107] To facilitate understanding of the present application, the specific process of using the deep reinforcement learning algorithm TD3 to train the neural network and intelligent agent in the Markov decision process to obtain the trained neural network and intelligent agent is described below.

[0108] Optionally, see Figure 4 , Figure 4 FIG. 1 is a schematic diagram showing a training method for a Markov decision process provided by an embodiment of the present application. Figure 4 As shown, the training method includes:

[0109] Step S401, generating multiple training scenarios for the reinforcement learning agent in the Markov decision process. Each of the multiple training scenarios includes but is not limited to line length, line topology and line speed limit information;

[0110] Step S402, initializing the TD3 algorithm neural network-level agent memory bank combined with LSTM;

[0111] Step S403, determining whether the training of the neural network and the agent is completed;

[0112] If the training of the neural network and the intelligent agent is not completed, step S404 is executed; if the training of the neural network and the intelligent agent is completed, step S408 is executed;

[0113] Step S404, randomly selecting a target training scenario from a plurality of training scenarios, and based on the target training scenario, resetting the reinforcement learning environment, and determining a maximum speed curve of the autonomous train and a speed curve of a preceding vehicle plan;

[0114] Step S405, the agent selects an action according to the state information of the current environment and outputs it to the environment. After executing the action, the environment updates the state information and calculates the reward value, and stores the action, the state information of the previous time step of the current environment, the state information of the current time step of the current environment and the reward value as a set of storage data in the memory bank;

[0115] Step S406, select a random target group storage data from the memory bank, and use the target storage data to update the neural network and the agent. The target group storage data is any set of storage data in the memory bank, and the target group storage data includes the target action, the state information of the previous time step of the corresponding environment, the state information of the current time step of the corresponding environment, and the target reward value;

[0116] Step S407, determining whether the autonomous train has reached the destination;

[0117] If the autonomous train has not reached the destination, the process returns to step S405; if the autonomous train has reached the destination, the process returns to step S403;

[0118] Step S408, the training of the neural network and the agent is completed.

[0119] For step S140, the trained neural network and intelligent agent are deployed in the autonomous train. The specific process of optimizing the speed curve of the autonomous train can be set according to actual needs, and the embodiments of the present application are not limited to this.

[0120] Optionally, the trained neural network and intelligent agent are deployed in the autonomous train, and an operation scenario that requires optimization of the autonomous train speed curve is generated. Then, the reinforcement learning environment is reset according to the scenario data, the maximum speed curve of the autonomous train and the speed curve of the preceding vehicle are calculated, and then it is determined whether the autonomous train has reached the destination at the current time step. If the autonomous train has not reached the destination at the current time step, the intelligent agent selects an action according to the state information of the current environment and outputs it to the environment. After executing the action, the environment updates the state information and calculates the reward value, and returns to execute the step of determining whether the autonomous train has reached the destination at the current time step until the autonomous train reaches the destination; if the autonomous train reaches the destination at the current time step, the optimized autonomous train speed curve and train control sequence are output, and the output results are visualized.

[0121] It should be noted here that the agent and the agent may be controlled by the same controller or not.

[0122] Therefore, with the help of the above technical solution, by constructing a speed curve optimization model of autonomous trains under virtual marshaling operation, and carefully considering the impact of the internal structure of the station on marshaling and disassembling, the safe and efficient operation of autonomous trains and the dynamic marshaling and disassembling process can be achieved.

[0123] Furthermore, the constructed model is converted into a Markov decision process, and the LSTM-TD3 algorithm is used to train the agent as the autonomous train controller to learn the control sequence generation strategy in a large number of operating scenarios to achieve real-time decision-making for autonomous trains.

[0124] In order to enable those skilled in the art to more clearly understand the technical solution of the present invention, the present technology will be described below in conjunction with specific scenarios.

[0125] Specifically, the scenarios and data for the training and application of the agent are selected from the real data of the Beijing-Shanghai High-Speed ​​Railway. After the agent is trained, four scenarios are selected to test the agent. The test scenario information is shown in Table 1 below, where the first column is the scenario sequence number, and the second column indicates the stop mode of the autonomous train and its leading vehicle at the front and rear stations, such as [(0,1), (1,1)] indicating that the leading vehicle does not stop at the first station but stops at the second station and the autonomous train stops at both the first station and the second station. The third column of data is the total length of the scenario, and the last column of data is the temporary speed limit information, such as [10000, 20000]: 200 indicates that the starting and ending points of the temporary speed limit interval are 10000m and 20000m respectively, and the speed limit value is 200km / h. In the third scenario, the starting and ending points of the temporary speed limit are both 20000m, and the speed limit value is 0, indicating that this scenario is an interruption scenario. The results obtained by the trained agent in the four test scenarios are shown in the following table. Figures 5A to 5D shown.

[0126] Table 1

[0127]

[0128]

[0129] It should be understood that the above-mentioned integrated train autonomous operation speed curve intelligent optimization method is only exemplary, and those skilled in the art can make various modifications based on the above-mentioned method, and the modified or deformed content is also within the scope of protection of this application.

[0130] See also Figure 6 , Figure 6 The block diagram of a system 600 for intelligent optimization of train autonomous speed curve for integration provided by an embodiment of the present application is shown. It should be understood that the system 600 can execute each step in the above method embodiment. The specific functions of the system 600 can be found in the description above. To avoid repetition, the detailed description is appropriately omitted here. The system 600 includes at least one software function module that can be stored in a memory in the form of software or firmware or solidified in the operating system (OS) of the system 600. Specifically, the system 600 includes:

[0131] A construction module 610 is used to construct an autonomous train speed curve optimization model under virtual marshaling based on discrete distance;

[0132] A conversion module 620, for converting the autonomous train speed curve optimization model into a Markov decision process;

[0133] A training module 630 is used to train the neural network and the intelligent agent using the deep reinforcement learning algorithm TD3 in the Markov decision process to obtain a trained neural network and intelligent agent;

[0134] A deployment module 640 is used to deploy the trained neural network and intelligent agent to the autonomous train to make an optimization decision on the speed curve of the autonomous train;

[0135] The autonomous train speed curve optimization model includes multiple constraints; the first constraint among the multiple constraints is:

[0136] d(s)+ebd(v a (s))+Δ l +Δ sm ≤P l (t(s))+ebd(V l (t(s)));

[0137]

[0138] Where d(s) represents the position of the end of segment s; ebd(v a (s)) represents the emergency braking distance of autonomous train a when the final speed of segment s is v(s); Δ l Indicates the length of the preceding vehicle; Δ sm Indicates the minimum safety interval; P l (t(s)) represents the position of the preceding vehicle at time t(s); t(s) represents the time when the autonomous train runs to the end of the segment s; V l (t(s)) represents the speed of the preceding vehicle at time t(s); s' represents the segment connected to and preceding the segment s; S represents the segment index set; Indicates the location of the outgoing turnout connection point; Indicates the location of the entry turnout connection point.

[0139] Since the system described in the above embodiment of the present invention is a system used to implement the method of the above embodiment of the present invention, a person skilled in the art can understand the specific structure and variation of the system / device based on the method described in the above embodiment of the present invention, and thus will not be described in detail here. All systems used in the method of the above embodiment of the present invention belong to the scope of protection of the present invention.

[0140] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0141] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions.

[0142] It should be noted that in the claims, any reference numerals placed between brackets shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention may be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In the claims enumerating several means, several of these means may be embodied by the same hardware. The use of the words first, second, third, etc., is for convenience of expression only and does not indicate any order. These words may be understood as part of the component name.

[0143] In addition, it should be noted that, in the description of this specification, the description of the terms "one embodiment", "some embodiments", "embodiment", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0144] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments after knowing the basic creative concept. Therefore, the claims should be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present invention.

[0145] Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention should also include these modifications and variations.

Claims

1. An integrated train autonomous running speed curve intelligent optimization method, characterized in that: include: Construct an autonomous train speed curve optimization model under virtual marshaling based on discrete distance; Converting the autonomous train speed curve optimization model into a Markov decision process; In the Markov decision process, the deep reinforcement learning algorithm TD3 is used to train the neural network and the intelligent agent to obtain the trained neural network and the intelligent agent; Deploying the trained neural network and intelligent agent into an autonomous train to make an optimization decision on a speed curve of the autonomous train; The autonomous train speed curve optimization model includes multiple constraints; the first constraint among the multiple constraints is: d(s)+ebd(v a (s))+Δ l +Δ sm ≤P l (t(s))+ebd(V l (t(s))); Where d(s) represents the position of the end of segment s; ebd(v a (s)) represents the emergency braking distance of autonomous train a when the final speed of the section s is v(s); Δ l Indicates the length of the preceding vehicle; Δ sm Indicates the minimum safety interval; P l (t(s)) represents the position of the preceding vehicle at time t(s); t(s) represents the time when the autonomous train runs to the end of the segment s; V l (t(s)) represents the speed of the preceding vehicle at time t(s); s' represents the segment connected to and preceding the segment s; S represents the segment index set; Indicates the location of the outgoing turnout connection point; Indicates the location of the entry turnout connection point.

2. The method according to claim 1, characterized in that The second constraint among the plurality of constraints is: In the formula, Indicates the section index where the outgoing turnout is located; Indicates the time when the autonomous train a arrives at the exit turnout; Indicates the time when the preceding vehicle leaves the exit switch; Δ sw Indicates the duration of the turnout switching action; Indicates the section index where the entry turnout is located; Indicates the time when the autonomous train a arrives at the station entry switch; Indicates the time when the preceding vehicle leaves the said station entry switch.

3. The method according to claim 2, characterized in that The autonomous train speed curve optimization model further includes an objective function; the objective function is: In the formula, J represents the minimum objective function value; α represents the first weight; rt a (s) represents the total running time of the autonomous train a in the section s; β represents the second weight; ec a (s) represents the total operating energy consumption of the autonomous train a in the section s; γ represents the third weight; It represents the accumulated value of the acceleration change of the autonomous train a in the section s.

4. The method according to claim 3, characterized in that In the Markov decision process, the deep reinforcement learning algorithm TD3 is used to train the neural network and the intelligent agent, including: Generate multiple training scenarios for the reinforcement learning agent in the Markov decision process; wherein each of the multiple training scenarios includes but is not limited to line length, line topology and line speed limit information; Determining whether the training of the neural network and the agent is completed; If the training of the neural network and the agent is not completed, a target training scene is randomly selected from the multiple training scenes, and based on the target training scene, the reinforcement learning environment is reset, and the maximum speed curve of the autonomous train and the speed curve of the preceding vehicle plan are determined; a storage step is performed: the agent selects an action according to the state information of the current environment and outputs it to the environment, and the environment updates the state information and calculates the reward value after executing the action, and stores the action, the state information of the previous time step of the current environment, the state information of the current time step of the current environment and the reward value as a group of storage data in a memory bank, and selects a random target group storage data from the memory bank, and uses the target group storage data to update the neural network parameters; determining whether the autonomous train has reached a destination; If the autonomous train has not reached the end point, the process returns to the execution and storage step; if the autonomous train reaches the end point, the process returns to the step of determining the completion of training until the training is completed, and then the cycle is stopped.

5. The method according to claim 4, characterized in that The state of the Markov decision process is: In the formula, φ s represents the state of the Markov decision process; v a (s') represents the final speed of the autonomous train a in the section s'; d(s') represents the position of the end of the section s'; l(s) represents the length of the section s; V a (s) represents the maximum speed of the autonomous train a in the section s; rrt a (s') represents the remaining running time of the autonomous train a to the next switch when the autonomous train a is at the end of the section s'; rd a (s') represents the distance between the autonomous train a and the next switch when it is at the end of the section s'; acc a (s') represents the acceleration of the autonomous train a at the end of the section s'; represents the maximum safe action of the autonomous train a in the section s; represents the minimum safety action of the autonomous train a in the section s.

6. The method according to claim 4, characterized in that The reward function of the Markov decision process is: In the formula, r s represents the reward value; case1 represents that the action selected by the agent does not violate the constraints of the first constraint condition or the second constraint condition; μ represents a preset negative number; case2 represents that the action selected by the agent violates the constraints of the first constraint condition or the second constraint condition.

7. The method according to claim 4, characterized in that The environment of the Markov decision process includes a virtual train dynamics simulation environment and a train operation environment.

8. An integrated train autonomous running speed curve intelligent optimization system, characterized in that: include: A construction module for constructing an autonomous train speed curve optimization model under virtual marshaling based on discrete distance; A conversion module, used for converting the autonomous train speed curve optimization model into a Markov decision process; A training module, used to train the neural network and the intelligent agent using the deep reinforcement learning algorithm TD3 in the Markov decision process to obtain the trained neural network and the intelligent agent; A deployment module, used to deploy the trained neural network and intelligent agent into the autonomous train to make an optimization decision on the speed curve of the autonomous train; The autonomous train speed curve optimization model includes multiple constraints; the first constraint among the multiple constraints is: d(s)+ebd(v a (s))+Δ l +Δ sm ≤P l (t(s))+ebd(V l (t(s))); Where d(s) represents the position of the end of segment s; ebd(v a (s)) represents the emergency braking distance of autonomous train a when the final speed of the section s is v(s); Δ l Indicates the length of the preceding vehicle; Δ sm Indicates the minimum safety interval; P l (t(s)) represents the position of the preceding vehicle at time t(s); t(s) represents the time when the autonomous train runs to the end of the segment s; V l (t(s)) represents the speed of the preceding vehicle at time t(s); s' represents the segment connected to and preceding the segment s; S represents the segment index set; Indicates the location of the outgoing turnout connection point; Indicates the location of the entry turnout connection point.

Images