Urban rail transit train energy saving optimization method, device and equipment and storage medium

Abstract

The application discloses a kind of urban rail transit train energy-saving optimization method, device, equipment and storage medium, method includes S10, obtains the state information under the train operating environment of current time and reward value Reward value is obtained using reward function, reward function includes the first reward function in the DDPG model and the second reward function that the work of traction force is done in the train operation process and the punctuality of train operation are combined;S20, based on the state information and reward value under train operating environment, select operation action and issue to train to make the train next time according to operation action train;S30, determine next time as current time, repeat the execution step S10-S20.In the application, train executes operation action every time, environment will immediately feed back a state information and reward value, guide the following manipulation sequence, to update and optimize train operation strategy, finally obtain a convergent ideal train operation strategy, reach the purpose of energy saving.

Description

Technical Field

[0001] This invention belongs to the field of rail transit technology, specifically relating to an energy-saving optimization method, device, equipment, and storage medium for urban rail transit trains. Background Technology

[0002] The term "reinforcement learning" originates from behavioral psychology and refers to the tendency of organisms to more frequently employ beneficial behavioral strategies in order to maximize their gains and minimize their losses. Reinforcement learning is a specific type of machine learning problem. In a reinforcement learning system, an agent observes its environment and takes actions based on those observations. After each action, the agent receives a reward and continuously improves its decision-making through these rewards. The process of reinforcement learning is essentially the process by which an agent maximizes its rewards through interaction with its environment.

[0003] In today's era, urban rail transit systems, represented by subways, have become an important choice for residents' daily travel due to their punctuality, speed, safety, and comfort. However, due to the large total capacity of rail transit systems, their energy consumption remains a problem to be solved. According to existing energy consumption data for urban rail systems, the largest proportion is the traction energy consumption of trains. Based on this, the energy consumption of the entire rail transit system can be reduced by adjusting the traction force of urban rail trains.

[0004] Therefore, research on energy-saving optimization of urban rail transit trains has significant practical implications. For example, the invention patent application No. 201910827467.6 discloses an online optimization method for energy-saving operation strategies of urban rail transit trains based on reinforcement learning. This method uses a deep Q-learning algorithm combined with a multi-objective speed adjustment model to solve the energy consumption model and achieve an energy-saving operation strategy for the train. However, this method uses the Q-Learning algorithm, and because the parameters of the Q-network are frequently updated with gradients while also being used to calculate the gradients of the Q-network and the policy network, the learning process is very unstable. Summary of the Invention

[0005] The present invention aims to provide a method, device, equipment and storage medium for energy-saving optimization of urban rail transit trains, so as to ensure the stability and efficiency of train energy-saving optimization.

[0006] The present invention solves the above-mentioned technical problems through the following technical means:

[0007] On one hand, embodiments of the present invention employ an energy-saving optimization method for urban rail transit trains, using a DDPG (Driving Energy Dissipation and Power Generation) model for train operation strategy selection. The method includes the following steps:

[0008] S10. Obtain the status information and reward value of the train operating environment at the current moment. The reward value is calculated using a reward function. The reward function includes the first reward function in the DDPG model and the second reward function that combines the work done by the traction force during the train's operation with the train's on-time performance.

[0009] S20. Based on the status information and reward value under the train operating environment, select an operating action and send it to the train so that the train will operate according to the operating action at the next moment.

[0010] S30. Determine the next moment as the current moment and repeat steps S10 to S20.

[0011] By employing a train operation energy consumption model, the state information and reward value of the train under the current operating environment are obtained. The reward function generating the reward value includes the first reward function from the DDPG model and a second reward function combining the work done by traction force during train operation with train punctuality. Based on the state information and reward value under the train operating environment, the model selects operational actions and issues them to the train's safety system to ensure the train operates according to the operational actions at the next moment. The next moment is defined as the current moment. The train operation energy consumption model uses real-time acquisition of the state information and reward value of the train operating environment fed back during operation. Each time the train executes an operational action, the environment immediately feeds back a state information and reward value, guiding subsequent operation sequences for updating and optimizing the train operation strategy. Ultimately, a convergent and ideal train operation strategy is obtained, achieving the goal of saving energy.

[0012] Furthermore, the formula for the reward function is expressed as follows:

[0013] R β (s)=ωR(s)+(1-ω)J(s)

[0014] Wherein, J(s) is the reward function in the DDPG model, R(s) is the reward function based on the combination of the work done by the traction force during the train's operation and the train's punctuality, ω is the proportion, and the value is 0-1. R(s) = α1(∫Fvdt) + α2(∑t-T0), α1+v2 = 1, ∫Fvdt is the work done by the traction force, F is the traction force, v is the train's speed, ∑t-T0 is the train's travel time deviation, t is the actual travel time, and T0 is the time in the train's operating section timetable.

[0015] Furthermore, when selecting and issuing a running action to the train based on the status information and reward value under the train's operating environment, the method also includes:

[0016] Random noise is introduced to convert the driving strategy into a random process, and the driving action is obtained by random sampling.

[0017] Furthermore, when the train receives the operation action, it also includes:

[0018] Determine whether the operation is dangerous based on the speed limit map of the driving section;

[0019] If so, an action request instruction is sent to reselect the running action;

[0020] If not, then perform the described operation.

[0021] Furthermore, the DDPG model for train energy consumption includes a critic network and an actor network. The critic network includes an Online policy network and a Target policy network, and the actor network includes an Online Q network and a Target Q network. When obtaining the state information and reward value of the train operating environment at the current moment, the model further includes:

[0022] The critic network transforms the state information and the reward value into state transition data and marks the priority of the state transition data, wherein the priority is marked from high to low according to the size of the reward value;

[0023] The state transition data after being marked with priority is stored in the replay memory buffer;

[0024] Data is extracted from the replay memory buffer in the order of priority for training the critic network and the actor network.

[0025] Furthermore, the method also includes:

[0026] The parameters of the Online Q network and the Online policy network are updated using stochastic gradient descent.

[0027] The parameters of the Target network and the Target policy network are updated using the soft update algorithm.

[0028] Secondly, embodiments of the present invention provide an energy-saving optimization device for urban rail transit trains, which uses a DDPG (Driving Energy Dissipation Model) for train operation strategy selection. The device includes:

[0029] The acquisition module is used to acquire the status information and reward value of the train operating environment at the current moment. The reward value is calculated using a reward function, which includes a first reward function in the DDPG model and a second reward function that combines the work done by the traction force during train operation with the train's on-time performance.

[0030] The decision-making module is used to select and issue a running action to the train based on the status information and reward value of the train operating environment so that the train will run according to the running action at the next moment.

[0031] The interactive feedback module is used to determine the next moment as the current moment and execute the actions of the acquisition module.

[0032] Furthermore, the formula for the reward function is expressed as follows:

[0033] R β (s)=ωR(s)+(1-ω)J(s)

[0034] Wherein, J(s) is the reward function in the DDPG model, R(s) is the reward function based on the combination of the work done by the traction force and the punctuality of the train during operation, ω is the proportion, and the value is 0-1. R(s) = α1(∫Fvdt) + α2(∑t-T0), α1 + α2 = 1, ∫Fvdt is the work done by the traction force, F is the traction force, v is the train speed, ∑t-T0 is the train travel time deviation, t is the actual travel time, and T0 is the time in the timetable of the train operating section.

[0035] Thirdly, embodiments of the present invention provide an energy-saving optimization device for urban rail transit trains, the device including a memory and a processor; wherein the processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the method described above.

[0036] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the method described above.

[0037] The advantages of this invention are:

[0038] (1) This invention obtains the state information and reward value of the train in the current operating environment by adopting a train operation energy consumption model. The reward function for generating the reward value includes the first reward function in the DDPG model and the second reward function that combines the work done by the traction force during train operation with the train's punctuality. By integrating the reinforcement learning DDPG algorithm into the urban rail train model, the environment will immediately provide a state information and reward value for each time the train performs an operation action, which will guide the subsequent operation sequence to update and optimize the train operation strategy, and finally obtain a convergent and ideal train operation strategy to achieve the purpose of saving energy.

[0039] (2) When selecting train operation strategies, random noise is introduced into the action decision-making mechanism to transform the action decision-making process from a deterministic process into a random process, and then the operation actions are sampled from this random process. By converting the original deterministic fixed-pattern selection of operation actions into random selection, all operation actions can be comprehensively explored.

[0040] (3) When the train safety system receives the operation action output by the model, it evaluates the safety of the strategy based on the speed limit map of the section, refuses to execute the operation strategy that exceeds the speed limit and imposes a penalty, and sends an action request instruction to the decision system to require the decision system to reselect the operation action to ensure the safety of train operation.

[0041] (4) Save the learned experience in the replay memory buffer (i.e., the experience pool) and mark the priority of the experience. Instead of randomly selecting samples from the experience pool, select samples for training based on priority. Reduce the priority of operations that endanger driving safety to the lowest level, and set the highest priority for samples that arrive on time but whose energy consumption has not been reduced to the lowest level. Select high-priority samples for priority training. By following the priority principle when selecting samples from the experience pool, the interference of invalid samples is avoided, the learning of potential samples is strengthened, and the learning efficiency and convergence speed of the system can be improved.

[0042] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0043] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings:

[0044] Figure 1 This is a flowchart of an embodiment of the energy-saving optimization method for urban rail transit trains according to the present invention;

[0045] Figure 2 This is a block diagram illustrating the principle of energy-saving optimization for urban rail transit trains according to an embodiment of the present invention;

[0046] Figure 3 This is a structural diagram of the DDPG model in this invention;

[0047] Figure 4 This is a structural diagram of an energy-saving optimization device for urban rail transit trains according to an embodiment of the present invention. Detailed Implementation

[0048] To further illustrate the features of the present invention, please refer to the following detailed description and accompanying drawings. The drawings are for reference and illustration only and are not intended to limit the scope of protection of the present invention.

[0049] like Figure 1 As shown in the figure, this embodiment discloses an energy-saving optimization method for urban rail transit trains, which uses the DDPG (Driving Energy Dissipation Model) for train operation strategy selection. The method includes the following steps:

[0050] S10. Obtain the status information and reward value of the train operating environment at the current moment. The reward value is calculated using a reward function. The reward function includes the first reward function in the DDPG model and the second reward function that combines the work done by the traction force during the train's operation with the train's on-time performance.

[0051] S20. Based on the status information and reward value under the train operating environment, select an operating action and send it to the train so that the train will operate according to the operating action at the next moment.

[0052] S30. Determine the next moment as the current moment and repeat steps S10 to S20.

[0053] It should be noted that this invention obtains the state information and reward value of the train in the current operating environment by using a train operation energy consumption model. The reward function for generating the reward value includes the first reward function in the DDPG model and the second reward function that combines the work done by the traction force during train operation with the train's punctuality. By integrating the reinforcement learning DDPG algorithm into the urban rail train model, the environment will immediately provide a state information and reward value after each train operation, guiding the subsequent operation sequence to update and optimize the train operation strategy, and finally obtain a convergent and ideal train operation strategy to achieve the goal of saving energy.

[0054] It should be noted that the DDPG (Deep Deterministic Policy Gradient) algorithm is a policy learning method that integrates deep learning neural networks into DPG. The DDPG algorithm is an offline, environment-free model algorithm that uses deep neural networks to approximate value functions. DDPG can be used to solve continuous action space problems.

[0055] like Figure 3 As shown, the DDPG algorithm includes an actor-critic framework, consisting of a critic network and an actor network. The critic network uses a neural network approximation function, taking the action and observation(a,s) as input and outputting Q(s,a). The actor network uses a neural network approximation policy function, taking the observation(s) as input and outputting the action(a).

[0056] The DDPG algorithm can update the critic network based on the reward, update the actor network along the direction suggested by the critic, train an ideal actor network, and finally obtain a satisfactory train operation strategy.

[0057] It should be noted that this embodiment integrates the reinforcement learning DDPG algorithm into the urban rail train model, and trains the train's operation strategy through the algorithm to achieve the effect of energy saving optimization.

[0058] In some embodiments, the reward function is expressed as:

[0059] R β (s)=ωR(s)+(1-ω)J(s)

[0060] Wherein, J(s) is the reward function in the DDPG model, R(s) is the reward function based on the combination of the work done by the traction force and the punctuality of the train during operation, ω is the proportion, and the value is 0-1. R(s) = α1(∫Fvdt) + α2(∑t-T0), α1 + α2 = 1, ∫Fvdt is the work done by the traction force, F is the traction force, v is the train speed, ∑t-T0 is the train travel time deviation, t is the actual travel time, and T0 is the time in the timetable of the train operating section.

[0061] It should be noted that, based on the train's section timetable and the train's energy consumption formula, the train's operation can be monitored in real time to ensure it meets the timetable's punctuality requirements and to track energy consumption. The energy consumption during train operation can be expressed as the work done by the train's traction force. This embodiment aims to reduce energy consumption by modifying the operating strategy as much as possible without affecting the train's punctuality. According to the train energy consumption model, the work done by the traction force during operation is substituted into the reward function of the DDPG algorithm. This, together with the original DDPG algorithm's reward function, forms a new reward function. The algorithm continuously improves along the path of energy conservation, ultimately obtaining a convergent and ideal train operating strategy.

[0062] Compared to Q-Learning and DQN algorithms frequently used in reinforcement learning, the learning process of the Q-network is highly unstable because its parameters are frequently updated with gradients while simultaneously calculating the gradients of both the Q-network and the policy network. The DDPG algorithm used in this embodiment creates two copies of the neural network for both the policy network and the Q-network, ensuring the stability and efficiency of the algorithm's learning, making convergence easier, and is particularly suitable for solving driving optimization problems with large action spaces and continuous computation.

[0063] In some embodiments, when selecting an action to send to the train based on the status information and reward value under the train operating environment, the method further includes:

[0064] Random noise is introduced to convert the driving strategy into a random process, and the driving action is obtained by random sampling.

[0065] It should be noted that, when selecting train operation strategies, random noise is introduced into the action decision-making mechanism, transforming the action decision-making process from a deterministic process into a stochastic process. The operational action 'a' is then sampled from this stochastic process. t By transforming the previously deterministic, fixed selection of actions into random choices, all possible actions can be fully explored.

[0066] In some embodiments, such as Figure 2 As shown, when the train receives the operation action, it also includes:

[0067] Determine whether the operation is dangerous based on the speed limit map of the driving section;

[0068] If so, an action request instruction is sent to reselect the running action;

[0069] If not, then perform the described operation.

[0070] Specifically, when receiving the train operation strategy, this embodiment first uses the judgment mechanism of the train safety system to draw a speed-time image of the trained train operation sequence. By comparing it with the speed limit-distance image of the train section, some results that do not meet the requirements are filtered out. That is, by adding the state of the previous moment to the action selection of the current moment, the train speed is calculated. The train speed is compared with the speed limit of the train section. If the train speed is greater than the speed limit of the section, it is determined that it constitutes dangerous operation and the operation action needs to be reselected to avoid dangerous operation and improve the training effect. At the same time, a larger penalty value is set for the train operation sequence that does not meet the safety specifications.

[0071] It should be noted that when the train safety system receives the operational actions output by the model, it evaluates the safety of the strategy based on the speed limit map of the section. If the driving strategy exceeds the speed limit, it refuses to execute it and imposes a penalty. At the same time, it sends an action request instruction to the decision-making system to require the decision-making system to reselect the operational actions to ensure driving safety.

[0072] In some embodiments, the environment performs action a t Then, the reward value and the new environment running status are returned. t+1 The online policy network in the actor network converts it into state transition data (s). t ,a t ,rt ,s t+1 ), stored in the replay memory buffer, as the dataset for training the online network, where s t ,a t ,r t ,s t+1 These are the segment status, the action performed, the reward value, and the status of the next segment.

[0073] In some embodiments, the method further includes:

[0074] The priority of the marked state transition data is determined by the reward value from high to low.

[0075] The state transition data after being marked with priority is stored in the replay memory buffer;

[0076] Data is extracted from the replay memory buffer in the order of priority for training the critic network and the actor network.

[0077] It should be noted that in this embodiment, the state transition data is prioritized and stored in the replay memory buffer. Operations that endanger driving safety are given the lowest priority, while samples that arrive on time but whose energy consumption is not reduced to the lowest priority are given the highest priority, forming a sum-tree structure with decreasing priority from top to bottom.

[0078] Data is extracted from the replay memory buffer in priority order to serve as small training sets for the Target Policy Network, Online Policy Network, TargetQ Network, and Online Q Network. A single training dataset is represented as (s t ,a t ,r t ,s t+1 ), s t ,a t ,r t ,S t+1 These represent the train's state at time t, the action taken, the reward function value obtained, and the state of the next segment, respectively.

[0079] It should be noted that the learned experience is saved in the replay memory buffer (i.e., the experience pool) and its priority is marked. Instead of randomly selecting samples from the experience pool, training is performed based on priority. Operations that endanger driving safety are given the lowest priority, while samples that arrive on time but do not minimize energy consumption are given the highest priority. High-priority samples are selected for training first. By following the priority principle in selecting samples from the experience pool, interference from invalid samples is avoided, and learning from potential samples is enhanced, thus improving the system's learning efficiency and convergence speed.

[0080] In some embodiments, the method further includes:

[0081] The parameters of the Online Q network and the Online policy network are updated using stochastic gradient descent.

[0082] The parameters of the Target network and the Target policy network are updated using the soft update algorithm.

[0083] like Figure 4 As shown, the second embodiment of the present invention also proposes an energy-saving optimization device for urban rail transit trains, which uses a DDPG (Driving Energy Dissipation Model) for train operation strategy selection. The device includes:

[0084] The acquisition module 10 is used to acquire the status information and reward value of the train operating environment at the current time. The reward value is calculated using a reward function, which includes a first reward function in the DDPG model and a second reward function that combines the work done by the traction force during train operation with the train's on-time performance.

[0085] Decision module 20 is used to select and issue a running action to the train based on the status information and reward value of the train operating environment so that the train will run according to the running action at the next moment;

[0086] The interactive feedback module 30 is used to determine the next moment as the current moment and execute the action of the acquisition module.

[0087] It should be noted that in this embodiment, every time the train performs a running action, the environment will immediately provide a status information and a reward value to guide the subsequent operation sequence, so as to update and optimize the train operation strategy, and finally obtain a converged and ideal train operation strategy to achieve the purpose of saving energy.

[0088] In some embodiments, the train energy consumption model adopts the DDPG model.

[0089] It should be noted that the DDPG (Deep Deterministic Policy Gradient) algorithm is a policy learning method that integrates deep learning neural networks into DPG. The DDPG algorithm is an offline, environment-free model algorithm that uses deep neural networks to approximate value functions. DDPG can be used to solve continuous action space problems.

[0090] It should be noted that this embodiment integrates the reinforcement learning DDPG algorithm into the urban rail train model, and trains the train's operation strategy through the algorithm to achieve the effect of energy saving optimization.

[0091] In some embodiments, the reward function is expressed as:

[0092] R β (s)=ωR(s)+(1-ω)J(s)

[0093] Wherein, J(s) is the reward function in the DDPG model, R(s) is the reward function based on the combination of the work done by the traction force and the punctuality of the train during operation, ω is the proportion, and the value is 0-1. R(s) = α1(∫Fvdt) + α2(∑t-T0), α1 + α2 = 1, ∫Fvdt is the work done by the traction force, F is the traction force, v is the train speed, ∑t-T0 is the train travel time deviation, t is the actual travel time, and T0 is the time in the timetable of the train operating section.

[0094] It should be noted that, based on the train's section timetable and the train's energy consumption formula, the train's operation can be monitored in real time to ensure it meets the timetable's punctuality requirements and to track energy consumption. The energy consumption during train operation can be expressed as the work done by the train's traction force. This embodiment aims to reduce energy consumption by modifying the operating strategy as much as possible without affecting the train's punctuality. According to the train energy consumption model, the work done by the traction force during operation is substituted into the reward function of the DDPG algorithm. This, together with the original DDPG algorithm's reward function, forms a new reward function. The algorithm continuously improves along the path of energy conservation, ultimately obtaining a convergent and ideal train operating strategy.

[0095] Compared to Q-Learning and DQN algorithms frequently used in reinforcement learning, the learning process of the Q-network is highly unstable because its parameters are frequently updated with gradients while simultaneously calculating the gradients of both the Q-network and the policy network. The DDPG algorithm used in this embodiment creates two copies of the neural network for both the policy network and the Q-network, ensuring the stability and efficiency of the algorithm's learning, making convergence easier, and is particularly suitable for solving driving optimization problems with large action spaces and continuous computation.

[0096] In some embodiments, the decision module 20 further includes:

[0097] The noise introduction unit is used to introduce random noise, convert the driving strategy into a random process, and randomly sample to obtain the operating action.

[0098] It should be noted that, when selecting train operation strategies, random noise is introduced into the action decision-making mechanism, transforming the action decision-making process from a deterministic process into a stochastic process. The operational action 'a' is then sampled from this stochastic process. t By transforming the previously deterministic, fixed selection of actions into random choices, all possible actions can be fully explored.

[0099] In some embodiments, a train safety system is added for receiving the driving strategy output by the decision module. The safety system is used for:

[0100] Determine whether the operation is dangerous based on the speed limit map of the driving section;

[0101] If so, an action request instruction is sent to reselect the running action;

[0102] If not, then perform the described operation.

[0103] Specifically, in this embodiment, when receiving the train operation strategy, it first utilizes the judgment mechanism of the train safety system to plot a speed-time graph of the trained train operation sequence. This graph is then compared with the speed limit-distance graph of the operating section to filter out non-compliant results, avoiding dangerous driving and improving training effectiveness. Simultaneously, a larger penalty value is set for train operation sequences that do not meet safety standards. It should be noted that when the train safety system receives the operational actions output by the model, it evaluates the safety of the strategy based on the speed limit graph of the section. Operation strategies exceeding the speed limit are rejected and penalized. Simultaneously, an action request command is sent to the decision-making system, requiring it to reselect an operational action to ensure train operation safety.

[0104] In some embodiments, the environment performs action a t Then, the reward value and the new environment running status are returned. t+1 The online policy network in the actor network converts it into state transition data (s). t ,a t ,r t ,s t+1 ), stored in the replay memory buffer, as the dataset for training the online network, where s t ,a t ,rt ,s t+1 These are the segment status, the action performed, the reward value, and the status of the next segment.

[0105] In some embodiments, the decision module 20 further includes:

[0106] A priority marking unit is used to mark the priority of state transition data, wherein the priority is marked from high to low according to the size of the reward value;

[0107] A storage unit is used to store the state transition data after priority marking into a replay memory buffer;

[0108] An extraction unit is configured to extract data from the replay memory buffer in order of the priority, for training the critic network and the actor network.

[0109] It should be noted that in this embodiment, the state transition data is prioritized and stored in the replay memory buffer. Operations that endanger driving safety are given the lowest priority, while samples that arrive on time but whose energy consumption is not reduced to the lowest priority are given the highest priority, forming a sum-tree structure with decreasing priority from top to bottom.

[0110] Data is extracted from the replay memory buffer in priority order to serve as small training sets for the Target Policy Network, Online Policy Network, TargetQ Network, and Online Q Network. A single training dataset is represented as (s t ,a t ,r t ,s t+1 ), s t ,a t ,r t ,s t+1 These represent the train's state at time t, the action taken, the reward function value obtained, and the state of the next segment, respectively.

[0111] It should be noted that the learned experience is saved in the replay memory buffer (i.e., the experience pool) and its priority is marked. Instead of randomly selecting samples from the experience pool, training is performed based on priority. Operations that endanger driving safety are given the lowest priority, while samples that arrive on time but do not minimize energy consumption are given the highest priority. High-priority samples are selected for training first. By following the priority principle in selecting samples from the experience pool, interference from invalid samples is avoided, and learning from potential samples is enhanced, thus improving the system's learning efficiency and convergence speed.

[0112] Furthermore, a third embodiment of the present invention discloses a dual most relevant recommendation device, the device including a memory and a processor; wherein the processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the method described in the above embodiments.

[0113] Furthermore, the fourth embodiment of the present invention discloses a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the method described in the above embodiments.

[0114] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0115] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0116] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0117] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0118] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for optimizing energy conservation in urban rail transit trains, characterized in that, The method for selecting driving strategies using the DDPG (Driving Energy Dissipation Model) includes the following steps: S10. Obtain the status information and reward value of the train operating environment at the current moment. The reward value is calculated using a reward function, which includes a first reward function in the DDPG model and a second reward function combining the work done by the traction force during train operation with the train's punctuality. The formula for the reward function is as follows: R β (s)=ωR(s)+(1-ω)J(s) Wherein, J(s) is the reward function in the DDPG model, R(s) is the reward function based on the combination of the work done by the traction force and the punctuality of the train during operation, ω is the proportion, and the value is 0-1. R(s) = α1(∫Fvdt) + α2(∑t-T0), α1 + α2 = 1, ∫Fvdt is the work done by the traction force, F is the traction force, v is the train speed, ∑t-T0 is the train travel time deviation, t is the actual travel time, and T0 is the time in the train's operating section timetable. S20. Based on the status information and reward value under the train operating environment, select an operating action and send it to the train so that the train will operate according to the operating action at the next moment. S30. Determine the next moment as the current moment and repeat steps S10 to S20.

2. The energy-saving optimization method for urban rail transit trains as described in claim 1, characterized in that, When selecting and issuing a running action to the train based on the status information and reward value under the train's operating environment, the method further includes: Random noise is introduced to convert the driving strategy into a random process, and the driving action is obtained by random sampling.

3. The energy-saving optimization method for urban rail transit trains as described in claim 1, characterized in that, When the train receives the operation action, it also includes: Determine whether the operation is dangerous based on the speed limit map of the driving section; If so, an action request instruction is sent to reselect the running action; If not, then perform the described operation.

4. The energy-saving optimization method for urban rail transit trains as described in claim 1, characterized in that, The DDPG model for train energy consumption includes a critic network and an actor network. The critic network includes an Online policy network and a Target policy network. The actor network includes an Online Q network and a Target Q network. When obtaining the state information and reward value of the train operating environment at the current moment, the model further includes: The critic network transforms the state information and the reward value into state transition data and marks the priority of the state transition data, wherein the priority is marked from high to low according to the size of the reward value; The state transition data after being marked with priority is stored in the replay memory buffer; Data is extracted from the replay memory buffer in the order of priority for training the critic network and the actor network.

5. The energy-saving optimization method for urban rail transit trains as described in claim 4, characterized in that, The method further includes: The parameters of the Online Q network and the Online policy network are updated using stochastic gradient descent. The parameters of the Target network and the Target policy network are updated using the soft update algorithm.

6. An energy-saving optimization device for urban rail transit trains, characterized in that, The device employs a DDPG (Driving Energy Dissipation-Driven Projection) model for driving strategy selection, and includes: The acquisition module is used to acquire the status information and reward value of the train operating environment at the current moment. The reward value is calculated using a reward function, which includes a first reward function in the DDPG model and a second reward function combining the work done by the traction force during train operation with the train's punctuality. The formula for the reward function is as follows: R β (s)=ωR(s)+(1-ω)J(s) Wherein, J(s) is the reward function in the DDPG model, R(s) is the reward function based on the combination of the work done by the traction force and the punctuality of the train during operation, ω is the proportion, and the value is 0-1. R(s) = α1(∫Fvdt) + α2(∑t-T0), α1 + α2 = 1, ∫Fvdt is the work done by the traction force, F is the traction force, v is the train speed, ∑t-T0 is the train travel time deviation, t is the actual travel time, and T0 is the time in the train's operating section timetable. The decision-making module is used to select and issue a running action to the train based on the status information and reward value of the train operating environment so that the train will run according to the running action at the next moment. The interactive feedback module is used to determine the next moment as the current moment and execute the actions of the acquisition module.

7. An energy-saving optimization device for urban rail transit trains, characterized in that, The device includes a memory and a processor; wherein the processor runs a program corresponding to the executable program code by reading executable program code stored in the memory, so as to implement the method as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-5.

Images