Freight train air-electric joint multi-target manipulation optimization method

Abstract

The invention provides a freight train air-power joint multi-target manipulation optimization method, and relates to the technical field of rail transit operation control and energy-saving optimization, and the method comprises the steps: setting training parameters, vehicle information and freight train line information, and initializing a reinforcement learning environment and variables; calculating a selection action by utilizing a strategy function; decomposing the action vector to obtain a new state; calculating the state value of the new state; updating a Q value related to the current state in the Q table; and for the condition that the current step number n exceeds the maximum round number N, judging whether the updated Q table is converged or not by using the total reward, if so, executing actions by the agent according to the Q table, calculating a new state, obtaining a final sequence working condition, and completing the air-power joint multi-target operation optimization of the freight train. The problem that in the prior art, air braking and electric braking are not well matched in the long downhill section circulating braking process of the heavy-duty train is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of rail transit operation control and energy-saving optimization, and in particular relates to an air-electric combined multi-objective operation optimization method for a freight train. Background Art

[0002] In the existing technology, the force characteristics of a single train during operation, the specific influence of air braking force and running resistance are studied. A mathematical model that can accurately describe the dynamic behavior of the train is constructed through longitudinal dynamics modeling. A fine equivalent model of the brake cylinder is constructed by simplifying fluid mechanics, and the complex nonlinear characteristics of the train operation process such as historical mitigation factors, altitude, and exhaust at the end of the train are considered. Combined with reinforcement learning theory, especially the Markov decision model (MPD) and Q learning algorithm, an air-electric joint operation control method specifically for freight trains on long and steep slopes is designed.

[0003] Disadvantages of existing technology:

[0004] 1) The traditional empirical model has insufficient description accuracy. The simplified processing of the empirical model leads to a large deviation from the actual braking performance in practical applications, especially in the case of long downhill sections where higher precision control is required, the model shows obvious limitations.

[0005] 2) Although the air brake model based on fluid dynamics can theoretically describe the braking process more accurately, it has high computational complexity and slow solution speed, making it difficult to apply to braking optimization or online control in real time.

[0006] 3) The air-electric brake matching strategy lacks optimization. Traditional braking strategies are mainly based on fixed logical rules or simple optimization algorithms, and fail to fully consider the synergistic effect between air brakes and electric brakes.

[0007] Existing research focuses on the optimization design of speed curves, mainly considering some safety factors, but it is relatively simple in actual application scenarios. In particular, there is a lack of targeted solutions for precise control of long and steep slopes and complex working conditions. Summary of the invention

[0008] In view of the above-mentioned deficiencies in the prior art, the present invention provides an air-electric combined multi-objective operation optimization method for freight trains, which solves the problem of poor matching between air braking and electric braking during the long downhill cyclic braking process of heavy-loaded trains in the prior art, and significantly improves the train operation performance.

[0009] In order to achieve the above objectives, the technical solution adopted by the present invention is: a freight train air-electric joint multi-objective operation optimization method, comprising the following steps:

[0010] S1, setting training parameters, vehicle information and freight train line information, and initializing reinforcement learning environment and variables;

[0011] S2, using the strategy function to calculate the selected action;

[0012] S3, based on the selected action, decompose the action vector to obtain the current step traction braking force and air brake state, and obtain a new state by calculating mechanical information;

[0013] S4. Calculate the state value of the new state;

[0014] S5. Update the Q value related to the current state in the Q table;

[0015] S6, determine whether the current number of steps n exceeds the maximum number of rounds N, if so, proceed to step S7, otherwise, return to step S3;

[0016] S7, using the total reward to determine whether the updated Q table converges, if so, proceed to step S9, otherwise, proceed to step S8;

[0017] S8, determine whether the current iteration number reaches the maximum iteration number, if so, proceed to step S9, otherwise, reset the train variable and return to step S3;

[0018] S9. The agent executes actions according to the Q table, calculates the new state, obtains the final sequence operating conditions, and completes the multi-objective operation optimization of the air-electric combined freight train.

[0019] The beneficial effects of the present invention are as follows: the present invention utilizes the Q-learning algorithm in reinforcement learning to optimize the matching strategy of air braking timing and electric braking force. Compared with the traditional method, the optimized strategy more accurately balances the effects of air braking and electric braking, and greatly reduces the energy loss in braking operation. Experimental results show that compared with the actual operation of the driver, the present invention can increase the average running speed of the train by about 10%, while reducing the cumulative change of control force by about 70%, significantly improving the speed following performance and operation stability. In addition, the optimization strategy of the present invention is simple to operate in actual operation, can adapt to different braking performance conditions, and significantly reduce the impact of braking shock on train stability, thereby improving the safety and reliability of train operation. Overall, the present invention has demonstrated significant technical effects in terms of performance improvement, energy consumption reduction and operation stability.

[0020] Furthermore, the step S2 comprises the following steps:

[0021] S201, randomly generate a random number p∈[0,1], and calculate the current agent according to the strategy function π(A s |S d );

[0022] S202, compare the random number p and the agent according to the strategy function π (A s |S d ), if the random number Then select action A with the largest state value in the current Q table d * ,like Then randomly select a current state S d Actions that can be performed.

[0023] The beneficial effect of the above further scheme is that through the random selection of the above steps, the mechanism has the exploration of different operations under the same working conditions, which ensures the extensiveness of the learning strategy.

[0024] Furthermore, the current agent follows the strategy function π(A s |S d ) is as follows:

[0025]

[0026] Among them, A d |S d Indicates that in S d Make A in the state d Action, A d * represents the optimal action for the current conditions, rand(A(S d )) means randomly selecting S d An action that the state agent can perform, δ represents the exploration rate decay rate, ep represents the number of learning iterations, and ep T represents the attenuation period, and ε0 represents the initial value of the random exploration rate.

[0027] The beneficial effect of the above further scheme is that by introducing the comprehensive design of random selection and strategy function π, the algorithm's exploration ability can be increased while ensuring the optimal action execution, avoiding falling into the local optimal solution. In addition, by introducing the attenuation mechanism, the scheme effectively balances the stability of strategy execution and learning efficiency, further improving the reliability and adaptability of the system.

[0028] Furthermore, step S3 includes the following steps:

[0029] S301, based on the selected action, decompose the action vector to obtain the current step traction braking force and air brake state;

[0030] S302, obtaining mechanical information by calculating the dynamics of the freight train according to the current step traction braking force and the air brake state;

[0031] S303, according to the mechanical information, by calculating the relevant state quantity of the freight train, a new state S is obtained. d+1 .

[0032] The beneficial effect of the above further scheme is that by decomposing the operation vector, the traction force and air braking force states can be accurately obtained, thereby providing more accurate data support for the dynamic modeling of the train and enhancing the system's perception of the train's operating status.

[0033] Furthermore, the expression for decomposing the motion vector is as follows:

[0034] A d =[F d ,Ab d ]

[0035] Among them, A d represents the d-th step action, F d represents the traction braking force in step d, Ab d Indicates the air brake status in step d.

[0036] The beneficial effect of the above further solution is that it can accurately distinguish between traction and air braking force, ensure the detailed expression of the mechanical state, and provide more reliable basic data for dynamic modeling.

[0037] Furthermore, the step S302 includes the following steps:

[0038] S3021, constructing a multi-parameter brake cylinder detailed model based on exponential function fitting and correction function;

[0039] S3022, calculating and obtaining corresponding air braking force according to the multi-parameter brake cylinder detailed model;

[0040] S3023. Based on the train kinematics equation, the basic running resistance of the freight train and the additional resistance of the whole vehicle are calculated, and the mechanical information is obtained in combination with the air braking force. The expression of the train kinematics equation is as follows:

[0041]

[0042] F b =∑F b,i (t B ,t R ,v)

[0043] Where v is the train speed, t is the system time, M is the vehicle weight, and F t Indicates the train traction, F eb represents the air braking force of the whole vehicle, W0 represents the basic running resistance of the whole vehicle, W j Indicates the additional resistance of the vehicle, Fb represents the air braking force of the rigid multi-mass model vehicle, F b,i represents the air braking force of the i-th carriage, t B Indicates the braking time of air brake, t R represents the release time of air brake, and x represents the displacement of the train.

[0044] The beneficial effect of the above further scheme is: the present invention can quickly and accurately solve the brake cylinder pressure and air braking force by establishing a multi-parameter brake cylinder equivalent model that takes into account the historical release time and altitude changes, thereby more accurately describing the changes in air brake performance during cyclic braking, and effectively overcoming the shortcomings of the traditional two-stage empirical model in terms of accuracy.

[0045] Furthermore, the expression of the multi-parameter brake cylinder detailed model is as follows:

[0046]

[0047] Among them, P i (t) represents the function of the pressure of the brake cylinder changing with time, P m,i Indicates the initial pressure of the brake cylinder, P s,i represents the steady-state pressure of the brake cylinder, β i represents the pressure increase characteristic function of the brake cylinder, t represents the system time, t delay,i Indicates the time when the brake cylinder pressure starts to increase;

[0048] The expression of the air brake force is as follows:

[0049]

[0050] Among them, F b,i (t) represents the function of air brake pressure changing with time, S h represents the piston area, α b represents the mechanical structure transmission ratio, eff represents the braking efficiency, Indicates the friction coefficient of the brake pad;

[0051] The expressions of the basic resistance of the freight train and the additional resistance of the whole vehicle are as follows:

[0052] W0=∑m i ·g·w0·10 -3

[0053] w0=a+bv+cv 2

[0054] W j =∑m i ·g·(w g,i +w r,i+w s,i )·10 -3

[0055] Among them, m i represents the mass of the i-th carriage, g represents the acceleration of gravity, w0 represents the basic resistance of the carriage unit, a, b, c represent the empirical coefficients, v represents the train speed, w g,i represents the slope resistance of the i-th carriage, w r,i represents the resistance of the i-th carriage over the curve, w s,i represents the air resistance of the i-th carriage.

[0056] The beneficial effect of the above further scheme is that through resistance modeling and mechanical parameter calculation, the accuracy of train dynamics analysis is enhanced, providing important data support for energy-saving operation and control optimization of trains.

[0057] Furthermore, the new state S d+1 The expression is as follows:

[0058]

[0059] Among them, S d+1 represents the new state of the system at step d+1, Indicates the friction coefficient of the brake pad, S d represents the train state at step d, g represents the acceleration due to gravity, A d Indicates the operation performed by the train, acc d () represents the calculation function of train acceleration, F t Indicates the train traction, V d represents the train speed at step d, W0 represents the basic running resistance of the whole vehicle, F eb Indicates the vehicle air braking force, W j Indicates the additional resistance of the vehicle, X d represents the current train position, γ represents the train rotation mass coefficient, V d+1 represents the speed of the train at the d+1th step, ΔX represents the discrete step length, Δt represents the time consumed to travel the distance ΔX, and pos d Indicates the joystick position of step d, pos d-1 Indicates the joystick level of step d-1, X d+1 represents the position of the train at step d+1, t rmax Indicates the maximum release time of air brake, t d represents the system time of step d, t d+1 Indicates the system time of step d+1, Ab d Indicates the air brake state, t bmax Indicates the maximum braking time of air brake.

[0060] The beneficial effects of the above further scheme are: by introducing the system state equation and the dynamic calculation of speed and position, the train operation status information can be updated in real time to ensure the accuracy and continuity of the data. And through the judgment and adjustment of air braking and relief time, as well as the comprehensive consideration of the system friction coefficient and additional resistance, the scheme can adapt to complex operating environments and improve the stability and safety of train operation.

[0061] Furthermore, the expression of the state value in step S4 is as follows:

[0062]

[0063] Among them, U d Indicates the current state S d The state value, that is, from the current state S d The expected total reward that can be obtained from starting, R d Indicates the current state S d The reward obtained immediately after executing the action, γ represents the discount factor, Q(S d+1 ,A d+1 ) indicates state S d+1 Execute action A d+1 The expected total return when .

[0064] The beneficial effect of the above further scheme is that through the introduction of the state value function, the expected return brought by performing actions under the current state can be quantified. The scheme takes into account both immediate returns and long-term benefits, providing an accurate evaluation basis for train operation decisions.

[0065] Furthermore, step S5 includes the following steps:

[0066] S501, for the continuous state S d Decompose into surrounding discrete states S0, S1, ..., S n , calculate the time difference error TD of all adjacent discrete states:

[0067] δ i =U d -Q(S i ,A d )

[0068] Among them, δ i Indicates the timing differential error TD, U d represents the state value of step d, A d represents the d-th step action, S i Indicates S d Discrete state after decomposition;

[0069] S502, based on the timing differential errors TD of all adjacent discrete states, update all discrete state Q tables using the following formula:

[0070] Q π (s i ,a D )←Q π (s i ,a D )+α·ω i ·δ i

[0071] Among them, Q π (s i ,a D ) represents the state-action value, a D represents the action, α represents the regenerative braking efficiency, ω i Indicates that S i The corresponding weight coefficient.

[0072] The beneficial effect of the above further scheme is that by calculating the TD (temporal difference) error, the system can quickly evaluate the difference between the current state and the target state, provide feedback to accelerate the convergence process of the strategy, and balance the update speed of new and old knowledge, thereby improving the learning efficiency and robustness of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0074] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0075] Example

[0076] Before describing the complete technology of the present invention, the theoretical results used in the following text are summarized and described.

[0077] (1) The detailed model of the multi-parameter brake cylinder based on exponential function fitting and correction function is as follows:

[0078]

[0079] Among them, P i (t) represents the function of the pressure of the brake cylinder changing with time, P m,i Indicates the initial pressure of the brake cylinder, Ps,i represents the steady-state pressure of the brake cylinder, β i represents the pressure increase characteristic function of the brake cylinder, t represents the system time, t delay,i Indicates the time when the brake cylinder pressure starts to increase.

[0080] The corresponding air brake force is calculated as follows:

[0081]

[0082] Among them, F b,i (t) represents the function of air brake pressure changing with time, P i (t) represents the function of the pressure of the brake cylinder changing with time, S h represents the piston area, α b represents the mechanical structure transmission ratio, eff represents the braking efficiency, Indicates the friction coefficient of the brake pad.

[0083] (2) Train kinematic equation:

[0084]

[0085] Where v is the train speed, t is the system time, and x is the train displacement. M is the vehicle weight, and F t Indicates the train traction, F eb represents the air braking force of the whole vehicle, W0 represents the basic running resistance of the whole vehicle, W j Indicates the additional resistance of the whole vehicle.

[0086] Basic resistance of vehicle operation:

[0087] W0=∑m i ·g·w0·10 -3 (4)

[0088] Among them, m i represents the mass of the i-th carriage, g represents the acceleration of gravity, and w0 represents the unit basic resistance of the carriage.

[0089] Basic resistance per unit weight:

[0090] w0=a+bv+cv 2 (5)

[0091] Among them, a, b, c represent empirical coefficients, which are selected according to the vehicle model, and v represents the train running speed.

[0092] Additional resistance of the vehicle:

[0093] W j =∑m i ·g·(w g,i +wr,i +w s,i )·10 -3 (6)

[0094] Among them, m i represents the mass of the i-th carriage, g represents the acceleration due to gravity, and w g,i represents the slope resistance of the i-th carriage, w r,i represents the resistance of the i-th carriage over the curve, w s,i represents the air resistance of the i-th carriage.

[0095] Calculation method of air braking force of rigid multi-particle model:

[0096] F b =∑F b,i (t B ,t R ,v) (7)

[0097] Among them, F b represents the air braking force of the rigid multi-mass model vehicle, F b,i represents the air braking force of the i-th carriage, t B Indicates the braking time of air brake, t R It indicates the release time of air brake, and v indicates the running speed of the train.

[0098] Based on the above theoretical analysis, in order to achieve the invention purpose of the present invention, the technical solution adopted is as follows:

[0099] like Figure 1 As shown, the present invention provides a freight train air-electric joint multi-objective operation optimization method, and its implementation method is as follows:

[0100] S1, setting training parameters, vehicle information and freight train line information, and initializing reinforcement learning environment and variables;

[0101] In this embodiment, basic program training parameters (maximum number of iterations M, maximum number of steps in a round N, etc.), vehicle information (train weight, train traction / braking characteristics, etc.), and line information (speed limit, slope, curve, etc.) are set.

[0102] In this embodiment, variables such as the reinforcement learning environment (system time t, round number n, etc.), the Q table, etc. are initialized.

[0103] S2. Use the strategy function to calculate the selected action. The implementation method is as follows:

[0104] S201, randomly generate a random number p∈[0,1], and calculate the current agent according to the strategy function π(A s |S d );

[0105] S202, compare the random number p and the agent according to the strategy function π (A s |S d ), if the random number , then select the action A with the largest state value in the current Q table d * ,like , then randomly select a current state S d Actions that can be performed.

[0106] In this embodiment, the strategy function of formula (17) is used to calculate the selected action A d .

[0107] S3. Based on the selected action, the action vector is decomposed to obtain the current step traction braking force and air brake state, and the new state is obtained by calculating the mechanical information. The implementation method is as follows:

[0108] S301, based on the selected action, decompose the action vector to obtain the current step traction braking force and air brake state;

[0109] In this embodiment, the action vector is decomposed according to formula (15) to obtain the current step traction braking force and air brake state.

[0110] S302, according to the current step traction braking force and the air brake state, the mechanical information is obtained by calculating the dynamics of the freight train, and the implementation method is as follows:

[0111] S3021, constructing a multi-parameter brake cylinder detailed model based on exponential function fitting and correction function;

[0112] S3022, calculating and obtaining corresponding air braking force according to the multi-parameter brake cylinder detailed model;

[0113] S3023. Based on the train kinematics equation, the basic running resistance of the freight train and the additional resistance of the whole vehicle are calculated, and mechanical information is obtained in combination with the air braking force;

[0114] In this embodiment, the train dynamics are calculated according to equations (1) to (7) to obtain mechanical information.

[0115] S303, obtaining a new state by calculating relevant state quantities of the freight train according to the mechanical information;

[0116] In this embodiment, the train dynamics is calculated according to equations (1) to (7) to obtain the mechanical information S d+1 .

[0117] S4. Calculate the state value of the new state;

[0118] In this embodiment, the new state S is calculated according to formula (18): d The status value.

[0119] S5. Update the Q value related to the current state in the Q table. The implementation method is as follows:

[0120] S501, for the continuous state S d Decompose into surrounding discrete states S0, S1, ..., S n , calculate the time difference error TD of all adjacent discrete states: δ i =U d -Q(S i ,A d );

[0121] Among them, δ i Indicates the timing differential error TD, U d represents the state value of step d, A d represents the d-th step action, S i Indicates S d Discrete state after decomposition;

[0122] S502, based on the timing differential error TD of all adjacent discrete states, update all discrete state Q tables using the following formula: Q π (s i ,a D )←Q π (s i ,a D )+α·ω i ·δ i ;

[0123] Among them, Q π (s i ,a D ) represents the state-action value, a D represents the action, α represents the regenerative braking efficiency, ω i Indicates that S i The corresponding weight coefficient.

[0124] S6, determine whether the current number of steps n exceeds the maximum number of rounds N, if so, proceed to step S7, otherwise, return to step S3;

[0125] S7, using the total reward to determine whether the updated Q table converges, if so, proceed to step S9, otherwise, proceed to step S8;

[0126] S8, determine whether the current iteration number reaches the maximum iteration number, if so, proceed to step S9, otherwise, reset the train variable and return to step S3;

[0127] In this embodiment, resetting train variables includes location, environment, etc.

[0128] S9. The agent executes actions according to the Q table, calculates the new state, obtains the final sequence operating conditions, and completes the multi-objective operation optimization of the air-electric combined freight train.

[0129] In this embodiment, the agent decision is guided by estimating the value function. The reinforcement learning goal is to maximize the reward during the interaction between the agent and the environment to determine the optimal action and obtain a better air-electric joint control method.

[0130] Since the above computational model is difficult to linearize and transform into a standard planning algorithm, reinforcement learning, as a widely used model-free optimization method, can achieve multi-objective optimization under complex models through the interaction and learning between the agent and the environment. The target reward function of reinforcement learning is the learning guidance basis of the agent in the present invention. It gives different rewards according to the different operations made by the agent in different states. The reward function R of the present invention is d The specific form is:

[0131] R d (S d ,A d )=w s R s +w v R v +w f (R f +R b )+w e R e (8)

[0132] Among them, S d Indicates the current state of the train, A d represents the operation performed by the train, w s Represents the safety index weight coefficient, R s represents the safety index, w v Represents the speed index weight coefficient, R v Indicates the speed index, w f Represents the weight coefficient of the comprehensive index of control force and air brake, R f Represents the train control index, R b Indicates the air brake index.

[0133] Safety indicators:

[0134]

[0135] Among them, r v represents the safety index penalty, V d+1 represents the next state speed, v lim (X d+1 ) represents the position of the next state Xd+1 Speed ​​limit on the route, Indicates the accumulated braking release time of the train in the next state.

[0136] Energy saving index R e :

[0137]

[0138] Among them, X d Indicates the current state of the train position, X d+1 Indicates the position of the train in the next state, F d Indicates the current train speed, η t ,η b represents the traction and braking energy transmission efficiency, and α represents the regenerative braking efficiency.

[0139] Speed ​​Index R v :

[0140]

[0141] Among them, V target represents the target speed, ΔV represents the discrete value of speed reward, v lim (X d+1 ) indicates the maximum speed limit during the braking phase.

[0142] Train control index R f :

[0143]

[0144] Among them, λ1,λ2 represent the control power reward coefficient, pos d Indicates the joystick position of step d, pos d-1 Indicates the joystick level of step d-1, v lim (X d+1 ) indicates X d+1 The maximum speed allowed.

[0145] Air brake index R b :

[0146] R b =-500 t d <0&Ab d =1 (13)

[0147] Among them, t d <0&Ab d =1 means air brake is used or the release time is less than 0.

[0148] The reinforcement learning Markov decision process problem is constructed as follows:

[0149] State space: refers to the part of the train operation information observed by the agent, which represents the current train operation status. The present invention defines the state space as follows:

[0150] s d =[x d ,v d ,t d ,pos d-1 ] (14)

[0151] Among them, s d represents the system state at step d, x d represents the d-th step position, v d Indicates the speed of the dth step, pos d-1 Indicates the traction force or electric braking force gear in the d-1 step decision stage.

[0152] Action space: a set of different actions that an agent performs according to different environments. The present invention defines the following discrete state traction braking force and air brake sequence set:

[0153] A d =[F d ,Ab d ] (15)

[0154] Among them, A d represents the d-th step action, F d represents the traction braking force in step d, Ab d Indicates the air brake status in step d.

[0155] Environment: In the heavy-load train MDP model designed by the present invention, the simulation solution model of the train object is a rigid multi-particle model. After converting it into a discrete process, the state transfer process of the heavy-load train MDP model of the present invention is as follows:

[0156]

[0157] Among them, S d+1 represents the new state of the system at step d+1, Indicates the friction coefficient of the brake pad, S d represents the train state at step d, g represents the acceleration due to gravity, A d Indicates the operation performed by the train, acc d () represents the calculation function of train acceleration, F t Indicates the train traction, V d represents the train speed at step d, W0 represents the basic running resistance of the whole vehicle, F eb Indicates the vehicle air braking force, W j Indicates the additional resistance of the vehicle, X drepresents the current train position, γ represents the train rotation mass coefficient, V d+1 represents the speed of the train at the d+1th step, ΔX represents the discrete step length, Δt represents the time consumed to travel the distance ΔX, and pos d Indicates the joystick position of step d, pos d-1 Indicates the joystick level of step d-1, X d+1 represents the position of the train at step d+1, t rmax Indicates the maximum release time of air brake, t d represents the system time of step d, t d+1 Indicates the system time of step d+1, Ab d Indicates the air brake state, t bmax Indicates the maximum braking time of air brake.

[0158] Strategy: The agent follows the policy function π(A s |S d ) Select the corresponding action, the strategy function designed by the present invention is as follows:

[0159]

[0160] Among them, A d |S d Indicates that in S d In this state, make A d Action, π(A s |S d ) indicates that the current agent follows the policy function, A d * represents the optimal action for the current conditions, rand(A(S d )) means randomly selecting S d An action that the state agent can perform, δ represents the exploration rate decay rate (δ∈[0,1]), ep represents the number of learning iterations, ep T represents the attenuation period, and ε0 represents the initial value of the random exploration rate (ε0∈[0,1]).

[0161] Q table (state-action expected return table): records the expected return Q value that can be obtained by selecting different actions under all states.

[0162] Current Status d The state value can be expressed as:

[0163]

[0164] Among them, U d Indicates the current state S d The state value indicates that the state S d The expected total reward that can be obtained from starting, Rd Indicates the current state S d The reward obtained immediately after executing the action, γ represents the discount factor, which indicates the degree of weight decay of future rewards, Q(S d+1 ,A d+1 ) indicates state S d+1 Execute action A d+1 The expected total return when .

[0165] There is a recursive relationship between the action value function, that is, the current action value function can be expressed by the action value function of the next stage. The Q learning of the present invention adopts the time difference update method and the recursive formula under the optimal strategy to update the Q table:

[0166] Q(S d ,A d )=Q(S d ,A d )+α[U d -Q(S d ,A d )] (19)

[0167] Among them, Q(S d ,A d ) indicates state S d Execute action A d The expected total return when , α represents the learning rate.

[0168] In the process of reinforcement learning Q-learning training, in order to determine whether the strategy has converged or whether the Q table has stabilized, this paper introduces an early termination criterion based on the change of total reward. After each training round, the total reward value of this round will be calculated. And the total reward value of the previous round Make a comparison.

[0169] When the Q learning algorithm continuously iterates and updates the Q table, if the state-action value distribution in the Q table gradually approaches the optimal strategy, the total reward value obtained by the agent under the same or similar operating conditions will tend to be stable. To this end, the present invention sets a convergence threshold ε tol (For example, ε tol It can be taken as 10 or adjusted according to actual operation experience). If the following conditions are met:

[0170]

[0171] It is considered that the performance of the strategy in this round of training has changed little from the previous training, and the Q value in the Q table has basically converged. At this point, the training process can be terminated without further unnecessary iterations.

[0172] In summary, the present invention proposes a multi-objective optimization method for air-electric joint control of freight trains on long downhill slopes using a reinforcement learning method and considering a fine equivalent model of the brake cylinder. Specifically, it includes:

[0173] 1. Optimization strategy design: Through the Q-learning algorithm in reinforcement learning, the matching strategy of air braking timing and electronic control power in the cyclic braking process is optimized. The improved Q-table iteration mechanism is introduced to solve the problem of continuous state changes and make the strategy more adaptable.

[0174] 2. Multi-parameter brake cylinder equivalent model: In order to solve the problem of inaccurate description of the traditional two-stage empirical model in the cyclic braking process, a brake cylinder equivalent model that can be solved quickly is designed. This model can more accurately describe the air braking force and brake cylinder pressure under different release time and altitude changes, making up for the shortcomings of the traditional model.

[0175] 3. Adaptability to complex operating conditions: Taking into account the characteristics of long and steep slopes, the model fully considers the operating disturbance, discreteness of air brakes, and accuracy requirements of cyclic brake control, thus improving the practicality of the model in complex scenarios.

Claims

1. A freight train air-electric joint multi-objective operation optimization method, characterized in that: The following steps are involved: S1, setting training parameters, vehicle information and freight train line information, and initializing reinforcement learning environment and variables; S2, using the strategy function to calculate the selected action; S3, based on the selected action, decompose the action vector to obtain the current step traction braking force and air brake state, and obtain a new state by calculating mechanical information; S4. Calculate the state value of the new state; S5. Update the Q value related to the current state in the Q table; S6, determine whether the current number of steps n exceeds the maximum number of rounds N, if so, proceed to step S7, otherwise, return to step S3; S7, using the total reward to determine whether the updated Q table converges, if so, proceed to step S9, otherwise, proceed to step S8; S8, determine whether the current iteration number reaches the maximum iteration number, if so, proceed to step S9, otherwise, reset the train variable and return to step S3; S9. The agent executes actions according to the Q table, calculates the new state, obtains the final sequence operating conditions, and completes the multi-objective operation optimization of the air-electric combined freight train.

2. The freight train air-electric joint multi-objective operation optimization method according to claim 1 is characterized in that: The step S2 comprises the following steps: S201, randomly generate a random number p∈[0,1], and calculate the current agent according to the strategy function π(A s |S d ); S202, compare the random number p and the agent according to the strategy function π (A s |S d ) size, if the random number p≤1-ε0δ ep / epT , then select the action A with the largest state value in the current Q table d * , if p>1-ε0δ ep / epT , then randomly select a current state S d Actions that can be performed.

3. The freight train air-electric joint multi-objective operation optimization method according to claim 2 is characterized in that: The current agent follows the policy function π(A s |S d ) is as follows: Among them, A d |S d Indicates that in S d Make A in the state d Action, A d * represents the optimal action for the current conditions, rand(A(S d )) means randomly selecting S d An action that the state agent can perform, δ represents the exploration rate decay rate, ep represents the number of learning iterations, and ep T represents the attenuation period, and ε0 represents the initial value of the random exploration rate.

4. The freight train air-electric joint multi-objective operation optimization method according to claim 1 is characterized in that: The step S3 comprises the following steps: S301, based on the selected action, decompose the action vector to obtain the current step traction braking force and air brake state; S302, obtaining mechanical information by calculating the dynamics of the freight train according to the current step traction braking force and the air brake state; S303, according to the mechanical information, by calculating the relevant state quantity of the freight train, a new state S is obtained. d+1 .

5. The freight train air-electric joint multi-objective operation optimization method according to claim 4 is characterized in that: The expression for decomposing the action vector is as follows: A d =[F d ,Ab d ] Among them, A d represents the d-th step action, F d represents the traction braking force in step d, Ab d Indicates the air brake status in step d.

6. The freight train air-electric joint multi-objective operation optimization method according to claim 4 is characterized in that: The step S302 includes the following steps: S3021, constructing a multi-parameter brake cylinder detailed model based on exponential function fitting and correction function; S3022, calculating and obtaining corresponding air braking force according to the multi-parameter brake cylinder detailed model; S3023. Based on the train kinematics equation, the basic running resistance of the freight train and the additional resistance of the whole vehicle are calculated, and the mechanical information is obtained in combination with the air braking force. The expression of the train kinematics equation is as follows: F b =∑F b,i (t B ,t R ,v) Where v is the train speed, t is the system time, M is the vehicle weight, and F t Indicates the train traction, F eb represents the air braking force of the whole vehicle, W0 represents the basic running resistance of the whole vehicle, W j Indicates the additional resistance of the vehicle, F b represents the air braking force of the rigid multi-mass model vehicle, F b,i represents the air braking force of the i-th carriage, t B Indicates the braking time of air brake, t R represents the release time of air brake, and x represents the displacement of the train.

7. The freight train air-electric joint multi-objective operation optimization method according to claim 6 is characterized in that: The expression of the multi-parameter brake cylinder detailed model is as follows: Among them, P i (t) represents the function of the pressure of the brake cylinder changing with time, P m,i Indicates the initial pressure of the brake cylinder, P s,i represents the steady-state pressure of the brake cylinder, β i represents the pressure increase characteristic function of the brake cylinder, t represents the system time, t delay,i Indicates the time when the brake cylinder pressure starts to increase; The expression of the air brake force is as follows: Among them, F b,i (t) represents the function of air brake pressure changing with time, S h represents the piston area, α b represents the mechanical structure transmission ratio, eff represents the braking efficiency, Indicates the friction coefficient of the brake pad; The expressions of the basic resistance of the freight train and the additional resistance of the whole vehicle are as follows: W0=∑m i ·g·w0·10 -3 w0=a+bv+cv 2 W j =∑m i ·g·(w g,i +w r,i +w s,i )·10 -3 Among them, m i represents the mass of the i-th carriage, g represents the acceleration of gravity, w0 represents the basic resistance of the carriage unit, a, b, c represent the empirical coefficients, v represents the train speed, w g,i represents the slope resistance of the i-th carriage, w r,i represents the resistance of the i-th carriage over the curve, w s,i represents the air resistance of the i-th carriage.

8. The freight train air-electric joint multi-objective operation optimization method according to claim 4 is characterized in that: The new state S d+1 The expression is as follows: Among them, S d+1 represents the new state of the system at step d+1, Indicates the friction coefficient of the brake pad, S d represents the train state at step d, g represents the acceleration due to gravity, A d Indicates the operation performed by the train, acc d () represents the calculation function of train acceleration, F t Indicates the train traction, V d represents the train speed at step d, W0 represents the basic running resistance of the whole vehicle, F eb Indicates the vehicle air braking force, W j Indicates the additional resistance of the vehicle, X d represents the current train position, γ represents the train rotation mass coefficient, V d+1 represents the speed of the train at the d+1th step, ΔX represents the discrete step length, Δt represents the time consumed to travel the distance ΔX, and pos d Indicates the joystick position of step d, pos d-1 Indicates the joystick level of step d-1, X d+1 represents the position of the train at step d+1, t rmax Indicates the maximum release time of air brake, t d represents the system time of step d, t d+1 Indicates the system time of step d+1, Ab d Indicates the air brake state, t bmax Indicates the maximum braking time of air brake.

9. The freight train air-electric joint multi-objective operation optimization method according to claim 1 is characterized in that: The expression of the state value in step S4 is as follows: Among them, U d Indicates the current state S d The state value, that is, from the current state S d The expected total reward that can be obtained from starting, R d Indicates the current state S d The reward obtained immediately after executing the action, γ represents the discount factor, Q(S d+1 ,A d+1 ) indicates state S d+1 Execute action A d+1 The expected total return when .

10. The freight train air-electric joint multi-objective operation optimization method according to claim 1 is characterized in that: The step S5 comprises the following steps: S501, for the continuous state S d Decompose into surrounding discrete states S0, S1, ..., S n , calculate the time difference error TD of all adjacent discrete states: δ i =U d -Q(S i ,A d ) Among them, δ i Indicates the timing differential error TD, U d represents the state value of step d, A d represents the d-th step action, S i Indicates S d Discrete state after decomposition; S502, based on the timing differential errors TD of all adjacent discrete states, update all discrete state Q tables using the following formula: Q π (s i ,a D )←Q π (s i ,a D )+a·ω i ·d i Among them, Q π (s i ,a D ) represents the state-action value, a D represents the action, α represents the regenerative braking efficiency, ω i Indicates that S i The corresponding weight coefficient.

Images