A safety reinforcement learning control method, device and medium for a maglev train suspension system

Abstract

The present invention discloses a method, device, and medium for safe reinforcement learning control of a maglev train suspension system, relating to the field of maglev control technology. The method comprises the following steps: S1, constructing a suspension system dynamics model as a training environment; S2, modeling the suspension system control problem as a constrained optimization problem; S3, converting the constrained optimization problem into an unconstrained optimization problem, and iteratively solving the objective function; S4, constructing a cost function to accelerate the solution of the unconstrained optimization problem, and designing a penalty term to guide the system state away from the safety boundary; S5, in a reinforcement learning environment, iteratively training and solving the unconstrained optimization problem in combination with the cost function. The present invention does not rely on a precise suspension system model, and does not require the design of a separate control strategy for each specific challenge. The optimal control strategy can be learned through interactive adaptive learning with the suspension system, exhibiting strong robustness and flexibility.

Description

Technical Field

[0001] The present invention relates to the field of magnetic levitation control technology, and in particular to a safety reinforcement learning control method, equipment and medium for a magnetic levitation train suspension system. Background Art

[0002] Maglev trains achieve contactless, high-speed operation through electromagnetic forces between the train and rails. They bridge the speed gap between high-speed rail (300 km / h) and air transport (800 km / h), and are crucial for rapidly connecting major urban agglomerations and promoting cross-regional integration and development. Due to its magnetic circuit characteristics, the suspension system is a highly nonlinear, open-loop unstable system, requiring a controller to implement active control. However, the suspension system operates in a complex environment, subject to external disturbances caused by track irregularities and impacts from aerodynamic loads. Furthermore, the suspension air gap is bounded, with a rated gap of only approximately 10 mm, and the control current is subject to unidirectional and saturation constraints. Furthermore, the induced electromotive force generated by the inductance of the suspension electromagnet coils hinders rapid changes in the control current, resulting in significant hysteresis in the adjustment of the electromagnetic force, further exacerbating control difficulties. Especially at speeds of 600 km / h and above, the suspension system faces even more stringent requirements. The performance of the suspension controller directly determines the safety and smoothness of train operation. Developing a highly adaptable and robust suspension controller is a critical requirement and a major challenge.

[0003] A Chinese invention patent application, publication number CN118790056A, discloses a deep reinforcement learning collaborative control method for a maglev train suspension system. This method first constructs a deep reinforcement learning environment for the unilateral suspension frame system based on a system dynamics model. A reinforcement learning algorithm and an initial reinforcement learning agent are then designed and integrated with the deep reinforcement learning environment to form a deep reinforcement learning collaborative control algorithm. The deep reinforcement learning environment and collaborative control algorithm are then used to train the reinforcement learning agent, ultimately yielding a trained reinforcement learning agent controller. This method effectively controls the unilateral suspension frame system under various disturbance conditions.

[0004] However, the reinforcement learning agent established by this method directly optimizes the control algorithm by maximizing cumulative rewards, without considering the stability requirements of the control system. This theoretically fails to guarantee the stability of the control system and cannot prevent the agent from making dangerous movements. Furthermore, this method fails to consider the bounded constraints between the vehicle and the track, which can lead to accidents such as "track-smashing" and fail to guarantee the safety of the suspension control system. Summary of the Invention

[0005] The present invention overcomes the deficiencies of the prior art and provides a safety reinforcement learning control method, device and medium for a maglev train suspension system.

[0006] To achieve the above-mentioned object, the present invention adopts a technical solution: a safety reinforcement learning control method for a maglev train suspension system, comprising the following steps:

[0007] S1. Construct a dynamic model of the suspension system that takes into account various external disturbances, time delays, and track irregularities as a training environment for the reinforcement learning algorithm.

[0008] S2. Model the control problem of the suspension system as a constrained optimization problem, define the Lyapunov candidate function and stability constraints;

[0009] S3, convert the constrained optimization problem in step S2 into an unconstrained optimization problem, and iteratively solve the objective function by gradient descent method;

[0010] S4. Construct a cost function based on the barrier function to speed up the solution of the unconstrained optimization problem in step S3. Combined with the physical boundary constraints of the suspension system, a penalty term is designed to guide the system state away from the safety boundary.

[0011] S5. In the reinforcement learning environment established in step S1, combined with the cost function in step S4, the unconstrained optimization problem in step S3 is iteratively trained and solved to obtain a reinforcement learning agent with a stability-guaranteed and safety-constrained suspension system.

[0012] In a preferred embodiment of the present invention, in step S1, the suspension system dynamics model includes the following sub-models: a magnetic circuit equation established based on Maxwell's equations and Biot-Savart's law, a system dynamics model including electromagnetic force equations, voltage equations and vertical dynamics equations, a time-varying disturbance model for describing track irregularity interference, and an actuator model considering chopper characteristics and time lag.

[0013] In a preferred embodiment of the present invention, the magnetic circuit equation can be obtained based on Maxwell's equations and Biot-Savart's law:

[0014] ;

[0015] in, is the number of coil turns; is the coil current; is the magnetic flux; is the magnetic resistance; It is the suspended air gap between the magnetic poles and the track; is the magnetic pole area; is the magnetic permeability of air;

[0016] The electromagnetic force equation is:

[0017] ;

[0018] in, It is a controllable electromagnetic force; is the total energy stored in the magnetic field; Yes Find partial derivatives; is the differential symbol;

[0019] Based on Kirchhoff's voltage law, the voltage equation can be obtained:

[0020] ;

[0021] in, is the electromagnet coil winding voltage; is the coil resistance; is the coil inductance;

[0022] Further vertical dynamic analysis of the suspension system can be performed to obtain the system equation;

[0023] ;

[0024] in, is the suspended mass; is the weight acting on a suspended object;

[0025] The impact of track irregularity on the system can be described as:

[0026] ;

[0027] in, is the distance between the electromagnet and the reference plane of the track; It is the suspended air gap; It’s the uneven track;

[0028] To develop the chopper equation, substitute the voltage equation:

[0029] ;

[0030] in, is the coil current; is a sign function; is the control current, given by the control algorithm; is the maximum rate of change of current, determined by the coil inductance and chopper performance;

[0031] Taking into account time delay and other disturbances, the system equation can be rewritten as:

[0032] ;

[0033] in, It is the interference force; is the actuator lag; It is a control strategy; is the state feedback lag.

[0034] In a preferred embodiment of the present invention, in step S2, the control problem of the suspension system is constructed as a constrained optimization problem in a constrained Markov decision process by defining the Lyapunov candidate function:

[0035] ;

[0036] ;

[0037] ;

[0038] in, It is to search; is the optimal strategy to be found; is maximized; It’s a reward; yes Rewards of the moment; Is constrained to; and They are the current state and the state at the next moment; and They are the current action and the action at the next moment; It is the experience replay pool; From the experience replay pool The reward for sampling Seek expectations; It’s the price; is a bounded positive definite cost function; is a positive definite Lyapunov-like function with parameters Neural Network describe; is a constant;

[0039] The actions made by the agent are Action Network Generate the mean of a Gaussian distribution and variance , reparameterization yields:

[0040] ;

[0041] in, is the action of the agent; is from a standard Gaussian distribution The noise variables collected in .

[0042] In a preferred embodiment of the present invention, in step S3, the objective function of the unconstrained optimization problem is:

[0043] ;

[0044] in, and is the Lagrange multiplier; is the observation value received by the agent in the environment; From the experience replay pool The reward for sampling Seek expectations; yes Rewards of the moment; It’s the price;

[0045] Considering that ensuring stability can satisfy the maximum reward, the above formula is simplified to:

[0046] ;

[0047] Update via gradient descent:

[0048] ;

[0049] in, Yes Find the gradient; Yes Find the gradient;

[0050] The Lagrange multipliers are updated by maximizing the following objective:

[0051] ;

[0052] in, From the experience replay pool expectations gained from

[0053] and is restricted to positive values:

[0054] ;

[0055] in, is the learning rate; Yes Find the gradient;

[0056] The loss function of the Lyapunov critic is defined as:

[0057] ;

[0058] in, is a Lyapunov candidate function, as Updated objective function;

[0059] Further, Select as a finite time window The price within and:

[0060] ;

[0061] in, is the cost function; yes The system status at a moment.

[0062] In a preferred embodiment of the present invention, in step S4, the design of the barrier function includes:

[0063] Define the security set for the system and measure system status Is it Functions within ,satisfy:

[0064] ;

[0065] ;

[0066] ;

[0067] in, It is a safety boundary; yes dimensional Euclidean space; is the set of real numbers; It is a measure of the system state;

[0068] Construct the barrier function, which satisfies that when the system approaches the safety boundary, the barrier function value tends to infinity, and the barrier function for:

[0069] ;

[0070] in, is the safety boundary of the suspension system, and the above formula can meet the requirements of the barrier function:

[0071] ;

[0072] ;

[0073] in, is for any system state;

[0074] The barrier function is combined with the error-sensitive term to form the comprehensive cost function:

[0075] Increased consideration , improve the cost value near the equilibrium point Designed to:

[0076] ;

[0077] in, is the base number for adjusting the sensitivity of errors; , , is an adjustable coefficient;

[0078] The total cost for:

[0079] ;

[0080] in, , , , , , , , All are adjustable coefficients; It is the suspended air gap; is the air gap velocity;

[0081] Pick , , , , , , , which is the constructed cost function.

[0082] In a preferred embodiment of the present invention, in step S5, the training parameters of the reinforcement learning agent include:

[0083] The action network and Lyapunov critic network adopt a multi-layer neural network structure;

[0084] Set the minimum batch size, experience replay pool capacity, and finite time window length;

[0085] Filter and save qualified agents based on the average cost threshold.

[0086] In a preferred embodiment of the present invention, the hidden layer structure of the action network and the Lyapunov critic network is two layers, each layer contains 256 neurons, and the activation function adopts a linear rectification function.

[0087] The present invention provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor; the memory stores a computer program executable by the processor, and the computer program is executed by the processor so that the processor can execute the safety reinforcement learning control method for a maglev train suspension system as described in any one of the above.

[0088] The present invention provides a computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a processor to implement the safety reinforcement learning control method for a maglev train suspension system as described in any one of the above-mentioned methods when executed.

[0089] The present invention solves the defects existing in the background technology and has the following beneficial effects:

[0090] The present invention provides a safety reinforcement learning control method, device and medium for the suspension system of a maglev train. Based on the Lyapunov function, a stability constrained optimization problem of the suspension system is constructed, which is converted into an unconstrained optimization objective through the Lagrange multiplier method. Combined with the control objective of the suspension system, the unconstrained optimization problem is simplified, reducing the amount of computation while ensuring stability constraints. It does not rely on an accurate suspension system model and does not require the design of a separate control strategy for each specific challenge. The optimal control strategy can be learned through interactive adaptive learning with the suspension system, and has strong robustness and flexibility.

[0091] In the present invention, the constraints of the action network are constructed based on Lyapunov stability theory. Compared with the existing suspension control method based on reinforcement learning, this method avoids the action network from making dangerous actions, and theoretically guarantees the progressive stability of the suspension system when facing complex disturbances.

[0092] In the present invention, a barrier function is introduced into the design of the cost function, so that the unconstrained optimization problem can be quickly moved away from the physical boundary of the suspension system during training and solving, which speeds up the solution of the unconstrained optimization problem while ensuring the safety of the suspension system, effectively limiting the state of the system to the feasible region. The designed cost function can provide a large gradient when the system is near the physical constraint boundary and the equilibrium point, guiding the suspension system to quickly stabilize.

[0093] In the present invention, a universal chopper model is given by establishing a suspension system model that takes into account multiple interferences and time delays, so that the training environment of the algorithm is closer to the real object, reducing the difficulty of migrating the trained reinforcement learning agent with stability guarantee and safety constraints from simulation to real vehicle application. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments described in the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive efforts.

[0095] Figure 1 This is a block diagram of a safety reinforcement learning control method for a maglev train suspension system according to a preferred embodiment of the present invention;

[0096] Figure 2 This is a model diagram of a single-point suspension system according to a preferred embodiment of the present invention;

[0097] Figure 3 This is the air gap response diagram of the control method of the present invention and the PID control method in the static floating with time lag in working condition 1;

[0098] Figure 4 This is a partial enlarged view of the air gap response diagram of the control method of the present invention and the PID control method in the static floating with time lag in working condition 1;

[0099] Figure 5 This is the electromagnet current diagram of the control method of the present invention and the PID control method in the static floating with time lag in working condition 1;

[0100] Figure 6 This is a partial enlarged diagram of the electromagnet current diagram of the control method of the present invention and the PID control method in the static floating with time lag in working condition 1;

[0101] Figure 7 1 is the air gap response diagram of the control method of the present invention and the PID control method when there is time lag and impact interference in working condition 2;

[0102] Figure 8 1 is the electromagnet current diagram of the control method of the present invention and the PID control method when there is time lag and impact interference in working condition 2;

[0103] Figure 9 1 is the air gap response diagram of the control method of the present invention and the PID control method when there is time lag and uneven interference in working condition 3;

[0104] Figure 10 This is the electromagnet current diagram of the control method of the present invention and the PID control method when there is time lag and uneven interference in working condition 3. DETAILED DESCRIPTION

[0105] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0106] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0107] Application Overview:

[0108] Numerous researchers have proposed various solutions to the challenges faced by suspension systems, including strong nonlinearity, response lag, track irregularity, and bounded constraints. These solutions include fuzzy control, sliding mode control, and model predictive control. However, existing approaches are often patchwork solutions designed for specific problems. These solutions rely on precise plant models and require tailored control strategies, which often lack sufficient adaptability.

[0109] Unlike traditional control methods, reinforcement learning, as a data-driven approach, directly learns optimal control strategies through interaction with the environment, eliminating the need for separate modeling and controller design for each specific problem. However, existing suspension control methods based on reinforcement learning directly learn control strategies from data and lack rigorous stability analysis. Direct application to suspension systems cannot guarantee control system stability.

[0110] In addition, existing suspension control strategies based on reinforcement learning do not consider the safety limitations of the suspension air gap, which may lead to accidents such as "track smashing" and cannot guarantee the safety of the suspension control system.

[0111] Therefore, this paper designs a secure reinforcement learning control method for a maglev train suspension system with stability guarantees and safety constraints. Based on Lyapunov theory, a stability constraint is constructed. Through rigorous mathematical analysis, the proposed method proves that it can ensure the stability of the suspension system. Furthermore, a cost function is constructed by considering physical constraints, and a barrier function is introduced to guide the suspension system away from physical boundaries, thus ensuring its safety.

[0112] like Figure 1 As shown, a safety reinforcement learning control method for a maglev train suspension system includes the following steps:

[0113] S1. Construct a dynamic model of the suspension system that takes into account various external disturbances, time delays, and track irregularities as a training environment for the reinforcement learning algorithm.

[0114] S2. Model the control problem of the suspension system as a constrained optimization problem, define the Lyapunov candidate function and stability constraints;

[0115] S3, convert the constrained optimization problem in step S2 into an unconstrained optimization problem, and iteratively solve the objective function by gradient descent method;

[0116] S4. Construct a cost function based on the barrier function to speed up the solution of the unconstrained optimization problem in step S3. Combined with the physical boundary constraints of the suspension system, a penalty term is designed to guide the system state away from the safety boundary.

[0117] S5. In the reinforcement learning environment established in step S1, combined with the cost function in step S4, the unconstrained optimization problem in step S3 is iteratively trained and solved to obtain a reinforcement learning agent with a stability-guaranteed and safety-constrained suspension system.

[0118] It should be noted that the present invention constructs the stability constrained optimization problem of the suspension system based on the Lyapunov function, converts it into an unconstrained optimization objective through the Lagrange multiplier method, and combines it with the control objective of the suspension system to simplify the unconstrained optimization problem, thereby reducing the amount of computation while ensuring stability constraints.

[0119] like Figure 2 As shown, in some specific embodiments, in step S1, the suspension system dynamics model includes the following sub-models: a magnetic circuit equation established based on Maxwell's equations and Biot-Savart's law, a system dynamics model including an electromagnetic force equation, a voltage equation, and a vertical dynamics equation, as well as a time-varying disturbance model for describing track irregularity interference and an actuator model considering chopper characteristics and time lag;

[0120] Based on Maxwell's equations and Biot-Savart's law, the magnetic circuit equation can be obtained:

[0121] ;

[0122] in, is the number of coil turns; is the coil current; is the magnetic flux; is the magnetic resistance; It is the suspended air gap between the magnetic poles and the track; is the magnetic pole area; is the magnetic permeability of air;

[0123] The electromagnetic force equation is:

[0124] ;

[0125] in, It is a controllable electromagnetic force; is the total energy stored in the magnetic field; Yes Find partial derivatives; is the differential symbol;

[0126] Based on Kirchhoff's voltage law, the voltage equation can be obtained:

[0127] ;

[0128] in, is the electromagnet coil winding voltage; is the coil resistance; is the coil inductance;

[0129] Further vertical dynamic analysis of the suspension system can be performed to obtain the system equation;

[0130] ;

[0131] in, is the suspended mass; is the weight acting on a suspended object;

[0132] The impact of track irregularities on the system can be described as:

[0133] ;

[0134] in, is the distance between the electromagnet and the reference plane of the track; It is the suspended air gap; It’s the uneven track;

[0135] Design the chopper equations, replacing the voltage equations:

[0136] ;

[0137] in, is the coil current; is a sign function; is the control current, given by the control algorithm; is the maximum rate of change of current, determined by the coil inductance and chopper performance;

[0138] Taking into account time delay and other disturbances, the system equation can be rewritten as:

[0139] ;

[0140] in, It is the interference force; is the actuator lag; It is a control strategy; is the state feedback lag.

[0141] It should be noted that by establishing a suspension system model that takes into account multiple interferences and time delays, a universal chopper model is given, which makes the algorithm's training environment closer to the real object and reduces the difficulty of migrating the trained reinforcement learning agent with stability guarantees and safety constraints from simulation to real vehicle application.

[0142] In some specific implementations, in step S2, the control problem of the suspension system is constructed as a constrained optimization problem in a constrained Markov decision process by defining a Lyapunov candidate function:

[0143] ;

[0144] ;

[0145] ;

[0146] in, It is to search; is the optimal strategy to be found; is maximized; It’s a reward; yes Rewards of the moment; Is constrained to; and They are the current state and the state at the next moment; and They are the current action and the action at the next moment; It is the experience replay pool; From the experience replay pool The reward for sampling Seek expectations; It’s the price; is a bounded positive definite cost function; is a positive definite Lyapunov-like function with parameters Neural Network describe; is a constant;

[0147] The actions made by the agent are Action Network Generate the mean of a Gaussian distribution and variance , reparameterization yields:

[0148] ;

[0149] in, is the action of the agent; is from a standard Gaussian distribution The noise variables collected in .

[0150] In some specific implementations, in step S3, the objective function of the unconstrained optimization problem is:

[0151] ;

[0152] in, and is the Lagrange multiplier; is the observation value received by the agent in the environment; From the experience replay pool The reward for sampling Seek expectations; yes Rewards of the moment; It’s the price;

[0153] Considering that ensuring stability can satisfy the maximum reward, the above formula is simplified to:

[0154] ;

[0155] Update via gradient descent:

[0156] ;

[0157] in, Yes Find the gradient; Yes Find the gradient;

[0158] The Lagrange multipliers are updated by maximizing the following objective:

[0159] ;

[0160] in, From the experience replay pool expectations gained from

[0161] and is restricted to positive values:

[0162] ;

[0163] in, is the learning rate; Yes Find the gradient;

[0164] The loss function of the Lyapunov critic is defined as:

[0165] ;

[0166] in, is a Lyapunov candidate function, as Updated objective function;

[0167] Further, Select as a finite time window The price within and:

[0168] ;

[0169] in, is the cost function; yes The system status at a moment.

[0170] In some specific implementations, in step S4, the design of the barrier function includes:

[0171] Define the security set for the system and measure system status Is it Functions within ,satisfy:

[0172] ;

[0173] ;

[0174] ;

[0175] in, It is a safety boundary; yes dimensional Euclidean space; is the set of real numbers; It is a measure of the system state;

[0176] Construct a barrier function that satisfies the requirement that when the system approaches the safety boundary, the barrier function value tends to infinity. for:

[0177] ;

[0178] in, is the safety boundary of the suspension system, and the above formula can meet the requirements of the barrier function:

[0179] ;

[0180] ;

[0181] in, is for any system state;

[0182] Combine the barrier function with the error-sensitive term to form a comprehensive cost function:

[0183] Increased consideration , improve the cost value near the equilibrium point Designed to:

[0184] ;

[0185] in, is the base number for adjusting the sensitivity of errors; , , is an adjustable coefficient;

[0186] The total cost for:

[0187] ;

[0188] in, , , , , , , , All are adjustable coefficients; It is the suspended air gap; is the air gap velocity;

[0189] Pick , , , , , , , which is the constructed cost function.

[0190] It should be noted that by constructing a safety barrier function for the suspension system, the unconstrained optimization problem can be quickly solved away from the physical boundaries of the suspension system, which speeds up the solution of the unconstrained optimization problem while ensuring the safety of the suspension system.

[0191] In some specific embodiments, in step S5, the training parameters of the reinforcement learning agent include:

[0192] The action network and Lyapunov critic network adopt a multi-layer neural network structure;

[0193] Set the minimum batch size, experience replay pool capacity, and finite time window length;

[0194] Filter and save qualified agents based on the average cost threshold;

[0195] Among them, the hidden layer structure of the action network and the Lyapunov critic network is two layers, each layer contains 256 neurons, and the activation function uses a linear rectification function.

[0196] Specifically, the maximum number of training steps in a single round is designed to be 1000, and an evaluation is saved every 5000 steps. The simulation step is 0.001 s, and the limited time window is 30. Both the action network and the Lyapunov critic network are designed with two hidden layers, each with 256 neurons, and a linear rectifier activation function. The learning rate of the action network is , the learning rate of the Lyapunov critic network is The minimum batch size is designed to be 256, and the maximum experience replay pool is , by saving the agents with an average cost less than 50, we can obtain a reinforcement learning agent with a stability-guaranteed and safety-constrained suspension system.

[0197] The present invention provides an electronic device, which includes: at least one processor, and a memory communicatively connected to the at least one processor; the memory stores a computer program executable by the processor, and the computer program is executed by the processor so that the processor can execute any of the above-mentioned safety reinforcement learning control methods for a maglev train suspension system.

[0198] The present invention provides a computer-readable storage medium storing computer instructions, which are used to enable a processor to implement any of the above-mentioned safety reinforcement learning control methods for a maglev train suspension system when executed.

[0199] To further illustrate the present invention, numerical simulation is used to compare the safety reinforcement learning control method of the present invention with the proportional-integral-differential (PID) control algorithm currently used in real vehicles under three different working conditions. Root mean square error (RMSE) and maximum error are used for quantitative analysis. The PID parameter is selected as the proportional gain. , integral gain , differential gain .

[0200] Specifically, the three working conditions are:

[0201] Condition 1: Static floating, with a state lag of 2 ms and an actuator lag of 3 ms, a maximum chopper adjustment time of 30 ms, and no interference.

[0202] Condition 2: There is a state lag of 2 ms and an actuator lag of 3 ms, with a maximum chopper adjustment time of 30 ms. When impact interference.

[0203] Condition 3: There is a 2 ms state lag and a 3 ms actuator lag, a maximum chopper adjustment time of 30 ms, and interference from track irregularities and random noise. Track irregularities are simulated using a sine wave, and the expression is:

[0204] ;

[0205] in, is the amplitude of long-wave irregularity; is the train speed; The span of the bridge.

[0206] In working condition 1, the suspension air gap response is as follows Figure 3 and Figure 4 As shown in the figure, in the first 0.3 s adjustment phase, the PID control shows obvious oscillation due to the time lag, which basically reaches stability after 0.44 s. The proposed method is not significantly affected by the time lag and is more stable than the PID control. Figure 5 and Figure 6 As shown in the figure, the electromagnet current under PID control fluctuates widely around 0.05 s, and oscillates gradually over the next 3 s. The proposed method is more stable, completely eliminating the jitter after about 0.23 s, and the electromagnet current stabilizes at 21.8 A.

[0207] In working condition 2, the suspension air gap response is as follows Figure 7 As shown, When the interference is small, both PID and the proposed method can resist the impact well. When the disturbance increases, the PID control deviates more from the equilibrium point than the proposed method, with a maximum deviation of 0.64 mm, while the proposed method is only 0.32 mm. Moreover, it takes a longer time to return to the equilibrium state under PID control after the disturbance disappears. When the interference reaches the maximum, the maximum error of the suspension air gap under PID control reaches 1.24 mm, while the maximum error of the proposed method is only 0.46 mm, which is 63.2% less than that of PID control. When the interference disappears, Figure 8 As shown in the electromagnet current, the PID takes longer to recover to the equilibrium state than the proposed method. In summary, in working condition 2, the proposed method has stronger robustness than the PID control.

[0208] In working condition 3, the influence of track unevenness on the suspension air gap is as follows: Figure 9As shown in the figure, the proposed method has smaller fluctuation of suspension air gap than PID control, and the RMSE and maximum error of the proposed method are reduced by 14.1% and 49.9% respectively compared with PID, and is less affected by uneven interference. Figure 10 It can be clearly seen from the electromagnet current shown that the fluctuation of the proposed method is much smaller than that of PID, and the response is more stable.

[0209] In summary, the safe reinforcement learning control method for the suspension system of a maglev train of the present invention does not rely on a precise suspension system model, and does not require the design of a separate control strategy for each specific challenge. It can adaptively learn the optimal control strategy through interaction with the suspension system, and has strong robustness and flexibility. Compared with the existing suspension control method based on reinforcement learning, the present invention constructs the constraint conditions of the action network based on Lyapunov stability theory, which prevents the action network from making dangerous actions, and theoretically guarantees the asymptotic stability of the suspension system in the face of complex disturbances. At the same time, the present invention introduces a barrier function in the design of the cost function, which effectively restricts the state of the system to a feasible region. The designed cost function can provide a larger gradient when the system is at the physical constraint boundary and near the equilibrium point, guiding the suspension system to quickly stabilize.

[0210] The above description is based on the ideal embodiment of the present invention. It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, the embodiments should be considered as illustrative and non-restrictive in all respects. The scope of the present invention is defined by the appended claims, not the foregoing description, and it is intended that all variations within the meaning and range of equivalents of the claims be encompassed within the present invention. Any reference signs in the claims should not be construed as limiting the claim to which they relate.

[0211] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A safety reinforcement learning control method for a maglev train suspension system, characterized in that: The following steps are involved: S1. Construct a dynamic model of the suspension system that takes into account various external disturbances, time delays, and track irregularities as a training environment for the reinforcement learning algorithm. S2. Model the control problem of the suspension system as a constrained optimization problem, define the Lyapunov candidate function and stability constraints; S3, convert the constrained optimization problem in step S2 into an unconstrained optimization problem, and iteratively solve the objective function by gradient descent method; S4. Construct a cost function based on the barrier function to speed up the solution of the unconstrained optimization problem in step S3. Combined with the physical boundary constraints of the suspension system, a penalty term is designed to guide the system state away from the safety boundary. S5. In the reinforcement learning environment established in step S1, combined with the cost function in step S4, iteratively train and solve the unconstrained optimization problem in step S3 to obtain a reinforcement learning agent with a stability-guaranteed and safety-constrained suspension system; In step S3, the objective function of the unconstrained optimization problem is: ； in, and is the Lagrange multiplier; is the observation value received by the agent in the environment; From the experience replay pool The reward for sampling Seek expectations; yes Rewards of the moment; It’s the price; Considering that ensuring stability can satisfy the maximum reward, the above formula is simplified to: ； Update via gradient descent: ； in, Yes Find the gradient; Yes Find the gradient; The Lagrange multipliers are updated by maximizing the following objective: ； in, From the experience replay pool expectations gained from and is restricted to positive values: ； in, is the learning rate; Yes Find the gradient; The loss function of the Lyapunov critic is defined as: ； in, is a Lyapunov candidate function, as Updated objective function; Further, Select as a finite time window The price within and: ； in, is the cost function; yes System status at the moment; In the step S4, the design of the barrier function includes: Define the security set for the system and measure system status Is it Functions within ,satisfy: ； ； ； in, It is a safety boundary; yes dimensional Euclidean space; is the set of real numbers; It is a measure of the system state; Construct the barrier function, which satisfies that when the system approaches the safety boundary, the barrier function value tends to infinity, and the barrier function for: ； in, is the safety boundary of the suspension system, and the above formula can meet the requirements of the barrier function: ； ； in, is for any system state; The barrier function is combined with the error-sensitive term to form the comprehensive cost function: Increased consideration , improve the cost value near the equilibrium point Designed to: ； in, is the base number for adjusting the sensitivity of errors; , , is an adjustable coefficient; The total cost for: ； in, , , , , , , , All are adjustable coefficients; It is the suspended air gap; is the air gap velocity; Pick , , , , , , , which is the constructed cost function.

2. The safety reinforcement learning control method for a maglev train suspension system according to claim 1, characterized in that: In step S1, the suspension system dynamics model includes the following sub-models: a magnetic circuit equation based on Maxwell's equations and Biot-Savart's law, a system dynamics model including electromagnetic force equations, voltage equations and vertical dynamics equations, a time-varying disturbance model for describing track irregularity interference, and an actuator model considering chopper characteristics and time lag.

3. The safety reinforcement learning control method for a maglev train suspension system according to claim 2, characterized in that: Based on Maxwell's equations and Biot-Savart's law, the magnetic circuit equation can be obtained: ； in, is the number of coil turns; is the coil current; is the magnetic flux; is the magnetic resistance; It is the suspended air gap between the magnetic poles and the track; is the magnetic pole area; is the magnetic permeability of air; The electromagnetic force equation is: ； in, It is a controllable electromagnetic force; is the total energy stored in the magnetic field; Yes Find partial derivatives; is the differential symbol; Based on Kirchhoff's voltage law, the voltage equation can be obtained: ； in, is the electromagnet coil winding voltage; is the coil resistance; is the coil inductance; Further vertical dynamic analysis of the suspension system can be performed to obtain the system equation; ； in, is the suspended mass; is the weight acting on a suspended object; The impact of track irregularity on the system can be described as: ； in, is the distance between the electromagnet and the reference plane of the track; It is the suspended air gap; It’s the uneven track; To develop the chopper equation, substitute the voltage equation: ； in, is the coil current; is a sign function; is the control current, given by the control algorithm; is the maximum rate of change of current, determined by the coil inductance and chopper performance; Taking into account time delay and other disturbances, the system equation can be rewritten as: ； in, It is the interference force; is the actuator lag; It is a control strategy; is the state feedback lag.

4. The safety reinforcement learning control method for a maglev train suspension system according to claim 1, characterized in that: In step S2, by defining the Lyapunov candidate function, the control problem of the suspension system is constructed as a constrained optimization problem in a constrained Markov decision process: ； ； ； in, It is to search; is the optimal strategy to be found; is maximized; It’s a reward; yes Rewards of the moment; Is constrained to; and They are the current state and the state at the next moment; and They are the current action and the action at the next moment; It is the experience replay pool; From the experience replay pool The reward for sampling Seek expectations; It’s the price; is a bounded positive definite cost function; is a positive definite Lyapunov-like function with parameters Neural Network describe; is a constant; The actions made by the agent are Action Network Generate the mean of a Gaussian distribution and variance , reparameterization yields: ； in, is the action of the agent; is from a standard Gaussian distribution The noise variables collected in .

5. The safety reinforcement learning control method for a maglev train suspension system according to claim 1, characterized in that: In step S5, the training parameters of the reinforcement learning agent include: The action network and Lyapunov critic network adopt a multi-layer neural network structure; Set the minimum batch size, experience replay pool capacity, and finite time window length; Filter and save qualified agents based on the average cost threshold.

6. The safety reinforcement learning control method for a maglev train suspension system according to claim 5, characterized in that: The hidden layer structure of the action network and the Lyapunov critic network is two layers, each layer contains 256 neurons, and the activation function adopts a linear rectification function.

7. An electronic device, characterized in that: The electronic device includes: at least one processor, and a memory communicatively connected to at least one of the processors; the memory stores a computer program executable by the processor, and the computer program is executed by the processor so that the processor can execute the safety reinforcement learning control method for the maglev train suspension system according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the safety reinforcement learning control method for a maglev train suspension system according to any one of claims 1 to 6 when executed.

Images