Double-pendulum crown block self-adaptive control method based on online reinforcement learning

Abstract

The invention discloses a double-pendulum crown block adaptive control method based on online reinforcement learning, and relates to the technical field of crown block motion control, and the method comprises the following steps: S1, employing an Euler-Lagrange method to establish a dynamic model of a crown block according to the specific parameters of the crown block; s2, converting the kinetic model into a linearized system model; s3, designing a self-adaptive control system framework based on RL; s4, solving the weight of the AN through the iteration of the RL; s5, the weight of the AN is output in real time through the updated AN to serve as the optimal gain parameter generated at the next moment, online reinforcement learning is directly applied to the physical double-pendulum crown block system, an accurate model and offline training are not needed, the complex coupling effect between a lifting hook and goods is effectively overcome, and the reliability of the physical double-pendulum crown block system is improved. And meanwhile, the problem of performance degradation of a traditional reinforcement learning method from simulation to practical application is avoided, so that the control system can keep stable and reliable performance under various working conditions, and the control precision is improved.

Description

Technical Field

[0001] This invention relates to the field of overhead crane motion control technology, specifically an adaptive control method for a double-pendulum overhead crane based on online reinforcement learning. Background Technology

[0002] Overhead cranes are critical transport equipment in manufacturing workshops, construction sites, and port terminals. They consist of a trolley, hook, rope, drive motor, and lifting device. Due to their underactuated characteristics, the movement of the trolley driven by the motor inevitably causes the cargo to sway, reducing operational efficiency and potentially leading to safety accidents.

[0003] Early control methods primarily employed open-loop control strategies such as input shaping, filtering, and command smoothing (e.g., patent CN201910781611.7). While simple in structure, these methods relied excessively on precise system models and lacked feedback regulation, resulting in limited performance. Subsequently, closed-loop control methods such as PID control and sliding mode control were widely adopted, with PID control becoming the mainstream due to its simple structure and convenient engineering deployment. For example, patent CN202210592475.9 proposed a PID controller for a vehicle's speed loop, used to track the desired trajectory and suppress cargo swaying. However, the parameters of traditional PID controllers are typically fixed values, making it difficult to adapt to dynamic operating conditions.

[0004] To improve adaptability, researchers have combined intelligent algorithms such as fuzzy logic, neural networks (NN), and reinforcement learning (RL) with PID control. For example, patent CN201711492808.6 uses NN as an online estimator to optimize controller parameters in real time. However, these methods still have significant limitations: 1) they do not fully consider the double pendulum coupling effect caused by the hook mass; 2) they rely on predefined rules or network parameters, resulting in insufficient flexibility under unknown operating conditions; 3) RL-based methods face the "Sim-to-Realgap" problem, making it difficult to directly transfer from simulation to the physical system.

[0005] Therefore, it is necessary to provide an adaptive control method for a double-pendulum overhead crane based on online reinforcement learning to solve the technical problems mentioned in the background. Summary of the Invention

[0006] To address the aforementioned problems, this invention provides an adaptive control method for a double-pendulum overhead crane based on online reinforcement learning, thereby solving the technical problems mentioned in the background section that require reliance on accurate models and offline training.

[0007] To achieve the above objectives, the present invention provides the following technical solution: an adaptive control method for a double-pendulum overhead crane based on online reinforcement learning, comprising the following steps:

[0008] S1. Based on the specific parameters of the overhead crane, the dynamic model of the overhead crane is established using the Euler-Lagrange method;

[0009] S2. Transform the dynamic model into a linearized system model;

[0010] S3. Design of an adaptive control system framework based on RL;

[0011] S4. Solve for the weights of AN through RL iteration;

[0012] S5. Use the updated AN to output the weights of AN in real time as the parameters to generate the optimal gain parameters at the next time step.

[0013] In one possible implementation, the dynamic model includes: driving force This causes the overhead crane to reciprocate on the beam frame. The lower end of the overhead crane is connected to the cargo via rope one, hook, rope two, and so on, forming a two-stage swing structure.

[0014] (1)

[0015] The masses of the overhead crane, hook, and cargo are respectively... , and The length of rope one is denoted as The length of rope two is denoted as , Let gravitational acceleration be the acceleration due to gravity. The horizontal displacement of the overhead crane, the swing angle of the hook, and the swing angle of the cargo are denoted as follows: , and The system's damping parameters are used , and It is stated that the driving force of the crane is .

[0016] In one possible implementation, the dynamic model shows relatively small system oscillations during normal transportation, which can be approximated using a small angle:

[0017] (2)

[0018] The resulting linearized system model is as follows:

[0019] (3)

[0020] The system is also subject to air damping. , and friction The impact.

[0021] In one possible implementation, step S3 includes:

[0022] S31. Construct the state variables of the double pendulum crane system:

[0023] (4)

[0024] (5)

[0025] in, This represents the horizontal displacement of the overhead crane. Let be the expected displacement of the overhead crane. For the hook swing angle, Given the cargo swing angle, the control objective in equation (5) can be integrated into an optimal adjustment objective;

[0026] S32. The control input for the overhead crane system is generated using an incremental PID controller:

[0027] (6)

[0028] in, , , and To control the gain, This represents the displacement tracking error of the overhead crane.

[0029] S33. Define the performance index function:

[0030] (7)

[0031] (8)

[0032] (9)

[0033] in, and These represent runtime and sampling time, respectively. , ;

[0034] The index shown in Equation (7) directly reflects the performance of the system. The smaller the index value, the better the system performance and the lower the energy consumption.

[0035] Furthermore, the control problem in equation (5) can be rearranged as follows:

[0036] (10)

[0037] In one possible implementation, step S4 includes:

[0038] S41. Construct two neural networks:

[0039] In equation (10), due to the presence of the perturbation term, it is difficult to solve directly. Therefore, an online iterative update method is used to handle this problem, and equation (7) can be written as:

[0040] (11)

[0041] The optimal index is defined as:

[0042] (12)

[0043] Based on equation (12), the optimal RL control strategy can be derived:

[0044] (13)

[0045] Based on the universal approximation principle, equation (13) can be rewritten in the form of NN:

[0046] (14)

[0047] in, For the ideal weight vector, It is from arrive Nonlinear activation function of the mapping It is a deviation factor;

[0048] The evaluation network (CN) is used to approximate the optimal performance index, and its output is:

[0049] (15)

[0050] in, for The estimated vector, selected , for and The second mapping;

[0051] Similarly, in order to solve the optimal control strategy in equation (13) Build another NN:

[0052] (16)

[0053] in, The set maximum gain parameter value. , For the ideal weight vector, It is from arrive The nonlinear activation function of the mapping It is a deviation factor;

[0054] Action Network (AN): Used to approximate the optimal performance index, its output is:

[0055] (17)

[0056] in, for The estimated vector, selected , ;

[0057] S42. Based on the time-series difference method, the network approximation error of equation (15) can be obtained as follows:

[0058] (18)

[0059] and

[0060] (19)

[0061] in, It is a discount factor;

[0062] Furthermore, by minimizing the error of equation (18), the CN weight update law is obtained:

[0063] (20)

[0064] in, , ;

[0065] Based on the deterministic policy gradient algorithm and equation (20), the AN weight update law can be obtained:

[0066] (twenty one)

[0067] in, , .

[0068] In one possible implementation, the control gain , and The initial value is any positive value, and the system automatically converges to the optimal value through online reinforcement learning.

[0069] In one possible implementation, the method is applicable to non-zero initial swing conditions, wherein the initial hook swing angle is... (0) and cargo angle (0) are all not 0.

[0070] In one possible implementation, the weight update period for CN and AN is less than or equal to 0.1s.

[0071] Compared with existing technologies, this invention provides an adaptive control method for a double-pendulum overhead crane based on online reinforcement learning, which has the following advantages:

[0072] This invention directly applies online reinforcement learning to a physical double-pendulum crane system, eliminating the need for precise models and offline training. It effectively overcomes the complex coupling between the hook and the cargo, while avoiding the performance degradation problem of traditional reinforcement learning methods from simulation to actual application. This allows the control system to maintain stable and reliable performance under various operating conditions, thereby improving control accuracy. Attached Figure Description

[0073] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0074] Figure 1 This is a schematic diagram of the motion model of a double pendulum crane in an adaptive control method for a double pendulum crane based on online reinforcement learning.

[0075] Figure 2 This is a schematic diagram of the adaptive control system framework based on online reinforcement learning in a double pendulum crane adaptive control method.

[0076] Figure 3 This is a schematic diagram of the trolley displacement in the first transportation scenario of a double-pendulum crane system response in an adaptive control method for double-pendulum cranes based on online reinforcement learning.

[0077] Figure 4 This is a schematic diagram of the hook swing of the double pendulum crane system response in the first transportation scenario of a double pendulum crane adaptive control method based on online reinforcement learning;

[0078] Figure 5 This is a schematic diagram of the cargo swinging response of a double-pendulum crane system in the first transportation scenario of an adaptive control method for double-pendulum cranes based on online reinforcement learning.

[0079] Figure 6 This is a schematic diagram of the network changes and parameter optimization CN weight changes in the first transportation scenario of a double-pendulum crane adaptive control method based on online reinforcement learning.

[0080] Figure 7 This is a schematic diagram of the network changes and parameter optimization of the AN weights in the first transportation scenario of a double-pendulum crane adaptive control method based on online reinforcement learning.

[0081] Figure 8 This is a schematic diagram of online parameter optimization for network changes and parameter optimization in the first transportation scenario of a double-pendulum crane adaptive control method based on online reinforcement learning.

[0082] Figure 9 This is a schematic diagram of the trolley displacement in the second transportation scenario of a double-pendulum crane system response in an adaptive control method for double-pendulum cranes based on online reinforcement learning.

[0083] Figure 10 This is a schematic diagram of the hook swing of the double pendulum crane system response under the second transportation scenario in an adaptive control method for double pendulum cranes based on online reinforcement learning.

[0084] Figure 11 This is a schematic diagram of the cargo swinging response of a double-pendulum crane system in a second transportation scenario, based on an adaptive control method for double-pendulum cranes using online reinforcement learning.

[0085] Figure 12 This is a schematic diagram of the network changes and parameter optimization CN weight changes in the second transportation scenario of a double-pendulum crane adaptive control method based on online reinforcement learning.

[0086] Figure 13 This is a schematic diagram of the network changes and parameter optimization of the AN weights in the second transportation scenario of a double-pendulum crane adaptive control method based on online reinforcement learning.

[0087] Figure 14 This is a schematic diagram of online parameter optimization for network changes and parameter optimization in the second transportation scenario of a double-pendulum crane adaptive control method based on online reinforcement learning;

[0088] Attached reference numerals: 1. Overhead crane; 2. Beam frame; 3. Rope 1; 4. Hook; 5. Rope 2; 6. Cargo. Detailed Implementation

[0089] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0090] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature; in the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0091] Please see Figures 1-14This invention provides an adaptive control method for a double-pendulum overhead crane based on online reinforcement learning, comprising the following steps:

[0092] S1. Based on the specific parameters of crane 1, the dynamic model of crane 1 is established using the Euler-Lagrange method;

[0093] S2. Transform the dynamic model into a linearized system model;

[0094] S3. Design of an adaptive control system framework based on RL;

[0095] S4. Solve for the weights of AN through RL iteration;

[0096] S5. Use the updated AN to output the weights of AN in real time as the parameters to generate the optimal gain parameters at the next time step.

[0097] In one possible implementation, the dynamic model includes: driving force This causes the overhead crane 1 to reciprocate on the beam frame 2. The lower end of the overhead crane 1 is connected to rope 3, hook 4, rope 5 and cargo 6 in sequence, forming a two-stage swing structure.

[0098] (1)

[0099] The masses of the overhead crane 1, hook 4, and cargo 6 are respectively... , and The length of rope 3 is denoted as The length of rope 2.5 is denoted as , Let gravitational acceleration be the acceleration due to gravity. The horizontal displacement of the crane 1, the swing angle of the hook 4, and the swing angle of the cargo 6 are denoted as follows: , and The system's damping parameters are used , and It is stated that the driving force of crane 1 is .

[0100] In one possible implementation, the dynamic model shows relatively small system oscillations during normal transportation, which can be approximated using a small angle:

[0101] (2)

[0102] The resulting linearized system model is as follows:

[0103] (3)

[0104] The system is also subject to air damping. , and friction The impact.

[0105] In one possible implementation, step S3 includes:

[0106] S31. Construct the state variables of the double pendulum crane system:

[0107] (4)

[0108] (5)

[0109] in, For the horizontal displacement of crane 1, Let be the desired displacement of the crane 1. The hook has a 3-angle swing. For a cargo swing angle of 5, the control objective in equation (5) can be integrated into an optimal adjustment objective;

[0110] S32. The control input for the overhead crane system is generated using an incremental PID controller:

[0111] (6)

[0112] in, , , and To control the gain, This represents the displacement tracking error of crane 1.

[0113] S33. Define the performance index function:

[0114] (7)

[0115] (8)

[0116] (9)

[0117] in, and These represent runtime and sampling time, respectively. , ;

[0118] The index shown in Equation (7) directly reflects the performance of the system. The smaller the index value, the better the system performance and the lower the energy consumption.

[0119] Furthermore, the control problem in equation (5) can be rearranged as follows:

[0120] (10)

[0121] In one possible implementation, step S4 includes:

[0122] S41. Construct two neural networks:

[0123] In equation (10), due to the presence of the perturbation term, it is difficult to solve directly. Therefore, an online iterative update method is used to handle this problem, and equation (7) can be written as:

[0124] (11)

[0125] The optimal index is defined as:

[0126] (12)

[0127] Based on equation (12), the optimal RL control strategy can be derived:

[0128] (13)

[0129] Based on the universal approximation principle, equation (13) can be rewritten in the form of NN:

[0130] (14)

[0131] in, For the ideal weight vector, It is from arrive The nonlinear activation function of the mapping It is a deviation factor;

[0132] The evaluation network (CN) is used to approximate the optimal performance index, and its output is:

[0133] (15)

[0134] in, for The estimated vector, selected , for and The second mapping;

[0135] Similarly, in order to solve the optimal control strategy in equation (13) Build another NN:

[0136] (16)

[0137] in, The set maximum gain parameter value. , For the ideal weight vector, It is from arrive The nonlinear activation function of the mapping It is a deviation factor;

[0138] Action Network (AN): Used to approximate the optimal performance index, its output is:

[0139] (17)

[0140] in, for The estimated vector, selected , ;

[0141] S42. Based on the time-series difference method, the network approximation error of equation (15) can be obtained as follows:

[0142] (18)

[0143] and

[0144] (19)

[0145] in, It is a discount factor;

[0146] Furthermore, by minimizing the error of equation (18), the CN weight update law is obtained:

[0147] (20)

[0148] in, , ;

[0149] Based on the deterministic policy gradient algorithm and equation (20), the AN weight update law can be obtained:

[0150] (twenty one)

[0151] in, , .

[0152] In one possible implementation, the control gain , and The initial value is any positive value, and the system automatically converges to the optimal value through online reinforcement learning.

[0153] In one possible implementation, the method is applicable to non-zero initial swing conditions, wherein the initial hook swing angle is 3. (0) and the cargo at a 5-degree angle (0) are all non-zero, making them fit the real operation scenario.

[0154] In one possible implementation, the weight update period of CN and AN is less than or equal to 0.1s, which can ensure the real-time stability and anti-disturbance capability of the control system and achieve the optimal balance between industrial deployment feasibility and performance.

[0155] In practice, the overhead crane system parameters were selected as follows: , , , , and Other parameters are selected as follows: , , , , To verify the superiority of the algorithm, this section compares the neural network adaptive control algorithm from the paper (Neural network adaptive PID-like coupling control for double pendulum cranes with time-varying input constraints, 2024) with the model predictive control method from the paper (Anti-sway design of overhead cranes based on model predictive control, 2020). Consider the following two working scenarios: non-zero initial sway (the initial angles are set as follows). , ) and variable-condition transportation (cargo weight, length of rope 1 and rope 2 reduced to respectively) , and ).

[0156] Depend on Figures 3-5 Observations show that the proposed algorithm has superior control performance. For displacement, the motion is smoother and the convergence speed is faster. Furthermore, the maximum value and settling time of the hook and cargo swing angles are significantly reduced. Figure 4 , 5 As shown. Specifically, the maximum angle and adjustment time of the hook swing are 6.75° and 5.90s, respectively, while the maximum angle and adjustment time of the load swing are reduced to 6.90° and 5.80s, respectively.

[0157] also, Figure 6 , Figure 7 The changes in network weights under this scenario are presented, and the results show that the online learning network updates quickly and exhibits small fluctuations, which also contributes to the good speed and stability of parameter optimization. Figure 8As shown above, the proposed online RL adaptive control algorithm exhibits superior dynamic performance and robustness under non-zero initial oscillation conditions.

[0158] Depend on Figures 9-11 As can be seen, even under these changing operating conditions, the proposed algorithm can still achieve rapid trolley positioning and cargo sway suppression. For displacement, the curve is smoother and the settling time is shorter. For system sway, this method significantly reduces both the maximum swing angle and the settling time. Specifically, the maximum angle and settling time of the hook sway are reduced to 0.86° and 6.50s, respectively, and the maximum angle and settling time of the load sway are reduced to 0.88° and 6.43s, respectively. Figure 12 , Figure 13 This indicates that the online RL network can achieve fast and stable convergence, which also guarantees its superior performance in parameter optimization, such as... Figure 14 As shown in the figure. The above analysis demonstrates that the proposed algorithm possesses superior performance and adaptability.

[0159] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A double swing sheave adaptive control method based on online reinforcement learning, characterized in that, Includes the following steps: S1. Based on the specific parameters of the overhead crane (1), the dynamic model of the overhead crane (1) is established using the Euler-Lagrange method; S2. Transform the dynamic model into a linearized system model; S3. Design of an adaptive control system framework based on RL; S4. Solve for the weights of AN through RL iteration; S5. Use the updated AN to output the weights of AN in real time as the parameters to generate the optimal gain parameters at the next time step.

2. The adaptive control method for a double-pendulum overhead crane based on online reinforcement learning according to claim 1, characterized in that, The kinetic model includes: driving force The crane 1 reciprocates on the beam 2, and the lower end of the crane (1) sequentially passes through the rope one (3), the hook (4), the rope two (5), and the connected goods (6), forming a two-stage swing structure: (1) The masses of the overhead crane (1), hook (4), and cargo (6) are respectively , and The length of rope 1 (3) is denoted as The length of rope two (5) is denoted as , Let gravitational acceleration be the acceleration due to gravity. The horizontal displacement of the crane (1), the swing angle of the hook (4), and the swing angle of the cargo (6) are denoted as follows: , and The system's damping parameters are used , and This indicates that the driving force of the overhead crane (1) is .

3. The adaptive control method for a double-pendulum overhead crane based on online reinforcement learning according to claim 2, characterized in that, The dynamic model shows minimal system oscillation during normal transportation, which can be approximated using a small angle. (2) The resulting linearized system model is as follows: (3) The system is also subject to air damping. , and friction The impact.

4. The adaptive control method for a double-pendulum overhead crane based on online reinforcement learning according to claim 1, characterized in that, Step S3 includes: S31. Construct the state variables of the double pendulum crane system: (4) (5) in, For the horizontal displacement of crane 1, For the expected displacement of the crane (1), For the swing angle of the hook (3), Given the swing angle of the cargo (5), the control objectives in equation (5) can be integrated into an optimal adjustment objective; S32. The control input for the overhead crane system is generated using an incremental PID controller: (6) in, , , and To control the gain, The displacement tracking error of the overhead crane (1); S33. Define the performance index function: (7) (8) (9) in, and These represent runtime and sampling time, respectively. , ; The index shown in Equation (7) directly reflects the performance of the system. The smaller the index value, the better the system performance and the lower the energy consumption. Furthermore, the control problem in equation (5) can be rearranged as follows: (10)。 5. The adaptive control method for a double-pendulum overhead crane based on online reinforcement learning according to claim 1, characterized in that, The S4 step includes: S41. Construct two neural networks: In equation (10), due to the presence of the perturbation term, it is difficult to solve directly. Therefore, an online iterative update method is used to handle this problem, and equation (7) can be written as: (11) The optimal index is defined as: (12) Based on equation (12), the optimal RL control strategy can be derived: (13) Based on the universal approximation principle, equation (13) can be rewritten in the form of NN: (14) in, For the ideal weight vector, It is from arrive Nonlinear activation function of the mapping It is a deviation factor; The evaluation network (CN) is used to approximate the optimal performance index, and its output is: (15) in, for The estimated vector, selected , for and The second mapping; Similarly, in order to solve the optimal control strategy in equation (13) Build another NN: (16) in, The set maximum gain parameter value. , For the ideal weight vector, It is from arrive Nonlinear activation function of the mapping It is a deviation factor; Action Network (AN): Used to approximate the optimal performance index, its output is: (17) in, for The estimated vector, selected , ; S42. Based on the time-series difference method, the network approximation error of equation (15) can be obtained as follows: (18) and (19) in, It is a discount factor; Furthermore, by minimizing the error of equation (18), the CN weight update law is obtained: (20) in, , ; Based on the deterministic policy gradient algorithm and equation (20), the AN weight update law can be obtained: (21) in, , .

6. The adaptive control method for a double-pendulum overhead crane based on online reinforcement learning according to claim 4, characterized in that, The control gain , and The initial value is any positive value, and the system automatically converges to the optimal value through online reinforcement learning.

7. The adaptive control method for a double-pendulum overhead crane based on online reinforcement learning according to claim 2, characterized in that, The method is applicable to non-zero initial swing conditions, where the initial hook (3) swing angle (0) and the angle of the goods (5) (0) are all not 0.

8. The adaptive control method for a double-pendulum overhead crane based on online reinforcement learning according to claim 5, characterized in that, The weight update period for CN and AN is less than or equal to 0.1s.

Images