Suspension bridge aerial spinning method construction traction system control method based on reinforcement learning

Abstract

The application belongs to the technical field of bridge construction, and particularly relates to a suspension bridge aerial spinning method construction traction system control method based on reinforcement learning. The specific steps of the method are as follows: modeling the AS method spinning process through Simulink; building a reinforcement learning environment, wherein the state space comprises the spinning wheel position, speed, acceleration and the height of the dynamic pulley of the tension balance tower from the ground, the action space comprises the PID controller parameters of the traction and spinning equipment, the reward function comprises the trajectory smoothness reward, the speed stability reward, the speed mutation penalty, the pulley height penalty and the pulley height mutation penalty; training is performed by using a DDPG reinforcement learning model; the trained DDPG strategy network is deployed to hardware control, the dynamic optimization of the PID parameters is realized according to the state data collected by the sensor, and the intelligent control of the spinning process is realized. The application realizes the accurate regulation and control of the spinning wheel running speed, significantly improves the control precision and construction efficiency of the main cable traction process, and reduces the demand for human intervention.

Description

Technical Field

[0001] This invention belongs to the field of bridge construction technology, specifically a control method for the traction system of the aerial spinning method for suspension bridge construction based on reinforcement learning. Background Technology

[0002] Currently, when using the AS method for erecting the main cable strands of suspension bridges, the control of the traction winch and wire unwinding machine mainly relies on the classic PID algorithm. PID (Proportional-Integral-Derivative) control algorithm is an automatic control algorithm widely used in industrial control. It can adjust the control quantity according to the system error (i.e., the difference between the set value and the actual value), thereby achieving system control.

[0003] However, with the gradual increase in bridge span, existing control theories are difficult to adapt to the complex and ever-changing construction environment and the requirements for high-quality construction. Due to the lag in manual parameter adjustment, there are problems such as low accuracy of traction speed control, poor traction-wire release coordination, low traction efficiency, and insufficient intelligence. There is an urgent need to propose an intelligent control method for wire traction that is suitable for the AS method to ensure stable and efficient system operation and high-precision coordination of traction and wire release. Summary of the Invention

[0004] To address the problems in existing suspension bridge aerial spinning construction control, such as large fluctuations in spinning wheel speed, poor stability of the traction process, and asynchrony between traction and unloading equipment caused by the lag in manual parameter adjustment, the present invention aims to provide a reinforcement learning-based control method for the traction system of suspension bridge aerial spinning construction. By constructing a multi-objective reinforcement learning framework including trajectory smoothness reward, speed stability reward, speed mutation penalty, pulley height penalty, and pulley height mutation penalty, and combining the DDPG algorithm to dynamically optimize the PID control parameters of the traction and unloading equipment, precise control of the spinning wheel speed is achieved, significantly improving the control accuracy and construction efficiency of the main cable traction process, while reducing the need for human intervention.

[0005] The technical solution of this invention is: a control method for a traction system in the aerial spinning method of suspension bridge construction based on reinforcement learning, comprising the following steps: S1: Model the AS method wire feeding process using Simulink; S2: Build a reinforcement learning environment and define the state space and action space. The state space includes the position, speed, acceleration of the spinning wheel and the height of the tension balance tower pulley from the ground. The action space includes the PID controller parameters of the traction and unloading equipment, specifically including reward functions for trajectory smoothness, speed stability, speed mutation penalty, pulley height penalty and pulley height mutation penalty. S3: Use the DDPG reinforcement learning model for training, initialize the neural network, and set the training parameters; S4: Deploy the trained DDPG policy network into the hardware control system, and dynamically optimize the PID parameters based on the state data collected by the sensors to achieve intelligent control of the spinning process.

[0006] In step S1, Simulink is used to model the traction equipment, yarn feeding equipment, spinning wheel, pulley block, tension balance tower pulley block, counterweight, working load and traction cable of the spinning method construction traction system, and to build a simulation environment for AS method spinning.

[0007] In step S2, a reinforcement learning environment is built. Combined with the construction process of the AS method, the spinning traction process is described by the kinematic state space S(x, z, v, a, h) of the spinning wheel, where x and z are the horizontal and vertical coordinates of the spinning wheel, respectively, v is the speed of the spinning wheel, a is the acceleration of the spinning wheel, and h is the height of the moving pulley of the balance tower from the ground; the motion space is A(K p1 K i1 K d1 K p2 K i2 K d2 ), R total Reward function and sampling interval, where K p1 K i1 K d1 These are the proportional coefficient, integral coefficient, and differential coefficient of the traction equipment, respectively; K p2 K i2 K d2 These are the proportional coefficient, integral coefficient, and differential coefficient of the wire feeding equipment, respectively.

[0008] The reward function R in step S2 total It is the feedback from the environment to the agent's actions, used to quantify the quality of action choices, and the reward function R. total The settings take into account the operating speed and stability of the spinning wheel, and include the following components: R total =R smooth +R acc +R vel +R h +R dh (1) (2) (3) (4) (5) (6) Among them, R total For the total score, R smooth R is the reward for trajectory smoothness. acc As a penalty for speed mutation, R dh For the penalty of sudden changes in pulley height, R vel For velocity stability constraints, R h Penalty for pulley height; k1, k2, k3, c1, c2, c3, and c4 are their respective coefficients, calibrated based on model training performance, v min To allow the minimum speed, v max The maximum permissible speed is set according to the spinning process requirements and winch performance; v design For the designed operating speed, h min h is the minimum allowable height of the movable pulley of the balance tower. max The maximum allowable height of the balancing tower's moving pulley is determined according to the spinning operation requirements. dt is the sampling interval, N is the total number of time steps, representing the time window length for monitoring the spinning wheel's trajectory; t is the time step number; kt is the trajectory curvature of the spinning wheel at time t, and the absolute difference |kt| t -k t-1 |Measures the magnitude of the change in trajectory curvature between adjacent time steps; the larger the difference, the less smooth the trajectory.

[0009] In step S3, the DDPG reinforcement learning model is used for training, the neural network is initialized, and the training parameters are set. The specific steps are as follows: S31: Initialize the network structure, create four neural networks: an online actor network (policy network, used to generate deterministic actions), a target actor network (used for stable training, with parameters synchronized with the online actor), an online critic network (Q-value network, used to evaluate the value of state-action pairs), and a target critic network (calculate the target Q-value, with parameters synchronized with the online critic), and initialize the experience replay pool to store interaction samples; S32: Set the number of training sessions and the number of steps per session, set the batch size for each training group, and give the initial state S of the traction system. t (x) t , z t v t a t h t ); S33: Agent training, the specific process is as follows: S331: Using state S as input to the model, an online actor network generates deterministic actions A. t (K) p1t Ki1t K d1t K p2t K i2t K d2t ), and add exploration noise; S332: Based on K after adding noise p1t K i1t K d1t K p2t K i2t K d2t The deviation between the actual speed and the expected speed Δv, and the deviation between the actual height and the expected height of the pulley at the balance tower Δh are used for PID calculation to calculate and send the control parameters to the traction equipment and the unwinding equipment. S333: The Simulink model updates the state space based on the motor's operating parameters to obtain S t+1 Simultaneously calculate the single-step reward r t This data (S) t A t r t S t+1 ) Store the samples in the experience replay pool. If the number of samples in the experience replay pool is less than the set number of batches, repeat steps S331-S333. S334: Training and updating the network, specifically including: Critic network update: A batch of samples is randomly sampled from the experience replay pool. The target actor generates the target action based on the next state. The target critic calculates the target Q value, calculates the loss function (mean squared error) of the critic, and updates the parameters of the online critic network through gradient descent. Actor Network Update: The goal of the online Actor is to maximize the Q-value and update the parameters of the online Actor network through gradient ascent; A soft update strategy is adopted to slowly synchronize the target network parameters; S335: Repeat S331-S334. Stop training when the network converges or reaches the preset number of iterations. The online actor network (policy model) is the final training result.

[0010] The technical advantages of this invention are as follows: 1. The suspension bridge aerial spinning method traction system control method based on reinforcement learning can automatically adjust the PID parameters of the equipment according to the speed and stability of the spinning wheel and the height of the pulley of the balance tower. This effectively solves the problem of poor parameter tuning flexibility in traditional PID control, significantly improves the control accuracy and stability of cable traction, and enhances construction efficiency. 2. This invention defines a comprehensive state space including the position, speed, acceleration of the spinning wheel, and the height of the moving pulley of the balance tower, as well as an action space including the proportional, integral, and derivative coefficients of the PID controller. This provides clear inputs and outputs for the reinforcement learning model, enabling the model to more accurately perceive the construction status and make control decisions. 3. This invention designs a multi-objective reward function that includes penalties for trajectory smoothness, speed stability, speed mutation, and height mutation. This allows the model to consider multiple performance indicators simultaneously during training, balancing the requirements of construction accuracy, efficiency, and stability.

[0011] The following will provide further explanation in conjunction with the accompanying drawings. Attached Figure Description

[0012] Figure 1 This is a flowchart of the intelligent control process for cable traction in this invention. Detailed Implementation

[0013] Example 1 A reinforcement learning-based control method for the traction system of a suspension bridge aerial spinning method includes the following steps: S1: Model the AS method wire feeding process using Simulink; S2: Build a reinforcement learning environment and define the state space and action space. The state space includes the position, speed, acceleration of the spinning wheel and the height of the tension balance tower pulley from the ground. The action space includes the PID controller parameters of the traction and unloading equipment, specifically including reward functions for trajectory smoothness, speed stability, speed mutation penalty, pulley height penalty and pulley height mutation penalty. S3: Use the DDPG reinforcement learning model for training, initialize the neural network, and set the training parameters; S4: Deploy the trained DDPG policy network into the hardware control system, and dynamically optimize the PID parameters based on the state data collected by the sensors to achieve intelligent control of the spinning process.

[0014] In step S1, Simulink is used to model the traction equipment, yarn feeding equipment, spinning wheel, pulley block, tension balance tower pulley block, counterweight, working load and traction cable of the spinning method construction traction system, and to build a simulation environment for AS method spinning.

[0015] In step S2, a reinforcement learning environment is built. Combined with the construction process of the AS method, the spinning traction process is described by the kinematic state space S(x, z, v, a, h) of the spinning wheel, where x and z are the horizontal and vertical coordinates of the spinning wheel, respectively, v is the speed of the spinning wheel, a is the acceleration of the spinning wheel, and h is the height of the moving pulley of the balance tower from the ground; the motion space is A(K p1 K i1 K d1 K p2 K i2 K d2 ), R total Reward function and sampling interval, where K p1 K i1 K d1 These are the proportional coefficient, integral coefficient, and differential coefficient of the traction equipment, respectively; K p2 K i2 K d2 These are the proportional coefficient, integral coefficient, and differential coefficient of the wire feeding equipment, respectively.

[0016] The reward function R in step S2 total It is the feedback from the environment to the agent's actions, used to quantify the quality of action choices, and the reward function R. total The settings take into account the operating speed and stability of the spinning wheel, and include the following components: R total =R smooth +R acc +R vel +R h +R dh (1) (2) (3) (4) (5) (6) Among them, R total For the total score, R smooth R is the reward for trajectory smoothness. acc As a penalty for speed mutation, R dh For the penalty of sudden changes in pulley height, R vel For velocity stability constraints, R h Penalty for pulley height; k1, k2, k3, c1, c2, c3, and c4 are their respective coefficients, calibrated based on model training performance, v min To allow the minimum speed, v maxThe maximum permissible speed is set according to the spinning process requirements and winch performance; v design For the designed operating speed, h min h is the minimum allowable height of the movable pulley of the balance tower. max The maximum allowable height of the balancing tower's moving pulley is determined according to the spinning operation requirements. dt is the sampling interval, N is the total number of time steps, representing the time window length for monitoring the spinning wheel's trajectory; t is the time step number; kt is the trajectory curvature of the spinning wheel at time t, and the absolute difference |kt| t -k t-1 |Measures the magnitude of the change in trajectory curvature between adjacent time steps; the larger the difference, the less smooth the trajectory.

[0017] In step S3, the DDPG reinforcement learning model is used for training, the neural network is initialized, and the training parameters are set. The specific steps are as follows: S31: Initialize the network structure, create four neural networks: an online actor network (policy network, used to generate deterministic actions), a target actor network (used for stable training, with parameters synchronized with the online actor), an online critic network (Q-value network, used to evaluate the value of state-action pairs), and a target critic network (calculate the target Q-value, with parameters synchronized with the online critic), and initialize the experience replay pool to store interaction samples; S32: Set the number of training sessions and the number of steps per session, set the batch size for each training group, and give the initial state S of the traction system. t (x) t , z t v t a t h t ); S33: Agent training, the specific process is as follows: S331: Using state S as input to the model, an online actor network generates deterministic actions A. t (K) p1t K i1t K d1t K p2t K i2t K d2t ), and add exploration noise; S332: Based on K after adding noise p1t K i1t K d1t K p2t K i2t K d2tThe deviation between the actual speed and the expected speed Δv, and the deviation between the actual height and the expected height of the pulley at the balance tower Δh are used for PID calculation to calculate and send the control parameters to the traction equipment and the unwinding equipment. S333: The Simulink model updates the state space based on the motor's operating parameters to obtain S t+1 Simultaneously calculate the single-step reward r t This data (S) t A t r t S t+1 ) Store the samples in the experience replay pool. If the number of samples in the experience replay pool is less than the set number of batches, repeat steps S331-S333. S334: Training and updating the network, specifically including: Critic network update: A batch of samples is randomly sampled from the experience replay pool. The target actor generates the target action based on the next state. The target critic calculates the target Q value, calculates the loss function (mean squared error) of the critic, and updates the parameters of the online critic network through gradient descent. Actor Network Update: The goal of the online Actor is to maximize the Q-value and update the parameters of the online Actor network through gradient ascent; A soft update strategy is adopted to slowly synchronize the target network parameters; S335: Repeat S331-S334. Stop training when the network converges or reaches the preset number of iterations. The online actor network (policy model) is the final training result.

[0018] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A control method for the traction system of the aerial spinning method for suspension bridge construction based on reinforcement learning, characterized in that: Includes the following steps: S1: Model the AS method yarn feeding process using Simulink; In step S1, Simulink is used to model the traction equipment, yarn feeding equipment, spinning wheel, pulley block, tension balance tower pulley block, counterweight, working load and traction cable of the yarn spinning method construction traction system, and build a simulation environment for AS method spinning. S2: Establish a reinforcement learning environment and define the state space and action space. The state space includes the position, velocity, acceleration of the spinning wheel, and the height of the moving pulley of the tension balancing tower from the ground. The action space includes the PID controller parameters of the traction and unloading equipment, specifically including reward functions for trajectory smoothness, velocity stability, velocity mutation, pulley height, and pulley height mutation. In step S2, a reinforcement learning environment is established. Combined with the construction process of the AS method, the spinning traction process is described by the kinematic state space S(x, z, v, a, h) of the spinning wheel, where x and z are the horizontal and vertical coordinates of the spinning wheel, respectively, v is the running speed of the spinning wheel, a is the acceleration of the spinning wheel, and h is the height of the moving pulley of the balancing tower from the ground. The action space is A(K) p1 K i1 K d1 K p2 K i2 K d2 ), R total Reward function and sampling interval, where K p1 K i1 K d1 These are the proportional coefficient, integral coefficient, and differential coefficient of the traction equipment, respectively; K p2 K i2 K d2 These are the proportional coefficient, integral coefficient, and differential coefficient of the wire feeding device; the reward function R total It is the feedback from the environment to the agent's actions, used to quantify the quality of action choices, and the reward function R. total The settings take into account the operating speed and stability of the spinning wheel, and include the following components: R total =R smooth +R acc +R vel +R h +R dh (1) （2） （3） （4） （5） （6） Among them, R total For the total score, R smooth R is the reward for trajectory smoothness. acc As a penalty for speed mutation, R dh For the penalty of sudden changes in pulley height, R vel For velocity stability constraints, R h Penalty for pulley height; k1, k2, k3, c1, c2, c3, and c4 are their respective coefficients, calibrated based on model training performance, v min To allow the minimum speed, v max The maximum permissible speed is set according to the spinning process requirements and winch performance; v design For the designed operating speed, h min h is the minimum allowable height of the movable pulley of the balance tower. max The maximum allowable height of the balancing tower's moving pulley is determined according to the spinning operation requirements. dt is the sampling interval, N is the total number of time steps, representing the time window length for monitoring the spinning wheel's trajectory; t is the time step number; kt is the trajectory curvature of the spinning wheel at time t, and the absolute difference |kt| t -k t-1 |Measures the magnitude of the change in trajectory curvature between adjacent time steps; the larger the difference, the less smooth the trajectory. S3: Use the DDPG reinforcement learning model for training, initialize the neural network, and set the training parameters; S4: Deploy the trained DDPG policy network into the hardware control system, and dynamically optimize the PID parameters based on the state data collected by the sensors to achieve intelligent control of the spinning process.

2. The control method for the traction system of the suspension bridge aerial spinning method construction based on reinforcement learning as described in claim 1, characterized in that: In step S3, the DDPG reinforcement learning model is used for training, the neural network is initialized, and the training parameters are set. The specific steps are as follows: S31: Initialize the network structure, create four neural networks, namely the online actor network, the target actor network, the online critic network, and the target critic network, and initialize the experience replay pool to store interaction samples; S32: Set the number of training sessions and the number of steps per session, set the number of batches per training group for the model, and give the initial state S of the traction system. t (x) t , z t v t a t h t ); S33: Agent training, the specific process is as follows: S331: Using state S as input to the model, an online actor network generates deterministic actions A. t (K) p1t K i1t K d1t K p2t K i2t K d2t ), and add exploration noise; S332: Based on K after adding noise p1t K i1t K d1t K p2t K i2t K d2t The deviation between the actual speed and the expected speed Δv, and the deviation between the actual height and the expected height of the pulley at the balance tower Δh are used for PID calculation to calculate and send the control parameters to the traction equipment and the unwinding equipment. S333: The Simulink model updates the state space based on the motor's operating parameters to obtain S t+1 Simultaneously calculate the single-step reward r t This data (S) t A t r t S t+1 ) Store the samples in the experience replay pool. If the number of samples in the experience replay pool is less than the set number of batches, repeat steps S331-S333. S334: Training and updating the network, specifically including: Critic network update: Randomly sample a batch of samples from the experience replay pool, the target actor generates the target action based on the next state, the target Critic calculates the target Q value, calculates the Critic's loss function, and updates the parameters of the online Critic network through gradient descent; Actor Network Update: The goal of the online Actor is to maximize the Q-value and update the parameters of the online Actor network through gradient ascent; A soft update strategy is adopted to slowly synchronize the target network parameters; S335: Repeat S331-S334. Stop training when the network converges or reaches the preset number of iterations. The online actor network is the final training result.

Images