Crane anti-sway positioning control method based on near-end strategy optimization

Abstract

This invention discloses a crane anti-sway positioning control method based on near-end strategy optimization. It is characterized by comprising a trolley and cable mechanism, a rope length detection module, a position and velocity detection module, a swing angle detection module, a deep reinforcement learning controller, and a control panel. The rope length detection module measures the rope length. The position and velocity detection module detects the trolley position, crane position, trolley velocity, and crane velocity. The swing angle detection module outputs the swing angle and angular velocity of the suspended object. The control panel outputs the target position and mass of the suspended object. The deep reinforcement learning controller inputs the data collected by the above modules as state information, processes and analyzes the data, calculates the optimal control signal, and transmits the control signal to the trolley, crane, and cable mechanism.

Description

Technical Field

[0001] This invention relates to the field of motion control methods, and more specifically to a crane anti-sway positioning control method based on proximal strategy optimization. Background Technology

[0002] Deep reinforcement learning algorithms have achieved remarkable control results in fields such as industrial robots and autonomous driving. This algorithm does not require precise system modeling; it autonomously learns and optimizes control strategies through continuous interaction between the agent and its environment. Deep reinforcement learning combines the perceptual capabilities of deep learning with the decision-making abilities of reinforcement learning, offering significant advantages in handling complex perception and decision-making problems.

[0003] Traditional crane control methods, such as PID, linear quadratic regulator (LQR), and model predictive control (MPC), have limited performance in complex environments, rely on accurate modeling, and have poor adaptability to environmental changes.

[0004] In summary, there is an urgent need to provide a crane anti-sway positioning control method based on near-end strategy optimization. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a crane anti-sway positioning control method based on near-end strategy optimization. This method exhibits stable performance under different load, disturbance, and task conditions and can be transferred to new scenarios, demonstrating significantly enhanced adaptability compared to traditional methods.

[0006] The technical solution of the present invention is: a crane anti-sway positioning control method based on near-end strategy optimization, including a crane and cable mechanism, a rope length detection module, a position and speed detection module, a swing angle detection module, a deep reinforcement learning controller and a control panel;

[0007] The rope length detection module is used to measure the rope length. The position and speed detection module is used to detect the position of the vehicle. Large vehicle position Car speed large vehicle speed The swing angle detection module is used to output the swing angle of the suspended object. and angular velocity The control panel outputs the target position of the suspended object. and the weight of the suspended object ;

[0008] The deep reinforcement learning controller takes the data collected by the above modules as state information input, processes and analyzes it, calculates the optimal control signal, and sends the control signal to the large and small trolleys and the cable mechanism.

[0009] Furthermore, the deep reinforcement learning controller training process includes:

[0010] Step 1: Initialize the working environment;

[0011] Step 2: Construct a deep reinforcement learning controller based on proximal policy optimization, which includes an action network and a value network;

[0012] Step 3: The crane controller continuously interacts with the working environment to collect experience datasets;

[0013] Step 4: Train the action network and update its parameters;

[0014] Step 5: Train the value network and update its parameters;

[0015] Step 6: Repeat steps 3-5 until the parameter changes of the action network and value network are less than a certain threshold, or the cumulative reward no longer increases significantly. Stop training and save the latest action network and value network parameters.

[0016] In further step 1, the initialization of the working environment is based on the initial position of the trolley target position in the crane's working environment. target location of the large vehicle Car position Car speed Large vehicle position large vehicle speed Swing angle angular velocity of the pendulum Cable length The weight of the suspended object, m, is used as status information of the working environment and is denoted as... s =[ x r , y r , x , y , x ̇ , y ̇ , θ , θ , ̇ l , m ] .

[0017] Furthermore, in step 2, the crane controller and the working environment follow the current strategy. Interact with the system to generate a series of experiential data. ;

[0018] in, yes Real-time status information of the work environment In the current state The lower crane controller makes an action. New status information generated by the work environment;

[0019] a t =[ ω t 1 , ω t 2 , ω t 3 ] ,in, for The speed of the car motor is constantly changing. for The speed of the main vehicle motor is constantly monitored. for Constant cable motor speed; reward yes The reward obtained is calculated using the reward function at any time;

[0020] The crane controller continuously interacts with the working environment until a sufficient amount of empirical data is collected to form an empirical dataset. .

[0021] Furthermore, the reward Including location rewards and sway angle reward The total reward is the sum of the position reward and the swing angle reward. ;

[0022] The location reward is given when the crane reaches the destination position, based on the error between the destination position and the target position, as follows:

[0023]

[0024] in The target location of the suspended object. , for The location of the car at any time for The location of the large vehicle at any given time. This indicates a calculation of Euclidean distance. To achieve the required positional accuracy, , ;

[0025] The swing angle reward is given according to the size of the swing angle, as shown in the following formula:

[0026]

[0027] In the formula, for The angle of the suspended object at any given moment. To achieve the required swing angle accuracy, , .

[0028] Furthermore, step 4 specifically includes the following steps:

[0029] Step 4.1: From the empirical dataset Randomly sample N empirical data points;

[0030] Step 4.2, through value networks To assess the current state The value below ;

[0031] Step 4.3: Calculate the estimated value of the advantage function. :

[0032]

[0033] in, Discount factor;

[0034] Step 4.4: Calculate the loss of the action network. :

[0035]

[0036] in, It is the new and old action networks at all times The probability ratio of taking an action. These are the motion network parameters before the update. For clipping functions, These are trimming parameters used to limit the difference between new and old actions;

[0037] This loss function is used to measure the difference between the current strategy and the strategy that maximizes cumulative rewards.

[0038] Step 4.5: Using the Adam optimization algorithm, update the parameters of the action network based on the loss function to optimize the action network in a direction that can obtain greater cumulative rewards.

[0039] Furthermore, step 5 specifically includes the following steps:

[0040] Step 5.1: Based on the rewards in the sampled data And the value estimation of the next state Calculate the target value ;

[0041] Step 5.2: Calculate the loss of the value network. :

[0042]

[0043] Step 5.3: Using the Adam optimization algorithm, update the parameters of the value network based on the loss function so that the value network can more accurately evaluate the state value.

[0044] The beneficial technical effects of this invention are:

[0045] 1. This invention proposes a crane anti-sway positioning control method based on near-end strategy optimization, combining the perception capabilities of deep learning and the decision-making capabilities of reinforcement learning. Deep learning is used to accurately perceive the crane's working environment, load characteristics, and motion state, extracting useful feature information. Reinforcement learning is used to enable the crane to interact with the external working environment to collect information and autonomously learn an anti-sway positioning control strategy. Through continuous trial and error and learning, the crane can gradually learn the optimal anti-sway positioning control strategy, reducing swaying during operation and accurately positioning itself to the set location.

[0046] 2. Traditional crane control often relies on pre-established precise dynamic mathematical models. However, this method utilizes reinforcement learning to autonomously learn anti-sway positioning control strategies through the interaction between the crane and its external environment. This autonomous learning capability frees the crane from dependence on precise dynamic mathematical models, avoiding the difficulty of accurate modeling due to the complexity of crane systems and the variability of operating conditions, thus lowering the technical implementation threshold. It also reduces control deviations caused by model errors, further improving the reliability and stability of control. Furthermore, it can adjust strategies in real time according to actual working conditions, adapting to different operating scenarios and greatly enhancing the flexibility and adaptability of the control strategy.

[0047] 3. Deep learning methods enable precise perception of the crane's working environment, load characteristics, and motion state, effectively extracting key feature information. Compared to traditional methods, this significantly improves the ability to understand complex working conditions, providing solid input data for subsequent control decisions. Combined with the autonomous decision-making capabilities of reinforcement learning, the crane can continuously optimize its anti-sway positioning strategy, significantly improving positioning accuracy and ensuring that the load accurately reaches the target position.

[0048] 4. Through continuous trial and error and learning, cranes can gradually master the optimal anti-sway positioning control strategy. In actual operation, this significantly reduces swaying during operation, while accurately moving to the set position, improving operational safety and accuracy, reducing the risk of damage to the load caused by swaying, and mitigating the problem of low operational efficiency due to positioning deviations.

[0049] 5. Precise anti-sway positioning control can reduce the adjustment time of the crane during the lifting process, so that each lifting task can be completed more quickly, thereby improving the overall operation efficiency and saving a lot of time and labor costs in industrial production and other fields.

[0050] 6. This method enables cranes to have intelligent capabilities, reduces manual intervention by operators in complex lifting tasks, reduces reliance on operators' experience and skill levels, alleviates operators' workload, and also reduces the possibility of human error.

[0051] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, the preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings. Attached Figure Description

[0052] Figure 1 This is a schematic diagram of the overall architecture of the present invention;

[0053] Figure 2 This is a diagram of the deep reinforcement learning algorithm framework of the present invention;

[0054] Figure 3 This is a schematic diagram of the action network structure of the present invention;

[0055] Figure 4 This is a schematic diagram of the value network structure of the present invention. Detailed Implementation

[0056] In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, the specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0057] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate for the embodiments of this application described herein.

[0058] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship described in the embodiments and shown in the accompanying drawings, or the orientation or positional relationship that the product of this invention is usually placed in during use. They are only for the convenience of describing this invention and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0059] like Figure 1 As shown, the present invention specifically relates to a crane anti-sway positioning control method based on near-end strategy optimization, including a crane and cable mechanism, a rope length detection module, a position and speed detection module, a swing angle detection module, a deep reinforcement learning controller, and a control panel.

[0060] The rope length detection module is used to measure the rope length. The position and speed detection module is used to detect the position of the vehicle. Large vehicle position Car speed large vehicle speed .

[0061] The swing angle detection module is used to output the swing angle of the suspended object. and angular velocity .

[0062] The control panel outputs the target position of the suspended object. and the weight of the suspended object .

[0063] The deep reinforcement learning controller takes the data collected by the above modules as state information input, processes and analyzes it, calculates the optimal control signal, and sends the control signal to the large and small trolleys and the cable mechanism.

[0064] Furthermore, such as Figure 2 As shown, the training process of the deep reinforcement learning controller includes:

[0065] Step 1: Initialize the working environment;

[0066] Furthermore, in step 1, the initialization of the working environment is based on the initial position of the trolley target in the crane's working environment. target location of the large vehicle Car position Car speed Large vehicle position large vehicle speed Swing angle angular velocity of the pendulum Cable length The weight of the suspended object, m, is used as status information of the working environment and is denoted as... s =[ x r , y r , x , y , x ̇ , y ̇ , θ , θ , ̇ l , m ] .

[0067] Step 2: Construct a deep reinforcement learning controller based on proximal policy optimization, which includes an action network and a value network;

[0068] Among them, such as Figure 3 As shown, the action network consists of an input layer containing a 10-dimensional vector with state information, a fully connected layer with 256 neurons, layer normalization, a ReLU activation function, and an output layer.

[0069] Among them, such as Figure 4 As shown, the value network consists of an input layer containing a 10-dimensional vector with state information, a fully connected layer with 256 neurons, layer normalization, ReLU activation function, a fully connected layer with 1 neuron, and an output layer.

[0070] Furthermore, in step 2, the crane controller and the working environment follow the current strategy. Interact with the system to generate a series of experiential data. ;

[0071] in, yes Real-time status information of the work environment In the current state The lower crane controller makes an action. New status information generated by the work environment;

[0072] a t =[ ω t 1 , ω t 2 , ω t 3 ] ,in, for The speed of the car motor is constantly changing. for The speed of the main vehicle motor is constantly monitored. for Constant cable motor speed; reward yes The reward obtained is calculated using the reward function at any time;

[0073] The crane controller continuously interacts with the working environment until a sufficient amount of empirical data is collected to form an empirical dataset. .

[0074] Furthermore, the reward Including location rewards and sway angle reward The total reward is the sum of the position reward and the swing angle reward. ;

[0075] The location reward is given when the crane reaches the destination position, based on the error between the destination position and the target position, as follows:

[0076]

[0077] in The target location of the suspended object. , for The location of the car at any time for The location of the large vehicle at any given time. This indicates a calculation of Euclidean distance. To achieve the required positional accuracy, , ;

[0078] The swing angle reward is given according to the size of the swing angle, as shown in the following formula:

[0079]

[0080] In the formula, for The angle of the suspended object at any given moment. To achieve the required swing angle accuracy, , .

[0081] Setting different error ranges can help the agent more easily find the required accuracy.

[0082] Step 3: The crane controller continuously interacts with the working environment to collect experience datasets;

[0083] Step 4: Train the action network and update its parameters;

[0084] Furthermore, step 4 specifically includes the following steps:

[0085] Step 4.1: From the empirical dataset Randomly sample N empirical data points;

[0086] Step 4.2, through value networks To assess the current state The value below ;

[0087] Step 4.3: Calculate the estimated value of the advantage function. :

[0088]

[0089] in, Discount factor;

[0090] Step 4.4: Calculate the loss of the action network. :

[0091]

[0092] in, It is the new and old action networks at all times The probability ratio of taking an action. These are the motion network parameters before the update. For clipping functions, These are trimming parameters used to limit the difference between new and old actions;

[0093] This loss function is used to measure the difference between the current strategy and the strategy that maximizes cumulative rewards.

[0094] Step 4.5: Using the Adam optimization algorithm, update the parameters of the action network based on the loss function to optimize the action network in a direction that can obtain greater cumulative rewards.

[0095] Step 5: Train the value network and update its parameters.

[0096] Furthermore, step 5 specifically includes the following steps:

[0097] Step 5.1: Based on the rewards in the sampled data And the value estimation of the next state Calculate the target value ;

[0098] Step 5.2: Calculate the loss of the value network. :

[0099]

[0100] Step 5.3: Using the Adam optimization algorithm, update the parameters of the value network based on the loss function so that the value network can more accurately evaluate the state value.

[0101] Step 6: Repeat steps 3-5 until the parameter changes of the action network and value network are less than a certain threshold, or the cumulative reward no longer increases significantly. Stop training and save the latest action network and value network parameters.

[0102] The above embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and are not intended to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the scope of the technology disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention.

Claims

1. A crane anti-sway positioning control method based on near-end strategy optimization, characterized in that, It includes a crane and cable mechanism, a rope length detection module, a position and speed detection module, a swing angle detection module, a deep reinforcement learning controller, and a control panel; The rope length detection module is used to measure the rope length. The position and speed detection module is used to detect the position of the vehicle. Large vehicle position Car speed Large vehicle speed ; The swing angle detection module is used to output the swing angle of the suspended object. and angular velocity The control panel outputs the target position of the vehicle. target location of the large vehicle and the weight of the suspended object ; The deep reinforcement learning controller takes the data collected by the above modules and the data output by the control panel as the status information input. After processing and analysis by the deep reinforcement learning controller, it calculates the optimal control signal and sends the control signal to the large and small trolleys and the cable mechanism. The deep reinforcement learning controller training process includes: Step 1: Initialize the working environment; Step 2: Construct a deep reinforcement learning controller based on proximal policy optimization, which includes an action network and a value network; Step 3: The crane controller continuously interacts with the working environment to collect experience datasets; Step 4: Train the action network and update its parameters; Step 5: Train the value network and update its parameters; Step 6: Repeat steps 3-5 until the parameter changes of the action network and value network are less than a certain threshold, or the cumulative reward no longer increases significantly. Stop training and save the latest action network and value network parameters. In step 1, the initialization of the working environment is based on the initial position of the trolley target in the crane's working environment. target location of the large vehicle Car position Car speed Large vehicle position large vehicle speed Swing angle angular velocity of the pendulum Cable length The weight of the suspended object, m, is used as status information of the working environment and is denoted as... ; In step 2, the crane controller and the working environment follow the current strategy. Interact with the system to generate a series of experiential data. ; in, yes Real-time status information of the work environment In the current state The lower crane controller makes an action. New status information generated by the working environment; ,in, for The speed of the car motor is constantly changing. for The speed of the main vehicle motor is constantly monitored. for Constant cable motor speed; reward yes The reward obtained is calculated using the reward function at any time; The crane controller continuously interacts with the working environment until a sufficient amount of empirical data is collected to form an empirical dataset. .

2. The crane anti-sway positioning control method based on near-end strategy optimization according to claim 1, characterized in that, The reward Including location rewards and sway angle reward The total reward is the sum of the position reward and the angle reward. ; The location reward is given when the crane reaches the destination position, based on the error between the destination position and the target position, as follows: ； in The target location of the suspended object. , for The location of the car at any time for The location of the large vehicle at any given time. This indicates a search for Euclidean distance. To achieve the required positional accuracy, , ; The swing angle reward is given according to the size of the swing angle, as shown in the following formula: ； In the formula, for The angle of the suspended object at any given moment. To achieve the required swing angle accuracy, , .

3. The crane anti-sway positioning control method based on near-end strategy optimization according to claim 1, characterized in that, Step 4 specifically includes the following steps: Step 4.1: From the empirical dataset Random sampling N One set of empirical data; Step 4.2, through value networks To assess the current state The value below ; Step 4.3: Calculate the estimated value of the advantage function. : ； in, Discount factor; Value estimation for the next state; Step 4.4: Calculate the loss of the action network. : ； in, It is the new and old action networks at all times The probability ratio of taking an action. These are the motion network parameters before the update. For clipping functions, These are trimming parameters used to limit the difference between new and old actions; The loss function is used to measure the difference between the current strategy and the strategy that maximizes cumulative rewards; Step 4.5: Using the Adam optimization algorithm, update the parameters of the action network based on the loss function to optimize the action network in a direction that can obtain greater cumulative rewards.

4. The crane anti-sway positioning control method based on near-end strategy optimization according to claim 3, characterized in that, Step 5 specifically includes the following steps: Step 5.1: Based on the rewards in the sampled data And the value estimation of the next state Calculate the target value ; Step 5.2: Calculate the loss of the value network. : ； Step 5.3: Using the Adam optimization algorithm, update the parameters of the value network based on the loss function so that the value network can more accurately evaluate the state value.

Images