A deep reinforcement learning driven crane trajectory planning method
By employing a deep reinforcement learning-driven crane trajectory planning method, a crane operation state space and motion space are constructed. A dense reward function is designed, and the Actor-Critic network and HER experience playback strategy are improved. This solves the problem of traditional cranes relying on driver experience, realizes autonomous loading and unloading operations of cranes, and enhances safety and intelligence levels.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANDONG UNIV
- Filing Date
- 2026-05-19
- Publication Date
- 2026-06-16
AI Technical Summary
Traditional crane control relies on the operator's experience, making it difficult to guarantee safety. In particular, it lacks autonomous trajectory planning capabilities in complex, changeable, or high-risk operating environments, making it difficult to achieve intelligent operation.
A deep reinforcement learning-driven crane trajectory planning method is adopted to construct the crane's operation state space and action space, design a dense reward function, and improve the Actor-Critic network architecture and HER experience playback strategy to achieve autonomous loading and unloading operations.
Reducing human intervention improves the real-time performance and adaptability of crane operations, thus promoting the intelligentization of crane operations.
Smart Images
Figure CN122221699A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of crane trajectory planning, specifically involving a deep reinforcement learning-driven crane trajectory planning method. Background Technology
[0002] Cranes are widely used in cargo loading and unloading operations, offshore equipment maintenance, and other fields. Traditional crane control methods mainly rely on manual operation by the driver, which is highly dependent on the driver's experience and makes it difficult to guarantee safety. In the face of complex, changing, or high-risk operating environments, the ability to autonomously plan its trajectory is key to achieving intelligent operation. Summary of the Invention
[0003] This application proposes a deep reinforcement learning-driven crane trajectory planning method. The method designs the crane's operation state space, action space, and dense reward function; constructs an improved Actor-Critic network architecture; and refines the HER experience replay strategy to more effectively utilize failed trajectory learning. This method enables autonomous loading and unloading operations for cranes, reduces human intervention, and offers strong real-time performance and adaptability, thus promoting the intelligentization of crane operations. The technical solution is as follows: A deep reinforcement learning-driven crane trajectory planning method includes the following steps: S1. Construct a 3D model of the crane's working environment and import it into the simulation environment; S2. For the current task round, the first... t state of time step Input actor network to get the first t Time step action ; S3. Obtain t Time step action Then, by interacting with the environment in the simulation environment, one can obtain the results. t +1 time step status Based on t +1 time step status Calculate the first t Time step rewards , put in t Experience gained from time steps Stored in the experience pool; simultaneously, the improved HER experience replay method is used to process the new experiences gained. It is also stored in the experience pool; S4. Randomly extract a specified number of experiences from the experience pool to update and improve the model parameters of the actor-critic network; S5. Repeat the reinforcement learning training process from S2 to S4 continuously, and deploy the converged Actor network to the real-time control system. Generate motion trajectories in real time based on the current state.
[0004] Preferred, the first t state of time step Including the t Crane lifting device position at time step , No. t Crane lifting posture at time step Target location Target attitude , No. t Position of each moving joint of the crane in time step ; No. t Time step action , for the first t Desired position increments of each joint at each time step .
[0005] Preferred, the first t Time step rewards Includes the following parts: No. t The hoisting position at the time step is close to the reward, as shown in the following formula: ; in, For the first t The hoisting position in the time step is close to the reward. For the first t The position of the lifting device at the time step is the lifting position. The installation location is specified during installation. ; The first coefficient for the hoisting position. The second coefficient for the hoisting position. The third coefficient for the hoisting position. Assuming the hoisting position is close to the threshold, A continuity coefficient is awarded for the hoisting position; No. t The formula for the hoisting attitude constraint reward at the time step is as follows: ; in, For the first t Time-step hoisting posture constraint reward The quaternion for the current spreader attitude. For the target pose quaternion, The attitude constraint coefficient; No. t The joint trajectory smoothness penalty for the time step is calculated as follows: ; in, For the first t Time-step joint trajectory smoothness penalty For the first j The joint in the t The position increment of the time step, J The total number of joints. To smooth out the penalty coefficient; No. t The collision penalty for each time step is calculated using the following formula: ; in, For the first t Collision penalty of time step This is the collision penalty coefficient; No. t The system overload penalty at each time step is calculated using the following formula: ; in, For the first t System overload penalty at time step For the current load, For rated load, This is the overload penalty coefficient; No. t The total reward for the time step is: .
[0006] Preferably, the improved actor-critic network includes an Actor network, a Target Actor network, a multi-depth Critic network, and a multi-depth Target Critic network, with the following specific structures: The Actor network input is in t state of time step The output is in t Time step action The structure is as follows: ; in, It is the ReLU activation function. For the Tanh activation function, Indicates a fully connected layer. Represents a fully connected block; The target actor network input is in t +1 time step status The output is in t +1 time step action The network structure is the same as that of the Actor network. Multiple deep Critic networks have N Critic networks containing varying numbers of fully connected blocks, the th n The number of fully connected blocks in a Critic network is Each Critic network input is in t state of time step and in t Time step action The splicing of the data outputs the network in [the following context is missing from the original text] t time step ; The multi-depth Target Critic network structure is the same as the Critic network, with each Target Critic network taking input from... t +1 time step status and in t +1 time step action The output is the network in t +1 time step .
[0007] Preferably, the multi-depth Critic network ultimately... t time step value for n Weighted sum of network outputs: ; in, In order to be in t The first step of time n The output of the Critic network value, These are the corresponding learnable adaptive weights.
[0008] Preferably, multi-depth target critic networks are used in... t +1 time step value The weighted sum of the outputs of each network: ; in, In order to be in t +1 time step n The output of the Target Critic network value, These are the corresponding learnable adaptive weights.
[0009] Preferred approaches include improving HER experience replay strategies: Step 1: If the current round of the task fails, obtain the complete trajectory data of the current round of the task; Step 2: For a complete trajectory, divide it into fixed time intervals. There are several stage points, and the minimum time interval is required to be [number]. The formula is as follows: ; in Let be the time interval, satisfying , For the index of the stage point, The number of stage points, For the time step of the phase point, the first Each stage point marks the end of the trajectory; The path from the starting point to the first stage point is designated as the first sub-path; the path from the starting point to the second stage point is designated as the second sub-path; and so on, for a total of... The trajectory of the strip; Step 3: For each sub-trajectory, first sample it at equal intervals and retain... One data point; Among the sampled points, find a point that minimizes the sum of its distances to all other sampled points, and set this point as the new target point, as shown in the following formula: Equally spaced sampling point set: ; New target location: ; in, For any sub-trajectory, the set of points is sampled at equal intervals. The set of points after equal interval sampling. For the new target location, and All The position of any point in it, The operation is to find the value that minimizes the subsequent summation formula. .
[0010] Step 4: Based on the new target point location Actions in sub-trajectories Recalculate the state and reward at each time step in this sub-trajectory to obtain It is also stored in the experience pool, along with the previously stored experience. Together, they are used for network updates.
[0011] Preferably, this includes network parameter updates: During training, a specified batch of experiences is randomly sampled from the experience pool, and the Target Actor network input state is... Output action The Target Critic network is based on and Output value ,based on value and rewards in the experience pool Calculate the target value of the Q-value (TD target). (Critic network output) value The parameters are updated with the goal of approximating the TD target; the Actor network is based on empirical data. Output Action And use the Critic network to output value To maximize value Update parameters for the target. The parameters of the Target Actor network and Target Critic network are slowly tracked by a soft update method to track changes in the parameters of the Actor network and Critic network.
[0012] Compared with the prior art, the beneficial effects of this application are as follows: An Actor-Critic network architecture was constructed, and an improved experience replay strategy was designed to more effectively utilize failure trajectory learning. This allows for real-time strategy adjustments based on load and environmental changes, adapting to the complex and ever-changing operating conditions of ports. This has promoted the intelligentization of crane operations, reduced the impact of human factors, and demonstrated strong real-time performance and adaptability. Attached Figure Description
[0013] Figure 1 A framework for a crane trajectory planning method driven by deep reinforcement learning; Figure 2 A schematic diagram of the improved actor-critic network structure; Figure 3 A diagram illustrating the improvement of experience playback strategies. Detailed Implementation
[0014] The present invention will now be described in detail with reference to the accompanying drawings, and the technical solutions in the embodiments of the present invention will be clearly and completely described. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.
[0015] A deep reinforcement learning-driven crane trajectory planning method includes the following steps: S1. Construct a 3D model of the crane's working environment and import it into the simulation environment; S2. For the current task round, the first... t state of time step Input actor network to get the first t Time step action ; S3. Obtain t Time step action Then, by interacting with the environment in the simulation environment, one can obtain the results. t +1 time step status Based on t +1 time step status Calculate the first t Time step rewards , put in t Experience gained from time steps Stored in the experience pool; simultaneously, the improved HER experience replay method is used to process the new experiences gained. It is also stored in the experience pool; S4. Randomly extract a specified number of experiences from the experience pool to update and improve the model parameters of the actor-critic network; S5. Repeat the reinforcement learning training process from S2 to S4 continuously, and deploy the converged Actor network to the real-time control system. Generate motion trajectories in real time based on the current state.
[0016] Among them, the t state of time step , including the t Crane lifting device position at time step , No. t Crane lifting posture at time step Target location (Lifting position during hoisting) The installation location is specified during installation. ), target attitude , No. t Position of each moving joint of the crane in time step wait.
[0017] Among them, the t Time step action , for the first t Desired position increments of each joint at each time step .
[0018] Among them, the t The reward and penalty functions for each time step include the following components: No. t The hoisting position at the time step is close to the reward, as shown in the following formula: ; in, For the first t The hoisting position in the time step is close to the reward. For the first tThe position of the lifting device at the time step. The target location (the lifting position during lifting) The installation location is specified during installation. ), The first coefficient for the hoisting position. The second coefficient for the hoisting position. The third coefficient for the hoisting position. Assuming the hoisting position is close to the threshold, A continuity coefficient is awarded for the hoisting position.
[0019] No. t The formula for the hoisting attitude constraint reward at the time step is as follows: ; in, For the first t Time-step hoisting posture constraint reward The quaternion for the current spreader attitude. For the target pose quaternion, This represents the attitude constraint coefficient.
[0020] No. t The joint trajectory smoothness penalty for the time step is calculated as follows: ; in, For the first t Time-step joint trajectory smoothness penalty For the first j The joint in the t The position increment of the time step, J The total number of joints. To smooth out the penalty coefficient.
[0021] No. t The collision penalty for each time step is calculated using the following formula: ; in, For the first t Collision penalty of time step This represents the collision penalty coefficient.
[0022] No. t The system overload penalty at each time step is calculated using the following formula: ; in, For the first t System overload penalty at time step For the current load, For rated load, This is the overload penalty coefficient.
[0023] No. t The total reward for the time step is: ; The improved actor-critic network includes constructing an Actor network, a Target Actor network, a multi-depth Critic network, and a multi-depth Target Critic network. Actor Network: The Actor network input is in t state of time step The output is in t Time step action The structure is as follows: ; in, It is the ReLU activation function. For the Tanh activation function, This indicates a fully connected layer. This represents a fully connected block.
[0024] Target Actor Network: The structure is the same as the Actor network, and the input is... t +1 time step status The output is in t +1 time step action .
[0025] Multiple deep Critic networks: have N Critic networks containing varying numbers of fully connected blocks, the th n The number of fully connected blocks in a Critic network is Each Critic network input is in t state of time step and in t Time step action The splicing of the data outputs the network in [the following context is missing from the original text] t time step value .
[0026] Multiple deep Critic networks ultimately in t time step value for n Weighted sum of network outputs: ; in, In order to be in tThe first step of time n The output of the Critic network value, These are the corresponding learnable adaptive weights.
[0027] Multiple deep target critic networks: The structure is the same as the Critic network, with each Target Critic network input being... t +1 time step status and in t +1 time step action The output is the network in t +1 time step value .
[0028] Finally in t +1 time step value The weighted sum of the outputs of each network: ; in, In order to be in t +1 time step n The output of the Target Critic network value, Consistent with the Critic network.
[0029] Improving HER experience replay includes the following steps: Step 1: If the current round of the task fails, obtain the complete trajectory data of the current round of the task; Step 2: For a complete trajectory, divide it into fixed time intervals. K There are several stage points, and the minimum time interval is required to be [number]. The formula is as follows: ; in Let be the time interval, satisfying , For the index of the stage point, The number of stage points, For the time step of the phase point, the first Each stage point marks the end of the trajectory.
[0030] The path from the starting point to the first stage point is designated as the first sub-path; the path from the starting point to the second stage point is designated as the second sub-path; and so on, for a total of... The trajectory of the strip.
[0031] Step 3: For each sub-trajectory, first sample it at equal intervals and retain... U There are 10 data points. Among the sampled points, find a point that minimizes the sum of its distances to all other sampled points, and set this point as the new target point. The formula is as follows: Equally spaced sampling point set: ; New target location: ; in, For any sub-trajectory, the set of points is sampled at equal intervals. The set of points after equal interval sampling. For the new target location, and All The position of any point in it, The operation is to find the value that minimizes the subsequent summation formula. .
[0032] Step 4: Based on the new target point location Actions in sub-trajectories Recalculate the state and reward at each time step in this sub-trajectory to obtain It is also stored in the experience pool, along with the previously stored experience. Together, they are used for network updates.
[0033] The network parameter update method is as follows: During training, a specified batch of experiences is randomly sampled from the experience pool, and the Target Actor network input state is... Output action The Target Critic network is based on and Output value ,based on value and rewards in the experience pool Calculate the target value of the Q-value (TD target). (Critic network output) value The parameters are updated with the goal of approximating the TD target; the Actor network is based on empirical data. Output Action And use the Critic network to output value To maximize value Update parameters for the target. The parameters of the Target Actor network and Target Critic network are slowly tracked by a soft update method to track changes in the parameters of the Actor network and Critic network.
Claims
1. A deep reinforcement learning-driven crane trajectory planning method, characterized in that, Includes the following steps: S1. Construct a 3D model of the crane's working environment and import it into the simulation environment; S2. For the current task round, the first... t state of time step Input actor network to get the first t Time step action ; S3. Obtain t Time step action Then, by interacting with the environment in the simulation environment, one can obtain the results. t +1 time step status Based on t +1 time step status Calculate the first t Time step rewards , put in t Experience gained from time steps Store in the experience pool; At the same time, the improved HER experience playback method was used to extract new experiences. It is also stored in the experience pool; S4. Randomly extract a specified number of experiences from the experience pool to update and improve the model parameters of the actor-critic network; S5. Repeat the reinforcement learning training process from S2 to S4 continuously, and deploy the converged Actor network to the real-time control system to generate motion trajectories in real time based on the current state.
2. The deep reinforcement learning-driven crane trajectory planning method according to claim 1, characterized in that, No. t state of time step Including the t Crane lifting device position at time step , No. t Crane lifting posture at time step Target location Target attitude , No. t Position of each moving joint of the crane in time step ; No. t Time step action , for the first t Desired position increments of each joint at each time step .
3. The deep reinforcement learning-driven crane trajectory planning method according to claim 1, characterized in that, No. t Time step rewards Includes the following parts: No. t The hoisting position at the time step is close to the reward, as shown in the following formula: ； in, For the first t The hoisting position in the time step is close to the reward. For the first t The position of the lifting device at the time step is the lifting position. The installation location is specified during installation. ; The first coefficient for the hoisting position. The second coefficient for the hoisting position. The third coefficient for the hoisting position. As the hoisting position approaches the threshold, A continuity coefficient is awarded for the hoisting position; No. t The formula for the hoisting attitude constraint reward at the time step is as follows: ； in, For the first t Time-step hoisting attitude constraint reward The quaternion for the current spreader attitude. For the target pose quaternion, The attitude constraint coefficient; No. t The joint trajectory smoothness penalty for the time step is calculated as follows: ； in, For the first t Time-step joint trajectory smoothness penalty For the first j The joint in the t The position increment of the time step, J The total number of joints. To smooth out the penalty coefficient; No. t The collision penalty for each time step is calculated using the following formula: ； in, For the first t Collision penalty of time step This is the collision penalty coefficient; No. t The system overload penalty at each time step is calculated using the following formula: ； in, For the first t System overload penalty at time step For the current load, For rated load, This is the overload penalty coefficient; No. t The total reward for the time step is: 。 4. The deep reinforcement learning-driven crane trajectory planning method according to claim 1, characterized in that, The improved actor-critic network includes an Actor network, a Target Actor network, a multi-depth Critic network, and a multi-depth Target Critic network, with the following specific structures: The Actor network input is in t state of time step The output is in t Time step action The structure is as follows: ； in, It is the ReLU activation function. For the Tanh activation function, Indicates a fully connected layer. Represents a fully connected block; The target actor network input is in t +1 time step status The output is in t +1 time step action The network structure is the same as that of the Actor network. Multiple deep Critic networks have N Critic networks containing varying numbers of fully connected blocks, the th n The number of fully connected blocks in a Critic network is Each Critic network input is in t state of time step and in t Time step action The splicing of the data outputs the network in [the following context is missing from the original text] t time step ; The multi-depth Target Critic network structure is the same as the Critic network, with each Target Critic network taking input from... t +1 time step status and in t +1 time step action The output is the network's performance. t +1 time step .
5. The deep reinforcement learning-driven crane trajectory planning method according to claim 4, characterized in that, Multiple deep Critic networks ultimately in t time step value for n Weighted sum of network outputs: ； in, In order to be in t The first step of time n The output of the Critic network value, These are the corresponding learnable adaptive weights.
6. The deep reinforcement learning-driven crane trajectory planning method according to claim 5, characterized in that, Multiple deep target critic networks in t +1 time step value The weighted sum of the outputs of each network: ； in, In order to be in t +1 time step n The output of the Target Critic network value, These are the corresponding learnable adaptive weights.
7. The deep reinforcement learning-driven crane trajectory planning method according to claim 5, characterized in that, This includes improving the HER experience replay strategy: Step 1: If the current round of the task fails, obtain the complete trajectory data of the current round of the task; Step 2: For a complete trajectory, divide it into fixed time intervals. There are several stage points, and the minimum time interval is required to be [number]. The formula is as follows: ； in Let be the time interval, satisfying , For the index of the stage point, The number of stage points, For the time step of the phase point, the first Each stage point marks the end of the trajectory; The path from the starting point to the first stage point is designated as the first sub-path; the path from the starting point to the second stage point is designated as the second sub-path; and so on, for a total of... The trajectory of the strip; Step 3: For each sub-trajectory, first sample it at equal intervals and retain... One data point; Among the sampled points, find a point that minimizes the sum of its distances to all other sampled points, and set this point as the new target point, as shown in the following formula: Equally spaced sampling point set: ; New target location: ; in, For any sub-trajectory, the set of points is sampled at equal intervals. The set of points after equal interval sampling. For the new target location, and All The position of any point in it, The operation is to find the value that minimizes the subsequent summation formula. ; Step 4: Based on the new target point location Actions in sub-trajectories Recalculate the state and reward at each time step in this sub-trajectory to obtain It is also stored in the experience pool, along with the previously stored experience. Together, they are used for network updates.
8. The deep reinforcement learning-driven crane trajectory planning method according to claim 5, characterized in that, Including network parameter updates: During training, a specified batch of experiences is randomly sampled from the experience pool, and the Target Actor network input state is... Output action The Target Critic network is based on and Output value ,based on value and rewards in the experience pool The target value for calculating the Q-value; the output of the Critic network. value The parameters are updated with the goal of approximating the TD target; the Actor network is based on empirical data. Output Action and output using the Critic network. value To maximize value To update the parameters for the target, the parameters of the Target Actor network and TargetCritic network are slowly tracked by a soft update method.