PDPG-based ship continuous steering intelligent collision avoidance decision-making system
By using the main Actor, target Actor, main Critic and target Critic networks of the PDPG model in the ship's intelligent collision avoidance system, the problem of difficulty in quickly avoiding obstacles under high risk situations in the prior art is solved, and safe and efficient ship navigation is achieved.
Patent Information
- Application Number
- CN202510048942.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-13
- Publication Date
- 2025-04-25
AI Technical Summary
The prior art ignores safety in pursuit of the shortest path or the fastest speed, making it difficult to quickly avoid obstacles under high risk situations, especially in complex or uncertain marine environments.
A PDPG-based ship continuous steering intelligent collision avoidance decision-making system includes a main Actor network, a target Actor network, a main Critic network and a target Critic network. Through the combination and weight adjustment of these networks, the system can adaptively balance the exploration and utilization of strategies in different environments, ensuring the optimal security and benefits of collision avoidance strategies.
It realizes the ability to quickly avoid obstacles under high risk situations, while ensuring the safety and efficiency of ship navigation and adapting to the ever-changing marine environment.
Smart Images
Figure CN119882743A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of ship automatic control, and in particular relates to a ship continuous steering intelligent collision avoidance decision system based on PDPG. Background Art
[0002] The terrain near the island cape is complex, and the geographical features form curved channels, such as the Luotou Channel in Zhoushan. The water flow conditions at the bend are complex, and it is easy to capsize and run aground, which brings challenges to ship navigation. Curved channels include single bends, continuous bends, and forked bends, which require continuous steering of the ship. Curved channels are usually narrower than straight channels, which means that the ship's maneuvering space in the channel is limited. Ships must ensure safe passage in narrow channels to avoid collisions with other ships or touching the shore.
[0003] In the existing technology, Actor-Critic neural network is used for collision avoidance. However, the evaluation index of the Critic network in the existing network is relatively simple and mainly focuses on benefits. It is difficult to achieve an adaptive balance between exploration and utilization in an uncertain marine environment. It is easy to ignore safety when pursuing the shortest path or the fastest speed, and it is difficult to quickly avoid obstacles in high-risk situations. Summary of the invention
[0004] The purpose of the present invention is to provide a PDPG-based ship continuous steering intelligent collision avoidance decision system to solve the technical problems in the prior art of ignoring safety when pursuing the shortest path or the fastest speed and making it difficult to quickly avoid obstacles in high-risk situations.
[0005] The PDPG-based ship continuous steering intelligent collision avoidance decision system includes a PDPG model and a simulation environment. The PDPG model includes a main Actor network, a target Actor network, a main Critic network and a target Critic network. The main Actor network is used to generate the main steering decision to consider the current optimal collision avoidance strategy; the target Actor network is used to generate different emergency steering actions in the exploration phase to try to cope with complex or uncertain collision avoidance situations; the main Critic network updates the network parameters through back propagation and evaluates the overall benefit of the current state and the current action; the target critic network updates its own network parameters at a slower rate to evaluate the safety of the current steering action, trains the PDPG model based on the simulation environment, adjusts the adaptive weights of the main Actor network and the target Actor network according to the simulation results, and fine-tunes the network parameters of the main Critic network and the target Critic network to ensure that the safety and benefits of the ship collision avoidance strategy in different environments are optimized.
[0006] Preferably, during the training of the PDPG model, the PDPG model is first initialized and the corresponding network parameters are randomized; the current state s of the ship is obtained from the simulation environment, and the main Actor network and the target Actor network generate actions a1 and a2 respectively according to the current state; the main Actor network adopts a four-layer neural network to extract the most effective collision avoidance information from the complex state input, and the output layer of the main Actor network uses a hyperbolic tangent activation function to generate a continuous steering angle in the range of [-30°, 30°]; the structure of the target Actor network adopts a three-layer neural network, and the output layer of the target Actor network also uses a hyperbolic tangent activation function to output the steering angle range. Compared with the main Actor network, the steering action generated by the target Actor network is more conservative, and the output strategy tends to reduce the deviation from the current navigation trajectory.
[0007] Preferably, after the PDPG model generates actions a1 and a2, the actions generated by the two Actor networks are combined according to a certain weight, and the formula is: a=w1a1+w2a2, w1 and w2 are weights, a is the action actually output by the PDPG model, and the weights w1 and w2 are dynamically adjusted based on the complexity of the environment; in the simulation environment, the combined action a is executed, the navigation state of the ship is updated, and the next state s' is obtained.
[0008] Preferably, both the main critic network and the target critic network adopt a four-layer deep neural network, and the network parameters of the main critic network and the target critic network are updated by minimizing the loss function; among them, the network parameters of the main critic network are updated at a higher frequency, and it can respond to environmental changes more quickly; the network parameters of the target critic network are updated more slowly, and are used to generate a more stable strategy.
[0009] Preferably, the main critic network is used to evaluate the direct Q value based on the current state-action combination to measure the overall benefit of the current strategy; the target critic network is used to evaluate the safety index of the action, and then the evaluation index is fused to obtain a more comprehensive value evaluation through weighted summation or conservative minimum Q value method; then, through policy gradient, the feedback of the main critic network and the target critic network is used to update the network parameters of the main actor network and the target actor network.
[0010] Preferably, a state space and an action space are defined in the simulation environment. The state space includes the speed and heading of the ship and the distance between the ship and the obstacle. The current state includes the position, speed and heading information of the ship. The state space is defined as:
[0011] State=(S o ,S t ),
[0012]
[0013]
[0014] Among them, ex o ∈{0,1},ex t ∈{0,1},-l<Δx to <L,t≤5;S o This is our ship, S t are the target ships around, and ex is a variable about the ship. 1 means the ship exists, and 0 means it does not exist. x represents the lateral position of the ship, and y represents the longitudinal position of the ship. Indicates the lateral speed of own ship. Indicates the longitudinal speed of the ship; δ o represents the rudder angle of the ship, Δ represents the relative value between the own ship and the target ship, l represents the length of the ship, L represents the maximum detection distance, and t represents the number of ships detected nearby; the lateral and longitudinal movements are controlled by steering and speed; the action space includes continuous steering angle adjustment, and the action includes the acceleration a0 of the ship and the rudder angle δ. The action space is defined as: Action=(a0,δ),a0∈[-3,3]m / s 2 ,δ∈(-45°,45°).
[0015] Preferably, the reward is the benefit, which is used to evaluate the quality of the ship agent's execution of actions. The evaluation of rewards in this project mainly includes safety rewards R collision , effectiveness reward R effic , Comfort Reward R com and rule reward R rule ; Each reward is defined as:
[0016] R=R collision +R effic +R com +R rule ,
[0017]
[0018] R com =k2,
[0019] R rule =k3,
[0020] Among them, v, v min and v maxis the current speed, minimum speed and maximum speed of the ship agent. k1, k2 and k3 are constants. During training, rewards are calculated based on the collision avoidance effect. Positive rewards include: safe collision avoidance, speed and steering angle within the specified range, and reaching the target position; while negative rewards include: approaching obstacles, collision risk, and deviation from the route.
[0021] Preferably, after each training interaction, the experience data (s, a, r, s') generated by the current interaction is stored in the experience replay pool, s is the current state, a is the action, r is the reward, and s' is the next state; the ship continuous steering intelligent collision avoidance decision-making system randomly samples batch experience data (s, a, r, s') from the experience replay pool in a key experience priority manner for training the dual Actor and dual Critic networks.
[0022] The advantages of the present invention are as follows: in the present invention, the two Actor networks have different functional divisions, the main Actor network is used to generate the main collision avoidance strategy, and the target Actor network is used for collision avoidance strategies in special situations such as multi-ship encounters and complex dynamic environments. The dual Actors in the improved structure explore and utilize the dynamic balance strategy of adaptive exploration rate, so that the ship has both safety and conservatism when performing actions, and flexibility, and adapts to the ever-changing marine environment.
[0023] After the improvement of this scheme, the dual critic network not only independently evaluates the Q value of the same state-action pair, but also evaluates different value indicators separately. This improvement allows the network to not only consider the maximization of benefits, but also adds security constraints to the value evaluation, so as to balance the benefits and risks in the strategy generation process. The improved dual critic network integrates the evaluation indicators during training, and obtains a more comprehensive value evaluation by weighting or conservatively taking the minimum Q value. This can effectively avoid the biased decision that may be caused by a single Q value, ensuring that the collision avoidance strategy has both high benefits and minimizes the risk of collision.
[0024] In addition, by adaptively adjusting the weights between the dual actors and dual critics, the direction of policy learning can be flexibly controlled, allowing the network to converge to the optimal safe collision avoidance strategy more quickly. The dual critic network provides a more stable evaluation, avoiding unstable policy updates due to fluctuations in value evaluation, thereby improving the robustness of training and policy execution. The target critic's feedback on safety evaluation can help ships avoid local optimal solutions and prompt the strategy to explore and optimize globally. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 It is a flow chart of the ship continuous steering intelligent collision avoidance decision system based on PDPG of the present invention. DETAILED DESCRIPTION
[0026] The specific implementation modes of the present invention are further explained in detail below by describing the embodiments with reference to the accompanying drawings, so as to help those skilled in the art to have a more complete, accurate and in-depth understanding of the inventive concept and technical solution of the present invention.
[0027] like Figure 1 As shown, the present invention provides a ship continuous steering intelligent collision avoidance decision system based on PDPG, including a PDPG (path derived policy gradient algorithm) model and a simulation environment, wherein the PDPG model includes a main Actor network, a target Actor network, a main Critic network and a target Critic network, wherein the main Actor network is used to generate the main steering decision to consider the current optimal collision avoidance strategy; the target Actor network is used to generate different emergency steering actions in the exploration phase to try to cope with complex or uncertain collision avoidance situations; the main Critic network frequently updates network parameters through back propagation to minimize the TD error (the difference between the target value and the predicted value) and evaluate the overall benefit of the current state and the current action; the target critic network updates its own network parameters at a slower rate to evaluate the safety of the current steering action, and the evaluation content includes the shortest distance between the ship itself and the obstacle and the steering smoothness. The PDPG model is trained based on the simulation environment, and the adaptive weights of the main Actor network and the target Actor network are adjusted according to the simulation results, and the network parameters of the main Critic network and the target Critic network are fine-tuned to ensure that the safety and benefits of the ship collision avoidance strategy in different environments are optimized.
[0028] The simulation environment includes the initial position of the ship, the position of dynamic obstacles, the speed of the ship, and the ocean meteorological conditions. The state space and action space are defined in the simulation environment. The state space includes the ship's speed, heading, and the distance between the ship and the obstacle, and the action space includes continuous steering angle adjustment.
[0029] The current state s is the observation of the ship agent on the continuous steering collision avoidance simulation environment, and is the basis for intelligent collision avoidance of the ship steering control. In this work, the current state includes the position, speed and heading information of the ship. The state space is defined as:
[0030] State=(S o ,S t ),
[0031]
[0032] Among them, ex o ∈{0,1},ex t ∈{0,1},-l<Δx to <L,t≤5.o This is our ship, S t are the target ships around, and ex is a variable about the ship. 1 means the ship exists, and 0 means it does not exist. x represents the lateral position of the ship, and y represents the longitudinal position of the ship. v xo Indicates the lateral speed of the ship, v yo Indicates the longitudinal speed of the own ship. o Indicates the rudder angle of the ship. Δ indicates the relative value of the own ship and the target ship. l represents the length of the ship, and L represents the maximum detection distance. t represents the number of ships detected nearby. Lateral and longitudinal movement can be controlled by steering and speed.
[0033] Action refers to the decision taken by the ship to complete the steering control collision avoidance task. In the research work of this project, the action includes the acceleration a and the rudder angle δ of the ship, and the action space is defined as:
[0034] Action=(a,δ),a∈[-3,3]m / s 2 ,δ∈(-45°,45°).
[0035] The PDPG model makes a trade-off between exploration and exploitation during training. Exploration refers to trying new unknown actions to discover better strategies, while exploitation refers to selecting the currently considered optimal action based on existing experience. Introducing random strategies in the Actor network can increase the exploration behavior of the ship agent, thereby making it more likely to discover the optimal strategy and avoid falling into the local optimal solution too early. The introduction of random strategies can increase the stability of the algorithm and make the gradient descent process smoother, thereby improving the convergence and stability of the algorithm. Random strategies include categorical policies and diagonal Gaussian policies. Categorical policies are often used in discrete action spaces, while diagonal Gaussian policies are used in continuous action spaces. a=π θ (s) (i.e., steering angle adjustment value), during the exploration process, noise (diagonal Gaussian) is added to balance exploration and exploitation. The main actor network and the target actor network select actions and continuously update strategies. The target actor network has a lower update frequency and strategy complexity than the main actor network when updating strategies. The entire actor network reduces its dependence on a single actor when updating strategies, which helps to avoid too fast changes in strategies due to short-term feedback, and is conducive to more stable improvement of overall performance.
[0036] Rewards are benefits, which are used to evaluate the quality of the actions performed by the ship agent. The evaluation of the actions taken in a certain state takes into account four aspects. Safety refers to avoiding collisions with other ships, effectiveness refers to driving the ship within a specific speed range, comfort refers to turning the ship's rudder angle within a specific range, and rules refer to sailing on the center line of the ship's channel. The evaluation of rewards in this project mainly includes safety rewards R collision , effectiveness reward R effic , Comfort Reward R com and rule reward R rule . Each reward is defined as:
[0037] R=R collision +R effic +R com +R rule ,
[0038]
[0039] R com =k2,
[0040] R rule =k3,
[0041] Among them, v, v min and v max is the current speed, minimum speed and maximum speed of the ship agent. k1, k2 and k3 are constants. During training, rewards are calculated based on the collision avoidance effect. Positive rewards include: safe collision avoidance, speed and steering angle within the specified range, and reaching the target position; while negative rewards include: approaching obstacles, collision risk, and deviation from the route. After each training interaction, the experience data (s, a, r, s') generated by the current interaction is stored in the experience replay pool, where s is the current state, a is the action, r is the reward, and s' is the next state.
[0042] In this scheme, the critic network (including the main critic network and the target critic network) updates (Q value evaluation) and uses the target network to calculate the target Q value as follows; Where γ is the discount factor, and the Critic network updates the Critic network parameters by minimizing the loss function. The formula for updating (strategy optimization) the Actor network (including the main Actor network and the target Actor network) is: It uses path derivative policy gradient to optimize the Actor network and guides policy improvement through the Critic network. The algorithms for updating the target network parameters include: This algorithm can improve stability.
[0043] During training, the PDPG model is initialized first, and the corresponding network parameters are randomized; the current state s of the ship is obtained from the simulation environment: the current state s includes the current position, speed, and relative position and speed of the ship to the obstacle. The main actor network and the target actor network generate actions a1 and a2 respectively according to the current state. The main actor network has multiple hidden layers to handle the complex ship navigation environment. Specifically, the main actor network uses a four-layer neural network to ensure that the most effective collision avoidance information is extracted from the complex state input. The output layer of the main actor network uses a hyperbolic tangent activation function to generate a continuous steering angle in the range of [-30°, 30°]. The main actor network directly outputs the steering angle in the current state to control the real-time collision avoidance of the ship. On the other hand, the structure of the target actor network is slightly simplified compared to the main actor network. A three-layer neural network is used to reduce the update frequency during training and reduce the complexity of the strategy, making it smoother when generating strategies. The output layer of the target actor network also uses a hyperbolic tangent activation function to output the steering angle range. Compared with the main Actor network, the steering actions generated by the target Actor network are more conservative, and the output strategy usually tends to reduce the deviation from the current navigation trajectory to avoid excessive steering actions.
[0044] After the PDPG model generates actions a1 and a2, the actions generated by the two Actor networks are combined according to a certain weight, and the formula is a=w1a1+w2a2, where w1 and w2 are weights, a is the action actually output by the PDPG model, and the weights w1 and w2 are dynamically adjusted based on the complexity of the environment to achieve more flexible decision-making. In the simulation environment, the combined action a is executed, the navigation state of the ship is updated, and the next state s' is obtained. Among the above two actions, the action generated by the main Actor network is the main collision avoidance strategy, that is, it provides the optimal turning action during navigation; while the action generated by the target Actor network is applied in special circumstances, such as multi-ship encounters, complex dynamic environments, etc. In addition, the dual Actor network composed of the main Actor network and the target Actor network can adaptively balance the exploration and utilization of collision avoidance strategies. Let the main Actor network generate actions in a way that is more inclined to utilize existing strategies to ensure that the ship is in a safe state most of the time; let the target Actor network have higher exploratory nature, especially in uncertain states, to try different turning decisions, thereby enhancing the flexibility of the overall strategy. This dual strategy network enables the ship to be both safe and conservative while also being flexible when performing actions, allowing it to better adapt to the ever-changing marine environment.
[0045] Then, the main critic network and the target critic network evaluate action a. Both the main critic network and the target critic network use four-layer deep neural networks, and the network parameters of the main critic network and the target critic network are updated by minimizing the loss function. Among them, the network parameters of the main critic network are updated more frequently, which can respond to environmental changes more quickly and is suitable for handling dynamic collision avoidance scenarios; the network parameters of the target critic network are updated more slowly and are used to generate a more stable strategy, which can help the model avoid unstable ship movement caused by too frequent strategy changes in ship collision avoidance. This scheme updates the network parameters of the main critic network and the target critic network by minimizing the loss function.
[0046] The dual critic network composed of the main critic network and the target critic network not only independently evaluates the Q value of the same state-action pair of the Actor-Critic neural network, but can evaluate different value indicators separately. The main critic network is used to evaluate the direct Q value based on the current state-action combination to measure the overall benefit of the current strategy; while the target critic network is used to evaluate the safety indicators of the action, such as the shortest distance between the ship and the obstacle, the smoothness of the turn, etc., so that the network not only considers the maximization of benefits, but also adds safety constraints to the value evaluation. After that, the evaluation indicators are fused, and a more comprehensive value evaluation is obtained by weighted summation or conservatively taking the minimum Q value. This can effectively avoid the biased decision that may be caused by a single Q value, ensure that the collision avoidance strategy has both high benefits and minimizes the risk of collision, so as to balance the benefits and risks in the strategy generation process. After that, the feedback of the main critic network and the target critic network is used to update the network parameters of the main actor network and the target actor network through the policy gradient.
[0047] The dual critic network provides feedback on the strategy in multiple dimensions, and adjusts the direction of the Actor network strategy generation based on the feedback of the evaluation results. Through the dual feedback mechanism of the dual critic network, the two Actor networks can better learn strategies that are both adaptable to the environment and avoid risks as much as possible when generating new strategies.
[0048] In the above training process, batch experience data (s, a, r, s') are randomly sampled from the experience replay pool in a key experience priority manner for training the dual actor and dual critic networks. This can reduce the problems caused by data correlation. During training, the parameters of the target actor and target critic are updated with a smaller step size, so that the target network remains stable during the training process. Key experience priority further improves the efficiency of experience utilization, allowing the model to prioritize learning samples that contribute more to strategy improvement.
[0049] The present invention is described above by way of example in conjunction with the accompanying drawings. It is obvious that the specific implementation of the present invention is not limited to the above-mentioned method. As long as various non-substantial improvements are made using the inventive concept and technical solution of the present invention, or the inventive concept and technical solution are directly applied to other occasions without improvement, they are all within the protection scope of the present invention.
Claims
1. The ship continuous steering intelligent collision avoidance decision system based on PDPG is characterized by: It includes a PDPG model and a simulation environment. The PDPG model includes a main Actor network, a target Actor network, a main Critic network and a target Critic network. The main Actor network is used to generate the main steering decision to consider the current optimal collision avoidance strategy; the target Actor network is used to generate different emergency steering actions in the exploration phase to try to deal with complex or uncertain collision avoidance situations; the main Critic network updates the network parameters through back propagation and evaluates the overall benefit of the current state and the current action; the target critic network updates its own network parameters at a slower rate to evaluate the safety of the current steering action. The PDPG model is trained based on the simulation environment, and the adaptive weights of the main Actor network and the target Actor network are adjusted according to the simulation results. At the same time, the network parameters of the main Critic network and the target Critic network are fine-tuned to ensure that the safety and benefit of the ship collision avoidance strategy in different environments are optimized.
2. The PDPG-based ship continuous steering intelligent collision avoidance decision system according to claim 1 is characterized by: During the training of the PDPG model, the PDPG model is first initialized and the corresponding network parameters are randomized; the current state s of the ship is obtained from the simulation environment, and the main Actor network and the target Actor network generate actions a1 and a2 respectively according to the current state; the main Actor network adopts a four-layer neural network to extract the most effective collision avoidance information from the complex state input, and the output layer of the main Actor network uses a hyperbolic tangent activation function to generate a continuous steering angle in the range of [-30°, 30°]; the structure of the target Actor network adopts a three-layer neural network, and the output layer of the target Actor network also uses a hyperbolic tangent activation function to output the steering angle range. Compared with the main Actor network, the steering action generated by the target Actor network is more conservative, and the output strategy tends to reduce the deviation from the current navigation trajectory.
3. The PDPG-based ship continuous steering intelligent collision avoidance decision system according to claim 2 is characterized by: After the PDPG model generates actions a1 and a2, the actions generated by the two Actor networks are combined according to a certain weight. The formula is: a=w1a1+w2a2, w1 and w2 are weights, a is the action actually output by the PDPG model, and the weights w1 and w2 are dynamically adjusted based on the complexity of the environment; in the simulation environment, the combined action a is executed, the navigation status of the ship is updated, and the next state s' is obtained.
4. The PDPG-based ship continuous steering intelligent collision avoidance decision system according to claim 3 is characterized by: Both the main critic network and the target critic network use four-layer deep neural networks, and the network parameters of the main critic network and the target critic network are updated by minimizing the loss function. Among them, the network parameters of the main critic network are updated more frequently, and it can respond to environmental changes more quickly; the network parameters of the target critic network are updated more slowly, which is used to generate a more stable strategy.
5. The PDPG-based ship continuous steering intelligent collision avoidance decision system according to claim 4 is characterized by: The main critic network is used to evaluate the direct Q value based on the current state-action combination to measure the overall benefit of the current strategy; the target critic network is used to evaluate the safety index of the action. The evaluation indicators are then fused to obtain a more comprehensive value assessment through weighted summation or conservative minimum Q value method; then, through policy gradient, the feedback of the main critic network and the target critic network is used to update the network parameters of the main actor network and the target actor network.
6. The PDPG-based ship continuous steering intelligent collision avoidance decision system according to claim 1 is characterized by: The state space and action space are defined in the simulation environment. The state space includes the ship's speed, heading, and the distance between the ship and the obstacle. The current state includes the ship's position, speed, and heading information. The state space is defined as: State=(S o ,S t ), Among them, ex o ∈{0,1},ex t ∈{0,1},-l<Δx to <L,t≤5;S o This is our ship, S t are the target ships around, and ex is a variable about the ship. 1 means the ship exists, and 0 means it does not exist. x represents the lateral position of the ship, and y represents the longitudinal position of the ship. Indicates the lateral speed of own ship. Indicates the longitudinal speed of the ship; δ o represents the rudder angle of the ship, Δ represents the relative value between the own ship and the target ship, l represents the length of the ship, L represents the maximum detection distance, and t represents the number of ships detected nearby; the lateral and longitudinal movements are controlled by steering and speed; the action space includes continuous steering angle adjustment, and the action includes the acceleration a0 of the ship and the rudder angle δ. The action space is defined as: Action=(a0,δ),a0∈[-3,3]m / s 2 ,δ∈(-45°,45°).
7. The PDPG-based ship continuous steering intelligent collision avoidance decision system according to claim 6 is characterized by: Rewards are benefits, which are used to evaluate the performance of the ship agent. The evaluation of rewards in this project mainly includes safety rewards R collision , effectiveness reward R effic , Comfort Reward R com and rule reward R rule ; Each reward is defined as: R=R collision +R effic +R com +R rule , R com =k2, R rule =k3, Among them, v, v min and v max is the current speed, minimum speed and maximum speed of the ship agent. k1, k2 and k3 are constants. During training, rewards are calculated based on the collision avoidance effect. Positive rewards include: safe collision avoidance, speed and steering angle within the specified range, and reaching the target position; while negative rewards include: approaching obstacles, collision risk, and deviation from the route.
8. The PDPG-based ship continuous steering intelligent collision avoidance decision system according to claim 7 is characterized by: After each training interaction, the experience data (s, a, r, s') generated by the current interaction is stored in the experience replay pool, where s is the current state, a is the action, r is the reward, and s' is the next state; the ship continuous steering intelligent collision avoidance decision-making system randomly samples batch experience data (s, a, r, s') from the experience replay pool in a key experience priority manner for training the dual Actor and dual Critic networks.