Track tracking control parameter matching method for unmanned surface vehicle

Training PID or PD parameter matchers through deep reinforcement learning algorithms solves the problem that traditional PID controllers cannot adjust adaptively, improves the robustness and control accuracy of unmanned boat tracking, and achieves rapid in interference situations Track tracking.

CN118259672BActive Publication Date: 2024-11-08WUHAN UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410324877.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-21
Publication Date
2024-11-08
Estimated Expiration
2044-03-21

AI Technical Summary

Technical Problem

Traditional PID controllers cannot adaptively adjust control parameters in unmanned boat tracking, resulting in poor control accuracy and insufficient anti-interference.

Method used

The PID or PD parameter matcher is trained using deep reinforcement learning algorithm, and the parameter matcher is automatically trained through neural networks to realize adaptive adjustment of the changes in the navigation state of the unmanned boat.

Benefits of technology

It improves the robustness and control accuracy of unmanned boats in track tracking, and can quickly track the line of sight trajectory in interference situations, reducing manual intervention.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118259672B_ABST
    Figure CN118259672B_ABST
Patent Text Reader

Abstract

The present invention belongs to the field of unmanned boat path tracking, and discloses a method for matching control parameters of surface unmanned boat track tracking, which uses an improved DDPG algorithm to reasonably predict PID or PD parameters, so that the unmanned boat can "learn" to dynamically adjust the PID parameters according to the current state. The method includes: S1, building an unmanned boat motion simulation model and defining the unmanned boat motion parameters; S2, designing a framework combining deep reinforcement learning with the line of sight method; S3, designing a prediction of PID based on the basic framework of deep reinforcement learning, and intending to use a single-layer fully connected neural network as the basic structure of the four networks of the improved DDPG; S4, training the model designed in step S3 and saving the training parameters. The method of the present invention enables the unmanned boat to obtain higher precision and faster track tracking under interference.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the field of unmanned watercraft track tracking control, and in particular relates to an unmanned watercraft track tracking control parameter matching method based on a soft update algorithm. Background Art

[0002] In the field of unmanned boat track tracking, PID control has a wider range of applications and is easier to adjust parameters than other control methods. However, the setting of PID control parameters is more complicated. The control parameters of traditional PID controllers cannot be adaptively adjusted as the navigation state of the unmanned boat changes. In addition, the PID controller with fixed parameters has problems such as poor control accuracy and large overshoot, which leads to poor anti-interference ability of the unmanned boat during track tracking. Summary of the invention

[0003] The purpose of the present invention is to provide a method for matching the control parameters of a surface unmanned boat track tracking, which uses a deep reinforcement learning algorithm to train a PID or PD parameter matcher, and then uses the output PID or PD parameters as the input of the surface unmanned boat track tracking PID or PD controller, solving the problem of adaptive adjustment of control parameters as the surface unmanned boat navigation changes, while improving the robustness of the control, that is, the line of sight track can still be quickly tracked under interference. The method proposed by the present invention replaces mathematical derivation with a neural network approximation, and the parameter matcher is automatically trained by the program without manual intervention.

[0004] According to a first aspect of the embodiment, a method for matching control parameters of a surface unmanned boat track tracking is provided. The method uses an agent output PID or PD parameter trained by deep reinforcement learning as the input of the unmanned boat track tracking PID or PD controller, and the observation state s of the agent is: Where u is the x-axis speed of the unmanned boat, v is the y-axis speed, r is the angular velocity, is the heading angle, y e is the lateral distance between the unmanned boat and the target route, α k is the inclination angle of the target route, e psi is the heading angle, the deviation of the target course inclination angle is, d is the rudder angle, is the derivative of the rudder angle, is the derivative of the heading angle deviation, K p , K d are PID or PD parameters respectively; the agent action a is the PD parameter [K p , K d ].

[0005] The Actor-Critic model is used to train the agent. The method includes:

[0006] Initialize the parameters of the Actor network and Critic network: the number of iterations epoch, the maximum number of steps per iteration steps, the soft update parameter τ, the network learning rate, the decay factor γ, the entropy coefficient α, and assign the network parameters to the Target Actor and Target Critic;

[0007] Initialize the experience pool R;

[0008] Get the t-th step observation state s t :

[0009] Actor network input is state s t , the output is action a t , the agent gets reward r after performing actions and interacting with the environment t and the next state s t+1 , will (s t , a t , r t ,s t+1 , done) is stored in the experience pool R, done represents whether the task is completed, and the value is 0 or 1. Then the experience pool is divided into a failed experience pool and a successful experience pool according to the value of done;

[0010] When the current training reaches the maximum execution step steps, [σ*batch_size] samples are randomly selected from the failure experience pool and the success experience pool, and the agent starts learning, where σ is the output value of the adaptive sampling batch function and batch_size is the number of sample data initially set;

[0011] The state s t , action a t Input to the Critic network, get the current state-action value Q, and the next state s t+1 Input into the Tatget-Actor network to get the next action a t+1 , and similarly t+1 , a t+1 Input to the Target-Critic network to get the state-action value Q', and update the Critic network by gradient descent:

[0012] Take the data extracted from the experience pool (s t , a t , r t ,s t+1 , done) to update the Actor network:

[0013] Update Target-Actor and Target-Critic networks:

[0014] Save the training parameters.

[0015] According to a second aspect of the embodiment, there is provided an unmanned boat, comprising: a processor configured to control the unmanned boat by using the surface unmanned boat track tracking control parameter matching method.

[0016] According to a third aspect of the embodiment, a storage medium is provided for storing non-temporary instructions, which, when executed by a processor, can implement the method for matching control parameters of a track tracking control of an unmanned surface vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a control block diagram of an unmanned boat provided by an embodiment of the present invention.

[0018] Figure 2 This is the principle diagram of the LOS line of sight method.

[0019] Figure 3 This is the Actor neural network structure diagram.

[0020] Figure 4 This is the structure diagram of the Critic neural network.

[0021] Figure 5 It is the flowchart of the improved DDPG algorithm. DETAILED DESCRIPTION

[0022] Reinforcement learning is a branch of machine learning. The most significant feature of reinforcement learning is learning from interaction. Deep learning has strong perception capabilities, but lacks certain decision-making capabilities; while reinforcement learning has decision-making capabilities, but is helpless in the face of perception problems. Therefore, combining the two and complementing each other's strengths provides a solution to the perception and decision-making problems of complex systems. Agents continuously learn knowledge based on the rewards or penalties they receive in their interactions with the environment, and become more adaptable to the environment. The paradigm of reinforcement learning is very similar to the process of knowledge learning by humans, and for this reason, reinforcement learning is considered an important way to achieve general AI.

[0023] The present invention uses the DDPG deep reinforcement learning algorithm to adaptively adjust PID or PD parameters. Through deep reinforcement learning end-to-end, model-free control, the unmanned boat can autonomously learn the control rate, avoid manually establishing a precise model and designing the control rate, achieve precise control of the unmanned boat, and improve the robustness and practicality of the controller. Figure 1 The control block diagram of the unmanned boat is shown below. Figure 1 The unmanned boat track tracking control parameter matching method is explained in detail.

[0024] Step 1: Build an unmanned boat motion simulation model and define motion parameters.

[0025] The motion parameters of the unmanned boat are defined as: the x-axis speed u, y-axis speed v, angular velocity r, heading angle The lateral distance y between the unmanned boat and the target route e 、The inclination angle α of the target route k , the deviation between the heading angle and the target course inclination angle is e psi , the desired heading of the unmanned boat ψ d , the visual distance Δ of the unmanned boat.

[0026] Step 2: Design the agent’s observation state, actions, manually design reward functions, and improve the experience pool.

[0027] The observed state s is defined as: Where d is the rudder angle, is the derivative of the rudder angle, is the derivative of the heading angle deviation, K p , K d are PD parameters respectively.

[0028] Agent action a (2D) is the PD parameter [K p , K d ], K p Range [-0.5, 0.5], K d Range [-50, 50].

[0029] The reward value setting is divided into two parts: angle error reward r psi and lateral distance bonus

[0030]

[0031] where e psi_last is the expected heading angle deviation at the previous moment.

[0032]

[0033] Improve the experience pool, adopt the separation experience pool priority experience replay method, divide the experience pool into failure experience pool and success experience pool, and design an adaptive sampling batch function σ:

[0034]

[0035] Among them, replay_len represents the current number of experience pools, and replay_max represents the maximum number of experience pools.

[0036] Step 3: Design a deep reinforcement learning framework combined with the line of sight method.

[0037]

[0038]

[0039]

[0040] d=K p *e psi +K d *e psi_last

[0041] Where (x desire ,y desire ) is the target end point, (x0, y0) is the target starting point, Δ=2*L, L is the length of the ship, K p , K d PD parameters output by the deep reinforcement learning trainer.

[0042] Step 4: Design a basic framework based on the deep reinforcement learning model, using a single-layer fully connected neural network as the basic structure of the Actor network, Critic network, Target-Actor network, and Target-Critic network.

[0043] The Actor network has the same structure as the Target-Actor network, including input layer, hidden layer and output layer. The activation functions in the middle are connected through the Relu function. The last layer is connected through the Tanh function, and the number of neuron nodes is 300. The Critic network has the same structure as the Target-Critic network, including input layer, hidden layer and output layer. The activation functions are connected through the Relu function. The last layer does not use an activation function, and the number of neuron nodes is 300.

[0044] (a) The input of the Actor network is the state, and the output is the action probability distribution parameter π(a t |s t ), where a t is the action at time t, s t is the state at time t, and then the action is obtained by sampling according to the probability distribution.

[0045] (b) The output of the Critic network is q(s t , a t ), represents the estimate of the value of the action-state pair.

[0046] Step 5: Train the model designed in step 4 and save the training parameters.

[0047] 1) Initialize the parameters of the Actor network and the Critic network: the number of iterations epoch, the maximum number of steps per iteration steps, the soft update parameter τ, the network learning rate, the decay factor γ, the entropy coefficient α, and assign the network parameters to the TargetActor and Target Critic.

[0048] 2) Initialize the experience pool R, initialize the simulation environment, and initialize the random noise n.

[0049] 3) Obtain the t-th step observation state s from the simulation environment t :

[0050] 4) Actor network input is state s t , the output is action a t , the agent gets reward r after performing actions and interacting with the environment t and the next state s t+1 , and (s t , a t , r t ,s t+1 , done) is stored in the experience pool R, where s t+1 Represents the state at time t+1, done represents whether the task is completed, and its value is 0 or 1. Then, according to the value of done, the experience pool is divided into a failed experience pool and a successful experience pool.

[0051] 5) When the current training reaches the maximum number of execution steps, randomly select from the failure experience pool and the success experience pool The agent starts learning after receiving samples of , where σ is the output value of the adaptive sampling batch function, and batch_size is the number of sample data initially set.

[0052] 6) Set the state s t , action a t Input to the Critic network, get the current state-action value, and set the next state s t+1 Input into the Tatget-Actor network to get the next action a t+1 , and similarly t+1 , a t+1 Input to the Target-Critic network to get the state-action value Q', set y t =r t +γQ′(s t+1 ,|μ′(s t+1 |θ μ ′)|θ Q ′), where r t is the reward value at time t, μ′ is the action distribution of the Target-Actor network, θμ ′ is the Target-Actor network parameter, θ Q ′ is the Target-Critic network parameter.

[0053] Update the Critic network by minimizing the loss function L:

[0054]

[0055] where θ Q is the Critic network parameter.

[0056] 7) Take the data extracted from the experience pool (s t , a t , r t ,s t+1 , done) to update the Actor network, the gradient calculation formula is as follows:

[0057]

[0058] The above formula includes the gradient of the action value function to the action and the gradient of the action strategy to the parameter. That is, the gradient of the action a made by the Actor network is relatively large, so the parameter update is updated in the direction of this action gradient.

[0059] 8) Update Target-Actor and Target-Critic networks:

[0060] θ μ ′←τθ μ +(1-τ)θ μ '

[0061] θ Q ′←τθ Q +(1-τ)θ Q '

[0062] Where τ is the soft margin update coefficient.

[0063] The above steps 1 to 5 are completed on a computer.

[0064] Step 6: Perform comparative simulation experiments and actual boat experiments for the same initial state.

[0065] Under the same conditions, simulation experiments and actual boat experiments were conducted to compare the control of PID adjustment using deep reinforcement learning method and adaptive PID.

[0066] The improved DDPG algorithm divides the failed experience pool and the successful experience pool to solve the problems of non-convergence and slow training speed. The reward design of the reinforcement learning algorithm adopts a segmented manually designed reward function to encourage the unmanned boat to fit the trajectory as quickly as possible to prevent excessive overshoot. The sampling mechanism of the strongly improved DDPG algorithm designs an adaptive sampling function. As the number of experience bars in the experience pool increases, the sampling batch size gradually increases until it is twice the initial set sampling batch. The strongly improved DDPG algorithm delays the update of the Actor network and Target-Actor network parameters, that is, after updating the Critic network a certain number of times, the Actor network and Target-Actor network are updated.

[0067] The present invention also provides an embodiment of an unmanned boat. The unmanned boat includes a processor and a memory. The memory is used to store the program of the intelligent agent trained in the above embodiment. The processor is used to run the program of the intelligent agent, and when the program of the intelligent agent is run by the processor, the parameter K of the unmanned boat track tracking PID or PD controller is output. p , K d The memory and processor may be interconnected via a bus system and / or other forms of connection mechanisms.

[0068] For example, the processor may be a central processing unit (CPU), a graphics processing unit (GPU), or other forms of processing units with data processing capabilities and / or program execution capabilities. For example, the central processing unit (CPU) may be an X86 or ARM architecture, etc. The processor may be a general-purpose processor or a dedicated processor, and may control other components in the electronic device to perform desired functions.

[0069] For example, the memory may be a volatile memory and / or a non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, an erasable programmable read-only memory (EPROM), a compact disc read-only memory (CD-ROM), a USB memory, a flash memory, etc. One or more program modules may be stored in the memory, and the processor may run the one or more program modules to implement various functions of the electronic device.

[0070] The present invention also provides an embodiment of a storage medium. The storage medium is used to store the program of the intelligent agent, and when the program of the intelligent agent is executed by the processor, the parameter K of the unmanned boat track tracking PID or PD controller can be generated. p , K d For the relevant description of the storage medium, please refer to the corresponding description of the memory of the unmanned boat above, which will not be repeated here.

Claims

1. A method for matching control parameters of a surface unmanned vehicle track tracking, characterized in that: This method uses the PID or PD parameters output by the intelligent agent trained by deep reinforcement learning as the input of the unmanned boat track tracking PID or PD controller. The observed state s of the agent is: Where u is the x-axis speed of the unmanned boat, v is the y-axis speed, r is the angular velocity, is the heading angle, y e is the lateral distance between the unmanned boat and the target route, α k is the inclination angle of the target route, e psi is the heading angle, the deviation of the target course inclination angle is, d is the rudder angle, is the derivative of the rudder angle, is the derivative of the heading angle deviation, K p ,K d They are PID or PD parameters respectively; Agent action a is the PD parameter [K p , K d ]; The deep reinforcement learning framework is: d=K p *e psi +K d *e psi_last Where (x desire ,y desire ) is the target end point, (x0, y0) is the target starting point, Δ is the visual distance of the unmanned boat, K p , K d PID or PD parameters output by the agent trained by deep reinforcement learning; The Actor-Critic model is used to train the agent. The method includes: Initialize the parameters of the Actor network and Critic network: the number of iterations epoch, the maximum number of steps per iteration steps, the soft update parameter τ, the network learning rate, the decay factor γ, the entropy coefficient α, and assign the network parameters to the Target Actor and Target Critic; Initialize the experience pool R; Get the t-th step observation state s t : Actor network input is state s t , the output is action a t , the agent gets reward r after performing actions and interacting with the environment t and the next state s t+1 , will (s t ,a t ,r t ,s t+1 , done) is stored in the experience pool R, done represents whether the task is completed, and its value is 0 or 1. Then the experience pool is divided into a failed experience pool and a successful experience pool according to the value of done; When the current training reaches the maximum execution step, randomly select from the failure experience pool and the success experience pool The agent starts learning after sampling of , where σ is the output value of the adaptive sampling batch function and batch_size is the number of sample data set initially; The state s t , action a t Input to the Critic network, get the current state-action value Q, and the next state S t+1 Input into the Tatget-Actor network to get the next action a t+1 , and similarly t+1 ,a t+1 Input to the Target-Critic network to get the state-action value Q', and update the Critic network by gradient descent: Take the data extracted from the experience pool (s t ,a t ,r t ,s t+1 ,done) to update the Actor network: Update Target-Actor and Target-Critic networks: Save the training parameters.

2. The method according to claim 1, characterized in that The reward values ​​include: Angle error reward r psi and lateral distance bonus where e psi_last is the expected heading angle deviation at the previous moment, 3. The method according to claim 1, characterized in that: Adaptive sampling batch function σ: Among them, replay_len represents the current number of experience pools, and replay_max represents the maximum number of experience pools.

4. The method according to claim 1, characterized in that: As the number of experience bars in the experience pool increases, the sampling batch size gradually increases until it reaches twice the initial set sampling batch.

5. An unmanned boat, characterized in that: include: A processor is configured to control the unmanned boat using the surface unmanned boat track tracking control parameter matching method described in any one of claims 1-4.

6. A storage medium for storing non-temporary instructions, characterized in that: When the non-temporary instruction is executed by the processor, the surface unmanned boat track tracking control parameter matching method described in any one of claims 1-4 can be implemented.