Autonomous underwater vehicle trajectory tracking method and system based on deep reinforcement learning

Abstract

This invention belongs to the field of autonomous underwater vehicle (AUV) control, and provides a method and system for AUV trajectory tracking based on deep reinforcement learning. The method includes: constructing a control objective for the vehicle trajectory tracking controller and defining it as the AUV trajectory tracking control objective within a deep reinforcement learning framework based on a policy-evaluation network; forming transition tuples and storing them in an experience buffer; randomly sampling the transition tuples in the experience buffer twice to obtain two initial datasets of a set size; then selecting several top-ranked transition tuples from one of the initial datasets according to the score of the current complete exploration process to obtain a filtered dataset; randomly replacing the same number of transition tuples in the other initial dataset with the filtered dataset to obtain the final training dataset for iterative training of the policy-evaluation network; and using the iteratively trained policy-evaluation network as a control network to control the AUV.

Description

Technical Field

[0001] This invention belongs to the field of autonomous underwater vehicle control, and particularly relates to an autonomous underwater vehicle trajectory tracking method and system based on deep reinforcement learning. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Autonomous underwater vehicles (AUVs) can replace manual labor in various tasks such as marine environmental exploration, autonomous underwater operations, marine hydrological environment monitoring, and seabed topography search. Beyond civilian applications, AUVs have enormous potential for use in underwater early warning, underwater reconnaissance, swarm escort, mine detection, and intelligence gathering. To ensure AUVs can successfully complete their assigned tasks, motion control systems are crucial. Dynamic positioning, path tracking, and trajectory tracking are three fundamental motion control problems faced by AUVs in practical missions. Trajectory tracking is one of the most important research areas, ensuring that the AUV reaches its designated location along the desired trajectory. Trajectory tracking involves environmental perception, decision-making, and action execution, and has significant applications in scientific exploration (such as seabed topography mapping) and commercial activities (such as the inspection and maintenance of underwater facilities).

[0004] In real-world underwater missions, such as environmental exploration and shipwreck searches, environmental factors (e.g., water flow, obstacles, underwater topography) are highly dynamic and unpredictable. While current deep reinforcement learning methods have shown some adaptability in the absence of a model, their convergence speed in rapidly changing environments still falls short of practical requirements. Therefore, due to sample efficiency issues, further improvements to existing algorithms are needed.

[0005] Algorithms that rapidly adapt to environmental changes are of significant research and application value for improving the autonomy and mission success rate of AUVs in complex and ever-changing environments. Currently, mainstream reinforcement learning algorithms are designed and optimized based on the Actor-Critic framework. The Soft Actor-Critic (SAC) algorithm is now widely used and has achieved good results due to its excellent performance. However, existing SAC algorithms suffer from slow convergence speeds due to sample efficiency issues. Low sample efficiency means that the algorithm requires a large amount of interaction data to learn an effective policy, which is impractical in many practical applications, especially in scenarios where data acquisition is costly or time-sensitive. AUVs are expensive, thus data acquisition is costly; therefore, the trajectory tracking control algorithm needs to converge as quickly as possible during the training phase to reduce resource waste. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides a method and system for tracking the trajectory of autonomous underwater vehicles based on deep reinforcement learning. This method can achieve faster convergence speed when training the network with limited datasets, thereby reducing the waste of time and resources in practical engineering applications.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] The first aspect of the present invention provides a method for tracking the trajectory of an autonomous underwater vehicle based on deep reinforcement learning.

[0009] In one or more embodiments, a method for tracking the trajectory of an autonomous underwater vehicle based on deep reinforcement learning is provided, including:

[0010] Under the autonomous underwater vehicle system, the state information of the autonomous underwater vehicle is used as input, and the thrust of the thruster and the vertical rudder angle are used as output to construct the control target of the vehicle trajectory tracking controller.

[0011] Based on the predefined state vector, action vector, and reward function of the autonomous underwater vehicle, the control objective of the vehicle trajectory tracking controller is transformed into the autonomous underwater vehicle trajectory tracking control objective under the deep reinforcement learning framework based on the policy-evaluation network.

[0012] Based on the current state vector of the autonomous underwater vehicle, the current action vector and its corresponding reward function value, as well as the next state vector, are sampled, forming a transition tuple and stored in the experience buffer pool D;

[0013] The transition tuples in the experience buffer pool are randomly sampled twice to obtain two initial datasets of a set size. Then, according to the score of the current complete exploration process, a number of transition tuples ranked at the top are selected from one of the initial datasets to obtain the filtered dataset. The filtered dataset is then randomly replaced with the same number of transition tuples in the other initial dataset to obtain the final training dataset for iterative training of the policy-evaluation network.

[0014] The iteratively trained policy-evaluation network is used as a control network to control the autonomous underwater vehicle.

[0015] As one implementation method, the control objective of the vehicle trajectory tracking controller is to solve for the optimal control strategy so that the objective function R... t Maximize (τ);

[0016] R t (τ)=∑ i≥t γ i-t r;

[0017] Where γ is the discount factor, r is the reward function, and τ is the system output, i.e., the thruster thrust and the vertical rudder angle; R t (τ) is a function related to τ; t is time.

[0018] As one implementation method, the reward function at time t is defined as r. t The reward function represents the action a taken at time t. t The reward obtained; the reward function is set according to the current position, desired position, current heading angle, desired heading angle, and output action of the autonomous underwater vehicle.

[0019]

[0020] in, Indicates the reward for approaching the expected trajectory. This indicates the bonus for approaching the desired heading angle. This indicates a reward for the action.

[0021] As one implementation method, the objective function J of the policy network in the policy-evaluation network π The expression for (φ) is:

[0022]

[0023] Where, π φ (a t |s t ) is the policy network, φ is the network parameter; Q θ (s t ,a t ) represents the evaluation network, θ represents the evaluation network parameters; α is the temperature coefficient; S t It is a state vector; a t It is an action vector; a t ∽π φ Indicates action a t It is based on the current strategy π φ Given state s t The distribution of sampling at that time; s t ∽D represents state s t It was sampled from the experience buffer pool D.

[0024] As one implementation method, the objective function J of the evaluation network in the policy-evaluation network Q The expression for (θ) is:

[0025]

[0026] Among them, Q θ (s t ,a t ) represents the evaluation network, θ represents the evaluation network parameters; πφ (a t |s t ) is the policy network, and α is the temperature coefficient; Represents the value function. Represents the network parameters for evaluating the objective; γ is the discount factor; s t It is a state vector; a t It is an action vector; (s t a t )∽D represents the state (s) t a t ) is sampled from the experience buffer pool D; a t ∽π represents action a t It is based on the current strategy π φ The distribution sampled under a given state is obtained; s t+1 ∽P represents the observed state at the next moment; r(s) t ,a t () represents the immediate reward obtained by the AUV after taking an action at time t; This represents the value output by the target evaluation network based on the state and actions taken at time t.

[0027] As one implementation method, the objective function of the temperature coefficient α is J α The expression is:

[0028]

[0029] in, The entropy value is the distribution of actions in the policy network. This indicates that the expectation is about actions, and these actions are based on the current policy π. φ Given state s t The distribution of sampling at that time.

[0030] As one implementation method, the score for the current complete exploration process is calculated by accumulating the reward values ​​obtained from each step of the current exploration.

[0031] A second aspect of the present invention provides an autonomous underwater vehicle trajectory tracking system based on deep reinforcement learning.

[0032] In one or more embodiments, an autonomous underwater vehicle trajectory tracking system based on deep reinforcement learning includes:

[0033] The control target construction module is used to construct the control target of the autonomous underwater vehicle trajectory tracking controller under the autonomous underwater vehicle system, taking the state information of the autonomous underwater vehicle as input and the thruster thrust and vertical rudder angle as output.

[0034] The control target transformation module is used to transform the control target of the vehicle trajectory tracking controller into the autonomous underwater vehicle trajectory tracking control target under the deep reinforcement learning framework based on the policy-evaluation network, based on the predefined state vector, action vector and reward function of the autonomous underwater vehicle.

[0035] The transfer tuple construction module is used to sample the current action vector and its corresponding reward function value and the next state vector based on the current state vector of the autonomous underwater vehicle, form a transfer tuple and store it in the experience buffer pool.

[0036] The policy-evaluation network training module is used to randomly sample transition tuples from the experience buffer twice to obtain two initial datasets of a set size. Then, according to the score of the current complete exploration process, a number of transition tuples ranked at the top are selected from one of the initial datasets to obtain the filtered dataset. The filtered dataset is then used to randomly replace the same number of transition tuples in the other initial dataset to obtain the final training dataset for iterative training of the policy-evaluation network.

[0037] The vehicle control module is used to control the autonomous underwater vehicle by using the iteratively trained policy-evaluation network as the control network.

[0038] A third aspect of the present invention provides a computer-readable storage medium.

[0039] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the deep reinforcement learning-based autonomous underwater vehicle trajectory tracking method described above.

[0040] A fourth aspect of the present invention provides an electronic device.

[0041] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in the deep reinforcement learning-based autonomous underwater vehicle trajectory tracking method described above.

[0042] Compared with the prior art, the beneficial effects of the present invention are:

[0043] This invention transforms the control objective of an autonomous underwater vehicle (AUV) trajectory tracking controller into a deep reinforcement learning framework based on a policy-evaluation network. It eliminates the need for complex modeling by randomly sampling transition tuples from the experience pool twice to obtain two initial datasets of a predetermined size. Then, based on the score of the current complete exploration process, several top-ranked transition tuples are selected from one of the initial datasets to obtain a filtered dataset. This filtered dataset then randomly replaces the same number of transition tuples in the other initial dataset, resulting in the final training dataset for iterative training of the policy-evaluation network. By considering the superiority of the exploration round in which the randomly sampled transition tuples are located, and by adding a priority experience set, the convergence speed of the policy-evaluation network training is accelerated. In practical engineering applications, this can reduce time and resource waste, thereby improving the trajectory tracking efficiency of autonomous underwater vehicles. Attached Figure Description

[0044] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0045] Figure 1 This is a schematic diagram of the trajectory tracking control principle of an autonomous underwater vehicle according to an embodiment of the present invention;

[0046] Figure 2(a) shows the environment exploration process of the improved SAC algorithm according to an embodiment of the present invention;

[0047] Figure 2(b) shows an improved specific network update process according to an embodiment of the present invention;

[0048] Figure 3 This is the SAC algorithm network structure according to an embodiment of the present invention;

[0049] Figure 4 This is the Actor network update process according to an embodiment of the present invention;

[0050] Figure 5 This is the Critic network update process according to an embodiment of the present invention;

[0051] Figure 6 This is a flowchart of the autonomous underwater vehicle trajectory tracking method based on deep reinforcement learning according to an embodiment of the present invention;

[0052] Figure 7 This is a schematic diagram of the structure of an autonomous underwater vehicle trajectory tracking system based on deep reinforcement learning according to an embodiment of the present invention;

[0053] Figure 8 This is a schematic diagram of the electronic device structure according to an embodiment of the present invention. Detailed Implementation

[0054] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0055] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0056] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0057] Figure 1 This is a schematic diagram of the trajectory tracking control principle of an autonomous underwater vehicle according to an embodiment of the present invention; Figure 6 This is a flowchart of the autonomous underwater vehicle trajectory tracking method based on deep reinforcement learning according to an embodiment of the present invention.

[0058] Combination Figure 1 and Figure 6 The deep reinforcement learning-based autonomous underwater vehicle trajectory tracking method in this embodiment may include:

[0059] S601, under the autonomous underwater system, takes the state information of the autonomous underwater vehicle as input and the thruster thrust and vertical rudder angle as output to construct the control target of the vehicle trajectory tracking controller.

[0060] S602, based on the predefined state vector, action vector and reward function of the autonomous underwater vehicle, transforms the control objective of the vehicle trajectory tracking controller into the trajectory tracking control objective of the autonomous underwater vehicle under the deep reinforcement learning framework based on the policy-evaluation network;

[0061] S603: Based on the current state vector of the autonomous underwater vehicle, sample the current action vector and its corresponding reward function value and the next state vector, form a transition tuple and store it in the experience buffer pool.

[0062] S604. Randomly sample the transition tuples in the experience buffer twice to obtain two initial datasets of a set size. Then, select a number of transition tuples ranked at the top from one of the initial datasets according to the score of the current complete exploration process to obtain the filtered dataset. Randomly replace the same number of transition tuples in the other initial dataset with the filtered dataset to obtain the final training dataset for iterative training of the policy-evaluation network.

[0063] S605 uses the iteratively trained policy-evaluation network as a control network to control the autonomous underwater vehicle.

[0064] This embodiment transforms the control objective of the vehicle trajectory tracking controller into an autonomous underwater vehicle trajectory tracking control objective within a deep reinforcement learning framework based on a policy-evaluation network. It eliminates the need for complex modeling, randomly sampling transition tuples from the experience buffer twice to obtain two initial datasets of a set size. Then, based on the score of the current complete exploration process, several top-ranked transition tuples are selected from one of the initial datasets to obtain a filtered dataset. This filtered dataset then randomly replaces the same number of transition tuples in the other initial dataset, resulting in the final training dataset for iterative training of the policy-evaluation network. By considering the superiority of the exploration round in which the randomly sampled transition tuples are located, and adding a priority experience set, the convergence speed of the policy-evaluation network training is accelerated. In practical engineering applications, this can reduce time and resource waste, thereby improving the trajectory tracking efficiency of autonomous underwater vehicles.

[0065] In step S601, under the autonomous underwater vehicle (AUV) system, the state information of the AUV includes position and velocity; position [x] t ,y t ,z t , θ t ,ψ t ], where x t ,y t ,z t This indicates the position of the AUV (Autonomous Underwater Vehicle) under fixed-line conditions at time t. θ t ,ψ t These represent the roll angle, pitch angle, and yaw angle of the AUV in a fixed frame; and the velocity [u] in an inertial coordinate system. t ,v t ,w t ,p t ,q t ,r t ], u t v t w tp represents the linear velocity of the AUV in three directions at time t in the moving frame. t ,q t ,r t These represent the roll rate, pitch rate, and yaw rate, respectively.

[0066] This embodiment addresses the trajectory tracking problem of an AUV in two dimensions. Therefore, the vertical motion of the AUV is not considered, and the final system input becomes: position [x] t ,y t ,ψ t ], speed [u t ,v t ,r t ].

[0067] The system output is τ t =[f t ,δ t ,] including the thrust f of the aircraft's propulsion system t and vertical rudder angle δ t .

[0068] Determine the position error:

[0069] x-axis position error at time t y-axis position error

[0070] The position error at time t is expressed as: It is the difference between the sensor position information and the reference position information.

[0071] The control objective of the vehicle trajectory tracking controller is to find the optimal control strategy that makes the objective function R... t Maximize (τ);

[0072] R t (τ)=∑ i≥t γ i-t r;

[0073] Where γ is the discount factor, r is the reward function, and τ is the system output, i.e., the thruster thrust and the vertical rudder angle; R t (τ) is a function related to τ; t is time.

[0074] In step S602, the state vector is defined:

[0075] The state vector is

[0076] Current AUV location information x t ,y t ,ψ t The speed information of the AUV at the current moment, u t,v t ,r t The expected location information of the AUV at the current moment. Current position error e between AUV and desired position t The AUV's actuator output τ at the previous moment t-1 This includes the thrust of the propellers and the vertical rudder angle.

[0077] Define action vectors:

[0078] Define the action vector at time t as the system output a at time t. t =τ t , where τ t =[f t ,δ t ,].

[0079] Define the reward function:

[0080] Define the reward function at time t as r t The reward function represents the action a taken at time t. t The reward obtained; the reward function is set according to the current position, desired position, current heading angle, desired heading angle, and output action of the autonomous underwater vehicle.

[0081]

[0082] in, Indicates the reward for approaching the expected trajectory. This indicates the bonus for approaching the desired heading angle. This indicates a reward for the action.

[0083] The objective function J of the policy network in the policy-evaluation network π The expression for (φ) is:

[0084]

[0085] Where, π φ (a t |s t ) is the policy network, φ is the network parameter; Q θ (s t ,a t ) represents the evaluation network, θ represents the evaluation network parameters; α is the temperature coefficient; s t It is a state vector; a t It is an action vector; a t ∽π φ Indicates action a t It is based on the current strategy π φ Given state a t The distribution of sampling at that time; s t∽D represents state s t It was sampled from the experience buffer pool D.

[0086] Among them, the objective function J of the evaluation network in the policy-evaluation network is... Q The expression for (θ) is:

[0087]

[0088] Among them, Q θ (s t ,a t ) represents the evaluation network, θ represents the evaluation network parameters; π φ (a t |s t ) is the policy network, and α is the temperature coefficient; Represents the value function. Represents the network parameters for evaluating the objective; γ is the discount factor; s t It is a state vector; a t It is an action vector; (s t a t )∽D represents the state (s) t a t ) is sampled from the experience buffer pool D; a t ∽π represents action a t It is based on the current strategy π φ The distribution sampled under a given state is obtained; s t+1 ∽P represents the observed state at the next moment; r(s) t ,a t () represents the immediate reward obtained by the AUV after taking an action at time t; This represents the value output by the target evaluation network based on the state and actions taken at time t.

[0089] The objective function of temperature coefficient α is J α The expression is:

[0090]

[0091] in, The entropy value is the distribution of actions in the policy network. This indicates that the expectation is about actions, and these actions are based on the current policy π. φ Given state s t The distribution of sampling at that time.

[0092] As shown in Figure 2(a), Figure 2(b) and Figure 3 As shown, the construction process of the Policy-Evaluation Network (SAC Network) is as follows:

[0093] Constructing an Actor network:

[0094] By constructing an Actor network, the current action is output based on the current input state. To make the output strategy more stable, the clip function is used to limit the range of change in the output action of the new strategy.

[0095] like Figure 4 As shown, the Actor network consists of one input layer, two hidden layers, and two output layers, each of which is a fully connected layer. The input to the input layer is the observation s. t The number of neurons is the observation quantity s. t The dimension is ; the number of neurons in the hidden layer is 300; the output layer outputs action a. t The mean and standard deviation are used to establish a normal distribution, and then reparameter sampling is performed to obtain the final output action.

[0096] The input and hidden layers use the ReLU function for activation, while the output layer uses the tanh function to limit the output range.

[0097] Building the Critic network:

[0098] By constructing a Critic network, the actions output by the current Actor network are evaluated, and the value Q is obtained.

[0099] like Figure 5 As shown, the Critic network consists of four networks: Q1 network, Q2 network, Q1_target network, and Q2_target network. Each network has the same structure.

[0100] Each Critic network consists of one input layer, two hidden layers, and one output layer. Each layer is fully connected. The input to the input layer is the current state s. t and the action taken a t The number of neurons is the state s t And action a t The total dimension; the number of neurons in the hidden layer is 300; the output layer outputs the evaluation of the current action, with a dimension of 1.

[0101] The activation functions of the input and hidden layers are sampled from the ReLU function, and the weights and bias parameters of the fully connected layer are initialized with a uniform distribution in the output layer.

[0102] Define the target strategy:

[0103] When the average reward per round stabilizes at a certain time step t, the target policy can be considered to have been obtained, and the policy learned after time step t can be used as the output.

[0104] Combining steps S603 and S604, the iterative training strategy-evaluation network process is as follows:

[0105] (1) Parameter settings:

[0106] The optimizer learning rate is lr, the discount factor is γ, the size of the experience buffer D is M, the size of the dataset randomly sampled in each training session is b, the size of the advantage dataset is k, the target Q network update parameters are f, the maximum number of steps during training is T, and the number of steps that satisfy the training start condition is T_train.

[0107] (2) Initialize the Soft Actor-Critic network parameters:

[0108] Randomly initialize the Actor network parameters φ, the two Critic networks Q1 and Q2 parameters θ1 and θ2, and the two target networks Q1_target and Q2_target parameters. Initializing the experience buffer pool means creating a new empty training set.

[0109] (3) Obtain the state transition tuple into the experience buffer:

[0110] Initialization state s t The iteration begins. First, input the current state s. t In the Actor network, action a is sampled. t and reward value r(s) t ,a t ), and at the same time obtain the next state s t+1 The resulting transition tuple {s t a t ,r(s t ,a t ), s t+1 The data is stored in the experience buffer pool D; in addition, this method introduces a new parameter ρ based on the original state transition array. e This parameter represents the score of the e-th exploration process. This is for a complete exploration, not the score of each step. Therefore, the ρ in the transition tuple during an exploration process... e The values ​​are the same; therefore, the new transition tuple is as follows: {s} t a t ,r(s t ,a t ), s t+1 , ρ e Repeat the above process until the set number of steps T is met.

[0111] Among them, the score ρ of the current complete exploration process e The calculation process is as follows: the reward value obtained from each step of the current exploration is accumulated.

[0112] (4) Obtain the training dataset:

[0113] When the number of iterations meets the set number of steps T_train, it indicates that the experience buffer has collected a certain number of transition tuples, and training can begin. A training set is formed by sampling from the experience buffer D to update the neural network. The specific operation process is as follows: First, randomly sample the transition tuples in the experience buffer to obtain a dataset B1 of size b; then randomly sample the transition tuples in the experience buffer to obtain a dataset B2 of size b; according to the parameter ρ... e Prioritize the data in B2 and select ρ from the B2 dataset. e The highest k The transferred tuples are used to form a new dataset B2_prior; the resulting dataset B2_prior is then merged with dataset B1 to form a new dataset B, which is used for subsequent training.

[0114] (5) Critic network update:

[0115] Once the training set is ready, the neural network is updated. First, based on J... Q The (θ) function is used to perform gradient descent to update the network parameters θ1 and θ2 of the two Critic networks Q1 and Q2.

[0116]

[0117] (6) Actor Network Update:

[0118] Then according to J π The (φ) function is used to update the parameters φ of the Actor network through gradient descent.

[0119]

[0120] (7) Temperature coefficient α update:

[0121] Then update the temperature coefficient α.

[0122]

[0123] (8) Target network update:

[0124] Finally, perform a soft update on the parameters of the two target networks, Q1_target and Q2_target.

[0125]

[0126] After the neural network is fully updated, the next iteration process begins, repeating steps (7) to (9) until the iteration ends.

[0127] The iteration stops, the improved SAC algorithm of this embodiment is updated, and the final θ1, θ2, φ are output.

[0128] The learned Actor and Critic networks are used as the control network to achieve trajectory tracking control of the autonomous underwater vehicle.

[0129] Figure 7 This is a schematic diagram of the structure of an autonomous underwater vehicle trajectory tracking system based on deep reinforcement learning in an embodiment of the present invention. This embodiment is similar to... Figure 6 Corresponding to the deep reinforcement learning-based autonomous underwater vehicle trajectory tracking method, such as... Figure 7 As shown, the deep reinforcement learning-based autonomous underwater vehicle trajectory tracking system in this embodiment may include:

[0130] The control target construction module 701 is used to construct the control target of the vehicle trajectory tracking controller under the autonomous underwater vehicle system by taking the position, speed and other state information of the autonomous underwater vehicle as input and the thrust of the thruster and the vertical rudder angle as output.

[0131] The control target conversion module 702 is used to convert the control target of the vehicle trajectory tracking controller into the autonomous underwater vehicle trajectory tracking control target under the deep reinforcement learning framework based on the policy-evaluation network, based on the predefined state vector, action vector and reward function of the autonomous underwater vehicle.

[0132] The transfer tuple construction module 703 is used to sample the current action vector and its corresponding reward function value and the next state vector based on the current state vector of the autonomous underwater vehicle, form a transfer tuple and store it in the experience buffer pool.

[0133] The policy-evaluation network training module 704 is used to randomly sample the transition tuples in the experience buffer twice to obtain two initial datasets of a set size. Then, according to the score of the current complete exploration process, a number of transition tuples ranked at the top are selected from one of the initial datasets to obtain the filtered dataset. The filtered dataset is then randomly replaced with the same number of transition tuples in the other initial dataset to obtain the final training dataset for iterative training of the policy-evaluation network.

[0134] The vehicle control module 705 is used to control the autonomous underwater vehicle by using the iteratively trained policy-evaluation network as a control network.

[0135] It should be noted here that, Figure 7 The various modules in the deep reinforcement learning-based autonomous underwater vehicle trajectory tracking system, and... Figure 6Each step in the deep reinforcement learning-based autonomous underwater vehicle trajectory tracking method corresponds to the previous one, and their specific implementation process is the same, so it will not be repeated here.

[0136] Reference Figure 8 A schematic diagram of an electronic device is provided. It should be noted that... Figure 8 The illustrated electronic device 800 is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0137] like Figure 8 As shown, the electronic device 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage section 808 into a random access memory (RAM) 803. The RAM 803 also stores various programs and data required for system operation. The CPU 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0138] The following components are connected to I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a local area network (LAN) card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 810 as needed so that computer programs read from it can be installed into storage section 808 as needed.

[0139] When the central processing unit 801 in the electronic device of this embodiment executes the program, it achieves the following: Figure 6 The steps in the deep reinforcement learning-based autonomous underwater vehicle trajectory tracking method are shown.

[0140] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including functions for executing... Figure 6The program code for the method shown. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 809, and / or installed from a removable medium 811. When the computer program is executed by the central processing unit 801, it performs the various functions defined in the apparatus of this application.

[0141] in, Figure 6 The computer program instructions corresponding to the method shown may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in the process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0142] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0143] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A trajectory tracking method for autonomous underwater vehicles based on deep reinforcement learning, characterized in that, include: Under the autonomous underwater vehicle system, the state information of the autonomous underwater vehicle is used as input, and the thrust of the thruster and the vertical rudder angle are used as output to construct the control target of the vehicle trajectory tracking controller. Based on the predefined state vector, action vector, and reward function of the autonomous underwater vehicle, the control objective of the vehicle trajectory tracking controller is transformed into the autonomous underwater vehicle trajectory tracking control objective under the deep reinforcement learning framework based on the policy-evaluation network. Based on the current state vector of the autonomous underwater vehicle, the current action vector, its corresponding reward function value, and the next state vector are sampled, forming a transition tuple and storing it in the experience buffer pool. The transition tuples in the experience buffer pool are randomly sampled twice to obtain two initial datasets of a set size. Then, according to the score of the current complete exploration process, a number of transition tuples ranked at the top are selected from one of the initial datasets to obtain the filtered dataset. The filtered dataset is then randomly replaced with the same number of transition tuples in the other initial dataset to obtain the final training dataset for iterative training of the policy-evaluation network. The iteratively trained policy-evaluation network is used as the control network to control the autonomous underwater vehicle. The objective function of the policy network in the policy-evaluation network The expression is: in, It is a policy network. These are network parameters; It is an evaluation network. It is to evaluate network parameters; It is the temperature coefficient; It is a state vector; It is an action vector; Indicates action Based on the current strategy Given a state The distribution of sampling at that time; Representing state It was sampled from the experience buffer pool D; The objective function of the evaluation network in the strategy-evaluation network The expression is: [ ] in, It is an evaluation network. It is to evaluate network parameters; It is a policy network. It is the temperature coefficient; Represents the value function. Indicate the network parameters for evaluating the target; It is a discount factor; It is a state vector; It is an action vector; Representing state It was sampled from the experience buffer pool D; Indicates action Based on the current strategy The distribution is sampled under a given state; Indicates the observed state at the next moment; This represents the immediate reward obtained by the AUV after it takes an action at time t; This represents the value output by the target evaluation network based on the state and actions taken at time t.

2. The method for tracking the trajectory of an autonomous underwater vehicle based on deep reinforcement learning as described in claim 1, characterized in that, The control objective of the aircraft trajectory tracking controller is to find the optimal control strategy that makes the objective function... maximize; ； in, As a discount factor, For the reward function, This refers to the system output, namely the thruster thrust and the vertical rudder angle; for Related functions; For a moment.

3. The trajectory tracking method for autonomous underwater vehicles based on deep reinforcement learning as described in claim 1, characterized in that, Define the reward function at time t as follows: The reward function represents the state. Take action at the time The reward obtained; the reward function is set according to the current position, desired position, current heading angle, desired heading angle, and output action of the autonomous underwater vehicle. in, Indicates the reward for approaching the expected trajectory. This indicates the bonus for approaching the desired heading angle. This indicates a reward for the action.

4. The method for tracking the trajectory of an autonomous underwater vehicle based on deep reinforcement learning as described in claim 1, characterized in that, Temperature coefficient objective function The expression is: in, The entropy value is the distribution of actions in the policy network. This indicates that the expectation is about actions, and these actions are based on the current strategy. Given a state The distribution of sampling at that time.

5. The method for tracking the trajectory of an autonomous underwater vehicle based on deep reinforcement learning as described in claim 1, characterized in that, The score for the current complete exploration process is calculated by summing the reward values ​​obtained from each step of the current exploration.

6. A trajectory tracking system for an autonomous underwater vehicle based on deep reinforcement learning, characterized in that, The method for tracking the trajectory of an autonomous underwater vehicle based on deep reinforcement learning, as described in any one of claims 1-5, is adopted, including: The control target construction module is used to construct the control target of the autonomous underwater vehicle trajectory tracking controller under the autonomous underwater vehicle system, taking the state information of the autonomous underwater vehicle as input and the thruster thrust and vertical rudder angle as output. The control target transformation module is used to transform the control target of the vehicle trajectory tracking controller into the autonomous underwater vehicle trajectory tracking control target under the deep reinforcement learning framework based on the policy-evaluation network, based on the predefined state vector, action vector and reward function of the autonomous underwater vehicle. The transfer tuple construction module is used to sample the current action vector and its corresponding reward function value and the next state vector based on the current state vector of the autonomous underwater vehicle, form a transfer tuple and store it in the experience buffer pool. The policy-evaluation network training module is used to randomly sample transition tuples from the experience buffer twice to obtain two initial datasets of a set size. Then, according to the score of the current complete exploration process, a number of transition tuples ranked at the top are selected from one of the initial datasets to obtain the filtered dataset. The filtered dataset is then used to randomly replace the same number of transition tuples in the other initial dataset to obtain the final training dataset for iterative training of the policy-evaluation network. The vehicle control module is used to control the autonomous underwater vehicle by using the iteratively trained policy-evaluation network as the control network.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the deep reinforcement learning-based autonomous underwater vehicle trajectory tracking method as described in any one of claims 1-5.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the deep reinforcement learning-based autonomous underwater vehicle trajectory tracking method as described in any one of claims 1-5.

Images