Trajectory tracking control method, device and equipment of unmanned underwater vehicle and medium

Abstract

The application provides a trajectory tracking control method, device and equipment of an unmanned underwater vehicle and a medium. The method constructs a trajectory generator and a trajectory tracking controller for a virtual unmanned underwater vehicle based on a reinforcement learning network, respectively. The trajectory generator and the trajectory tracking controller are alternately trained in multiple rounds to obtain a trained trajectory generator and a trained trajectory tracking controller. In the alternately training, the trajectory generator is used to generate trajectory data samples until a reward value obtained by the trajectory tracking controller in tracking the trajectory data samples is minimum. The trajectory tracking controller is used to generate control data samples with a maximum reward value based on the trajectory data samples. The trained trajectory generator and the trained trajectory tracking controller are deployed to a real unmanned underwater vehicle for control. The application can improve the control accuracy of the real unmanned underwater vehicle.

Description

Technical Field

[0001] This application relates to the field of unmanned underwater vehicle control technology, and more specifically, to a trajectory tracking control method, device, equipment, and storage medium for an unmanned underwater vehicle. Background Technology

[0002] Unmanned Underwater Vehicles (UUVs) possess the capability to perform challenging tasks such as exploration, surveying, deployment, and recovery in complex marine environments. Their three-dimensional trajectory tracking and control performance is closely related to the successful completion of these tasks. In practical applications, UUVs often need to navigate in waters with multiple obstacles or limited space. In such cases, their trajectory must not only closely adhere to a pre-set path but also possess excellent smoothness and stability to ensure navigational safety and improve mission accuracy. Especially when performing high-precision tasks, UUVs must not only operate along a predetermined trajectory but also maintain high precision in attitude stability and position control throughout the entire voyage.

[0003] In recent years, deep reinforcement learning has gradually attracted attention in UUV trajectory control research. However, in high-dimensional and complex tasks, if there are distribution biases in the sampling of states and actions during training or insufficient sample diversity, it is easy for the policy or reward function to overfit on limited experience, thus lacking the ability to generalize to unknown trajectories or new environments, resulting in a significant decrease in control accuracy. Therefore, how to improve the control accuracy of UUVs is an urgent problem to be solved. Summary of the Invention

[0004] In view of this, this application provides a trajectory tracking and control method, apparatus, device and storage medium for an unmanned underwater vehicle, so as to at least solve the problems existing in the related technologies.

[0005] Specifically, this application is implemented through the following technical solution: This application provides a trajectory tracking and control method for an unmanned underwater vehicle, including: A trajectory generator and a trajectory tracking controller for a virtual unmanned underwater vehicle are constructed based on a reinforcement learning network. The trajectory generator and the trajectory tracking controller are trained in multiple rounds to obtain a trained trajectory generator and a trained trajectory tracking controller. Each round of training includes adversarial alternation training for the trajectory generator and the trajectory tracking controller. During the adversarial alternation training, the trajectory generator generates trajectory data samples until the reward value obtained by tracking the trajectory data samples by the trajectory tracker is minimized. The trajectory tracking controller generates control data samples with the maximum reward value based on the trajectory data samples. The system acquires navigation information of the real unmanned underwater vehicle corresponding to the virtual unmanned underwater vehicle, generates a target trajectory based on the navigation information using the trained trajectory generator, generates a trajectory tracking control strategy based on the target trajectory using the trained trajectory tracking controller, and controls the real unmanned underwater vehicle according to the trajectory tracking strategy.

[0006] In some implementations, the adversarial alternation training of the trajectory generator and the trajectory tracking controller includes: The network parameters of one of the trajectory generator and the trajectory tracking controller are fixed in sequence, and the other is trained multiple times to adjust the network parameters of the other.

[0007] In some implementations, the step of sequentially fixing the network parameters of one of the trajectory generator and the trajectory tracking controller, and then training the other multiple times to adjust the network parameters of the other includes: Using the trajectory generator, a first trajectory is generated based on the virtual navigation information of the virtual unmanned underwater vehicle; Using the trajectory tracking controller, control data samples are generated based on the first trajectory; The control data sample set is updated based on the control data sample, and a trajectory data sample is generated based on the first trajectory, and the trajectory data sample set is updated based on the trajectory data sample. During the training of the trajectory generator, the network parameters of the trajectory tracking controller are fixed, and the network parameters of the trajectory generator are adjusted based on the trajectory data sample set; or... During the training of the trajectory tracking controller, the network parameters of the trajectory generator are fixed, and the network parameters of the trajectory tracking controller are adjusted based on the control data sample set.

[0008] In some implementations, generating control data samples based on the first trajectory using the trajectory tracking controller includes: The trajectory tracking controller generates and executes a trajectory tracking action based on the first trajectory and the current trajectory tracking state, thereby obtaining a new trajectory tracking state and a control reward value. The control data sample is generated based on the current trajectory tracking state, the trajectory tracking action, the new trajectory tracking state, and the control reward value; The step of generating trajectory data samples based on the first trajectory and updating the trajectory data sample set based on the trajectory data samples includes: The trajectory data sample is generated based on the first trajectory and the corresponding trajectory reward value, and the trajectory data sample set is updated according to the trajectory data sample; the trajectory reward value is determined based on the control reward value.

[0009] In some implementations, the trajectory reward value and the control reward value are opposites of each other.

[0010] In some implementations, generating a first trajectory based on the virtual navigation information of the virtual unmanned underwater vehicle using the trajectory generator includes: Construct the trajectory state space and trajectory action space of the virtual unmanned underwater vehicle; the trajectory state space includes position state, velocity state and angular velocity state, and the trajectory action space includes thruster speed and rudder angle; Within the trajectory state space and trajectory action space, the trajectory generator generates the first trajectory based on the virtual navigation information.

[0011] In some implementations, the virtual unmanned underwater vehicle has the same dynamic model as the real unmanned underwater vehicle.

[0012] This application also provides a trajectory tracking and control device for an unmanned underwater vehicle, comprising: The module is used to build a trajectory generator and a trajectory tracking controller for the virtual unmanned underwater vehicle based on a reinforcement learning network. A training module is used to perform multiple rounds of training on the trajectory generator and the trajectory tracking controller to obtain a trained trajectory generator and a trained trajectory tracking controller. Each round of training includes adversarial alternating training for the trajectory generator and the trajectory tracking controller. During the adversarial alternating training, the trajectory generator generates trajectory data samples until the reward value obtained by tracking the trajectory data samples by the trajectory tracker is minimized, and the trajectory tracking controller generates control data samples with the maximum reward value based on the trajectory data samples. The control module is used to acquire the navigation information of the real unmanned underwater vehicle corresponding to the virtual unmanned underwater vehicle, generate a target trajectory based on the navigation information using the trained trajectory generator, generate a trajectory tracking control strategy based on the target trajectory using the trained trajectory tracking controller, and control the real unmanned underwater vehicle according to the trajectory tracking strategy.

[0013] This application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the trajectory tracking control method for any of the unmanned underwater vehicles described in the foregoing embodiments.

[0014] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the trajectory tracking control method for any of the unmanned underwater vehicles described in the foregoing embodiments.

[0015] This application also provides a computer program product, including a computer program that, when run by a processor, performs the steps of any of the possible trajectory tracking control methods for unmanned underwater vehicles described above.

[0016] The technical solutions provided by the embodiments of this application may include the following beneficial effects: In this embodiment, multiple rounds of training are conducted for the trajectory generator and the trajectory tracking controller. Each round of training includes adversarial alternating training for the trajectory generator and the trajectory tracking controller. During the adversarial alternating training, the trajectory generator generates trajectory data samples until the reward value obtained by tracking the trajectory data samples by the trajectory tracker is minimized. The trajectory tracking controller generates control data samples with the maximum reward value based on the trajectory data samples. That is, through the above-mentioned adversarial alternating training mechanism, the trajectory generator can continuously generate trajectories with tracking difficulty, thereby increasing the diversity of trajectory samples. This drives the trajectory tracking controller to continuously optimize its trajectory tracking capability based on the trajectory generated by the trajectory generator, thereby improving the generalization ability and control accuracy of the trajectory tracking controller. In turn, based on this mechanism, the control accuracy of the real unmanned underwater vehicle can be improved.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating an exemplary embodiment of the present application of a trajectory tracking control method for an unmanned underwater vehicle; Figure 2 This is a flowchart illustrating a single training process as shown in an exemplary embodiment of this application; Figure 3 This is a structural block diagram of a trajectory generator shown in an exemplary embodiment of this application; Figure 4 This is a schematic diagram illustrating an adversarial alternation training process according to an exemplary embodiment of this application; Figure 5 This is a schematic diagram of the structure of a trajectory tracking control device for an unmanned underwater vehicle, as shown in an exemplary embodiment of this application; Figure 6 This is a hardware structure diagram of a computer device illustrated in an exemplary embodiment of this application. Detailed Implementation

[0019] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0020] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0021] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0022] Unmanned Underwater Vehicles (UUVs) possess the capability to perform challenging tasks such as exploration, surveying, deployment, and recovery in complex marine environments. Their three-dimensional trajectory tracking and control performance is closely related to the successful completion of these tasks. In practical applications, UUVs often need to navigate in waters with multiple obstacles or limited space. In such cases, their trajectory must not only closely adhere to a pre-set path but also possess excellent smoothness and stability to ensure navigational safety and improve mission accuracy. Especially when performing high-precision tasks, UUVs must not only operate along a predetermined trajectory but also maintain high precision in attitude stability and position control throughout the entire voyage.

[0023] Traditional pathpoint tracking algorithms typically target discrete positions for control, making it difficult to accurately track an entire continuous trajectory. This is especially true in environments with complex disturbances or unstructured environments, where the control effectiveness of such algorithms is significantly limited. When performing trajectory tracking tasks in three-dimensional space, UUVs need to achieve coupled control in both the horizontal and vertical directions simultaneously. However, since UUVs are generally underactuated systems, significant kinematic coupling exists between their degrees of freedom, further increasing the difficulty of controller design. To reduce modeling and design complexity, some studies have adopted simplified dynamic models to design controllers; however, this method often suffers from insufficient dynamic response performance and low tracking accuracy in practical applications, limiting its applicability in complex environments.

[0024] In recent years, deep reinforcement learning has gradually gained attention in UUV trajectory control research, with typical algorithms including DQN, DDPG, TD3, PPO, and SAC. However, in high-dimensional and complex tasks, if there are distribution biases in state and action sampling or insufficient sample diversity during training, the policy or reward function is prone to overfitting on limited experience, thus lacking the ability to generalize to unknown trajectories or new environments, resulting in a significant decrease in control accuracy. Therefore, how to improve the control accuracy of UUVs is an urgent problem to be solved.

[0025] Based on the above research, this disclosure provides a trajectory tracking control method for an unmanned underwater vehicle (UUV). This method constructs a trajectory generator and a trajectory tracking controller for a virtual UUV using a reinforcement learning network. The trajectory generator and the trajectory tracking controller are trained in multiple rounds to obtain a trained trajectory generator and a trained trajectory tracking controller. Each round of training includes adversarial alternating training for the trajectory generator and the trajectory tracking controller. During the adversarial alternating training, the trajectory generator generates trajectory data samples until the reward value obtained by tracking the trajectory data samples by the trajectory tracker is minimized. The trajectory tracking controller then generates control data samples with the maximum reward value based on the trajectory data samples. The method acquires navigation information of a real UUV corresponding to the virtual UUV, uses the trained trajectory generator to generate a target trajectory based on the navigation information, and uses the trained trajectory tracking controller to generate a trajectory tracking control strategy based on the target trajectory. Finally, the method controls the real UUV according to the trajectory tracking strategy.

[0026] In this embodiment, multiple rounds of training are conducted for the trajectory generator and the trajectory tracking controller. Each round of training includes adversarial alternating training for the trajectory generator and the trajectory tracking controller. During the adversarial alternating training, the trajectory generator generates trajectory data samples until the reward value obtained by tracking the trajectory data samples by the trajectory tracker is minimized. The trajectory tracking controller generates control data samples with the maximum reward value based on the trajectory data samples. That is, through the above-mentioned adversarial alternating training mechanism, the trajectory generator can continuously generate trajectories with tracking difficulty, thereby increasing the diversity of trajectory samples. This drives the trajectory tracking controller to continuously optimize its trajectory tracking capability based on the trajectory generated by the trajectory generator, thereby improving the generalization ability and control accuracy of the trajectory tracking controller. In turn, based on this mechanism, the control accuracy of the real unmanned underwater vehicle can be improved.

[0027] To facilitate understanding of this embodiment, a detailed description of the trajectory tracking and control method for an unmanned underwater vehicle (UUV) disclosed in this disclosure is provided first. The execution entity of the trajectory tracking and control method for UUVs provided in this disclosure is generally a computer device. This computer device can be a server, which can be an independent physical server, a server cluster composed of multiple physical servers, or a distributed system. It can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, big data, and artificial intelligence platforms. In other embodiments, the computer device can also be a terminal device, which can be a mobile device, terminal, handheld device, computing device, vehicle-mounted device, etc.

[0028] In other embodiments, the method can also be applied to an implementation environment consisting of computer equipment and servers, or an implementation environment consisting of terminal equipment and servers. Furthermore, the trajectory tracking control method for this unmanned underwater vehicle can also be implemented by a processor calling computer-readable instructions stored in memory.

[0029] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0030] Please see the appendix Figure 1 The above is a flowchart illustrating a trajectory tracking control method for an unmanned underwater vehicle, as shown in an exemplary embodiment of this application. Figure 1 As shown, the trajectory tracking and control method for the unmanned underwater vehicle in this embodiment may include the following steps S101~S103: S101: Construct a trajectory generator and a trajectory tracking controller for the virtual unmanned underwater vehicle based on a reinforcement learning network.

[0031] The virtual unmanned underwater vehicle is modeled based on the real unmanned underwater vehicle. Specifically, the virtual unmanned underwater vehicle has the same kinematic model as the real unmanned underwater vehicle.

[0032] Here, it can be understood that the trajectory generator and trajectory tracking controller to be trained are built on reinforcement learning networks, and they can respectively implement corresponding functions through intelligent agents. Specifically, the trajectory generator is used to generate trajectory data samples and adjust its own network parameters based on the feedback of the trajectory data samples; the trajectory tracking controller is used to track the trajectory data samples and generate control data samples, and adjust its own network parameters based on the feedback of the control data samples.

[0033] In this application, the trajectory generator and trajectory tracking controller can be virtual software. Of course, in some embodiments, the trajectory generator and trajectory tracking controller can also be hardware devices, which is not limited here.

[0034] In this embodiment, the reinforcement learning network is implemented based on the TwinDelayed Deep Deterministic Policy Gradient (TD3) algorithm, which is an algorithm designed specifically for solving the problem of continuous action space in reinforcement learning.

[0035] Among them, the TD3 algorithm is based on the Actor-Critic architecture. The Actor network is responsible for generating actions, and the Critic network is responsible for evaluating the value (Q value) of the actions. The TD3 algorithm adopts a double Q network, which, combined with its own delayed policy update, double Q network and target smoothing mechanism, can improve the stability of training and solve the overestimation bias and policy oscillation problems of traditional reinforcement learning control in high-dimensional tasks.

[0036] S102: Perform multiple rounds of training on the trajectory generator and the trajectory tracking controller to obtain a trained trajectory generator and a trained trajectory tracking controller; wherein, each round of training includes adversarial alternating training for the trajectory generator and the trajectory tracking controller; during the adversarial alternating training process, the trajectory generator is used to generate trajectory data samples until the reward value obtained by tracking the trajectory data samples by the trajectory tracker is minimized, and the trajectory tracking controller is used to generate control data samples with the maximum reward value based on the trajectory data samples.

[0037] Here, it can be understood that the trajectory generator and trajectory tracking controller will undergo multiple rounds of training. Each round of training includes adversarial alternating training of the trajectory generator and trajectory tracking controller. In adversarial alternating training, the training objective of the trajectory generator is to continuously generate trajectory data samples until the reward value obtained by the trajectory tracker in tracking the trajectory data samples is minimized. In simple terms, the training objective of the trajectory generator can be to generate the trajectory that makes the trajectory tracking controller's control effect the worst (or to generate the trajectory that makes it difficult for the trajectory tracking controller to track). In this way, the diversity of trajectory data samples can be enriched.

[0038] Similarly, the trajectory tracking controller tracks and controls the trajectory data samples to generate control data samples. Its training objective is to generate control data samples with the highest reward value based on the trajectory data samples. The higher the accuracy of the control data samples, the greater the corresponding reward. In simple terms, the goal of the trajectory tracking controller is to maximize the reward obtained during the tracking process, thereby improving the tracking accuracy of the trajectory tracking controller.

[0039] In this application, during the adversarial alternation training of the trajectory generator and the trajectory tracking controller, the network parameters of one of the trajectory generator and the trajectory tracking controller can be fixed sequentially, and the other can be trained multiple times to adjust the network parameters of the other.

[0040] Specifically, the network parameters of the trajectory tracking controller can be fixed, and the network parameters of the trajectory generator can be trained multiple times. Then, the network parameters of the trajectory generator can be fixed, and the trajectory tracking controller can be trained multiple times.

[0041] In order to improve the accuracy of the trajectory generated by the trajectory generator, this application first trains the trajectory generator multiple times to adjust the network parameters of the trajectory generator. Then, the adjusted network parameters of the trajectory generator are fixed, and the trajectory tracking controller is trained multiple times.

[0042] For specific details regarding each training session, please refer to [link / reference]. Figure 2 The flowchart provided for an exemplary embodiment of this application illustrates a training process, such as... Figure 2 As shown, the specific contents include the following S201~S204: S201: Using the trajectory generator, a first trajectory is generated based on the virtual navigation information of the virtual unmanned underwater vehicle.

[0043] Virtual navigation information can include virtual environment information (such as virtual obstacle information, virtual water flow speed information, etc.) and virtual mission objectives (such as destination location, reaching the destination in the shortest time, etc.).

[0044] Here, virtual navigation information can be artificially constructed or constructed based on the historical navigation information of real unmanned underwater vehicles; no limitation is made here.

[0045] In this way, the trajectory generator can be used to generate the first trajectory based on the virtual navigation information of the virtual unmanned underwater vehicle.

[0046] In this application, when generating the first trajectory, the trajectory generator does not directly generate the first trajectory, but first generates and executes trajectory actions based on the current trajectory state and virtual navigation information, generates the next trajectory state, and then generates the first trajectory in three-dimensional space based on the dynamic model of the virtual unmanned underwater vehicle and each trajectory state.

[0047] Please see Figure 3 This is a structural block diagram of a trajectory generator provided as an exemplary embodiment of this application. Figure 3 As shown, the trajectory generator can be regarded as controlling the movement of the virtual UUV in space through the trajectory generation strategy network. Therefore, it is necessary to design its reward function, trajectory state space and trajectory action space. Based on the above, this application pre-constructs the trajectory state space and trajectory action space of the virtual unmanned underwater vehicle. The trajectory state space includes position state, velocity state and angular velocity state, and the trajectory action space includes thruster speed and rudder angle. The rudder angle includes horizontal rudder angle and vertical rudder angle.

[0048] Subsequently, in the trajectory state space and trajectory action space, the trajectory generator is used to generate the first trajectory based on the virtual navigation information.

[0049] Please refer to formulas (1) and (2), which are the expressions for the trajectory state space and trajectory action space, respectively: in, For the trajectory state space, The x-axis of the virtual unmanned underwater vehicle The vertical coordinate of the virtual unmanned underwater vehicle. The vertical coordinates of the virtual unmanned underwater vehicle. The lateral velocity of the virtual unmanned underwater vehicle. The longitudinal velocity of the virtual unmanned underwater vehicle. The vertical velocity of the virtual unmanned underwater vehicle. For the pitch angular velocity of the virtual unmanned underwater vehicle, The heading angular velocity of the virtual unmanned underwater vehicle.

[0050] (2) in, For trajectory action space, For the thruster rotation speed, For horizontal rudder angle, This refers to the rudder angle.

[0051] S202: Using the trajectory tracking controller, generate control data samples based on the first trajectory.

[0052] Here, it can be understood that the trajectory tracking controller controls the virtual unmanned underwater vehicle to track the first trajectory and generate control data samples.

[0053] Specifically, the trajectory tracking controller can generate and execute a trajectory tracking action based on the first trajectory and the current trajectory tracking state to obtain a new trajectory tracking state and a control reward value. Then, based on the current trajectory tracking state, the trajectory tracking action, the new trajectory tracking state, and the control reward value, the control data sample can be generated.

[0054] For example, control data samples may include ,in, This represents the current trajectory tracking status. For trajectory tracking actions, To control the reward value, This is the new trajectory tracking state.

[0055] In this embodiment, a control state space and a control action space for trajectory tracking are designed. The control state space may include three-dimensional coordinate control quantities and velocity control quantities. The three-dimensional coordinate control quantities include horizontal coordinate control quantities, vertical coordinate control quantities, and vertical coordinate control quantities. The velocity control quantities include horizontal velocity control quantities, vertical velocity control quantities, and vertical direction control quantities.

[0056] The control action space can include thruster speed control quantity and rudder angle control quantity, wherein the rudder angle control quantity includes horizontal rudder angle control quantity and directional rudder angle control quantity.

[0057] For example, please refer to formulas (4) and (5), which are expressions for the control state space and control action space, respectively: (4) in, To control the state space, This is the horizontal axis control value. This is the control value for the vertical axis. For vertical coordinate control quantities, For lateral speed control, For longitudinal speed control, This is the vertical control quantity.

[0058] (5) in, To control the motion space, For thruster speed control, This refers to the horizontal rudder angle control value. This refers to the rudder angle control value.

[0059] S203: Update the control data sample set based on the control data sample, generate trajectory data samples based on the first trajectory, and update the trajectory data sample set based on the trajectory data samples.

[0060] Since control data samples are generated during each training session, the control data sample set can be updated based on these control data samples.

[0061] Optionally, when generating trajectory data samples based on the first trajectory, the trajectory data samples can be generated based on the first trajectory and the corresponding trajectory reward value.

[0062] The trajectory reward value is determined based on the control reward value, and the trajectory reward value is the opposite of the control reward value.

[0063] As mentioned above, the trajectory generator and the trajectory tracking controller are trained in an adversarial manner. That is, the training objectives of the trajectory generator and the trajectory tracking controller can be understood as opposite. Therefore, in this application, the reward value function is not designed for the trajectory reward value, but is determined based on the control reward value.

[0064] As shown in formula (3), it is the expression for the relationship between trajectory reward value and control reward value.

[0065] (3) in, To control the reward value, This is the trajectory reward value.

[0066] After determining the trajectory reward value, trajectory data samples can be generated based on the first trajectory and the corresponding trajectory reward value. The trajectory data sample set can then be updated based on these trajectory data samples. It can be understood that since the trajectory generator training involves multiple sessions, each training session generates a trajectory data sample. Therefore, the trajectory data sample set can be updated based on the trajectory data samples generated in each training session, and the network parameters of the trajectory generator can be adjusted based on the trajectory data sample set.

[0067] S204: During the training of the trajectory generator, the network parameters of the trajectory tracking controller are fixed, and the network parameters of the trajectory generator are adjusted based on the trajectory data sample set; or... During the training of the trajectory tracking controller, the network parameters of the trajectory generator are fixed, and the network parameters of the trajectory tracking controller are adjusted based on the control data sample set.

[0068] Based on the above, when adjusting the network parameters of the trajectory generator and the trajectory tracking controller, the network parameters of the trajectory tracking controller can be fixed during the training of the trajectory generator, and the network parameters of the trajectory generator can be adjusted based on the trajectory data sample set. Here, the trajectory data sample set can be sampled to adjust the network parameters of the trajectory generator.

[0069] Alternatively, when training the trajectory tracking controller, the network parameters of the trajectory generator can be fixed, and the network parameters of the trajectory tracking controller can be adjusted based on the control data sample set. Similarly, the network parameters of the trajectory tracking controller can be adjusted by sampling the control data sample set.

[0070] Please see Figure 4 This is a schematic diagram illustrating an adversarial alternation training process, provided as an exemplary embodiment of this application. Figure 4 As shown, the trajectory generator generates the first trajectory, the trajectory tracking controller tracks the first trajectory, and generates the trajectory tracking status and reward value.

[0071] On the one hand, the negative of the reward value is used as the trajectory reward value of the first trajectory. Trajectory data samples are generated based on the first trajectory and the trajectory reward value, and the trajectory data samples are stored in the trajectory data sample set. Then, the network parameters of the trajectory generator are adjusted based on the trajectory data sample set.

[0072] On the other hand, the reward value is used as the control reward value. Control data samples are generated based on the current trajectory tracking state, trajectory tracking action, new trajectory tracking state, and trajectory tracking action. The control data samples are stored in the control data sample set, and the network parameters of the trajectory tracking controller are adjusted based on the control data sample set.

[0073] according to Figure 4 It can be seen that the training objective of the trajectory tracking controller is to track the first trajectory and maximize the reward value of trajectory tracking, while the training objective of the trajectory generator is to generate the first trajectory in a way that minimizes the reward value of the trajectory tracking controller for tracking this first trajectory. Therefore, in Figure 4 In the diagram, the reward value for the trajectory tracking controller is r, and the reward value for the trajectory generator is -r.

[0074] The following pseudocode illustrates the above multi-round training.

[0075] 1. Initialize the trajectory tracking controller Agent: 2. Initialize the trajectory generator Agent: 3. Initialize the control data sample set Trajectory data sample set 4 for t=1 to T do 5. Network parameters of the fixed trajectory tracking controller 6 for i=1 to K do 7. The trajectory generator generates the target trajectory: 8. The trajectory tracking controller generates actions based on the trajectory tracking strategy: 9. Based on the current state Execute actions To obtain a new state and control reward value 10 Data Samples Stored in control data sample set middle 11 Data Samples Stored in trajectory data sample set middle 12 Use Update the network parameters of the trajectory generator 13 end for 14. Network parameters of a fixed trajectory generator 15 for j=1 to N 16. The trajectory generator generates the target trajectory: 17. The trajectory tracking controller generates actions based on the trajectory tracking strategy: 18 Based on the current state Execute actions To obtain a new state and control reward value 19 Data Samples Stored in control data sample set middle 20 data samples Stored in trajectory data sample set middle 21 Using control data sample sets Update the network parameters of the trajectory tracking controller 22 end for 23 end for As can be seen from the above, in this application, the trajectory generator and the trajectory tracking controller are trained for T rounds. In each round of training, the network parameters of the trajectory tracking controller are fixed and the trajectory generator is trained K times. Then, the network parameters of the trajectory generator are fixed and the trajectory tracking controller is trained N times. In this way, the trained trajectory generator and the trained trajectory tracking controller can be obtained.

[0076] The values ​​of T, K, and N can be set according to actual needs and are not limited here.

[0077] S103: Obtain the navigation information of the real unmanned underwater vehicle corresponding to the virtual unmanned underwater vehicle, generate a target trajectory based on the navigation information using the trained trajectory generator, generate a trajectory tracking control strategy based on the target trajectory using the trained trajectory tracking controller, and control the real unmanned underwater vehicle according to the trajectory tracking strategy.

[0078] Here, the navigation information of a real unmanned underwater vehicle can include environmental information (such as obstacle information, water flow speed information, etc.) and mission objectives (such as destination location, reaching the destination in the shortest possible time, etc.).

[0079] In practice, the trained trajectory generator and the trained trajectory tracking controller can be deployed on a real unmanned underwater vehicle. In this way, the trained trajectory generator can generate a target trajectory based on the navigation information, and the trained trajectory tracking controller can generate a trajectory tracking control strategy based on the target trajectory. The trajectory tracking control strategy is used to control the real unmanned underwater vehicle, such as controlling the speed, rudder angle direction (horizontal and vertical), and thruster torque.

[0080] Thus, the trained trajectory generator and trajectory tracking controller obtained based on the above-mentioned adversarial alternating training mechanism can control the real unmanned underwater vehicle, thereby achieving obstacle avoidance while completing the mission objective, which can improve the control accuracy of the real unmanned underwater vehicle.

[0081] Corresponding to the aforementioned embodiments of the trajectory tracking control method for unmanned underwater vehicles, this application also provides embodiments of a trajectory tracking control device for unmanned underwater vehicles.

[0082] Please refer to Figure 5 This is a schematic diagram illustrating the structure of a trajectory tracking control device for an unmanned underwater vehicle, as shown in an exemplary embodiment of this application. Figure 5 As shown, the trajectory tracking and control device 500 of the unmanned underwater vehicle includes: Module 510 is used to build a trajectory generator and a trajectory tracking controller for a virtual unmanned underwater vehicle based on a reinforcement learning network. Training module 520 is used to perform multiple rounds of training on the trajectory generator and the trajectory tracking controller to obtain a trained trajectory generator and a trained trajectory tracking controller; wherein, each round of training includes adversarial alternating training for the trajectory generator and the trajectory tracking controller; during the adversarial alternating training process, the trajectory generator is used to generate trajectory data samples until the reward value obtained by tracking the trajectory data samples by the trajectory tracker is minimized, and the trajectory tracking controller is used to generate control data samples with the maximum reward value based on the trajectory data samples; The control module 530 is used to acquire the navigation information of the real unmanned underwater vehicle corresponding to the virtual unmanned underwater vehicle, generate a target trajectory based on the navigation information using the trained trajectory generator, generate a trajectory tracking control strategy based on the target trajectory using the trained trajectory tracking controller, and control the real unmanned underwater vehicle according to the trajectory tracking strategy.

[0083] In some implementations, the training module 520 is specifically used for: The network parameters of one of the trajectory generator and the trajectory tracking controller are fixed in sequence, and the other is trained multiple times to adjust the network parameters of the other.

[0084] In some implementations, the training module 520 is specifically used for: Using the trajectory generator, a first trajectory is generated based on the virtual navigation information of the virtual unmanned underwater vehicle; Using the trajectory tracking controller, control data samples are generated based on the first trajectory; The control data sample set is updated based on the control data sample, and a trajectory data sample is generated based on the first trajectory, and the trajectory data sample set is updated based on the trajectory data sample. During the training of the trajectory generator, the network parameters of the trajectory tracking controller are fixed, and the network parameters of the trajectory generator are adjusted based on the trajectory data sample set; or... During the training of the trajectory tracking controller, the network parameters of the trajectory generator are fixed, and the network parameters of the trajectory tracking controller are adjusted based on the control data sample set.

[0085] In some implementations, when the training module 520 generates control data samples based on the first trajectory using the trajectory tracking controller, it is specifically used for: The trajectory tracking controller generates and executes a trajectory tracking action based on the first trajectory and the current trajectory tracking state, thereby obtaining a new trajectory tracking state and a control reward value. The control data sample is generated based on the current trajectory tracking state, the trajectory tracking action, the new trajectory tracking state, and the control reward value; When the training module 520 generates trajectory data samples based on the first trajectory and updates the trajectory data sample set based on the trajectory data samples, it is specifically used for: The trajectory data sample is generated based on the first trajectory and the corresponding trajectory reward value, and the trajectory data sample set is updated according to the trajectory data sample; the trajectory reward value is determined based on the control reward value.

[0086] In some implementations, the trajectory reward value and the control reward value are opposites of each other.

[0087] In some implementations, the training module 520 is specifically used for: Construct the trajectory state space and trajectory action space of the virtual unmanned underwater vehicle; the trajectory state space includes position state, velocity state and angular velocity state, and the trajectory action space includes thruster speed and rudder angle; Within the trajectory state space and trajectory action space, the trajectory generator generates the first trajectory based on the virtual navigation information.

[0088] In some implementations, the virtual unmanned underwater vehicle has the same dynamic model as the real unmanned underwater vehicle.

[0089] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0090] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0091] Corresponding to the trajectory tracking and control method for unmanned underwater vehicles described above, this disclosure also provides a computer device, such as... Figure 6 The diagram shown is a structural schematic of a computer device provided in an embodiment of this disclosure. Figure 6As shown, the computer device 600 includes a processor 610, an internal bus 620, memory 630, a network interface 640, and non-volatile memory 660, and may also include other hardware required for its functions. One or more embodiments of this specification can be implemented in software, for example, the processor 610 reads the corresponding computer program from the non-volatile memory 660 into the memory 630 and then runs it. Of course, besides software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0092] The memory 630, also known as internal memory, is used to temporarily store the computational data in the processor 610, as well as the data exchanged with non-volatile memory 660 such as hard disk. The processor 610 exchanges data with the non-volatile memory 660 through the memory 630.

[0093] In this embodiment, memory 630 is specifically used to store application code that executes the solution of this application, and its execution is controlled by processor 610. That is, when the computer device is running, processor 610 communicates with network interface 640, memory 630 and non-volatile memory 660 through internal bus 620, so that processor 610 executes the application code stored in memory 630 and non-volatile memory 660, thereby executing the trajectory tracking control method of unmanned underwater vehicle described in the above method embodiment.

[0094] Processor 610 may be an integrated circuit chip with signal processing capabilities. The aforementioned processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware microservices. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor.

[0095] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the computer device 600. In other embodiments of this application, the computer device 600 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0096] This disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the trajectory tracking control method for an unmanned underwater vehicle described in the above-described method embodiments. The storage medium can be either volatile or non-volatile computer-readable storage.

[0097] This disclosure also provides a computer program product carrying program code. The program code includes instructions that can be used to execute the steps of the trajectory tracking control method for the unmanned underwater vehicle in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0098] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0099] The embodiments of the subject matter and functional operation described in this specification can be implemented in the following ways: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. Alternatively or additionally, the program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information and transmit it to a suitable receiving device for execution by the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof.

[0100] The processing and logic flow described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform corresponding functions by operating on input data and generating output. The processing and logic flow can also be executed by dedicated logic circuitry—such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and the device can also be implemented as dedicated logic circuitry.

[0101] Computers suitable for executing computer programs include, for example, general-purpose and / or special-purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory and / or random access memory. Basic computer microservices include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or the computer will be operatively coupled to such mass storage devices to receive data from or transfer data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0102] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0103] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0104] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and microservices in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program microservices and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0105] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0106] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A trajectory tracking and control method for an unmanned underwater vehicle, characterized in that, include: A trajectory generator and a trajectory tracking controller for a virtual unmanned underwater vehicle are constructed based on a reinforcement learning network. The trajectory generator and the trajectory tracking controller are trained in multiple rounds to obtain a trained trajectory generator and a trained trajectory tracking controller. Each round of training includes adversarial alternation training for the trajectory generator and the trajectory tracking controller. During the adversarial alternation training, the trajectory generator generates trajectory data samples until the reward value obtained by tracking the trajectory data samples by the trajectory tracking controller is minimized. The trajectory tracking controller then generates control data samples with the maximum reward value based on the trajectory data samples. The system acquires navigation information of the real unmanned underwater vehicle corresponding to the virtual unmanned underwater vehicle, generates a target trajectory based on the navigation information using the trained trajectory generator, generates a trajectory tracking control strategy based on the target trajectory using the trained trajectory tracking controller, and controls the real unmanned underwater vehicle according to the trajectory tracking control strategy. The adversarial alternation training for the trajectory generator and the trajectory tracking controller includes: Using the trajectory generator, a first trajectory is generated based on the virtual navigation information of the virtual unmanned underwater vehicle. Then, using the trajectory tracking controller, a trajectory tracking action is generated and executed based on the first trajectory and the current trajectory tracking state to obtain a new trajectory tracking state and a control reward value. Based on the current trajectory tracking state, the trajectory tracking action, the new trajectory tracking state, and the control reward value, the control data sample is generated and stored in the control data sample set; The trajectory data sample is generated based on the first trajectory and the corresponding trajectory reward value, and the trajectory data sample set is updated according to the trajectory data sample; the trajectory reward value is determined based on the control reward value; During the training of the trajectory generator, the network parameters of the trajectory tracking controller are fixed, and the network parameters of the trajectory generator are adjusted based on the trajectory data sample set; or... During the training of the trajectory tracking controller, the network parameters of the trajectory generator are fixed, and the network parameters of the trajectory tracking controller are adjusted based on the control data sample set.

2. The method according to claim 1, characterized in that, The trajectory reward value and the control reward value are opposites of each other.

3. The method according to claim 1, characterized in that, The step of generating a first trajectory based on the virtual navigation information of the virtual unmanned underwater vehicle using the trajectory generator includes: Construct the trajectory state space and trajectory action space of the virtual unmanned underwater vehicle; the trajectory state space includes position state, velocity state and angular velocity state, and the trajectory action space includes thruster speed and rudder angle; Within the trajectory state space and trajectory action space, the trajectory generator generates the first trajectory based on the virtual navigation information.

4. The method according to claim 1, characterized in that, The virtual unmanned underwater vehicle has the same dynamic model as the real unmanned underwater vehicle.

5. A trajectory tracking and control device for an unmanned underwater vehicle, characterized in that, include: The module is used to build a trajectory generator and a trajectory tracking controller for the virtual unmanned underwater vehicle based on a reinforcement learning network. A training module is used to perform multiple rounds of training on the trajectory generator and the trajectory tracking controller to obtain a trained trajectory generator and a trained trajectory tracking controller. Each round of training includes adversarial alternating training for the trajectory generator and the trajectory tracking controller. During the adversarial alternating training, the trajectory generator generates trajectory data samples until the reward value obtained by tracking the trajectory data samples by the trajectory tracking controller is minimized. The trajectory tracking controller then generates control data samples with the maximum reward value based on the trajectory data samples. The control module is used to acquire the navigation information of the real unmanned underwater vehicle corresponding to the virtual unmanned underwater vehicle, generate a target trajectory based on the navigation information using the trained trajectory generator, generate a trajectory tracking control strategy based on the target trajectory using the trained trajectory tracking controller, and control the real unmanned underwater vehicle according to the trajectory tracking control strategy. The training module is specifically used for: Using the trajectory generator, a first trajectory is generated based on the virtual navigation information of the virtual unmanned underwater vehicle. Then, using the trajectory tracking controller, a trajectory tracking action is generated and executed based on the first trajectory and the current trajectory tracking state to obtain a new trajectory tracking state and a control reward value. Based on the current trajectory tracking state, the trajectory tracking action, the new trajectory tracking state, and the control reward value, the control data sample is generated and stored in the control data sample set; The trajectory data sample is generated based on the first trajectory and the corresponding trajectory reward value, and the trajectory data sample set is updated according to the trajectory data sample; the trajectory reward value is determined based on the control reward value; During the training of the trajectory generator, the network parameters of the trajectory tracking controller are fixed, and the network parameters of the trajectory generator are adjusted based on the trajectory data sample set; or... During the training of the trajectory tracking controller, the network parameters of the trajectory generator are fixed, and the network parameters of the trajectory tracking controller are adjusted based on the control data sample set.

6. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the trajectory tracking control method for any one of claims 1-4 for an unmanned underwater vehicle.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the trajectory tracking control method for any one of claims 1-4 for an unmanned underwater vehicle.

Images