Trajectory tracking obstacle avoidance control method and device for AUV (Autonomous Underwater Vehicle) carrying front sonar and medium

Abstract

The invention discloses a trajectory tracking obstacle avoidance control method and device for an AUV (Autonomous Underwater Vehicle) carrying a front sonar and a medium, and relates to the field of AUV control. The objective of the invention is to solve the problem that the existing trajectory tracking and obstacle avoidance tasks cannot be freely switched. According to the trajectory tracking obstacle avoidance control method for the AUV carrying the front sonar, the state of the AUV is input into the controller, the action corresponding to the state is obtained, and control over the AUV is achieved; the state of the AUV is a state under a trajectory tracking task or a state under an obstacle avoidance task.

Description

Technical Field

[0001] This application belongs to the field of AUV control technology. Background Art

[0002] Currently, as AUVs (untethered underwater vehicles) have become a strategic development priority for various countries in both military and civilian fields, AUV tracking and obstacle avoidance control have become crucial as the basic capabilities and safety prerequisites for AUVs to perform various complex tasks. In actual tasks, AUVs are required to quickly track established trajectories and avoid unknown obstacles based on information detected by the forward sonar. Traditional control methods are often limited by precise mathematical models. When faced with new working environments, a large number of controller parameters need to be adjusted, and the ability to freely switch between tracking and obstacle avoidance tasks within the same environmental framework cannot be achieved, which greatly increases the time and complexity of the project. However, the development of existing deep reinforcement learning methods in the field of AUV control is restricted by the instability of the training process and the weak ability to adapt to the environment.

[0003] A review of current domestic and international references shows that relevant research mainly focuses on single-task environments of tracking or obstacle avoidance, and less consideration is given to AUV thruster failures. In a 2020 invention patent published by Wang Weiran, Yan Jinghao, Zhang Qian, and Tan Fei, titled "Autonomous Underwater Vehicle Trajectory Tracking Control Method Based on Deep Reinforcement Learning" (patent application number: CN201810535773.8; publication number: CN108803321B), a method combining the expected trajectory error state with the deep reinforcement learning framework is introduced. This method is only applicable to the ideal environment of a single tracking task without thruster failures and external disturbances. In a 2023 invention patent, Zhao Zhongyuan, Lü Zhangfei, Guo Dali, Gao Wang, Wang Yan, and Zhao Haihua published in 2023, titled "An Event-Triggered Trajectory Tracking Control Method for Autonomous Underwater Vehicles" (patent application number: CN202310243719.7; publication number: CN116224798A), introduced an event-triggered mechanism into AUV trajectory tracking control and designed a controller combining backstepping and global sliding mode control. However, the patent simplified the AUV mathematical model to three degrees of freedom, making it less adaptable to real-world underwater environments. In the field of AUV obstacle avoidance, an invention patent published in 2022 by Zhang Feihu, Yang Dianyu, Cheng Chensheng, Wang Can, and Wang Zhiliang, titled "A Method for Autonomous Obstacle Avoidance of AUVs Based on Reinforcement Learning" (patent application number: CN202110115197.3; publication number: CN112947421B), introduces an obstacle avoidance strategy that comprehensively considers multiple factors such as mission execution time, deviation angle from the main route, and obstacle distance. In the same year, a patent published by Hao Kun, Zhao Jiale, and Wang Beibei, titled "A Local Obstacle Avoidance Method for AUVs Based on Collision Detection and Artificial Potential Field Method," uses a collision detection model to screen key path points in the global path and combines the artificial potential field method to guide the AUV to avoid obstacles locally. Although the above obstacle avoidance method can avoid obstacles in specific environments, it is still limited by various boundary conditions and cannot be directly combined with tracking tasks.

[0004] In summary, the existing AUV trajectory tracking and obstacle avoidance control methods have the following problems:

[0005] 1. In trajectory tracking tasks, classical control methods are limited by precise mathematical models and the strong sensitivity of parameters to environmental changes, which inevitably increases the project workload. The current development of deep reinforcement learning methods is also limited to ideal environments.

[0006] 2. For local obstacle avoidance tasks, existing deep reinforcement learning methods typically create environmental models by setting a starting point and an end point and placing obstacles along the path. This model does not conform to the actual underwater environment. Furthermore, applications suffer from low obstacle avoidance success rates and low training efficiency.

[0007] 3. Existing methods lack a solution for the continuous task of trajectory tracking and obstacle avoidance within the same time domain. Performing this task continuously is highly risky when the two task state spaces and algorithms are completely separated. Therefore, a method is needed to enable the flexible switching between the two task states within the same environment. Summary of the Invention

[0008] This application aims to solve the problems that the existing AUV trajectory tracking task is limited by mathematical models and parameter changes, resulting in a large workload; the environmental model in the obstacle avoidance task is inconsistent with the actual situation; and the trajectory tracking and obstacle avoidance tasks cannot be switched freely. Now, a trajectory tracking and obstacle avoidance control method for AUV equipped with a front sonar is provided.

[0009] The trajectory tracking and obstacle avoidance control method of an AUV equipped with a front sonar includes:

[0010] Input the state of the AUV into the controller, obtain the action corresponding to the state, and realize the control of the AUV;

[0011] The state of the AUV is the state under the trajectory tracking task or the state under the obstacle avoidance task;

[0012] The expression of the controller is:

[0013]

[0014] Among them, π * represents the optimal control strategy of AUV, π represents the control strategy of AUV, V(s) is the state value function, Q(s,a) is the state-action value function, S and A are the observable state space and action space respectively, s and a are the state and action of AUV respectively.

[0015] Furthermore, the expression of the above state value function V(s) is:

[0016]

[0017] The expression of the state-action value function Q(s,a) is:

[0018]

[0019] Where, represents the sum of the expected rewards at the current time t, γ is the discount factor and γ∈(0,1], s k and a k are the state and action of AUV at time k, s t and a t are the state and action of AUV at time t, and r(s,a) represents the reward function.

[0020] Furthermore, the above reward function r(s,a) is the reward function r under the trajectory tracking task s Or the reward function under the obstacle avoidance task

[0021] The reward function r under the trajectory tracking task s :

[0022]

[0023] The reward function under the obstacle avoidance task

[0024]

[0025] Where, ||e s ||1 represents the error state e of the trajectory tracking task s 1-norm, ζ e and ζ c are weight coefficients,

[0026] It represents the function that penalizes the behavior of AUV approaching the dangerous area and entering the collision range.

[0027] represents a function that rewards the AUV for approaching the reference trajectory without entering the dangerous area.

[0028] A function that penalizes the AUV for not reaching the target area before the state collection ends.

[0029] Furthermore, the above function punishes the AUV for approaching the dangerous area and entering the collision range. The expression is:

[0030]

[0031] in, is the relative distance between the AUV and the obstacle, Indicates the scope of the danger zone. represents the collision range, φ0 represents the radius of the obstacle, ζ o and are weight parameters, and are the relative pitch angle and relative yaw angle between the AUV and the obstacle, E r A trigger flag indicating whether the AUV has entered a dangerous area, and includes:

[0032]

[0033] Furthermore, the above function rewards the AUV for approaching the reference trajectory without entering the dangerous area. The expression is:

[0034]

[0035] Among them, s for The weight parameter, h s is the set boundary constant, sgn(*) represents the sign function, and ρ is the relative distance between the AUV and the trajectory point.

[0036] Furthermore, the function r that penalizes the AUV for not reaching the target area before the state collection ends is t The expression is:

[0037]

[0038] Among them, t for The weight parameter, E t A trigger flag indicating whether the AUV has entered the target area, and has:

[0039]

[0040] ρ′ is the relative distance between the AUV and the center of the target area, φ t is the radius of the target area.

[0041] Furthermore, the state expression of the above AUV is:

[0042]

[0043] Among them, s1 represents the state under the trajectory tracking task, and the expression is:

[0044] s1=[g s f s h s e s ],

[0045] g s 、f s 、h s and e s They are speed state, actuator fault state, disturbance state and error state of trajectory tracking task respectively;

[0046] s2 represents the state under the obstacle avoidance task, and its expression is:

[0047]

[0048] ρ is the relative distance between the AUV and the trajectory point, ρ′ is the relative distance between the AUV and the center of the target area, and are the relative distance, relative pitch angle, and relative yaw angle between the AUV and the obstacle;

[0049] E d Indicates whether the AUV detects an obstacle. d When the value changes from 0 to 1, it means that the AUV detects an obstacle.

[0050] Furthermore, the training method of the controller includes:

[0051] In the trajectory tracking task, the TD3 algorithm is used to train the controller;

[0052] In the obstacle avoidance task, the controller is trained using a maximum entropy reinforcement learning strategy.

[0053] An AUV trajectory tracking and obstacle avoidance control device equipped with a front sonar, the AUV trajectory tracking and obstacle avoidance control device equipped with a front sonar includes a processor and a memory, the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the above-mentioned AUV trajectory tracking and obstacle avoidance control method equipped with a front sonar.

[0054] A computer storage medium stores at least one instruction, which is loaded and executed by a processor to implement the above-mentioned AUV trajectory tracking and obstacle avoidance control method equipped with a front sonar.

[0055] The beneficial effects of this application are:

[0056] This application takes the trajectory tracking and obstacle avoidance task as an example. Based on the event trigger mechanism, an obstacle-related state is generated when an obstacle is detected, and the event trigger mechanism is integrated into the setting of the obstacle avoidance task reward function.

[0057] To ensure the unity and continuity of the state space, event trigger flags are used to segment the different states required by the two tasks during the same joint task training process, thereby realizing state classification collection and automatic task switching under continuous tasks in training and experiments.

[0058] Based on the multi-experience pool mechanism, the fusion of two different deep reinforcement learning algorithms in an event-triggered framework helps to give full play to the advantages of the algorithms in different task environments, accelerates the overall convergence speed and improves sample utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1Schematic diagram of the AUV mission environment;

[0060] Figure 2 Schematic diagram of the AUV collision avoidance mechanism;

[0061] Figure 3 This is a schematic diagram of simulation parameters for an embodiment of the present application;

[0062] Figure 4 Schematic diagram of the average reward during the AUV 3D tracking task training process, where method 1 is the TD3 algorithm and method 2 is the SAC algorithm;

[0063] Figure 5 Schematic diagram of the average reward of the AUV 3D obstacle avoidance task training process, where method 1 is the SAC algorithm and method 2 is the TD3 algorithm;

[0064] Figure 6 A roadmap for the S-TD3 algorithm proposed in this application for the AUV 3D tracking and obstacle avoidance task;

[0065] Figure 7 This is a graph showing the force and torque changes of the AUV output during the AUV 3D tracking and obstacle avoidance task using the S-TD3 algorithm proposed in this application.

[0066] Figure 8 This is the schematic diagram of the trajectory tracking and obstacle avoidance control principle of an AUV equipped with a front sonar. DETAILED DESCRIPTION

[0067] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other in the absence of conflict.

[0068] Reference Figures 1 to 8 Specifically describing this embodiment, a trajectory tracking and obstacle avoidance control method for an AUV equipped with a front-mounted sonar under the influence of a thruster failure is described in this embodiment. Based on the motion state of the 5-DOF AUV and the description of the thruster failure, a complete AUV trajectory tracking and obstacle avoidance rule and training strategy are established within the deep reinforcement learning framework. The optimal control strategy is obtained by training under the corresponding state space and reward function. The details are as follows:

[0069] Step 1: Use coordinate transformation and system modeling methods to obtain the kinematic and dynamic models of the underactuated AUV, thereby establishing the mission environment model and Markov decision process. The details are as follows:

[0070] Step 1.1: The velocity directions of the five-degree-of-freedom AUV in its own coordinate system are the sway velocity u, the lateral velocity v, the heave velocity w, the pitch angular velocity q, and the bow angular velocity r. Since the actuator of the underactuated AUV only provides longitudinal thrust, pitch moment, and bow moment, let τ i (i=u,q,r) is the input of the actuator in the tail rudder and two rudders, a total of three directions. Therefore, the five-degree-of-freedom kinematic and dynamic model of the underactuated AUV is described as:

[0071]

[0072] Where, It represents the state of the AUV in the earth-fixed coordinate system, namely surge, roll, heave displacement, pitch angle and yaw angle.

[0073] Combined with the above thruster failure, the five-degree-of-freedom dynamic model of the underactuated AUV can be written as:

[0074]

[0075] Among them, m 11 、m 22 and m 33 are the moments of inertia about the x, y and z axes, m 55 represents the inertial coupling term along the xy plane, m 66 represents the inertial coupling term along the yz plane;

[0076] f u 、f v 、f w 、f q 、f r are the sum of the fluid resistance and friction terms of the AUV in u, v, w, q, and r degrees of freedom, respectively;

[0077] τ u represents the thrust generated by the actuator (propeller) in the velocity direction u, τ q and τ r They represent the torques generated by the actuator (cross rudder) in the speed directions q and r, respectively. Since the actuator may encounter failures in actual operation, ω i (i=u,q,r) is represented as the input of the actuator, then the output of the actuator with fault can be described as:

[0078]

[0079] Among them, λ ih ∈(0,1] represents the multiplicative actuator faults in the u, q, and r directions. Indicates additive actuator faults generated in the u, q, and r directions;

[0080] GM L 、 g、ρ e They represent longitudinal metacentric height, displacement, gravitational acceleration and water density respectively.

[0081] Step 1.2: In order to enable the AUV to autonomously avoid unknown obstacles based on the reference trajectory, it is necessary to model the two mission states separately and introduce an event trigger mechanism for conversion. The specific process is as follows:

[0082] According to the attached Figure 1 As shown in Figure 1, the mission environment state consists of two parts: the relative position relationship between the AUV and the point on the reference trajectory and the relative position relationship between the AUV and the detected obstacle, including relative distance, relative pitch angle and relative yaw angle. At the same time, each unknown obstacle is simplified as a sphere, and the position coordinates of the detected obstacle are (x o ,y o ,z o ). Figure 1 The angle and distance parameters in can be obtained:

[0083]

[0084] Among them, (x r ,y r ,z r ) represents the coordinate difference of the trajectory point relative to the AUV in the carrier coordinate system; represents the transformation matrix between the earth-fixed coordinate system and the carrier coordinate system; (Δx, Δy, Δz) represents the coordinate difference of the trajectory point relative to the AUV in the earth-fixed coordinate system, Δx = x m -x, Δy=y m -y, Δz=z m -z,(x m ,y m ,z m ) represents the coordinates of the trajectory point, and (x, y, z) represents the coordinates of the AUV in the earth-fixed coordinate system; Indicates the coordinate difference between the AUV and the obstacle in the carrier coordinate system; Indicates the coordinate difference between the AUV and the obstacle in the earth's fixed coordinate system.

[0085] Combined with attachment Figure 1 The analysis of the three-dimensional geometric relationship finally yields the relative distance ρ, relative pitch angle β, and relative yaw angle α between the AUV and the trajectory point:

[0086]

[0087] Among them, arctan2(a,b) is used to calculate the arc tangent value of a / b within the limited range (-π,π]. Similarly, the relative relationship between the AUV and the obstacle can be obtained.

[0088] Step 1.3: Apply reinforcement learning to the continuous task decision-making problem in continuous state space and action space, and model the AUV's detection process in an unknown environment as a partially observable Markov decision search, which can be defined by a four-tuple [S, A, p(s, s′, a), r(s, a)], where S and A are the observable state space and action space, respectively; p(s, s′, a): S×S×A→[0,∞) represents the probability density of executing action a∈A in the current state s∈S to reach the next state s′∈S; r(s, a) represents the reward function, and the corresponding reward is given by the environment at each state transition.

[0089] The goal of reinforcement learning is to update the policy network π(s) by maximizing the sum of expected rewards E at the current time t t ) approximates the optimal strategy, and the obtained optimal control strategy π * (s t ) is the AUV motion controller, which can be expressed as:

[0090]

[0091] Among them, ρ π Denotes the strategy π(s t ) includes the state edge of the trajectory distribution, and γ∈(0,1] is the discount factor.

[0092] The state value function V(s) is defined as:

[0093]

[0094] The state-action value function Q(s,a) can be defined as:

[0095]

[0096] Finally, formula (5) can be equivalent to:

[0097]

[0098] Using the reinforcement learning method, the strategy network π is trained under the conditions of the corresponding environment state and action to converge to the optimal strategy π to achieve the task goal. * .

[0099] Step 2: Set the transition conditions between the trajectory tracking task and the obstacle avoidance task based on the event trigger mechanism, and obtain the motion controller π * (s t ) Design appropriate controller input states and reward functions.

[0100] Step 2.1: For AUV motion control, the input state selection should be based on the speed state, actuator fault state, and disturbance state that directly affect the AUV's motion, and other states should be selected as supplementary based on the actual mission requirements. This implementation method customizes two different state spaces and explains their specific application strategies in AUV motion control. The details are as follows:

[0101] (1) Speed ​​state, actuator fault state and disturbance state

[0102] Referring to the underactuated AUV model, the general velocity state of the AUV can be defined as:

[0103]

[0104] The fault conditions of the three rudder thrusters can be defined as:

[0105]

[0106] The five-degree-of-freedom AUV has disturbances from five directions. After training the network model, the control strategy needs to learn to output appropriate actions a to indirectly offset the impact of the disturbance. The disturbance state can be defined as:

[0107] h s =[τ du τ dv τ dw τ dq τ dr ] (11).

[0108] (2) Trajectory tracking task related status

[0109] To complete the task of trajectory tracking, the error between the actual motion trajectory of the AUV and the reference trajectory needs to approach the set value. In the selection of the error state, according to the definition of error in control theory, the error state of the trajectory tracking task is obtained as:

[0110]

[0111] Among them, j d Indicates the UUV position status expected value.

[0112] For the error state of the trajectory tracking task, select:

[0113]

[0114] Where ρ, β, and α represent the relative distance, relative pitch angle, and relative yaw angle relative to the reference trajectory in the line-of-sight guidance (Los) method, respectively.

[0115] The combination of the above error state, speed state and disturbance state s1=[g s f s h s e s ] is the complete input state of the controller in the trajectory tracking task state.

[0116] (3) Obstacle avoidance task related status

[0117] The goal of real-time obstacle avoidance in trajectory tracking tasks is to control the AUV from identifying an obstacle to navigating to a target point away from the obstacle, avoiding collisions while maintaining a minimum distance from the reference trajectory. To effectively simulate the process of an AUV detecting obstacles, an event triggering mechanism is introduced and the state is constructed based on the relative distance to the obstacle, relative pitch angle, and relative yaw angle.

[0118] By the attached Figure 2 As shown, it is assumed that the maximum detection range of the AUV's front sonar is ρ max , E d As an event trigger flag, that is, when E d When the value changes from 0 to 1, it means that the AUV detects an obstacle and:

[0119]

[0120] in, represents the relative distance from the AUV to the obstacle, and φ0 represents the radius of the obstacle.

[0121] After detecting an obstacle, the AUV’s autonomous obstacle avoidance task can be summarized as: Figure 2 Find a path to reach the set target area within the safe area. The target area is simplified to a radius of φ t The center of the sphere is selected at the point on the reference trajectory 15 seconds after the intersection point q of the reference trajectory and the outermost spherical surface. This ensures that the vehicle is far away from obstacles and prevents the waste of resources caused by the distance from the reference trajectory after completing the obstacle avoidance. t As an event trigger mark, when the AUV enters the target area, E t The value of changes from 0 to 1:

[0122]

[0123] Note: The AUV studied in this embodiment only considers the front sonar. When the AUV passes through the maximum detection range for the second time, Ed Remain unchanged. Starting from the moment the AUV detects an obstacle through the front sonar, E d =1 will remain until step=step end At this time, the control flag E d becomes 0.

[0124] In the above formula, ρ′ represents the relative distance of the AUV relative to the center of the target area, β′ and α′ represent the relative pitch angle and relative yaw angle relative to the center of the target area, respectively. The calculation method is consistent with the line-of-sight guidance (Los) method introduced in step 1.1.

[0125] Finally, combining the speed state, actuator fault state, and disturbance state, the input state of the controller in the obstacle avoidance task state is obtained as follows:

[0126]

[0127] (4) Final state

[0128] In order to simulate the real obstacle avoidance situation, the AUV can only obtain the state related to obstacle avoidance after detecting the obstacle. d The final input state of the controller is:

[0129]

[0130] Note: Assume that the center of the target area is in the time domain, which is the end mark of the collision avoidance state collection in the network model training step. end , and the system state before the AUV detects the obstacle and after the collision avoidance mission is completed is the state related to the trajectory tracking mission. end Only used in the process of obtaining the state related to the obstacle avoidance task and training the network model. In the actual simulation environment, the obstacle avoidance task end flag and event trigger flag E t That is, in the simulation, E t When it becomes 1, the obstacle avoidance mission ends.

[0131] Step 2.2: In this embodiment, directly define the AUV motion controller π * (s t ) is the control force and torque in the underactuated AUV model.

[0132]

[0133] Due to the independence and continuity of the tracking task and the collision avoidance task, a hierarchical reward function is designed. Two sets of reward functions are designed in combination with the event trigger mechanism to evaluate the actions of the AUV under two different task states.

[0134] First, for an AUV in a trajectory tracking state, the error with the reference trajectory and the energy consumption of the AUV itself should be considered. The reward function can be set as:

[0135]

[0136] Among them, ||e s ||1 represents e s The 1-norm of the reference trajectory is larger, the smaller the reward is. In addition, in order to reduce energy loss without affecting the control performance, that is, to reduce the force and torque value of the actuator, The term also imposes a negative reward as a penalty for excessive energy consumption. e With ζ c Represents the weight coefficient.

[0137] Secondly, for the AUV in the collision avoidance task state, the reward function should be set to take into account:

[0138] ①AUV approaches the dangerous area and enter the collision range punishing behavior;

[0139] ② Reward the AUV for approaching the reference trajectory without entering the dangerous area;

[0140] ③For AUV at time node step end Penalty for not reaching the target area before the target area is reached;

[0141] Regarding ①, Figure 3 The introduction event trigger flag E r To indicate whether the AUV has entered a dangerous area.

[0142]

[0143] This part of the reward function can be set as:

[0144]

[0145] Among them, o and Both are weight parameters, giving AUVs that travel within the collision radius a larger constant penalty, while in the danger zone, the closer the AUV is to the obstacle, the smaller the reward. and The term indicates that the reward decreases when the AUV's heading direction (including pitch angle and yaw angle) is facing the obstacle.

[0146] Regarding ②, the AUV is required to plan a path close to the reference trajectory during the obstacle avoidance process, as shown in the attached figure. Figure 2The actual trajectory (2) in the reference trajectory is considered, while the actual trajectory (1) that is relatively far from the reference trajectory is discarded. The reward function can be set as:

[0147]

[0148] Among them, s for The weight parameter, h s is the constant indicating the set boundary value, and sgn(*) is the sign function.

[0149] In the obstacle avoidance task training, the distance between the AUV and the reference trajectory is kept as low as possible. s Instead of making the AUV closer and closer to the reference trajectory during model training, which would cause excessive interference to the obstacle avoidance task and reduce the generalization performance of the network model, the reward is kept within the range of the boundary (that is, if the condition is met, a fixed value reward is given).

[0150] For ③, combined with event trigger flag E t You can get:

[0151]

[0152] In the obstacle avoidance task training, if at the time node step end Front E t If it is always 0, the AUV will be penalized until it can successfully reach the target area.

[0153] Therefore, the complete reward function during the collision avoidance task is:

[0154]

[0155] Step 3: Improve the single-task training strategy based on the deep reinforcement learning framework, and select the state space and reward function corresponding to the task to train the optimal policy network.

[0156] In view of the complexity of the task space and reward function, this implementation introduces multiple experience pools and event triggering mechanisms into the algorithm, and proposes the S-TD3 algorithm that integrates the TD3 algorithm and the SAC algorithm in the same task environment.

[0157] First, in the trajectory tracking task, the policy network π of the TD3 algorithm ω (a t |s t ) and state-action value network They interact with each other during training and continuously change the network parameters θ1, θ2, ω to make the policy network converge to the optimal state.

[0158] When the AUV does not detect an obstacle (E d= 0), the collected states are used to train the AUV to track the reference trajectory, and the state of the obstacles detected by the AUV (E d = 1) until the AUV completes the obstacle avoidance mission and reaches the target area (E t =1), and the states collected during this period are used to train the AUV's obstacle avoidance capability.

[0159] Considering the powerful exploration capability and robustness of the Maximum Entropy Reinforcement Learning (SAC) strategy, it has significant advantages in dealing with the complex state space and reward function of obstacle avoidance. Therefore, it is used in obstacle avoidance tasks.

[0160] This implementation proposes a method for using an event-triggered mechanism to generate obstacle-related states in the continuous AUV trajectory tracking and obstacle avoidance task, as well as a method for using this mechanism as a marker for hierarchical training in deep reinforcement learning. This method also establishes the complete state space required to complete the task. Furthermore, a comprehensive set of rules for AUV trajectory tracking and obstacle avoidance is proposed, taking into account AUV actuator failures. A reward function is set based on the vehicle's own state and its relative state to obstacles, combined with an event-triggered mechanism. This achieves state classification and collection in continuous tasks, automatic task switching, and training to obtain an optimal deep reinforcement learning controller.

[0161] Example:

[0162] This application is further explained by taking a specific AUV trajectory tracking and obstacle avoidance continuous task problem in an underwater environment as an example and combining with simulation drawings.

[0163] example Figure 3 Introduce the setting of simulation parameters in this embodiment, including network parameters, weight parameters in reward function, etc. Figure 4 with example Figure 5 The training of trajectory tracking and obstacle avoidance tasks using the deep reinforcement learning framework is shown. It can be seen that after a period of training, the average reward converges to near 0, and the optimal strategy for both tasks is found.

[0164] The simulation results are as follows Figure 6 As shown, the randomly selected reference trajectory is:

[0165]

[0166] As can be seen from the figure, the AUV quickly approaches the reference trajectory from the starting point, starts the obstacle avoidance task after detecting the obstacle, and continues the trajectory tracking task after completing the obstacle avoidance task, showing good performance and adaptability. Figure 7 The action values ​​output by the policy network over time during this process are shown.

[0167] Although the present application is described herein with reference to specific embodiments, it should be understood that these embodiments are merely illustrative of the principles and applications of the present application. It should be understood that many modifications may be made to the illustrative embodiments, and that other arrangements may be devised, without departing from the spirit and scope of the present application as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in ways other than those described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be used in conjunction with other described embodiments.

Claims

1. A trajectory tracking and obstacle avoidance control method for an AUV equipped with a front-facing sonar, characterized in that: include: Input the state of the AUV into the controller, obtain the action corresponding to the state, and realize the control of the AUV; The state of the AUV is the state under the trajectory tracking task or the state under the obstacle avoidance task; The expression of the controller is: Among them, π * represents the optimal control strategy of AUV, π represents the control strategy of AUV, V(s) is the state value function, Q(s,a) is the state-action value function, S and A are the observable state space and action space respectively, s and a are the state and action of AUV respectively.

2. The AUV trajectory tracking and obstacle avoidance control method equipped with a front sonar according to claim 1 is characterized in that: The expression of the state value function V(s) is: The expression of the state-action value function Q(s,a) is: Where, represents the sum of the expected rewards at the current time t, γ is the discount factor and γ∈(0,1], s k and a k are the state and action of AUV at time k, s t and a t are the state and action of AUV at time t, and r(s,a) represents the reward function.

3. The AUV trajectory tracking and obstacle avoidance control method equipped with a front sonar according to claim 2 is characterized in that: The reward function r(s,a) is the reward function r under the trajectory tracking task s Or the reward function under the obstacle avoidance task The reward function r under the trajectory tracking task s : The reward function under the obstacle avoidance task Where, ||e s ||1 represents the error state e of the trajectory tracking task s 1-norm, ζ e and ζ c are weight coefficients, It represents the function that penalizes the behavior of AUV approaching the dangerous area and entering the collision range. represents a function that rewards the AUV for approaching the reference trajectory without entering the dangerous area. A function that penalizes the AUV for not reaching the target area before the state collection ends.

4. The AUV trajectory tracking and obstacle avoidance control method equipped with a front sonar according to claim 3 is characterized in that: The function that penalizes the AUV's behavior of approaching the dangerous area and entering the collision range The expression is: in, is the relative distance between the AUV and the obstacle, Indicates the scope of the danger zone. represents the collision range, φ0 represents the radius of the obstacle, ζ o and are weight parameters, and are the relative pitch angle and relative yaw angle between the AUV and the obstacle, E r A trigger flag indicating whether the AUV has entered a dangerous area, and includes:

5. The AUV trajectory tracking and obstacle avoidance control method equipped with a front sonar according to claim 4 is characterized in that: The function that rewards the AUV for approaching the reference trajectory without entering the dangerous area The expression is: Among them, s for The weight parameter, h s is the set boundary constant, sgn(*) represents the sign function, and ρ is the relative distance between the AUV and the trajectory point.

6. The AUV trajectory tracking and obstacle avoidance control method equipped with a front-mounted sonar according to claim 5, characterized in that: Function to punish the AUV for not reaching the target area before the state collection end node The expression is: Among them, t for The weight parameter, E t A trigger flag indicating whether the AUV has entered the target area, and has: ρ′ is the relative distance between the AUV and the center of the target area, φ t is the radius of the target area.

7. The AUV trajectory tracking and obstacle avoidance control method equipped with a front-mounted sonar according to claim 1, 2, 3, 4, 5 or 6, characterized in that: The state expression of the AUV is: Among them, s1 represents the state under the trajectory tracking task, and the expression is: s1=[g s f s h s e s ], g s 、f s 、h s and e s They are speed state, actuator fault state, disturbance state and error state of trajectory tracking task respectively; s2 represents the state under the obstacle avoidance task, and its expression is: ρ is the relative distance between the AUV and the trajectory point, ρ′ is the relative distance between the AUV and the center of the target area, and are the relative distance, relative pitch angle, and relative yaw angle between the AUV and the obstacle; E d Indicates whether the AUV detects an obstacle. d When the value changes from 0 to 1, it means that the AUV detects an obstacle.

8. The AUV trajectory tracking and obstacle avoidance control method equipped with a front-mounted sonar according to claim 7, characterized in that: The training method of the controller includes: In the trajectory tracking task, the TD3 algorithm is used to train the controller; In the obstacle avoidance task, the controller is trained using a maximum entropy reinforcement learning strategy.

9. AUV trajectory tracking and obstacle avoidance control equipment equipped with front sonar, characterized by: The AUV trajectory tracking and obstacle avoidance control device equipped with a front sonar includes a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the AUV trajectory tracking and obstacle avoidance control method equipped with a front sonar according to any one of claims 1 to 8.

10. A computer storage medium, characterized in that The computer storage medium stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the AUV trajectory tracking and obstacle avoidance control method equipped with a front sonar as claimed in any one of claims 1 to 8.

Images