A real-time path planning method for underwater robots based on wide reinforcement learning

Abstract

A real-time path planning method for an underwater robot based on width reinforcement learning, comprising: establishing an underwater robot simulation environment; modeling the underwater robot real-time path planning problem as a Markov decision process; constructing a width evaluation network and a width target network, and initializing parameters; designing a priori strategies to guide the underwater robot to select actions; calculating the reward after the action under the environmental state at the previous moment, obtaining complete experience and storing it in an experience pool; defining a loss function based on samples in the experience pool, and updating the width network parameters using a conjugate gradient method; updating the width target network at predetermined intervals until the training is completed. The present invention is based on a width reinforcement learning method, reduces computational complexity, designs a priori strategies to guide the robot to learn, has the characteristics of fast training speed and small amount of computation, and is suitable for real-time path planning tasks of underwater robots.

Description

Technical Field

[0001] The invention relates to the field of underwater robots, and in particular to a real-time path planning method for underwater robots based on width reinforcement learning. Background Art

[0002] Autonomous underwater vehicle (AUV), also known as underwater robot, is a highly efficient and autonomous ocean exploration equipment. It can be equipped with sonar, camera and other equipment. It is simple to operate, flexible to deploy, low in manufacturing cost, with high maneuverability and autonomy. It can replace manual work to complete tasks such as marine environment exploration, underwater autonomous operation, underwater search and rescue, fishing, etc., and has huge application potential.

[0003] During the AUV mission, path planning is a key technology, which plays an important role in underwater search and rescue, fishing and other tasks. The underwater environment is complex, with obstacles such as reefs. The AUV needs to approach the target area while avoiding obstacles. The commonly used method is the artificial potential field method, which uses the superposition of gravitational field and repulsive field to enable the robot to move toward the target while avoiding obstacles. However, when the robot encounters an irregular obstacle or the target is close to the obstacle, local convergence and deadlock are prone to occur.

[0004] In recent years, the application of model-free reinforcement learning methods in the field of AUV has attracted the attention of many scholars. It can overcome the difficulties of uncertain model parameters and environmental factors, so that the robot can learn the optimal strategy through autonomous exploration and achieve autonomous decision-making. The deep reinforcement learning technology combined with deep learning has outstanding strategy learning and optimization capabilities, and has been studied in the robot's path planning, path tracking control, obstacle avoidance and other aspects. However, most deep reinforcement learning algorithms have high computational complexity and slow convergence speed. Compared with deep reinforcement learning, the wide reinforcement learning method has the characteristics of simple network structure and fast calculation speed, but it also has the problem of slow convergence speed, which restricts its application in the field of real-time path planning of underwater robots. Therefore, the present invention proposes a real-time path planning method for underwater robots based on wide reinforcement learning, and improves the convergence speed of the algorithm to achieve real-time, efficient and safe path planning of underwater robots in complex environments. Summary of the invention

[0005] Based on the above background, the present invention proposes a real-time path planning method for underwater robots based on wide reinforcement learning to address the problems of global information loss, local convergence and slow convergence speed in existing AUV path tracking methods, so as to realize autonomous obstacle avoidance and target approach of the robot in a complex obstacle environment.

[0006] The technical solution adopted by the present invention is:

[0007] A real-time path planning method for underwater robots based on width reinforcement learning, the steps are as follows:

[0008] S1. Establishing the underwater robot AUV simulation environment, including AUV motion model, obstacle setting, obstacle detection model and simulation time step;

[0009] S2. Model the real-time path planning problem of AUV in an obstacle environment as a Markov decision process, and determine its state variables, control variables, reward function and objective function;

[0010] S3, construct a width evaluation network and a width target network and initialize network parameters;

[0011] S4, initializing the position and velocity of the AUV;

[0012] S5, the AUV action selection adopts the ε greedy strategy, and designs a priori strategy for guidance;

[0013] S6. Collect the data of the interaction between the AUV and the environment and transfer it to the experience pool, where the interaction data includes the environment state at the previous moment, the action taken by the AUV, the immediate reward obtained by the AUV, and the environment state at the next moment;

[0014] S7, randomly extract some data from the experience pool, define the loss function of the width network, and use the conjugate gradient method to update the network parameters;

[0015] S8, every Interval simulation time steps, use the weight of the evaluation network to update the weight of the target network, where Interval ≥ 5;

[0016] S9, determine whether the AUV collides with an obstacle or reaches the target point, if so, enter S10, otherwise return to S5;

[0017] S10, determine whether the maximum number of training times has been reached, if so, end the training and obtain the final strategy; otherwise, return to S4.

[0018] In one embodiment, in S1, the AUV motion model is as follows:

[0019]

[0020] in, are the position and attitude vectors, x, y are the AUV coordinates in the inertial coordinate system, is the yaw angle, is the velocity vector, u is the longitudinal velocity, v is the lateral velocity, r is the yaw angular velocity, is the driving force vector, τ u Propulsion force provided by the longitudinal thruster, τr The yaw moment provided to the vertical rudder, is the external disturbance force vector, τ ud , τ vd , τ rd are the longitudinal disturbance force, lateral disturbance force and disturbance yaw force respectively, M is the inertia matrix, C(v) is the Coriolis force matrix, D(v) is the viscous resistance matrix, g(η) is the restoring force vector, and J(η) is the Jacobi matrix, which are described as follows:

[0021]

[0022] In one embodiment, in S1, the obstacle detection model is as follows:

[0023] Assume that the field of view angle is 2α, the number of sensors is n, and the detection directions of n sensors are evenly distributed in a sector with the AUV coordinate position as the center and an angle of 2α. The maximum detection distance is d m , the distance detected by the sensor is the distance from the AUV coordinate position to the contact position of the ultrasonic wave and the obstacle, and finally forms n distances, which are normalized to form a distance sequence:

[0024] D={d1,d2,…,d n},

[0025] Among them, d i The front distance value measured by each ranging sensor and the maximum detection distance d m The ratio of i ∈[0,1], i=1,2,...,n;

[0026] In the simulation environment, the detection distance of the beam can be calculated by geometric methods. The specific steps are:

[0027] If the distance between the obstacle and the AUV is greater than or equal to d m , then the detection distance is d m ;

[0028] Define the coordinate position of the underwater robot as point A, the end position of the beam as point B, the center position of the obstacle as point O, and calculate the projection g1 of vector AO on vector AB. If g1≤0, the detection distance is d m ;

[0029] If g1>0, calculate the distance g2 from the obstacle point O to the straight line AB where the beam is located. If g2≥obstacle radius d o , then the detection distance is d m ;

[0030] If g2 <d o, calculate the dot product g4 of vector OB and vector AB, and calculate the modulus g3 of vector OB; if g3 <d o Or g4>0, the detection distance is Otherwise, the detection distance is d m .

[0031] In one embodiment, S2 may include:

[0032] S21, determine the state variable, according to the detection information D, the speed information in the hull coordinate system Yaw angle in inertial coordinate system and the desired yaw angle The build status variables are: The expected yaw angle is calculated by the line of sight method. Let the coordinates of the target point be (x d ,y d ), the AUV coordinates are (x, y), then the expected yaw angle is

[0033] S22, determine the control variable, set the control variable to the change of the AUV yaw angle, and discretize it as the action space A of the AUV, that is,

[0034] S23, determine the transfer model, the transfer model is a time-discrete black box function, the expression is: s t+1 =h(s t ,a t ,Δ t ), where Δ t is the uncertain disturbance in the transfer model at time t. The transfer model represents the state variable s at a given time t. t and control variable a t , returns the state variable s at the next moment t+1 .

[0035] S24. Determine the reward function, the reward r obtained by the AUV after performing the action t It is determined based on whether the AUV collides with an obstacle, approaches a target, or reaches a target. Specifically:

[0036]

[0037] S25. Determine the objective function, which can be expressed as

[0038]

[0039] Among them, π represents the strategy with the largest cumulative reward generated by the AUV in the decision-making process, strategy π(a|s) is the mapping from state space to action space, which represents the probability of taking action a in state s, γ is the discount factor, and r t It is the immediate reward obtained by the AUV after performing an action.

[0040] In one embodiment, in S3, the wide network includes an input layer and an output layer, and the input layer and the output layer are connected by a weight matrix W m Connection; wherein the input layer contains p groups of feature nodes and q groups of enhanced nodes, each group of feature nodes contains c1 number of nodes, and each group of enhanced nodes contains c2 number of nodes;

[0041] The construction method of the feature node is:

[0042]

[0043] in, is a nonlinear mapping function, W fi is a random weight matrix with a dimension of (n+5)×c1; β fi is a random bias matrix with dimension N p ×c1; all feature nodes are merged to get Z p =[Z1,Z2,…,Z p ], whose dimension is N p ×pc1,N p is the number of input samples, W fi , β fi All are generated using the randn() function;

[0044] The construction method of the enhanced node is:

[0045] H j =ξ(Z n W ej +β ej ),j=1,...,c2,

[0046] Among them, ξ(·) is a nonlinear mapping function, W ej is a random weight matrix with dimension pc1×c2; β ej is a random bias matrix with dimension N p ×c2; all enhanced nodes are merged to obtain H q =[H1,H2,…,H q ], whose dimension is N p ×qc2,W ej , β ej All are generated using the randn() function;

[0047] The feature matrix is ​​combined with the enhancement matrix to obtain the input layer [Z p |H q ], whose dimension is N p ×(pc1+qc2).

[0048] In one embodiment, S5 includes:

[0049] S51, construct a priori strategy, assuming that the detection field of view is 2α, and the distance detection sequence D = {d1, d2, ..., d n}, perform interpolation fitting to obtain the function g(x), where x represents the angle between each detection beam and the axis of the underwater robot in the hull coordinate system, x∈[-α, α];

[0050] Assume that the expected yaw angle calculated by the line of sight method is Then the constructor f(x) is

[0051]

[0052] Assume that the target approaches the weight coefficient w of the prior knowledge, and combine the two to obtain the prior probability function p(x), that is,

[0053] p(x)=w·f(x)+(1-w)·g(x);

[0054] S52, use the random function to generate a random number δ in the interval [0,1]. If δ>ε, then the current state s t And every action a in the action space i Composition state action pair (s t ,a i ) is input into the evaluation network to predict the value of each action, and the action with the largest value is selected as the action a at the current moment t ; If δ<ε, then use the constructed prior model p(x), input each action in the action space as the value of the independent variable into p(x), calculate the prior value of each action, input it into the softmax function to obtain the probability of each action, and sample to obtain the action a at the current moment t .

[0055] S53, based on the ε attenuation factor Δε given when the algorithm is initialized, ε is updated, that is,

[0056] ε=ε-Δε。

[0057] In one embodiment, S7 includes:

[0058] S71. When the number of experiences in the experience pool is greater than the batch size batch_size, batch_size pieces of experience data are randomly extracted from the experience pool {(s t,a t ,r t ,s t+1 )} P ;

[0059] S72、For a batch of experience {(s t ,a t ,r t ,s t+1 )} P , the action a in the action space i and the next moment state s t+1 The state-action pair {(s t+1 ,a i )} P , input to the evaluation network, for a i Predict the action value of

[0060] S73. Select the action with the greatest value, denoted as a max , the state-action pair {(s t+1 ,a max )} P Input to the target network, predict its value, and get {max a Q(s t+1 ,a)} P ;

[0061] S74. Calculate the state s according to the Bellman formula t Next, perform action a t The action value is:

[0062]

[0063] The state action pair {(s t ,a t )} P As input data, {q(s t ,a t )} P As a label, the width evaluation network is updated.

[0064] In one embodiment, S74 specifically includes: setting the state action pair {(s t ,a t )} P Input width network, after feature node mapping and enhanced node mapping, the input layer of the width network is S, and the output is SW m , the expected output is Y = {q(s t ,a t )} P , define the loss function as

[0065]

[0066] To W m Find the partial derivative and get the gradient of the loss function:

[0067]

[0068] The gradient is solved by iteration, and the evaluation network weight matrix W is completed by the conjugate gradient method. m Updates.

[0069] The beneficial effect of the present invention is that compared with the existing real-time path planning method based on deep reinforcement learning, it has lower computational complexity and faster convergence speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0071] Figure 1 A schematic flow chart of a real-time path planning method for an underwater robot based on width reinforcement learning provided by the present invention.

[0072] Figure 2 Schematic diagram of an obstacle detection model according to an embodiment of the present invention.

[0073] Figure 3 Schematic diagram of the width network structure of an embodiment of the present invention.

[0074] Figure 4 It is a schematic diagram of the path planning effect of an embodiment of the present invention.

[0075] Figure 5 It is the reward curve of the training process of the present invention.

[0076] Figure 6 The reward curve of the Double DQN algorithm training process in the prior art. DETAILED DESCRIPTION

[0077] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0078] Now combined Figure 1-Figure 6The preferred embodiments of the present invention are described. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0079] A method for underwater robot path planning based on width reinforcement learning. The specific process is as follows: Figure 1 As shown, including:

[0080] S1. Build a simulation environment for underwater robots;

[0081] (1.1) Establishing the motion model of the underwater robot: Since the underwater obstacle environment is usually at the bottom of the water or in a shallow beach, the motion amplitude in the horizontal plane is much greater than the motion amplitude in the vertical plane. Therefore, the underwater robot moves at a certain depth. The simulation time step is defined as T. The three-degree-of-freedom motion model of the AUV in the horizontal plane is established:

[0082]

[0083] in, is the posture vector in the inertial coordinate system, x is the east coordinate, y is the north coordinate, is the yaw angle, is the velocity vector in the hull coordinate system, u is the longitudinal velocity, v is the lateral velocity, r is the yaw angular velocity, is the driving force vector, τ u Propulsion force provided by the longitudinal thruster, τ r The yaw moment provided to the vertical rudder, is the external disturbance force vector, τ ud , τ vd , τ rd are the longitudinal disturbance force, lateral disturbance force and disturbance yaw moment, M is the inertia matrix, C(v) is the Coriolis force matrix, D(v) is the viscous resistance matrix, g(η) is the restoring force vector, and J(η) is the Jacobi matrix, which are described as follows:

[0084]

[0085] In order to enable the AUV to run according to the desired heading planned in real time, the heading controller of the AUV is designed using the PD control method. The propulsion force on the AUV is set to a constant value. The state of the AUV after performing a certain action is calculated based on the kinematic and dynamic model of the AUV.

[0086] (1.2) Setting obstacles: Underwater obstacles are set as cylinders. The radius, height, number and coordinates of the center point of each obstacle in the inertial coordinate system are set to determine the number of obstacles and the size and position of each obstacle in the simulation environment. The set cylindrical obstacles together constitute an inaccessible area. The underwater robot performs real-time path planning based on the perception of obstacles to avoid obstacles.

[0087] (1.3) Establish obstacle detection model: The underwater robot mainly detects obstacle information through underwater ultrasonic ranging sensors. Assume that the field of view angle is 2α, the number of sensors is n, and the detection directions of n sensors are evenly distributed in a sector with an angle of 2α and the AUV position as the center. The maximum detection distance is d m , the detected distance is the distance from the AUV coordinate position to the contact position of the ultrasonic wave and the obstacle. Finally, n distance values ​​are formed, which are normalized to form a distance sequence:

[0088] D={d1,d2,…,d n}

[0089] Among them, d i The front distance value measured by each ranging sensor and the maximum detection distance d m The ratio of i ∈[0,1], i=1,2,...,n.

[0090] In the simulation environment, the detection distance of the ranging sensor can be calculated by geometric method. The specific steps are as follows:

[0091] 1. If the distance between the obstacle and the AUV is greater than or equal to d m , then the detection distance is d m ;

[0092] Define the AUV position as point A, the beam end position as point B, and the obstacle center position as point O. Calculate the projection g1 of vector AO on vector AB. If g1≤0, the detection distance is d m .

[0093] 2. If g1>0, calculate the distance g2 from the obstacle point O to the straight line AB where the beam is located. If g2≥obstacle radius d o , then the detection distance is d m .

[0094] 3. If g2 <d o , calculate the dot product g4 of vector OB and vector AB, and calculate the modulus g3 of vector OB. If g3 <d o Or g4>0, the detection distance is Otherwise, the detection distance is d m .

[0095] S2. Model the real-time path planning problem of AUV in an obstacle environment as a Markov decision process. The Markov decision process model includes state variables, control variables, transfer model and reward function. Specifically, it includes:

[0096] S21, determining state variables;

[0097] In order to achieve obstacle avoidance and target approaching of underwater robots, the state variables should include obstacle detection information, target information and self-state information. Yaw angle information in inertial coordinate system and target direction information The build status variables are: The dimension n of the state variables s =n+5. The expected yaw angle is calculated by the line of sight method. Let the coordinates of the target point be (x d ,y d ), the AUV coordinates are (x, y), then the expected yaw angle is

[0098]

[0099] In the implementation process, the number of detected distances is determined by the number of underwater ranging sensors used. For example, if 10 underwater ranging sensors are used to measure obstacle information at different angles, the number of beams is 10.

[0100] S22, determine the control variables;

[0101] The control variable is set to the change of the AUV yaw angle, which is discretized as the action space A of the AUV, that is, In the implementation process, the size of the action space and the specific value of the action can be determined according to actual needs. Since the yaw angular velocity of the underwater robot is small, the change of its yaw angle per unit time is small. Taking the maximum yaw angle change per unit time as 20° as an example, its action space can be defined as A = {-20°, -10°, 0°, 10°, 20°}.

[0102] S23, determining a transfer model;

[0103] The transfer model is a time-discrete black box function, expressed as: s t+1 =h(s t ,a t ,Δ t ), where Δ t is the uncertain disturbance in the transfer model at time t. The transfer model represents the state variable s at a given time t. t and control variable a t, returns the state variable s at the next moment t+1 During implementation, the transfer model is determined by the constructed simulation environment.

[0104] S24, determining a reward function;

[0105] The reward r obtained by the underwater robot after performing the action t It is determined based on whether the AUV collides with an obstacle, approaches a target, or reaches a target. Specifically:

[0106]

[0107] S25, determining the objective function;

[0108] The objective function can be expressed as

[0109]

[0110] Among them, π represents the strategy with the largest cumulative reward generated in the decision-making process, and the strategy π(a|s) is the mapping from the state space to the action space, which represents the probability of taking action a in state s. γ is the discount factor, r t It is the immediate reward obtained after performing an action.

[0111] S3. Initialize the training round episode = 0, build the width evaluation network and the width target network and initialize the network parameters.

[0112] The width network consists of an input layer and an output layer, and the input layer and the output layer are connected by a weight matrix W. m The input layer contains p groups of feature nodes and q groups of enhanced nodes, where each group of feature nodes contains c1 nodes, and each group of enhanced nodes contains c2 nodes.

[0113] The construction method of the feature node is:

[0114]

[0115] in, is a nonlinear mapping function, W fi is a random weight matrix with a dimension of (n+5)×c1; β fi is a random bias matrix with dimension N p ×c1. All feature nodes are merged to get Z p =[Z1,Z2,…,Z p ], whose dimension is N p ×pc1,N p is the number of input samples. fi , β fi All are generated using the randn() function.

[0116] The construction method of the enhanced node is:

[0117] H j =ξ(Z n W ej +β ej ),j=1,...,c2

[0118] Among them, ξ(·) is a nonlinear mapping function, W ej is a random weight matrix with dimension pc1×c2; β ej is a random bias matrix with dimension N p ×c2. All enhanced nodes are merged to obtain H q =[H1,H2,…,H q ], whose dimension is N p ×qc2.W ej , β ej All are generated using the randn() function.

[0119] The feature matrix is ​​combined with the enhancement matrix to obtain the input layer S = [Z p |H q ], whose dimension is N p ×(pc1+qc2).

[0120] The relationship between the output layer and the input layer is: X = SW m .

[0121] S4. Initialize the position and velocity of the AUV, specifically, including:

[0122] Set the AUV's position in the inertial coordinate system:

[0123] Set the speed of the AUV in the hull coordinate system: v = (u0, v0, r0) T .

[0124] S5. The underwater robot action selection adopts the ε greedy strategy and designs a priori strategy to guide it to improve the convergence speed. Specifically, it includes:

[0125] S51. Construct a priori strategy. The specific method is:

[0126] Assume that the field of view of the underwater ranging sensor array is 2α, and the distance detection sequence D obtained in step S1 is D={d1, d2, …, d n}, the Lagrangian method is used for interpolation fitting to obtain the function g(x), where x represents the angle between each heading and the AUV axis in the hull coordinate system, x∈[-α,α].

[0127] Assume that the expected yaw angle calculated by the line of sight method is Then the constructor f(x) is

[0128]

[0129] In autonomous navigation, the two subtasks of obstacle avoidance and target approach need to be considered comprehensively. Let the weight coefficient w of the target approach prior knowledge be combined to obtain the prior probability function p(x):

[0130] p(x)=w·f(x)+(1-w)·g(x)

[0131] S52, use the rand() function to generate a random number δ. If δ>ε, then the current state s t And every action a in the action space i Composition state action pair (s t ,a i ) is input into the width evaluation network to predict the value of each action and select the action with the largest value as the action a at the current moment t If δ<ε, the prior model p(x) constructed by S51 is used to input the angle between the heading and the AUV axis corresponding to each action in the action space as the value of the independent variable into the function p(x), calculate the prior value of each action, input it into the softmax function to obtain the probability of each action, and sample to obtain the action a at the current moment. t .

[0132] S53, based on the ε attenuation factor Δε given when the algorithm is initialized, ε is updated, that is,

[0133] ε=ε-Δε。

[0134] S6. Collect the data of the interaction between the AUV and the environment and transfer it to the experience pool. The interaction data includes the environment state at the previous moment, the action taken by the AUV, the immediate reward obtained by the AUV, and the environment state at the next moment. Specifically, calculate the environment state s at the previous moment t and action a t Reward after action r t , and the state s at the next moment t+1 , get a set of experience (s t ,a t ,r t ,s t+1 ) and stored in the experience pool.

[0135] S7. Randomly extract some data from the experience pool, define the loss function of the wide network, and use the conjugate gradient method to update the network parameters. Specifically, when the number of experiences in the experience pool is greater than the batch size batch_size, randomly extract some data from the experience pool to evaluate the update of the network parameters. t ,a t ,r t ,s t+1 )} P , the action a in the action space i and the next moment state s t+1 The state-action pair {(s t+1 ,a i )} P , input to the evaluation network, for a i The action value of is predicted. The action with the largest value is selected and recorded as a max . The state action pair {(s t+1 ,a max )} P Input to the target network, predict its value, and get {max a Q(s t+1 ,a)} P . Calculate the state s according to the Bellman formula t Next, perform action a t The action value is:

[0136]

[0137] The state action pair {(s t ,a t )} P As input data, {q(s t ,a t )} P As a label, the width evaluation network is updated. Specifically, the state-action pair {(s t ,a t )} P Input width network, after feature node mapping and enhanced node mapping, the input layer of the width network is S, and the output is SW m , the expected output Y = {q(s t ,a t )} P The loss function is defined as

[0138]

[0139] To W m Find the partial derivative and get the gradient of the loss function:

[0140]

[0141] The gradient is solved by iterative method, and the evaluation network weight matrix W is completed by conjugate gradient method. m The update algorithm is as follows:

[0142] (7.1) Note s0=-r0.

[0143] (7.2)W m =W m +αs0.

[0144] (7.3) Initialize k=0.

[0145] (7.4)r=-S T (Y-SW m ).

[0146] (7.5)

[0147] (7.6)W m =W m +αs.

[0148] (7.7)r0=r,s0=s.

[0149] (7.8)k=k+1.

[0150] If k < the number of iterations m, return to (7.4); otherwise, end the iteration and complete the evaluation network weight matrix W m Updates.

[0151] S8. Every Interval simulation time steps, the weight of the target network is updated with the weight of the evaluation network. The value of Interval can be set as needed, Interval ≥ 5, and can usually be 5, 10, 20, etc.

[0152] S9: Determine whether the AUV has collided with an obstacle or reached the target point. If so, this round ends, episode = episode + 1, and enters S10, otherwise returns to S5. If the minimum value in the distance sequence detected by the AUV is 0, it is determined that the AUV has collided with an obstacle. If the distance between the AUV and the target position is less than 0.5m, it is determined that the AUV has reached the target position.

[0153] S10. Determine whether the maximum number of training times has been reached, that is, determine whether episode is equal to the maximum number of training times max_iter. If so, end the training and obtain the final strategy; otherwise, return to S4.

[0154] Simulation verification:

[0155] Figure 2-Figure 6 The simulation verification process of the method of the present invention is shown. The simulation time step T is set to 0.1s. The number of ranging sensors n is 10, the field of view angle is 90°, and the obstacle detection model is as follows: Figure 2 As shown in Figure 2, the obstacle diameter is 2 and the number is 8. The action space is set to A = {-20°, -10°, 0, 10°, 20°}, and the width network structure is as follows: Figure 3 As shown, the number of feature nodes c1 = 30, the number of feature node groups p = 10, the number of enhanced nodes c2 = 200, and the number of enhanced node groups q = 30. The maximum number of training times max_iter = 200, the update interval of the target network Interval = 10, ε = 1, Δε = 0.005, the learning rate α = 0.01, the number of iterations of the evaluation network update m = 2, and the number of experience extraction batches batch_size = 64. The AUV performs path planning based on the detected distance information, its own state information and target information, determines the navigation direction, and then uses the output torque of the underlying controller to control the AUV to move along the desired heading. The path planning effect is shown in Figure 4 As shown in FIG. 1 , it can be seen that the method of the present invention can plan a safe and efficient path in an obstacle environment, so that the AUV can smoothly reach the target area. The reward curve of the training process of the present invention is shown in FIG. Figure 5 As shown in the figure, the reward curve of the training process of the existing method Double DQN algorithm is as follows Figure 6 As shown, it can be seen that the average reward of the training process of the present invention is higher and it can converge faster. Compared with the existing real-time path planning method based on the Double DQN algorithm, the advantages of the present invention are lower computational complexity and faster convergence speed.

[0156] The above shows and describes the basic principle, main features and application effects of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A real-time path planning method for underwater robots based on wide reinforcement learning, characterized in that: The following steps are involved: S1. Establish an underwater robot AUV simulation environment, including an AUV motion model, obstacle setting, obstacle detection model and simulation time step, wherein the obstacle detection model is as follows: Assume that the field of view angle is 2α, the number of sensors is n, and the detection directions of n sensors are evenly distributed in a sector with the AUV coordinate position as the center and an angle of 2α. The maximum detection distance is d m , the distance detected by the sensor is the distance from the AUV coordinate position to the contact position of the ultrasonic wave and the obstacle, and finally forms n distances, which are normalized to form a distance sequence: D={d1,d2,…,d n }, Among them, d i The front distance value measured by each ranging sensor and the maximum detection distance d m The ratio of i ∈[0,1], i=1,2,...,n; In the simulation environment, the detection distance of the beam is calculated by geometric method. The specific steps are as follows: If the distance between the obstacle and the AUV is greater than or equal to d m , then the detection distance is d m ; Define the coordinate position of the underwater robot as point A, the end position of the beam as point B, the center position of the obstacle as point O, and calculate the projection g1 of vector AO on vector AB. If g1≤0, the detection distance is d m ; If g1>0, calculate the distance g2 from the obstacle point O to the straight line AB where the beam is located. If g2≥obstacle radius d o , then the detection distance is d m ; If g2 <d o , calculate the dot product g4 of vector OB and vector AB, and calculate the modulus g3 of vector OB; if g3 <d o Or g4>0, the detection distance is Otherwise, the detection distance is d m ; S2. Model the real-time path planning problem of AUV in an obstacle environment as a Markov decision process, and determine its state variables, control variables, reward function and objective function; S3, construct a width evaluation network and a width target network and initialize network parameters; S4, initializing the position and velocity of the AUV; S5, the AUV action selection adopts the ε greedy strategy, and designs a priori strategy for guidance; S6. Collect the data of the interaction between the AUV and the environment and transfer it to the experience pool, where the interaction data includes the environment state at the previous moment, the action taken by the AUV, the immediate reward obtained by the AUV, and the environment state at the next moment; S7, randomly extract some data from the experience pool, define the loss function of the width network, and use the conjugate gradient method to update the network parameters; S8, every Interval simulation time steps, use the weight of the evaluation network to update the weight of the target network, where Interval ≥ 5; S9, determine whether the AUV collides with an obstacle or reaches the target point, if so, enter S10, otherwise return to S5; S10, determine whether the maximum number of training times has been reached, if so, end the training and obtain the final strategy; otherwise, return to S4.

2. The method according to claim 1, characterized in that In S1, the AUV motion model is as follows: in, are the position and attitude vectors, x, y are the AUV coordinates in the inertial coordinate system, is the yaw angle, is the velocity vector, u is the longitudinal velocity, v is the lateral velocity, r is the yaw angular velocity, is the driving force vector, τ u Propulsion force provided by the longitudinal thruster, τ r The yaw moment provided to the vertical rudder, is the external disturbance force vector, τ ud , τ vd , τ rd are the longitudinal disturbance force, lateral disturbance force and disturbance yaw force respectively, M is the inertia matrix, C(v) is the Coriolis force matrix, D(v) is the viscous resistance matrix, g(η) is the restoring force vector, and J(η) is the Jacobi matrix, which are described as follows:

3. The method according to claim 2, characterized in that S2 include: S21, determine the state variable, according to the detection information D, the speed information in the hull coordinate system Yaw angle in inertial coordinate system and the desired yaw angle The build status variables are: The expected yaw angle is calculated by the line of sight method. Let the coordinates of the target point be (x d ,y d ), the AUV coordinates are (x, y), then the expected yaw angle is S22, determine the control variable, set the control variable to the change of the AUV yaw angle, and discretize it as the action space A of the AUV, that is, S23, determine the transfer model, the transfer model is a time-discrete black box function, the expression is: s t+1 =h(s t ,a t ,Δ t ), where Δ t is the uncertain disturbance in the transfer model at time t. The transfer model represents the state variable s at a given time t. t and control variable a t , returns the state variable s at the next moment t+1 ; S24. Determine the reward function, the reward r obtained by the AUV after performing the action t It is determined based on whether the AUV collides with an obstacle, approaches a target, or reaches a target. Specifically: S25. Determine the objective function, which is expressed as Among them, π represents the strategy with the largest cumulative reward generated by the AUV in the decision-making process, strategy π(a|s) is the mapping from state space to action space, which represents the probability of taking action a in state s, γ is the discount factor, and r t It is the immediate reward obtained by the AUV after performing an action.

4. The method according to claim 3, characterized in that In S3, the wide network consists of an input layer and an output layer, and the input layer and the output layer are connected by a weight matrix W. m Connection; wherein the input layer contains p groups of feature nodes and q groups of enhanced nodes, each group of feature nodes contains c1 number of nodes, and each group of enhanced nodes contains c2 number of nodes; The construction method of the feature node is: in, is a nonlinear mapping function, W fi is a random weight matrix with a dimension of (n+5)×c1; β fi is a random bias matrix with dimension N p ×c1; all feature nodes are merged to get Z p =[Z1,Z2,…,Z p ], whose dimension is N p ×pc1,N p is the number of input samples, W fi , β fi All are generated using the randn() function; The construction method of the enhanced node is: H j =ξ(Z n IN ej +β ej ),j=1,...,c2, Among them, ξ(·) is a nonlinear mapping function, W ej is a random weight matrix with dimension pc1×c2; β ej is a random bias matrix with dimension N p ×c2; all enhanced nodes are merged to obtain H q =[H1,H2,…,H q ], whose dimension is N p ×qc2,W ej , β ej All are generated using the randn() function; The feature matrix is ​​combined with the enhancement matrix to obtain the input layer [Z p |H q ], whose dimension is N p ×(pc1+qc2).

5. The method according to claim 4, characterized in that S5 include: S51, construct a priori strategy, the detection field angle is 2α, according to the distance detection sequence D = {d1, d2, ..., d n }, perform interpolation fitting to obtain the function g(x), where x represents the angle between the detection beam of each ranging sensor and the axis of the underwater robot in the hull coordinate system, x∈[-α, α]; Assume that the expected yaw angle calculated by the line of sight method is Then the constructor f(x) is Assume that the target approaches the weight coefficient w of the prior knowledge, and fuse g(x) and f(x) to obtain the prior probability function p(x), that is, p(x)=w·f(x)+(1-w)·g(x); S52, use the random function to generate a random number δ in the interval [0,1]. If δ>ε, then the current state s t And every action a in the action space i Composition state action pair (s t ,a i ) is input into the evaluation network to predict the value of each action, and the action with the largest value is selected as the action a at the current moment t ; If δ<ε, then use the constructed prior model p(x), input each action in the action space as the value of the independent variable into p(x), calculate the prior value of each action, input it into the softmax function to obtain the probability of each action, and sample to obtain the action a at the current moment t ; S53, based on the ε attenuation factor Δε given when the algorithm is initialized, ε is updated, that is, ε=ε-Δε。 6. The method according to claim 5, characterized in that S7 includes: S71. When the number of experiences in the experience pool is greater than the batch size batch_size, batch_size pieces of experience data are randomly extracted from the experience pool {(s t ,a t ,r t ,s t+1 )} P ; S72、For a batch of experience {(s t ,a t ,r t ,s t+1 )} P , the action a in the action space i and the next moment state s t+1 The state-action pair {(s t+1 ,a i )} P , input to the evaluation network, for a i Predict the action value of S73. Select the action with the greatest value, denoted as a max , the state-action pair {(s t+1 ,a max )} P Input to the target network, predict its value, and get {max a Q(s t+1 ,a)} P ; S74. Calculate the state s according to the Bellman formula t Next, perform action a t The action value is: The state action pair {(s t ,a t )} P As input data, {q(s t ,a t )} P As a label, the width evaluation network is updated.

7. The method according to claim 6, characterized in that S74 specifically includes: setting the state action pair {(s t ,a t )} P Input width network, after feature node mapping and enhanced node mapping, let the input layer of width network be S and the output be SW m , the expected output is Y = {q(s t ,a t )} P , define the loss function as To W m Find the partial derivative and get the gradient of the loss function: The gradient is solved by iteration, and the evaluation network weight matrix W is completed by the conjugate gradient method. m Updates.

Images