Underwater robot real-time path planning method based on width reinforcement learning

Abstract

The invention discloses an underwater robot real-time path planning method based on width reinforcement learning. The method comprises the following steps: establishing an underwater robot simulation environment; modeling a real-time path planning problem of the underwater robot as a Markov decision process; constructing a width evaluation network and a width target network, and initializing parameters; designing a priori strategy to guide the underwater robot to perform action selection; calculating a reward after action in the environment state at the last moment, obtaining complete experience, and storing the complete experience into an experience pool; defining a loss function based on samples in the experience pool, and updating width network parameters by a conjugate gradient method; and updating the width target network every predetermined time until the training is completed. Based on the width reinforcement learning method, the calculation complexity is reduced, a prior strategy is designed to guide the robot to learn, and the method has the advantages of being high in training speed and small in calculation amount and is suitable for the real-time path planning task of the underwater robot.

Description

Technical Field

[0001] The invention relates to the field of underwater robots, and in particular to a real-time path planning method for underwater robots based on width reinforcement learning. Background Art

[0002] An autonomous underwater vehicle (AUV), also known as an underwater robot, is an efficient and autonomous ocean exploration equipment that can be equipped with sonar, cameras and other equipment. It is easy to operate, flexible to deploy, and has low manufacturing cost. It has high maneuverability and autonomy, and can replace manual labor to complete tasks such as marine environment exploration, underwater autonomous operations, underwater search and rescue, and fishing. It has huge application potential.

[0003] During the AUV mission, path planning is a key technology, which plays an important role in underwater search and rescue, fishing and other tasks. The underwater environment is complex, with obstacles such as reefs. The AUV needs to approach the target area while avoiding obstacles. The commonly used method is the artificial potential field method, which uses the superposition of gravitational field and repulsive field to enable the robot to move toward the target while avoiding obstacles. However, when the robot encounters an irregular obstacle or the target is close to the obstacle, local convergence and deadlock are prone to occur.

[0004] In recent years, the application of model-free reinforcement learning methods in the field of AUV has attracted the attention of many scholars. It can overcome the difficulties of uncertain model parameters and environmental factors, so that the robot can learn the optimal strategy through autonomous exploration and achieve autonomous decision-making. The deep reinforcement learning technology combined with deep learning has outstanding strategy learning and optimization capabilities, and has been studied in the robot's path planning, path tracking control, obstacle avoidance and other aspects. However, most deep reinforcement learning algorithms have high computational complexity and slow convergence speed. Compared with deep reinforcement learning, the wide reinforcement learning method has the characteristics of simple network structure and fast calculation speed, but it also has the problem of slow convergence speed, which restricts its application in the field of real-time path planning of underwater robots. Therefore, the present invention proposes a real-time path planning method for underwater robots based on wide reinforcement learning, and improves the convergence speed of the algorithm to achieve real-time, efficient and safe path planning of underwater robots in complex environments. Summary of the invention

[0005] Based on the above background, the present invention proposes a real-time path planning method for underwater robots based on wide reinforcement learning to address the problems of global information loss, local convergence and slow convergence speed in existing AUV path tracking methods, so as to realize autonomous obstacle avoidance and target approach of the robot in a complex obstacle environment.

[0006] The technical solution adopted by the present invention is: A real-time path planning method for underwater robots based on width reinforcement learning, the steps are as follows: S1. Establishing the underwater robot AUV simulation environment, including AUV motion model, obstacle setting, obstacle detection model and simulation time step; S2. Model the real-time path planning problem of AUV in an obstacle environment as a Markov decision process, and determine its state variables, control variables, reward function and objective function; S3, construct a width evaluation network and a width target network and initialize network parameters; S4, initializing the position and velocity of the AUV; S5, AUV action selection e Greedy strategy, and design a priori strategy to guide; S6. Collect the data of the interaction between the AUV and the environment and transfer it to the experience pool, where the interaction data includes the environment state at the previous moment, the action taken by the AUV, the immediate reward obtained by the AUV, and the environment state at the next moment; S7, randomly extract some data from the experience pool, define the loss function of the width network, and use the conjugate gradient method to update the network parameters; S8, every Interval simulation time steps, and use the weights of the evaluation network to update the weights of the target network, where Interval ≥5; S9, determine whether the AUV collides with an obstacle or reaches the target point, if so, enter S10, otherwise return to S5; S10, determine whether the maximum number of training times has been reached ， If yes, end the training and get the final strategy; otherwise, return to S4.

[0007] In one embodiment, in S1, the AUV motion model is as follows: , in, or =[ x , y, f ] T are position and attitude vectors, x , y is the AUV coordinate in the inertial coordinate system, f is the yaw angle, v =[ u , v , r ] T is the velocity vector, u is the longitudinal velocity, v is the lateral velocity, r is the yaw angular velocity, t =[ tu , 0, t r ] T is the driving force vector, t u Propulsion provided by the longitudinal thruster, t r The yaw moment provided to the vertical rudder, t d =[ t ud , t vd , t rd ] T is the external disturbance force vector, t ud , t vd , t rd They are longitudinal disturbance force, lateral disturbance force and disturbance yaw force respectively, M is the inertia matrix, C ( v ) is the Coriolis force matrix, D ( v ) is the viscous resistance matrix, g ( or ) is the restoring force vector, J ( or ) is the Jacobian matrix, which is described as follows: .

[0008] In one embodiment, in S1, the obstacle detection model is as follows: Assume the field of view is 2 α , the number of sensors is n , n The detection directions of the sensors are evenly distributed in a circle with the AUV coordinate position as the center and an angle of 2 α The maximum detection distance is d m , the distance detected by the sensor is the distance from the AUV coordinate position to the contact position of the ultrasonic wave and the obstacle, and finally forms n distances, normalize them to form a distance sequence: D ={ d 1, d 2, … , d n}, in, d i The front distance value measured by each ranging sensor and the maximum detection distance are dm The ratio of d i ∈[0,1], i =1, 2, ... , n ; In the simulation environment, the detection distance of the beam can be calculated by geometric methods. The specific steps are: If the distance between the obstacle and the AUV is greater than or equal to d m , then the detection distance is d m ; Define the coordinate position of the underwater robot as point A, the end position of the beam as point B, and the center position of the obstacle as point O. Calculate the projection g1 of vector AO on vector AB. If g1≤ 0, the detection distance is d m ; like g 1>0, calculate the distance from the obstacle point O to the straight line AB where the beam is located g 2. If g2 ≥ obstacle radius d o , then the detection distance is d m ; like g 2< d o , calculate the dot product of vector OB and vector AB g 4. Calculate the modulus of vector OB g 3. If g 3< d o or g 4>0, the detection distance is ; Otherwise, the detection distance is d m .

[0009] In one embodiment, S2 may include: S21, determine the state variables, according to the detection information D , speed information in the hull coordinate system v =[ u , v , r ] T , yaw angle in inertial coordinate system f and the desired yaw angle f d The build status variables are: S ={ d 1, d 2, … , d n , u ,v , r , f , f d}, where the expected yaw angle is calculated by the line of sight method, and the coordinates of the target point are ( x d , y d ), the AUV coordinates are ( x , y ), then the expected yaw angle is ; S22, determine the control variable, set the control variable to the change of the AUV yaw angle, and discretize it as the action space of the AUV A ,Right now A ={-Δ f 1, -Δ f 2,…,0,…,Δ f 2, Δ f 1}; S23, determine the transfer model, the transfer model is a time-discrete black box function, and the expression is: s t+1 = h ( s t , a t , Δ t ), where Δ t for t The uncertain disturbances in the time transfer model represent the t State variables at a moment s t and control variables a t , returns the state variable of the next moment s t+1 .

[0010] S24. Determine the reward function, the reward that the AUV receives after performing the action r t It is determined based on whether the AUV collides with an obstacle, approaches a target, or reaches a target. Specifically: ; S25. Determine the objective function, which can be expressed as , in, π represents the strategy with the largest cumulative reward generated by the AUV during the decision-making process. π ( a|s) is the mapping from state space to action space, indicating that in state s Take action a The probability of c is the discount factor, r t It is the immediate reward obtained by the AUV after performing an action.

[0011] In one embodiment, in S3, the wide network includes an input layer and an output layer, and the input layer and the output layer are connected by a weight matrix W m connection; where the input layer contains p Group feature nodes and q Group enhancement nodes, each group of feature nodes contains the number of nodes c 1, each group of enhanced nodes contains the number of nodes c 2; The construction method of feature nodes is: , in, f (·) is a nonlinear mapping function, W fi is a random weight matrix with dimension ( n +5)× c 1; β fi is a random bias matrix with dimension N p × c 1; All feature nodes are merged to obtain Z p =[ Z 1, Z 2, …, Z p ], whose dimension is N p × pc 1, N p is the number of input samples, W fi , β fi All are generated using the randn() function; The construction method of the enhanced node is: , in, x (·) is a nonlinear mapping function, W ej is a random weight matrix with dimension pc 1× c 2; β ejis a random bias matrix with dimension N p × c 2; All feature nodes are merged to obtain H q =[ H 1, H 2, …, H q ], whose dimension is N p × qc 2, W ej , β ej All are generated using the randn() function; The feature matrix is ​​combined with the enhancement matrix to obtain the input layer as [ Z p |H q ], whose dimension is N p ×( pc 1+ qc 2).

[0012] In one embodiment, S5 includes: S51, construct a priori strategy, assuming the detection field of view is 2 α , according to the distance detection sequence obtained by S1 D ={ d 1, d 2,… , d n}, perform interpolation fitting and get the function g ( x ), x It represents the angle between each detection beam and the axis of the underwater robot in the hull coordinate system, x ∈[- α , α ]; Assume that the expected yaw angle calculated by the line of sight method is f d , then the constructor f ( x )for ; Assume that the target approaches the weight coefficient of prior knowledge w , the two are combined to obtain the prior probability function p ( x ),Right now ; S52, using random function to generate random numbers in the interval [0,1] d,like d > e , then the current state s t and every action in the action space a i Composition state action pair ( s t , a i ) is input into the evaluation network to predict the value of each action and select the action with the largest value as the action at the current moment a t ;like d < e , then use the constructed prior model p ( x ), each action in the action space is input as the value of the independent variable into p ( x ), calculate the prior value of each action, input it into the softmax function to get the probability of each action, and sample it to get the action at the current moment a t .

[0013] S53, based on the given initialization of the algorithm e Attenuation factor Δ e ,right e To update, e = e -Δ Yes.

[0014] In one embodiment, S7 includes: S71. When the number of experiences in the experience pool is greater than the batch size batch_size Then, randomly select from the experience pool batch_size Empirical data {( s t , a t , r t , s t+1 )} P ; S72. For a batch of experience {( s t , a t , r t , s t+1 )} P , the action in the action space a i The next moment states t+1 The state-action pair {( s t+1 , a i )} P , input to the evaluation network, a i Predict the action value of S73. Select the action with the greatest value, recorded as , the state-action pair {( s t+1 , a max )} P Input to the target network, predict its value, and get {max a Q ( s t+1 , a )} P ; S74. Calculate the state according to Bellman formula s t Next action a t The action value is: , The state action pair {( s t , a t )} P As input data, { q ( s t , a t )} P As a label, the width evaluation network is updated.

[0015] In one embodiment, S74 specifically includes: setting the state action pair {( s t , a t )} P Input width network. After feature node mapping and enhanced node mapping, the input layer of the width network is S , the output is SW m , the expected output is Y ={ q ( s t , a t )} P , define the loss function as ; right W m Find the partial derivative and get the gradient of the loss function: ; The gradient is solved by iteration, and the evaluation network weight matrix is ​​completed using the conjugate gradient method. W m Updates.

[0016] The beneficial effect of the present invention is that compared with the existing real-time path planning method based on deep reinforcement learning, it has lower computational complexity and faster convergence speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0018] Figure 1 A schematic flow chart of a real-time path planning method for an underwater robot based on width reinforcement learning provided by the present invention.

[0019] Figure 2 Schematic diagram of an obstacle detection model according to an embodiment of the present invention.

[0020] Figure 3 Schematic diagram of the width network structure of an embodiment of the present invention.

[0021] Figure 4 It is a schematic diagram of the path planning effect of an embodiment of the present invention.

[0022] Figure 5 It is the reward curve of the training process of the present invention.

[0023] Figure 6 The reward curve of the Double DQN algorithm training process in the prior art. DETAILED DESCRIPTION

[0024] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0025] Now combined Figure 1-Figure 6The preferred embodiments of the present invention are described. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0026] A path planning method for underwater robots based on width reinforcement learning. The specific process is as follows: Figure 1 As shown, including: S1. Build a simulation environment for underwater robots; (1.1) Establish the motion model of the underwater robot: Since the underwater obstacle environment is usually at the bottom of the water or in shallow water, the motion amplitude in the horizontal plane is much larger than the motion amplitude in the vertical plane. Therefore, the underwater robot moves at a certain depth, and the simulation time step is defined as T , establish the three-degree-of-freedom motion model of AUV in the horizontal plane:

[0027] in, or =[ x , y, f ] T is the pose vector in the inertial coordinate system, x is the easting coordinate, y is the north coordinate, f is the yaw angle, v =[ u , v , r ] T is the velocity vector in the hull coordinate system, u is the longitudinal velocity, v is the lateral velocity, r is the yaw angular velocity, t =[ t u , 0, t r ] T is the driving force vector, t u Propulsion provided by the longitudinal thruster, t r The yaw moment provided to the vertical rudder, t d =[ t ud , t vd , t rd ] T is the external disturbance force vector, t ud , t vd , t rdare the longitudinal disturbance force, lateral disturbance force and disturbance yaw moment respectively, M is the inertia matrix, C ( v ) is the Coriolis force matrix, D ( v ) is the viscous resistance matrix, g ( or ) is the restoring force vector, J ( or ) is the Jacobian matrix, which is described as follows:

[0028] In order to enable the AUV to run according to the desired heading planned in real time, the heading controller of the AUV is designed using the PD control method. The propulsion force on the AUV is set to a constant value. The state of the AUV after performing a certain action is calculated based on the kinematic and dynamic model of the AUV.

[0029] (1.2) Setting obstacles: Underwater obstacles are set as cylinders, and their radius, height, number, and the coordinates of the center point of each obstacle in the inertial coordinate system are set to determine the number of obstacles and the size and position of each obstacle in the simulation environment. The set cylindrical obstacles together constitute an inaccessible area. The underwater robot performs real-time path planning based on its perception of obstacles to avoid obstacles.

[0030] (1.3) Establish obstacle detection model: The underwater robot mainly detects obstacle information through underwater ultrasonic ranging sensors. Assume that the field of view angle is 2 α , the number of sensors is n , n The detection directions of the sensors are evenly distributed in a circle with the AUV position as the center and an angle of 2 α The maximum detection distance is d m , the detected distance is the distance from the AUV coordinate position to the contact position of the ultrasonic wave and the obstacle. n distance values, normalize them to form a distance sequence: D ={ d 1, d 2, … , d n} in, d i The front distance value measured by each ranging sensor and the maximum detection distance are d m The ratio of d i ∈[0,1], i =1, 2, ... ,n .

[0031] In the simulation environment, the detection distance of the ranging sensor can be calculated by geometric method. The specific steps are as follows: 1. If the distance between the obstacle and the AUV is greater than or equal to d m , then the detection distance is d m ; Define the AUV position as point A, the beam end position as point B, and the obstacle center position as point O. Calculate the projection g1 of vector AO on vector AB. If g1≤ 0, the detection distance is d m .

[0032] 2. If g 1>0, calculate the distance from the obstacle point O to the straight line AB where the beam is located g 2. If g2 ≥ obstacle radius d o , then the detection distance is d m .

[0033] 3. If g 2< d o , calculate the dot product of vector OB and vector AB g 4. Calculate the modulus of vector OB g 3. If g 3< d o or g 4>0, the detection distance is ; Otherwise, the detection distance is d m .

[0034] S2. Model the real-time path planning problem of AUV in an obstacle environment as a Markov decision process. The Markov decision process model includes state variables, control variables, transfer model and reward function. Specifically, it includes: S21, determining state variables; In order to achieve obstacle avoidance and target approaching of underwater robots, the state variables should contain obstacle detection information, target information and self-state information. D , speed information in the hull coordinate system v =[ u , v , r ] T , yaw angle information in inertial coordinate system f and target direction information f dThe build status variables are: O ={ d 1, d 2, … , d n , u , v , r , f , f d}, the dimension of the state variable n s = n +5. The expected yaw angle is calculated by the line of sight method. Let the coordinates of the target point be ( x d , y d ), the AUV coordinates are ( x , y ), then the expected yaw angle is

[0035] In the implementation process, the number of detected distances is determined by the number of underwater ranging sensors used. For example, if 10 underwater ranging sensors are used to measure obstacle information at different angles, the number of beams is 10.

[0036] S22, determine the control variables; The control variable is set to the change in the AUV yaw angle, which is discretized as the action space of the AUV. A ,Right now A ={-Δ f 1, -Δ f 2,…,0,…,Δ f 2, Δ f 1}. In the implementation process, the size of the action space and the specific value of the action can be determined according to actual needs. Since the yaw angular velocity of the underwater robot is small, the change of its yaw angle per unit time is small. Taking the maximum yaw angle change per unit time as 20° as an example, its action space can be defined as A ={-20°, -10°,0°, 10°, 20°}.

[0037] S23, determining a transfer model; The transfer model is a time-discrete black box function, expressed as: s t+1 = h ( s t , a t , Δ t ), where Δ t for tThe uncertain disturbances in the time transfer model. The transfer model represents the t State variables at a moment s t and control variables a t , returns the state variable of the next moment s t+1 During implementation, the transfer model is determined by the constructed simulation environment.

[0038] S24, determining a reward function; The reward the underwater robot receives after performing an action r t It is determined based on whether the AUV collides with an obstacle, approaches a target, or reaches a target. Specifically: .

[0039] S25, determining the objective function; The objective function can be expressed as

[0040] in, π represents the strategy with the largest cumulative reward in the decision-making process. π ( a|s ) is the mapping from state space to action space, indicating that in state s Take action a probability. c is the discount factor, r t It is the immediate reward obtained after performing an action.

[0041] S3. Initialize the training round episode = 0, build the width evaluation network and the width target network and initialize the network parameters.

[0042] The wide network consists of an input layer and an output layer, with a weight matrix between the input layer and the output layer. W m The input layer contains p Group feature nodes and q Group enhancement nodes, where each group of feature nodes contains the number of nodes c 1, each group of enhanced nodes contains the number of nodes c 2.

[0043] The construction method of feature nodes is:

[0044] in, f (·) is a nonlinear mapping function, W fiis a random weight matrix with dimension ( n +5)× c 1; β fi is a random bias matrix with dimension N p × c 1. All feature nodes are merged to obtain Z p =[ Z 1, Z 2, …, Z p ], whose dimension is N p × pc 1, N p is the number of input samples. W fi , β fi All are generated using the randn() function.

[0045] The construction method of the enhanced node is:

[0046] in, x (·) is a nonlinear mapping function, W ej is a random weight matrix with dimension pc 1× c 2; β ej is a random bias matrix with dimension N p × c 2. All feature nodes are merged to obtain H q =[ H 1, H 2, …, H q ], whose dimension is N p × qc 2. W ej , β ej All are generated using the randn() function.

[0047] The feature matrix is ​​combined with the enhancement matrix to obtain the input layer S =[ Z p |H q ], whose dimension is N p ×( pc 1+ qc 2).

[0048] The relationship between the output layer and the input layer is: X = SW m .

[0049] S4. Initialize the position and velocity of the AUV, specifically, including: Set the AUV's position in the inertial coordinate system: or =( x 0, y 0, f 0) T ; Set the speed of the AUV in the hull coordinate system: v =( u 0, v 0, r 0) T .

[0050] 5. Selection and adoption of underwater robot actions e Greedy strategy, design a priori strategy to guide, in order to improve the convergence speed. Specifically include: S51. Construct a priori strategy. The specific method is: Assume that the field of view of the underwater ranging sensor array is 2 α , according to the distance detection sequence obtained in step S1 D ={ d 1, d 2,… , d n}, use the Lagrange method to perform interpolation fitting and get the function g ( x ), x It represents the angle between each heading and the AUV axis in the hull coordinate system. x ∈[- α , α ].

[0051] Assume that the expected yaw angle calculated by the line of sight method is f d , then the constructor f ( x )for

[0052] In autonomous navigation, the two subtasks of obstacle avoidance and target approach need to be considered comprehensively. Suppose the weight coefficient of the target approach prior knowledge is w , the two are combined to obtain the prior probability function p ( x )

[0053] S52. Generate random numbers using the rand() function d .like d > e , then the current state s t and every action in the action space a i Composition state action pair ( s t , a i ) is input into the width evaluation network to predict the value of each action and select the action with the largest value as the action at the current moment a t ;like d < e , then the prior model constructed by S51 is used p ( x ), the angle between the heading and the AUV axis corresponding to each action in the action space is input into the function as the value of the independent variable p ( x ), calculate the prior value of each action, input it into the softmax function to get the probability of each action, and sample it to get the action at the current moment a t .

[0054] S53, based on the given initialization of the algorithm e Attenuation factor Δ e ,right e To update, e = e -Δ e .

[0055] S6. Collect the data of the interaction between the AUV and the environment and transfer it to the experience pool. The interaction data includes the environment state at the previous moment, the action taken by the AUV, the immediate reward obtained by the AUV, and the environment state at the next moment. Specifically, calculate the environment state at the previous moment s t and actions a t Reward after effect r t , and the state at the next moment s t+1 , and obtain a set of experiences ( s t , a t , r t ,s t+1 ) and stored in the experience pool.

[0056] S7. Randomly extract some data from the experience pool, define the loss function of the wide network, and use the conjugate gradient method to update the network parameters. Specifically, when the number of experiences in the experience pool is greater than the batch size batch_size After that, some data are randomly extracted from the experience pool to evaluate the update of network parameters. s t , a t , r t , s t+1 )} P , the action in the action space a i The next moment state s t+1 The state-action pair {( s t+1 , a i )} P , input to the evaluation network, a i The action value of is predicted. The action with the largest value is selected and recorded as . Set the state action pair {( s t+1 , a max )} P Input to the target network, predict its value, and get {max a Q ( s t+1 , a )} P . Calculate the state according to Bellman formula s t Next action a t The action value is:

[0057] The state action pair {( s t , a t )} P As input data, { q ( s t , a t )} PAs a label, update the width evaluation network. Specifically, the state action pair {( s t , a t )} P Input width network. After feature node mapping and enhanced node mapping, the input layer of the width network is S , the output is SW m , expected output Y ={ q ( s t , a t )} P The loss function is defined as

[0058] right W m Find the partial derivative and get the gradient of the loss function: .

[0059] The gradient is solved by iterative method, and the evaluation network weight matrix is ​​completed by conjugate gradient method. W m The update algorithm is as follows: (7.1) Note , s 0=- r 0.

[0060] (7.2) W m = W m + αs 0.

[0061] (7.3) Initialize k=0.

[0062] (7.4) r =- S T ( Y - SW m ).

[0063] (7.5) .

[0064] (7.6) W m = W m + αs .

[0065] (7.7) r 0=r , s 0= s .

[0066] (7.8)k=k+1.

[0067] If k < number of iterations m , then return to (7.4); otherwise, end the iteration and complete the evaluation network weight matrix W m Updates.

[0068] S8, every Interval The weights of the target network are updated with the weights of the evaluation network. Interval The value can be set as needed. Interval≥ 5 ， Usually you can take 5, 10, 20, etc.

[0069] S9: Determine whether the AUV has collided with an obstacle or reached the target point. If so, this round ends, episode = episode + 1, and enters S10, otherwise returns to S5. If the minimum value in the distance sequence detected by the AUV is 0, it is determined that the AUV has collided with an obstacle. If the distance between the AUV and the target position is less than 0.5m, it is determined that the AUV has reached the target position.

[0070] S10: Determine whether the maximum number of training times has been reached, that is, determine whether the episode is equal to the maximum number of training times. max_ iter, If yes, end the training and get the final strategy; otherwise, return to S4.

[0071] Simulation verification: Figure 2-Figure 6 The simulation verification process of the method of the present invention is shown. Setting the simulation time step T =0.1s. Number of distance sensors n =10, the field of view is 90°, and the obstacle detection model is as follows Figure 2 As shown in Figure 2, the obstacle diameter is 2 and the number is 8. The action space is set to A ={-20°, -10°,0, 10°,20°}, the width network structure is as follows Figure 3 As shown, the number of characteristic nodes c 1=30, number of feature node groups p =10, enhance the number of nodes c 2=200, increase the number of node groups q =30. Maximum number of training sessions max_iter =200, target network update interval Interval =10, e =1,Δ e =0.005, learning rate α = 0.01, the number of iterations to evaluate network updates m =2, number of experience extraction batches batch_size =64. The AUV plans the path based on the detected distance information, its own state information and target information, determines the navigation direction, and then uses the output torque of the underlying controller to control the AUV to move along the desired heading. The path planning effect is as follows Figure 4 As shown in FIG. 1 , it can be seen that the method of the present invention can plan a safe and efficient path in an obstacle environment, so that the AUV can smoothly reach the target area. The reward curve of the training process of the present invention is shown in FIG. Figure 5 As shown in Figure 2, the reward curve of the training process of the existing method DoubleDQN algorithm is as follows: Figure 6 As shown, it can be seen that the average reward of the training process of the present invention is higher and it can converge faster. Compared with the existing real-time path planning method based on the Double DQN algorithm, the advantages of the present invention are lower computational complexity and faster convergence speed.

[0072] The above shows and describes the basic principle, main features and application effects of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A real-time path planning method for underwater robots based on wide reinforcement learning, characterized in that: The following steps are involved: S1. Establishing the underwater robot AUV simulation environment, including AUV motion model, obstacle setting, obstacle detection model and simulation time step; S2. Model the real-time path planning problem of AUV in an obstacle environment as a Markov decision process, and determine its state variables, control variables, reward function and objective function; S3, construct a width evaluation network and a width target network and initialize network parameters; S4, initialize the position and velocity of AUV; S5, AUV action selection ε Greedy strategy, and design a priori strategy to guide; S6. Collect the data of the interaction between the AUV and the environment and transfer it to the experience pool, where the interaction data includes the environment state at the previous moment, the action taken by the AUV, the immediate reward obtained by the AUV, and the environment state at the next moment; S7, randomly extract some data from the experience pool, define the loss function of the width network, and use the conjugate gradient method to update the network parameters; S8, every Interval simulation time steps, and use the weights of the evaluation network to update the weights of the target network, where Interval ≥5; S9, determine whether the AUV collides with an obstacle or reaches the target point, if so, enter S10, otherwise return to S5; S10: Determine whether the maximum number of training times has been reached ， If yes, end the training and get the final strategy; otherwise, return to S4.

2. The method according to claim 1, characterized in that In S1, the AUV motion model is as follows: ； in, η =[ x , y, φ ] T are position and attitude vectors, x , y is the AUV coordinate in the inertial coordinate system, φ is the yaw angle, v =[ u , v , r ] T is the velocity vector, u is the longitudinal velocity, v is the lateral velocity, r is the yaw angular velocity, τ =[ τ u , 0, τ r ] T is the driving force vector, τ u Propulsion provided by the longitudinal thruster, τ r The yaw moment provided to the vertical rudder, τ d =[ τ ud , τ vd , τ rd ] T is the external disturbance force vector, τ ud , τ vd , τ rd They are longitudinal disturbance force, lateral disturbance force and disturbance yaw force respectively, M is the inertia matrix, C ( v ) is the Coriolis force matrix, D ( v ) is the viscous resistance matrix, g ( η ) is the restoring force vector, J ( η ) is the Jacobian matrix, which is described as follows: 。 3. The method according to claim 2, characterized in that In S1, the obstacle detection model is as follows: Assume the field of view is 2 α , the number of sensors is n , n The detection directions of the sensors are evenly distributed in a circle with the AUV coordinate position as the center and an angle of 2 α The maximum detection distance is d m , the distance detected by the sensor is the distance from the AUV coordinate position to the contact position of the ultrasonic wave and the obstacle, and finally forms n distances, normalize them to form a distance sequence: D ={ d 1, d 2, … , d n}， in, d i The front distance value measured by each ranging sensor and the maximum detection distance are d m The ratio of d i ∈[0,1], i =1, 2, ... , n ; In the simulation environment, the detection distance of the beam can be calculated by geometric methods. The specific steps are: If the distance between the obstacle and the AUV is greater than or equal to d m , then the detection distance is d m ; Define the coordinate position of the underwater robot as point A, the end position of the beam as point B, and the center position of the obstacle as point O. Calculate the projection g1 of vector AO on vector AB. If g1≤ 0, the detection distance is d m ; like g 1>0, calculate the distance from the obstacle point O to the straight line AB where the beam is located g 2. If g2 ≥ obstacle radius d o , then the detection distance is d m ; like g 2< d o , calculate the dot product of vector OB and vector AB g 4. Calculate the modulus of vector OB g 3. If g 3< d o or g 4>0, the detection distance is ; Otherwise, the detection distance is d m .

4. The method according to claim 3, characterized in that S2 include: S21, determine the state variables, according to the detection information D , speed information in the hull coordinate system v =[ u , v , r ] T , yaw angle in inertial coordinate system φ and the desired yaw angle φ d The build status variables are: S ={ d 1, d 2, … , d n , u , v , r , φ , φ d }, where the expected yaw angle is calculated by the line of sight method, and the coordinates of the target point are ( x d , y d ), the AUV coordinates are ( x , y ), then the expected yaw angle is ; S22, determine the control variable, set the control variable to the change of the AUV yaw angle, and discretize it as the action space of the AUV A ,Right now A ={-Δ φ 1, -Δ φ 2,…,0,…,Δ φ 2, Δ φ 1}; S23, determine the transfer model, the transfer model is a time-discrete black box function, and the expression is: s t+1 = h ( s t , a t , Δ t ), where Δ t for t The uncertain disturbances in the time transfer model represent the t State variables at a moment s t and control variables a t , returns the state variable of the next moment s t+1 ; S24. Determine the reward function, the reward that the AUV receives after performing the action r t It is determined based on whether the AUV collides with an obstacle, approaches a target, or reaches a target. Specifically: ； S25. Determine the objective function, which can be expressed as ， in, π represents the strategy with the largest cumulative reward generated by the AUV during the decision-making process. π ( a|s ) is the mapping from state space to action space, indicating that in state s Take action a The probability of γ is the discount factor, r t It is the immediate reward obtained by the AUV after performing an action.

5. The method according to claim 4, characterized in that In S3, the wide network consists of an input layer and an output layer, and the input layer and the output layer are connected by a weight matrix W m connection; where the input layer contains p Group feature nodes and q Group enhancement nodes, each group of feature nodes contains the number of nodes c 1, each group of enhanced nodes contains the number of nodes c 2; The construction method of feature nodes is: ， in, φ (·) is a nonlinear mapping function, W fi is a random weight matrix with dimension ( n +5)× c 1; β fi is a random bias matrix with dimension N p × c 1; All feature nodes are merged to obtain Z p =[ Z 1, Z 2, …, Z p ], whose dimension is N p × pc 1, N p is the number of input samples, W fi , β fi All are generated using the randn() function; The construction method of the enhanced node is: ， in, ξ (·) is a nonlinear mapping function, W ej is a random weight matrix with dimension pc 1× c 2; β ej is a random bias matrix with dimension N p × c 2; All feature nodes are merged to obtain H q =[ H 1, H 2, …, H q ], whose dimension is N p × qc 2, W ej , β ej All are generated using the randn() function; The feature matrix is ​​combined with the enhancement matrix to obtain the input layer as [ Z p |H q ], whose dimension is N p ×( pc 1+ qc 2).

6. The method according to claim 5, characterized in that S5 include: S51, construct a priori strategy, the detection field of view is 2 α , according to the distance detection sequence obtained by S1 D ={ d 1, d 2, … , d n }, perform interpolation fitting and get the function g ( x ), x It represents the angle between the detection beam of each ranging sensor and the axis of the underwater robot in the hull coordinate system. x ∈[- α , α ]; Assume that the expected yaw angle calculated by the line of sight method is φ d , then the constructor f ( x )for ； Assume that the target approaches the weight coefficient of prior knowledge w , the two are combined to obtain the prior probability function p ( x ),Right now ； S52, using random function to generate random numbers in the interval [0,1] δ ,like δ > ε , then the current state s t and every action in the action space a i Composition state action pair ( s t , a i ) is input into the evaluation network to predict the value of each action and select the action with the largest value as the action at the current moment a t ;like δ < ε , then use the constructed prior model p ( x ), each action in the action space is input as the value of the independent variable into p ( x ), calculate the prior value of each action, input it into the softmax function to get the probability of each action, and sample it to get the action at the current moment a t ; S53, based on the given initialization of the algorithm ε Attenuation factor Δ ε ,right ε To update, ε = ε -D ε.

7. The method according to claim 6, characterized in that S7 include: S71. When the number of experiences in the experience pool is greater than the batch size batch_size Then, randomly select from the experience pool batch_ size Empirical data {( s t , a t , r t , s t+1 )} P ; S72. For a batch of experience {( s t , a t , r t , s t+1 )} P , the actions in the action space a i The next moment state s t+1 The state-action pair {( s t+1 , a i )} P , input to the evaluation network, a i Predict the action value of S73. Select the action with the greatest value, recorded as , the state-action pair {( s t+1 , a max )} P Input to the target network, predict its value, and get {max a Q ( s t+1 , a )} P ; S 74. Calculate the state according to Bellman formula s t Next action a t The action value is: ， The state action pair {( s t , a t )} P As input data, { q ( s t , a t )} P As a label, the width evaluation network is updated.

8. The method according to claim 7, characterized in that S74 specifically includes: setting the state action pair {( s t , a t )} P Input width network, after feature node mapping and enhanced node mapping, let the input layer of width network be S , the output is SW m , the expected output is Y ={ q ( s t , a t )} P , define the loss function as ； right W m Find the partial derivative and get the gradient of the loss function: ； The gradient is solved by iteration, and the evaluation network weight matrix is ​​completed using the conjugate gradient method. W m Updates.

Images