An unmanned surface vehicle obstacle avoidance method and system

Abstract

The application discloses an unmanned ship obstacle avoidance method and system, after obtaining the position, motion and attitude information of the unmanned ship and obstacle information, the collected information is normalized by using a normalization formula and stored in an experience pool in the form of a quadruple; the information stored in the experience pool is calculated by using a neural network algorithm, the obtained decision action is input into the motor of the unmanned ship for execution, so that the unmanned ship executes an obstacle avoidance action, compared with other algorithms, the method designs a state space, a reward function and an action space, increases the exploration action of the unmanned ship in the early stage, makes the neural network algorithm converge faster during training, and after training, the environment is wide, and the method is suitable for scenes with wind wave flow interference, multiple obstacles and dynamic obstacles and has high robustness.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of unmanned ship obstacle avoidance, and in particular to an unmanned ship obstacle avoidance method and system. BACKGROUND

[0002] With the rapid development of economy and the expansion of energy production, in the case of the increasingly prominent contradiction between the global food, resource, energy supply and the rapid population growth, the development of marine resources is inevitable in the development of history. At present, more and more countries pay attention to the development of deep-sea resources, and the development of deep-sea oil and gas resources, seabed mineral resources, deep-sea biological gene resources and offshore fishery resources has become a hot spot of international marine resource development. The marine environment has higher uncertainty and danger compared with the land environment. The unmanned ship is a small offshore platform with environmental perception, autonomous navigation capability and the ability to autonomously complete corresponding tasks, which has been widely used in offshore scientific investigation, offshore search and rescue and marine energy exploration in recent years. Because the task environment of the unmanned ship is very complex, it not only contains static obstacles, but also is affected by sea waves and currents and other dynamic obstacles, so collision avoidance has become a key technical factor affecting the autonomous navigation of the unmanned ship. It has become one of the main research hotspots in the industry and the goal of global scholars to continuously explore and optimize.

[0003] The unmanned ship obstacle avoidance is based on the real-time planning of an obstacle-free path satisfying the dynamics constraints of the unmanned ship based on the environmental perception and the state information of the unmanned ship. In the process of the unmanned ship navigation, the information collected by the sensor in real time and the information about the obstacles, ships and sudden information near the ship body informed by the communication system in real time enable the unmanned ship to leave the original path and reasonably avoid while ensuring that the original task can be completed. Common obstacle avoidance algorithms include artificial potential field method, ant colony algorithm and bacterial foraging optimization. However, in actual tasks, the environmental space often has uncertainty, such as incomplete perception of environmental information, noise of sensor data and control error, so that the above methods are not suitable for solving the unmanned ship obstacle avoidance problem in complex environments. SUMMARY

[0004] The present application provides an unmanned ship obstacle avoidance method and system to solve the technical problem that the unmanned ship can stably avoid obstacles and stably reach the target in a complex environment such as wave flow interference.

[0005] To solve the above technical problem, the present application provides an unmanned ship obstacle avoidance method, comprising:

[0006] obtain the state information of the unmanned ship itself and the external environment information;

[0007] process the state information of the unmanned ship itself and the external environment information according to a preset obstacle avoidance execution model to obtain a decision action; wherein the obstacle avoidance execution model is provided with a state space, a reward function and an action space;

[0008] According to the decision action, the motor of the unmanned ship is controlled to make the unmanned ship perform the obstacle avoidance action.

[0009] By acquiring the position, motion and attitude information of the unmanned ship and the obstacle information, the collected information is normalized and stored in the experience pool, the collected data is calculated, the decision action is obtained, and the decision action is input into the motor of the unmanned ship to perform the action, the method has wide application environment, is suitable for scenes with wind wave flow interference, multiple obstacles and dynamic obstacles, and has high robustness.

[0010] As a preferred solution, the current state information of the unmanned ship and the external environment information are processed according to a preset obstacle avoidance execution model to obtain a decision action, specifically:

[0011] The current state information of the unmanned ship and the external environment information are normalized to obtain processed information;

[0012] The experience pool is used to store the processed information in the form of a four-tuple;

[0013] The neural network algorithm is used to calculate the processed information to obtain a decision action.

[0014] In the embodiment of the application, the collected unmanned ship ranging data, attitude data, heading angle data, speed data and unmanned ship distance to the end point data are normalized by using a normalization formula, and then stored in the experience pool in the form of a four-tuple. Then, the algorithm process is executed to calculate the information stored in the experience pool to obtain a decision action. The exploration strategy is added before the neural network algorithm training starts, which is beneficial to the faster convergence of the neural network algorithm during training.

[0015] As a preferred solution, the current state information of the unmanned ship and the external environment information are normalized to obtain processed information, specifically:

[0016] The current state information of the unmanned ship includes attitude information, inclination angle information, speed information and distance information of the unmanned ship to the target point;

[0017] The external environment information of the unmanned ship includes unmanned ship ranging information;

[0018] The normalization is performed by using a normalization formula, and the normalization formula is:

[0019]

[0020]

[0021]

[0022]

[0023] Wherein, DistanceTerminal represents distance terminal data, MaxCheckSize represents the maximum detection distance of the sensor, R represents the lateral tilt angle of the unmanned ship, P represents the left-right tilt angle of the unmanned ship, and velocity represents the speed data.

[0024] According to the preferred embodiment of the present application, by using the designed state space, the collected current position, motion, attitude information and obstacle data of the unmanned ship are normalized and input into the network, which is beneficial to the faster convergence of the neural network algorithm during training.

[0025] The obstacle avoidance execution model is obtained by training the neural network algorithm, and specifically comprises:

[0026] Initializing the neural network parameters and the experience pool;

[0027] According to the comparison between the size of the experience in the experience pool and the preset threshold, it is judged whether to take the exploration strategy, and the unmanned ship thrust value is obtained;

[0028] The current state of the unmanned ship is obtained, the decision action is selected, and the decision action is obtained by adding noise and thrust value;

[0029] The decision action is executed, and the reward value and the new state of the unmanned ship are obtained;

[0030] The current state of the unmanned ship, the decision action, the reward value and the new state of the unmanned ship are stored in the experience pool;

[0031] According to the current state of the unmanned ship, the decision action, the reward value and the new state of the unmanned ship, the neural network is trained;

[0032] The trained neural network algorithm is deployed in the obstacle avoidance execution model, so as to obtain the obstacle avoidance execution model.

[0033] As a preferred solution, according to the comparison between the size of the experience in the experience pool and the preset threshold, it is judged whether to take the exploration strategy, and the unmanned ship thrust value is obtained, and specifically comprises:

[0034] If the experience size of the current experience pool is less than the preset threshold, a random walk strategy is executed, and the thrust value is calculated, and the calculation formula is:

[0035]

[0036] a rl =N(0,max(1000,σ-i*400))

[0037]

[0038] a e =[a forw ,a rl ,a backw ]

[0039] Wherein, a forw represents forward action, a rl represents lateral action, a backw represents backward action, a e represents exploration action, epsilon~N(0, sigma) represents normal distribution with expected value 0 and standard deviation sigma, Maxthrust = 35000, MaxAction = Maxthrust, ExploreMin = Maxthrust * 0.5, and i is the current iteration number;

[0040] The thrust value is a value that decays with the iteration number;

[0041] If the experience size of the current experience pool is greater than the preset threshold, the random walk strategy is not executed, and a plurality of sample data in the experience pool are collected to train the neural network algorithm.

[0042] As a preferred solution, the decision action is executed to obtain the reward value of the current state and the new state of the unmanned ship, specifically: the unmanned ship executes the decision action to obtain the reward value of the current state, and the formula is as follows:

[0043] Reward = BR + OR + RR + WR + VR + DT + DDTReward + DAR + (-0.05)

[0044] Wherein, BR represents that the unmanned ship collides, BR =-2; OR represents that the unmanned ship overturns, OverturnReward =-1; RR represents that the unmanned ship reaches the end point, RR = 20; WR represents that the distance sensor of the unmanned ship detects an obstacle less than the safe distance of the unmanned ship, WR =-1; VR represents that the speed of the unmanned ship is less than the preset value, VR =-2, otherwise VR = 1; DT represents the distance between the current position of the unmanned ship and the target point, DDTR represents the difference between the distance between the current position of the unmanned ship and the target point and the distance between the previous position of the unmanned ship and the target point, DDTR = DDT * 200; DAR represents the heading angle between the unmanned ship and the target point, DAR =-abs(DA).

[0045] By setting the reward value, the reward function can give the reward value of the current state before the unmanned ship enters the next state after making an action, which can help the neural network algorithm converge faster during training.

[0046] As a preferred solution, the neural network is trained according to the current state, the decision action, the reward value and the new state, specifically:

[0047] The sample data of the current state, the decision action, the reward value and the new state are taken, and the expected return of the decision action is calculated by evaluating the target network, and the calculation formula is:

[0048] a t ~ π θ (s t ) + ε + select, a t ~ clip(-1000, 1000)

[0049] y j ~ r j + γminQ ω′ (s j+1 , a t )

[0050] Wherein, a t represents an action, π θ (s t ) represents an action selected under the policy network input state s t , Clip(x,y) represents limiting the value to the range of x to y, less than x is calculated as x, greater than y is calculated as y, y j represents the estimated value of the evaluation network, r j represents the current reward value, and γ represents the discount coefficient.

[0051] The evaluation network parameters, the policy network parameters and the target network parameters are updated.

[0052] As a preferred solution, the evaluation network parameters, the action network parameters and the target network parameters are updated, specifically:

[0053] The evaluation network parameters are updated, and the formula is as follows:

[0054]

[0055] Wherein, m represents the number of collected data, y j represents the estimated value of the evaluation network.

[0056] The network parameters are updated by the deterministic policy gradient, and the formula is as follows:

[0057]

[0058] The target network parameters are updated, and the formula is as follows:

[0059] θ i '← τθ i + (1-τ)θi

[0060] ω' <- ω + (1-τ)'ω' <- ω + (1-τ)'ω' <- ω + (1-τ)'ω' <- ω + (1-τ) '

[0061] To solve the same technical problems, the embodiment of the present application also provides an unmanned ship obstacle avoidance system, comprising: a state environment data acquisition module, an obstacle avoidance execution module;

[0062] The state data acquisition module is used for acquiring unmanned state information and external environment information;

[0063] The obstacle avoidance execution module is used for processing the current self-state information of the unmanned ship and the external environment, obtaining a decision action, and controlling the motor of the unmanned ship according to the decision action, so that the unmanned ship performs an obstacle avoidance action; wherein the obstacle avoidance execution model is provided with a state space, a reward function and an action space.

[0064] As a preferred scheme, the state environment data acquisition module comprises a state information acquisition unit and an external environment information acquisition unit;

[0065] The state information acquisition unit is used for acquiring unmanned ship position, motion and attitude information;

[0066] The external environment information acquisition unit is used for acquiring unmanned ship surrounding obstacle information through a laser radar or a millimeter wave radar carried by the unmanned ship itself.

[0067] The obstacle avoidance execution module comprises an experience pool unit and a neural network unit;

[0068] The experience pool unit is used for storing information after normalizing the current self-state information of the unmanned ship and the external environment;

[0069] The neural network unit is used for calculating the processed information to obtain a decision action. BRIEF DESCRIPTION OF DRAWINGS

[0070] Figure 1 : a flowchart of an embodiment of an unmanned ship obstacle avoidance method provided by the present application;

[0071] Figure 2 : an unmanned ship state space diagram of an embodiment of an unmanned ship obstacle avoidance method provided by the present application;

[0072] Figure 3 : an unmanned ship state ranging diagram of an embodiment of an unmanned ship obstacle avoidance method provided by the present application;

[0073] Figure 4 : a structural schematic diagram of another embodiment of an unmanned ship obstacle avoidance system provided by the present application. DETAILED DESCRIPTION ​

[0074] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative effort belong to the protection scope of the present application.

[0075] Embodiment one

[0076] Please refer to Figure 1 The unmanned ship obstacle avoidance method provided in the embodiments of the present application includes steps 01 to 03, and the specific steps are as follows:

[0077] S01: Obtain information of current unmanned ship state and external environment information.

[0078] In the embodiment, the information of the unmanned ship state includes unmanned ship position, motion and attitude information, which is obtained by GPS or Beidou and inertial navigation sensors; the external environment information mainly includes obstacle information, which can be obtained by the laser radar or millimeter wave radar carried by the unmanned ship itself.

[0079] S02: Process the current state information of the unmanned ship and the external environment information according to a preset obstacle avoidance execution model to obtain a decision action; wherein the obstacle avoidance execution model is provided with a state space, a reward function and an action space.

[0080] In the embodiment, after obtaining the information of the current state of the unmanned ship and the external environment information, the current state information of the unmanned ship and the external environment information are processed and stored in an experience pool, and a neural network algorithm is used to calculate the processed information stored in the experience pool, so as to obtain a decision action.

[0081] Optionally, the current state information of the unmanned ship and the external environment information are processed according to a preset obstacle avoidance execution model to obtain a decision action, specifically:

[0082] The current state information of the unmanned ship and the current external environment information of the unmanned ship are normalized to obtain processed information;

[0083] The experience pool is used to store the processed information in the form of a four-tuple;

[0084] The processed information is calculated using a neural network algorithm to obtain a decision action.

[0085] In the embodiment, the current state information of the unmanned ship includes attitude information, inclination angle information, speed information and current distance information of the unmanned ship from the target point.

[0086] The external environment information of the unmanned ship includes distance data information;

[0087] Normalization processing is performed through a normalization formula, which is:

[0088]

[0089]

[0090]

[0091]

[0092] wherein DistanceTerminal represents distance terminal data, MaxCheckSize represents the maximum detection distance of the sensor, R represents the left-right tilt angle of the unmanned ship, P represents the front-rear tilt angle of the unmanned ship, and velocity represents speed data;

[0093] The normalized information is stored in the experience pool in the form of a quadruple.

[0094] Optionally, the obstacle avoidance execution model is obtained by training a neural network algorithm, and specifically comprises:

[0095] Initializing the neural network parameters and the experience pool;

[0096] Comparing the size of the experience of the experience pool with a preset threshold to determine whether to adopt an exploration strategy, and obtaining the unmanned ship thrust value;

[0097] Obtaining the current state of the unmanned ship, selecting a decision action, adding noise and the thrust value to obtain the decision action;

[0098] Executing the decision action to obtain a reward value and a new state of the unmanned ship;

[0099] Storing the current state, the decision action, the reward value and the new state in the experience pool;

[0100] Training the neural network according to the current state, the decision action, the reward value and the new state;

[0101] Deploying the trained neural network algorithm to the obstacle avoidance execution model, thereby obtaining the obstacle avoidance execution model.

[0102] In this embodiment, the Critic network Q ω1 and Q ω2and Actor network parameters ω1, ω2, θ, initialize target network ω'1←ω1, ω'2←ω2, θ←θ, initialize experience pool M, the default structure is list, for each stage, get the initial state, if the unmanned ship has not ended the voyage and the voyage step number step is less than the maximum limit number, then execute the following process for each step of the unmanned ship:

[0103] According to the size of the experience in the experience pool and the preset threshold, if the size of the experience in the current experience pool is less than the preset threshold, action exploration a is performed forw , a rl and a backw , the action exploration is a two-dimensional vector, with a probability of 0.6 to perform the forward action of a forw , a probability of 0.3 to perform the left and right actions of a rl , and a probability of 0.1 to perform the backward action of a backw , and its calculation formula is:

[0104]

[0105] a rl =N(0, max(1000, σ-i*400))

[0106]

[0107] a e =[a forw , a rl , a backw ]

[0108] Wherein, a forw represents the forward action, a rl represents the lateral action, a backw represents the backward action, and ε~N(0, σ) represents a normal distribution with an expected value of 0 and a standard deviation of σ, Maxthrust=35000, MaxAction=Maxthrust; ExploreMin=Maxthrust*0.5, i is the current iteration number;

[0109] The thrust is a value that decays with the iteration number;

[0110] If the size of the experience in the current experience pool is greater than the preset threshold, the random walk strategy is not performed, and the neural network algorithm is trained by collecting a plurality of sample data in the experience pool;

[0111] Then, according to the current state of the unmanned ship, a running action is selected and noise and thrust values are added to obtain a decision action, and its formula is:

[0112] a t ~π θ (st )+ε+select

[0113] wherein a t represents action, π θ (s t ) represents the input state s t selected action, and ε represents the normally distributed selected noise, and select represents the action value generated by the random walk strategy.

[0114] The decision action is executed, the reward value of the current state is obtained through the reward function, and the new state of the unmanned ship is obtained.

[0115] wherein the reward function formula is as follows:

[0116] Reward = BR + OR + RR + WR + VR + DT + DDTReward + DAR + (-0.05)

[0117] wherein BR represents that the unmanned ship collides, BR = -2; OR represents that the unmanned ship overturns, OverturnReward = -1; RR represents that the unmanned ship reaches the end point, RR = 20; WR represents that the distance sensor of the unmanned ship detects the obstacle less than the safe distance of the unmanned ship navigation, WR = -1; VR represents that the speed of the unmanned ship is less than the preset value, VR = -2, otherwise VR = 1; DT represents the distance between the current position of the unmanned ship and the target point, DDTR represents the difference between the distance between the current position of the unmanned ship and the target point and the distance between the previous position of the unmanned ship and the target point, DDTR = DDT * 200; DAR represents the heading angle between the unmanned ship and the target point, DAR = -abs(DA).

[0118] The current state, decision action, reward value and new state are stored in the experience pool in the form of a four-tuple, sample data is randomly collected from the experience pool, the expected return of the decision action is calculated through the evaluation target network, and the calculation formula is:

[0119] a t ~ π θ (s t ) + ε + select, a t ~ clip(-1000, 1000)

[0120] y j ~ r j + γmin Q ω′ (s j+1 , a t )

[0121] wherein a t represents action, π θ (st represents the input state s under the policy network t represents the selected action, Clip(x,y) represents limiting the numerical value in the range from x to y, less than x is calculated as x, greater than y is calculated as y j represents the estimated value of the evaluation network, r j represents the current reward value, and γ represents the discount factor.

[0122] Then the evaluation target network parameters are updated, and the formula is as follows:

[0123]

[0124] Wherein, m represents the number of collected data, y j represents the estimated value of the evaluation network;

[0125] Every 2 steps, the policy network parameters are updated by deterministic policy gradient, and the formula is as follows:

[0126]

[0127] The target network parameters are updated, and the formula is as follows:

[0128] θ i ←τθ i +(1-τ)θ i ′

[0129] ω′←τω+(1-τ)′

[0130] S03: According to the decision action, the motor of the unmanned ship is controlled to make the unmanned ship execute the obstacle avoidance action.

[0131] In this embodiment, the unmanned ship adopts catamaran structure, and left and right two motors are used as propellers. The obtained decision action is input into the left and right motors of the unmanned ship in the form of PWM to execute for 0.5 seconds, so that the unmanned ship executes the obstacle avoidance action, so that the unmanned ship can stably avoid obstacles and further reach the target point under the interference of waves and currents.

[0132] Embodiment two

[0133] Correspondingly, referring to Figure 4 , Figure 4 is the structure schematic diagram of the unmanned ship obstacle avoidance system provided by the application. As Figure 4 shown, the unmanned ship obstacle avoidance system comprises a state environment data acquisition module 401 and an obstacle avoidance execution module 402, wherein the specific units of each module are as follows:

[0134] Among them, the state data acquisition module is used for acquiring unmanned state information and external environment information;

[0135] The obstacle avoidance execution module is configured to process the current state information of the unmanned ship and the external environment, obtain a decision action, and control a motor of the unmanned ship according to the decision action, so that the unmanned ship performs an obstacle avoidance action.

[0136] In the embodiment, the state environment data acquisition module includes a state information acquisition unit 4011 and an external environment information acquisition unit 4012.

[0137] The state information acquisition unit 4011 is configured to acquire the position, motion and attitude information of the unmanned ship through a GPS or Beidou and an inertial navigation sensor.

[0138] The external environment information acquisition unit 4012 is configured to acquire the obstacle information around the unmanned ship through a laser radar or a millimeter wave radar carried by the unmanned ship.

[0139] In the embodiment, the obstacle avoidance execution module includes an experience pool unit 4021 and a neural network unit 4022.

[0140] The experience pool unit 4021 is configured to store the information obtained by normalizing the current state information of the unmanned ship and the external environment.

[0141] The neural network unit 4022 is configured to calculate the processed information to obtain a decision action.

[0142] The obstacle avoidance method of the unmanned ship can be further understood with reference to the above description.

[0143] Compared with the prior art, by constructing an unmanned ship obstacle avoidance execution model, after acquiring the position, motion, attitude information and obstacle information of the unmanned ship, the collected information is normalized by using a normalization formula and then stored in an experience pool in the form of a quadruple, and then the information stored in the experience pool is calculated by using a neural network algorithm to obtain a decision action, the decision action is input into a motor of the unmanned ship for execution, so that the unmanned ship performs an obstacle avoidance action. Compared with other algorithms, the method designs a state space, a reward function and an action space, increases the exploration action of the agent in the early stage, makes the neural network algorithm converge faster during training, and is suitable for scenarios with wind, wave and current interference, multiple obstacles and dynamic obstacles after training, and has high robustness.

[0144] The above embodiments further illustrate the purpose, technical solutions and advantages of the present application. It should be understood that the above embodiments are only specific embodiments of the present application and are not used to limit the protection scope of the present application. It is particularly pointed out that any modification, equivalent replacement, improvement, etc. made by those skilled in the art within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. An unmanned surface vehicle obstacle avoidance method, characterized in that, The method comprises the following steps: acquiring current self-state information and external environment information of the unmanned ship; processing the current self-state information and the external environment information of the unmanned ship according to a preset obstacle avoidance execution model to obtain a decision action, wherein the obstacle avoidance execution model is provided with a state space, a reward function and an action space; controlling a motor of the unmanned ship according to the decision action to make the unmanned ship execute an obstacle avoidance action; the obstacle avoidance execution model is obtained by training a neural network algorithm, and specifically comprises the following steps: initializing neural network parameters and an experience pool; comparing the size of experience in the experience pool with a preset threshold to determine whether to adopt an exploration strategy and obtain an unmanned ship thrust value; obtaining a current state of the unmanned ship, selecting a decision action, adding noise and the thrust value to obtain the decision action; executing the decision action to obtain a reward value and a new state of the unmanned ship; storing the current state of the unmanned ship, the decision action, the reward value and the new state of the unmanned ship in the experience pool; training the neural network according to the current state of the unmanned ship, the decision action, the reward value and the new state of the unmanned ship; deploying the trained neural network algorithm to the obstacle avoidance execution model to obtain the obstacle avoidance execution model; the comparison of the size of experience in the experience pool with the preset threshold to determine whether to adopt the exploration strategy and obtain the unmanned ship thrust value specifically comprises the following steps: if the size of experience in the current experience pool is smaller than the preset threshold, a random walk strategy is executed, and an action value is calculated according to the following formula: wherein, represents a forward action, represents a lateral action, represents a backward action, represents an exploration action, represents a compliance with a desired value of 0, a standard deviation of a normal distribution, , , is the current iteration number; the thrust is a value that decays with the number of iterations; if the size of experience in the current experience pool is larger than the preset threshold, the random walk strategy is not executed, and a plurality of sample data in the experience pool are collected to train the neural network algorithm.

2. The unmanned surface vehicle obstacle avoidance method of claim 1, wherein, the processing of the current self-state information and the external environment information of the unmanned ship according to the preset obstacle avoidance execution model to obtain the decision action specifically comprises the following steps: normalizing the current self-state information and the external environment information of the unmanned ship to obtain processed information; storing the processed information in the experience pool in the form of a four-tuple; calculating the processed information by using the neural network algorithm to obtain the decision action.

3. The method of claim 2, wherein, the normalization of the current self-state information and the external environment information of the unmanned ship to obtain the processed information specifically comprises the following steps: the current self-state information of the unmanned ship comprises attitude information, inclination angle information, speed information and current distance information of the unmanned ship from a target point; the current external environment information of the unmanned ship comprises distance data information; the normalization is performed by using a normalization formula, and the normalization formula is as follows: wherein, represents distance to goal data, represents maximum detection distance of sensor, represents lateral tilt angle of USV, represents longitudinal tilt angle of USV, represents speed data.

4. The unmanned surface vehicle obstacle avoidance method of claim 1, wherein, the execution of the decision action to obtain the reward value of the current state and the new state of the unmanned ship specifically comprises the following steps: wherein, represents that the unmanned boat collides, ; represents that the unmanned boat capsizes, ; represents that the unmanned boat reaches the end point, ; represents that the distance sensor of the unmanned boat detects that the obstacle is less than the safe distance of the unmanned boat, ; represents that the speed of the unmanned boat is less than the preset value , otherwise ; represents the distance between the current position of the unmanned boat and the target point; represents the difference between the distance between the current position of the unmanned boat and the target point and the distance between the previous position of the unmanned boat and the target point, ; represents the heading angle between the unmanned boat and the target point, .

5. The unmanned surface vehicle obstacle avoidance method of claim 1, wherein, the unmanned ship executes the decision action to obtain the reward value of the current state, and the formula is as follows: the training of the neural network according to the current state, the decision action, the reward value and the new state specifically comprises the following steps: sample data of the current state, the decision action, the reward value and the new state are taken from the experience pool, the expected return of the decision action is calculated by evaluating a target network, and the calculation formula is as follows: wherein, represents an action, represents an input state under a policy network selected action, Clip (x, y) represents limiting the numerical value in the range from x to y, less than x by x, and greater than y by y, represents an estimate of the evaluation network, represents a current reward value, represents a discount factor; The evaluation network parameters, the policy network parameters, and the target network parameters are updated.

6. The unmanned surface vehicle obstacle avoidance method of claim 5, wherein, The evaluation network parameters, the policy network parameters, and the target network parameters are updated. The evaluation network parameters are updated, and the formula is as follows: wherein ; The network parameters are updated by a deterministic policy gradient, and the formula is as follows: The target network parameters are updated, and the formula is as follows: 。 7. An unmanned surface vehicle obstacle avoidance system, characterized by, The unmanned ship obstacle avoidance system for executing the unmanned ship obstacle avoidance method according to any one of claims 1-6 comprises a state environment data acquisition module and an obstacle avoidance execution module. The state data acquisition module is used to acquire unmanned state information and external environment information. The obstacle avoidance execution module is used to process the current state information of the unmanned ship and the external environment to obtain a decision action, and control the motor of the unmanned ship according to the decision action, so that the unmanned ship performs an obstacle avoidance action.

8. The unmanned surface vehicle obstacle avoidance system of claim 7, wherein, The state environment data acquisition module comprises a state information acquisition unit and an external environment information acquisition unit. The state information acquisition unit is used to acquire the position, motion, and attitude information of the unmanned ship. The external environment information acquisition unit is used to acquire the obstacle information around the unmanned ship through a laser radar or a millimeter wave radar carried by the unmanned ship. The obstacle avoidance execution module comprises an experience pool unit and a neural network unit. The experience pool unit is used to store the information after the current state information of the unmanned ship and the external environment are normalized. The neural network unit is used to calculate the processed information to obtain a decision action.

Images