Unmanned ship countermeasure method based on depth strategic gradient

By adopting deep strategic gradient algorithm and single-neuron proportional adaptive control method in unmanned boat systems, the problems of low tracking and interception efficiency and insufficient safety of unmanned boats in complex sea conditions are solved, and more efficient and stable are achieved and robust interception effects.

CN119065369APending Publication Date: 2024-12-03NORTHWESTERN POLYTECHNICAL UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411138351.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-19
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The existing unmanned boat tracking and interception systems are inefficient, robust and safe in complex sea conditions, making it difficult to intercept mobile ships quickly and effectively.

Method used

The unmanned boat countermeasure method based on deep strategic gradients is used to pre-train the policy network through a single-neuron proportional adaptive control method, and the security Liyapunov deep strategic gradient algorithm based on the actor-criticist solution is used. Track and intercept.

Benefits of technology

It improves the tracking and interception efficiency of the unmanned boat interception control system, reduces the interception time, enhances the robustness, safety and stability of the system, and provides more reliable safe operation guarantees.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119065369A_ABST
    Figure CN119065369A_ABST
Patent Text Reader

Abstract

The invention discloses an unmanned ship countermeasure method based on a depth strategic gradient, and the method comprises the steps: (1) providing a single neuron proportion adaptive control method to carry out the pre-training of a strategy network in order to accelerate the model training speed; and (2) in order to realize safe and stable tracking of the target ship, a safe Lyapunov depth strategic gradient algorithm based on an actor-commentator scheme is adopted. Compared with the prior art, the tracking and intercepting efficiency of the system is improved, the intercepting time consumption is reduced, the tracking and intercepting robustness, safety and stability of the system are effectively improved, and a solution is provided for reliable and safe operation of the unmanned ship intercepting control system.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned boats, and in particular relates to an unmanned boat countermeasure method based on deep strategic gradient. Background Art

[0002] Exploring advanced unmanned boat tracking and interception technologies and building an efficient and reliable unmanned boat intelligent tracking and interception system are of great significance for improving the tracking and interception capabilities of unmanned boats in complex sea conditions. As an important part of the unmanned system at sea, an efficient and safe tracking and interception system is the key to ensuring the security of the maritime territory. Researching the tracking and interception methods of unmanned boat fleets can not only provide theoretical support for the intelligent and autonomous operation of the new generation of unmanned boats, but also provide solid technical support for the tracking and interception of unmanned boats at sea.

[0003] With the development of the new generation of autonomous driving technology, unmanned boats have been able to assist or even replace manual handling of high-risk tasks, such as combating maritime smuggling, drug trafficking, piracy and other maritime crimes, which is conducive to reducing casualties. In these tasks, unmanned boats need to intercept moving ships as quickly as possible in a complex and changing marine environment to ensure the success rate. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the present invention provides a method for countermeasures against unmanned boats based on deep strategic gradients. (1) To speed up the model training speed, a single neuron proportional adaptive control method is proposed to pre-train the policy network; (2) To achieve safe and stable tracking of the target ship, a safe Lyapunov deep strategic gradient algorithm based on the actor-critic scheme is adopted. Compared with the prior art, the present invention improves the tracking and interception efficiency of the system, reduces the interception time, effectively improves the robustness, safety and stability of the system tracking and interception, and provides a solution for the reliable and safe operation of the unmanned boat interception control system.

[0005] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0006] Step 1: Construct a single neuron proportional adaptive control tracking method based on proportional guidance and single neuron adaptive control;

[0007] Step 2: Construct a deep policy gradient algorithm based on the Lyapunov function of the actor-critic scheme.

[0008] Furthermore, the step 1 is specifically as follows:

[0009] Step 1-1: Using the speed and position information of the interceptor and intruder boats as reference quantities, calculate the next predicted state of the interceptor boat according to the kinematic relationship: x' D ,y' D ,ψ'D ;

[0010] Step 1-2: The track deviation between the interceptor boat’s predicted state and the intruder boat’s current state is used as the neuron’s input variable. After passing through the proportional-integral controller, the control law u(k) is output to obtain the motion torque to control the interceptor boat’s motion.

[0011] Step 1-3: By providing feedback on the current state of the intercepting boat, the speed and heading of the intercepting boat are adjusted to track the intruding boat;

[0012] Step 1-4: By inputting the state information and action information of the intercepting boat and the invading boat, the neural network model is trained to obtain the initial parameters and complete the pre-training of the policy network.

[0013] Furthermore, the step 2 is specifically as follows:

[0014] Step 2-1: Based on environmental information t Select a from the pre-trained output state t , set as the initial state of the training model, and obtain action a through environmental interaction t The corresponding reward r t and the next state s t+1 , and obtain a new data set (s t ,a t ,r t ,c t ,s t+1 ), forming experience data and storing it in the experience pool;

[0015] Step 2-2: When the experience data in the experience pool reaches the preset amount, randomly sample experience data from the experience pool as sample data, input the current state and action into the online actor network, calculate the value function Q(s,a), and convert the next state s t+1 Input into the actor-target network and select the corresponding action a t+1 , the target critic network inputs the next state and action, calculates the value function Q'(s,a), calculates the value gradient of the value network by the value function of the target critic network and the current network, and updates all parameters of the current critic network through the gradient back propagation of the neural network;

[0016] Step 2-3: If the output data is unsafe, the Lyapunov boundary function is used to train the online actor network to minimize the loss function, calculate the policy gradient of the policy network, and update all parameters of the current actor network through the gradient back propagation of the neural network;

[0017] Step 2-4: Continuously adjust the parameters of the actor and critic networks through soft updates until the boundary is reached and the path planning model is obtained.

[0018] The beneficial effects of the present invention are as follows:

[0019] (1) The present invention pre-trains the strategy network by designing a proportional guidance method and a single neuron adaptive control strategy, thereby shortening the interception time in the backward tracking scenario and having stronger robustness in the forward interception scenario, thereby reducing the training steps and improving the initial reward.

[0020] (2) The present invention introduces a secure Lyapunov network with a higher initial reward and a smooth reward convergence curve, thus providing support for unmanned boats to intercept moving targets on water. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a diagram for explaining the pre-training process of the unmanned boat of the present invention;

[0022] Figure 2 A motion relationship diagram of the unmanned boat and the intercepted target of the present invention;

[0023] Figure 3 This is a diagram explaining the secure Lyapunov deep strategic gradient algorithm of the present invention;

[0024] Figure 4 It is a schematic diagram of the pre-training backward tracking result in the experimental example of the present invention;

[0025] Figure 5 This is a schematic diagram of the pre-trained forward interception result in the experimental example of the present invention;

[0026] Figure 6 A schematic diagram of the backward tracking result of the deep strategic gradient algorithm in an embodiment of the present invention;

[0027] Figure 7 Schematic diagram of the forward interception result of the deep strategic gradient algorithm in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0029] The present invention develops an unmanned boat tracking and interception method based on a deep strategic gradient algorithm, which shortens the learning and training time of the decision network and improves the safety, stability and robustness of the system.

[0030] like Figure 1 As shown, the purpose of the present invention is to provide an unmanned boat countermeasure method based on deep strategic gradient in order to solve the problem of safe and stable tracking and intercepting targets in the above-mentioned mission scenarios.

[0031] A method for countering unmanned boats based on deep strategic gradients, the method comprising the following steps:

[0032] (1) Based on the known positions, headings, and speeds of the interceptor and intruder boats, the strategy network is pre-trained using the single neuron proportional adaptive control method;

[0033] According to the current speed, position and heading angle of the interceptor and intruder boats, the proportional guidance method is used to calculate the next predicted state of the defense boat. By feeding back the current state of the interceptor boat, the speed and heading of the interceptor boat are adjusted to track the intruder boat. The initial parameters are obtained by inputting the state information and action information of the interceptor and intruder boats.

[0034] The single neuron proportional adaptive control pre-training method is adopted, which includes line of sight proportional guidance and neuron adaptive control. The line of sight angle proportional guidance law is used to calculate the next target position of the unmanned boat, and the neuron adaptive control is used for the interceptor boat to track the position of the intruder boat. The single neuron network with adaptive parameter update uses the adaptive law to adjust the neuron parameters.

[0035] The policy network has five parts: online actor network, target actor network, online critic network, target critic network and safe Lyapunov network. The online network and the target network have the same network structure but different hyperparameters. The parameters of the online network are updated at each iteration step, while the parameters of the target network are updated by a soft update method.

[0036] (2) In response to the real-time changes in the position and speed of the invading boat, a control plan is formulated using a deep strategic gradient method based on the safe Lyapunov function of the actor-critic scheme to track and intercept the invading boat.

[0037] A secure Lyapunov network is introduced to identify unsafe data sets, and then the unsafe data sets are collected and put into the experience pool. The unsafe data set samples are trained intensively to improve the recognition sensitivity of unsafe states and reduce the output of unsafe control volume.

[0038] The unmanned boat introduces a continuous non-negative safety cost function within the operating range to measure the degree of constraint satisfaction. If the expectation of the constraint function has an upper bound, it means that the unmanned boat can recover from an unsafe state within a finite time. The safety cost function is used to evaluate the safety range of actions in a state. If the expectation of the safety cost function satisfies the ultimate uniform bound within a finite time, the controlled object has the ability to recover from an unsafe state to a safe state.

[0039] Example:

[0040] like Figure 2 As shown, taking an intercepting boat tracking and intercepting an invading boat as an example, the specific implementation steps of the present invention are unfolded, and the implementation method of the scheme of the present invention is described in detail in conjunction with the accompanying drawings.

[0041] like Figure 3 As shown, the present invention proposes an unmanned boat countermeasure method based on deep strategic gradient, which includes two steps.

[0042] Step 1: In the pre-training stage, the speed, heading angle and position information of the intruder and interceptor boats are collected, the speed, heading angle and angular velocity of the interceptor boat in the next state are calculated, and the tracking interception trajectory is generated. By inputting the state information and action information of the interceptor and intruder boats, the neural network model is trained to obtain the initial parameters. The pre-trained interception trajectory of this experimental example is as follows: Figure 4 , Figure 5 shown.

[0043] Step 2: During the model training phase, based on the state changes of the intruder boat, the interceptor boat updates the policy network and value network parameters. Under the supervision of the secure Lyapunov network, after multiple cycles, a tracking and interception trajectory is generated safely and stably.

[0044] The final solution of the tracking interception path in this embodiment is as follows Figure 6 , Figure 7 shown.

[0045] The present invention realizes the tracking of intruder boats by designing a single neuron proportional adaptive control method and pre-training the strategy network, which greatly improves the tracking and interception efficiency of the system; by introducing a safe Lyapunov network, the robustness, safety and stability of the tracking and interception system are effectively improved.

Claims

1. A method for countering unmanned boats based on deep strategic gradients, characterized in that: The steps include: Step 1: Construct a single neuron proportional adaptive control tracking method based on proportional guidance and single neuron adaptive control; Step 2: Construct a deep policy gradient algorithm based on the Lyapunov function of the actor-critic scheme.

2. The unmanned boat countermeasure method based on deep strategic gradient according to claim 1 is characterized in that: The step 1 is specifically as follows: Step 1-1: Using the speed and position information of the interceptor and intruder boats as reference quantities, calculate the next predicted state of the interceptor boat according to the kinematic relationship: x' D ,y' D ,ψ' D ; Step 1-2: The track deviation between the interceptor boat’s predicted state and the intruder boat’s current state is used as the neuron’s input variable. After passing through the proportional-integral controller, the control law u(k) is output to obtain the motion torque to control the interceptor boat’s motion. Step 1-3: By providing feedback on the current state of the intercepting boat, the speed and heading of the intercepting boat are adjusted to track the intruding boat; Step 1-4: By inputting the state information and action information of the intercepting boat and the invading boat, the neural network model is trained to obtain the initial parameters and complete the pre-training of the policy network.

3. The unmanned boat countermeasure method based on deep strategic gradient according to claim 2 is characterized in that: The step 2 is specifically as follows: Step 2-1: Based on environmental information t Select a from the pre-trained output state t , set as the initial state of the training model, and obtain action a through environmental interaction t The corresponding reward r t and the next state s t+1 , and obtain a new data set (s t ,a t ,r t ,c t ,s t+1 ), forming experience data and storing it in the experience pool; Step 2-2: When the experience data in the experience pool reaches the preset amount, randomly sample experience data from the experience pool as sample data, input the current state and action into the online actor network, calculate the value function Q(s,a), and convert the next state s t+1 Input into the actor-target network and select the corresponding action a t+1 , the target critic network inputs the next state and action, calculates the value function Q'(s,a), calculates the value gradient of the value network by the value function of the target critic network and the current network, and updates all parameters of the current critic network through the gradient back propagation of the neural network; Step 2-3: If the output data is unsafe, the Lyapunov boundary function is used to train the online actor network to minimize the loss function, calculate the policy gradient of the policy network, and update all parameters of the current actor network through the gradient back propagation of the neural network; Step 2-4: Continuously adjust the parameters of the actor and critic networks through soft updates until the boundary is reached and the path planning model is obtained.