An unmanned ship path tracking control method based on knowledge and data fusion

Abstract

The application discloses a kind of unmanned ship path tracking control methods of knowledge and data fusion, comprising the following steps: 1) training environment building;2) establish unmanned ship path tracking control model;3) pre-training to network;4) model training;5) obtain the pre-trained unmanned ship path tracking control model;6) using pre-trained unmanned ship path tracking control model carries out unmanned ship path tracking control.The method disclosed in the application introduces prior knowledge, updates more quickly, and the effect is more stable: the introduction of knowledge-driven auxiliary reduces the solution space complexity of deep reinforcement learning model, which is conducive to algorithm convergence;The knowledge-driven method provides guidance for the update of the data-driven deep reinforcement learning model, ensuring the stability of the model effect.

Description

Technical Field

[0001] The present invention relates to unmanned boat control technology, and in particular to an unmanned boat path tracking control method integrating knowledge and data. Background Art

[0002] Traditional unmanned boat swarm path tracking control methods mainly include the following categories: 1. Leader-follower control method; 2. Behavior-based control method: pre-define the individual behavior rules and local control schemes of the intelligent agent, so that the intelligent agent can independently complete multiple behaviors; 3. Artificial potential field-based control method: by defining artificial potential functions and minimizing individual potential energy to control the formation, it has strong real-time computing capabilities and the ability to prevent sudden threats.

[0003] Traditional unmanned boat cluster path tracking control algorithms rely on rule knowledge in specific application fields and are knowledge-driven swarm intelligence algorithms. They have clear theories, are easy to integrate knowledge from various disciplines, and have high algorithm interpretability. However, they also have the disadvantages of high knowledge acquisition costs, incomplete solution space exploration, and difficulty in continuous learning and evolution.

[0004] Machine learning algorithms such as deep learning and reinforcement learning use data for iterative updates and are data-driven swarm intelligence algorithms. These algorithms do not require precise modeling of the problem and are highly versatile. They can fully explore the solution space and achieve continuous learning and evolution. However, they also have problems such as difficult theoretical analysis, poor algorithm interpretability, and the dimensionality disaster of the solution space. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a path tracking control method for an unmanned boat by integrating knowledge and data in order to address the defects in the prior art.

[0006] The technical solution adopted by the present invention to solve the technical problem is: a knowledge and data fusion unmanned boat path tracking control method, comprising the following steps:

[0007] 1) Training environment construction;

[0008] 2) Establishing the unmanned boat path tracking control model

[0009] Initialize the deep network θ and deep network Φ of each unmanned ship path tracking control; the deep network θ and deep network Φ correspond to two multi-layer neural network structures;

[0010] Set the parameter value of epsilon-clip in the model strategy update;

[0011] 3) Pre-train the network θ;

[0012] 3.1) θ network pre-training uses expert knowledge to build an unmanned boat path tracking control expert system using the LOS line of sight algorithm, which serves as an expert system to provide auxiliary training for the θ network pre-training;

[0013] 3.2) Initialize the training environment;

[0014] 3.3) At the time slot [t, t+1), the path tracking control expert system performs deduction based on the state transmitted from the training environment to generate the steering command and jet speed of the unmanned boat in the time slot;

[0015] 3.4) At time slot [t, t+1), the training scenario is simulated based on the steering commands and jet speeds of each unmanned boat sent by the path tracking expert system to generate the reward r at time t t , the state s at time t+1 t+1 , termination signal done;

[0016] 3.5) At time slot [t, t+1), the trajectory formed by the interaction between the path tracking expert system and the training environment is stored in the experience buffer for use in network θ pre-training; the trajectory refers to the temporal sequence of actions and states obtained by the interaction between the path tracking expert system and the environment;

[0017] 3.6) When the experience buffer is not full, select the next time slot to continue experience accumulation; when the buffer is full, perform θ network pre-training;

[0018] 3.7) Shuffle the records in the experience buffer and then extract the state s from it t 、Action a t , reward r t ,θ network parametersπ θ 、Future value expectation V Φ (s t ) quintuple;

[0019] 3.8) According to the state s t Generating the motion of the unmanned boat using the theta network

[0020] 3.9) Generate the action output corresponding to the θ network in each state by using the shuffled records in the buffer one by one;

[0021] 3.10) Update the network θ using the loss function;

[0022] 3.11) Repeat steps 3.2) to 3.10) according to the preset pre-training rounds until the θ network pre-training is completed;

[0023] 4) Model training;

[0024] 4.1) Initialize the training environment;

[0025] Select the water environment and the path to be tracked; set the initial position and initial bow direction of the unmanned boat; use the current water environment, the path to be tracked, the position of the unmanned boat and the initial bow direction information in the training environment as the initial state s0 at time t = 0, and send the state of the training environment at time t to the unmanned boat cluster path tracking control model;

[0026] 4.2) At the time slot [t, t+1), the UAV path tracking control model performs deduction based on the received state and generates the UAV steering command and jet speed:

[0027] 4.3) At time slot [t, t+1), the state s formed by the interaction between the unmanned boat path tracking control model and the training environment is t and action a t , reward r t ,θ network parametersπ θ 、Future value expectation V Φ (s t ) for training and updating of the unmanned boat path tracking control model:

[0028] 5) Evaluate the effect of the updated unmanned boat path tracking control model. If the effect meets the unmanned boat path tracking control requirements, end the unmanned boat path tracking control model training and obtain the pre-trained unmanned boat path tracking control model;

[0029] 6) Use the pre-trained unmanned boat path tracking control model to perform unmanned boat path tracking control.

[0030] According to the above scheme, in step 1), the training environment is set up as follows:

[0031] Use sensors to obtain wind, wave, and current data in different waters and weather conditions, which serve as the water environment for the training environment of the unmanned vehicle path tracking control model;

[0032] Use mechanism modeling or experimental methods to model the impact of different water environments on the trajectory of the unmanned boat;

[0033] Generate multiple paths to be tracked;

[0034] Based on the water environment modeling and the path to be tracked, the gym library in Python is used to build a training environment for the unmanned boat path tracking control model;

[0035] Receive wind, wave, and current data from the test environment as the scene water environment setting;

[0036] Set the initial position and initial bow direction of the unmanned boat;

[0037] The current water environment, the path to be tracked, the position of the unmanned boat and the bow direction information are taken as the state s0 at time t=0.

[0038] According to the above scheme, in step 3.10), the network θ is updated using the mean square error loss function.

[0039] According to the above scheme, step 4.2) is as follows:

[0040] 4.2.1) Calculate the target heading for the UAV path tracking control. Specifically, first set the radius of the LOS circle to the distance d between the UAV and the path to be tracked plus twice the ship length l, and the center of the circle to the UAV coordinates. Then, find the intersection of the LOS circle and the UAV path to be tracked. Then, set the starting point of the vector α to the UAV coordinates and the end point to the intersection of the LOS circle and the UAV path to be tracked. The vector α is the target heading for the UAV path tracking control.

[0041] 4.2.2) At time slot [t, t+1), obtain the state of the unmanned boat at time t s according to the state of the unmanned boat transmitted by the training environment t ;

[0042] The state includes the current water environment, the path to be tracked, the position of the unmanned boat, the bow heading information and the target heading;

[0043] The target heading vector α of the unmanned boat path tracking control is incorporated into the state s t ;

[0044] 4.2.3) Set the state s t Input the unmanned boat's θ network π θ , generate the action a of the unmanned boat t ;

[0045] 4.2.4) Encode action a t Decode and obtain the steering command and jet speed of the unmanned boat;

[0046] 4.2.5) According to the training environment and state s t and action a t , generate the reward r at time t t , the state s at time t+1 t+1 , termination signal done; details are as follows:

[0047] First, the position and bow direction of the boat at time t+1 are calculated based on the influence of wind, waves, and current on the movement of the unmanned boat and the steering command and jet speed of the unmanned boat.

[0048] Connect the position coordinates of the UAV at time t and time t+1, and calculate the actual heading of the UAV in the time slot [t, t+1);

[0049] The opposite of the absolute value of the difference between the actual heading and the target heading calculated by the LOS algorithm is used as the reward r for this time slot. t ;

[0050] The current water environment of the unmanned boat at time t+1, the path to be tracked, the position of the unmanned boat and the bow direction information, and the actual heading are taken as the state s t+1 ;

[0051] The unmanned boat path tracking control model determines whether the unmanned boat has reached the end point. If the unmanned boat has reached the end point, the current scene termination signal done = True is sent to the unmanned boat path tracking control model, and the next time slot is continued.

[0052] According to the above scheme, in step 4.3), the unmanned boat path tracking control model is updated as follows;

[0053] Get the loss function of the θ network and use this loss function to update the network θ;

[0054] Calculate the real future value function using time series difference

[0055] Using the future value function The network Φ is updated using the loss function.

[0056] Update the parameters of the θ network and the Φ network one by one using the shuffled records in the buffer;

[0057] According to the above scheme, in step 4.3), the loss function used in model training is:

[0058] According to the epsilon-clip principle, the advantage loss function after clipping is calculated using formula (2):

[0059]

[0060] Where B is the number of batches for training, π θ is the strategy of the unmanned boat numbered k, It is a demonstration strategy for the unmanned boat to interact with the environment. i is the advantage function calculated for the i-th batch of training, the clip function implements epsilon-clip, and ε is the hyperparameter of the selected epsilon-clip;

[0061] The entropy function is calculated using formula (3) to encourage iterative policy updates:

[0062]

[0063] Among them, B is the number of batches for batch training, o i is the observation obtained by the UAV in the i-th batch of training. The S function calculates the entropy change of the action obtained by the UAV when observing the environment;

[0064] Combine the functions calculated by formula (2) and formula (3) to obtain the loss function of the θ network, as shown in formula (4), and use this loss function to update the network θ;

[0065]

[0066] Among them, σ is the proportional coefficient of the combination of the two loss functions.

[0067] According to the above scheme, in step 4.3), the future value function is used The network Φ is updated using the following loss function;

[0068]

[0069] Among them, B is the number of batches for training, V Φ (s t ) is the expected future value of the network Φ, It is the real future value calculated by time series difference.

[0070] The beneficial effects produced by the present invention are:

[0071] 1. The method of the present invention uses a deep reinforcement learning algorithm as the core, accumulates experience by interacting with the virtual simulation environment, and obtains the operating effect of the current strategy in real time. By exploring the solution space, a better planning and control strategy is discovered, and the model is automatically iterated and updated.

[0072] 2. The method described in the present invention introduces prior knowledge, which makes updates faster and the effects more stable: the introduction of knowledge-driven assistance reduces the complexity of the solution space of the deep reinforcement learning model and is conducive to algorithm convergence; the knowledge-driven method provides guidance for the update of the data-driven deep reinforcement learning model, ensuring the stability of the model's effects.

[0073] 3. This invention introduces the LOS algorithm in the field of ship path tracking and uses the target heading calculated by LOS as the target for the unmanned boat path tracking algorithm to control the moving direction of the unmanned boat. Compared with the end-to-end reinforcement learning algorithm, this solution has stronger interpretability and faster algorithm convergence speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0075] Figure 1 It is a flow chart of a method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0076] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0077] like Figure 1 As shown, a knowledge and data fusion unmanned boat path tracking control method includes the following steps:

[0078] 1) Training environment establishment;

[0079] Use sensors to obtain wind, wave, and current data in different waters and weather conditions, which serve as the water environment for the training environment of the unmanned vehicle path tracking control model;

[0080] Use mechanism modeling or experimental methods to model the impact of different water environments on the trajectory of the unmanned boat;

[0081] Generate multiple paths to be tracked;

[0082] Based on the water environment modeling and the path to be tracked, the gym library in Python is used to build a training environment for the unmanned boat path tracking control model;

[0083] Receive wind, wave, and current data from the test environment as the scene water environment setting;

[0084] Set the initial position and initial bow direction of the unmanned boat;

[0085] The current water environment, the path to be tracked, the position of the unmanned boat and the bow direction information are taken as the state s0 at time t=0.

[0086] 2) Establishing the unmanned boat path tracking control model

[0087] Initialize the deep network θ and deep network Φ of each unmanned ship path tracking control; the deep network θ and deep network Φ correspond to two multi-layer neural network structures;

[0088] Set the parameter value of epsilon-clip in the model strategy update;

[0089] 3) Pre-train the network θ;

[0090] 3.1) θ network pre-training uses expert knowledge to build an unmanned boat path tracking control expert system using the LOS line of sight algorithm, which serves as an expert system to provide auxiliary training for the θ network pre-training;

[0091] 3.2) Initialize the training environment;

[0092] 3.3) In the time slot [t, t+1), the path tracking control expert system generates the rudder order and jet speed of the unmanned surface vehicle in the time slot according to the state transmitted from the training environment, and performs reasoning;

[0093] 3.4) In the time slot [t, t+1), the training scene is reasoned according to the rudder order and jet speed of each unmanned surface vehicle sent by the path tracking expert system, and the reward r at time t is generated t , the state s at time t+1 t+1 , the termination signal done;

[0094] 3.5) In the time slot [t, t+1), the trajectory formed by the interaction of the path tracking expert system and the training environment is stored in the experience buffer for the pre-training of the network θ; the trajectory refers to the time series of actions and states obtained by the interaction of the path tracking expert system and the environment;

[0095] 3.6) When the experience buffer is not full, the next time slot is selected to continue experience accumulation; when the buffer is full, the θ network is pre-trained;

[0096] 3.7) The records in the experience buffer are shuffled in order, and then the state s t , the action a t , the reward r t , the θ network parameter π θ , the future value expectation V Φ (s t ) five-tuple is taken out;

[0097] 3.8) According to the state s t , the action a of the unmanned surface vehicle is generated using the θ network

[0098] 3.9) The records in the buffer are used to generate the action output of the θ network corresponding to each state;

[0099] 3.10) The network θ is updated using the loss function;

[0100]

[0101] The θ network pre-training module uses the mean square error to update the network θ using the loss function of formula (1);

[0102] 3.11) According to the preset pre-training round, steps 3.2) to 3.10) are repeatedly looped until the θ network pre-training is completed;

[0103] 4) Model training;

[0104] 4.1) Training environment initialization;

[0105] Select the water environment and the path to be tracked; set the initial position and initial bow direction of the unmanned boat; use the current water environment, the path to be tracked, the position of the unmanned boat and the initial bow direction information in the training environment as the initial state s0 at time t = 0, and send the state of the training environment at time t to the unmanned boat cluster path tracking control model;

[0106] 4.2) At the time slot [t, t+1), the UAV path tracking control model performs deduction based on the received state and generates the UAV steering command and jet speed:

[0107] 4.2.1) Calculate the target heading for the UAV path tracking control. Specifically, first set the radius of the LOS circle to the distance d between the UAV and the path to be tracked plus twice the ship length l, and the center of the circle to the UAV coordinates. Then, find the intersection of the LOS circle and the UAV path to be tracked. Then, set the starting point of the vector α to the UAV coordinates and the end point to the intersection of the LOS circle and the UAV path to be tracked. The vector α is the target heading for the UAV path tracking control.

[0108] 4.2.2) At time slot [t, t+1), obtain the state of the unmanned boat at time t s according to the state of the unmanned boat transmitted by the training environment t ;

[0109] The state includes the current water environment, the path to be tracked, the position of the unmanned boat, the bow heading information and the target heading;

[0110] The target heading vector α of the unmanned boat path tracking control is incorporated into the state s t ;

[0111] 4.2.3) Set the state s t Input the unmanned boat's θ network π θ , generate the action a of the unmanned boat t ;

[0112] 4.2.4) Encode action a t Decode and obtain the steering command and jet speed of the unmanned boat;

[0113] 4.2.5) According to the training environment and state s t and action a t , generate the reward r at time t t , the state s at time t+1 t+1 , termination signal done; details are as follows:

[0114] First, the position and bow direction of the boat at time t+1 are calculated based on the influence of wind, waves, and current on the movement of the unmanned boat and the steering command and jet speed of the unmanned boat.

[0115] Connect the position coordinates of the UAV at time t and time t+1, and calculate the actual heading of the UAV in the time slot [t, t+1);

[0116] The opposite of the absolute value of the difference between the actual heading and the target heading calculated by the LOS algorithm is used as the reward r for this time slot. t ;

[0117] The current water environment of the unmanned boat at time t+1, the path to be tracked, the position of the unmanned boat and the bow direction information, and the actual heading are taken as the state s t+1 ;

[0118] The unmanned boat path tracking control model determines whether the unmanned boat has reached the end point. If the unmanned boat has reached the end point, the current scene termination signal done = True is sent to the unmanned boat path tracking control model, and the next time slot is continued.

[0119] 4.3) At time slot [t, t+1), the state s formed by the interaction between the unmanned boat path tracking control model and the training environment is t and action a t , reward r t ,θ network parametersπ θ 、Future value expectation V Φ (s t ) for training and updating of the unmanned boat path tracking control model:

[0120] The process of updating the unmanned boat path tracking control model is as follows:

[0121] According to the epsilon-clip principle, the advantage loss function after clipping is calculated using formula (2):

[0122]

[0123] Where B is the number of batches for training, π θ is the strategy of the unmanned boat numbered k, It is a demonstration strategy for the unmanned boat to interact with the environment. i is the advantage function calculated for the i-th batch of training, the clip function implements epsilon-clip, and ε is the hyperparameter of the selected epsilon-clip;

[0124] The entropy function is calculated using formula (3) to encourage iterative policy updates:

[0125]

[0126] Among them, B is the number of batches for batch training, o iis the observation obtained by the UAV in the i-th batch of training. The S function calculates the entropy change of the action obtained by the UAV when observing the environment;

[0127] Combine the functions calculated by formula (2) and formula (3) to obtain the loss function of the θ network, as shown in formula (4), and use this loss function to update the network θ;

[0128]

[0129] Among them, σ is the proportional coefficient of the combination of the two loss functions.

[0130] Get the loss function of the θ network and use this loss function to update the network θ;

[0131] Calculate the real future value function using time series difference

[0132] Using the future value function The network Φ is updated using the loss function.

[0133]

[0134] Among them, B is the number of batches for batch training, V Φ (s t ) is the expected future value of the network Φ, It is the real future value calculated by time series difference.

[0135] Update the parameters of the θ network and the Φ network one by one using the shuffled records in the buffer;

[0136] 5) Evaluate the effect of the updated unmanned boat path tracking control model. If the effect meets the unmanned boat path tracking control requirements, end the unmanned boat path tracking control model training and obtain the pre-trained unmanned boat path tracking control model;

[0137] 6) Use the pre-trained unmanned boat path tracking control model to perform unmanned boat path tracking control.

[0138] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all such improvements and changes should fall within the scope of protection of the appended claims of the present invention.

Claims

1. A knowledge and data fusion unmanned vehicle path tracking control method, characterized in that: The following steps are involved: 1) Training environment construction; 2) Establishing the unmanned boat path tracking control model Initialize the deep network θ and deep network Φ of each unmanned ship path tracking control; the deep network θ and deep network Φ correspond to two multi-layer neural network structures; Set the parameter value of epsilon-clip in the model strategy update; 3) Pre-train the network θ; 3.1) θ network pre-training uses expert knowledge to build an unmanned boat path tracking control expert system using the LOS line of sight algorithm, which serves as an expert system to provide auxiliary training for the θ network pre-training; 3.2) Initialize the training environment; 3.3) At the time slot [t, t+1), the path tracking control expert system performs deduction based on the state transmitted from the training environment to generate the steering command and jet speed of the unmanned boat in the time slot; 3.4) At time slot [t, t+1), the training scenario is simulated based on the steering commands and jet speeds of each unmanned boat sent by the path tracking expert system to generate the reward r at time t t , the state s at time t+1 t+1 , termination signal done; 3.5) At time slot [t, t+1), the trajectory formed by the interaction between the path tracking expert system and the training environment is stored in the experience buffer for use in network θ pre-training; the trajectory refers to the temporal sequence of actions and states obtained by the interaction between the path tracking expert system and the environment; 3.6) When the experience buffer is not full, select the next time slot to continue experience accumulation; when the buffer is full, perform θ network pre-training; 3.7) Shuffle the records in the experience buffer and then extract the state st, action at, and reward r from it t ,θ network parametersπ θ 、Future value expectation V Φ (s t ) quintuple; 3.8) According to the state s t Generating the motion of the unmanned boat using the theta network 3.9) Generate the action output corresponding to the θ network in each state by using the shuffled records in the buffer one by one; 3.10) Update the network θ using the loss function; 3.11) Repeat steps 3.2) to 3.10) according to the preset pre-training rounds until the θ network pre-training is completed; 4) Model training; 4.1) Initialize the training environment; Select the water environment and the path to be tracked; set the initial position and initial bow direction of the unmanned boat; The current water environment, the path to be tracked, the position of the unmanned boat, and the bow direction information in the training environment are used as the initial state s0 at time t = 0, and the state of the training environment at time t is sent to the unmanned boat cluster path tracking control model; 4.2) At the time slot [t, t+1), the UAV path tracking control model performs deduction based on the received state and generates the UAV steering command and jet speed: 4.3) At time slot [t, t+1), the state s formed by the interaction between the unmanned boat path tracking control model and the training environment is t and action a t , reward r t ,θ network parametersπ θ 、Future value expectation V Φ (s t ) for training and updating of the unmanned boat path tracking control model: 5) Evaluate the effect of the updated unmanned boat path tracking control model. If the effect meets the unmanned boat path tracking control requirements, end the unmanned boat path tracking control model training and obtain the pre-trained unmanned boat path tracking control model; 6) Use the pre-trained unmanned boat path tracking control model to perform unmanned boat path tracking control.

2. The knowledge and data fusion unmanned vehicle path tracking control method according to claim 1 is characterized in that: In step 1), the training environment is set up as follows: Use sensors to obtain wind, wave, and current data in several different waters and weather conditions, which serve as the water environment for the unmanned vehicle path tracking control model training environment; Use mechanism modeling or experimental methods to model the impact of different water environments on the trajectory of the unmanned boat; Generate multiple paths to be tracked; Based on the water environment modeling and the path to be tracked, the gym library in Python is used to build a training environment for the unmanned boat path tracking control model; Receive wind, wave, and current data from the test environment as the scene water environment setting; Set the initial position and initial bow direction of the unmanned boat; The current water environment, the path to be tracked, the position of the unmanned boat and the bow direction information are taken as the state s0 at time t=0.

3. The knowledge and data fusion unmanned vehicle path tracking control method according to claim 1 is characterized in that: In step 3.10), the network θ is updated using the mean square error loss function.

4. The knowledge and data fusion unmanned vehicle path tracking control method according to claim 1 is characterized in that: Step 4.2) is as follows: 4.2.1) Calculate the target heading for the UAV path tracking control. Specifically, first set the radius of the LOS circle to the distance d between the UAV and the path to be tracked plus twice the ship length l, and the center of the circle to the UAV coordinates. Then, find the intersection of the LOS circle and the UAV path to be tracked. Then, set the starting point of the vector α to the UAV coordinates and the end point to the intersection of the LOS circle and the UAV path to be tracked. The vector α is the target heading for the UAV path tracking control. 4.2.2) At time slot [t, t+1), obtain the state of the unmanned boat at time t s according to the state of the unmanned boat transmitted by the training environment t ; The state includes the current water environment, the path to be tracked, the position of the unmanned boat, the bow direction information and the target heading; Incorporate the target heading vector α of the unmanned boat path tracking control into the state st; 4.2.3) Set the state s t Input the unmanned boat's θ network π θ , generate the action a of the unmanned boat t ; 4.2.4) Encode action a t Decode and obtain the steering command and jet speed of the unmanned boat; 4.2.5) According to the training environment and state s t and action a t , generate the reward r at time t t , the state s at time t+1 t+1 , termination signal done; details are as follows: First, the position and bow direction of the boat at time t+1 are calculated based on the influence of wind, waves, and current on the movement of the unmanned boat and the steering command and jet speed of the unmanned boat. Connect the position coordinates of the UAV at time t and time t+1 to calculate the actual heading of the UAV in the time slot [t, t+1); The opposite of the absolute value of the difference between the actual heading and the target heading calculated by the LOS algorithm is used as the reward r for this time slot. t ; The current water environment of the unmanned boat at time t+1, the path to be tracked, the position of the unmanned boat and the bow direction information, and the actual heading are taken as the state s t+1 ; The unmanned boat path tracking control model determines whether the unmanned boat has reached the end point. If the unmanned boat has reached the end point, the current scene termination signal done = True is sent to the unmanned boat path tracking control model, and the next time slot is continued.

5. The knowledge and data fusion unmanned vehicle path tracking control method according to claim 1 is characterized in that: In step 4.3), the path tracking control model of the unmanned boat is updated as follows; Get the loss function of the θ network and use this loss function to update the network θ; Calculate the real future value function using time series difference Using the future value function Update the network Φ using the loss function; The parameters of the θ network and the Φ network are updated one by one using the shuffled records in the buffer.

6. The knowledge and data fusion unmanned vehicle path tracking control method according to claim 5 is characterized in that: In step 4.3), the loss function used in model training is: According to the epsilon-clip principle, the advantage loss function after clipping is calculated using formula (2): Where B is the number of batches for training, π θ is the strategy of the unmanned boat numbered k, It is a demonstration strategy for the unmanned boat to interact with the environment. i is the advantage function calculated for the i-th batch of training, the clip function implements epsilon-clip, and ε is the hyperparameter of the selected epsilon-clip; The entropy function is calculated using formula (3) to encourage iterative policy updates: Among them, B is the number of batches for batch training, o i is the observation obtained by the UAV in the i-th batch of training. The S function calculates the entropy change of the action obtained by the UAV when observing the environment; Combine the functions calculated by formula (2) and formula (3) to obtain the loss function of the θ network, as shown in formula (4), and use this loss function to update the network θ; Among them, σ is the proportional coefficient of the combination of the two loss functions.

7. The knowledge and data fusion unmanned vehicle path tracking control method according to claim 5 is characterized in that: In step 4.3), the future value function is used The network Φ is updated using the following loss function; Among them, B is the number of batches for training, V Φ (s t ) is the expected future value of the network Φ, It is the real future value calculated by time series difference.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images