Knowledge and data fused unmanned ship path tracking control method

Abstract

The invention discloses an unmanned ship path tracking control method based on knowledge and data fusion. The unmanned ship path tracking control method comprises the following steps: 1) building a training environment; 2) establishing an unmanned ship path tracking control model; 3) pre-training the network; 4) model training; 5) obtaining a pre-trained unmanned ship path tracking control model; and 6) performing unmanned ship path tracking control by using the pre-trained unmanned ship path tracking control model. According to the method, priori knowledge is introduced, updating is faster, the effect is more stable, the solution space complexity of a deep reinforcement learning model is reduced through introduction of knowledge-driven assistance, and algorithm convergence is facilitated; the knowledge driving method provides guidance for updating of the deep reinforcement learning model based on data driving, and the stability of the effect of the model is ensured.

Description

Technical Field

[0001] The invention relates to unmanned boat control technology, and in particular to an unmanned boat path tracking control method integrating knowledge and data. Background Art

[0002] Traditional unmanned boat swarm path tracking control methods mainly include the following categories: 1. Leader-follower control method; 2. Behavior-based control method: define the individual behavior rules and local control schemes of the intelligent agent in advance, so that the intelligent agent can independently complete a variety of behaviors; 3. Artificial potential field-based control method: by defining artificial potential functions and minimizing individual potential energy to control the formation, it has strong real-time computing capabilities and the ability to prevent sudden threats.

[0003] The traditional unmanned boat swarm path tracking control algorithm relies on the rule knowledge of specific application fields and is a knowledge-driven swarm intelligence algorithm. It has clear theory, is easy to integrate knowledge from various disciplines, and has high algorithm interpretability. However, it also has the disadvantages of high knowledge acquisition cost, incomplete solution space exploration, and difficulty in continuous learning and evolution.

[0004] Machine learning algorithms such as deep learning and reinforcement learning use data for iterative updates and are data-driven swarm intelligence algorithms. These algorithms do not require precise modeling of the problem and are highly versatile. They can fully explore the solution space and achieve continuous learning and evolution. However, they also have problems such as difficulty in theoretical analysis, poor algorithm interpretability, and the dimensionality disaster of the solution space. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a knowledge and data fusion unmanned boat path tracking control method in view of the defects in the prior art.

[0006] The technical solution adopted by the present invention to solve the technical problem is: a knowledge and data fusion unmanned boat path tracking control method, comprising the following steps:

[0007] 1) Training environment construction;

[0008] 2) Establish the path tracking control model of unmanned boat

[0009] Initialize the deep network θ and the deep network Φ of each unmanned ship path tracking control; the deep network θ and the deep network Φ correspond to two multi-layer neural network structures;

[0010] Set the parameter value of epsilon-clip in model strategy update;

[0011] 3) Pre-train the network θ;

[0012] 3.1) θ network pre-training uses expert knowledge to build an unmanned boat path tracking control expert system using the LOS line of sight algorithm, which serves as an expert system that provides auxiliary training for θ network pre-training;

[0013] 3.2) Initialize the training environment;

[0014] 3.3) In the time slot [t, t+1), the path tracking control expert system performs deduction based on the state transmitted from the training environment to generate the steering command and jet speed of the unmanned boat in the time slot;

[0015] 3.4) At time slot [t, t+1), the training scenario is deduced based on the steering commands and jet speeds of each unmanned boat sent by the path tracking expert system to generate the reward r at time t t , the state s at time t+1 t+1 , termination signal done;

[0016] 3.5) At time slot [t, t+1), the trajectory formed by the interaction between the path tracking expert system and the training environment is stored in the experience buffer for network θ pre-training; the trajectory refers to the temporal actions and states obtained by the interaction between the path tracking expert system and the environment;

[0017] 3.6) When the experience buffer is not full, select the next time slot to continue experience accumulation; when the buffer is full, perform θ network pre-training;

[0018] 3.7) Shuffle the order of records in the experience buffer and then extract the state s from it t 、Action a t , Reward t ,θ network parametersπ θ 、Future value expectation V Φ (s t ) quintuple;

[0019] 3.8) According to the state s t Using the theta network to generate the actions of the unmanned boat

[0020] 3.9) Generate the action output corresponding to the θ network in each state by using the shuffled records in the buffer one by one;

[0021] 3.10) Update the network θ using the loss function;

[0022] 3.11) According to the preset pre-training rounds, repeat steps 3.2) to 3.10) in a loop until the θ network pre-training is completed;

[0023] 4) Model training;

[0024] 4.1) Initialize the training environment;

[0025] Select the water environment and the path to be tracked; set the initial position and initial bow direction of the unmanned boat; use the current water environment, the path to be tracked, the position of the unmanned boat and the bow direction information in the training environment as the initial state s0 at time t=0, and send the state at time t in the training environment to the unmanned boat cluster path tracking control model;

[0026] 4.2) At the time slot [t, t+1), the path tracking control model of the unmanned boat is deduced based on the received state to generate the steering command and jet speed of the unmanned boat:

[0027] 4.3) At time slot [t, t+1), the state s formed by the interaction between the unmanned boat path tracking control model and the training environment is t and action a t , reward r t ,θ network parametersπ θ 、Future value expectation V Φ (s t ) for training and updating of the unmanned boat path tracking control model:

[0028] 5) Evaluate the effect of the updated unmanned boat path tracking control model. If the effect meets the unmanned boat path tracking control requirements, end the unmanned boat path tracking control model training and obtain the pre-trained unmanned boat path tracking control model;

[0029] 6) Use the pre-trained unmanned boat path tracking control model to perform unmanned boat path tracking control.

[0030] According to the above scheme, in step 1), the training environment is set up as follows:

[0031] Use sensors to obtain wind, wave, and current data in different waters and weather conditions as the water environment in the training environment of the unmanned boat path tracking control model;

[0032] Use mechanism modeling or experimental methods to model the impact of different water environments on the trajectory of the unmanned boat;

[0033] Generate multiple paths to be tracked;

[0034] According to the water environment modeling and the path to be tracked, the gym library in Python is used to build a training environment for the unmanned boat path tracking control model;

[0035] Receive wind, wave and current data from the test environment as scene water environment settings;

[0036] Set the initial position and initial bow direction of the unmanned boat;

[0037] The current water environment, the path to be tracked, the position of the unmanned boat and the bow direction information are taken as the state s0 at time t=0.

[0038] According to the above scheme, in step 3.10), the network θ is updated using the mean square error loss function.

[0039] According to the above scheme, step 4.2) is as follows:

[0040] 4.2.1) Calculate the target heading of the unmanned boat path tracking control; the specific method is to first set the radius of the LOS circle to the distance d between the unmanned boat and the path to be tracked plus twice the length of the boat l, and set the center of the circle to the coordinates of the unmanned boat, then find the intersection of the LOS circle and the path to be tracked by the unmanned boat, and then set the starting point of the vector α to the coordinates of the unmanned boat and the end point to the intersection of the LOS circle and the path to be tracked by the unmanned boat. The vector α is the target heading of the unmanned boat path tracking control;

[0041] 4.2.2) At time slot [t, t+1), obtain the state s of the unmanned boat at time t according to the state of the unmanned boat transmitted by the training environment t ;

[0042] The state includes the current water environment, the path to be tracked, the position of the unmanned boat, the bow direction information and the target heading;

[0043] The target heading vector α of the unmanned boat path tracking control is incorporated into the state s t ;

[0044] 4.2.3) Set the state s t Input the unmanned boat's θ network π θ , generate the action a of the unmanned boat t ;

[0045] 4.2.4) Encode action a t Decode and obtain the steering order and jet speed of the unmanned boat;

[0046] 4.2.5) According to the training environment and state s t and action a t , generate the reward r at time t t , the state s at time t+1 t+1 , termination signal done; the details are as follows:

[0047] First, according to the influence of wind, waves and current on the movement of the unmanned boat in the current water environment and the steering order and jet speed of the unmanned boat, the position and bow direction of the boat at time t+1 are calculated;

[0048] Connect the position coordinates of the unmanned boat at time t and time t+1, and calculate the actual heading of the unmanned boat in the time slot [t, t+1);

[0049] The opposite of the absolute value of the difference between the actual heading and the target heading calculated by the LOS algorithm is taken as the reward r for this time slot. t ;

[0050] The current water environment of the unmanned boat at time t+1, the path to be tracked, the position and bow direction information of the unmanned boat, and the actual heading are taken as the state s t+1 ;

[0051] The unmanned boat path tracking control model determines whether the unmanned boat has reached the end point. If the unmanned boat has reached the end point, the current screen termination signal done = True is set, and the model training environment is reinitialized. Otherwise, the current screen termination signal done = False is sent to the unmanned boat path tracking control model to continue to the next time slot.

[0052] According to the above scheme, in step 4.3), the unmanned boat path tracking control model is updated, and the process is as follows;

[0053] Get the loss function of the θ network and use this loss function to update the network θ;

[0054] Calculate the real future value function using time difference method

[0055] Using the Future Value Function The network Φ is updated using the loss function.

[0056] Using the shuffled records in the buffer one by one, update the parameters of the θ network and the Φ network;

[0057] According to the above scheme, in step 4.3), the loss function used in model training is:

[0058] According to the epsilon-clip principle, the clipped advantage loss function is calculated using formula (2):

[0059]

[0060] Where B is the number of batches for batch training, π θ is the strategy of the unmanned boat numbered k, is a demonstration strategy for an unmanned boat to interact with the environment. i is the calculated advantage function of the i-th batch of training, the clip function implements epsilon-clip, and ε is the selected hyperparameter of epsilon-clip;

[0061] The entropy function is calculated using formula (3) to encourage iterative updates of the strategy:

[0062]

[0063] Among them, B is the number of batches for batch training, o i is the observation obtained by the unmanned boat in the i-th batch of training. The S function calculates the entropy change of the action obtained by the unmanned boat when observing the environment;

[0064] Combine the functions calculated by formula (2) and formula (3) to obtain the loss function of the θ network, as shown in formula (4), and use the loss function to update the network θ;

[0065]

[0066] Among them, σ is the proportional coefficient of the combination of the two loss functions.

[0067] According to the above scheme, in step 4.3), the future value function is used The network Φ is updated using the following loss function;

[0068]

[0069] Among them, B is the number of batches for batch training, V Φ (s t ) is the expected future value of the network Φ, It is the real future value calculated by time series difference.

[0070] The beneficial effects produced by the present invention are:

[0071] 1. The method of the present invention uses a deep reinforcement learning algorithm as the kernel, accumulates experience by interacting with the virtual simulation environment, obtains the operating effect of the current strategy in real time, discovers better planning and control strategies by exploring the solution space, and realizes automatic iterative updating of the model.

[0072] 2. The method described in the present invention introduces prior knowledge, which makes updates faster and the effects more stable: the introduction of knowledge-driven assistance reduces the complexity of the solution space of the deep reinforcement learning model and is conducive to algorithm convergence; the knowledge-driven method provides guidance for the update of the data-driven deep reinforcement learning model, ensuring the stability of the model's effects.

[0073] 3. The present invention introduces the LOS algorithm in the field of ship path tracking, and uses the target heading calculated by LOS as the target of the unmanned boat path tracking algorithm to control the moving direction of the unmanned boat. Compared with the end-to-end reinforcement learning algorithm, this solution has stronger interpretability and faster algorithm convergence speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0075] Figure 1 It is a method flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0076] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0077] like Figure 1 As shown, a knowledge and data fusion unmanned boat path tracking control method includes the following steps:

[0078] 1) Training environment construction;

[0079] Use sensors to obtain wind, wave, and current data in different waters and weather conditions as the water environment in the training environment of the unmanned boat path tracking control model;

[0080] Use mechanism modeling or experimental methods to model the impact of different water environments on the trajectory of the unmanned boat;

[0081] Generate multiple paths to be tracked;

[0082] According to the water environment modeling and the path to be tracked, the gym library in Python is used to build a training environment for the unmanned boat path tracking control model;

[0083] Receive wind, wave and current data from the test environment as scene water environment settings;

[0084] Set the initial position and initial bow direction of the unmanned boat;

[0085] The current water environment, the path to be tracked, the position of the unmanned boat and the bow direction information are taken as the state s0 at time t=0.

[0086] 2) Establish the path tracking control model of unmanned boat

[0087] Initialize the deep network θ and the deep network Φ of each unmanned ship path tracking control; the deep network θ and the deep network Φ correspond to two multi-layer neural network structures;

[0088] Set the parameter value of epsilon-clip in model strategy update;

[0089] 3) Pre-train the network θ;

[0090] 3.1) θ network pre-training uses expert knowledge to build an unmanned boat path tracking control expert system using the LOS line of sight algorithm, which serves as an expert system that provides auxiliary training for θ network pre-training;

[0091] 3.2) Initialize the training environment;

[0092] 3.3) In the time slot [t, t+1), the path tracking control expert system performs deduction based on the state transmitted from the training environment to generate the steering command and jet speed of the unmanned boat in the time slot;

[0093] 3.4) At time slot [t, t+1), the training scenario is deduced based on the steering commands and jet speeds of each unmanned boat sent by the path tracking expert system to generate the reward r at time t t , the state s at time t+1 t+1 , termination signal done;

[0094] 3.5) At time slot [t, t+1), the trajectory formed by the interaction between the path tracking expert system and the training environment is stored in the experience buffer for network θ pre-training; the trajectory refers to the temporal actions and states obtained by the interaction between the path tracking expert system and the environment;

[0095] 3.6) When the experience buffer is not full, select the next time slot to continue experience accumulation; when the buffer is full, perform θ network pre-training;

[0096] 3.7) Shuffle the order of records in the experience buffer and then extract the state s from it t 、Action a t , Reward t ,θ network parametersπ θ 、Future value expectation V Φ (s t ) quintuple;

[0097] 3.8) According to the state s t Using the theta network to generate the actions of the unmanned boat

[0098] 3.9) Generate the action output corresponding to the θ network in each state by using the shuffled records in the buffer one by one;

[0099] 3.10) Update the network θ using the loss function;

[0100]

[0101] The θ network pre-training module uses the mean square error and the loss function of formula (1) to update the network θ;

[0102] 3.11) According to the preset pre-training rounds, repeat steps 3.2) to 3.10) in a loop until the θ network pre-training is completed;

[0103] 4) Model training;

[0104] 4.1) Initialize the training environment;

[0105] Select the water environment and the path to be tracked; set the initial position and initial bow direction of the unmanned boat; use the current water environment, the path to be tracked, the position of the unmanned boat and the bow direction information in the training environment as the initial state s0 at time t=0, and send the state at time t in the training environment to the unmanned boat cluster path tracking control model;

[0106] 4.2) At the time slot [t, t+1), the path tracking control model of the unmanned boat is deduced based on the received state to generate the steering command and jet speed of the unmanned boat:

[0107] 4.2.1) Calculate the target heading of the unmanned boat path tracking control; the specific method is to first set the radius of the LOS circle to the distance d between the unmanned boat and the path to be tracked plus twice the length of the boat l, and set the center of the circle to the coordinates of the unmanned boat, then find the intersection of the LOS circle and the path to be tracked by the unmanned boat, and then set the starting point of the vector α to the coordinates of the unmanned boat and the end point to the intersection of the LOS circle and the path to be tracked by the unmanned boat. The vector α is the target heading of the unmanned boat path tracking control;

[0108] 4.2.2) At time slot [t, t+1), obtain the state s of the unmanned boat at time t according to the state of the unmanned boat transmitted by the training environment t ;

[0109] The state includes the current water environment, the path to be tracked, the position of the unmanned boat, the bow direction information and the target heading;

[0110] The target heading vector α of the unmanned boat path tracking control is incorporated into the state s t ;

[0111] 4.2.3) Set the state s t Input the unmanned boat's θ network π θ , generate the action a of the unmanned boat t ;

[0112] 4.2.4) Encode action a t Decode and obtain the steering order and jet speed of the unmanned boat;

[0113] 4.2.5) According to the training environment and state s t and action a t , generate the reward r at time t t , the state s at time t+1 t+1 , termination signal done; the details are as follows:

[0114] First, according to the influence of wind, waves and current on the movement of the unmanned boat in the current water environment and the steering order and jet speed of the unmanned boat, the position and bow direction of the boat at time t+1 are calculated;

[0115] Connect the position coordinates of the unmanned boat at time t and time t+1, and calculate the actual heading of the unmanned boat in the time slot [t, t+1);

[0116] The opposite of the absolute value of the difference between the actual heading and the target heading calculated by the LOS algorithm is taken as the reward r for this time slot. t ;

[0117] The current water environment of the unmanned boat at time t+1, the path to be tracked, the position and bow direction information of the unmanned boat, and the actual heading are taken as the state s t+1 ;

[0118] The unmanned boat path tracking control model determines whether the unmanned boat has reached the end point. If the unmanned boat has reached the end point, the current screen termination signal done = True is set, and the model training environment is reinitialized. Otherwise, the current screen termination signal done = False is sent to the unmanned boat path tracking control model to continue to the next time slot.

[0119] 4.3) At time slot [t, t+1), the state s formed by the interaction between the unmanned boat path tracking control model and the training environment is t and action a t , reward r t ,θ network parametersπ θ 、Future value expectation V Φ (s t ) for training and updating of the unmanned boat path tracking control model:

[0120] The path tracking control model of the unmanned boat is updated, and the process is as follows;

[0121] According to the epsilon-clip principle, the clipped advantage loss function is calculated using formula (2):

[0122]

[0123] Where B is the number of batches for batch training, π θ is the strategy of the unmanned boat numbered k, is a demonstration strategy for an unmanned boat to interact with the environment. i is the calculated advantage function of the i-th batch of training, the clip function implements epsilon-clip, and ε is the selected hyperparameter of epsilon-clip;

[0124] The entropy function is calculated using formula (3) to encourage iterative updates of the strategy:

[0125]

[0126] Among them, B is the number of batches for batch training, o iis the observation obtained by the unmanned boat in the i-th batch of training. The S function calculates the entropy change of the action obtained by the unmanned boat when observing the environment;

[0127] Combine the functions calculated by formula (2) and formula (3) to obtain the loss function of the θ network, as shown in formula (4), and use the loss function to update the network θ;

[0128]

[0129] Among them, σ is the proportional coefficient of the combination of the two loss functions.

[0130] Get the loss function of the θ network and use this loss function to update the network θ;

[0131] Calculate the real future value function using time difference method

[0132] Using the Future Value Function The network Φ is updated using the loss function.

[0133]

[0134] Among them, B is the number of batches for batch training, V Φ (s t ) is the expected future value of the network Φ, It is the real future value calculated by time series difference.

[0135] Using the shuffled records in the buffer one by one, update the parameters of the θ network and the Φ network;

[0136] 5) Evaluate the effect of the updated unmanned boat path tracking control model. If the effect meets the unmanned boat path tracking control requirements, end the unmanned boat path tracking control model training and obtain the pre-trained unmanned boat path tracking control model;

[0137] 6) Use the pre-trained unmanned boat path tracking control model to perform unmanned boat path tracking control.

[0138] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all these improvements and changes should fall within the scope of protection of the appended claims of the present invention.

Claims

1. A knowledge and data fusion unmanned boat path tracking control method, characterized in that: The following steps are involved: 1) Training environment construction; 2) Establish the path tracking control model of unmanned boat Initialize the deep network θ and the deep network Φ of each unmanned ship path tracking control; the deep network θ and the deep network Φ correspond to two multi-layer neural network structures; Set the parameter value of epsilon-clip in model strategy update; 3) Pre-train the network θ; 3.1) θ network pre-training uses expert knowledge to build an unmanned boat path tracking control expert system using the LOS line of sight algorithm, which serves as an expert system that provides auxiliary training for θ network pre-training; 3.2) Initialize the training environment; 3.3) In the time slot [t, t+1), the path tracking control expert system performs deduction based on the state transmitted from the training environment to generate the steering command and jet speed of the unmanned boat in the time slot; 3.4) At time slot [t, t+1), the training scenario is deduced based on the steering commands and jet speeds of each unmanned boat sent by the path tracking expert system to generate the reward r at time t t , the state s at time t+1 t+1 , termination signal done; 3.5) At time slot [t, t+1), the trajectory formed by the interaction between the path tracking expert system and the training environment is stored in the experience buffer for network θ pre-training; the trajectory refers to the temporal actions and states obtained by the interaction between the path tracking expert system and the environment; 3.6) When the experience buffer is not full, select the next time slot to continue experience accumulation; when the buffer is full, perform θ network pre-training; 3.7) Shuffle the order of records in the experience buffer, and then extract the state st, action at, and reward r from it t 、θ network parameters π θ 、Future value expectation V Φ (s t ) quintuple; 3.8) According to the state s t Using the theta network to generate the actions of the unmanned boat 3.9) Generate the action output corresponding to the θ network in each state by using the shuffled records in the buffer one by one; 3.10) Update the network θ using the loss function; 3.11) According to the preset pre-training rounds, repeat steps 3.2) to 3.10) in a loop until the θ network pre-training is completed; 4) Model training; 4.1) Initialize the training environment; Select the water environment and the path to be tracked; set the initial position and initial bow direction of the unmanned boat; The current water environment, the path to be tracked, the position of the unmanned boat and the bow direction information in the training environment are taken as the initial state s0 at time t=0, and the state at time t in the training environment is sent to the unmanned boat cluster path tracking control model; 4.2) At the time slot [t, t+1), the path tracking control model of the unmanned boat is deduced based on the received state to generate the steering command and jet speed of the unmanned boat: 4.3) At time slot [t, t+1), the state s formed by the interaction between the unmanned boat path tracking control model and the training environment is t and action a t , reward r t ,θ network parametersπ θ 、Future value expectation V Φ (s t ) for training and updating of the unmanned boat path tracking control model: 5) Evaluate the effect of the updated unmanned boat path tracking control model. If the effect meets the unmanned boat path tracking control requirements, end the unmanned boat path tracking control model training and obtain the pre-trained unmanned boat path tracking control model; 6) Use the pre-trained unmanned boat path tracking control model to perform unmanned boat path tracking control.

2. The knowledge and data fusion unmanned boat path tracking control method according to claim 1 is characterized in that: In step 1), the training environment is set up as follows: Use sensors to obtain wind, wave, and current data in different waters and weather conditions as the water environment in the training environment of the unmanned boat path tracking control model; Use mechanism modeling or experimental methods to model the impact of different water environments on the trajectory of the unmanned boat; Generate multiple paths to be tracked; According to the water environment modeling and the path to be tracked, the gym library in Python is used to build a training environment for the unmanned boat path tracking control model; Receive wind, wave and current data from the test environment as scene water environment settings; Set the initial position and initial bow direction of the unmanned boat; The current water environment, the path to be tracked, the position of the unmanned boat and the bow direction information are taken as the state s0 at time t=0.

3. The knowledge and data fusion unmanned boat path tracking control method according to claim 1 is characterized in that: In step 3.10), the network θ is updated using the mean square error loss function.

4. The knowledge and data fusion unmanned boat path tracking control method according to claim 1 is characterized in that: Step 4.2) is as follows: 4.2.1) Calculate the target heading of the unmanned boat path tracking control; the specific method is to first set the radius of the LOS circle to the distance d between the unmanned boat and the path to be tracked plus twice the length of the boat l, and set the center of the circle to the coordinates of the unmanned boat, then find the intersection of the LOS circle and the path to be tracked by the unmanned boat, and then set the starting point of the vector α to the coordinates of the unmanned boat and the end point to the intersection of the LOS circle and the path to be tracked by the unmanned boat. The vector α is the target heading of the unmanned boat path tracking control; 4.2.2) At time slot [t, t+1), obtain the state s of the unmanned boat at time t according to the state of the unmanned boat transmitted by the training environment t ; The state includes the current water environment, the path to be tracked, the position of the unmanned boat, the bow direction information and the target heading; Incorporate the target heading vector α of the unmanned boat path tracking control into the state st; 4.2.3) Set the state s t Input the unmanned boat's θ network π θ , generate the action a of the unmanned boat t ; 4.2.4) Encode action a t Decode and obtain the steering order and jet speed of the unmanned boat; 4.2.5) According to the training environment and state s t and action a t , generate the reward r at time t t , the state s at time t+1 t+1 , termination signal done; the details are as follows: First, according to the influence of wind, waves and current on the movement of the unmanned boat in the current water environment and the steering order and jet speed of the unmanned boat, the position and bow direction of the boat at time t+1 are calculated; Connect the position coordinates of the unmanned boat at time t and time t+1, and calculate the actual heading of the unmanned boat in the time slot [t, t+1); The opposite of the absolute value of the difference between the actual heading and the target heading calculated by the LOS algorithm is taken as the reward r for this time slot. t ; The current water environment of the unmanned boat at time t+1, the path to be tracked, the position and bow direction information of the unmanned boat, and the actual heading are taken as the state s t+1 ; The unmanned boat path tracking control model determines whether the unmanned boat has reached the end point. If the unmanned boat has reached the end point, the current screen termination signal done = True is set, and the model training environment is reinitialized. Otherwise, the current screen termination signal done = False is sent to the unmanned boat path tracking control model to continue to the next time slot.

5. The knowledge and data fusion unmanned boat path tracking control method according to claim 1 is characterized in that: In step 4.3), the unmanned boat path tracking control model is updated, and the process is as follows; Get the loss function of the θ network and use this loss function to update the network θ; Calculate the real future value function using time difference method Using the Future Value Function The network Φ is updated using the loss function. The parameters of the θ network and the Φ network are updated one by one using the shuffled records in the buffer.

6. The unmanned boat path tracking control method based on knowledge and data fusion according to claim 5 is characterized in that: In step 4.3), the loss function used in model training is: According to the epsilon-clip principle, the clipped advantage loss function is calculated using formula (2): Where B is the number of batches for batch training, π θ is the strategy of the unmanned boat numbered k, is a demonstration strategy for an unmanned boat to interact with the environment. i is the calculated advantage function of the i-th batch of training, the clip function implements epsilon-clip, and ε is the selected hyperparameter of epsilon-clip; The entropy function is calculated using formula (3) to encourage iterative updates of the strategy: Among them, B is the number of batches for batch training, o i is the observation obtained by the unmanned boat in the i-th batch of training. The S function calculates the entropy change of the action obtained by the unmanned boat when observing the environment; Combine the functions calculated by formula (2) and formula (3) to obtain the loss function of the θ network, as shown in formula (4), and use the loss function to update the network θ; Among them, σ is the proportional coefficient of the combination of the two loss functions.

7. The knowledge and data fusion unmanned boat path tracking control method according to claim 5 is characterized in that: In step 4.3), the future value function is used The network Φ is updated using the following loss function; Among them, B is the number of batches for batch training, V Φ (s t ) is the expected future value of the network Φ, It is the real future value calculated by time series difference.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images