Unmanned ship cluster path planning method based on fusion of reinforcement learning and domain knowledge

Abstract

The invention discloses an unmanned ship cluster path planning method based on fusion of reinforcement learning and domain knowledge. The method comprises the following steps: 1) building an unmanned ship cluster path planning model training scene environment; 2) establishing an unmanned ship path planning model; initializing a deep network theta < k > and a deep network phi < k > of path planning of each unmanned ship; 3) pre-training the network theta k; 4) training the unmanned ship path planning model; 5) evaluating the effect of the updated unmanned ship cluster path planning model, and if the effect meets the unmanned ship cluster path planning requirement, ending the training of the unmanned ship cluster path planning model to obtain a pre-trained unmanned ship cluster path planning model; and 6) solving unmanned ship cluster path planning by using the trained unmanned ship cluster path planning model. According to the method, priori knowledge is introduced for model pre-training, so that the solution space complexity of the model is reduced; and meanwhile, guidance is provided for updating of the model, and the stability of the effect is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of path planning, and in particular to an unmanned boat cluster path planning method integrating reinforcement learning with domain knowledge. Background Art

[0002] The path planning algorithms for unmanned boat swarms mainly include the following categories: 1. Graphic search algorithms: such as Dijkstra algorithm, A* algorithm, etc., which are classic single-source shortest path search algorithms in computer graphics; 2. Virtual potential field method: such as artificial potential field method and fast marching method, this type of method constructs a virtual potential field in the map, and then uses the gradient descent method to generate a path; 3. Random sampling algorithm: such as probabilistic roadmap method and rapid expansion random tree method, which uses a given initial point as the root node of the random tree to quickly and effectively search the feasible domain space, and guide the search to the blank area and to the target point by generating random points. 4. Traditional intelligent algorithms: such as genetic algorithms, particle swarm algorithms, ant colony algorithms, etc., so that the population as a whole is continuously iterated and converges towards the optimal solution. Traditional optimization methods usually require the acquisition of rule knowledge for specific application fields in advance. The planning effect is highly dependent on the accuracy of expert prior knowledge, and it is difficult to continuously learn evolution and iterative update. Debugging to the best effect requires a lot of manpower and material resources.

[0003] The traditional unmanned boat swarm path planning algorithm relies on the rule knowledge of specific application fields and is a knowledge-driven swarm intelligence algorithm. It has clear theory, is easy to integrate knowledge from various disciplines, and has high algorithm interpretability. However, it also has the disadvantages of high knowledge acquisition cost, incomplete exploration of solution space, and difficulty in continuous learning and evolution. Machine learning algorithms such as deep learning and reinforcement learning use data for iterative updates and are data-driven swarm intelligence algorithms. This type of algorithm does not require precise modeling of the problem and has strong versatility. It can fully explore the solution space and achieve continuous learning and evolution. However, it also has problems such as difficult theoretical analysis, poor algorithm interpretability, and dimensional disaster of solution space. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a path planning method for a cluster of unmanned boats by integrating reinforcement learning with domain knowledge in view of the defects in the prior art.

[0005] The technical solution adopted by the present invention to solve the technical problem is: a method for unmanned boat cluster path planning integrating reinforcement learning and domain knowledge, comprising the following steps:

[0006] 1) Build a training scenario for the unmanned boat swarm path planning model;

[0007] 1.1) Use 3D reconstruction technology to obtain the hydrological and geographical information of a river, including the edge of the river, the water depth at each location of the river, etc., as the basic scene of the model training environment;

[0008] 1.2) Obtain the shape and kinematic constraints of the controlled unmanned ship, calculate the safe navigation area of ​​the unmanned ship in the waters according to the basic scenario, mark the unsafe area as a no-go zone, and abstract it as a static obstacle in the model training environment;

[0009] 1.3) Collect the shape information of several buoys and ships, obtain their kinematic constraints such as moving speed, and add them to the candidate dynamic obstacle library as candidate dynamic obstacles;

[0010] 1.4) Select several obstacles from the candidate dynamic obstacle library, design the motion mode of the dynamic obstacle according to its kinematic constraints, and use these dynamic obstacles with set motion modes as dynamic obstacles in the model training environment;

[0011] 1.5) Use the gym library in Python to build a model training environment based on the basic environment, static obstacles, and dynamic obstacles;

[0012] 2) Establish an unmanned boat path planning model;

[0013] Initialize the deep network θ for each unmanned ship path planning k and the deep network Φ k , where k is the number of the unmanned ship; the depth network θ k and the deep network Φ k The corresponding ones are two multi-layer neural network structures;

[0014] Set the parameter value of epsilon-clip in the policy update;

[0015] 3) For the network θ k Conduct pre-training;

[0016] 3.1) Network θ k Pre-training, using expert knowledge, using the artificial potential field method to build an unmanned ship cluster path planning expert system, for the network θ k The pre-training module provides an expert system to assist in training;

[0017] 3.2) Initialize the training environment;

[0018] 3.3) In the time slot [t, t+1), the path planning expert system deduce the state transmitted from the training environment and generate the path that each unmanned ship should take in the time slot;

[0019] 3.4) In the time slot [t, t+1), the network θ k Pre-training: Based on the movement of each unmanned ship sent by the path planning expert system, the training scenario is deduced to generate the reward r at time t. t k, the state at time t+1 Termination signal done;

[0020] 3.5) At time slot [t, t+1), the trajectory formed by the interaction between the path planning expert system and the model training environment is stored in the experience buffer for network θ k Pre-training use;

[0021] 3.6) When the experience buffer is not full, select the next time slot to continue accumulating experience; when the buffer is full, perform network θ k Pre-training;

[0022] 3.7) Shuffle the records in the experience buffer and then retrieve the state from it action Reward t k ,θ k Network parameters Future value expectations quintuple;

[0023] 3.8) Network θ k Pre-training uses temporal difference to calculate the true future value function

[0024] 3.9) Using the Future Value Function The network Φ is trained using the following loss function k Make updates;

[0025]

[0026] Among them, B is the number of batches for batch training, and n is the number of unmanned ships;

[0027] 3.10) Network θ k Pre-training uses the shuffled records in the buffer one by one to train the network θ in the model k Update the parameters of

[0028] 3.11) According to the preset pre-training rounds, repeat steps 3.2) to 3.10) until the network θ k Pre-training is finished;

[0029] Step 4) training the unmanned boat path planning model;

[0030] 4.1) Model training environment initialization:

[0031] The information of each unmanned ship, static obstacle, and dynamic obstacle in the current waters of the training environment is used as the initial state at time t = 0 Send to the unmanned ship cluster path planning model;

[0032] 4.2) In the time slot [t, t+1), the unmanned ship cluster path planning model is deduced from the received initial state to generate the path that each unmanned ship should take in the time slot:

[0033] 4.2.1) Receive the status of each unmanned ship, static obstacle, and dynamic obstacle information in the waters of the training environment at time t

[0034] 4.2.2) Change the status Input the θ of each unmanned ship k network Generate the actions of each unmanned ship

[0035] 4.2.3) Encoding Action Decode and calculate the position coordinates of the unmanned ship at time t+1, and connect the positions of the unmanned ship at time t and time t+1 to obtain the path of the unmanned ship in this time slot;

[0036] 4.2.4) Obtain the movement status of each unmanned ship;

[0037] 4.3) At time slot [t, t+1), the model training scenario is deduced based on the movement of each unmanned ship to generate the reward r at time t t k and the state at time t+1 or a termination signal:

[0038] 4.3.1) Receive the movement status of each unmanned ship;

[0039] 4.3.2) Calculate the position of each dynamic obstacle at time t+1 according to the motion pattern of each dynamic obstacle;

[0040] 4.3.3) Determine whether a collision occurs based on the movement of each unmanned ship, the coordinates of the static obstacle, and the movement of the dynamic obstacle. If a collision occurs, a negative reward r is generated for the collision. t k and the termination signal done, and the training environment is initialized again; if no collision occurs, the information of each unmanned ship, static obstacle, and dynamic obstacle in the current waters at time t+1 is used as the state Send to the unmanned ship cluster path planning system;

[0041] 4.3.4) Determine whether all unmanned ships have safely reached the destination. If all unmanned ships have safely reached the destination, the positive reward r for safe arrival will be t The termination signal True of this scene is sent to the unmanned ship cluster path planning system to re-initialize the model training environment; otherwise, the intrinsic reward r of each unmanned ship moving between the time slots [t, t+1) is sent to the unmanned ship cluster path planning system.t k And the current screen termination signal False;

[0042] 4.4) The trajectory quintuple generated by the time slot [t, t+1) is: the current state Select Action Get rewards t k ,θ k Network parameters and future value expectations The data is packaged and stored in the experience buffer as experience for training and updating of the unmanned ship cluster path planning model:

[0043] The update process is as follows:

[0044] 4.4.1) Shuffle the records in the experience buffer and extract the state from it action Reward t k ,θ k Network parameters Future value expectations quintuple;

[0045] 4.4.2) Calculate the advantage function A using the generalized advantage estimation algorithm;

[0046] 4.4.3) According to the epsilon-clip principle, use formula (2) to calculate the advantage loss function after clipping:

[0047]

[0048] Among them, B is the number of batches for batch training, n is the number of unmanned ships, is the strategy of the unmanned ship numbered k, is the demonstration strategy of the unmanned boat numbered k that interacts with the environment, is the calculated advantage function of the i-th batch of training, the clip function implements epsilon-clip, and ε is the selected hyperparameter of epsilon-clip;

[0049] 4.4.4) Calculate the entropy function to encourage iterative updates of the strategy:

[0050]

[0051] Among them, B is the number of batches for batch training, n is the number of unmanned ships, is the observation obtained by the unmanned ship numbered k in the i-th batch of training. The S function calculates the entropy change of the action obtained by the strategy of the unmanned ship when observing the environment;

[0052] 4.4.5) Combine the functions to get θ k The loss function of the network is used to k To update:

[0053] Among them, σ is the proportional coefficient of the combination of the two loss functions;

[0054]

[0055] 4.4.6) Calculate the real future value function using time series difference

[0056] 4.4.7) Using the Future Value Function The network Φ is trained using the following loss function k To update:

[0057]

[0058] Among them, B is the number of batches for batch training, n is the number of unmanned ships, Is the network Φ k Estimated future value expectations, It is the real future value calculated by time series difference;

[0059] 4.4.8) Use the shuffled records in the buffer one by one to calculate the θ in the model k Network and Φ k Network parameters are updated;

[0060] 5) Evaluate the effect of the updated unmanned ship cluster path planning model. If the effect meets the unmanned ship cluster path planning requirements, end the unmanned ship cluster path planning model training and obtain the pre-trained unmanned ship cluster path planning model. Otherwise, return to step 3) to accumulate experience and continue training;

[0061] 6) Use the trained unmanned ship cluster path planning model to solve the unmanned ship cluster path planning problem.

[0062] The beneficial effects produced by the present invention are:

[0063] 1) With the introduction of prior knowledge, the model is updated more quickly and the effect is more stable:

[0064] The introduction of knowledge-driven assistance reduces the complexity of the solution space of the deep reinforcement learning model and helps to accelerate the convergence speed; the knowledge-driven method provides guidance for the update of the data-driven deep reinforcement learning model and ensures the stability of the effect.

[0065] 2) Accumulate experience through interaction with the virtual simulation environment, obtain the operating effect of the current model strategy in real time, discover better planning strategies through exploration of the solution space, and realize automatic iterative update of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0067] Figure 1 It is a flow chart of a method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0069] like Figure 1 As shown, a method for unmanned boat cluster path planning by integrating reinforcement learning and domain knowledge includes the following steps:

[0070] 1) Build a training scenario for the unmanned boat swarm path planning model;

[0071] 1.1) Use 3D reconstruction technology to obtain the hydrological and geographical information of a river, including the edge of the river, the water depth at each location of the river, etc., as the basic scene of the model training environment;

[0072] 1.2) Obtain the shape and kinematic constraints of the controlled unmanned ship, calculate the safe navigation area of ​​the unmanned ship in the waters according to the basic scenario, mark the unsafe area as a no-go zone, and abstract it as a static obstacle in the model training environment;

[0073] 1.3) Collect the shape information of several buoys and ships, obtain their kinematic constraints such as moving speed, and add them to the candidate dynamic obstacle library as candidate dynamic obstacles;

[0074] 1.4) Select several obstacles from the candidate dynamic obstacle library, design the motion mode of the dynamic obstacle according to its kinematic constraints, and use these dynamic obstacles with set motion modes as dynamic obstacles in the model training environment;

[0075] 1.5) Use the gym library in Python to build a model training environment based on the basic environment, static obstacles, and dynamic obstacles;

[0076] 2) Establish an unmanned boat path planning model;

[0077] Initialize the deep network θ for each unmanned ship path planning k and the deep network Φ k, where k is the number of the unmanned ship; the depth network θ k and the deep network Φ k The corresponding ones are two multi-layer neural network structures;

[0078] Set the parameter value of epsilon-clip in the policy update;

[0079] 3) For the network θ k Conduct pre-training;

[0080] 3.1) Network θ k Pre-training, using expert knowledge, using the artificial potential field method to build an unmanned ship cluster path planning expert system, for the network θ k The pre-training module provides an expert system to assist in training;

[0081] 3.2) Initialize the training environment;

[0082] 3.3) In the time slot [t, t+1), the path planning expert system deduce the state transmitted from the training environment and generate the path that each unmanned ship should take in the time slot;

[0083] 3.4) In the time slot [t, t+1), the network θ k Pre-training: Based on the movement of each unmanned ship sent by the path planning expert system, the training scenario is deduced to generate the reward r at time t. t k , the state at time t+1 Termination signal done;

[0084] 3.5) At time slot [t, t+1), the trajectory formed by the interaction between the path planning expert system and the model training environment is stored in the experience buffer for network θ k Pre-training use;

[0085] 3.6) When the experience buffer is not full, select the next time slot to continue accumulating experience; when the buffer is full, perform network θ k Pre-training;

[0086] 3.7) Shuffle the records in the experience buffer and then retrieve the state from it action Reward t k ,θ k Network parameters Future value expectations quintuple;

[0087] 3.8) Network θ k Pre-training uses temporal difference to calculate the true future value function

[0088] 3.9) Using the Future Value Function The network Φ is trained using the following loss function k Make updates;

[0089]

[0090] Among them, B is the number of batches for batch training, and n is the number of unmanned ships;

[0091] 3.10) Network θ k Pre-training uses the shuffled records in the buffer one by one to train the network θ in the model k Update the parameters of

[0092] 3.11) According to the preset pre-training rounds, repeat steps 3.2) to 3.10) until the network θ k Pre-training is finished;

[0093] Step 4: Train the unmanned boat path planning model;

[0094] 4.1) Model training environment initialization:

[0095] The information of each unmanned ship, static obstacle, and dynamic obstacle in the current waters of the training environment is used as the initial state at time t = 0 Send to the unmanned ship cluster path planning model;

[0096] 4.2) In the time slot [t, t+1), the unmanned ship cluster path planning model is deduced from the received initial state to generate the path that each unmanned ship should take in the time slot:

[0097] 4.2.1) Receive the status of each unmanned ship, static obstacle, and dynamic obstacle information in the waters of the training environment at time t

[0098] 4.2.2) Change the status Input the θ of each unmanned ship k network Generate the actions of each unmanned ship

[0099] 4.2.3) Encoding Action Decode and calculate the position coordinates of the unmanned ship at time t+1, and connect the positions of the unmanned ship at time t and time t+1 to obtain the path of the unmanned ship in this time slot;

[0100] 4.2.4) Obtain the movement status of each unmanned ship;

[0101] 4.3) At time slot [t, t+1), the model training scenario is deduced based on the movement of each unmanned ship to generate the reward r at time tt k and the state at time t+1 or a termination signal:

[0102] 4.3.1) Receive the movement status of each unmanned ship;

[0103] 4.3.2) Calculate the position of each dynamic obstacle at time t+1 according to the motion pattern of each dynamic obstacle;

[0104] 4.3.3) Determine whether a collision occurs based on the movement of each unmanned ship, the coordinates of the static obstacle, and the movement of the dynamic obstacle. If a collision occurs, a negative reward r is generated for the collision. t k and the termination signal done, and the training environment is initialized again; if no collision occurs, the information of each unmanned ship, static obstacle, and dynamic obstacle in the current waters at time t+1 is used as the state Send to the unmanned ship cluster path planning system;

[0105] 4.3.4) Determine whether all unmanned ships have safely reached the destination. If all unmanned ships have safely reached the destination, the positive reward r for safe arrival will be t The termination signal True of this scene is sent to the unmanned ship cluster path planning system to re-initialize the model training environment; otherwise, the intrinsic reward r of each unmanned ship moving between the time slots [t, t+1) is sent to the unmanned ship cluster path planning system. t k And the current screen termination signal False;

[0106] 4.4) The trajectory quintuple generated by the time slot [t, t+1) is: the current state Select Action Get rewards t k ,θ k Network parameters and future value expectations The data is packaged and stored in the experience buffer as experience for model update and for training and updating of the unmanned ship cluster path planning model:

[0107] The update process is as follows:

[0108] 4.4.1) Shuffle the records in the experience buffer and extract the state from it action Reward t k ,θ k Network parameters Future value expectations quintuple;

[0109] 4.4.2) Calculate the advantage function A using the generalized advantage estimation algorithm;

[0110] 4.4.3) According to the epsilon-clip principle, use formula (2) to calculate the advantage loss function after clipping:

[0111]

[0112] Among them, B is the number of batches for batch training, n is the number of unmanned ships, is the strategy of the unmanned ship numbered k, is the demonstration strategy of the unmanned boat numbered k that interacts with the environment, is the calculated advantage function of the i-th batch of training, the clip function implements epsilon-clip, and ε is the selected hyperparameter of epsilon-clip;

[0113] 4.4.4) Calculate the entropy function to encourage iterative updates of the strategy:

[0114]

[0115] Among them, B is the number of batches for batch training, n is the number of unmanned ships, is the observation obtained by the unmanned ship numbered k in the i-th batch of training. The S function calculates the entropy change of the action obtained by the strategy of the unmanned ship when observing the environment;

[0116] 4.4.5) Combine the functions to get θ k The loss function of the network is used to k To update:

[0117] Among them, σ is the proportional coefficient of the combination of the two loss functions;

[0118]

[0119] 4.4.6) Calculate the real future value function using time series difference

[0120] 4.4.7) Using the Future Value Function The network Φ is trained using the following loss function k To update:

[0121]

[0122] Among them, B is the number of batches for batch training, n is the number of unmanned ships, Is the network Φ k Estimated future value expectations, It is the real future value calculated by time series difference;

[0123] 4.4.8) Use the shuffled records in the buffer one by one to calculate the θ in the model k Network and Φ k Network parameters are updated;

[0124] 5) Evaluate the effect of the updated unmanned ship cluster path planning model. If the effect meets the unmanned ship cluster path planning requirements, end the unmanned ship cluster path planning model training and obtain the pre-trained unmanned ship cluster path planning model. Otherwise, return to step 3) to accumulate experience and continue training;

[0125] 6) Use the trained unmanned ship cluster path planning model to solve the unmanned ship cluster path planning problem.

[0126] According to the mission, the target coordinates of the unmanned ship's movement are set and the boundaries of the unmanned ship's navigation area are determined;

[0127] The network θ in the pre-trained unmanned ship cluster path planning model is k Network parameter import;

[0128] Set the initial time t=0;

[0129] At time t, the coordinates and shape information of static obstacles such as islands and dynamic obstacles such as buoys in the navigation area of ​​the unmanned ship are obtained and input into the path planning model as states;

[0130] At the time slot [t, t+1), the path planning module uses the trained unmanned ship cluster path planning model to plan the movement path of each unmanned ship within this period;

[0131] If at time t+1, the path planning module determines that the paths of all the planned unmanned ships have reached the destination, the process ends if all the unmanned ships have reached the destination, thus completing the task of automatic planning of the unmanned ship cluster.

[0132] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all these improvements and changes should fall within the scope of protection of the appended claims of the present invention.

Claims

1. A method for unmanned boat cluster path planning based on the integration of reinforcement learning and domain knowledge, characterized in that: The following steps are involved: 1) Build a training scenario environment for the unmanned boat swarm path planning model; 2) Establish an unmanned boat path planning model; Initialize the deep network θ for each unmanned ship path planning k and the deep network Φ k , where k is the number of the unmanned ship; the depth network θ k and the deep network Φ k The corresponding ones are two multi-layer neural network structures; Set the parameter value of epsilon-clip in the policy update; 3) For the network θ k Conduct pre-training; 3.1) Network θ k Pre-training, using expert knowledge, using the artificial potential field method to build an unmanned ship cluster path planning expert system, for the network θ k The pre-training module provides an expert system to assist in training; 3.2) Initialize the training environment; 3.3) In the time slot [t, t+1), the path planning expert system deduce the state transmitted from the training environment and generate the path that each unmanned ship should take in the time slot; 3.4) In the time slot [t, t+1), the network θ k Pre-training: Based on the movement of each unmanned ship sent by the path planning expert system, the training scenario is deduced to generate the reward r at time t. t k , the state at time t+1 Termination signal done; 3.5) At time slot [t, t+1), the trajectory formed by the interaction between the path planning expert system and the model training environment is stored in the experience buffer for network θ k Pre-training use; 3.6) When the experience buffer is not full, select the next time slot to continue accumulating experience; when the buffer is full, perform network θ k Pre-training; 3.7) Shuffle the records in the experience buffer and then retrieve the state from it action Reward t k ,θ k Network parameters Future value expectations quintuple; 3.8) Network θ k Pre-training uses temporal difference to calculate the true future value function 3.9) Using the Future Value Function Use the loss function to train the network Φ k Make updates; 3.10) Network θ k Pre-training uses the shuffled records in the buffer one by one to train the network θ in the model k Update the parameters of 3.11) According to the preset pre-training rounds, repeat steps 3.2) to 3.10) until the network θ k Pre-training is finished; Step 4) training the unmanned boat path planning model; 4.1) Model training environment initialization: The information of each unmanned ship, static obstacle, and dynamic obstacle in the current waters of the training environment is used as the initial state at time t = 0 Send to the unmanned ship cluster path planning model; 4.2) In the time slot [t, t+1), the unmanned ship cluster path planning model is deduced from the received initial state to generate the path that each unmanned ship should take in the time slot: 4.3) At time slot [t, t+1), the model training scenario is deduced based on the movement of each unmanned ship to generate the reward r at time t t k and the state at time t+1 or a termination signal: 4.4) The trajectory quintuple generated by the time slot [t, t+1) is: the current state Select Action Get rewards t k ,θ k Network parameters and future value expectations The data is packaged and stored in the experience buffer as experience for training and updating of the unmanned ship cluster path planning model: 5) Evaluate the effect of the updated unmanned ship cluster path planning model. If the effect meets the unmanned ship cluster path planning requirements, end the unmanned ship cluster path planning model training and obtain the pre-trained unmanned ship cluster path planning model. Otherwise, return to step 3) to accumulate experience and continue training; 6) Use the trained unmanned ship cluster path planning model to solve the unmanned ship cluster path planning problem.

2. The unmanned boat swarm path planning method integrating reinforcement learning and domain knowledge according to claim 1 is characterized in that: In step 1), a model training scenario environment is constructed as follows: 1.1) Use 3D reconstruction technology to obtain the hydrological and geographical information of a river, including the edge of the river and the water depth at various locations in the river, as the basic scene of the model training environment; 1.2) Obtain the shape and kinematic constraints of the controlled unmanned ship, calculate the safe navigation area of ​​the unmanned ship in the waters according to the basic scenario, mark the unsafe area as a no-go zone, and abstract it as a static obstacle in the model training environment; 1.3) Collect the shape information of several buoys and ships, obtain their kinematic constraints such as moving speed, and add them to the candidate dynamic obstacle library as candidate dynamic obstacles; 1.4) Select several obstacles from the candidate dynamic obstacle library, design the motion mode of the dynamic obstacle according to its kinematic constraints, and use these dynamic obstacles with set motion modes as dynamic obstacles in the model training environment; 1.5) Use the gym library in Python to build a model training environment based on basic scenarios, static obstacles, and dynamic obstacles.

3. The unmanned boat swarm path planning method integrating reinforcement learning and domain knowledge according to claim 1 is characterized in that: In step 3.9), the following loss function is used to adjust the network Φ k Make updates; Among them, B is the batch number of batch training, and n is the number of unmanned ships.

4. The unmanned boat swarm path planning method integrating reinforcement learning and domain knowledge according to claim 1 is characterized in that, in the step 4.2), the specific steps are as follows: 4.2.1) Receive the status of each unmanned ship, static obstacle, and dynamic obstacle information in the waters of the training environment at time t 4.2.2) Change the status Input the θ of each unmanned ship k network Generate the actions of each unmanned ship 4.2.3) Encoding Action Decode and calculate the position coordinates of the unmanned ship at time t+1, and connect the positions of the unmanned ship at time t and time t+1 to obtain the path of the unmanned ship in this time slot; 4.2.4) Obtain the movement status of each unmanned ship.

5. The unmanned boat swarm path planning method integrating reinforcement learning and domain knowledge according to claim 1 is characterized in that: In the step 4.3), the details are as follows: 4.3.1) Receive the movement status of each unmanned ship; 4.3.2) Calculate the position of each dynamic obstacle at time t+1 according to the motion pattern of each dynamic obstacle; 4.3.3) Determine whether a collision occurs based on the movement of each unmanned ship, the coordinates of the static obstacle, and the movement of the dynamic obstacle. If a collision occurs, a negative reward r is generated for the collision. t k and the termination signal done, and the training environment is initialized again; if no collision occurs, the information of each unmanned ship, static obstacle, and dynamic obstacle in the current waters at time t+1 is used as the state Send to the unmanned ship cluster path planning system; 4.3.4) Determine whether all unmanned ships have safely reached the destination. If all unmanned ships have safely reached the destination, the positive reward r for safe arrival will be t The termination signal True of this scene is sent to the unmanned ship cluster path planning system to re-initialize the model training environment; otherwise, the intrinsic reward r of each unmanned ship moving between the time slots [t, t+1) is sent to the unmanned ship cluster path planning system. t k And the current screen termination signal is False.

6. The unmanned boat swarm path planning method integrating reinforcement learning and domain knowledge according to claim 1 is characterized in that: In step 4.4), the updating process is as follows: 4.4.1) Shuffle the records in the experience buffer and extract the state from it action Reward t k ,θ k Network parameters Future value expectations quintuple; 4.4.2) Calculate the advantage function A using the generalized advantage estimation algorithm; 4.4.3) According to the epsilon-clip principle, use formula (2) to calculate the clipped advantage loss function: in, B is the number of batches for batch training, n is the number of unmanned ships, is the strategy of the unmanned ship numbered k, is the demonstration strategy of the unmanned boat numbered k that interacts with the environment, is the calculated advantage function of the i-th batch of training, the clip function implements epsilon-clip, and ε is the selected hyperparameter of epsilon-clip; 4.4.4) Calculate the entropy function to encourage iterative updates of the strategy: Among them, B is the number of batches for batch training, n is the number of unmanned ships, is the observation obtained by the unmanned ship numbered k in the i-th batch of training. The S function calculates the entropy change of the action obtained by the strategy of the unmanned ship when observing the environment; 4.4.5) Combine the functions to get θ k The loss function of the network is used to k To update: Among them, σ is the proportional coefficient of the combination of the two loss functions; 4.4.6) Calculate the real future value function using time series difference 4.4.7) Using the Future Value Function The network Φ is trained using the following loss function k To update: in, B is the number of batches for batch training, n is the number of unmanned ships, Is the network Φ k Estimated future value expectations, It is the real future value calculated by time series difference; 4.4.8) Use the shuffled records in the buffer one by one to calculate the θ in the model k Network and Φ k The network parameters are updated.

7. An electronic device, characterized in that: include: one or more processors; as well as a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are caused to execute the method according to any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images