A method and system for planning the assembly path of multiple unmanned boats

Abstract

The present invention relates to a method and system for planning the assembly path of multiple unmanned boats, the method comprising the following steps: first, constructing a correction matrix based on environmental point cloud information obtained by a laser radar to form a zero-space correction navigation vector field, and then obtaining obstacle avoidance instructions for each unmanned boat; in an emergency, introducing the COLREGs rule to dynamically adjust the core reaction coefficient in the correction matrix; in a non-emergency, using a multi-agent deep reinforcement learning algorithm and a centralized training distributed execution strategy, the core reaction coefficient of the correction matrix is ​​adaptively adjusted to guide the unmanned boats to avoid various obstacles and achieve simultaneous assembly. The present invention controls the simultaneous movement of multiple unmanned boats based on the corrected navigation vector, and realizes that multiple unmanned boats arrive at the assembly area at the same time in the shortest time under the constraints of obstacle avoidance / collision avoidance, and can avoid collisions between multiple boats in a complex marine environment, thereby improving the safety and standardization of the operation of the unmanned boat system.

Description

Technical Field

[0001] The present invention belongs to the field of marine energy technology, and in particular relates to a method and system for planning the assembly path of multiple unmanned boats. Background Art

[0002] In recent years, in order to further explore and develop marine resources, marine unmanned systems have been increasingly developing in the direction of intelligence. Among them, unmanned surface vessels (USVs) play a very important role in water quality sampling, mapping, marine equipment inspections and environmental protection due to their advantages of intelligence, modularity, high flexibility and small size.

[0003] Due to the limitations of a single unmanned boat in terms of navigation distance, sensor detection range and volume, the research on multi-unmanned boat systems has gradually received widespread attention in recent years. Compared with the path planning of a single unmanned boat, the collaborative path planning of multiple unmanned boats is more complex. It must not only meet the basic needs such as obstacle avoidance / collision avoidance, communication maintenance, etc., but also meet the task requirements of task allocation, formation maintenance, and simultaneous assembly.

[0004] There are some studies in the existing technology on the task of assembling multiple unmanned boats at the same time. However, most of the existing studies are only applicable to simple scenarios, lacking comprehensive consideration of factors such as unknown / irregular obstacles, COLREGs, and simultaneous assembly under wind / wave / current interference, and cannot adapt to complex marine environments. Summary of the invention

[0005] The purpose of the present invention is to solve one of the above-mentioned technical problems and to provide a method and system for planning the assembly path of multiple unmanned boats. Based on the international regulations for avoiding collisions at sea and multi-agent deep reinforcement learning, the navigation vectors of multiple unmanned boats are adjusted in different situations, so that multiple unmanned boats can start from their respective starting points and arrive at the assembly area at the same time in the shortest time while satisfying the constraints of obstacle avoidance / collision avoidance.

[0006] To achieve the above object, the technical solution adopted by the present invention is:

[0007] A method for planning a multi-unmanned boat assembly path includes the following steps:

[0008] Construct the initial navigation vectors of each unmanned boat in a free and barrier-free environment to form an initial navigation vector field;

[0009] In an environment with obstacles, use LiDAR to obtain obstacle point cloud information and build a correction matrix based on the obstacle point cloud information;

[0010] Correcting the initial navigation vector of each unmanned boat based on the correction matrix to obtain a corrected navigation vector, and calculating the obstacle avoidance path of each unmanned boat based on the corrected navigation vector;

[0011] Each unmanned boat moves based on an obstacle avoidance path, and during the movement, it detects the distance to other unmanned boats in real time to determine whether there are other unmanned boats within its predetermined safety distance;

[0012] If there are other unmanned boats within the predetermined safety distance, the core reaction coefficient of the correction matrix will be dynamically adjusted based on the International Regulations for Preventing Collisions at Sea, and then the obstacle avoidance path will be dynamically adjusted;

[0013] If there are no other unmanned boats within the predetermined safety distance, the multi-agent deep reinforcement learning algorithm is used to adaptively adjust the core reaction coefficient of the correction matrix, and then dynamically adjust its own obstacle avoidance path.

[0014] In some embodiments of the present invention, the following steps are further included:

[0015] After obtaining the obstacle point cloud information, preprocessing the obtained obstacle point cloud information is performed to obtain K pieces of preprocessed point cloud information;

[0016] After obtaining the preprocessed point cloud information, calculate the threat level of each feature point :

[0017] ;

[0018] in, For the The distance of feature point detection, For the The orientation of the feature point detection;

[0019] Sort the threat values ​​of each feature point from large to small to determine the priority of each feature point ;

[0020] Introduce the null space strategy to define the correction matrix of each feature point :

[0021] ;

[0022] in, is the rejection coefficient; is the tangential reaction coefficient;

[0023] is the radial quantity of the vertical section, The expression is:

[0024] ;

[0025] in, is the heading angle of the unmanned boat;

[0026] is the tangent vector corresponding to the vertical tangent plane, The expression is:

[0027] ;

[0028] in, is the directional response coefficient;

[0029] Correction matrix based on each feature point Determine the comprehensive correction matrix :

[0030] ;

[0031] in, is the weight coefficient, The expression is:

[0032] ;

[0033] Based on the comprehensive correction matrix Calculate the corrected navigation vector :

[0034] ;

[0035] in, is the initial navigation vector, The expression is:

[0036] ;

[0037] in, is the current position of the unmanned boat, , is the target position of the unmanned boat, is the Euclidean distance between the current position of the unmanned boat and the target position, The cruising speed of the unmanned boat.

[0038] In some embodiments of the present invention, preprocessing the acquired obstacle point cloud information specifically includes the following steps:

[0039] Use CropBox filter to remove invalid points, perform through filtering along the X-axis, Y-axis, and Z-axis respectively, enclose a box-shaped area based on a predetermined threshold, and remove all point cloud data in it;

[0040] The PointPillars algorithm is used to extract the features of point cloud data, convert the point cloud in each grid into a cylinder, and obtain data in the form of a pseudo-image.

[0041] In some embodiments of the present invention, dynamically adjusting the core reaction coefficient of the correction matrix based on the International Regulations for Preventing Collisions at Sea specifically includes the following steps:

[0042] The shortest encounter distance based on the regulations for preventing collisions at sea , shortest encounter time , and the speed ratio of the other unmanned boat to the main unmanned boat Defining collision threat :

[0043] ;

[0044] in, , , is the corresponding weight;

[0045] Distance Threat Time Threat They are defined as:

[0046] ;

[0047] ;

[0048] in, , , is the length of the unmanned boat, and The expressions are:

[0049] ;

[0050] ;

[0051] in, For the Unmanned boats relative to current unmanned boats The relative speed of

[0052] Define the weights of unmanned boats for different ship encounter types ;

[0053] The collision threat and unmanned boat weight The rejection coefficients uniformly set in the correction matrix and the tangential reaction coefficient middle:

[0054] ;

[0055] in, It is the upper limit constant of the obstacle avoidance reaction coefficient.

[0056] In some embodiments of the present invention, the specific steps of using a multi-agent deep reinforcement learning algorithm to adaptively adjust the core reaction coefficient of the correction matrix include:

[0057] The Markov process is used to model the decision-making process of each unmanned boat to adjust the correction matrix, obtain a multi-agent model, and define the corresponding state space, action space and reward function;

[0058] The Markov process consists of six tuples express;

[0059] in, Discount factor for calculating cumulative returns;

[0060] for The state information of the agents is integrated, including The local observation information of each agent, ; The agents correspond to unmanned boats;

[0061] for The state space of the fusion of agents, ,in, is the location information of each unmanned boat, , is the posture information of each unmanned boat, , is the speed information of each unmanned boat, ;

[0062] for The action space of the fusion of agents, It is expressed by the repulsive reaction coefficient, tangential reaction coefficient and directional reaction coefficient of each unmanned boat:

[0063] ;

[0064] is the transition probability;

[0065] for The reward function of an agent is ;

[0066] Reward Function Including mainline rewards , continuous rewards , Collision Reward , Stable Rewards , Progress Rewards and network rewards ; Reward function for an agent It is expressed as:

[0067] ;

[0068] Reward Function Middle The reward function of an agent is :

[0069] ;

[0070] in, For the The main line reward for each agent, For the The continuous reward of each agent, For the The collision reward of each agent, For the The stable reward of each agent, For the The progress reward for each agent, For the Network rewards for each agent; , , , , , are the weight coefficients of the corresponding reward items respectively;

[0071] No. Mainline reward for agents The expression is:

[0072] ;

[0073] in, To determine whether the main task is completed. is a constant;

[0074] No. Continuous rewards for agents The expression is:

[0075] ;

[0076] No. Collision reward for each agent The expression is:

[0077] ;

[0078] No. Stable rewards for each agent The expression is:

[0079] ;

[0080] in, is the preset threshold value of the heading angle change rate;

[0081] No. Progress reward for each agent The expression is:

[0082] ;

[0083] in, for The progress of task completion at all times; The expression is:

[0084] ;

[0085] Network Rewards The expression is:

[0086] ;

[0087] in, For the communication efficiency of the network, is the network communication consumption, and are the weight coefficients corresponding to communication efficiency and communication consumption, For the Unmanned boat The number of communication connections that exist.

[0088] In some embodiments of the present invention, the specific steps of adaptively adjusting the core reaction coefficient of the correction matrix using the multi-agent deep reinforcement learning algorithm further include:

[0089] Train multi-agent models;

[0090] In the training phase, each agent in the multi-agent model sets up its own Actor network, and uses the Clip function to calculate the network parameters of each Actor. , and make updates ;in, for The joint state of all agents at the moment, ; for The joint action of all agents at the moment, ;

[0091] Each agent uses the same central node's Critic network to evaluate the corresponding state-action pair. The central node's Critic network evaluates the corresponding state-action pair based on the joint state. , Joint Action and joint rewards Update network parameters , ;

[0092] In the execution phase, each agent in the multi-agent model independently makes decisions and selects actions.

[0093] In some embodiments of the present invention, the following steps are further included:

[0094] Build a simulation platform based on Gazebo, conduct multi-agent model training in the simulation platform, and verify the collaborative path planning effect of multiple unmanned boats.

[0095] In some embodiments of the present invention, the unmanned boat platform in the simulation platform adopts a catamaran structure and is equipped with an articulated suspension system and an inflatable buoy;

[0096] The simulation platform sets environmental factors to generate interference force on the unmanned boat and uses the anti-disturbance controller as the underlying controller of the unmanned boat.

[0097] In some embodiments of the present invention, the simulation platform includes:

[0098] The marine environment module is used to provide the unmanned boat with interference forces generated by marine environmental factors;

[0099] The environment perception module is used to control each unmanned boat to use laser radar to perceive environmental information and extract point cloud information, and at the same time control each unmanned boat to share perceived data and location information through the communication network;

[0100] The path planning module is used to plan the navigation vector of each unmanned boat based on the data sensed by each unmanned boat and calculate the obstacle avoidance path of each unmanned boat.

[0101] Some embodiments of the present invention further provide a multi-unmanned boat assembly path planning system, including:

[0102] at least one processor;

[0103] at least one memory for storing at least one program;

[0104] When at least one program is executed by at least one processor, the at least one processor implements the above-mentioned multi-unmanned boat assembly path planning method.

[0105] The beneficial effects of the present invention are:

[0106] 1. The present invention introduces the zero space strategy into the path planning process of multiple unmanned boats, corrects the navigation vector of each unmanned boat through the correction matrix, and then forms a corrected zero space corrected navigation vector field (NSGVF). Based on the corrected navigation vector, multiple unmanned boats are controlled to move simultaneously, and multiple unmanned boats are achieved to arrive at the assembly area at the same time in the shortest time under the constraints of obstacle avoidance / collision avoidance;

[0107] 2. In non-emergency situations, the present invention adjusts the core reaction coefficient of the correction matrix through the COLREGs rule to regulate the collision avoidance behavior between unmanned boats, which can avoid collisions between multiple boats in a complex marine environment, thereby improving the safety and standardization of the operation of the unmanned boat system;

[0108] 3. The present invention uses a multi-agent deep reinforcement learning (MADRL) algorithm and combines it with a centralized training distributed execution strategy to adaptively adjust the core reaction coefficient of the correction matrix, thereby improving the flexibility of the unmanned boat system in coping with complex environments.

[0109] 4. The virtual simulation platform constructed by the present invention has a high degree of realism and can improve the credibility of the training results of the unmanned boat simulation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0110] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the specific embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0111] Figure 1 It is a flow chart of a method for planning the assembly path of multiple unmanned boats;

[0112] Figure 2 It is a schematic diagram of the correction effect on point cloud data;

[0113] Figure 3 This is a schematic diagram of the four types of ship encounters in the COLREGs rules;

[0114] Among them, (a) is a schematic diagram of a head-on encounter; (b) is a schematic diagram of a starboard crossing; (c) is a schematic diagram of a port crossing; (d) is a schematic diagram of an overtaking;

[0115] Figure 4 This is a schematic diagram of the network structure of the MAPPO algorithm;

[0116] Figure 5 It is a flow chart of collaborative path planning for a multi-unmanned boat system;

[0117] Figure 6 It is the architecture diagram of the simulation platform;

[0118] Figure 7This is the unmanned boat path result diagram of the algorithm combining MAPPO and NSGVF;

[0119] Figure 8 This is the unmanned boat speed result diagram of the algorithm combining MAPPO and NSGVF;

[0120] Fig. 9 This is the result diagram of the remaining time of the unmanned boat using the algorithm combining MAPPO and NSGVF;

[0121] Fig.10 This is the unmanned boat path result diagram of the pure NSGVF algorithm;

[0122] Fig.11 This is the unmanned boat speed result diagram of the pure NSGVF algorithm;

[0123] Fig.12 This is the remaining time result diagram of the unmanned boat using the pure NSGVF algorithm. DETAILED DESCRIPTION

[0124] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0125] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0126] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0127] The technical solution of the present invention is described in detail below in conjunction with specific embodiments and the accompanying drawings.

[0128] As attached Figure 1 -Attached Figure 8 As shown, in an exemplary embodiment of a method for assembling a path for multiple unmanned boats of the present invention, the path planning method includes the following steps:

[0129] S1: Construct NSGVF algorithm:

[0130] Firstly, the initial navigation vectors of each unmanned boat in a free and obstacle-free environment are constructed to form an initial navigation vector field.

[0131] In an environment where obstacles exist, lidar is used to obtain obstacle point cloud information, and a correction matrix is ​​constructed based on the obstacle point cloud information to quantitatively describe the impact of obstacles on each unmanned boat.

[0132] The initial navigation vector of each unmanned boat is corrected based on the correction matrix to obtain a corrected navigation vector. The corrected navigation vectors of each unmanned boat form a zero space corrected navigation vector field (NSGVF).

[0133] The corrected navigation vector is integrated to obtain the obstacle avoidance path of each unmanned boat.

[0134] In some embodiments of the present invention, step S1 specifically includes the following steps:

[0135] S11: First, define the initial navigation vector field in a free obstacle-free environment, where the initial navigation vector Always point to the target point, initial navigation vector The expression is:

[0136] ;

[0137] in, Current position of the unmanned boat With target location The Euclidean distance between The cruising speed of the unmanned boat.

[0138] S12: In an environment with obstacles, the laser radar installed on the unmanned boat is used to obtain obstacle point cloud information.

[0139] S13: Preprocess the acquired obstacle point cloud information to obtain K pieces of preprocessed point cloud information.

[0140] Since the point cloud data is unevenly distributed, after obtaining the obstacle point cloud information in step S12, the obtained obstacle point cloud information needs to be preprocessed to remove invalid points therein and reduce the amount of subsequent calculations.

[0141] In some embodiments of the present invention, the method for preprocessing the acquired obstacle point cloud information specifically includes the following steps.

[0142] First, the CropBox filter is used to remove invalid points, and through filtering is performed along the X-axis, Y-axis, and Z-axis directions respectively. That is, a box-shaped area is enclosed based on the predetermined X-axis, Y-axis, and Z-axis thresholds, and all point cloud data therein are eliminated, thereby reducing the amount of calculation in advance.

[0143] The PointPillars algorithm is used to extract the features of point cloud data. The algorithm converts the point cloud in each grid into a cylinder, and finally obtains data in the form of a pseudo-image so as to be added to the construction of the subsequent correction matrix.

[0144] S14: Obtain the pre-processed point cloud information and calculate the threat level of each feature point :

[0145] .

[0146] in, For the The distance of feature point detection, For the The direction of the feature point detection is the angle between the straight line connecting the feature point position and the unmanned boat position and the unmanned boat heading.

[0147] Sort the threat values ​​of each feature point from large to small to determine the priority of each feature point .

[0148] S15: Introducing the null space strategy to define the correction matrix of each feature point , where the correction matrix with high priority is retained in full, and the correction matrix with low priority is only retained in its tangential effect, and the correction matrix The specific definition of is:

[0149] .

[0150] in, is the rejection coefficient; is the tangential reaction coefficient. The repulsion reaction coefficient and the tangential reaction coefficient jointly affect the obstacle avoidance timing and range of the unmanned boat.

[0151] is the radial quantity of the vertical section, The expression is:

[0152] .

[0153] in, is the heading angle of the unmanned boat, that is, the angle between the heading of the unmanned boat and the x-axis direction of the inertial coordinate system.

[0154] is the tangent vector corresponding to the vertical tangent plane, The expression is:

[0155] .

[0156] is the directional reaction coefficient. When the directional reaction coefficient is 1, the unmanned boat turns right. When the directional reaction coefficient is -1, the unmanned boat turns left.

[0157] vector , The expression is as follows Figure 2 shown.

[0158] Correction matrix based on each feature point Determine the comprehensive correction matrix :

[0159] .

[0160] in, is the weight coefficient. The closer the distance, the greater the weight coefficient. The expression is:

[0161] .

[0162] S16: Based on the comprehensive correction matrix Calculate the corrected navigation vector , the corrected navigation vector That is, the expected velocity vector of the unmanned boat and the collision avoidance instruction of the unmanned boat, which is expressed as:

[0163] .

[0164] in, is the initial navigation vector, The expression is:

[0165] .

[0166] in, is the current position of the unmanned boat, , is the target position of the unmanned boat, is the Euclidean distance between the current position of the unmanned boat and the target position, The cruising speed of the unmanned boat.

[0167] S17: Calculate the obstacle avoidance path of each unmanned boat based on the corrected navigation vector of each unmanned boat. Perform an integration based on the navigation vector of the current position of the unmanned boat to obtain the position of the unmanned boat at the next moment, and repeat the iteration on this basis to obtain the complete obstacle avoidance path of each unmanned boat.

[0168] S2: Each unmanned boat moves based on the obstacle avoidance path, detects the distance to other unmanned boats in real time during the movement, and determines whether there are other unmanned boats within its predetermined safety distance.

[0169] The length of the unmanned boat is ,by As the lethal distance, that is, when the distance between the boats is less than When a ship is involved in a collision, it is inevitable due to the disturbance of waves and the inertia of ships.

[0170] In this embodiment, the predetermined safety distance is set to , that is, when the distance between the boats is arrive When the distance between the boats is greater than It is a non-emergency situation.

[0171] S3: If there are other unmanned boats within the predetermined safety distance, it is determined that the current situation is an emergency, and the core reaction coefficient of the correction matrix is ​​dynamically adjusted based on the International Regulations for Preventing Collisions at Sea, thereby dynamically adjusting the obstacle avoidance path of the unmanned boat.

[0172] That is, in an emergency, the unmanned boats regard each other as virtual dynamic obstacles, and firstly, the collision avoidance instructions of each boat are calculated by the method of step S1.

[0173] Since the selection of the repulsive reaction coefficient, tangential reaction coefficient and directional reaction coefficient in the correction matrix usually depends on human settings, if the parameters are not selected properly, the maritime collision avoidance rules (COLREGs) may be violated, or even collision avoidance may fail. In an emergency, the COLREGs rules need to be introduced to dynamically adjust the core reaction coefficients in the correction matrix.

[0174] It should be noted that the International Regulations for Preventing Collisions at Sea are designed to ensure safe and orderly navigation while reducing the risk of collision. The COLREGs rules divide ship encounters into four types: port crossing, starboard crossing, head-on encounter and overtaking. Figure 3 shown.

[0175] Whether two ships collide when they meet is closely related to the distance between the ships and the relative speed between the ships. The International Regulations for Preventing Collisions at Sea generally define them as the shortest distance to meet (DCPA) and the shortest time to meet (TCPA). The calculation formulas are:

[0176] ;

[0177] ;

[0178] in, Indicates The coordinates of the unmanned boat, Indicates Unmanned boats relative to current unmanned boats The relative speed, angle Indicates Unmanned boats and current unmanned boats The connection with The angle between.

[0179] In some embodiments of the present invention, in order to improve the collision avoidance path quality of unmanned boats and the calculation efficiency of algorithms and realize collision avoidance between multiple unmanned boats, a method for dynamically adjusting the core reaction coefficient of the correction matrix based on the International Regulations for Preventing Collisions at Sea specifically includes the following steps:

[0180] The shortest encounter distance based on the regulations for preventing collisions at sea , shortest encounter time , and the speed ratio of the other unmanned boat to the main unmanned boat Defining collision threat :

[0181] .

[0182] in, , , are the weights of distance, time and speed respectively.

[0183] is the distance threat degree, is the time threat level. Time Threat They are defined as:

[0184] ;

[0185] .

[0186] in, and They are the safe meeting distance for ships and the safe passing distance for ships. , , is the length of the unmanned boat, and They are the safe meeting time of ships and the safe passing time of ships. and The expressions are:

[0187] ;

[0188] .

[0189] in, For the Unmanned boats relative to current unmanned boats relative speed.

[0190] Based on the COLREGs rule, different weights are assigned to unmanned boats of different ship encounter types. , the boat authority is the passage authority of different types of boats, and different types of ship encounters include port crossing, starboard crossing, encountering and overtaking. In this embodiment, the weights of unmanned boats of different types of encounters are shown in Table 1.

[0191] Table 1 Weights of unmanned boats of different encounter types

[0192]

[0193] In order to simplify the coupling of collision avoidance amplitude and collision avoidance timing, the collision threat degree and unmanned boat weight The rejection coefficients uniformly set in the correction matrix and the tangential reaction coefficient middle:

[0194] .

[0195] in, It is the upper limit constant of the obstacle avoidance reaction coefficient, and its value depends on the size of the unmanned boat.

[0196] If there are no other unmanned boats within the predetermined safety distance, it is determined that the current situation is not an emergency. The multi-agent deep reinforcement learning (MADRL) algorithm is used in combination with the centralized training distributed execution strategy to adaptively adjust the core reaction coefficient of the correction matrix, thereby dynamically adjusting the obstacle avoidance path of the unmanned boat.

[0197] MADRL can learn different adaptive strategies through more diverse environments, and further optimize strategies through continuous interaction with the environment to cope with changes and uncertainties. Among them, Multi-Agent Proximal Policy Optimization (MAPPO) is a MADRL algorithm based on policy optimization, which is suitable for continuous action space and relies on importance sampling to avoid drastic fluctuations during policy updates, improving the stability of the algorithm and making higher use of experience samples.

[0198] In some embodiments of the present invention, the specific steps of using a multi-agent deep reinforcement learning algorithm to adaptively adjust the core reaction coefficient of the correction matrix include:

[0199] In an unknown environment, each unmanned boat is limited by the detection distance and obstacles and can only make decisions based on partial information, which can be modeled as a partially observable Markov decision process (POMDP).

[0200] Therefore, the Markov process is first used to model the decision-making process of each unmanned boat adjusting the correction matrix, obtain the multi-agent model, and define the corresponding state space, action space and reward function.

[0201] The Markov process consists of six tuples express.

[0202] in, Discount factor for calculating cumulative returns.

[0203] for The state information of the agents is integrated, including The local observation information of each agent, ; The agents correspond to Unmanned boats, That is, the preprocessed point cloud information reflects the environmental information around the intelligent agent.

[0204] for The state space of the fusion of agents, ,in, is the location information of each unmanned boat, , is the posture information of each unmanned boat, , is the speed information of each unmanned boat, .

[0205] for The action space of the fusion of agents, It is expressed by the repulsive reaction coefficient, tangential reaction coefficient and directional reaction coefficient of each unmanned boat:

[0206] .

[0207] It should be noted that by adjusting the repulsive reaction coefficient and the tangential reaction coefficient, the correction matrix can be adjusted to correct the speed of each unmanned boat. In addition, in the process of adjusting the correction matrix, the impact of obstacles on the unmanned boat should be comprehensively considered, where the speed correction is not only reflected in the obstacle avoidance direction, but also includes constraints in time and space dimensions. In addition, the directional reaction coefficient determines the collision avoidance direction of the unmanned boat (turn left or right). Therefore, in this embodiment, the repulsive reaction coefficient, tangential reaction coefficient and directional reaction coefficient in the correction matrix are used as actions.

[0208] is the transition probability, for example Indicates from the state Take Action Arrival Status probability.

[0209] is the agent’s reward function, In reinforcement learning, the core role of the reward function is to indicate the goal of the agent, and the reward function needs to be carefully designed to maximize the cumulative return of the agent.

[0210] Reward function for an agent It can be summarized as:

[0211] .

[0212] This includes mainline rewards , continuous rewards , Collision Reward , Stable Rewards , Progress Rewards and network rewards .

[0213] Therefore, for The reward function for The reward function of an agent is :

[0214] .

[0215] in, For the The main line reward for each agent, For the The continuous reward of each agent, For the The collision reward of each agent, For the The stable reward of each agent, For the The progress reward for each agent, For the Network rewards for each agent; , , , , , are the weight coefficients of the corresponding reward items respectively.

[0216] Mainline Rewards It is mainly used to guide the agent to complete the main task. When the unmanned boat reaches the designated target point, the agent will receive the corresponding reward. Mainline reward for agents The expression is:

[0217] .

[0218] in, To determine whether the main task is completed. is a small constant.

[0219] Continuous Rewards A small negative reward will be provided at each step to encourage the agent to complete the task as quickly as possible and avoid unintentional score-boosting actions. Continuous rewards for agents The expression is:

[0220] ;

[0221] in, A small positive number.

[0222] Collision Rewards The distance between the unmanned boat and the nearest obstacle will be Give a certain negative reward. The closer the distance, the greater the penalty. When the distance is not greater than the safety threshold Give a larger collision penalty , it is deemed that the collision cannot be avoided by normal maneuvers, Collision reward for each agent The expression is:

[0223] .

[0224] in, is a small positive number, The shortest distance between the unmanned boat and the obstacle. is a large positive number, is the safety distance threshold.

[0225] Stable Rewards Used to evaluate the stability of unmanned boats, based on the rate of change of the heading angle As the evaluation criteria. When the heading angle change rate is larger and exceeds the corresponding set threshold , the greater the penalty, the Stable rewards for each agent The expression is:

[0226] .

[0227] in, It is the preset threshold value of heading angle change rate.

[0228] Progress Rewards It is designed based on the completion of the assembly task. Progress reward for each agent The expression is:

[0229] .

[0230] in, for The progress of task completion at all times; The expression is:

[0231] .

[0232] in, for The distance between the unmanned boat and the assembly area at all times.

[0233] Network Rewards It is used to evaluate the network communication performance of the system, network rewards The expression is:

[0234] .

[0235] in, For the communication efficiency of the network, is the network communication consumption, and are the weight coefficients corresponding to communication efficiency and communication consumption, For the Unmanned boat The number of communication connections that exist.

[0236] In some embodiments of the present invention, the specific steps of adaptively adjusting the core reaction coefficient of the correction matrix using the multi-agent deep reinforcement learning algorithm further include:

[0237] The multi-agent model is trained using the MAPPO algorithm based on the CTDE framework.

[0238] In the training phase, each agent in the multi-agent model sets up its own Actor network, and uses the Clip function to calculate the network parameters of each Actor. , and make updates ;in, for The joint state of all agents at the moment, ; for The joint action of all agents at the moment, ;

[0239] Each agent uses the same central node's Critic network to evaluate the corresponding state-action pair. The central node's Critic network evaluates the corresponding state-action pair based on the joint state. , Joint Action and joint rewards Update network parameters , ;

[0240] In the execution phase of the multi-agent model, each agent makes independent decisions and selects actions without sharing information.

[0241] In the above exemplary embodiment, the path planning process of assembling multiple unmanned boats simultaneously based on the hierarchical architecture is as follows: Figure 5 shown.

[0242] In some embodiments of the present invention, in order to train the network model and test the collaborative path planning effect of multiple unmanned boats, the following steps are further included:

[0243] S4: Build a high-fidelity virtual simulation platform based on Gazebo, conduct multi-agent model training in the simulation platform, and verify the collaborative path planning effect of multiple unmanned boats.

[0244] In some embodiments of the present invention, the unmanned boat platform in the simulation platform adopts a catamaran structure and is installed with an articulated suspension system and an inflatable buoy.

[0245] The simulation platform sets the marine environmental factors to generate interference forces / torques on the unmanned boat, including wind (considering both steady wind and variable gusts), waves (considering the Pierson-Moscowitz spectrum and using the linear superposition of Gerstner waves), flow (sinusoidal distribution) and other factors.

[0246] The simulation platform uses an anti-disturbance controller as the underlying controller of the unmanned boat, and uses islands, reefs, ships, etc. as obstacles.

[0247] In some embodiments of the present invention, the simulation platform includes:

[0248] The marine environment module is used to provide the unmanned boat with interference forces generated by marine environmental factors.

[0249] The environmental perception module is used to control each unmanned boat to use lidar to perceive environmental information and extract point cloud information, and at the same time control each unmanned boat to share perceived data and location information through the communication network.

[0250] The path planning module is used to plan the navigation vector of each unmanned boat based on the data sensed by each unmanned boat and calculate the obstacle avoidance path of each unmanned boat.

[0251] The workflow of each module in the simulation platform is as follows:

[0252] First, in the environmental perception module, each unmanned boat uses a 3D lidar to perceive the surrounding environment and extract point cloud data. At the same time, each unmanned boat shares perception data and location information through a communication network to ensure the consistency of the system.

[0253] Then, the path planning module plans the navigation commands and paths of each unmanned boat based on the above perception information: the module first evaluates the distance between the unmanned boats. If they are within the threat range (i.e., in an emergency), the system will enter the collision avoidance phase, which uses the algorithm combined with COLREGS and NSGVF to provide collision avoidance instructions to maintain a safe distance between the unmanned boats.

[0254] In other cases, the multi-unmanned boat system enters the normal navigation phase (i.e., non-emergency situations), and the algorithm combining MADRL and NSGVF enables the unmanned boats to avoid obstacles and reach the assembly area at the same time.

[0255] During this period, the underlying controller of each unmanned boat will track the above navigation commands and then calculate the propeller thrust. At the same time, the wind, wave and current simulation provided by the ocean environment module can enhance the realism of the simulation process.

[0256] In order to further illustrate the beneficial effects of the present invention, the algorithm combining MAPPO and NSGVF and the pure NSGVF algorithm were compared. The results are shown in the attached figure. Figure 7 -Attached Fig.12 shown.

[0257] In terms of path planning, the path generated by the algorithm combining MAPPO and NSGVF is smoother and more orderly, especially in the process of obstacle avoidance. The three unmanned boats can maintain a stable formation and avoid unnecessary path deviations, ensuring the safety and efficiency of the multi-unmanned boat system in complex environments, while the path stability of the unmanned boats in the pure NSGVF algorithm is poor. From the speed curve, the speed control of the algorithm combining MAPPO and NSGVF is more stable, and the speed fluctuations between the three unmanned boats are extremely small, effectively avoiding excessive speed adjustments, while the speed changes in the pure NSGVF algorithm are significantly larger. From the analysis of the completion of the assembly task, the remaining distance of the three unmanned boats in the algorithm combining MAPPO and NSGVF dropped to 0 at the same time at 90s, indicating that they arrived at the assembly area at the same time, while the pure NSGVF algorithm could not achieve this task.

[0258] Some embodiments of the present invention further provide a multi-unmanned boat assembly path planning system, including:

[0259] at least one processor;

[0260] at least one memory for storing at least one program;

[0261] When at least one program is executed by at least one processor, the at least one processor implements the above-mentioned multi-unmanned boat assembly path planning method.

[0262] Finally, it should be noted that: the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0263] The above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or some technical features can be replaced by equivalents without departing from the spirit of the technical solution of the present invention, which should be included in the scope of the technical solution for protection of the present invention.

Claims

1. A method for planning a multi-unmanned boat assembly path, characterized in that: The following steps are involved: Construct the initial navigation vectors of each unmanned boat in a free and barrier-free environment to form an initial navigation vector field; In an environment where obstacles exist, using a laser radar to obtain obstacle point cloud information, and constructing a correction matrix based on the obstacle point cloud information; The specific method of constructing the correction matrix includes: After obtaining the obstacle point cloud information, preprocessing the obtained obstacle point cloud information is performed to obtain K pieces of preprocessed point cloud information; After obtaining the preprocessed point cloud information, calculate the threat level of each feature point : ； in, For the The distance of feature point detection, For the The position of the feature point detection; Sort the threat values ​​of each feature point from large to small to determine the priority of each feature point ; Introduce the null space strategy to define the correction matrix of each feature point : ； in, is the identity matrix, is the feature point number, is the rejection coefficient; is the tangential reaction coefficient; is the radial quantity of the vertical section, The expression is: ； in, is the heading angle of the unmanned boat; is the tangent vector corresponding to the vertical tangent plane, The expression is: ； in, is the directional response coefficient; Correcting the initial navigation vector of each unmanned boat based on the correction matrix to obtain a corrected navigation vector, and calculating the obstacle avoidance path of each unmanned boat based on the corrected navigation vector; Among them, the specific method of correcting the initial navigation vector of each unmanned boat based on the correction matrix includes: Correction matrix based on each feature point Determine the comprehensive correction matrix : ； in, is the weight coefficient, The expression is: ； Based on the comprehensive correction matrix Calculate the corrected navigation vector : ； in, is the initial navigation vector, The expression is: ； in, is the current position of the unmanned boat, , is the target position of the unmanned boat, is the Euclidean distance between the current position of the unmanned boat and the target position, is the cruising speed of the unmanned boat; Each unmanned boat moves based on the obstacle avoidance path, detects the distance to other unmanned boats in real time during the movement, and determines whether there are other unmanned boats within its predetermined safety distance; If there are other unmanned boats within the predetermined safety distance, the core reaction coefficient of the correction matrix is ​​dynamically adjusted based on the international maritime collision avoidance regulations, thereby dynamically adjusting its own obstacle avoidance path; If there are no other unmanned boats within the predetermined safety distance, a multi-agent deep reinforcement learning algorithm is used to adaptively adjust the core reaction coefficient of the correction matrix, thereby dynamically adjusting its own obstacle avoidance path.

2. The method for planning the assembly path of multiple unmanned boats according to claim 1, characterized in that: The preprocessing of the acquired obstacle point cloud information specifically includes the following steps: Use CropBox filter to remove invalid points, perform through filtering along the X-axis, Y-axis, and Z-axis respectively, enclose a box-shaped area based on a predetermined threshold, and remove all point cloud data in it; The PointPillars algorithm is used to extract the features of point cloud data, convert the point cloud in each grid into a cylinder, and obtain data in the form of a pseudo-image.

3. The method for planning the assembly path of multiple unmanned boats according to claim 1, characterized in that: Dynamically adjusting the core reaction coefficient of the correction matrix based on the International Regulations for Preventing Collisions at Sea specifically includes the following steps: The shortest encounter distance based on the regulations for preventing collisions at sea , Shortest encounter time , and the speed ratio between the other unmanned boat and the main unmanned boat , define the collision threat : ； in, , , are the weights of distance, time and speed respectively; is the distance threat degree, The degree of threat is time; Distance Threat Time Threat They are defined as: ； ； in, and They are the safe meeting distance for ships and the safe passing distance for ships. , , is the length of the unmanned boat, and They are the safe meeting time of ships and the safe passing time of ships. and The expressions are: ； ； in, For the Unmanned boats relative to current unmanned boats The relative speed of Define the weights of unmanned boats for different ship encounter types ; The collision threat and the unmanned boat weight The rejection coefficients uniformly set in the correction matrix and the tangential reaction coefficient middle: ； in, It is the upper limit constant of the obstacle avoidance reaction coefficient.

4. The method for planning the assembly path of multiple unmanned boats according to any one of claims 1 to 3, characterized in that: The specific steps of adaptively adjusting the core reaction coefficient of the correction matrix using a multi-agent deep reinforcement learning algorithm include: The Markov process is used to model the decision-making process of each unmanned boat to adjust the correction matrix, obtain a multi-agent model, and define the corresponding state space, action space and reward function; The Markov process consists of six tuples express; in, Discount factor for calculating cumulative returns; for The state information of the agents is integrated, including The local observation information of each agent, ; The agents correspond to unmanned boats; for The state space of the fusion of agents, ,in, is the location information of each unmanned boat, , is the posture information of each unmanned boat, , is the speed information of each unmanned boat, ; for The action space of the fusion of agents, It is expressed by the repulsive reaction coefficient, tangential reaction coefficient and directional reaction coefficient of each unmanned boat: ； is the transition probability; for The reward function of an agent is ; The reward function Including mainline rewards , continuous rewards , Collision Reward , Stable Rewards , Progress Rewards and network rewards ; Reward function for an agent It is expressed as: ； The reward function Middle The reward function of an agent is : ； in, For the The main line reward for each agent, For the The continuous reward of each agent, For the The collision reward of each agent, For the The stable reward of each agent, For the The progress reward for each agent, For the Network rewards for each agent; , , , , , are the weight coefficients of the corresponding reward items; No. Mainline reward for agents The expression is: ； in, To determine whether the main task is completed. is a constant; No. Continuous rewards for agents The expression is: ； in, is a positive number; No. Collision reward for each agent The expression is: ； in, , are all positive numbers, The shortest distance between the unmanned boat and the obstacle. is the safety distance threshold; No. Stable rewards for each agent The expression is: ； in, is the preset threshold value of the heading angle change rate; No. Progress reward for each agent The expression is: ； in, for The progress of task completion at all times; The expression is: ； The network rewards The expression is: ； in, For the communication efficiency of the network, is the network communication consumption, and are the weight coefficients corresponding to communication efficiency and communication consumption, For the Unmanned boat The number of communication connections that exist.

5. The method for planning the assembly path of multiple unmanned boats according to claim 4, characterized in that: The specific steps of adaptively adjusting the core reaction coefficient of the correction matrix using a multi-agent deep reinforcement learning algorithm further include: Training the multi-agent model; In the training phase, each agent in the multi-agent model sets up its own Actor network, and uses the Clip function to calculate the network parameters of each Actor. , and make updates ;in, for The joint state of all agents at the moment, ; for The joint action of all agents at the moment, ; Each agent uses the same central node's Critic network to evaluate the corresponding state-action pair. The central node's Critic network evaluates the corresponding state-action pair based on the joint state. , Joint Action and joint rewards Update network parameters , ; During the execution phase, each agent in the multi-agent model independently makes decisions and selects actions.

6. The method for planning the assembly path of multiple unmanned boats according to claim 5, characterized in that: Further comprising the steps of: A simulation platform based on Gazebo was built to train the multi-agent model and verify the collaborative path planning effect of multiple unmanned boats.

7. The method for planning the assembly path of multiple unmanned boats according to claim 6, characterized in that: The unmanned boat platform in the simulation platform adopts a catamaran structure and is equipped with an articulated suspension system and an inflatable buoy; The simulation platform sets environmental factors to generate interference force on the unmanned boat and uses an anti-disturbance controller as the underlying controller of the unmanned boat.

8. The method for planning the assembly path of multiple unmanned boats according to claim 6 or 7, characterized in that: The simulation platform comprises: The marine environment module is used to provide the unmanned boat with interference forces generated by marine environmental factors; The environment perception module is used to control each unmanned boat to use laser radar to perceive environmental information and extract point cloud information, and at the same time control each unmanned boat to share perceived data and location information through the communication network; The path planning module is used to plan the navigation vector of each unmanned boat based on the data sensed by each unmanned boat and calculate the obstacle avoidance path of each unmanned boat.

9. A multi-unmanned boat assembly path planning system, characterized in that it includes: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the multi-unmanned boat assembly path planning method as described in any one of claims 1-8.

Images