POS-DQN-based maritime search and rescue path planning method for multiple unmanned surface ships

Through the deep reinforcement learning algorithm based on POS-DQN, the unmanned boat maritime search and rescue path planning is optimized, and the problems of low efficiency in traditional search and rescue path planning and incomplete repeated coverage of task areas are solved, achieving efficient and safe maritime search and rescue tasks.

CN118915730APending Publication Date: 2024-11-08DALIAN MARITIME UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410952628.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-16
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Traditional maritime search and rescue path planning is inefficient, and high-probability areas cannot be searched first, and the task area has duplication and cannot cover the entire search area, ignoring the priority of the task.

Method used

Using a deep reinforcement learning algorithm based on POS-DQN, the task allocation and path planning of unmanned boats are optimized by calculating the POC distribution map and probability weight reward function in the search area, ensuring that the task area of ​​each unmanned boat has priority and No repetitions, and priority is given to searching for high-probability areas in maritime search and rescue environments.

Benefits of technology

The rational allocation and path planning of unmanned boats in maritime search and rescue mission areas have been realized, the search and rescue efficiency has been improved, and the search and rescue missions have been ensured safe and efficient completion.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118915730A_ABST
    Figure CN118915730A_ABST
Patent Text Reader

Abstract

The invention discloses a multi-unmanned ship maritime search and rescue path planning method based on POS-DQN, and relates to the technical field of path planning. Aiming at the problem that multiple unmanned ships reasonably and efficiently complete maritime search and rescue tasks, the maritime search and rescue path planning method which is reasonable in task allocation and capable of preferentially searching a high-probability area is provided on the basis of an improved task allocation algorithm and a POS-DQN deep reinforcement learning algorithm. According to the method, firstly, an improved task allocation algorithm is used for integrally dividing a search area, so that the task area of each unmanned ship has priority and is not divided repeatedly; secondly, the POS of a search environment is considered, and a POS-DQN algorithm based on deep reinforcement learning is provided. The algorithm can adapt to a complex change environment of search and rescue, and a probability weight reward function is utilized to train multiple unmanned ships to obtain an optimal search path. And finally, on the basis of considering the comprehensive coverage of obstacle avoidance and collision avoidance, the search path can preferentially search a high-probability region, so that the search and rescue efficiency is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of path planning, and in particular to a method for planning paths for multi-unmanned boat maritime search and rescue based on POS-DQN. Background Art

[0002] As a core component of global logistics, international ocean cargo transportation has a huge impact on world trade. Due to the complex and changeable marine environment, maritime accidents caused by natural and human factors often occur during shipping. Efficient maritime search and rescue (SAR) is crucial to responding to maritime emergencies. In traditional search and rescue, the use of fixed search path planning is inefficient and cannot prioritize high-probability areas, which has great limitations. According to the priority-based task allocation method, high-probability areas are searched first, but the generated task areas are repeated and cannot cover the entire search area. According to the overall regional division task allocation method, there is no duplication in the task area and it covers the entire area, but the priority of the task is ignored. Summary of the invention

[0003] In view of the above problems, the purpose of the present invention is to provide a USVs maritime search and rescue path planning method based on POS (Probability of success, probability of successfully finding the search and rescue target in the search area)-DQN (deep Q-network) for the problems that the task area in the search and rescue situation ignores the priority of the task, is repeated and cannot cover the entire search area, the path planning is unreasonable, and the search and rescue efficiency is low. The present invention takes the search and rescue of multiple unmanned boats (USVs) as the research object, and proposes an improved task allocation algorithm and a deep reinforcement learning algorithm based on POS-DQN for task allocation and path planning, so as to achieve that while avoiding obstacles and collisions, the task area assigned to each USV meets the priority of the task, and is reasonably divided without duplication, and can give priority to searching high-probability areas in the search area, and complete the search and rescue mission safely and efficiently.

[0004] To this end, the present invention adopts the following technical solutions:

[0005] The present invention discloses a multi-unmanned boat maritime search and rescue path planning method based on POS-DQN, comprising:

[0006] Calculate the POC of the search area and obtain a POC distribution map;

[0007] The search area is divided as a whole according to the search initial point when the unmanned boat arrives at the search area, and the search area of ​​each unmanned boat is obtained, so that the mission area of ​​each unmanned boat has priority and no duplication.

[0008] The POC distribution map is combined with the grid map of the search area to generate a marine search and rescue environment model;

[0009] According to the POC distribution map, calculate POS, which is the probability of successfully finding the search and rescue target in the search area;

[0010] The POS-DQN-based reinforcement learning method is used to train unmanned boats, and the optimal search and rescue path for each unmanned boat is obtained in the maritime search and rescue environment model, so as to avoid obstacles and prioritize the search of high-probability areas.

[0011] Furthermore, the POC of the search area is calculated to obtain a POC distribution map, including:

[0012] The search area is subdivided into square grids of the same size, the POC of each grid unit is calculated, and the POC distribution map of the search area is obtained;

[0013] Among them, the calculation formula of POC is:

[0014]

[0015] Where m i is the number of particles in grid i, and M is the number of particles contained in the overall distribution area.

[0016] Further, determining the search initial point where the unmanned boat reaches the search area includes:

[0017] Calculate the time it takes to reach the search area at full speed based on the initial distance of the ath unmanned boat from the search area and its maximum speed

[0018] It is known that the total area of ​​the search area is S, and the search time covered by the a-th unmanned boat in the search area is Therefore, the total search time for the ath unmanned boat search mission is:

[0019] The best search and rescue multi-UAV collaboration should minimize the search and rescue time:

[0020] according to Get the area constraint set of the search and rescue unit {S1,S2,…,S a …S n};

[0021] Where D a is the initial distance between the a-th unmanned boat and the search area; n is the total number of unmanned boats; a is the a-th USV; is the maximum speed of the ath unmanned boat; T is the time taken for the entire mission, that is, the time taken for multiple unmanned boats to participate in the search and rescue mission; The time when the ath unmanned boat arrives at the search area at full speed; is the search time covered by the ath unmanned boat in the search area; T a is the total search time for the ath unmanned boat search mission; b a is the search capability of the ath unmanned boat; S a The search area allocated to the ath unmanned boat; S n The search area allocated to the nth unmanned boat;

[0022] Connect the initial position of the unmanned boat with the center point of the grid with the highest POC in the search area to obtain the ideal route for the unmanned boat to reach the search and rescue area;

[0023] If the unmanned boat is outside the search area, the intersection of the route and the boundary of the search area is the initial point of the search; if the unmanned boat is within the search area, the intersection of the extended line of the route and the boundary of the search area is the initial point of the search.

[0024] Furthermore, the search area is divided as a whole to obtain the search area of ​​each unmanned boat, including:

[0025] Sort the search initial points in counterclockwise order to obtain the first point set N = {N1, N2, N3, ..., N n};

[0026] Sort the search initial point and the search area vertices in counterclockwise order to obtain the second point set W = {W1, W2, W3, ..., W m};

[0027] Define l as a scan line, which means a straight line segment whose two endpoints are on the edge of the search area. For the scan line l = (l s ,l e ) to initialize, l s is the initial point of the scan line, l e is the end point of the scan line, N1=W k ,k=0,1,2…m,l←(W1,W k ), m is the sum of the number of vertices in the search area and the number of initial search points, W m is the mth point in the point set, k is the kth point in the point set; get the polygon and calculate its area S;

[0028] If S a ,l←(W1,W k+1 ), recalculate the area S; if S>S a , move the starting point of the scan line counterclockwise along the boundary of the search area until S = S a ;

[0029] Keep S=S on the zone line a Move l s ,l e ​until it is horizontal or vertical to the area line, and the assigned area of ​​the ath unmanned boat is obtained;

[0030] Repeat the above operation to obtain the search area of ​​each unmanned boat.

[0031] Furthermore, the reward function in the deep reinforcement learning algorithm based on POS-DQN is a probability weighted reward function.

[0032] Furthermore, the probability weighted reward function r t The formula is:

[0033] r t =R m ×R o ×R weight ;

[0034] Among them, R m Rewards for obstacles:

[0035]

[0036] R O Reward for visits:

[0037]

[0038] R weight For weighted rewards:

[0039] R weight =w×R POC ; R POC POC reward for grid cell color: R POC =100×POC; w is the weight, which is designed according to the POC size of the grid cell color and whether it passes through obstacles.

[0040] Furthermore, according to the POC distribution diagram, POS is calculated, including:

[0041] POS = POC × POD;

[0042] Among them, POD refers to the probability that the search and rescue target can be found in a given search area, and the calculation formula is:

[0043] POD=1-e -C ;

[0044] Among them, C coverage; Where W is the sea sweep width and R is the route spacing.

[0045] Advantages and positive effects of the present invention: Aiming at the problem of USVs completing maritime search and rescue missions reasonably and efficiently, the present invention provides a maritime search and rescue path planning method with reasonable task allocation and priority search for high-probability areas based on an improved task allocation algorithm and a POS-DQN deep reinforcement learning algorithm. The present invention first uses an improved task allocation algorithm to divide the search area as a whole, so that the task area of ​​each USV has a priority and non-repetitive division. Secondly, the present invention considers the POS of the search environment and proposes a POS-DQN algorithm based on deep reinforcement learning. The algorithm can adapt to the complex and changing environment of search and rescue, and uses a probability weighted reward function to train USVs agents to obtain the optimal search path. Finally, on the basis of considering the comprehensive coverage of obstacle avoidance and collision avoidance, the search path can prioritize the search of high-probability areas, thereby improving the efficiency of search and rescue. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0047] Figure 1 FIG. 1 is a diagram of a USVs search and rescue communication network architecture based on Geo Networking in an embodiment of the present invention;

[0048] Figure 2 A schematic diagram of a point set in an embodiment of the present invention;

[0049] Figure 3 This is a flow chart of reinforcement learning in an embodiment of the present invention;

[0050] Figure 4 Schematic diagram of probability weight reward function in an embodiment of the present invention;

[0051] Figure 5 A maritime search and rescue path planning model in an embodiment of the present invention;

[0052] Figure 6 A drift trajectory diagram in an embodiment of the present invention;

[0053] Figure 7 It is a marine search and rescue environment model in an embodiment of the present invention;

[0054] Figure 8 A single USV path map finally generated by the PA algorithm, CA method, A* algorithm, DQN algorithm, and POS-DQN algorithm in the embodiments of the present invention;

[0055] Fig. 9is an initial position diagram of the USV in an embodiment of the present invention;

[0056] Fig.10 A diagram showing the process of allocating regions and the final result in an embodiment of the present invention;

[0057] Fig.11 It is the USVs path diagram finally generated by the PA algorithm, CA method, A* algorithm, DQN algorithm, and POS-DQN algorithm in the embodiments of the present invention;

[0058] Fig.12 is the path of the search and rescue area in an embodiment of the present invention;

[0059] Fig.13 It is the POS, Single USV, USV(1), USV(2), and USV(3) accumulated in each step in the embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0061] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0062] The present invention uses an improved task allocation algorithm to divide the search area of ​​USVs according to the parameters of the maritime search and rescue environment and the search and rescue equipment. Finally, the task area assigned to each USV meets the priority of the task and is reasonably divided without duplication. A POS-DQN deep reinforcement learning algorithm is proposed to solve the maritime search and rescue path planning. It designs a probability weight reward function for search and rescue to train each USV to avoid obstacles and collisions and prioritize searching high-probability areas, so as to complete the search and rescue mission safely and efficiently.

[0063] In maritime search and rescue, the USVs search and rescue communication network architecture is established based on the Geo networking modality, such as Figure 1 As shown. In USVs maritime search and rescue, the division of the search area must take into account the priority of the task, and the task area must be fully covered without duplication to achieve the maximum probability of success efficiently and quickly. Therefore, the present invention combines the two methods of priority division and overall division, and proposes an improved task allocation algorithm. The POS-DQN algorithm is used to solve the search and rescue path planning in the search area. Based on the DQN reinforcement learning method, through the design of the reward function, the USV agent obtains the search path in the maritime search and rescue environment model according to the algorithm learning training, realizes avoiding obstacles and prioritizing the search of high probability areas, and avoids repeated full coverage.

[0064] An embodiment of the present invention provides a USVs maritime search and rescue path planning method based on POS-DQN, which specifically includes the following steps:

[0065] Step 1: Improved task allocation algorithm, taking into account the POS of the search area; the search area is divided as a whole according to the search initial point when the USV arrives at the search area, so as to realize the task allocation of USVs.

[0066] In specific implementation, the calculation steps of POS are as follows: POS is the probability of successfully finding the search and rescue target in the search area; POS is the main measurement indicator of the search and rescue operation, which is related to the inclusion probability and the discovery probability. The larger the POS value, the higher the search efficiency; the calculation formula is:

[0067] POS = POC × POD;

[0068] According to the optimal search theory, POC is the probability that the search target exists in the search area. The search area is expanded until the area where all particles exist is fully contained, and the POC of the area reaches 100%. In actual maritime search and rescue missions, the number of search and rescue teams is often limited, which requires them to give priority to areas with higher POC. Therefore, the search area is usually subdivided into square grids of the same size, and the probability of the existence of the search target in the search area is quantified by calculating the POC of each grid unit. The calculation formula of POC is:

[0069]

[0070] Where m i is the number of particles in grid i, and M is the number of particles contained in the overall distribution area.

[0071] POS is the probability of successfully finding the search and rescue target in the search area. POS is the main measurement indicator of search and rescue operations. It is related to the inclusion probability and the discovery probability. The larger the POS value, the higher the search efficiency. The calculation formula is:

[0072] POS = POC × POD;

[0073] POD refers to the probability that the search and rescue target can be found in a given search area. The calculation formula is:

[0074] POD=1-e -C ;

[0075] POD includes two important concepts, namely, C coverage and W sweep width. Coverage is a measure of the degree of coverage of the search area by the search equipment during the search process. The calculation formula is:

[0076]

[0077] The sea sweep width is a key parameter that determines the efficiency of maritime search and rescue. It is affected by many factors, including the changing ocean environment, the performance of the search and rescue equipment, and the characteristics of the search and rescue target. R is the route spacing. In the case where the sea sweep width of the unmanned boat considered by the present invention is consistent with the route spacing, POS is mainly related to POC.

[0078] In the specific implementation, the steps of the improved task allocation algorithm are as follows:

[0079] Step 1.1: Because each USV has a different initial position, maximum speed, and search capability, first calculate the time it takes to reach the search area at full speed based on the initial distance and maximum speed of the a-th USV from the search area, as shown in formula (1). Given that the total area of ​​the search area is S, the search time of the a-th USV in the search area is Therefore, the total search time of the ath USV search mission is as shown in formula (2). Then the optimal search and rescue USVs collaboration should minimize the search and rescue time, as shown in formula (3). Finally, according to formula (4), the area constraint set {S1, S2, …, S a …S n}.

[0080]

[0081] Where D a is the initial distance of the a-th USV from the search area; n is the total number of USVs; a is the a-th USV; is the maximum speed of the ath USV; T is the time taken for the entire mission execution = the execution time of USVs participating in the search and rescue mission; is the time when the ath USV arrives at the search area at full speed; is the search time covered by the ath USV in the search area; T a is the total search time of the ath USV search mission; b a is the search capability of the ath USV; S a The search area allocated to the a-th USV; S n The search area allocated to the nth USV.

[0082] Step 1.2: Connect the initial position of the USV to the center of the grid with the highest POC in the search area to obtain the ideal route for the USV to reach the search and rescue area. If the USV is outside the search area, the intersection of the route and the boundary of the search area is the search initial point; if the USV is within the search area, the intersection of the extension of the route and the boundary of the search area is the search initial point. Sort the search initial points in counterclockwise order to obtain a point set of N = {N1, N2, N3, ..., N n},like Figure 2 shown.

[0083] Step 1.3: Sort the search initial point and the search area vertices in counterclockwise order to obtain the point set W = {W1, W2, W3, ..., W m}.

[0084] Step 1.4: Define l as a scan line, which means a straight line segment whose two endpoints are on the edge of the search area. For the scan line l = (l s ,l e ) to initialize, l s is the initial point of the scan line, l e is the end point of the scan line, N1=W k ,k=0,1,2…m,l←(W1,W k ), m is the sum of the number of vertices in the search area and the number of initial search points, W m is the mth point in the point set, and k is the kth point in the point set; get the polygon and calculate its area S.

[0085] Step 1.5: Determine if S a ,l←(W1,W k+1 ), go to step 1.4; if S>S a , move the starting point of the scan line counterclockwise along the boundary of the search area until S = S a , go to step 1.6.

[0086] Step 1.6: Keep S=S on the zone line a Move l s ,l e The position of the a-th USV is obtained by moving the a-th USV horizontally or vertically to the area line. Repeat the above operation to obtain the search area for each USV.​

[0087] At this point, the entire process of USV task allocation is completed.

[0088] Step 2: Based on the DQN algorithm, considering POS, POS-DQN is proposed, and then a maritime search and rescue path planning model is established.

[0089] like Figure 3 As shown in the figure, based on the DQN reinforcement learning method, by designing the reward function, the USV agent obtains the search path in the maritime search and rescue environment model according to the algorithm learning training, and realizes avoiding obstacles and prioritizing the search of high probability areas.

[0090] The DQN algorithm consists of two networks: an evaluation network and a target network. The evaluation network is responsible for real-time learning of the mapping of the state of the environment to the value of the action. The Vessel agent observes the state of the environment at each time step, and then uses the evaluation network to calculate the predicted value of each possible action in that state, that is, the expected future benefit of the action Q(s,a:θ). The optimal value Q is calculated using the Bellman equation * (s,a) is:

[0091]

[0092] Where γ is a discount factor between 0 and 1, which is used to balance the importance of immediate rewards and future rewards; s is the current state; a is the current action; θ is the parameter of the evaluation network; r is the immediate reward; Q is the expected future reward; Q * is the optimal value of Q value; E is the expectation; s′ is the new state; a′ is the action in the new state.

[0093] The target network is a delayed update version of the evaluation network with the same structure, but the target network's θ - It is not updated in real time, but after a specific time interval, the evaluation network's θ is copied to θ - The purpose of this is to reduce instability during training. The target network is used to generate the target Q value, providing a more stable target Q value, which helps to reduce the variance during learning. The formula for the target Q value is:

[0094] y i =r+γmaxQ′(s′,a′:θ - );

[0095] where yi is the target Q value; θ - It is the parameter of the target network.

[0096] The USV agent of the present invention can choose four directions, and then select action a according to the greedy rule ε, {a1, a2, a3, a4} are four actions of moving up, down, left, and right; p is the probability. Get the reward and observe the new state s. The formula for selecting the action is:

[0097] random action={a1,a2,a3,a4};

[0098]

[0099] Then the formula for the action selection strategy π is:

[0100]

[0101] Continuously iterate to obtain a series of quadruple (s, a, r, s′) and store them in the experience replay pool D. In each round, the model selects a set of quadruple from the experience replay pool and calculates the loss function based on the mean square error. The mean square error (MSE) is used as the loss function to update the parameters of the evaluation network. The loss function is minimized by adjusting θ, and finally the future return expectation Q(s, a; θ) predicted by the evaluation network is improved. The formula of the loss function L(θ) is:

[0102] L(θ)=E (s,a,r,s′)-D [(y i -Q(s,a;θ) 2 ];

[0103] In the parameter update phase, the gradient descent method is used to adjust the θ of the evaluation network to minimize the loss function. The gradient formula for:

[0104]

[0105] The parameter θ of the evaluation network is updated according to the gradient formula. α is the learning rate; the update formula is:

[0106]

[0107] Furthermore, the POS-DQN algorithm includes:

[0108] In the USV maritime search and rescue path planning, it is necessary to ensure that the designed reward function can maximize the USV agent's benefits while also achieving comprehensive coverage by avoiding obstacles and prioritizing searches in high-probability areas. According to the above requirements, the present invention designs a probability weighted reward function, such as Figure 4 As shown, the probability weight reward function r t The formula is:

[0109] r t =R m ×R o ×R weight ;

[0110] Among them, safely avoiding obstacles is the basis of path planning and is crucial to ensuring effective search in complex environments, so the obstacle reward R is designed. m The formula is:

[0111]

[0112] To achieve efficient search area with the shortest search and rescue time, the search repetition rate should be reduced, so the visit reward formula R is designed. O for:

[0113]

[0114] Prioritize the search of high probability search areas, and quickly reach the grid with the highest POC value. When the scan width is fixed, the grid with large POC should be searched first, so the weight reward R is designed. weight The formula is:

[0115] R weight =w×R POC ;

[0116] The weighted reward is divided into two parts to satisfy the high probability priority search of the search area. The POC reward is designed according to the POC of the grid cell color. The larger the POC, the greater the reward. The POC reward formula for the grid cell color is:

[0117] R POC =100×POC;

[0118] The weight is designed based on the POC size of the grid cell color and whether or not to pass the obstacle. The larger the POC, the greater the weight. Because safely avoiding obstacles is the most important, the obstacle has the highest weight. The weight formula is:

[0119]

[0120] At this point, the POS-DQN deep reinforcement learning algorithm used for path planning has been established.

[0121] In the above-mentioned embodiments, the search and rescue process is carried out efficiently and reasonably to solve the problem of disappearance or death caused by too long a search time. The present invention enables USVs to perform search and rescue tasks reasonably and efficiently. For task allocation, an improved task allocation algorithm is used to divide the search area of ​​USVs according to the parameters of the maritime search and rescue environment and the search and rescue equipment. Finally, the task area assigned to each USV meets the priority of the task and is divided reasonably without duplication; for the path planning of the task area, a POS-DQN deep reinforcement learning algorithm is proposed to solve the maritime search and rescue path planning. This algorithm designs a probability weight reward function for search and rescue to train each USV to avoid obstacles and collisions and prioritize searching high-probability areas; ultimately, a safe and efficient path planning is obtained to complete the search and rescue mission.

[0122] In order to verify the effectiveness of the maritime search and rescue path planning model, a simulation experiment was conducted in the embodiment of the present invention for training analysis. The target particle distribution map at different times was obtained through drift calculation to determine the search area. The POC distribution map of the search area was combined with the grid map of the search area to generate a maritime search and rescue environment model, such as Figure 5 shown.

[0123] In order to verify the effectiveness of the maritime search and rescue path planning model, simulation experiments were conducted for training and analysis. By selecting the vicinity of Bohai Bay, a 12-hour drift simulation experiment was simulated for people in distress who collided with a ship. There are many factors affecting maritime search and rescue, including wind pressure, wind-driven currents, ocean currents, tidal currents, and factors of the drifting object itself, but the biggest influence is mainly wind and current. When determining the search area, the scanning width of the USV should be taken into account, which affects the inclusion probability of the area. In order to achieve efficient and rapid search, a scientific and reasonable calculation method must be adopted to determine the target search area. First, the drift trajectory is predicted using marine environmental data and the last known position. In the drift prediction process, objects in distress are regarded as particles. The present invention uses the Lagrangian particle tracking algorithm to calculate the displacement trajectory of particles under the influence of the marine environment.

[0124] The calculation formula is as follows:

[0125]

[0126] where is the displacement of the particle, x t is the drift coefficient, A(x t ) is the diffusion coefficient, Z n is a random number. Its drift trajectory is as follows Figure 6 shown.

[0127] After drift calculation, the target particle distribution map at different times is obtained to determine the search area. For the path planning of USV navigation, it is first necessary to understand the location of the obstacles. The grid method is usually used to process the electronic nautical chart of the sea environment. According to the grid method, the electronic nautical chart of the search area is divided into regular grids, and the size of the grid unit is designed according to the scanning width of the USV. Each grid is independent, and the coordinates of the center point of each grid are used as the position coordinates of the grid. The POC of each grid is calculated based on the particle distribution map obtained by drift. The grid unit color is assigned according to the size of the POC to obtain the POC distribution map of the search area. The POC distribution map is combined with the grid map of the search area to generate a maritime search and rescue environment model, such as Figure 7 As shown in the figure, a is the wind chart, flow chart and electronic nautical chart at that time; b is the POC distribution map and grid map; c is the final maritime search and rescue environment.

[0128] At this point, the USVs maritime search and rescue path planning based on POS-DQN is completed. A single USV is trained in a maritime search and rescue environment, and a comparative experiment is conducted using the parallel line scanning algorithm (PA), the lateral line scanning algorithm (CA), the A* algorithm, the DQN algorithm, and the POS-DQN algorithm used in the present invention. And the final generated path is as follows Figure 8 As shown in the figure. The training result of a single USV path using this method achieves full coverage without obstacles and no duplication. Although the path of this method can prioritize the search of high-probability areas, the effect is not optimal because it needs to consider full coverage. In addition, it takes a long time for a single USV to search the entire area, and the search and rescue is not timely enough. Therefore, USVs are usually used for search and rescue.

[0129] Select three USVs with different initial positions within 3nmi of the center point with the highest POC value in the search area at that time. Their initial positions are Fig. 9 As shown in the figure, USVs are assigned tasks according to the improved task assignment algorithm, and the final assigned area result is shown in the figure Fig.10 As shown in the figure, USVs are trained in a maritime search and rescue environment, and the parallel line scanning algorithm (PA), the lateral line scanning algorithm (CA), the A* algorithm, the DQN algorithm, and the POS-DQN algorithm are used for comparative experiments. And the final generated path is as follows Fig.11 The training results of the USVs path using the algorithm of the present invention achieves comprehensive coverage of avoiding obstacles and no duplication, and gives priority to searching high-probability areas. The final path in the search and rescue area is Fig.12 The present invention calculates and analyzes the cumulative POS at each step to evaluate the above methods. Fig.13 As shown, the growth rate of the algorithm used in the present invention changes the fastest and can preferentially search high-probability areas.

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-unmanned boat maritime search and rescue path planning method based on POS-DQN, It is characterized in that include: Calculate the POC of the search area and obtain a POC distribution map; POC is the probability that the search and rescue target exists in the search area; The search area is divided as a whole according to the search initial point when the unmanned boat arrives at the search area, and the search area of ​​each unmanned boat is obtained, so that the mission area of ​​each unmanned boat has priority and no duplication. The POC distribution map is combined with the grid map of the search area to generate a marine search and rescue environment model; According to the POC distribution map, calculate POS, which is the probability of successfully finding the search and rescue target in the search area; The POS-DQN-based reinforcement learning method is used to train unmanned boats, and the optimal search and rescue path for each unmanned boat is obtained in the maritime search and rescue environment model, so as to avoid obstacles and prioritize the search of high-probability areas.

2. According to the POS-DQN-based multi-unmanned boat maritime search and rescue path planning method of claim 1, It is characterized in that Calculate the POC of the search area and obtain the POC distribution map, including: The search area is subdivided into square grids of the same size, the POC of each grid unit is calculated, and the POC distribution map of the search area is obtained; Among them, the calculation formula of POC is: Where m i is the number of particles in grid i, and M is the number of particles contained in the overall distribution area.

3. According to claim 2, a method for planning a multi-unmanned boat maritime search and rescue path based on POS-DQN, It is characterized in that Determine the initial search point for the unmanned boat to arrive at the search area, including: Calculate the time it takes to reach the search area at full speed based on the initial distance of the ath unmanned boat from the search area and its maximum speed It is known that the total area of ​​the search area is S, and the search time covered by the a-th unmanned boat in the search area is Therefore, the total search time for the ath unmanned boat search mission is: The best search and rescue multi-UAV collaboration should minimize the search and rescue time: according to Get the area constraint set of the search and rescue unit {S 1 ,S 2 ,…,S a …S n }; Where D a is the initial distance between the a-th unmanned boat and the search area; n is the total number of unmanned boats; a is the a-th USV; is the maximum speed of the ath unmanned boat; T is the time taken for the entire mission, that is, the time taken for multiple unmanned boats to participate in the search and rescue mission; The time when the ath unmanned boat arrives at the search area at full speed; is the search time covered by the ath unmanned boat in the search area; T a is the total search time for the ath unmanned boat search mission; b a is the search capability of the ath unmanned boat; S a The search area allocated to the ath unmanned boat; S n The search area allocated to the nth unmanned boat; Connect the initial position of the unmanned boat with the center point of the grid with the highest POC in the search area to obtain the ideal route for the unmanned boat to reach the search and rescue area; If the unmanned boat is outside the search area, the intersection of the route and the boundary of the search area is the initial point of the search; if the unmanned boat is within the search area, the intersection of the extended line of the route and the boundary of the search area is the initial point of the search.

4. According to the POS-DQN-based multi-unmanned boat maritime search and rescue path planning method of claim 3, It is characterized in that The search area is divided as a whole to obtain the search area of ​​each unmanned boat, including: Sort the search initial points in counterclockwise order to obtain the first point set N = {N 1 ,N 2 ,N 3 ,…,N n }; Sort the search initial point and the search area vertices in counterclockwise order to obtain the second point set W = {W 1 ,W 2 ,W 3 ,…,W m }; Define l as a scan line, which means a straight line segment whose two endpoints are on the edge of the search area. For the scan line l = (l s ,l e ) to initialize, l s is the initial point of the scan line, l e is the end point of the scan line, N 1 =W k ,k=0,1,2…m,l←(W 1 ,W k ), m is the sum of the number of vertices in the search area and the number of initial search points, W m is the mth point in the point set, k is the kth point in the point set; get the polygon and calculate its area S; If S a ,l←(W 1 ,W k+1 ), recalculate the area S; if S>S a , move the starting point of the scan line counterclockwise along the boundary of the search area until S = S a ;​ Keep S=S on the zone line a Move l s ,l e until it is horizontal or vertical to the area line, and the assigned area of ​​the ath unmanned boat is obtained; Repeat the above operation to obtain the search area of ​​each unmanned boat.

5. According to the POS-DQN-based multi-unmanned boat maritime search and rescue path planning method of claim 1, It is characterized in that The reward function in the DQN-based deep reinforcement learning algorithm is a probability weighted reward function.

6. According to claim 5, a method for planning a multi-unmanned boat maritime search and rescue path based on POS-DQN, It is characterized in that Probability weighted reward function r t The formula is: r t =R m ×R o ×R weight ; Among them, R m Rewards for obstacles: R O Reward for visits: R weight For weighted rewards: R weight =w×R POC ; R POC POC reward for grid cell color: R POC =100×POC; w is the weight, which is designed according to the POC size of the grid cell color and whether it passes through obstacles.

7. According to the POS-DQN-based multi-unmanned boat maritime search and rescue path planning method of claim 1, It is characterized in that According to the POC distribution diagram, calculate POS, including: POS = POC × POD; Among them, POD refers to the probability that the search and rescue target can be found in a given search area, and the calculation formula is: POD=1-e -C ; Among them, C coverage; Where W is the sea sweep width and R is the route spacing.