Training method of search path planning model and offshore search path planning method

Abstract

The application provides a training method of a search path planning model, a maritime search path planning method, and the training method comprises the following steps: constructing a training sample set based on a mixed Gaussian probability distribution; initializing a maritime search path planning model; and using the training sample set to train the maritime search path planning model based on a threshold control scene switching training strategy until the loss value of the maritime search path planning model is less than a preset loss threshold value, so as to obtain the trained maritime search path planning model. The application can reduce search time consumption, improve the accuracy of path planning, and improve the rescue probability of a search target.

Description

Technical Field

[0001] This invention belongs to the field of maritime search technology, specifically involving a training method for a search path planning model and a maritime search path planning method. Background Technology

[0002] In recent years, extreme weather events at sea have become more frequent, maritime activities have increased, and the risk of maritime accidents has continued to rise, making the maritime rescue situation increasingly severe. The probability of drowning, hypothermia, or dehydration and death increases over time for those in distress awaiting rescue. Maritime search and rescue operations can be divided into two phases: search and rescue, with the search phase being the longest. Therefore, how to scientifically and efficiently plan search routes for search and rescue equipment to achieve rapid searches in complex marine environments is a critical issue that urgently needs to be addressed.

[0003] Existing research methods can be divided into four paradigms: traditional methods, exact algorithms, heuristic algorithms, and reinforcement learning (RL).

[0004] Traditional methods often combine other approaches to simplify the problem. Some researchers use the K-means method to first divide the search area for the helicopter, and then use a "plowing" approach to solve the problem; others use methods such as grid decomposition and Voronoi diagrams to allocate the search area, and use the "mowing" and "outer spiral" methods to plan the search path. Exact algorithms typically abstract the problem into models such as mixed-integer programming. Some researchers have proposed greedy algorithms with branch-and-bound processes to solve the extended search problem in weighted graphs; some have developed pseudo-polynomial dynamic programming methods to handle time-sensitive mission scenarios. Regarding heuristic algorithms, some researchers designed an A* algorithm based on prior information, successfully solving the underwater search problem for unmanned surface vessels; some researchers selected the best solution from 96 variants of the ant colony algorithm; some researchers proposed a particle swarm optimization algorithm based on co-evolutionary motion encoding to solve the search problem for multiple dynamic targets; and some researchers first use the A* algorithm to pre-plan the flight path, and then use a set full-coverage algorithm to update the path in real time.

[0005] RL (Research-Based Learning) is a method of autonomous learning through trial and error, and it has made many key advances in the field of maritime search path planning. Some researchers have improved the Q-Learning algorithm to QTraverse based on rasterized information maps; some researchers have designed a QLearning full-coverage path planning model that prioritizes searching high-probability areas, and proposed the rule of prioritizing searching high-probability areas for the first time; some researchers have proposed a maritime battlefield target search task planning method based on the Rainbow algorithm; some researchers have divided the region into crosses to define the search area for each agent, and combined the Deep Q Network (DQN) algorithm to solve the path for a single agent; some researchers have introduced a collision avoidance mechanism into DQN to solve the search problem in fixed scenarios.

[0006] While traditional methods perform stably in certain scenarios, their search patterns are largely based on pre-defined rules, making them inflexible and difficult to adapt to complex situations. Exact algorithms, although guaranteeing the optimal solution, have high computational complexity, failing to meet the rapid planning requirements of real-world search and rescue missions. Heuristic algorithms, while offering some flexibility, essentially only solve for a specific scenario, exhibiting weak cross-scenario transferability. These shortcomings reduce the probability of rescuing the target. Summary of the Invention

[0007] The technical problem to be solved by this invention is to provide a training method for a search path planning model and a maritime search path planning method, so as to reduce search time, improve the accuracy of path planning, and increase the probability of rescue of search and rescue targets.

[0008] In a first aspect, the present invention provides a method for training a search path planning model, the method comprising the following steps: A training sample set is constructed based on a Gaussian mixture probability distribution. The training sample set includes multiple training samples, which correspond one-to-one with multiple search scenarios. The search success rate distribution is different in different search scenarios. The search success rate is obtained by drift prediction of the search area based on the drift prediction framework. Initialize the maritime search path planning model; the maritime search path planning model is an improved PPO model. The improved PPO model includes a spatiotemporal perception module for extracting spatiotemporal features of the agent's state sequence. The spatiotemporal perception module is connected to the Actor network and the Critic network respectively. The spatiotemporal features are used to guide the action policy generation of the Actor network and the action policy update of the Critic network, representing the impact of the action executed at the current time step on the action executed in subsequent time steps. The threshold-based scene switching training strategy uses a training sample set to train the maritime search path planning model until the loss value of the maritime search path planning model is less than a preset loss threshold, thus obtaining the trained maritime search path planning model.

[0009] Optionally, a training sample set is constructed based on a Gaussian mixture probability distribution, including: The local search success rate distribution of the search target at each time step is simulated based on a two-dimensional Gaussian function, and the local search success rate distributions of multiple time steps are superimposed to obtain the global search success rate distribution of the search target during the search period. This global search success rate distribution is used as a training sample. Multiple training samples were obtained by adjusting the covariance matrix of the two-dimensional Gaussian function multiple times.

[0010] Optionally, the expression for the training samples is:

[0011]

[0012]

[0013] in, This represents the global search success rate distribution. Indicates the grid cells in the search area. , Indicates the position of the grid cell. Indicates the first Distribution of local search success rates at each time step , This represents the total number of time steps, indicating the search period. The covariance matrix represents the shape and orientation of the search success rate distribution. Indicates the first The mean vector of the Gaussian distribution corresponding to each time step corresponds to the center point of the search region. These represent the variances of the search region horizontally and vertically, respectively. The variance represents the covariance in the horizontal and vertical directions. The variances are randomly selected between 0 and 1 to simulate various distribution scenarios.

[0014] Optionally, the spatiotemporal perception module is a long short-term memory network; Extracting the spatiotemporal features of the agent's state sequence, including: Obtain the state sequence of the agents participating in the search at multiple time steps; The state sequence is input into a Long Short-Term Memory (LSTM) network and passed sequentially through a forget gate, an input gate, and an output gate. The output gate outputs the spatiotemporal features. The forget gate filters redundant noise, including searched regions with no search value (POS) and distant grid cells irrelevant to the current decision. The input gate retains key information, such as decision patterns in historical paths. The output gate adjusts the weights of forgetting and retaining. The spatiotemporal features include the grid cell search order, remaining search time, and the agent's spatial location.

[0015] Optional scene switching training strategies include: Step 1: Use a commercial solver to find the optimal path in all search scenarios, calculate the first cumulative search success rate corresponding to the optimal path, and construct a scenario switching threshold based on the first cumulative search success rate. Step II: Obtain the action policy generated by the Actor network in the current round; Step III: Calculate the second cumulative search success rate of the action policy in the search scenario corresponding to the current training sample; Step IV: If the second cumulative search success rate is greater than the scene switching threshold, then train the Actor network using other training samples besides the current training samples; otherwise, store the second cumulative search success rate in the experience replay pool, update the Actor network by sampling batches from the experience replay pool, obtain the new Actor network, and obtain the action policy generated by the new Actor network in the current round, then return to execute step III.

[0016] Optionally, the update step size of the Actor network is determined based on the probability ratio of the new action policy to the old action policy.

[0017] Optionally, the loss values ​​for the maritime search path planning model include strategy loss and evaluation loss; The expression for the policy loss is:

[0018]

[0019]

[0020]

[0021] in, The model parameters of the Actor network are: Strategy loss at that time The advantage function is the action value function. and state value function The difference, This indicates that the current action is better than average, and the probability of selecting this action should be increased. This indicates that the current action is worse than average, and the probability of selecting this action should be reduced. Used to evaluate in Take in state The expected cumulative reward of an action. Expressing expectations, Represents the trajectory. Representation strategy, Indicates the discount factor. Indicates time step Instant rewards Indicates the total time steps. Used for evaluation The quality of a state is characterized by the expected value of the value function of all possible actions that can be taken in that state. This represents the probability ratio between the new action strategy and the old action strategy. Represents the cutoff coefficient, used to... Limited to to Within a certain range, prevent the Actor network from updating too much; The expression for evaluating the loss is:

[0022] in, The model parameters of the Critic network are: The evaluation loss at that time The model parameters of the Critic network are: Predicting the value function of the current state. This represents the target value of the time-series difference. , This represents the immediate reward at the current time step. Indicates the state at the next time step. The projected value, This represents the discount factor, used to balance the weight between immediate rewards and rewards in the next time step.

[0023] Secondly, the present invention provides a method for planning a maritime search path, which uses a model trained by the above-mentioned training method to plan a maritime search path and obtain a maritime search path planning result.

[0024] Optionally, the model is used for maritime search path planning to obtain the maritime search path planning results, including: Obtain environmental information of the search area and parameter information of the agents participating in the search; The search area is rasterized based on environmental and parameter information to obtain multiple raster cells. The search success rate of each grid cell is calculated based on the drift prediction framework; By inputting multiple grid cells, search success rate, and parameter information into the model, the results of the maritime search path planning are obtained.

[0025] Optionally, the search area can be rasterized based on environmental and parameter information to obtain multiple raster cells, including: Calculate the effective search radius based on environmental and parameter information; The search area is rasterized based on the effective search radius to obtain multiple raster cells.

[0026] The beneficial effects of this invention are: The training method for the search path planning model provided by this invention constructs a training sample set based on a Gaussian mixture probability distribution, providing diverse training scenarios for model training, improving the model's generalization performance, and enhancing the accuracy of path planning. The maritime search path planning model is an improved PPO model, including a spatiotemporal perception module that extracts the spatiotemporal features of the agent's state sequence. This module can dynamically adjust the degree of attention paid to historical information, enabling the Actor network and Critic network to more effectively capture temporal decision information, adapt to complex marine environments, and achieve efficient search path planning. The threshold-controlled scene switching training strategy ensures that the model learns sufficiently in one scene before entering the next, reducing noise interference from inefficient strategies, lowering search time, improving the accuracy of path planning, and increasing the probability of rescuing the search target. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating a training method for a search path planning model in one embodiment of this application. Figure 2 This is a schematic diagram of the training sample set in one embodiment of this application; Figure 3 This is a schematic diagram of the experimental results of different center point distributions in one embodiment of this application; Figure 4 For detailed accident case information in one embodiment of this application; Figure 5 This is a schematic diagram of the path planning results of different algorithms in one embodiment of this application; Figure 6 This is a cumulative POS curve for different algorithms in one embodiment of this application. Detailed Implementation

[0028] To address the issues of high search time and low path planning accuracy in traditional methods, this invention discloses a training method for a search path planning model. This method constructs a training sample set based on a Gaussian mixture probability distribution, providing diverse training scenarios for the model, improving its generalization performance, and thus enhancing the accuracy of path planning. The maritime search path planning model is an improved PPO model, including a spatiotemporal perception module that extracts the spatiotemporal features of the agent's state sequence. This module can dynamically adjust the degree of attention paid to historical information, enabling the Actor network and Critic network to more effectively capture temporal decision information, adapt to complex marine environments, and achieve efficient search path planning. A threshold-controlled scene switching training strategy ensures that the model learns sufficiently in one scene before entering the next, reducing noise interference from inefficient strategies, lowering search time, improving path planning accuracy, and increasing the probability of rescuing the search target.

[0029] The training method of the search path planning model disclosed in this invention will be described below, such as... Figure 1 As shown, the training method includes: Step 11: Construct a training sample set based on a Gaussian mixture probability distribution.

[0030] In this embodiment of the invention, the training sample set includes multiple training samples, which correspond one-to-one with multiple search scenarios. The success rate distribution in different search scenarios is different, and the success rate is obtained by drift prediction of the search area based on the drift prediction framework.

[0031] In one feasible implementation, constructing a training sample set based on a mixture of Gaussian probability distributions includes steps 11.1 to 11.2.

[0032] Step 11.1: Simulate the local search success rate distribution of the search target at each time step based on the two-dimensional Gaussian function, and superimpose the local search success rate distributions of multiple time steps to obtain the global search success rate distribution of the search target during the search period. Use this global search success rate distribution as a training sample.

[0033] Specifically, the expression for the training samples is:

[0034]

[0035]

[0036] in, This represents the global search success rate distribution. Indicates the grid cells in the search area. , Indicates the position of the grid cell. Indicates the first Distribution of local search success rates at each time step , This represents the total number of time steps, indicating the search period. The covariance matrix represents the shape and orientation of the search success rate distribution. Indicates the first The mean vector of the Gaussian distribution corresponding to each time step corresponds to the center point of the search region. These represent the variances of the search region horizontally and vertically, respectively. The variance represents the covariance in the horizontal and vertical directions. The variances are randomly selected between 0 and 1 to simulate various distribution scenarios.

[0037] It should be noted that researchers typically combine the Monte Carlo method with drift prediction models to simulate and predict the possible movement paths of a target. The Monte Carlo method generates a large number of random particles in the approximate sea area where the accident occurred, and uses a drift prediction model to predict the drift trajectories of these particles, thereby estimating the probability distribution of the target's location. In this embodiment of the invention, based on sea wind and current data provided by the Marine Environmental Forecasting Center, it is input into the Leeway module of the open-source drift prediction framework OpenDrift. The drift of the distressed target is simulated hourly. Combined with the search radius of search and rescue equipment, navigation speed, and weather conditions, the effective search radius can be calculated. Subsequently, the task sea area is rasterized based on the effective search radius to construct a search area model.

[0038] In this embodiment of the invention, the search success rate POS is composed of the probability of containment (POC) and the probability of detection (POD): .in, This indicates the probability of the search target appearing in a given grid cell. , Indicates the first The number of particles contained in each grid cell Represents the total number of particles. This represents the probability that a search target can be successfully detected by a search agent (such as a drone or ship) given that the target is located within a certain grid cell. ,in, The coverage factor is the ratio of the distance between flight paths to the effective search radius.

[0039] Step 11.2 involves adjusting the covariance matrix of the two-dimensional Gaussian function multiple times to obtain multiple training samples.

[0040] Specifically, each two-dimensional Gaussian function can be viewed as a bell-shaped curve with an elliptical cross section, which can be controlled by... The value of can adjust the shape and orientation of the ellipse, where These controls the degree of stretching in the row and column directions respectively. The larger the value, the more dispersed the search success rate distribution. The direction of the distribution tilt is controlled to simulate asymmetric drift caused by ocean currents.

[0041] In one embodiment of the present invention, the main diagonal parameter from Uniform sampling, when the value is 0.5, results in a concentrated distribution of search success rates, with small and concentrated areas of high success rates. When the value is 1.0, the distribution of search success rates is dispersed, with large and dispersed areas of high success rates. (Off-diagonal parameter) from Uniform sampling This is used to simulate the impact of ocean current direction on the distribution of search success rate.

[0042] In one feasible implementation, the drift prediction of the search target is in hours. For example, when a person falls into the water, 500 particles are randomly generated at the point of fall to represent the possible location of the person. Therefore, each particle will have a location in each hour. Different training samples can be understood as the search success rate distribution in different hour intervals. Each two-dimensional Gaussian distribution represents the search success rate distribution in one hour.

[0043] For example, after performing step 11, the training sample set is constructed, which includes 6 training samples, specifically as follows: Figure 2 As shown.

[0044] Step 12: Initialize the maritime search path planning model.

[0045] The maritime search path planning model is an improved PPO model. The improved PPO model includes a spatiotemporal perception module for extracting spatiotemporal features of the agent's state sequence. The spatiotemporal perception module is connected to the Actor network and the Critic network respectively. The spatiotemporal features are used to guide the action policy generation of the Actor network and the action policy update of the Critic network, representing the impact of the action executed at the current time step on the action executed in subsequent time steps.

[0046] In this embodiment of the invention, the spatiotemporal perception module is a Long Short-Term Memory (LSTM) network. The process of maritime search path planning is essentially a temporal decision-making problem, requiring consideration of the impact of the current action on a series of subsequent steps. LSTM can dynamically adjust the degree of attention given to historical information. In this embodiment of the invention, it is necessary to prioritize searching high-probability regions while reducing the focus on zero-value regions (…). The focus is on the forget gate (0). Through the synergistic effect of the forget gate, input gate, and output gate, key information can be selectively retained and redundant noise can be suppressed, allowing for better perception of spatiotemporal changes and thus avoiding short-sighted decision-making.

[0047] The process of extracting the spatiotemporal features of the agent's state sequence is explained below.

[0048] Specifically, the process first acquires the state sequences of the agents participating in the search across multiple time steps. These state sequences are then input into a Long Short-Term Memory (LSTM) network. Through the coordinated action of forget gates, input gates, and output gates, key information is selectively retained while redundant noise is suppressed. The LTM network then outputs spatiotemporal features, including the grid cell search order, remaining search time, and the agent's spatial location.

[0049] In one feasible implementation, the state space in the maritime search path planning model... It consists of the following three parts. To improve generalization performance, all state features have been normalized: ;in, The observation window is the local information around the current position of the agent. Taking the current position as the center, the POS values ​​of the grid cells in the neighborhood are extracted to form an observation matrix for the agent to perceive the surrounding environment. It is the ratio of remaining time steps to total time steps, used to characterize the pressure on the remaining time of the search task; This indicates the current position of the agent.

[0050] Action space is defined as: ;in, These represent movement operations in the four directions: east, south, west, and north. In one feasible implementation, actions that cause the agent to move beyond the map boundaries are removed from the action space of the current location.

[0051] To guide the agent in its exploration and reduce ineffective search behavior, the number of times each grid is visited is recorded during training. The reward function consists of three parts:

[0052] in, To explore rewards, when searching for a grid with a non-zero POS value that has not been searched, the agent is given a positive reward, the value of which is the POS value of that grid. For repeated punishment, If the agent repeatedly searches for the same grid cell, a negative reward is applied, with the intensity of the penalty proportional to the number of repetitions. This is a repeated penalty factor; This represents a zero-value penalty. To reduce the agent's search in the zero-value region, a negative reward is applied to the agent when searching for grid cells with a POS value of 0. It is a zero-value penalty factor.

[0053] Step 13: Based on the threshold-controlled scene switching training strategy, the maritime search path planning model is trained using the training sample set until the loss value of the maritime search path planning model is less than the preset loss threshold, thus obtaining the trained maritime search path planning model.

[0054] Generally, the complete process of an agent moving from the initial state (initial position) to the final state (target position) is called a round, corresponding to the search path. That is, starting from the initial state, the final state is reached when the total time constraint is exceeded or the POS value of all grid cells is 0, which is one round.

[0055] However, traditional training methods have significant drawbacks. If a new scene is randomly switched each round, the model begins frequent updates even before learning simple search strategies, storing a large number of inefficient strategies in the experience replay pool, resulting in poor training performance. To address this issue, this invention proposes a threshold-based scene training strategy. This involves setting a performance threshold to ensure the model has been sufficiently learned in one scene before moving to the next, thus reducing noise interference.

[0056] Specifically, this includes steps I through IV.

[0057] Step 1: Use a commercial solver to find the optimal path in all search scenarios, calculate the first cumulative search success rate corresponding to the optimal path, and construct a scenario switching threshold based on the first cumulative search success rate.

[0058] In one feasible implementation, the commercial solver is Gurobi.

[0059] In one feasible implementation, the scene switching threshold is: ;in, Indicates the threshold coefficient. , This indicates the first cumulative search success rate.

[0060] Step II: Obtain the action policy generated by the Actor network in the current round.

[0061] Step III: Calculate the second cumulative search success rate of the action policy in the search scenario corresponding to the current training sample.

[0062] Step IV: If the second cumulative search success rate is greater than the scene switching threshold, then train the Actor network using training samples other than the current training samples. Otherwise, store the second cumulative search success rate in the experience replay pool, update the Actor network by sampling batches from the experience replay pool to obtain a new Actor network, obtain the action policy generated by the new Actor network in the current round, and return to execute step III.

[0063] It should be noted that when the second cumulative search success rate is greater than the scene switching threshold, it can be considered that the Actor network has roughly grasped the search pattern of the current search scene, and the Actor network can then switch to the new search scene.

[0064] In this embodiment of the invention, the loss value of the maritime search path planning model includes strategy loss and evaluation loss.

[0065] The expression for the policy loss is:

[0066]

[0067]

[0068]

[0069] in, The model parameters of the Actor network are: Strategy loss at that time The advantage function is the action value function. and state value function The difference, This indicates that the current action is better than average, and the probability of selecting this action should be increased. This indicates that the current action is worse than average, and the probability of selecting this action should be reduced. Used to evaluate in Take in state The expected cumulative reward of an action. Expressing expectations, Represents the trajectory. Representation strategy, Indicates the discount factor. Indicates time step Instant rewards Indicates the total time steps. Used for evaluation The quality of a state is characterized by the expected value of the value function of all possible actions that can be taken in that state. This represents the probability ratio between the new action strategy and the old action strategy. Represents the cutoff coefficient, used to... Limited to to Within a certain range, to prevent the Actor network from updating too much.

[0070] In an embodiment of the present invention, It represents the expected value; during training, the AI ​​continuously interacts with the environment based on its current policy, generating a large number of trajectories—that is, a process of constantly taking actions from the current state to reaching a new state and then taking another action. These trajectories can be used to estimate the action value function and the state value function. Specifically: Indicates time step The expected value. That is to the from All trajectories originating from a state are sampled, and the expected value of the cumulative reward for these trajectories is calculated to estimate the value of the action. From The state value is estimated by sampling different actions taken from a given state and calculating the expected value of the action value function corresponding to these actions.

[0071] The expression for evaluating the loss is:

[0072] in, The model parameters of the Critic network are: The evaluation loss at that time The model parameters of the Critic network are: Predict the value function of the current state, and then predict the state. The input is the Critic network, which is obtained through forward propagation. This represents the target value of the time-series difference. , This represents the immediate reward at the current time step (as described by the reward function mentioned earlier). get), Indicates the state at the next time step. The predicted value of the state The input is the Critic network, which is obtained through forward propagation. This represents the discount factor, used to balance the weight between immediate rewards and rewards in the next time step.

[0073] In one feasible embodiment, an optimizer can be used to update the network in the maritime search path planning model by setting a learning rate.

[0074] In one embodiment of the present invention, in order to verify the effectiveness of the training method of the search path planning model provided by the present invention, experiments were conducted on the generalization performance of the search path planning model in unknown scenarios: To comprehensively evaluate the generalization performance of the training method for the search path planning model provided in this invention in unknown scenarios, 100 scenarios were randomly selected from 1000 scenarios as a test set, covering three types: single-center, dual-center, and multi-center, with roughly equal numbers of scenarios for each type. Testing was then conducted on the test set, and the cumulative results were recorded. and The ratio is used as a performance score. The model is run 30 times in each test scenario, and the average value is taken as the final result. The model is trained to... After several rounds, the performance stabilized, and the generalization performance reached its optimal level. Experimental results for scenarios with different distributions of the number of center points are shown below. Figure 3 As shown.

[0075] Depend on Figure 3 As can be seen, the training method for the search path planning model provided by this invention achieves an average score of 96.05% on the training set and 94.87% on the test set, with an average performance score exceeding 93% in scenarios with different distributions of the number of center points. This indicates that the model obtained through this training method has learned the probabilistic characteristics of a Gaussian mixture probability distribution and possesses good generalization performance. Furthermore, the model obtained through this training method has an average planning time of 0.55 seconds in a single scenario, meeting the timeliness requirements of maritime search missions.

[0076] As can be seen, the training method of the search path planning model provided by this invention constructs a training sample set based on a Gaussian mixture probability distribution, providing diverse training scenarios for model training, improving the generalization performance of the model, and contributing to the improvement of path planning accuracy. The maritime search path planning model is an improved PPO model, including a spatiotemporal perception module that extracts the spatiotemporal features of the agent's state sequence, which can dynamically adjust the degree of attention to historical information, enabling the Actor network and Critic network to more effectively capture temporal decision information, adapt to complex marine environments, and achieve efficient search path planning. The threshold-controlled scene switching training strategy ensures that the model learns sufficiently in one scene before entering the next, reducing noise interference from inefficient strategies, reducing search time, improving the accuracy of path planning, and increasing the probability of rescue of the search and rescue target.

[0077] The present invention also provides a method for planning a maritime search path, which uses a model trained by the above-mentioned training method to plan a maritime search path and obtain the maritime search path planning result.

[0078] Specifically, the maritime search route planning method includes steps A through D.

[0079] Step A: Obtain environmental information of the search area and parameter information of the agents participating in the search.

[0080] The aforementioned environmental information includes meteorological and ocean current information, which can be obtained from publicly available meteorological and ocean current datasets. Parameter information includes the agent's search radius and navigation speed.

[0081] Step B involves rasterizing the search area based on environmental and parameter information to obtain multiple raster cells.

[0082] In one feasible implementation, sea wind and current data provided by the China National Marine Environmental Forecasting Center are input into the Leeway module of the open-source drift prediction framework OpenDrift. The drift of the distressed target is simulated hourly. Combined with the search radius of search and rescue equipment, navigation speed, and weather conditions, the effective search radius can be calculated. Subsequently, the task sea area is rasterized based on the effective search radius to obtain multiple raster cells.

[0083] Step C: Calculate the search success rate for each grid cell based on the drift prediction framework.

[0084] Step D involves inputting multiple grid cells, search success rate, and parameter information into the model to obtain the maritime search path planning results.

[0085] In one embodiment of the present invention, to test the application effect of the maritime search path planning method in a real complex environment, a case of a person falling overboard in the waters off Hong Kong, Macao and Zhuhai was selected as an experimental case. Detailed information about this case is as follows: Figure 4 As shown.

[0086] After inputting the marine environmental data of the incident area into OpenDrift for drift prediction, the search area is rasterized according to the effective search radius of the UAV, and the POS distribution of the search area is calculated.

[0087] Based on this, comparisons were made with Q-Learning, DQN, and Gurobi. The Q-Learning and DQN algorithms, after being trained for 5000 rounds in this scenario, both required more than 5 minutes of training time before testing. The search path planning results for 30 time steps starting from the drone's initial position are as follows: Figure 5 As shown, the Q-Learning algorithm suffers from redundant paths due to insufficient feature fitting ability, while the other methods do not exhibit redundant or invalid searches and can prioritize searching high-probability regions.

[0088] The cumulative POS of each algorithm's planned path is as follows: Figure 6As shown. The maritime search path planning method (SAPPO, Spatiotemporal-Aware Proximal Policy Optimization) provided by this invention has the same cumulative POS value as the DQN model, which can only work in a single scenario. Although it is still some distance from the optimal solution, it takes the least amount of time and is more in line with the timeliness requirements of maritime search path planning tasks.

[0089] In summary, the maritime search path planning method provided by this invention can reduce search time, improve the accuracy of path planning, and increase the probability of rescuing the search target.

[0090] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of protection of this application is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of one or more embodiments of this application as described above, which are not provided in detail for the sake of brevity.

[0091] One or more embodiments in this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of this application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments in this application should be included within the protection scope of this application.

Claims

1. A training method for a search path planning model, characterized in that, include: A training sample set is constructed based on a mixture Gaussian probability distribution; The training sample set includes multiple training samples, which correspond one-to-one with multiple search scenarios. The search success rate distribution is different in different search scenarios. The search success rate is obtained by drift prediction of the search area based on the drift prediction framework. Initialize the maritime search path planning model; the maritime search path planning model is an improved PPO model, the improved PPO model includes a spatiotemporal perception module for extracting spatiotemporal features of the agent's state sequence, the spatiotemporal perception module is connected to the Actor network and the Critic network respectively, the spatiotemporal features are used to guide the action policy generation of the Actor network and the action policy update of the Critic network, characterizing the impact of the action executed at the current time step on the action executed in subsequent time steps; The threshold-based scene switching training strategy uses the training sample set to train the maritime search path planning model until the loss value of the maritime search path planning model is less than a preset loss threshold, thus obtaining the trained maritime search path planning model.

2. The training method according to claim 1, characterized in that, The training sample set constructed based on the Gaussian mixture probability distribution includes: The local search success rate distribution of the search target at each time step is simulated based on a two-dimensional Gaussian function, and the local search success rate distributions of multiple time steps are superimposed to obtain the global search success rate distribution of the search target during the search period. This global search success rate distribution is used as a training sample. The covariance matrix in the two-dimensional Gaussian function is adjusted multiple times to obtain the multiple training samples.

3. The training method according to claim 2, characterized in that, The expression for the training samples is in, This represents the global search success rate distribution. Indicates the grid cells in the search area. , Indicates the position of the grid cell. Indicates the first Distribution of local search success rates at each time step , This represents the total number of time steps, indicating the search period. The covariance matrix represents the shape and orientation of the search success rate distribution. Indicates the first The mean vector of the Gaussian distribution corresponding to each time step corresponds to the center point of the search region. These represent the variances of the search region horizontally and vertically, respectively. The variance represents the covariance in the horizontal and vertical directions. The variances are randomly selected between 0 and 1 to simulate various distribution scenarios.

4. The training method according to claim 3, characterized in that, The spatiotemporal perception module is a long short-term memory network; The extraction of spatiotemporal features of the agent's state sequence includes: Obtain the state sequence of the search agent at multiple time steps; The state sequence is input into a Long Short-Term Memory (LSTM) network and passed sequentially through a forget gate, an input gate, and an output gate. The output gate outputs the spatiotemporal features. The forget gate filters redundant noise, including searched regions with no search value (POS) and regions with a POS of 0, as well as distant grid cells irrelevant to the current decision. The input gate retains key information, including decision patterns in historical paths. The output gate adjusts the weights of forgetting and retaining. The spatiotemporal features include the grid cell search order, remaining search time, and the agent's spatial location.

5. The training method according to claim 4, characterized in that, The scene switching training strategy includes: Step 1: Use a commercial solver to find the optimal path in all search scenarios, calculate the first cumulative search success rate corresponding to the optimal path, and construct a scenario switching threshold based on the first cumulative search success rate. Step II: Obtain the action policy generated by the Actor network in the current round; Step III: Calculate the second cumulative search success rate of the action strategy in the search scenario corresponding to the current training sample; Step IV: If the second cumulative search success rate is greater than the scene switching threshold, then train the Actor network using other training samples besides the current training samples; otherwise, store the second cumulative search success rate in the experience replay pool, update the Actor network by sampling batches from the experience replay pool to obtain a new Actor network, and obtain the action policy generated by the new Actor network in the current round, then return to execute step III.

6. The training method according to claim 5, characterized in that, The update step size of the Actor network is determined based on the probability ratio of the new action policy to the old action policy.

7. The training method according to claim 6, characterized in that, The loss values ​​mentioned include strategy loss and evaluation loss; The expression for the policy loss is: in, The model parameters of the Actor network are: Strategy loss at that time The advantage function is the action value function. and state value function The difference, This indicates that the current action is better than average, and the probability of selecting this action should be increased. This indicates that the current action is worse than average, and the probability of selecting this action should be reduced. Used to evaluate in Take in state The expected cumulative reward of an action. Expressing expectations, Represents the trajectory. Representation strategy, Indicates the discount factor. Indicates time step Instant rewards Indicates the total time steps. Used for evaluation The quality of a state is characterized by the expected value of the value function of all possible actions that can be taken in that state. This represents the probability ratio between the new action strategy and the old action strategy. Represents the cutoff coefficient, used to... Limited to to Within a certain range, prevent the Actor network from updating too much; The expression for the evaluation loss is: in, The model parameters of the Critic network are: The evaluation loss at that time The model parameters of the Critic network are: Predicting the value function of the current state. This represents the target value of the time-series difference. , This represents the immediate reward at the current time step. Indicates the state at the next time step. The projected value, This represents the discount factor, used to balance the weight between immediate rewards and rewards in the next time step.

8. A method for planning a search path at sea, characterized in that, The search path planning method uses a model trained by any one of the training methods in claims 1-7, and uses the model to plan a search path at sea to obtain the search path planning result.

9. The maritime search path planning method according to claim 8, characterized in that, The process of using the model to plan maritime search routes and obtaining the maritime search route planning results includes: Obtain environmental information of the search area and parameter information of the agents participating in the search; The search area is rasterized based on the environmental information and the parameter information to obtain multiple raster units; The search success rate of each grid cell is calculated based on the drift prediction framework; The multiple grid cells, the search success rate, and the parameter information are input into the model to obtain the marine search path planning result.

10. The maritime search path planning method according to claim 9, characterized in that, The search area is rasterized based on the environmental information and the parameter information to obtain multiple raster units, including: Calculate the effective search radius based on the environmental information and the parameter information; The search area is rasterized according to the effective search radius to obtain the plurality of raster units.

Images