Unmanned ship multipath optimization method for large-scale dynamic search and rescue tasks

Abstract

The invention discloses an unmanned ship multi-path optimization method for large-scale dynamic search and rescue tasks, and belongs to the technical field of unmanned ship path planning. According to the method, a multi-track optimization framework combined with a deep reinforcement learning algorithm is introduced, so that the efficiency and accuracy of path planning are effectively improved. Through a multi-trajectory sampling technology, the problem of trajectory data sampling deviation in a traditional method is solved, and the robustness and adaptability of an algorithm in a dynamic rescue environment are enhanced. Meanwhile, an encoder-decoder structure is designed, path exploration of the unmanned ship is optimized by utilizing reinforcement learning, and the exploration capability of the model to a complex solution space is greatly improved. According to the method, the optimal searching path can be quickly found, and the unmanned ship is prevented from falling into a local optimal solution. According to the invention, the unmanned ship can intelligently adapt to a large-scale emergency rescue scene, the search time is greatly shortened, the task success rate is improved, and key technical support is provided for the application of the unmanned ship in emergency rescue.

Description

Technical Field

[0001] The present invention relates to the technical fields of autonomous unmanned systems, artificial intelligence, navigation, etc., and in particular to the field of unmanned ship path planning, and specifically to a multi-path optimization method for unmanned ships for large-scale dynamic search and rescue missions. Background Art

[0002] In emergency rescue missions, time urgency determines the efficiency of search and rescue, which directly affects the safety and protection of life and property. Unmanned boats have been widely used in search and rescue missions in complex environments such as oceans and rivers due to their advantages of autonomous navigation, high flexibility, and convenient operation. In particular, they can assist search and rescue personnel in quickly locating in dangerous environments. However, with the expansion of the scale of rescue missions and the increase in environmental uncertainty, traditional search and path planning methods have gradually shown problems such as low operating efficiency and difficulty in coping with dynamic changes in complex environments. Especially in large-scale search scenarios, traditional methods take a long time and cannot quickly adapt to environmental changes and the dynamic position of search and rescue objects.

[0003] At present, the path planning of unmanned boats mostly relies on known environments or preset rules. It performs well in smaller-scale rescue missions, but it is incapable of coping with complex and changeable rescue missions. In addition, there is a data bias problem in traditional trajectory sampling technology, which further reduces the search efficiency and increases the time and difficulty of mission execution. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides a multi-path optimization method for unmanned ships in large-scale dynamic search and rescue missions. The present invention combines the deep reinforcement learning (DRL) algorithm with the multi-track sampling technology. By introducing the multi-track sampling technology, the present invention significantly reduces the deviation in data sampling and improves the efficiency and accuracy of track sampling. At the same time, using the deep reinforcement learning algorithm, the unmanned ship can continuously learn and optimize decisions in a complex dynamic environment, so as to quickly adapt to environmental changes and find the global optimal path.

[0005] The technical means adopted by the present invention are as follows:

[0006] A multi-path optimization method for unmanned ships in large-scale dynamic search and rescue missions comprises the following steps:

[0007] Step 1: Construct a Markov decision model based on a deep reinforcement learning algorithm to find the shortest path for maritime rescue. The Markov decision model includes state elements, action elements, reward elements, and state transition probability elements. Based on the multi-trajectory sampling technology, the trajectory data of the unmanned ship in the search mission is dynamically sampled. The multi-trajectory sampling technology explores from different starting points and samples several complete trajectories through Monte Carlo.

[0008] Step 2: Construct an encoder-decoder structure and combine it with a deep reinforcement learning algorithm to train the Markov decision model, which enhances the model's ability to explore the solution space, generates random training samples for the search area, and performs multi-path optimization of the unmanned ship based on the trained encoder-decoder structure.

[0009] Furthermore, the Markov decision model includes:

[0010] State: the arrangement relationship a of the two-dimensional path nodes visited before time T 1:T-1 for:

[0011]

[0012] Among them, 1,…,N represents the node number range, a t represents the node selected for action at time t, ∪{a t} represents the union of ordered nodes, Ψ represents the loop trajectory. Since each node in the traveling salesman problem can only be passed once, different nodes can only be selected at different times. Therefore, the node a(t1) at time t1 is different from the node a(t2) at time t2. The arrangement relationship of the nodes represents the path;

[0013] Action: Select the node target a at time T from the unvisited nodes T for:

[0014] a T ∈{{1,…,N}\{a 1:T-1}}

[0015] Among them, a 1:T-1 represents the path state at time T, {1,…,N}\{a 1:T-1} means excluding the set of visited nodes in the full set of nodes {1,…,N} 1:T-1};

[0016] Reward: The reward R for arranging nodes to form the current loop is:

[0017]

[0018] Among them, L(Ψ|s) represents the total path length, s represents the set of N nodes in the plane, and x(a t ) represents the coordinates of node at, ||x(a t+1 )-x(a t )||2 represents the distance from the starting point to the adjacent nodes on the path, ||x(a N )-x(a1)||2 represents the distance from the end point to the starting point;

[0019] State transition probability: probability P calculated according to trajectory Ψ θ(Ψ|s) is:

[0020]

[0021] where a 1:t-1 represents the path status before time t, P θ (a t |a 1:t-1 ,s) means selecting node a t The state transition probability, ΠP θ represents the cumulative multiplication of single-step probabilities, and θ represents the parameters of the policy network.

[0022] Furthermore, the multi-trajectory sampling technique explores from multiple initial points and samples M complete trajectories through Monte Carlo, where a single trajectory is:

[0023]

[0024] Where 1, 2, ..., M represents the trajectory number, and each trajectory is the arrangement relationship of N nodes. represents the i-th trajectory Ψ i The selected initial point; in the traveling salesman problem with N nodes, the number of each node in the trajectory is selected through a series of actions. Each trajectory consists of N actions, and each trajectory has a different initial point, where the i-th trajectory is represented by Ψ i , the j-th action of the ith trajectory is expressed as

[0025] Furthermore, the encoder-decoder structure includes at least two encoders and one decoder. The encoder inputs the node information at time T, aggregates the node features using a multi-head attention network of L layers, and outputs the graph features at time T. The encoder has 1 embedding layer and L attention layers. The embedding layer transforms the two-dimensional features of the node information into high-dimensional features through linear mapping.

[0026] Furthermore, the PG algorithm is used for model training. PG obtains trajectory data through sampling, optimizes the policy network based on the interaction between actions and the environment, learns and updates the solution strategy of the traveling salesman problem, calculates the reward value based on the path length of each trajectory, and then calculates the total reward value R(Ψ i ), the ultimate goal of the algorithm is to maximize the expectation of the total reward value;

[0027] During the training process, the policy network of the model is optimized using unbiased estimation of gradient learning. The policy network with parameter θ outputs probability p θ for:

[0028]

[0029] Among them, s represents the N serial numbers of the nodes, Ψ i represents the ith complete trajectory, represents the action sequence before time t, Indicates the output action at time t The single-step state transition probability, Πp θ Represents the cumulative multiplication result of the single-step probability, p θ (Ψ i |s) represents the trajectory Ψ i The probability of output.

[0030] Compared with the prior art, the present invention has the following advantages:

[0031] The present invention aims at the problem that the traditional search method has low operating efficiency due to the expansion of the search scale in large-scale emergency rescue missions of unmanned ships, and proposes an unmanned ship marine rescue method based on multi-trajectory strategy optimization. With the increasing application of unmanned ships in marine rescue, traditional path planning and search methods are difficult to adapt to complex and dynamic task scenes, and there are problems of long search time and low efficiency. To this end, the present invention innovatively introduces a multi-trajectory optimization framework combined with a deep reinforcement learning algorithm, which effectively improves the efficiency and accuracy of path planning. Through the multi-trajectory sampling technology, the problem of trajectory data sampling deviation in the traditional method is solved, and the robustness and adaptability of the algorithm in a dynamic rescue environment are enhanced. At the same time, the present invention designs an encoder-decoder structure, which optimizes the path exploration of the unmanned ship by using reinforcement learning, and greatly improves the model's ability to explore complex solution spaces. The method can quickly find the optimal search path, avoid the unmanned ship from falling into a local optimal solution, thereby significantly improving the execution efficiency of the search task and reducing the loss of life and property. Through the implementation of the present invention, the unmanned ship can intelligently adapt to large-scale emergency rescue scenarios, greatly shorten the search time, and improve the success rate of the task, providing key technical support for the application of unmanned ships in emergency rescue, and has important practical significance and application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0033] Figure 1 It is a schematic diagram of the action process of single track and multi-track of the present invention.

[0034] Figure 2 This is a flowchart of the multi-trajectory strategy optimization method of the present invention.

[0035] Figure 3 This is the overall framework of the multi-path optimization method for unmanned boats for large-scale dynamic search and rescue missions of the present invention. DETAILED DESCRIPTION

[0036] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0037] The present invention discloses a multi-path optimization method for an unmanned ship for a large-scale dynamic search and rescue mission, comprising the following steps:

[0038] Step 1: In order to solve the problem that the traditional path planning method has a long running time and low efficiency due to the expansion of the scale of the unmanned ship search mission, the present invention proposes a Markov decision model based on a deep reinforcement learning algorithm. By introducing multi-trajectory sampling technology, the trajectory data of the unmanned ship in the search mission is dynamically sampled to ensure the accuracy and diversity of the sampling. This technology significantly improves the efficiency and accuracy of path planning by reducing sampling deviations, enabling unmanned ships to quickly generate optimal paths in large-scale maritime rescue missions, shorten search time, and effectively improve mission execution efficiency.

[0039] The specific steps include:

[0040] Step 1.1 In order to achieve full coverage of the maritime rescue area, the present invention conducts research on maritime rescue path planning for unmanned ships. In order to find the shortest path for maritime rescue, the process of executing the shortest path is modeled as a Markov decision process of the traveling salesman problem, and the path optimization process mainly includes the following elements.

[0041] State: the arrangement relationship a of the two-dimensional path nodes (referred to as nodes) visited before time T 1:T-1 , as shown below. Among them, 1, L, N represent the node number range, a t Represents the node selected for action at time t, U{a t} represents the union of ordered nodes, and Ψ represents the loop trajectory. Since each node in the traveling salesman problem can only be passed once, different nodes can only be selected at different times. Therefore, the node a(t1) at time t1 is different from the node a(t2) at time t2, and the arrangement relationship of the nodes is used to represent the path.

[0042]

[0043] Action: Select the node target a at time T from the unvisited nodes T , as shown below. Among them, a 1:T-1 represents the path status at time T, {1,L,N}\{a 1:T-1} means excluding the set of visited nodes in the full set of nodes {1,L,N} 1:T-1}.

[0044] a T ∈{{1,L,N}\{a 1:T-1}}(2)

[0045] Reward: The reward R for arranging nodes to form the current loop is as follows. L(Ψ|s) represents the total path length, s represents the set of N nodes in the plane, and x(a t ) represents node a t The coordinates of ||x(a t+1 )-x(a t )||2 represents the distance from the starting point to the adjacent nodes on the path, ||x(a N )-x(a1)||2 represents the distance from the end point to the starting point.

[0046]

[0047] State transition probability: probability P calculated according to trajectory Ψ θ (Ψ|s), as shown below. 1:t-1 represents the path status before time t, P θ (a t |a 1:t-1 ,s) means selecting node a t The state transition probability, ΠP θ represents the cumulative multiplication of single-step probabilities, and θ represents the parameters of the policy network.

[0048]

[0049] Step 1.2 The process of the unmanned boat performing the maritime rescue mission can be equivalent to solving the problem based on the minimum closed path of the traveling salesman. To this end, the present invention introduces a multi-trajectory sampling technology that can make full use of symmetry. The multi-trajectory sampling technology explores from multiple initial points and samples M complete trajectories through Monte Carlo. The single trajectory is shown below. Where 1, 2, L, M represent the trajectory numbers, and each trajectory is the arrangement relationship of N nodes; represents the i-th trajectory Ψ i Select the initial point.

[0050]

[0051] The single-track and multi-track action process is shown in Figure 1In the traveling salesman problem with N nodes, the number of each node in the trajectory is selected through a series of actions, and each trajectory consists of N actions. Each trajectory has a different initial point. The i-th trajectory is represented by Ψ i , the j-th action of the ith trajectory is expressed as

[0052] Step 2: An encoder-decoder structure was designed and trained with a deep reinforcement learning algorithm to enhance the model's ability to explore the solution space. Through the optimization of this structure, the unmanned ship is prevented from falling into a local minimum in path planning, thereby ensuring that the optimal feasible path is found. This method can adapt to complex changes in the marine environment, enabling the unmanned ship to quickly complete path planning in marine rescue missions, significantly improving search efficiency, reducing loss of life and property, and ensuring the success rate of the mission.

[0053] This step is used to generate random training samples for the search area, and select data enhancement technology for data preprocessing according to application requirements. Specifically, the training samples are randomly generated points in the interval [0,1], representing multiple discrete nodes (i.e., rescue targets or locations) in the unmanned boat rescue mission. Restore these discrete nodes and reset the environment, obtain multiple trajectory data through multi-trajectory sampling, and input the data features into the network model; Based on the encoder, extract the problem features from the data features through the embedding layer, use L attention layers (multi-head attention layer and feedforward network layer) to encode the problem features, obtain the relationship between the nodes in the graph, output the graph encoding information, and involve the node mask information; Based on the decoder, according to the graph encoding information of multiple trajectories, use the multi-head attention layer, residual layer and single-head attention layer to obtain the transfer probability distribution of the remaining nodes, and use the probability distribution to update the parameters of the network model; Based on the policy gradient (PG) algorithm, greedily select the next path node according to the node probability distribution and the result of the interaction between the learning agent and the environment; Determine whether a loop is formed. If no loop is formed, feed the existing information back to the encoder, otherwise calculate the path length.

[0054] The specific steps include:

[0055] Step 2.1 Construct a multi-trajectory strategy optimization method. When performing a maritime rescue mission, the unmanned ship sails from the starting point according to the mission instructions, and tries to choose a shortest navigation path for maritime rescue. After all sub-target information is collected, it returns to the starting point to ensure that emergency personnel can obtain the rescue location in time and ensure that the rescue operation is carried out in a timely and orderly manner. The process of constructing the shortest path may produce multiple different route trajectories. Therefore, the invention introduces multi-trajectory sampling technology and designs a multi-path optimization method for unmanned ships based on large-scale dynamic search and rescue missions, such as Figure 2As shown. This method generates multiple trajectories with equal weights starting from the initial point to ensure that the information entropy of the initial point is maximized, enrich and optimize the decision-making mechanism of the model, and avoid generating biased decision-making behaviors. Its core content includes network model (encoding and decoding framework), policy gradient algorithm, and data enhancement technology. The steps of this method are executed in order as follows.

[0056] Step 2.2 Network model structure construction. The network model of the present invention adopts a coding and decoding framework, which is composed of multiple encoders and a decoder. The encoder inputs the node information at time T, aggregates the node features using the L-layer multi-head attention network, and outputs the graph features at time T. The encoder has 1 embedding layer and L attention layers. The embedding layer transforms the two-dimensional features of the node information into high-dimensional features through linear mapping, and the dimension d is 128. The mapping transformation is shown below. Where represents the initial problem features output by the embedding layer, x i represents the data characteristics of node i, W x and b x Represents the network parameters of the embedding layer. Different attention layers do not share parameters. Each attention layer consists of a multi-head attention sub-layer and a feed-forward network sub-layer. The multi-head attention sub-layer can calculate the weight of each attention head, merge the node information of all attention heads in this sub-layer, and output it to the feed-forward network sub-layer for processing.

[0057]

[0058] The function of the decoder is to input graph features and mask information, extract the query information Q of the visited nodes at the current moment through the multi-head attention sublayer, and distribute the pointing information K and node information V of the unvisited nodes to different attention heads to achieve parallel computing. The graph information output by the multi-head attention sublayer is strengthened through the residual sublayer, and its final probability distribution is calculated through the single-head attention sublayer. The decoder has only one attention layer, which consists of a multi-head attention sublayer, a residual sublayer based on a multi-layer perceptron, and a single-head attention sublayer. The multi-head attention sublayer assigns the unvisited nodes to each attention head according to the mask information, and quickly calculates the similarity of each node j to be visited in combination with the query information.

[0059] Step 2.3: Train the model using the PG algorithm. PG obtains trajectory data by sampling, optimizes the policy network based on the interaction between actions and the environment, and learns and updates the solution strategy of the traveling salesman problem using the policy gradient of the loss function. The reward value is calculated based on the path length of each trajectory, and then the total reward value R(Ψ) of multiple trajectories of the instance is calculated. i ), the ultimate goal of the algorithm is to maximize the expectation of the total reward value. During the training process, the present invention uses the policy network of the unbiased estimation optimization model of gradient learning, and the policy network output probability p with parameter θ θ, as shown below. s represents the N sequence numbers of the nodes, Ψ i represents the ith complete trajectory, represents the action sequence before time t, Indicates the output action at time t The single-step state transition probability, Πp θ Represents the cumulative multiplication result of the single-step probability, p θ (Ψ i |s) represents the trajectory Ψ i The probability of output.

[0060]

[0061] Through the execution of step 2 above, the environment will continue to interact with the environment according to the action, and use the policy gradient of the cross loss function to optimize the policy network, learn and update the solution strategy of the traveling salesman problem, such as Figure 3 shown.

[0062] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-path optimization method for unmanned ships in large-scale dynamic search and rescue missions, characterized in that: The following steps are involved: Step 1: Construct a Markov decision model based on a deep reinforcement learning algorithm to find the shortest path for maritime rescue. The Markov decision model includes state elements, action elements, reward elements, and state transition probability elements. Based on the multi-trajectory sampling technology, the trajectory data of the unmanned ship in the search mission is dynamically sampled. The multi-trajectory sampling technology explores from different starting points and samples several complete trajectories through Monte Carlo. Step 2: Construct an encoder-decoder structure and combine it with a deep reinforcement learning algorithm to train the Markov decision model, which enhances the model's ability to explore the solution space, generates random training samples for the search area, and performs multi-path optimization of the unmanned ship based on the trained encoder-decoder structure.

2. The multi-path optimization method for large-scale dynamic search and rescue mission unmanned boat according to claim 1 is characterized in that: The Markov decision model includes: State: the arrangement relationship a of the two-dimensional path nodes visited before time T 1:T-1 for: Among them, 1,…,N represents the node number range, a t represents the node selected for action at time t, ∪{a t } represents the union of ordered nodes, Ψ represents the loop trajectory. Since each node in the traveling salesman problem can only be passed once, different nodes can only be selected at different times. Therefore, the node a(t1) at time t1 is different from the node a(t2) at time t2. The arrangement relationship of the nodes represents the path; Action: Select the node target a at time T from the unvisited nodes T for: a T ∈{{1,…,N}\{a 1:T-1 }} Among them, a 1:T-1 represents the path state at time T, {1,…,N}\{a 1:T-1 } means excluding the set of visited nodes in the full set of nodes {1,…,N} 1:T-1 }; Reward: The reward R for arranging nodes to form the current loop is: Among them, L(Ψ|s) represents the total path length, s represents the set of N nodes in the plane, and x(a t ) represents the coordinates of node at, ||x(a t+1 )-x(a t )||2 represents the distance from the starting point to the adjacent nodes on the path, ||x(a N )-x(a1)||2 represents the distance from the end point to the starting point; State transition probability: probability P calculated according to trajectory Ψ θ (Ψ|s) is: where a 1:t-1 represents the path status before time t, P θ (a t |a 1:t-1 ,s) means selecting node a t The state transition probability, ΠP θ represents the cumulative multiplication of single-step probabilities, and θ represents the parameters of the policy network.

3. The multi-path optimization method for unmanned ships in large-scale dynamic search and rescue missions according to claim 1 is characterized in that: The multi-trajectory sampling technique explores from multiple initial points and samples M complete trajectories through Monte Carlo, where a single trajectory is: Where 1, 2, ..., M represents the trajectory number, and each trajectory is the arrangement relationship of N nodes. represents the i-th trajectory Ψ i The selected initial point; in the traveling salesman problem with N nodes, the number of each node in the trajectory is selected through a series of actions. Each trajectory consists of N actions, and each trajectory has a different initial point, where the i-th trajectory is represented by Ψ i , the j-th action of the ith trajectory is expressed as 4. The multi-path optimization method for large-scale dynamic search and rescue mission unmanned ship according to claim 1 is characterized in that: The encoder-decoder structure includes at least two encoders and one decoder. The encoder inputs the node information at time T, aggregates the node features using the L-layer multi-head attention network, and outputs the graph features at time T. The encoder has 1 embedding layer and L attention layers. The embedding layer transforms the two-dimensional features of the node information into high-dimensional features through linear mapping.

5. The multi-path optimization method for unmanned ships in large-scale dynamic search and rescue missions according to claim 1 is characterized in that: The model is trained through the PG algorithm. PG obtains trajectory data through sampling, optimizes the policy network based on the interaction between actions and the environment, learns and updates the solution strategy of the traveling salesman problem, calculates the reward value based on the path length of each trajectory, and then calculates the total reward value R(Ψ) of multiple trajectories. i ), the ultimate goal of the algorithm is to maximize the expectation of the total reward value; During the training process, the policy network of the model is optimized using unbiased estimation of gradient learning. The policy network with parameter θ outputs probability p θ for: Among them, s represents the N serial numbers of the nodes, Ψ i represents the ith complete trajectory, represents the action sequence before time t, Indicates the output action at time t The single-step state transition probability, Πp θ Represents the cumulative multiplication result of the single-step probability, p θ (Ψ i |s) represents the trajectory Ψ i The probability of output.

Images