A path planning method for offshore unmanned vehicle cluster based on TD-MA TD3

Abstract

This invention discloses a path planning method for maritime unmanned surface vessel (USV) swarms based on TD-MATD3. The method includes: decomposing the USV swarm path planning task into a navigation task module and an obstacle avoidance task module, and designing decomposed Actor-Critic networks for each; designing an optimization method based on formation-keeping rewards, which generates an additional distance reward signal by calculating the deviation between the actual relative distance between USVs and the expected optimal distance, and this signal participates in strategy optimization together with the navigation task reward and obstacle avoidance task reward; designing an experience replay mechanism based on learning potential, which calculates the learning potential score of samples using temporal difference error and instantaneous reward value, prioritizing samples with high learning potential scores for training; and combining the experience replay mechanism based on learning potential with the MATD3 algorithm to update the network parameters of the two task modules, ultimately generating the optimal path planning strategy for the USV swarm. This invention can generate the optimal path planning strategy for maritime USV swarms.

Description

Technical Field

[0001] This invention belongs to the field of unmanned surface vessel (USV) swarm path planning technology, specifically relating to a method for swarm path planning of unmanned surface vessels based on TD-MATD3. Background Technology

[0002] In recent years, artificial intelligence technology based on neural networks (NNs) has developed rapidly. It aims to simulate the learning process of the human brain to achieve feature extraction and intelligent processing of various types of information. Route planning is an important research direction in multi-agent reinforcement learning. Unmanned surface vessel (USV) swarms and other intelligent agents achieve autonomous navigation through flexible cooperation and adaptive planning, playing a crucial role in tasks such as marine environmental monitoring and collaborative disaster search and rescue. Traditional path planning methods have significant limitations, especially in handling dynamic environments. These limitations make the development of path planning methods capable of real-time environmental perception, rapid adaptation to changes, and efficient collaboration a key direction for solving the autonomous navigation challenges of USV swarms.

[0003] In recent years, with the rapid development of artificial intelligence technology, more and more scholars have begun to explore learning-based methods to solve path planning problems. Reinforcement Learning (RL), as an important branch of this field, has achieved certain results in solving path planning problems due to its powerful feature extraction capabilities and highly automated data processing flow. RL directly optimizes strategies through trial and error iteration with the environment, without requiring a large amount of prior knowledge. Especially in situations where complete environmental information is lacking or the environment is constantly changing, RL can gradually generate optimal path planning strategies through autonomous learning.

[0004] However, traditional RL methods are often limited by the dimensionality of the action space, making it difficult to handle complex, high-dimensional dynamic environments. Furthermore, RL algorithms have limitations in their scalability in multi-agent scenarios, particularly when dealing with cooperation and competition among multiple agents, where they often struggle to provide efficient solutions. Summary of the Invention

[0005] The purpose of this invention is to provide a path planning method for unmanned surface vessel (USV) swarms based on TD-MATD3. This method combines the concept of task decomposition with the MATD3 algorithm and designs an optimization method based on formation maintenance rewards and an experience playback mechanism based on learning potential to achieve path planning for USV swarms in dynamic and complex environments.

[0006] The technical solution adopted in this invention is as follows: The first aspect of the present invention provides a path planning method for a swarm of unmanned surface vessels (USVs) based on TD-MATD3, the method comprising: The path planning task for the unmanned surface vessel (USV) swarm is decomposed into two independent task modules: a navigation task module and an obstacle avoidance task module. The navigation task module guides the USVs to move to the target location, while the obstacle avoidance task module prevents the USVs from colliding with obstacles and other USVs. Decomposed Actor-Critic networks are designed based on the navigation task module and the obstacle avoidance task module respectively. The Actor-Critic network of the navigation task module receives environmental information related to the target and outputs navigation features. The Actor-Critic network of the obstacle avoidance task module receives the navigation features and fuses environmental information related to obstacles and other unmanned surface vessels to output the path planning strategy of the unmanned surface vessel. An optimization method based on formation-keeping rewards is designed. An additional distance reward signal is generated by calculating the deviation between the actual relative distance between unmanned surface vessels and the expected optimal distance. This signal is then used as part of the navigation mission reward to participate in strategy optimization. Acquire environmental information related to the target, as well as environmental information related to obstacles and other unmanned surface vessels; Environmental information related to the target is input into the Actor-Critic network of the navigation task module; We design an experience replay mechanism based on learning potential, which calculates the learning potential score of samples by using temporal difference error and instant reward value, so as to prioritize the selection of samples with high learning potential scores for training, and uses the Q value to avoid overestimation to update the network parameters of the Actor-Critic network of the navigation task module in order to extract high-quality navigation features. High-quality navigation features and environmental information related to obstacles and other unmanned surface vessels are input into the Actor-Critic network of the obstacle avoidance task module; By combining the experience replay mechanism based on learning potential with the MATD3 algorithm, the network parameters of the Actor-Critic network of the obstacle avoidance task module are updated using the Q value to avoid overestimation, thereby generating the optimal path planning strategy for the unmanned surface vessel swarm.

[0007] The optimization method based on formation maintenance rewards in the above scheme includes: Set a reward function for the actual distance between unmanned surface vessels and the expected distance:

[0008] In the formula, This is the distance reward value; For unmanned surface vessels and unmanned surface vessels The actual relative distance between them; For unmanned surface vessels and unmanned surface vessels The expected optimal distance between them; Distance bonus is part of the navigation mission reward.

[0009] The above scheme includes an experience playback mechanism based on learning potential, which includes: Calculate time difference error ; Get instant reward value This includes all rewards from the navigation task module and the obstacle avoidance task module; Based on time difference error and instant reward value The learning potential score of the sample is calculated using weighted averages.

[0010] In the formula, Score the learning potential of the sample; As the weight of the time difference error, As the weight of the instant reward value, and All are positive and ; This is the bias term, a very small positive value, to prevent time difference errors. and instant reward value The learning potential score for all zero-time samples is zero; Based on the learning potential scores of the samples, samples with high learning potential scores are selected for training.

[0011] In the above scheme, the weight of time difference error The weight of the instant reward value is 0.6. It is 0.4.

[0012] In the above scheme, a periodic recalculation strategy is set for the experience playback mechanism based on learning potential. That is, every certain number of training rounds, the temporal difference error of all or some samples is recalculated and their learning potential scores are updated.

[0013] In the above scheme, the time difference error is the difference between the estimated value of the value function or action value function of the current state and the estimated value of the value function or action value function of the next state, as well as the immediate reward value.

[0014] In the above scheme, the environmental information related to the target includes the real-time location of the unmanned surface vessel and the target location, and the environmental information related to obstacles and other unmanned surface vessels includes the location of obstacles and the location of other unmanned surface vessels in the cluster.

[0015] The second aspect of the present invention provides an unmanned surface vessel (USV) swarm that employs the steps of the TD-MATD3-based path planning method for maritime USV swarms as described in any one of the first aspects.

[0016] The third aspect of the present invention provides a computer device, comprising: a processor and a memory, wherein the memory stores a program or instructions executable on the processor, and the program or instructions, when executed by the processor, implement the steps of the TD-MATD3-based unmanned surface vessel swarm path planning method described in any one of the first aspects.

[0017] The fourth aspect of the present invention provides a computer-readable storage medium having a program or instructions stored thereon, which, when executed by a processor, implement the steps of the TD-MATD3-based unmanned surface vessel swarm path planning method described in any one of the first aspects.

[0018] Compared with the prior art, the present invention has the following advantages and beneficial effects: By combining the idea of ​​task decomposition with the MATD3 algorithm and designing a decomposed Actor-Critic structure, the complex behavior of multi-agent systems can be effectively coordinated, achieving high efficiency and safety in overall path planning.

[0019] A learning potential-based experience playback mechanism (LPER) is designed and combined with the MATD3 algorithm. This significantly improves the targeting of sample sampling, optimizes the convergence performance of the algorithm, and accelerates the learning process of the model in sparse reward environments. It provides a more efficient and reliable solution for path planning of unmanned surface vessel formations in dynamic and complex environments.

[0020] An optimization method for formation keeping reward (FMR) is designed. By dynamically adjusting and constraining the distance deviation, the FMR optimization method can guide the unmanned surface vessel to adaptively adjust its navigation position, thereby achieving precise control and stable maintenance of the formation. Attached Figure Description

[0021] Figure 1 A flowchart illustrating a method for swarm path planning of unmanned surface vessels based on TD-MATD3 provided in an embodiment of the present invention; Figure 2 This is a task decomposition structure diagram of a path planning method for a swarm of unmanned surface vessels based on TD-MATD3 provided in an embodiment of the present invention. Figure 3 A decomposed Actor-Critic network structure diagram based on TD-MATD3 is provided in this embodiment of the invention; Figure 4 A detailed structural diagram of a network model for a path planning model for a swarm of unmanned surface vessels based on TD-MATD3, provided in an embodiment of the present invention; Figure 5This is a simulation result diagram of unmanned surface vessel swarm route planning based on TD-MATD3, provided in an embodiment of the present invention. Figure 6 This is a schematic diagram of the hardware structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. All other embodiments obtained by those skilled in the art based on the embodiments provided by this invention without inventive effort are within the scope of protection of this invention.

[0023] Obviously, the accompanying drawings described below are merely some examples or embodiments of the present invention. Those skilled in the art can apply the present invention to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this invention, modifications to design, manufacturing, or production based on the technical content disclosed in this invention are merely conventional technical means and should not be construed as insufficient disclosure of the present invention.

[0024] In this invention, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this invention may be combined with other embodiments without conflict.

[0025] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "a," "an," "an," "the," and similar words used in this invention do not indicate quantity limitation and may indicate singular or plural. The terms "comprising," "including," "having," and any variations thereof used in this invention are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms "connected," "linked," "coupled," and similar words used in this invention are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "A plurality" used in this invention refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships may exist; for example, "A and / or B" can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects have an "or" relationship. The terms "first," "second," and "third" used in this invention are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0026] This invention provides a path planning method for a maritime unmanned surface vessel (USV) swarm based on TD-MATD3, comprising the following steps: Innovatively proposing to decompose the complex path planning problem into two independent task modules: a navigation task module and an obstacle avoidance task module; designing a decomposed Actor-Critic network structure based on the two independent task modules, wherein the Actor-Critic network of the navigation task module receives environmental information related to the target and outputs navigation features, and the Actor-Critic network of the obstacle avoidance task module receives the navigation features and fuses environmental information related to obstacles and other USVs to output a path planning strategy for the USVs; designing an FMR optimization method, which generates additional reward signals by calculating the deviation between the actual relative positions and desired positions of the USVs, and these signals participate in the strategy optimization process together with the rewards from the navigation and obstacle avoidance tasks. The system acquires obstacle locations, relevant environmental information, and the dynamic position information of other unmanned surface vessels (USVs) in the formation using a sensor system. Environmental information is used as input to the Actor and Critic networks of the navigation task module. A specially designed LPER mechanism is employed to improve the utilization rate of high-quality training samples, and Q-values ​​to avoid overestimation are used to update the network parameters of the Actor and Critic networks of the navigation task module, extracting high-quality navigation features. These high-quality navigation features, obstacle locations, and the dynamic position information of other USVs are used as input to the Actor and Critic networks of the obstacle avoidance task module. The LPER mechanism is combined with the MATD3 algorithm, and Q-values ​​to avoid overestimation are used to update the network parameters of the Actor and Critic networks of the obstacle avoidance task module, generating the optimal path planning strategy for the USV swarm. This invention innovatively proposes to combine the idea of ​​Task Decomposed (TD) with the MATD3 algorithm to realize path planning for unmanned surface vessel (USV) swarms. By decomposing the path planning problem and combining it with the MATD3 algorithm, and designing the FMR optimization method and LPER mechanism, compared with traditional path planning methods (such as MADDPG), the proposed method significantly alleviates the challenges caused by task complexity during the learning process, improves the overall performance and interpretability of the algorithm, and provides strong technical support for solving dynamic multi-agent path planning problems.

[0027] This invention provides a method for swarm path planning of unmanned surface vessels (USVs) based on TD-MATD3, such as... Figure 1 and Figure 2 As shown, the method includes the following steps: S1. Innovatively proposes to decompose the complex path planning problem into two independent task modules: a navigation task module and an obstacle avoidance task module.

[0028] The navigation module guides the unmanned surface vessel (USV) to the target location, while the obstacle avoidance module prevents the USV from colliding with obstacles or other USVs.

[0029] S2. Design a decomposed Actor-Critic network structure based on two independent task modules.

[0030] The Actor-Critic network of the navigation task module receives environmental information related to the target and outputs navigation features. The Actor-Critic network of the obstacle avoidance task module receives the navigation features and fuses environmental information related to obstacles and other unmanned surface vessels to output the path planning strategy of the unmanned surface vessel.

[0031] S3. Design an FMR optimization method that generates additional reward signals by calculating the deviation between the actual relative distance between unmanned surface vessels and the expected optimal distance, and participates in the strategy optimization process together with the rewards for navigation and obstacle avoidance tasks.

[0032] The FMR optimization method calculates the expected optimal distance between each pair of unmanned surface vessels (USVs) and the current actual distance, thereby generating a dynamic reward signal to ensure that the distance between each pair of USVs remains relatively stable. Specifically, in the reward design phase, the FMR optimization method is incorporated. By calculating the deviation between the actual relative distance between USVs and the expected formation distance, an additional reward signal is generated. This reward is then integrated with the rewards from navigation and obstacle avoidance tasks and participates in the optimization process of the policy network. This ensures that while completing the path planning task, a good formation structure and coordination are maintained.

[0033] S4. Obtain obstacle locations, relevant environmental information, and dynamic position information of other unmanned surface vessels in the formation through the sensor system.

[0034] The information acquired through the sensor system is processed and used for model training. Specifically, the Critic network input to the navigation task module consists of relevant environmental information acquired by the sensor system.

[0035] S5. Use environmental information as input to the Actor network and Critic network of the navigation task module.

[0036] S6. Improve the utilization rate of training samples by designing the LPER mechanism, and use Q-values ​​to avoid overestimation to update the network parameters of the Actor network and Critic network of the navigation task module, thereby extracting high-quality navigation features.

[0037] The navigation task module is responsible for guiding the unmanned surface vessel (USV) towards the target. It maximizes the efficiency of the USV reaching the target location based on the navigation task reward value and improves sample utilization using a designed LPER mechanism. Finally, the Actor network outputs high-quality navigation feature information. Specifically, during the training phase, the navigation task module's Critic network consists of two Critic networks. Through a designed Learning Potential Experience Replay (LPER) mechanism combined with the MATD3 algorithm, it prioritizes training samples with higher potential, effectively improving the utilization efficiency of training samples. Then, two target networks are used to update network parameters to avoid overestimating Q-values ​​and extract high-quality navigation features.

[0038] S7. High-quality navigation feature information, obstacle positions, and the dynamic position information of other unmanned surface vessels are used as inputs to the Actor network and Critic network of the obstacle avoidance task module.

[0039] The obstacle avoidance module takes high-quality navigation feature information, obstacle locations, and the dynamic positions of other unmanned surface vessels (USVs) as input. The module focuses on guiding USVs to avoid obstacles and other USVs to ensure navigational safety, and its reward value primarily measures the USV's performance in collision avoidance tasks.

[0040] S8. By combining the LPER mechanism with the MATD3 algorithm, the network parameters of the Actor network and Critic network of the obstacle avoidance task module are updated using the Q value to avoid overestimation, thereby generating the optimal path planning strategy for the unmanned surface vessel swarm.

[0041] During the training phase, the obstacle avoidance task module Critic network also consists of two Critic networks. The LPER mechanism is combined with the MATD3 algorithm. Two target networks are used to avoid overestimating the Q value to update the network parameters. The Actor network generates the optimal path planning strategy for the unmanned surface vessel.

[0042] Based on the above, this invention proposes a path planning method for unmanned surface vessel (USV) swarms based on TD-MATD3. It innovatively combines the concept of Task Decomposed (TD) with the MATD3 algorithm, and designs an FMR optimization method and LPER mechanism to construct a network model to achieve autonomous navigation of the USV swarm. By reasonably constructing the state space, action space, and reward function, it ensures that the multi-agent system can take into account both local and global optima, quickly generate route planning schemes, and help complete complex marine missions.

[0043] To more clearly illustrate the functional characteristics and structural parameters of the present invention, the following description is provided in conjunction with the accompanying drawings and specific embodiments. The proposed method is then compared with the commonly used path planning method MADDPG to verify the superiority of the proposed method. Finally, the effectiveness of the proposed method in unmanned surface vessel swarm path planning tasks is demonstrated.

[0044] In complex tasks, unmanned surface vessels (USVs) receive multifaceted observational information, making direct strategy optimization for the overall mission objective inefficient. In this invention, the path planning task can be viewed as a combination of two task modules: a navigation module, which guides the USV towards the target; and an obstacle avoidance module, which helps the USV avoid obstacles and collisions with other USVs. The navigation module is the primary module, and the obstacle avoidance module builds upon it to complete the entire path planning task.

[0045] For the navigation module, path planning is performed solely based on the UAV's tendency to move towards the target. Its input is environmental information relevant to the target, significantly reducing task complexity and facilitating efficient learning of the UAV's target-reaching skills. The obstacle avoidance module takes into account environmental information related to obstacles and other UAVs to avoid collisions. Furthermore, high-quality navigation features extracted from the navigation module are integrated into the obstacle avoidance module's network layer, providing clear direction and guiding the UAV to avoid obstacles on its journey to the target. Ultimately, through continuous interaction and learning with the environment, the UAV achieves the overall goal of reaching the target without collisions.

[0046] Based on the idea of ​​task decomposition, this invention proposes a TD-MATD3 algorithm, which decomposes the Actor-Critic structure according to the rewards of the two task modules mentioned above. For each Actor-Critic, the Critic is divided into a navigation module Critic_n and an obstacle avoidance module Critic_o, while the Actor is divided into a navigation module Actor and an obstacle avoidance module Actor. The navigation module is decomposed according to the reward value. The unmanned surface vessels are guided to maintain formation and approach the target, while the obstacle avoidance module determines the objective based on the reward value. Guide unmanned surface vessels to avoid obstacles and prevent collisions with other unmanned surface vessels.

[0047] The system processes the obstacle locations, relevant environmental information, and dynamic position information of other unmanned surface vessels in the formation obtained by the sensor system to construct the input data for the navigation and obstacle avoidance modules.

[0048] like Figure 3 and Figure 4As shown, in the decomposed Actor-Critic structure of the design, for each unmanned surface vessel, the Actor network consists of 5 fully-connected (FC) layers, with input nodes and output nodes being... , , , and For the Actor_o network of the unmanned surface vessel formation i, the navigation feature information of the Actor_t network is integrated into each layer of the Actor_o network. The final network structure consists of 5 fully connected (FC) layers, with input and output nodes as follows: , , , and The first four layers use ReLU for activation, and the last layer uses Tanh as the activation function to ensure that the action is within a feasible range.

[0049] Similarly, for each unmanned surface vessel (USV) swarm, the Critic network also consists of five fully connected (FC) layers, with input and output nodes being respectively... , , , and ,in This represents the joint state and actions of the navigation task module. For the Critic_o network of USV_i, the navigation feature information of the Critic_t network is integrated into each layer of the Critic_o network. The final network structure consists of 5 fully connected (FC) layers, with input and output nodes respectively. , , , and ,in This indicates the combined state and actions of the obstacle avoidance task module.

[0050] Maintaining formation stability is crucial for successful maritime reconnaissance missions by unmanned surface vessels (USVs). To address this issue, this invention proposes an optimization method based on the TD-MATD3 algorithm: Formation Maintenance Reward (FMR). This method designs a reward function that combines the expected optimal spacing between each pair of USVs with the current actual spacing, dynamically adjusting the reward signal to ensure relatively stable spacing between USVs and guide the USV formation to maintain a relatively stable formation.

[0051] During formation navigation, the formation is controlled to maintain a stable distance between each pair of unmanned surface vessels (USVs) at a reasonable level. Therefore, a distance-based approach is adopted to establish a framework for USV navigation. and unmanned surface vessels The reward function for the actual distance and the expected optimal distance between them:

[0052] In the formula, This is a distance reward value, which is part of the navigation mission reward. For unmanned surface vessels and unmanned surface vessels The actual relative distance between them; For unmanned surface vessels and unmanned surface vessels The expected optimal distance between them.

[0053] The reward value can be seen from the functional relationship. and It exhibits a quadratic function relationship. When When the value is 1, the quadratic function reaches its maximum value of 1, i.e., the unmanned surface vessel. and unmanned surface vessels When the actual distance and the expected optimal distance are equal, the reward value is... The maximum value is 1. The greater the deviation from 1, the more likely it is to be an unmanned surface vessel. and unmanned surface vessels The greater the deviation between the actual distance and the expected optimal distance, the lower the reward value. The smaller.

[0054] By dynamically adjusting and constraining distance deviations, the FMR optimization method can guide unmanned surface vessels (USVs) to adaptively adjust their navigation positions, thereby achieving precise control and stable maintenance of formation. This improvement not only effectively constrains the relative positions between USVs, ensuring that the formation meets mission requirements, but also further enhances the algorithm's stability and robustness, enabling the formation to maintain a specific formation structure even in complex environments. By incorporating FMR into a multi-agent reinforcement learning framework, the algorithm can implement formation constraints simultaneously with path planning, providing strong support for efficient collaboration and accurate reconnaissance of USV formations.

[0055] During the training phase of TD-MATD3, for unmanned surface vessel (USV) formation i, sufficient rewards are only awarded upon successfully reaching the target position. Prior to this, actions often lack clear guidance, resulting in limited training value for the training samples stored in the experience pool. To address this, a LearningPotential-based Experience Replay (LPER) mechanism was designed to quantitatively evaluate the importance of experience samples and assign a corresponding score to each sample. Since the temporal difference error and reward value of experience samples significantly influence their importance, these factors are selected as influencing factors. Finally, samples with higher potential are prioritized for training based on their scores, thereby effectively improving the utilization efficiency of training samples.

[0056] The time difference error represents the difference between the estimated value of the value function or action value function of the current state and the sum of the estimated values ​​of the value function or action value function of the next state (plus the actual reward). Its calculation formula is as follows:

[0057] in, This represents the actual reward the agent receives from the environment, including all rewards from the navigation task module and the obstacle avoidance task module. ; This represents the discount factor, used to measure the importance of future rewards; Indicates motion noise; Value functions or state-action pairs representing the current state The estimated value of the action-value function; Indicates the next state.

[0058] The reward value is an important input in reinforcement learning algorithms, used to evaluate the quality of the agent's behavior when interacting with the environment. Furthermore, the temporal difference error is relatively more important than the reward value in measuring the learning potential of experienced samples. A linear quantization function is designed to highlight the impact of the temporal difference error on sample priority, while also taking into account the important role of the reward value. The sample priority calculation formula is as follows:

[0059] in, This represents the learning potential score of a sample; This represents the time difference error of the sample; This represents the instantaneous reward value of the sample, which includes all rewards from the navigation task module and the obstacle avoidance task module; The bias term is a small positive number. Its purpose is to ensure that the model's learning potential is not zero when both the time difference error and the reward value are zero, so that the sample still has a chance to be sampled.

[0060] After the design was completed, the training process and experience pool structure of MATD3 were optimized to effectively embed LPER into the MATD3 algorithm. Specifically, the LPER mechanism plays a role in the experience sampling phase of TD-MATD3.

[0061] The experience samples generated by environmental interactions during each round of training. The data is stored in an experience pool structure, which, in addition to the basic data, also records the learning potential score for each sample. The score is determined by a weighted combination of temporal difference error and immediate reward value. During updates to the Actor and Critic networks, training batches are no longer uniformly sampled; instead, they are weighted based on the learning potential score of each sample. This ensures that key experiences with larger temporal difference errors or significant rewards are more frequently used in training, thereby strengthening the policy's ability to adjust at critical inflection points.

[0062] Furthermore, to avoid the "aging" problem of sample scores, the LPER mechanism employs a periodic recalculation strategy: every K training epochs, the temporal difference error of all or some samples in the experience pool is recalculated and the potential score is updated, ensuring that sample priority reflects the learning needs of the current policy network in real time. This mechanism not only improves the adaptability of experience replay but also effectively alleviates the sample estimation bias problem caused by the continuous evolution of the policy.

[0063] Then, noise is added to the target actions of the navigation task module and the obstacle avoidance task module respectively:

[0064]

[0065] like Figure 3 and Figure 4 As shown, the Critic network in the navigation task module contains two Critic networks. , The parameters are , and two Critic target networks , The parameters are , The objective Q-value of a navigation task can be represented by a reward. To calculate:

[0066] Two Critic Networks , Update using the smaller Q-value estimated by the two target networks:

[0067]

[0068]

[0069] The Critic network in the obstacle avoidance task module also includes two Critic networks. , The parameters are , and two Critic target networks , The parameters are , Target network and Use rewards and high-quality navigation feature information ,Depend on and Output to estimate the overall task objective Q value:

[0070] and The smaller Q-value update determined by the two target networks:

[0071]

[0072]

[0073] The Actor networks for both the navigation task module and the obstacle avoidance task module include Actor networks. , and Actor Target Network , The parameters are: of Depend on Q-value update for navigation task output from the network:

[0074] The parameters are of Depend on The Q-value of the navigation task output by the network is updated. High-quality navigation action characteristics output by the network Update:

[0075] The Actor network uses a delayed update strategy, meaning that it updates only once when the Critic network updates twice. and The target network parameters are updated as follows:

[0076]

[0077]

[0078]

[0079] The Critic_n and Actor_n networks in the navigation task module mutually constrain each other, updating and outputting more accurate navigation actions. Specifically, these two networks optimize the navigation strategy of the unmanned surface vessel (USV) formation through collaborative updates, ensuring that the formation can move efficiently towards the target direction. Subsequently, the high-quality navigation features extracted from the navigation task module are input into the corresponding obstacle avoidance task module to further generate the final execution actions of the USV.

[0080] MADDPG (Multi-Agent Deep Deterministic Policy Gradient) is a multi-agent deep reinforcement learning algorithm widely used in path planning problems. Table 1 shows the comparative experimental results between the proposed method and MADDPG. As can be seen from the table, the proposed method improves the path planning length by 61.19% compared to MADDPG within the same path planning time, significantly demonstrating the superiority of the proposed method.

[0081] Table 1. Comparative experimental results of TD-MATD3 and MADDPG

[0082] Figure 5 The figure shows the experimental results of the proposed method for unmanned surface vessel (USV) swarm route planning. As can be seen from the figure, the proposed method enables USVs to effectively avoid obstacles and quickly reach their destination.

[0083] In this example, the model can automatically adjust its parameters during training without requiring prior model setup or detailed parameter tuning, ensuring the generation of autonomous navigation strategies. Compared to traditional methods, this approach offers higher accuracy and avoids the hassle of manually setting hyperparameters beforehand. The model is developed using Python 3.8 and the accompanying PyTorch 2.0.0 deep learning development package, employing a TD-MATD3 path planning model based on task decomposition and MATD3 as the path planning generation scheme. In this embodiment, a dataset for unmanned surface vessel (USV) swarm path planning is generated based on a maritime USV swarm simulation model combined with expert knowledge. The sampling interval for each sample is 0.2 seconds, and the total simulation time can be set according to the task.

[0084] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0085] In addition, combined Figure 1 The TD-MATD3-based path planning method for swarms of unmanned surface vessels described in this application can be implemented by a computer device. Figure 6 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of this application. Figure 6 As shown, the device may include a processor 301 and a memory 302 storing computer program instructions.

[0086] Specifically, the processor 301 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0087] Memory 302 may include a large-capacity memory for data or instructions. For example, and not limitingly, memory 302 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), flash memory, an optical disk drive, a magneto-optical disk drive, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 302 may include removable or non-removable (or fixed) media. Where appropriate, memory 302 may be internal or external to a data processing device. In a particular embodiment, memory 302 is non-volatile memory. In a particular embodiment, memory 302 includes read-only memory (ROM) and random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable read-only memory (PROM), an erasable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), an electrically alterable read-only memory (EAROM), or flash memory, or a combination of two or more of these. Where appropriate, the RAM can be Static Random-Access Memory (SRAM) or Dynamic Random-Access Memory (DRAM). DRAM can be Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), Extended Data Out Dynamic Random-Access Memory (EDODRAM), Synchronous Dynamic Random-Access Memory (SDRAM), etc.

[0088] The memory 302 can be used to store or cache various data files that need to be processed and / or communicated, as well as possible computer program instructions executed by the processor 301.

[0089] The processor 301 reads and executes computer program instructions stored in the memory 302 to implement any of the TD-MATD3-based unmanned surface vessel swarm path planning methods in the above embodiments.

[0090] In some embodiments, the computer device may further include a communication interface 303 and a bus 300. For example, Figure 6 As shown, the processor 301, memory 302, and communication interface 303 are connected through bus 300 and complete communication with each other.

[0091] The communication interface 303 is used to enable communication between the various modules, devices, units, and / or equipment in the embodiments of this application. The communication interface 303 can also enable data communication with other components such as external devices, image / data acquisition devices, databases, external storage, and image / data processing workstations.

[0092] Bus 300 includes hardware, software, or both, that couples components of a computer device together. Bus 300 includes, but is not limited to, at least one of the following: data bus, address bus, control bus, expansion bus, and local bus. For example, and not as a limitation, bus 300 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 300 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, this application contemplates any suitable bus or interconnection.

[0093] The computer device can execute the TD-MATD3-based unmanned surface vessel swarm path planning method in the embodiments of this application, thereby achieving a combination of Figure 1 The paper describes a path planning method for unmanned surface vessels (USVs) swarms based on TD-MATD3.

[0094] Furthermore, in conjunction with the TD-MATD3-based path planning method for unmanned surface vessels (USVs) swarms described in the above embodiments, this application embodiment can provide a computer-readable storage medium for implementation. This computer-readable storage medium stores computer program instructions; when executed by a processor, these computer program instructions implement any of the TD-MATD3-based path planning methods for USVs swarms described in the above embodiments.

[0095] It should be noted that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification. In addition, depending on the implementation needs, the various steps / components described in this invention can be broken down into more steps / components, or two or more steps / components or parts of steps / components can be combined into new steps / components to achieve the purpose of this invention.

[0096] It will be readily understood by those skilled in the art that the above-described embodiments merely illustrate several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this patent should be determined by the appended claims.

Claims

1. A method for swarm path planning of unmanned surface vessels (USVs) based on TD-MATD3, characterized in that, The method includes: The path planning task for the unmanned surface vessel (USV) swarm is decomposed into two independent task modules: a navigation task module and an obstacle avoidance task module. The navigation task module guides the USVs to move to the target location, while the obstacle avoidance task module prevents the USVs from colliding with obstacles and other USVs. Decomposed Actor-Critic networks are designed based on the navigation task module and the obstacle avoidance task module respectively. The Actor-Critic network of the navigation task module receives environmental information related to the target and outputs navigation features. The Actor-Critic network of the obstacle avoidance task module receives the navigation features and fuses environmental information related to obstacles and other unmanned surface vessels to output the path planning strategy of the unmanned surface vessel. An optimization method based on formation-keeping rewards is designed. An additional distance reward signal is generated by calculating the deviation between the actual relative distance between unmanned surface vessels and the expected optimal distance. This signal is then used in conjunction with navigation task rewards and obstacle avoidance task rewards to optimize the strategy. Acquire environmental information related to the target, as well as environmental information related to obstacles and other unmanned surface vessels; Environmental information relevant to the target is input into the Actor-Critic network of the navigation task module; We design an experience replay mechanism based on learning potential, which calculates the learning potential score of samples by using temporal difference error and instant reward value, so as to prioritize the selection of samples with high learning potential scores for training, and uses the Q value to avoid overestimation to update the network parameters of the Actor-Critic network of the navigation task module in order to extract high-quality navigation features. High-quality navigation features and environmental information related to obstacles and other unmanned surface vessels are input into the Actor-Critic network of the obstacle avoidance task module; By combining the experience replay mechanism based on learning potential with the MATD3 algorithm, the network parameters of the Actor-Critic network of the obstacle avoidance task module are updated using the Q value to avoid overestimation, thereby generating the optimal path planning strategy for the unmanned surface vessel swarm.

2. The method for swarm path planning of unmanned surface vessels based on TD-MATD3 according to claim 1, characterized in that, Optimization methods based on formation retention rewards include: Set a reward function for the actual distance between unmanned surface vessels and the expected distance: In the formula, This is the distance reward value; For unmanned surface vessels and unmanned surface vessels The actual relative distance between them; For unmanned surface vessels and unmanned surface vessels The expected optimal distance between them; Distance bonus is part of the navigation mission reward.

3. The method for swarm path planning of unmanned surface vessels based on TD-MATD3 according to claim 1, characterized in that, Experience replay mechanisms based on learning potential include: Calculate time difference error ; Get instant reward value This includes rewards for navigation tasks and rewards for obstacle avoidance tasks; Based on time difference error and instant reward value The learning potential score of the sample is calculated using weighted averages. In the formula, Score the learning potential of the sample; As the weight of the time difference error, As the weight of the instant reward value, and All are positive and ; This is the bias term, a very small positive value, to prevent time difference errors. and instant reward value The learning potential score for all zero-time samples is zero; Based on the learning potential scores of the samples, samples with high learning potential scores are selected for training.

4. The method for swarm path planning of unmanned surface vessels based on TD-MATD3 according to claim 3, characterized in that, Weight of time difference error The weight of the instant reward value is 0.

6. It is 0.

4.

5. The method for swarm path planning of unmanned surface vessels based on TD-MATD3 according to claim 3, characterized in that, A periodic recalculation strategy is set for the experience playback mechanism based on learning potential, that is, every certain number of training rounds, the temporal difference error of all or some samples is recalculated and their learning potential scores are updated.

6. The method for swarm path planning of unmanned surface vessels based on TD-MATD3 according to claim 3, characterized in that, The time difference error is the difference between the estimated value of the value function or action value function of the current state and the estimated value of the value function or action value function of the next state, as well as the immediate reward value.

7. The method for swarm path planning of unmanned surface vessels based on TD-MATD3 according to claim 1, characterized in that, Environmental information related to the target includes the real-time location of the unmanned surface vessel (USV) and the target's location. Environmental information related to obstacles and other USVs includes the location of obstacles and the location of other USVs within the swarm.

8. A swarm of unmanned surface vessels, characterized in that, The unmanned surface vessel swarm adopts the steps of the TD-MATD3-based path planning method for marine unmanned surface vessel swarms as described in any one of claims 1 to 7.

9. A computer device, characterized in that, include: The processor and memory, wherein the memory stores programs or instructions that can run on the processor, and when the programs or instructions are executed by the processor, implement the steps of the TD-MATD3-based unmanned surface vessel swarm path planning method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores programs or instructions, which, when executed by a processor, implement the steps of the TD-MATD3-based unmanned surface vessel swarm path planning method as described in any one of claims 1 to 7.

Images