Unmanned ship closed-loop path planning method and system based on multi-agent potential field and deep Q learning

Abstract

The invention relates to the technical field of artificial intelligence, in particular to an unmanned ship closed-loop path planning method and system based on a multi-agent potential field and deep Q learning. According to the method, a multi-agent potential field including an attraction potential field and a repulsion potential field is constructed by using state information of multiple unmanned ships, and an action evaluation function is formed based on a potential field gradient and an epsilon-greedy strategy. And a single-boat target optimal action strategy is obtained by establishing a double-Q dynamic adjustment network model and combining deep Q learning iterative updating. On the basis, a collaborative strategy sharing mechanism of potential field gradient, position relation and state interaction among multiple agents is introduced, and cluster-level collaborative path planning is achieved. And finally, driving the unmanned ship to complete forward and return tasks according to a collaborative planning result, and generating a closed-loop optimal path in combination with a layered reward mechanism in a return stage. According to the invention, the autonomous decision-making capability, the whole-process autonomous navigation capability and the global cooperation efficiency of the unmanned ship cluster in a complex task sea area are improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method and system for closed-loop path planning of unmanned surface vessels based on multi-agent potential fields and deep Q-learning. Background Technology

[0002] In related technologies, with the rapid development of Unmanned Surface Vehicle (USV) technology in scenarios such as emergency search and rescue, maritime patrol, target reconnaissance, port and shipping management, and fleet collaborative operations, multi-agent path planning technology has gradually become one of the core foundations of maritime autonomous decision-making systems. In typical applications, USVs usually need to autonomously complete task point search, obstacle avoidance navigation, and swarm collaborative decision-making in unknown or semi-structured sea areas. However, most existing research focuses on one-time unidirectional path planning models from "start point to end point," with their objectives mainly concentrated on optimizing indicators such as shortest path, collision avoidance, or local environmental adaptability. These methods are relatively mature in scenarios such as port track generation and single task point planning, but they exhibit significant limitations in complex maritime tasks requiring "task execution—path regression—track closure."

[0003] However, USVs often face challenges such as dynamic obstacles, ocean current disturbances, multi-vehicle collision avoidance, and uneven distribution of mission points when performing missions in complex sea conditions. Traditional reinforcement learning path planning methods suffer from typical problems such as sparse rewards, slow convergence speed, and low sample utilization, making it difficult to form stable and effective strategies in complex environments. Existing collaborative mechanisms are also insufficient to solve path conflicts and overall efficiency optimization among multiple vessels in a shared environment. Furthermore, most existing technologies are geared towards "one-way path planning," that is, focusing only on generating the optimal path from the starting point to the designated mission point, lacking unified planning for the return path and the overall closed-loop trajectory after mission completion, making it difficult to meet the requirements of maritime missions for continuous path throughout the entire process and reliable return journey.

[0004] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0005] The main objective of this application is to propose a closed-loop path planning method and system for unmanned surface vessels (USVs) based on multi-agent potential fields and deep Q-learning, which can improve the autonomous decision-making capability, full-process autonomous navigation capability, and global collaborative efficiency of USV swarms in complex mission sea areas.

[0006] To achieve the above objectives, one aspect of this application proposes a closed-loop path planning method for unmanned surface vessels based on multi-agent potential fields and deep Q-learning, the method comprising: Acquire the status information of multiple unmanned surface vessels (USVs) clusters in the mission area; Based on the state information, an artificial potential field algorithm is used to construct a multi-agent potential field; the multi-agent potential field includes the target point attraction potential field and the obstacle repulsion potential field; Combining the multi-agent potential field, an action evaluation function is established using the artificial potential field gradient and the ε-greedy strategy; Construct a dual-Q dynamically adjusted network model; The action evaluation function and deep Q-learning algorithm are used to iteratively update the dual-Q dynamic adjustment network model and output the target optimal action strategy for a single unmanned surface vessel. The target optimal action strategy includes the outbound optimal action strategy and the return optimal action strategy. Based on the target optimal action strategy output by the dual-Q dynamic adjustment network model, and combined with the artificial potential field gradient, positional relationship and state interaction information of each surface unmanned vessel, a cluster collaborative strategy sharing mechanism is constructed to generate collaborative path planning decisions for multiple unmanned vessels. The collaborative path planning decision drives the unmanned surface vessel to perform outbound and return journeys. During the return journey, the optimal return action strategy is invoked, and a closed-loop determination is made in conjunction with a hierarchical reward mechanism to generate a closed-loop path.

[0007] In some embodiments, the status information includes task area grid information, task point location, obstacle distribution, location information of each unmanned surface vessel, and environmental dynamic parameters.

[0008] In some embodiments, the step of combining the multi-agent potential field and establishing an action evaluation function using an artificial potential field gradient and an ε-greedy policy includes: Based on the multi-agent potential field, the resultant force vector of the target point's attractive potential field and the obstacle's repulsive potential field is calculated, and an artificial potential field gradient representing the current motion trend of a single unmanned surface vessel is generated. Based on the artificial potential field gradient, the directional offset of the candidate action is weighted to obtain the weighted artificial potential field gradient. The weighted artificial potential gradient is embedded into the ε-greedy strategy to adjust the action probability and generate an action evaluation function.

[0009] In some embodiments, the dual-Q dynamic adjustment network model includes a Q_task network and a Q_return network; the Q_task network is used to generate an action strategy for the unmanned surface vessel (USV) to perform at least one task point search during the outward journey, and the Q_return network is used to generate an optimal action strategy for the USV to execute a return path during the return journey; the switching condition between the Q_task network and the Q_return network includes a comparison between the real-time distance of the USV and a preset distance; The Q_task network uses the current task point as the source of the attractive potential field, and modifies the action value based on the potential field function constructed from the attractive potential field of the target point and the repulsive potential field of the obstacle, so as to output the optimal action strategy for the outward journey oriented towards the task point. The Q_return network uses the return start point or a preset end point as its attraction potential source, and updates the value of the return action based on a hierarchical reward mechanism for the return phase to generate the optimal return action strategy. The hierarchical reward mechanism includes a return approach reward, a closed-loop completion reward, and a return timeout penalty.

[0010] In some embodiments, the formula for the determination criteria of the dual-Q dynamic adjustment network model is as follows: ; MS represents a binary decision variable. When MS=1, it indicates that the target point arrival condition in the closed-loop task is met, which is used to trigger the Q_return mode switching in the dual-Q dynamic adjustment network model. When MS=0, it indicates that the unmanned surface vessel is still in the outbound stage, which is used to trigger the Q_task network. This indicates the real-time position coordinates of the unmanned surface vessel. Represents the spatial coordinates of the final task point; This indicates the Euclidean distance between the current coordinates of the unmanned surface vessel and the final mission point; otherwise, it indicates that when the Euclidean distance between the unmanned surface vessel and the final mission point is greater than the threshold of 0.5, it has not yet entered the return trip determination state and is still in the outbound point search or path search stage.

[0011] In some embodiments, the step of iteratively updating the target optimal action strategy for a single unmanned surface vessel based on the dual-Q dynamic adjustment network model and the action evaluation function using a deep Q-learning algorithm includes: Based on the action evaluation function, the bias probability of each candidate action in the current state is calculated to form a bias-corrected action evaluation result. The action evaluation results are input into the dual-Q dynamic adjustment network model, and the target Q value of the corresponding state-action pair is constructed using a deep Q-learning algorithm; the target Q value includes immediate reward, discounted future reward, and action bias term. The parameters of the dual-Q dynamic adjustment network model are iteratively updated based on the target Q value; After completing the iterative update, select the action with the highest action value and output the target optimal action strategy for a single unmanned surface vessel.

[0012] In some embodiments, the target optimal action strategy output by the dual-Q dynamic adjustment network model, combined with the artificial potential field gradient, positional relationship, and state interaction of each surface unmanned surface vessel, constructs a cluster collaborative strategy sharing mechanism to generate collaborative path planning decisions for multiple unmanned surface vessels, including: Based on the artificial potential field gradient, positional relationship, and state interaction of each surface unmanned vessel, an interaction information set for multiple agents is constructed; the interaction information set includes the attractive and repulsive potential field components, relative distance, relative speed, and motion trend of neighboring unmanned vessels; Based on the interaction information set and the target optimal action strategy of the single unmanned surface vessel, a set of cooperative actions of multiple unmanned surface vessels is generated by joint weighting. A cluster collaboration strategy sharing mechanism is constructed based on the aforementioned set of collaborative actions; Based on the aforementioned cluster collaboration strategy sharing mechanism, the action selection of the surface unmanned surface vessel cluster is executed, generating collaborative path planning decisions for multiple unmanned surface vessels.

[0013] In some embodiments, the formula for the collaborative force in the cluster collaborative strategy sharing mechanism is as follows: ; in, Indicates the first The total artificial potential force experienced by a surface unmanned vessel at the current moment; N represents the total number of surface unmanned vessels in the swarm. Indicates unmanned surface vessels unmanned surface vessels Interaction forces between them; Represents the coefficient of the attractive potential field; Indicates the first The target point coordinates of the unmanned surface vessel; Indicates the first The current position coordinates of the unmanned surface vessel; Indicates unmanned surface vessels The Euclidean distance to the corresponding target point; M represents the number of obstacles in the environment; This represents the coefficient of the repulsive potential field; Indicates unmanned surface vessels and obstacles The current distance between them; Indicates unmanned surface vessels A safe distance threshold set for obstacles.

[0014] To achieve the above objectives, another aspect of this application proposes an unmanned surface vessel closed-loop path planning system, the system comprising: The acquisition module is used to acquire the status information of multiple unmanned surface vessels (USVs) clusters in the mission area. The potential field construction module is used to construct a multi-agent potential field based on the state information using an artificial potential field algorithm; the multi-agent potential field includes an attractive potential field for the target point and a repulsive potential field for obstacles; The action evaluation module is used to combine the potential field of the multi-agent and establish an action evaluation function using the artificial potential field gradient and the ε-greedy strategy. The dual-Q network construction module is used to build a dual-Q dynamically adjustable network model; The optimization module is used to dynamically adjust the network model and the action evaluation function based on the dual Q-learning algorithm, and iteratively update them using a deep Q-learning algorithm to output the target optimal action strategy for a single unmanned surface vessel. The collaborative strategy sharing module is used to dynamically adjust the target optimal action strategy output by the network model based on the dual Q, and combine the artificial potential field gradient, position relationship and state interaction of each surface unmanned vessel to build a cluster collaborative strategy sharing mechanism to generate collaborative path planning decisions for multiple unmanned vessels. The closed-loop path generation module is used to drive the unmanned surface vessel to perform outbound and return journeys based on the collaborative path planning decision, and to call the optimal return action strategy during the return journey phase. It also combines a hierarchical reward mechanism to determine the closed loop and generate a closed-loop path.

[0015] To achieve the above objectives, another aspect of this application proposes an unmanned surface vessel closed-loop path planning system. The system includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0016] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0017] To achieve the above objectives, another aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0018] The embodiments of this application include at least the following beneficial effects: This application provides a closed-loop path planning method and system for unmanned surface vessels (USVs) based on multi-agent potential fields and deep Q-learning. This scheme constructs a composite potential field that attracts target points and repels obstacles by using an artificial potential field, enabling rapid response to multi-source obstacles and multi-task points in a large sea area, improving the real-time performance and global reachability of path planning. Secondly, by establishing an action evaluation function based on the gradient of the artificial potential field and the ε-greedy policy, the USV achieves an adaptive balance between exploration and utilization, avoiding the problem of traditional methods easily getting trapped in local optima or having a single action choice, thus improving the convergence quality of policy search. The dual-Q dynamic adjustment network model constructed in this invention effectively weakens the overestimation bias of traditional Q-learning when estimating action value, and significantly enhances policy stability and learning robustness through the alternating update of the dual networks. In addition, the cluster collaborative policy sharing mechanism proposed in this invention can extend the optimal action policy of a single USV target to a multi-agent scenario, enabling each USV to couple information based on the gradient of the artificial potential field, positional relationships, and state interactions, achieving synchronous generation of global collaborative behavior, thereby improving the overall obstacle avoidance capability, path consistency, and task division efficiency of the cluster. The closed-loop determination method designed in this invention, which combines the optimal return action strategy with a hierarchical reward mechanism, enables unmanned surface vessels to maintain strategy consistency and convergence during round-trip missions, significantly improving the reliability, safety, and energy efficiency of closed-loop mission execution. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating a closed-loop path planning method for unmanned surface vessels based on multi-agent potential field and deep Q-learning provided in an embodiment of this application. Figure 2 This is an environmental modeling diagram under static conditions provided in the embodiments of this application; Figure 3 This is an environmental modeling diagram under a dynamic environment provided in the embodiments of this application; Figure 4 This is a closed-loop path planning diagram for multiple agents in a static scenario provided in this application embodiment; Figure 5 This is a closed-loop path planning diagram for multiple agents in a dynamic scenario provided in the embodiments of this application; Figure 6 This is a schematic diagram of a closed-loop path planning system for unmanned surface vessels based on multi-agent potential field and deep Q-learning, provided in an embodiment of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0021] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0022] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0024] This application provides a closed-loop path planning method and system for unmanned surface vessels (USVs) based on multi-agent potential fields and deep Q-learning. This scheme constructs a composite potential field that attracts target points and repels obstacles using an artificial potential field, enabling rapid response to multiple obstacles and task points across a wide sea area, improving the real-time performance and global reachability of path planning. Secondly, by establishing an action evaluation function based on the artificial potential field gradient and an ε-greedy policy, the USV achieves an adaptive balance between exploration and utilization, avoiding the problems of traditional methods easily getting trapped in local optima or having a single action choice, thus improving the convergence quality of policy search. The dual-Q dynamic adjustment network model constructed in this invention effectively reduces the overestimation bias in traditional Q-learning when estimating action value, and significantly enhances policy stability and learning robustness through alternating updates of the dual networks. Furthermore, the proposed cluster collaborative policy sharing mechanism extends the optimal action policy for a single USV target to a multi-agent scenario, enabling each USV to couple information based on the artificial potential field gradient, positional relationships, and state interactions, achieving synchronous generation of globally collaborative behaviors, thereby improving the overall obstacle avoidance capability, path consistency, and task allocation efficiency of the cluster. The closed-loop determination method designed in this invention, which combines the optimal return action strategy with a hierarchical reward mechanism, enables unmanned surface vessels to maintain strategy consistency and convergence during round-trip missions, significantly improving the reliability, safety, and energy efficiency of closed-loop mission execution.

[0025] This application provides a method and system for closed-loop path planning of unmanned surface vessels (USVs) based on multi-agent potential fields and deep Q-learning, relating to the field of artificial intelligence technology. The method and system provided in this application can be applied to terminals, servers, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the method and system for closed-loop path planning of USVs based on multi-agent potential fields and deep Q-learning, but is not limited to the above forms.

[0026] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0027] Please see Figures 1-6 As shown, this invention relates to a closed-loop path planning method and system for unmanned surface vessels based on multi-agent potential fields and deep Q-learning.

[0028] Figure 1 This is an optional flowchart of an unmanned surface vessel closed-loop path planning method based on multi-agent potential field and deep Q-learning provided in an embodiment of this application. Figure 1 The method may include, but is not limited to, steps S1 to S7: S1: Obtain the status information of the multi-UAV cluster in the mission area; the status information includes the mission area grid information, mission point location, obstacle distribution, the location information of each UAV, and environmental dynamic parameters.

[0029] In this embodiment, multiple unmanned surface vessels (USVs) depart from a preset starting point before the mission begins. They perform a full-scale state scan and initial perception of the mission area using an integrated environmental awareness module and pre-loaded mission data. To ensure the accuracy of subsequent potential field construction and reinforcement learning decisions, the state information collection covers three levels: static environmental information, dynamic environmental change trends, and the USVs' own kinematic information.

[0030] At the static environment level, each unmanned surface vessel (USV) reads the gridded structure of the mission area, divides the region into a two-dimensional grid with uniform or mission-adaptive resolution, and marks navigable and impassable areas. The USV simultaneously loads the position coordinates of all mission points, mission sequence requirements, and navigation risk levels of the mission point's neighborhood for subsequent construction of the attractive potential field. For known fixed obstacles, such as reefs, structures, and no-navigation zones, the USV maps their geometry, circumscribed boundary, and danger radius entirely into the environmental grid in the initial stage, ensuring that the obstacle's influence range is accurately reflected in the subsequent potential field.

[0031] At the dynamic environment level, unmanned surface vessels (USVs) predict ocean current direction, drift trajectories of floating objects, and their speed changes by utilizing short-term historical observations and the dynamic characteristics of the mission scenario. If moving targets or temporary obstacles exist, the USV records their positions, movement trends, and the extent of surrounding disturbances as dynamic state inputs, thus providing a more realistic environmental description for subsequent action evaluation. Furthermore, the dynamic environment information is updated in time-series format, providing continuous state awareness for deep Q-learning.

[0032] At the kinematic level, each unmanned surface vessel (USV) reports its precise geographical location, heading angle, speed, available energy, and turning ability in real time. USVs also exchange position information, forming a cluster-level state sharing mechanism. This allows each member to not only understand its own status but also predict the trajectory trends of neighboring USVs. This information will play a crucial role in subsequently developing collaborative strategies.

[0033] S2: Based on state information, an artificial potential field algorithm is used to construct a multi-agent potential field; wherein, the multi-agent potential field includes the target point attraction potential field and the obstacle repulsion potential field; In this embodiment, target attraction trends are created for each surface unmanned vessel (USV) based on the distribution of task points. By analyzing the surrounding environment of the task points, the USVs can perceive changes in distance to the task points; a shorter distance creates a stronger attraction trend, providing clear directional guidance for the USVs as they move forward. Furthermore, when a task point has already been prioritized by other USVs, its attraction trend is automatically weakened based on a swarm collaboration mechanism to prevent multiple USVs from competing for the same task point and improve task allocation efficiency.

[0034] In constructing the obstacle potential field, unmanned surface vessels (USVs) set different repulsion influence ranges for different types of obstacles. For fixed structures (such as reefs and shoals), the repulsion range is stable and has high influence intensity; for dynamic obstacles, the repulsion range changes with the possible direction and speed of movement, forming a "predictive repulsion zone." When USVs approach any obstacle, they automatically identify the most likely collision direction and generate a reverse avoidance tendency, thus forming a safe path tendency to avoid that direction.

[0035] In a multi-agent environment, unmanned surface vessels (USVs) also generate a slight repulsive potential field among themselves to avoid congestion and collisions. For example, when two USVs approach the same area, each USV calculates its own directional tendency to avoid the other or continue forward based on the other's dynamic trajectory, its own positional change trend, and the urgency of its current task. This prevents paths within the swarm from intersecting too densely, thus reducing the overall probability of conflict.

[0036] After the above processing, the resulting multi-agent potential field not only includes the two classic potential fields of task attraction and obstacle repulsion, but also includes an interactive potential field for multi-vessel collaborative scenarios, enabling the unmanned surface vessel to automatically tend towards a reasonable, safe and efficient trajectory direction based on complex environmental information during the subsequent action selection phase.

[0037] S3: Combining the multi-agent potential field, an action evaluation function is established using the artificial potential field gradient and the ε-greedy strategy; Among them, combining the multi-agent potential field, the action evaluation function is established using the artificial potential field gradient and the ε-greedy policy, including: Based on the multi-agent potential field, the resultant force vector of the attractive potential field of the target point and the repulsive potential field of the obstacle is calculated, and an artificial potential field gradient representing the current motion trend of a single unmanned surface vessel is generated. Based on the artificial potential field gradient, the directional offset of the candidate action is weighted to obtain the weighted artificial potential field gradient. The weighted artificial potential gradient is embedded into the ε-greedy policy to adjust the action probability and generate the action evaluation function.

[0038] In this embodiment, each unmanned surface vessel (USV) first reads the target point attraction trend and obstacle repulsion trend from the multi-agent potential field based on its current grid location, and then superimposes these two trends in direction to form an artificial potential field gradient for the current state. This gradient characterizes the USV's natural motion direction without external intervention. If the USV is in an open area near the mission point, the attraction trend dominates, and the potential field gradient tends to point towards the target point. If the USV is near obstacles or other densely populated areas of USVs, the repulsion trend strengthens, causing the potential field gradient to deviate from the potentially dangerous direction, thus forming a proactive avoidance directional tendency.

[0039] After obtaining the basic potential field gradient, the unmanned surface vessel (USV) analyzes the set of currently executable candidate actions, such as minor heading adjustments in multiple directions and straight-line propulsion. For each candidate action, its consistency with the potential field gradient direction is calculated to assess whether the action aligns with the current potential field trend. Subsequently, the candidate actions are weighted: actions consistent with or close to the artificial potential field gradient direction receive increased weight, giving them a more significant advantage in subsequent selections; actions deviating significantly from the potential field trend have their weights appropriately reduced, but still retain a certain probability to prevent the USV from completely losing its necessary exploration capabilities. The weighted potential field gradient result constitutes the USV's directional offset reference at the current moment.

[0040] After weighted processing, this embodiment embeds the artificial potential field gradient into the ε-greedy policy to form the final action evaluation function. When the policy is in the utilization phase, the selection probability of each action will be biased according to the enhanced potential field tendency, making the unmanned surface vessel more inclined to choose actions that are close to the target and have reasonable obstacle avoidance. When the policy is in the exploration phase, the mechanism of random action selection is still retained, but the bias influenced by the potential field is added, so that the exploration process is no longer completely random, but maintains a certain directionality, which helps to reduce ineffective detours and improve learning efficiency in complex sea areas.

[0041] It should be noted that the ε-greedy policy is a commonly used action selection mechanism in reinforcement learning. Its core idea is to strike a balance between "exploration" and "exploitation": randomly select an action with probability ε to explore potentially better but untried strategies; and randomly select an action with probability 1... ε selects the action with the highest estimated value, which is used to improve decision-making quality by leveraging existing knowledge. By gradually decreasing ε during training, unmanned surface vessels and other intelligent agents are encouraged to try more different operations in the early stages and gradually focus on the optimal strategy in the later stages, thereby improving learning efficiency and decision-making stability.

[0042] Based on the action evaluation function constructed using the artificial potential field gradient and the ε-greedy strategy, this embodiment further utilizes formulas (1) and (2) to more finely distinguish the directional characteristics of candidate actions. Specifically, the artificial potential field generates a resultant force direction in the current state based on the attraction trend of the target point and the repulsion trend of the obstacle. This direction not only represents the optimal motion trend of the unmanned surface vessel (USV) but also reflects the risk distribution in the current environment. By comparing the direction of the candidate action with this resultant force direction, the offset attributes of the action in different directions can be obtained, enabling the USV to identify which actions are more in line with the target convergence requirements and which actions help avoid obstacle areas.

[0043] The specific calculation formula is as follows: Formula (1); In the formula, The total potential force is a vector composed of the attractive force at the target point and the repulsive force from the obstacle, guiding the USV to move towards the target and avoid the obstacle; Represents the spatial coordinates of the target point (such as the coordinates of the task point or the return destination). This indicates the real-time position coordinates of the current USV; This represents the Euclidean distance between the unmanned surface vessel and the target point. represents the attraction potential coefficient, which controls the strength of the attraction force that moves the unmanned surface vessel (USV) toward the target point. The larger the value, the more the USV tends to accelerate toward the target point faster; M represents the number of obstacles in the environment, including static and dynamic obstacles. This represents the repulsive potential field coefficient, which controls the intensity of the repulsive force when the unmanned surface vessel approaches an obstacle. The larger the value, the stronger the obstacle avoidance behavior. Indicates unmanned surface vessels and the first The current distance between each obstacle, if The smaller the value, the greater the repulsive potential field. The repulsion approaches zero; This represents the safe distance threshold between the USV and the obstacle, used to determine whether a repulsive potential field is triggered. When the repulsive force increases, The repulsive force weakens or disappears over time.

[0044] Formula (2); In the formula, Indicates the current state Next action Then, the Q value after incorporating the directional correction of the artificial potential field; Indicates the state Select action The original Q value represents the expected value of the long-term cumulative reward; This represents the magnitude of the total resultant force of the artificial potential field; Represents the direction vector of the action; Represents the magnitude of the motion direction vector; This represents the potential field influence coefficient.

[0045] Based on this, this embodiment utilizes the probability bias concept embodied in formula (3) to embed the artificial potential field gradient as an offset in the action selection probability into the ε-greedy strategy. This allows action evaluation to move beyond the traditional "maximum reward action priority" logic and incorporate a directional bias based on the potential field. Thus, when the unmanned surface vessel (USV) is in the exploration phase, it still possesses random exploration capabilities, but will not engage in completely aimless random wandering. When in the utilization phase, actions with potential field advantages will receive a higher selection probability, forming a "greedy selection based on directional prior." Through this mechanism, the action evaluation function ensures sufficient exploration while maintaining the USV's action selection towards the target, away from obstacles, and with a clear dynamic trend, thereby significantly improving the stability and efficiency of path planning.

[0046] The specific calculation formula is as follows: Formula (3) ； In the formula, Indicates the state Select action The probability of; Indicates the probability coefficient of exploration; Indicates the current state The action that maximizes the Q value; This represents the exploration probability distribution guided by the artificial potential field gradient.

[0047] S4: Construct a dual-Q dynamically adjusted network model; The dual-Q dynamic adjustment network model includes a Q_task network and a Q_return network. The Q_task network is used to generate the action strategy for the unmanned surface vessel (USV) to perform at least one task point search during the outward journey, and the Q_return network is used to generate the optimal action strategy for the USV to execute the return path during the return journey. The switching condition between the Q_task network and the Q_return network includes the comparison relationship between the real-time distance of the USV and the preset distance. The Q_task network uses the current task point as the source of the attractive potential field, and modifies the action value based on the potential field function constructed from the attractive potential field of the target point and the repulsive potential field of the obstacle, so as to output the optimal action strategy for the outward journey oriented towards the task point. The Q_return network uses the return start point or a preset end point as its attraction potential source, and updates the value of the return action based on the hierarchical reward mechanism of the return stage to generate the optimal return action strategy.

[0048] In this embodiment, a dual-Q dynamic adjustment network model is constructed. This model includes a Q_task network for the task point search phase and a Q_return network for the return phase. Both are trained based on the same state space and action space, but differ in the setting of the attraction potential source, reward structure, and emphasis on value update, thereby optimizing the strategy for the outbound and return scenarios respectively.

[0049] During the outward journey, the Q_task network is activated to perform task point search. At this point, the attraction potential source for each unmanned surface vessel (USV) is set to the current task point location. The attraction trend of the target point serves as the primary guiding information, and together with the repulsion trend of obstacles, they construct the potential function for this task phase. When estimating action value, the Q_task network embeds this potential function into the value evaluation process, correcting the traditional reward-based action value assessment. This makes it more favorable to choose actions that accelerate approach to the task point while maintaining a safe distance, given equal rewards. Through long-term training, the Q_task network gradually develops an action strategy optimized for task points, enabling efficient search and allocation of multiple task points by USVs in complex sea areas.

[0050] When the unmanned surface vessel (USV) completes the exploration of the current task point and the distance to the final task point meets the preset judgment conditions, this embodiment triggers the dual-Q dynamic adjustment process, switching the action decision from the Q_task network to the Q_return network. After the switch, the source of the attractive potential field is no longer the task point, but the return start point or the preset end point. The potential field function in the return phase emphasizes the convergence trend towards the start point and obstacle avoidance safety during the return process. The Q_return network introduces a hierarchical reward mechanism during training and execution. Through a hierarchical structure such as return approach reward, closed-loop completion reward, and return timeout penalty, the value of the return action is updated, enabling the network to learn to complete the return closed loop with a shorter path and higher time efficiency while ensuring safe obstacle avoidance. When the USV meets the closed-loop completion conditions in the return phase, such as returning to the vicinity of the start point and meeting the convergence criterion, the dual-Q dynamic adjustment network model terminates the network switching process, determining that the closed-loop path planning task is completed. Through the aforementioned dual-Q structure and dynamic switching mechanism, this embodiment achieves collaborative optimization of outbound task point search and return path planning within the same framework, thereby improving the overall performance and feasibility of multi-unmanned surface vessel closed-loop path planning.

[0051] Specifically, the formula for the decision criterion of the dual-Q dynamic adjustment network model is as follows: Formula (4); MS represents a binary decision variable. When MS=1, it means that the target point arrival condition in the closed-loop task is met, which is used to trigger the Q_return mode switching in the dual-Q dynamic adjustment network model. When MS=0, it means that the unmanned surface vessel is still in the outbound stage, which is used to trigger the Q_task network. This indicates the real-time position coordinates of the unmanned surface vessel. Represents the spatial coordinates of the final task point; This indicates the Euclidean distance between the current coordinates of the unmanned surface vessel and the final mission point; otherwise, it indicates that when the Euclidean distance between the unmanned surface vessel and the final mission point is greater than the threshold of 0.5, it has not yet entered the return trip determination state and is still in the outbound point search or path search stage.

[0052] When executing task mode, the source of the attractive potential field is the current task point, and the potential field function at this time is: ,in Let be the weights in the task execution mode; when in return mode, the attractive potential source is the start / end point, and the potential function at this time is . ,in Weights are assigned to the return mode. To ensure smooth switching between the two modes in real-world scenarios, a dual-Q dynamic adjustment network model is used for dynamic switching, allowing the USV to automatically redirect to the start / end point after completing its task.

[0053] S5: The action evaluation function and deep Q-learning algorithm are used to iteratively update the dual-Q dynamic adjustment network model and output the target optimal action strategy for a single unmanned surface vessel. The target optimal action strategy includes the outbound optimal action strategy and the return optimal action strategy. Specifically, the action evaluation function and deep Q-learning algorithm are used to iteratively update the dual-Q dynamic adjustment network model, outputting the target optimal action strategy for a single unmanned surface vessel. The target optimal action strategy includes an optimal action strategy for the outward journey and an optimal action strategy for the return journey, including: Based on the action evaluation function, the bias probability of each candidate action in the current state is calculated to form a bias-corrected action evaluation result. The action evaluation results are input into the dual-Q dynamic adjustment network model, and the target Q value of the corresponding state-action pair is constructed using the deep Q learning algorithm; wherein, the target Q value includes the immediate reward, the discounted future reward, and the action bias term; The parameters of the dual-Q dynamically adjusted network model are iteratively updated based on the target Q value; After completing the iterative update, select the action with the highest action value and output the target optimal action strategy for a single unmanned surface vessel.

[0054] In this embodiment, the single unmanned surface vessel (USV) first calculates the bias probability of all candidate actions in the current state based on the action evaluation function constructed in the previous stage. This bias result is jointly determined by the artificial potential field gradient, obstacle repulsion tendency, and the ε-greedy exploration factor, enabling the USV to obtain an action evaluation quantity that combines directional guidance and exploration capabilities before action selection. Through this bias processing, the USV can prioritize actions that are close to the target point, have a significant obstacle avoidance effect, or approach the return start point, thereby improving the effectiveness of the strategy.

[0055] After obtaining the bias-corrected action evaluation result, it is input along with the current state into the dual-Q dynamic adjustment network model. Depending on whether the unmanned surface vessel (USV) is in the outbound or return phase, the Q_task network or Q_return network is automatically selected for value assessment. Subsequently, a deep Q-learning algorithm is used to calculate the target Q-value for the corresponding state-action pair. This target Q-value includes the immediate reward, the discounted future reward, and a bias term generated by potential field guidance information. The immediate reward reflects the quality of a single action; the discounted future reward reflects the long-term convergence trend; and the bias term ensures that the potential field direction continuously influences the value estimation, preventing the learning process from deviating from the overall path planning objective.

[0056] Subsequently, the network parameters are updated based on the target Q-value. By utilizing experience replay and gradient adjustment, the network iteratively approximates the action value function, making the Q-value estimation more accurate, thereby continuously improving the unmanned surface vessel's ability to judge the merits of actions at different stages and in different environmental states. As the number of training rounds increases, the Q_task network gradually masters the convergence characteristics in the task point search environment, while the Q_return network gradually learns the optimal behavior pattern of approaching the starting point, completing the closed loop, and improving obstacle avoidance efficiency during the return journey.

[0057] After completing the parameter update for the current round, the unmanned surface vessel (USV) directly selects the action with the highest value from the updated Q-network output and executes it. Through continuous iteration of the process of "bias evaluation → target Q-value construction → network update → optimal action output", the USV can form a stable, reliable, and scenario-adaptive target-optimal action strategy, ensuring that the entire closed-loop path planning process remains efficient, safe, and convergent in both the outbound and return phases.

[0058] After completing the iterative update based on the dual-Q network and action evaluation function, this embodiment further utilizes the value update mechanism embodied in formula (5) to strengthen the learning process of the Q value. In this mechanism, each action generates an immediate reward, while future potential gains are incorporated into the update target through a discount, enabling the unmanned surface vessel to not only focus on its current local performance during training but also consider the overall gains of subsequent path planning. At the same time, artificial potential field guidance information is embedded in the update term, allowing the network to automatically strengthen the trend-following direction and weaken the counter-trend direction during value iteration, thereby avoiding getting trapped in local optima in complex environments.

[0059] By integrating immediate rewards, discounted future rewards, and potential field guidance into value updates, the unmanned surface vessel (USV) gradually develops a stable convergence trend after multiple rounds of training: it can reliably approach the mission point or return start point while maintaining a balance between obstacle avoidance and time efficiency. Ultimately, the action strategy output based on this update mechanism can continuously improve the quality of path planning in dynamic environments.

[0060] The specific calculation formula is as follows: Formula (5); In the formula, This represents the new Q value after incorporating artificial potential field reward adjustments and being updated using Q-learning. Indicates the state Next action The current Q value (the old value before the update); Indicates the learning rate; This indicates the immediate reward for adjusting the artificial potential field. Indicates the discount factor; Indicates the next state The maximum value of Q among all possible actions is the optimal future value that USV can achieve in the new state.

[0061] S6: Based on the target optimal action strategy output by the dual-Q dynamic adjustment network model, and combined with the artificial potential field gradient, positional relationship and state interaction of each surface unmanned vessel, a cluster collaborative strategy sharing mechanism is constructed to generate collaborative path planning decisions for multiple unmanned vessels. Among them, based on the target optimal action strategy output by the dual-Q dynamic adjustment network model, and combined with the artificial potential field gradient, positional relationship, and state interaction information of each surface unmanned surface vessel, a cluster collaborative strategy sharing mechanism is constructed to generate collaborative path planning decisions for multiple unmanned surface vessels, including: Based on the artificial potential field gradient, positional relationship and state interaction of each surface unmanned vessel, an interaction information set of multiple agents is constructed; specifically, the interaction information set includes the attractive and repulsive potential field components, relative distance, relative speed and motion trend of neighboring unmanned vessels; Based on the interaction information set and the target optimal action strategy of a single unmanned surface vessel, a set of cooperative actions of multiple unmanned surface vessels is generated by joint weighting. A cluster collaboration strategy sharing mechanism is built based on a set of collaborative actions; Based on the cluster collaboration strategy sharing mechanism, the action selection of the surface unmanned surface vessel cluster is executed, and collaborative path planning decisions of multiple unmanned surface vessels are generated.

[0062] In this embodiment, after obtaining the target-optimal action strategy output by the dual-Q dynamic adjustment network, multiple surface unmanned surface vessels (USVs) do not immediately execute actions independently. Instead, they first combine the environmental perception results among the multiple agents to construct a unified interactive information set. Each USV periodically collects information such as the current artificial potential field gradient, relative distance and relative speed with neighboring USVs, heading change trends, and the force directions of neighboring USVs in the attracting and repulsive potential fields. Through information broadcasting, each USV can obtain a multi-agent interactive information set covering its own local neighborhood, thereby comprehensively understanding the movement intentions and potential conflicts of surrounding USVs.

[0063] After obtaining the aforementioned set of interactive information, this embodiment jointly weights it with the target optimal action strategy of a single surface unmanned surface vessel (USV). If a certain action direction conflicts with the movement direction of a neighboring USV, is too close, or may lead to path overlap, the priority of that action is reduced, thereby prompting the USV to choose a safer and more coordinated direction. Conversely, if certain actions enable the USV to avoid locally congested areas, balance task coverage, or avoid redundant searches, their weight is increased, giving that direction a greater advantage in strategy selection. The jointly weighted action set constitutes the collaborative action set of multiple USVs at the current moment, taking into account both the individual vessel's task benefits and the overall collaborative requirements.

[0064] Subsequently, this embodiment constructs a cluster collaborative strategy sharing mechanism based on the aforementioned set of collaborative actions. This mechanism defines the strategy transmission methods and collaborative behavior specifications among unmanned surface vessels (USVs), such as: sharing local optimal path trends, automatically adjusting priorities to avoid USVs, and dynamically coordinating track distribution during task point search or return phases. Through this mechanism, each USV can align with the overall strategy at the cluster level in real time before executing its own actions, ensuring that path planning no longer interferes with each other, but instead exhibits overall coordinated and clearly defined cluster motion characteristics.

[0065] Ultimately, based on the collaborative strategy sharing mechanism, each unmanned surface vessel (USV) selects the most suitable action command from the set of collaborative actions to form a collaborative path planning decision for the current moment. This decision is continuously iterated and updated in subsequent steps, enabling multiple USVs to gradually form a cluster trajectory with safety, coordination, and closed-loop feasibility in complex sea areas.

[0066] Specifically, the formula for the collaborative force in the cluster collaborative strategy sharing mechanism is as follows: Formula (6); in, Indicates the first The total artificial potential force experienced by a surface unmanned vessel at the current moment; N represents the total number of surface unmanned vessels in the swarm. Indicates unmanned surface vessels unmanned surface vessels Interaction forces between them; Represents the coefficient of the attractive potential field; Indicates the first The target point coordinates of the unmanned surface vessel; Indicates the first The current position coordinates of the unmanned surface vessel; Indicates unmanned surface vessels The Euclidean distance to the corresponding target point; M represents the number of obstacles in the environment; This represents the coefficient of the repulsive potential field; Indicates unmanned surface vessels and obstacles The current distance between them; Indicates unmanned surface vessels A safe distance threshold set for obstacles.

[0067] S7: Based on collaborative path planning decisions, the unmanned surface vessel (USV) executes outbound and return journeys. During the return journey, it invokes the optimal return action strategy and uses a tiered reward mechanism to determine the closed-loop path. The tiered reward mechanism includes a return journey approximation reward, a closed-loop completion reward, and a return journey timeout penalty.

[0068] In this embodiment, after completing the cooperative path planning, the unmanned surface vessel (USV) will execute its outbound journey step by step according to the generated path strategy. During the outbound phase, the USV advances from the starting point along a predetermined direction based on the cooperative path planning decision, maintaining real-time action updates and status monitoring. As the USVs gradually approach their assigned task points along the planned trajectory, they continuously perceive the surrounding environment, including the movement trends of dynamic obstacles, the avoidance behavior of neighboring USVs, and changes in the local water flow field, and accordingly make real-time fine adjustments to their current course, speed, and path nodes.

[0069] After successfully reaching the mission point and completing the outbound phase, the unmanned surface vessel (USV) will automatically trigger the return journey based on preset thresholds. During the return phase, a dedicated return strategy network outputs action decisions, prioritizing the feasibility, safety, and time efficiency of the path back to the starting point. Throughout this process, each USV continuously adjusts its orientation based on the return approach reward, gradually moving closer to the starting point. Simultaneously, as the USV maintains a stable approach to the starting point without deviation from the path, the return approach reward accumulates, contributing to the stable execution of the return maneuver.

[0070] During the return journey, the unmanned surface vessel (USV) also invokes a loop completion reward, which is used to monitor whether the return path has been closed as expected. When the USV's current position enters the loop completion detection interval and meets the return convergence condition, the loop completion reward is triggered, allowing the USV to confirm that it has completed the loop return. This reward mechanism ensures that the entire loop path not only has directional consistency but also behavioral termination criteria, enabling the training network to clearly distinguish between the two key stages of outbound completion and return loop closure.

[0071] To avoid excessive path detours or time delays during the return journey, this embodiment introduces a return timeout penalty. When the unmanned surface vessel (USV) continuously deviates from its return direction, repeatedly lingers in localized areas, or experiences a significant decrease in exploration efficiency due to dense obstacles, the return timeout penalty will increase rapidly, causing the USV to automatically adjust its course and avoid ineffective return journeys. The timeout penalty, along with the approach reward and closed-loop completion reward, constitutes a hierarchical reward mechanism that continuously adjusts the USV's path execution during the return journey, ensuring that the return decision-making possesses both the stability of gradual approach and the convergence of avoiding excessive detours.

[0072] During the return journey, the multiple unmanned surface vessels (USVs) maintain a shared collaborative strategy. Once one USV completes its return journey ahead of schedule and returns to its starting point, the remaining USVs can dynamically adjust their return routes to avoid path conflicts during the return phase, ensuring that the USVs are distributed reasonably and in an orderly manner near the starting point. If the return efficiency of the USVs decreases due to dynamic obstacles or other unforeseen circumstances, a local path planning process will be re-executed during the return phase to ensure the feasibility and safety of the return path.

[0073] Once all unmanned surface vessels (USVs) have successfully returned to the starting point, the closed-loop process officially ends. This embodiment realizes a continuous process from outward execution and return triggering to closed-loop completion, enabling USVs not only to complete task point exploration but also to return to the starting point in an efficient, safe, and stable manner, achieving the complete "closed-loop path planning" task objective.

[0074] Specifically, in this embodiment, multiple USVs start from a base point and aim to completely cover all task nodes within the water area. A multi-constraint, multi-objective optimization model is established, and obstacle avoidance constraints are embedded in the path planning process. The overall goal of multi-USV cooperative path planning is to enable all USVs to collaboratively complete the task, while minimizing the total path length and task completion time, and maximizing the collision avoidance success rate. Furthermore, a weight allocation objective function is designed, which... By balancing the total path length and the shortest task completion time, different weights are assigned to the global path length and the global task completion time, and the optimality of the training path under different weights is evaluated to improve the efficiency of ship navigation.

[0075] The specific calculation formula is as follows: Formula (7); Formula (8); In the formula, Indicates the first The USV executes the first The path length of each task segment. It refers to the number of task segments; Indicates the first The USV in the path planning The path length of the segment. It refers to the number of task segments; It is a weighting coefficient that balances the relative importance of path length and task execution time; It is the first A set of task execution times for each USV.

[0076] Formula (9); In the formula, It is the success rate of cluster collision avoidance; It is the collision avoidance success rate threshold; It is the first The actual path length of the USV; It is the first Minimum path length constraint for each USV; It is the first The mission completion time of the USV; It is the first Deadline for completion of each USV task; yes and The Euclidean distance; It is the minimum safe distance between USVs; yes Distance to the nearest obstacle; yes The obstacle safety radius; yes The heading angle; yes The linear velocity.

[0077] Specifically, formulas (7) and (8) are used to measure the path length performance of each surface unmanned vessel (USV) throughout the entire mission process. During the return phase, the cumulative path cost of the USV from the return start point to the destination is continuously estimated. When the USV's path is unnecessarily deflected due to dense obstacles or swarm avoidance, the path cost will increase accordingly, prompting the return strategy to readjust its orientation and reduce unnecessary distance consumption. Formula (9) is the objective function for the USV's mission execution, representing the minimum sum of the total path cost of all USVs and the mission completion time cost of all USVs. The larger the corresponding value, the greater the proportion of that item.

[0078] The time constraints and task segment execution duration involved in formula (9) are also used to evaluate return efficiency during the return phase. When the return speed of the unmanned surface vessel is low or it stagnates in a local area, the return time cost will increase rapidly. In this embodiment, this time cost is used as an influencing factor in the action decision-making process to ensure that the return path maintains time convergence and avoids situations where the vessel lingers in a certain area for a long time.

[0079] In this embodiment, the multi-objective reward total function in the hierarchical reward mechanism of formula (10) is designed. The hierarchical reward mechanism in this embodiment is used to balance path length, time efficiency, task completion and safety when planning multiple USVs. In order to ensure that the USV can return accurately, a closed-loop dedicated reward is set in the hierarchical reward to strengthen the integrity of the closed loop. It mainly includes return approach reward, closed-loop completion reward and return timeout penalty.

[0080] The formula for calculating the total multi-objective reward function is as follows: Formula (10); In the formula, This indicates the relative weight of goal-approach rewards and time efficiency rewards in the overall task planning; This represents the target proximity reward, which is the reward the USV receives for each step it takes closer to the target mission point (or return start point). The closer the distance, the higher the reward; the further away from the target direction the reward is, the lower or negative the reward becomes. This represents the time efficiency reward, reflecting the time saved by the USV during task execution. The faster the task is completed or the return trip is made, the greater the reward; if the speed is slow or the execution is delayed, the reward will be reduced or a penalty will be imposed. This indicates an obstacle avoidance reward, used to encourage USVs to successfully avoid obstacles and other USVs; This refers to the collaborative reward among multiple unmanned surface vessels (USVs), i.e., the collaborative effect of multiple USVs during mission execution. This indicates the mission completion reward, a large reward triggered when the USV successfully reaches the designated mission point; To provide a reward for getting closer to the starting point on the return journey, the USV is rewarded every time it gets closer to the starting point. To complete the closed-loop reward, the closed-loop determination condition is: Once the USV confirms that it has completed the closed loop, a substantial reward will be given. Penalty for exceeding the return time limit The timeout penalty coefficient is set to avoid return delays and ensure return timeliness by setting an exponential penalty to improve return efficiency.

[0081] In this embodiment, a simulation environment was built based on the MATLAB / Simulink simulation platform. For the cases of 3 and 4 USVs, simulation analysis was conducted on various scenarios with 30 and 40 task points, dynamic and static obstacles at different locations, and random task points distributed in various directions from the starting point. Figure 2 , Figure 3 Environmental modeling diagrams for static and dynamic environments. Figure 4 , Figure 5 This is a closed-loop path planning diagram for multiple ships in static and dynamic scenarios.

[0082] Specifically, Figure 2 This paper demonstrates a USV (Unmanned Aerial Vehicle) operating environment model constructed in a static scenario to verify the path planning performance of the invention under fixed obstacle conditions. The scenario uses a 100×100 regular grid to represent the marine space, where dark or solid-shaped areas correspond to impassable static obstacles such as reefs, fixed structures, or restricted waters. Task points are identified by different colors and assigned to multiple USVs. Each USV starts from the same or adjacent starting points, must visit all task points, and finally return to the starting point to achieve the closed-loop path planning objective. In this modeling environment, the obstacles do not change over time, thus effectively validating the path optimality, obstacle avoidance robustness, and closed-loop planning capability of the artificial potential field guidance, action evaluation function, and dual-Q dynamic adjustment network model in a classic static scenario. Figure 2 For subsequent static path results ( Figure 4 The basic modeling environment.

[0083] Figure 3 This paper demonstrates the environmental modeling method of a USV in dynamic sea areas, used to verify the adaptive capability of the invention in scenarios containing dynamic obstacles. Unlike static environments, the figure includes multiple dynamic obstacles that move in real-time, in addition to fixed obstacles. These may simulate other ships, floating objects on the sea surface, or mobile platforms performing tasks. The direction and velocity of the dynamic obstacles are continuously updated in the environmental model, forming time-varying constraint boundaries, requiring the USV to dynamically perceive its state and adjust its action strategy accordingly during path planning. The task point remains fixed in the model, but the navigable area of ​​the USV changes continuously at different time steps, effectively verifying the value of the potential field gradient guidance in the action evaluation function and the adaptability of the double-Q network to the dynamic environment during the return phase. This figure is... Figure 5 The modeling basis for the dynamic closed-loop path planning results shown is as follows.

[0084] Figure 4This paper demonstrates the closed-loop path planning results of this invention in a static environment using multiple USVs. In the figure, pink ellipses represent static obstacles, and points of different colors represent task points assigned to different USVs. Each USV starts from the same point and generates an optimal path under the combined guidance of an artificial potential field and a deep Q-learning strategy. The USVs can sequentially visit their task points without collisions, while avoiding all obstacles, avoiding local optima or ineffective detours. After completing the task, each USV's path automatically switches to a return strategy under the action of the dual-Q dynamic adjustment network model, planning a return trajectory and successfully returning to the starting point, thus forming a complete and effective closed-loop path. The smooth path and stable obstacle avoidance in the figure demonstrate that this invention has excellent convergence performance, path feasibility, and task execution efficiency in a static environment.

[0085] Figure 5 This paper demonstrates the closed-loop path planning results of multiple USVs in a dynamic environment. In this scenario, dynamic obstacles continuously move over time, so the USVs not only need to avoid fixed obstacles but also need to avoid the trajectories of dynamic obstacles in real time. The paths of each USV in the figure exhibit obvious adaptive change characteristics: when the dynamic environment changes, the USVs can adjust their action direction based on the APF potential force and the deep Q-network estimation, causing the path to automatically shift to avoid potential collisions. Simultaneously, after completing all task points, the USVs enter return mode, stably returning to the starting point under the guidance of return approach rewards and closed-loop completion rewards, achieving full-process closed-loop planning. As can be seen from the figure, regardless of how the obstacles move, the USVs can maintain a safe trajectory, demonstrating the algorithm's real-time performance, robustness, and collaborative capabilities in dynamic scenarios.

[0086] Please see Figure 6 This application also provides an unmanned surface vessel closed-loop path planning system, the system comprising: The acquisition module is used to acquire the status information of multiple unmanned surface vessels (USVs) clusters in the mission area. The potential field construction module is used to construct a multi-agent potential field based on state information and using an artificial potential field algorithm; the multi-agent potential field includes the target point attraction potential field and the obstacle repulsion potential field; The action evaluation module is used to combine the potential fields of multiple agents and establish an action evaluation function using artificial potential field gradients and ε-greedy strategies. The dual-Q network construction module is used to build a dual-Q dynamically adjustable network model; The optimization module is used to dynamically adjust the network model and action evaluation function based on double Q, and iteratively update the model using a deep Q-learning algorithm to output the target optimal action strategy for a single unmanned surface vessel. The collaborative strategy sharing module is used to dynamically adjust the target optimal action strategy output by the network model based on the dual Q. It combines the artificial potential field gradient, position relationship and state interaction of each surface unmanned vessel to build a cluster collaborative strategy sharing mechanism to generate collaborative path planning decisions for multiple unmanned vessels. The closed-loop path generation module is used to drive the unmanned surface vessel to perform outbound and return journeys based on collaborative path planning decisions. During the return journey, it calls the optimal return action strategy and combines a hierarchical reward mechanism to determine the closed loop and generate a closed-loop path.

[0087] It is understood that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0088] This application also provides an unmanned surface vessel (USV) closed-loop path planning system. The system includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0089] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0090] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0091] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0092] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0093] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0094] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0095] This application provides a closed-loop path planning method and system for unmanned surface vessels (USVs) based on multi-agent potential fields and deep Q-learning. This scheme constructs a composite potential field that attracts target points and repels obstacles using an artificial potential field, enabling rapid response to multiple obstacles and task points across a wide sea area, improving the real-time performance and global reachability of path planning. Secondly, by establishing an action evaluation function based on the artificial potential field gradient and an ε-greedy policy, the USV achieves an adaptive balance between exploration and utilization, avoiding the problems of traditional methods easily getting trapped in local optima or having a single action choice, thus improving the convergence quality of policy search. The dual-Q dynamic adjustment network model constructed in this invention effectively reduces the overestimation bias in traditional Q-learning when estimating action value, and significantly enhances policy stability and learning robustness through alternating updates of the dual networks. Furthermore, the proposed cluster collaborative policy sharing mechanism extends the optimal action policy for a single USV target to a multi-agent scenario, enabling each USV to couple information based on the artificial potential field gradient, positional relationships, and state interactions, achieving synchronous generation of globally collaborative behaviors, thereby improving the overall obstacle avoidance capability, path consistency, and task allocation efficiency of the cluster. The closed-loop determination method designed in this invention, which combines the optimal return action strategy with a hierarchical reward mechanism, enables unmanned surface vessels to maintain strategy consistency and convergence during round-trip missions, significantly improving the reliability, safety, and energy efficiency of closed-loop mission execution.

[0096] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0097] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0098] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0099] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0100] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0101] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A closed-loop path planning method for unmanned surface vessels based on multi-agent potential field and deep Q-learning, characterized in that, The method includes: Acquire the status information of multiple unmanned surface vessels (USVs) clusters in the mission area; Based on the state information, an artificial potential field algorithm is used to construct a multi-agent potential field; the multi-agent potential field includes the target point attraction potential field and the obstacle repulsion potential field; Combining the multi-agent potential field, an action evaluation function is established using the artificial potential field gradient and the ε-greedy strategy; Construct a dual-Q dynamically adjusted network model; The action evaluation function and deep Q-learning algorithm are used to iteratively update the dual-Q dynamic adjustment network model and output the target optimal action strategy for a single unmanned surface vessel. The target optimal action strategy includes the outbound optimal action strategy and the return optimal action strategy. Based on the target optimal action strategy output by the dual-Q dynamic adjustment network model, and combined with the artificial potential field gradient, positional relationship and state interaction information of each surface unmanned vessel, a cluster collaborative strategy sharing mechanism is constructed to generate collaborative path planning decisions for multiple unmanned vessels. The collaborative path planning decision drives the unmanned surface vessel to perform outbound and return journeys. During the return journey, the optimal return action strategy is invoked, and a closed-loop determination is made in conjunction with a hierarchical reward mechanism to generate a closed-loop path.

2. The method according to claim 1, characterized in that, The status information includes task area grid information, task point location, obstacle distribution, location information of each unmanned surface vessel, and environmental dynamic parameters.

3. The method according to claim 1, characterized in that, The step of combining the multi-agent potential field and establishing an action evaluation function using the artificial potential field gradient and the ε-greedy policy includes: Based on the multi-agent potential field, the resultant force vector of the target point's attractive potential field and the obstacle's repulsive potential field is calculated, and an artificial potential field gradient representing the current motion trend of a single unmanned surface vessel is generated. Based on the artificial potential field gradient, the directional offset of the candidate action is weighted to obtain the weighted artificial potential field gradient. The weighted artificial potential gradient is embedded into the ε-greedy strategy to adjust the action probability and generate an action evaluation function.

4. The method according to claim 1, characterized in that, The dual-Q dynamic adjustment network model includes a Q_task network and a Q_return network. The Q_task network is used to generate an action strategy for the unmanned surface vessel (USV) to perform at least one task point search during the outward journey, and the Q_return network is used to generate an optimal action strategy for the USV to execute the return path during the return journey. The switching condition between the Q_task network and the Q_return network includes a comparison between the real-time distance of the USV and a preset distance. The Q_task network uses the current task point as the source of the attractive potential field, and modifies the action value based on the potential field function constructed from the attractive potential field of the target point and the repulsive potential field of the obstacle, so as to output the optimal action strategy for the outward journey oriented towards the task point. The Q_return network uses the return start point or a preset end point as its attraction potential source, and updates the value of the return action based on a hierarchical reward mechanism for the return phase to generate the optimal return action strategy. The hierarchical reward mechanism includes a return approach reward, a closed-loop completion reward, and a return timeout penalty.

5. The method according to claim 4, characterized in that, The formula for determining the decision criteria of the dual-Q dynamic adjustment network model is as follows: ; MS represents a binary decision variable. When MS=1, it indicates that the target point arrival condition in the closed-loop task is met, which is used to trigger the Q_return mode switching in the dual-Q dynamic adjustment network model. When MS=0, it indicates that the unmanned surface vessel is still in the outbound stage, which is used to trigger the Q_task network. This indicates the real-time position coordinates of the unmanned surface vessel. Represents the spatial coordinates of the final task point; This indicates the Euclidean distance between the current coordinates of the unmanned surface vessel and the final mission point; otherwise, it indicates that when the Euclidean distance between the unmanned surface vessel and the final mission point is greater than the threshold of 0.5, it has not yet entered the return trip determination state and is still in the outbound point search or path search stage.

6. The method according to claim 1, characterized in that, The process of using the action evaluation function and deep Q-learning algorithm to dynamically adjust the dual-Q network model and iteratively update it to output the target optimal action strategy for a single unmanned surface vessel includes: Based on the action evaluation function, the bias probability of each candidate action in the current state is calculated to form a bias-corrected action evaluation result. The action evaluation results are input into the dual-Q dynamic adjustment network model, and the target Q value of the corresponding state-action pair is constructed using a deep Q-learning algorithm; the target Q value includes immediate reward, discounted future reward, and action bias term. The parameters of the dual-Q dynamic adjustment network model are iteratively updated based on the target Q value; After completing the iterative update, select the action with the highest action value and output the target optimal action strategy for a single unmanned surface vessel.

7. The method according to claim 1, characterized in that, The target optimal action strategy output by the dual-Q dynamic adjustment network model, combined with the artificial potential field gradient, positional relationship, and state interaction information of each surface unmanned surface vessel, constructs a cluster collaborative strategy sharing mechanism to generate collaborative path planning decisions for multiple unmanned surface vessels, including: Based on the artificial potential field gradient, positional relationship and state interaction of each surface unmanned vessel, an interaction information set of multiple agents is constructed; the interaction information set includes the attractive and repulsive potential field components, relative distance, relative speed and motion trend of neighboring unmanned vessels; Based on the interaction information set and the target optimal action strategy of the single unmanned surface vessel, a set of cooperative actions of multiple unmanned surface vessels is generated by joint weighting. A cluster collaboration strategy sharing mechanism is constructed based on the aforementioned set of collaborative actions; Based on the aforementioned cluster collaboration strategy sharing mechanism, the action selection of the surface unmanned surface vessel cluster is executed, generating collaborative path planning decisions for multiple unmanned surface vessels.

8. The method according to claim 7, characterized in that, The formula for the collaborative force in the cluster collaborative strategy sharing mechanism is as follows: ； in, Indicates the first The total artificial potential force experienced by a surface unmanned vessel at the current moment; N represents the total number of surface unmanned vessels in the swarm. Indicates unmanned surface vessels unmanned surface vessels Interaction forces between them; Represents the coefficient of the attractive potential field; Indicates the first The target point coordinates of the unmanned surface vessel; Indicates the first The current position coordinates of the unmanned surface vessel; Indicates unmanned surface vessels The Euclidean distance to the corresponding target point; M represents the number of obstacles in the environment; This represents the coefficient of the repulsive potential field; Indicates unmanned surface vessels and obstacles The current distance between them; Indicates unmanned surface vessels A safe distance threshold set for obstacles.

9. A closed-loop path planning system for unmanned surface vessels, characterized in that, The system includes: The acquisition module is used to acquire the status information of multiple unmanned surface vessels (USVs) clusters in the mission area. The potential field construction module is used to construct a multi-agent potential field based on the state information using an artificial potential field algorithm; the multi-agent potential field includes an attractive potential field for the target point and a repulsive potential field for obstacles; The action evaluation module is used to combine the potential field of the multi-agent and establish an action evaluation function using the artificial potential field gradient and the ε-greedy strategy; The dual-Q network construction module is used to build a dual-Q dynamically adjustable network model; The optimization module is used to iteratively update the dual-Q dynamic adjustment network model using the action evaluation function and deep Q learning algorithm, and output the target optimal action strategy for a single unmanned surface vessel. The target optimal action strategy includes the outbound optimal action strategy and the return optimal action strategy. The collaborative strategy sharing module is used to dynamically adjust the target optimal action strategy output by the dual-Q network model based on the dual-Q dynamic adjustment, and combine the artificial potential field gradient, position relationship and state interaction information of each surface unmanned vessel to build a cluster collaborative strategy sharing mechanism to generate collaborative path planning decisions for multiple unmanned vessels. The closed-loop path generation module is used to drive the unmanned surface vessel to perform outbound and return journeys based on the collaborative path planning decision, and to call the optimal return action strategy during the return journey phase. It also combines a hierarchical reward mechanism to determine the closed loop and generate a closed-loop path.

10. A closed-loop path planning system for unmanned surface vessels, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method as described in any one of claims 1-8.

Images