Unmanned ship path generation method based on graph neural network and deep reinforcement learning

Abstract

The application discloses an unmanned ship path generation method based on a graph neural network and deep reinforcement learning, and belongs to the technical field of path planning. The method first builds a path generation scene; sets the environment, action, reward and environment after the unmanned ship performs the action in the scene as parameters of a deep reinforcement learning method; then constructs a memory pool with a fixed capacity for storing experience; then constructs and trains an unmanned ship path generation model, the unmanned ship path generation model adopts a double deep Q network model structure, experience in the memory pool is used as training data, a current value network and a target value network adopt a graph convolutional neural network, and a deviation between the current value network and the target value network is used as a loss function; finally, environment information collected when the unmanned ship sails is input into the model to obtain an action to be performed by the unmanned ship. Simulation experiments prove that the method effectively improves the path planning success rate.

Description

Technical Field

[0001] This invention belongs to the field of trajectory planning technology, and more specifically, relates to a method for generating unmanned surface vessel trajectories based on graph neural networks and deep reinforcement learning. Background Technology

[0002] Surface unmanned surface vessel (USV) trajectory generation in complex sea conditions is a multidisciplinary applied technology, encompassing the pre-calculation, setting, and optimization of USV navigation directions and routes. With the increasing development and utilization of marine resources, USV trajectory generation technology has received continuous attention. For example, when USVs travel to specific locations to perform missions, they need to automatically plan their routes and avoid static and dynamic obstacles such as reefs and other underwater vehicles. Therefore, driven by practical application needs, research on USV trajectory generation methods in complex sea conditions has become a current research hotspot.

[0003] However, traditional path generation algorithms suffer from drawbacks such as strong dependence on environmental information, high computational resource requirements, and inability to perform real-time planning. For example, the A* algorithm suffers from uneven path planning and is not adaptable to complex dynamic environments. The RTT algorithm has some adaptability to complex environments, but it has high requirements for dynamic model constraints. Intelligent optimization algorithms such as ant colony optimization and genetic algorithms, although possessing strong self-learning and adaptive capabilities, still suffer from problems such as initialization optimization, early interruption, and high computational cost.

[0004] Furthermore, factors such as moving obstacles and destinations in a scene often result in complex topological structures, which traditional fully connected neural networks may not handle well. Additionally, factors such as distances or reachability between different nodes in a scene can influence action selection and value estimation, and traditional fully connected neural networks struggle to adequately address these factors. Summary of the Invention

[0005] In view of the above-mentioned defects or improvement needs of existing technologies, this invention provides an unmanned surface vessel trajectory generation method based on graph neural networks and deep reinforcement learning. Its purpose is to solve the technical problems of insufficient real-time performance, low success rate and low computational efficiency of existing technologies in complex and dynamic sea environments.

[0006] To achieve the above objectives, in a first aspect, the present invention provides a training method for an unmanned surface vessel (USV) trajectory generation model based on graph neural networks and deep reinforcement learning, the method comprising:

[0007] A trajectory generation scenario is constructed, which includes an unmanned surface vessel (USV), moving obstacles, and a target location; and the environment in the scenario, the actions performed by the USV, the rewards, and the environment after the USV performs the actions are set as parameters for a deep reinforcement learning method.

[0008] A memory pool with a fixed capacity is constructed to store experiences and the sampling probabilities of those experiences. Each experience includes the environment at a certain moment, the action performed by the unmanned surface vessel (USV), the reward, and the environment after the USV performs the action.

[0009] An unmanned surface vessel (USV) trajectory generation model is constructed and trained. The USV trajectory generation model adopts a dual-depth Q-network model structure, in which the experience in the memory pool is used as training data, the current value network and the target value network adopt graph convolutional neural networks, and the deviation between the current value network and the target value network is used as the loss function. Experience with high sampling probability is selected first to train the USV trajectory generation model.

[0010] Preferably, the environment includes the position coordinates of the unmanned surface vessel, moving obstacles, and target at various times; the actions performed by the unmanned surface vessel include up, down, left, right, and stop; the reward is:

[0011]

[0012] Where, r t d represents the reward at time t; mint =min(d_obs1) t d_obs2 t ,...,d_obsN t ); d_obs1 t d_obs2 t ,...,d_obsN t Let d represent the distances of the unmanned surface vessel (USV) from the N moving obstacles at time t; t d t-1 These represent the distances of the unmanned surface vessel (USV) from the target location at time t and time t-1, respectively; Obs_collided indicates that the USV collided with a moving obstacle; and Des_arrived indicates that the USV arrived at the target location.

[0013] Preferably, in the trajectory generation scenario, when the unmanned surface vessel is in environment s at time t... t At time t, the Q-values ​​of various actions of the unmanned surface vessel are calculated based on the action value estimation network, and the action a to be executed at time t is selected according to the greedy strategy. t And give the unmanned surface vessel a reward. t Simultaneous action a t This leads to the environment being affected by s t Change to s t+1 , elements (s t ,a t ,r t ,s t+1 This constitutes one instance of the experience described above; the action value estimation network is based on a graph convolutional neural network.

[0014] Preferably, the sampling probability of the experience is:

[0015]

[0016] Where, r t s t and a t s' and a' represent the reward, environment, and action at time t; γ represents the environment and action at the next time step after time t; Q represents the discount factor; target This represents the Q-value of the current value network.

[0017] Preferably, the environment is mapped based on experience, and the graph convolutional neural network is used to fit the value Q of the unmanned surface vessel performing different actions in the environment. The mapping specifically involves:

[0018] A graph is constructed based on the environmental information at a given moment, treating the target location, obstacles, and unmanned surface vessel (USV) as nodes. The target location and obstacles are unidirectionally connected to the USV, while the obstacles themselves are bidirectionally connected to each other. The feature vector of each node represents its coordinates. The relationships between nodes are represented by an adjacency matrix A.

[0019]

[0020] Where, the value A in the i-th row and j-th column ij =1 indicates that there exists an edge pointing from node i to node j, A ij =0 indicates that this edge does not exist; the weight of an edge is defined as the inverse square of the distance multiplied by a coefficient, where the coefficient β between the obstacle and the unmanned surface vessel is... OU for:

[0021]

[0022] The coefficient β between the target location and the unmanned surface vessel DU for:

[0023]

[0024] Coefficient between obstacles for:

[0025]

[0026] Where, x O y O The x-coordinate represents the coordinates of obstacle o; U y U Represents the coordinates of the unmanned surface vessel; x D y D Represents the coordinates of the target location; Indicates an obstacle oi The coordinates; Indicates an obstacle o y The coordinates.

[0027] Secondly, the present invention provides a method for generating unmanned surface vessel (USV) tracks based on graph neural networks and deep reinforcement learning, the method comprising:

[0028] When the unmanned surface vessel is navigating, it collects environmental data, including the coordinates of the unmanned surface vessel, the coordinates of obstacles, and the coordinates of the target point, and then constructs a map of the environmental data.

[0029] The mapping data is input into the trained unmanned surface vessel trajectory generation model; the unmanned surface vessel trajectory generation model is trained according to any one or more of the implementation methods of the first aspect above.

[0030] The unmanned surface vessel (USV) trajectory generation model outputs the Q-values ​​of various actions performed by the USV, and the USV selects the actions to be performed based on a greedy strategy.

[0031] Thirdly, the present invention provides a training system for an unmanned surface vessel trajectory generation model based on graph neural networks and deep reinforcement learning, the system comprising:

[0032] The scene building module is used to build a trajectory generation scene, which includes an unmanned surface vessel (USV), moving obstacles, and a target location; and sets the environment in the scene, the actions performed by the USV, the reward, and the environment after the USV performs the actions as parameters for a deep reinforcement learning method.

[0033] The training data selection module is used to construct a memory pool with a fixed capacity to store experience and the sampling probability of the experience. Each experience includes the environment at a certain moment, the action performed by the unmanned surface vessel, the reward, and the environment after the unmanned surface vessel performs the action.

[0034] The model building and training module is used to construct and train the unmanned surface vessel (USV) trajectory generation model. The USV trajectory generation model adopts a dual-deep Q-network model structure, in which the experience in the memory pool is used as training data, the current value network and the target value network adopt graph convolutional neural networks, and the deviation between the current value network and the target value network is used as the loss function. Experience with high sampling probability is selected first to train the USV trajectory generation model.

[0035] Preferably, the environment includes the position coordinates of the unmanned surface vessel, moving obstacles, and target at various times; the actions performed by the unmanned surface vessel include up, down, left, right, and stop; the reward is:

[0036]

[0037] Where, r t d represents the reward at time t;mint =min(d_obs1) t d_obs2 t ,...,d_obsN t ); d_obs1 t d_obs2 t ,...,d_obsN t Let d represent the distances of the unmanned surface vessel (USV) from the N moving obstacles at time t; t d t-1 These represent the distances of the unmanned surface vessel (USV) from the target location at time t and time t-1, respectively; Obs_collided indicates that the USV collided with a moving obstacle; and Des_arrived indicates that the USV arrived at the target location.

[0038] Preferably, the environment is mapped based on experience, and the graph convolutional neural network is used to fit the value Q of the unmanned surface vessel performing different actions in the environment. The mapping specifically involves:

[0039] A graph is constructed based on the environmental information at a given moment, treating the target location, obstacles, and unmanned surface vessel (USV) as nodes. The target location and obstacles are unidirectionally connected to the USV, while the obstacles themselves are bidirectionally connected to each other. The feature vector of each node represents its coordinates. The relationships between nodes are represented by an adjacency matrix A.

[0040]

[0041] Where, the value A in the i-th row and j-th column ij =1 indicates that there exists an edge pointing from node i to node j, A ij =0 indicates that this edge does not exist; the weight of an edge is defined as the inverse square of the distance multiplied by a coefficient, where the coefficient β between the obstacle and the unmanned surface vessel is... OU for:

[0042]

[0043] The coefficient β between the target location and the unmanned surface vessel DU for:

[0044]

[0045] Coefficient between obstacles for:

[0046]

[0047] Where, x O y O The x-coordinate represents the coordinates of obstacle o; U y U Represents the coordinates of the unmanned surface vessel; xD y D Represents the coordinates of the target location; Indicates an obstacle o i The coordinates; Indicates an obstacle o y The coordinates.

[0048] Fourthly, the present invention provides an unmanned surface vessel trajectory generation system based on graph neural networks and deep reinforcement learning, the system comprising:

[0049] An environmental data acquisition unit is used to collect environmental data during the navigation of the unmanned surface vessel (USV). The environmental data includes the coordinates of the USV, the coordinates of obstacles, and the coordinates of the target point. The unit also constructs a map of the environmental data.

[0050] A data input unit is used to input the mapping data into the trained unmanned surface vessel trajectory generation model; the unmanned surface vessel trajectory generation model is trained according to any one or more of the implementation methods of the first aspect above.

[0051] The trajectory calculation unit is used to output the Q values ​​of various actions performed by the unmanned surface vessel (USV) from the trajectory generation model of the USV. The USV selects the actions to be performed based on a greedy strategy.

[0052] In summary, the technical solutions conceived by this invention have the following beneficial effects compared with the prior art:

[0053] (1) This invention proposes a method for generating unmanned surface vessels based on graph neural networks and deep reinforcement learning. It combines the ability of deep reinforcement learning to explore unknown environments with the powerful processing effect of graph convolutional neural networks on complex topological data. Compared with traditional methods, it has advantages such as better handling of scene topology, relationships between nodes, and higher performance and efficiency. Simulation experiments show that the method proposed in this invention has a higher success rate and a shorter planning path length compared with traditional trajectory planning methods.

[0054] (2) Based on the relationship between dynamic obstacles, unmanned surface vessels and target locations, this invention constructs a directed weighted graph and uses a graph neural network to integrate the information of neighboring nodes into the feature representation of the target node, thereby better handling the interdependence and topological relationships between nodes, thus improving the performance and robustness of the algorithm.

[0055] (3) When training the unmanned surface vessel trajectory generation model, this invention selects experience from the memory pool based on the sampling probability of experience to participate in the model training. Experience with a high sampling probability will participate in the model training more frequently, thereby improving the utilization rate of important experience and further enabling more effective use of the information in the experience buffer to accelerate the convergence of the model. Attached Figure Description

[0056] Figure 1 This is a flowchart of the unmanned surface vessel trajectory generation method in an embodiment of the present invention;

[0057] Figure 2 This is a schematic diagram of the scenario for building a trajectory generation scenario in an embodiment of the present invention;

[0058] Figure 3 This is a schematic diagram of the mapping based on environmental information in an embodiment of the present invention;

[0059] Figure 4 This is a schematic diagram of the dual-depth Q-network model structure built in this embodiment of the invention;

[0060] Figure 5 This is a schematic diagram of the effective flight path generated in the verification experiment of this invention. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0062] like Figure 1 The diagram shows a flowchart of the training method for the unmanned surface vessel trajectory generation model based on graph neural networks and deep reinforcement learning in this invention, which specifically includes the following steps:

[0063] S1: Construct a typical scenario for trajectory generation that includes unmanned surface vessels, moving obstacles, and a target location, as shown in the attached diagram. Figure 2 As shown. To ensure the model's generalization and practicality, the starting and ending positions of dynamic obstacles and the starting and destination positions of unmanned surface vessels (USVs) in the scenario are set to random values ​​within a fixed range. The simulated marine environment is set as a square area with a side length of 20m. Meanwhile, to enhance the actual combat capability of the USVs, a certain degree of active blocking capability is added to the dynamic obstacles: when the dynamic obstacles do not detect the USVs within 4m, they will use the A* algorithm for path planning and the ORCA algorithm for simple collision avoidance; however, when the USVs appear within 4m of the dynamic obstacles, the dynamic obstacles will change their original movement logic and move towards the USVs, interfering with their actions.

[0064] S2: To enable the unmanned surface vessel (USV) to perceive state changes over a period of time, thereby better judging the current state and making decisions, the USV's observation space is set as the set of two-dimensional coordinates of the USV, the target location, and various obstacles in three time-stamped scenarios, defined as:

[0065]

[0066] Among them, the status of unmanned surface vessels is s U,t It consists of its two-dimensional coordinates at times t-2, t-1, and t, and is defined as follows:

[0067] s U,t =(p U,t-2 ,p U,t-1 ,p U,t )

[0068] p U,t =(x U,t ,y U,t )

[0069] Where, x U,t ,y U,t and are the horizontal and vertical coordinates of the unmanned surface vessel at time t, respectively.

[0070] The motion state of the i-th dynamic obstacle in the scene is also composed of its two-dimensional coordinates at times t-2, t-1, and t, defined as:

[0071]

[0072]

[0073] in, and are the horizontal and vertical coordinates of the i-th dynamic obstacle at time t, respectively.

[0074] The status of the target location s D,t Its two-dimensional coordinates at time t are represented as follows:

[0075] s D,t =(x D,t ,y D,t )

[0076] Where, x D,t ,y D,t and represent the x and y coordinates of the target at time t, respectively.

[0077] In order to reduce the training cost of reinforcement learning algorithms, this embodiment simplifies the action space to five actions: up, down, left, right, and stop. The motion speed of the unmanned surface vessel is set to a fixed value of 1 m / s.

[0078] In typical trajectory generation scenarios, the environmental reward r t Set to:

[0079]

[0080] Where, r td represents the reward at time t; mint =min(d_obs1) t d_obs2 t ,...,d_obsN t ); d_obs1 t d_obs2 t ,...,d_obsN t Let d represent the distances of the unmanned surface vessel (USV) from the N moving obstacles at time t; t d t-1 These represent the distances of the unmanned surface vessel (USV) from the target location at time t and time t-1, respectively; Obs_collided indicates that the USV collided with a moving obstacle; and Des_arrived indicates that the USV arrived at the target location.

[0081] S3: To eliminate the strong correlation between actions in reinforcement learning and enhance the stability of reinforcement learning training, a memory pool with a fixed capacity was constructed to store samples and random sampling during model training. First, the memory pool P was initialized, and its fixed capacity was defined as N. During the trajectory generation process, when the unmanned surface vessel is in a certain environment s at time t... t At time t, the Q-value of each action is calculated using the action value estimation network, and the action a to be executed at time t is selected according to the ε-greedy policy. t Action a t This will cause the environmental state to change from s t Change to s t+1 And give the unmanned surface vessel a reward. t These elements together constitute an experience (s t ,a t ,r t ,s t+1 This data will be stored in the memory pool. Furthermore, to accelerate training, sampling probabilities are determined based on the temporal difference error (TD error) to enable priority experience replay.

[0082] Prioritized Experience Replay (PER) is an augmentation algorithm designed to improve sample efficiency and convergence speed. In traditional Experience Replay, the agent stores its experienced state transitions in a replay buffer and randomly selects samples from it for training. A problem with this approach is that each sample is treated as equally important; that is, regardless of its contribution to the agent's training, each sample is selected with the same probability. In contrast, Prioritized Experience Replay introduces a priority system to select important samples for training. The priority can be calculated based on the sample's Temporal Difference Error (TD) or other metrics, where TD measures the difference between the agent's predicted value and the observed value under the current policy. Higher-priority samples are selected more frequently, increasing their training probability and improving the utilization of important samples. For example, for a sample (s... t ,a t ,r t ,s t+1 In the dual-depth Q-network model, the TD error is:

[0083]

[0084] Where, r t s t and a t s' and a' represent the reward, environment, and action at time t; γ represents the environment and action at the next time step after time t; Q represents the discount factor; target This represents the Q-value of the current value network.

[0085] During training, Priority Experience Replay employs an importance sampling weight to adjust the training weight of each sample. These weights are calculated based on the sample's priority, giving greater emphasis to higher-priority samples. By adjusting the sample weights, Priority Experience Replay can more effectively utilize the information in the experience buffer, accelerating the algorithm's convergence.

[0086] S4: Construct a graph of the environmental information acquired by the unmanned surface vessel (USV), and use a graph convolutional neural network (GNN) to fit the value of different actions performed by the USV at different times. A graph neural network is a machine learning model for processing graph data, inspired by graph theory and neural networks. Its goal is to learn about nodes and edges in a graph to perform tasks such as node classification, predicting node attributes, and predicting links. In trajectory generation tasks, compared to traditional reinforcement learning algorithms, graph-based methods can better handle the impact of factors such as moving obstacles and destinations on action selection and value estimation. Furthermore, because graph neural networks can consider dependencies between nodes, they can better handle complex topological structures in the scene.

[0087] The composition of the scene at different times is shown in the attached figure. Figure 3 As shown, the target location and each obstacle are unidirectionally connected to the unmanned surface vessel (USV), while the obstacles are bidirectionally connected to each other. For the graph consisting of the target location D, the USV U, and obstacles O1, O2, and O3, the eigenvector of each node is its coordinates (x, y), and the relationships between nodes are represented by an adjacency matrix A.

[0088]

[0089] Among them, A ij =1 indicates that there exists an edge pointing from node i to node j, A ij =0 indicates that this edge does not exist; the weight of an edge is defined as the inverse square of the distance multiplied by a coefficient, where the coefficient β between the obstacle and the unmanned surface vessel is... OU for:

[0090]

[0091] The coefficient β between the target location and the unmanned surface vessel DU for:

[0092]

[0093] Coefficient between obstacles for:

[0094]

[0095] Where, x O y O The x-coordinate represents the coordinates of obstacle o; U y U Represents the coordinates of the unmanned surface vessel; x D y D Represents the coordinates of the target location; Indicates an obstacle o i The coordinates; Indicates an obstacle oy The coordinates.

[0096] The specific network model parameter settings are shown in Table 1:

[0097] Table 1

[0098] Network Structure Input Dimensions Output Dimension GCN_1 5*2 5*16 GCN_2 5*16 5*8 Reshape 5*8 1*40 FC 40 5

[0099] The model's input is scene state information s consisting of five two-dimensional coordinates, and the output is the value estimate (a, Q) of each action of the unmanned surface vessel in the current state.

[0100] Step S5 describes a method that includes: constructing and training a dual-depth Q-network model with a priority playback mechanism by combining environmental mapping; using the environmental state as the model input and the actions of the unmanned surface vessel as the model output; and using the deviation between the actual action value function Q and the fitted action value function Q as the loss function to train the model. The structure of the constructed dual-depth Q-network model is shown in the attached figure. Figure 4 As shown, both the current value network and the target value network are graph neural network models built in step S4. During the initialization phase, both networks are set up as neural networks with the same structure and weights, and can be initialized at any time. When the experience pool reaches a predetermined amount, the network update process begins: a batch of experiences is randomly sampled from the experience replay buffer, including the state, action, next state, and reward. The target value network is used to calculate the maximum Q-value Q of the next state. target_next And use it to update the Q value of the current value network Q. target Target Q value Q target The calculation method is as follows:

[0101] Q target =r t +α*Q target_next

[0102] Here, α is the discount factor, which balances immediate and future rewards to learn a better policy. To ensure the stability of the target network, the parameters of the current value network need to be periodically copied to the target value network. The proposed model uses two neural networks to separate the calculation of the target Q-value from the policy of selecting actions, avoiding the overestimation phenomenon in Q-learning. Through experience replay, the proposed model can also reduce data correlation and improve training stability. By periodically updating the target network, the proposed model can maintain tracking of the optimal policy and further improve training performance. The hyperparameter settings of the model during training are shown in Table 2.

[0103] Table 4

[0104] hyperparameters Setting value Learning rate 0.005 Batch_size 256 Buffer_size 40000 Discount rate 0.99

[0105] The method described in step S6 includes: when the unmanned surface vessel (USV) is navigating, using a trained model, it selects the action to be performed based on the environmental state at different times according to a greedy strategy, thereby generating a trajectory. The USV acquires the scene state s at time t. t Then, it is used as the input to the model in step S5, and the action a that the unmanned surface vessel should perform at this time is calculated. t And the corresponding actions are performed by unmanned surface vessels to generate a flight path.

[0106] To verify the effectiveness of the proposed unmanned surface vessel trajectory generation method, this implementation case combines actual simulation scenarios for model training and performance verification.

[0107] The specific steps are as follows:

[0108] (1) Scene Construction

[0109] A typical trajectory generation scenario was constructed, incorporating unmanned surface vessels (USVs), moving obstacles, and target locations. To ensure the model's generalization and practicality, the start and end points of the dynamic obstacles, as well as the start and destination points of the USVs, were set to random values ​​within a certain range. Furthermore, to enhance the USVs' actual combat capabilities, a certain degree of active blocking capability was added to the dynamic obstacles: when a USV appears within a certain range of a dynamic obstacle, the obstacle will change its original movement logic and move towards the USV.

[0110] (2) Model construction and training

[0111] The environmental information acquired by the unmanned surface vessel (USV) at different times is mapped, and a graph convolutional neural network is used to fit the value of different actions performed by the USV at different times. Based on this, a dual-depth Q-network model with a priority playback mechanism is built and trained in conjunction with the environmental mapping. The environmental state is used as the model input, the USV's actions are used as the model output, and the deviation between the actual action value function and the fitted action value function is used as the loss function to train the model.

[0112] (3) Track generation

[0113] When an unmanned surface vessel (USV) is navigating, after acquiring the scene state at time t, it is used as the input to the trained model. The USV then calculates the action it should take at that time and selects the action to be taken according to a greedy strategy, thereby generating a trajectory.

[0114] Two comparison methods were used to evaluate the validity of this study. The results of 100 randomized experiments are shown in Table 5.

[0115] Table 5

[0116] method Success rate of trajectory planning Path length Artificial potential field method 73％ 16.88m Dueling DQN 81％ 18.49m PER-DDQN 83％ 18.86m Method of the present invention 91％ 17.82m

[0117] Experimental results show that the method proposed in this invention further improves the success rate of generating effective tracks, while effectively reducing the path length, and has the highest success rate in the entire path planning task.

[0118] Therefore, this invention, based on the characteristics of reinforcement learning, deep neural networks, and graph neural networks, achieves real-time trajectory generation for unmanned surface vessels (USVs) in complex marine environments, effectively solving the problems of speed and stability in trajectory planning. The method proposed in this invention is based on the ideas of reinforcement learning, deep learning, and graph neural networks. Using the trajectory generation simulation of USVs in a dynamic marine environment as an experimental example, it can be extended to underwater vehicles, underwater robots, and other unmanned equipment in practical applications, demonstrating good engineering applicability.

[0119] The above content is readily understood by those skilled in the art. The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A training method for an unmanned surface vessel (USV) trajectory generation model based on graph neural networks and deep reinforcement learning, characterized in that, The method includes the following steps: A trajectory generation scenario is constructed, which includes an unmanned surface vessel (USV), moving obstacles, and a target location; and the environment in the scenario, the actions performed by the USV, the rewards, and the environment after the USV performs the actions are set as parameters for a deep reinforcement learning method. A memory pool with a fixed capacity is constructed to store experiences and the sampling probabilities of those experiences. Each experience includes the environment at a certain moment, the action performed by the unmanned surface vessel (USV), the reward, and the environment after the USV performs the action. An unmanned surface vessel (USV) trajectory generation model is constructed and trained. The USV trajectory generation model adopts a dual deep Q-network model structure, in which the experience in the memory pool is used as training data, the current value network and the target value network adopt a graph convolutional neural network, and the deviation between the current value network and the target value network is used as the loss function. Experience with high sampling probability is selected to train the USV trajectory generation model. The environment is mapped based on experience, and the graph convolutional neural network is used to fit the value Q of the unmanned surface vessel performing different actions in the environment. The mapping process specifically involves: A graph is constructed based on the environmental information at a given moment, treating the target location, obstacles, and unmanned surface vessel (USV) as nodes. The target location and obstacles are unidirectionally connected to the USV, while the obstacles themselves are bidirectionally connected to each other. The feature vector of each node represents its coordinates. The relationships between nodes are represented by an adjacency matrix A. Among them, the i OK j Column values This represents the existence of nodes. i Pointing to node j The edge, This indicates that the edge does not exist; the weight of the edge is defined as the inverse square of the distance multiplied by a coefficient, where the coefficient between the obstacle and the unmanned surface vessel is... for: Coefficient between target location and unmanned surface vessel for: Coefficient between obstacles for: in, Indicates obstacles o The coordinates; Represents the coordinates of the unmanned surface vessel; Represents the coordinates of the target location; Indicates obstacles The coordinates; Indicates obstacles The coordinates.

2. The method according to claim 1, characterized in that, The environment includes the unmanned surface vessel (USV), moving obstacles, and the target's position coordinates at various times; the actions performed by the USV include up, down, left, right, and stop; the reward is: in, Represents the reward at time t; ; These represent unmanned surface vessels in t Time Distance The distance of the moving obstacle; These represent unmanned surface vessels in t time, t- Distance from the target location at any given moment; This indicates that the unmanned surface vessel collided with a moving obstacle; This indicates that the unmanned surface vessel has arrived at the target location.

3. The method according to claim 1, characterized in that, In the aforementioned trajectory generation scenario, when the unmanned surface vessel is in t Always in the environment At this point, the Q-values ​​of various actions of the unmanned surface vessel are calculated based on the action value estimation network, and a greedy strategy is used to select... t Actions that should be performed at any time And give the unmanned surface vessel a reward. Simultaneous action Caused the environment to be Change to , elements ( , , , This constitutes one instance of the experience described above; the action value estimation network is based on a graph convolutional neural network.

4. The method according to claim 1, characterized in that, The sampling probability of the experience is: in, , and This represents the reward, environment, and action at time t; and This represents the environment and actions one time after time t; Indicates the discount factor; This represents the Q-value of the current value network.

5. A method for generating unmanned surface vessel (USV) tracks based on graph neural networks and deep reinforcement learning, characterized in that, The method includes: When the unmanned surface vessel is navigating, it collects environmental data, including the coordinates of the unmanned surface vessel, the coordinates of obstacles, and the coordinates of the target point, and then constructs a map of the environmental data. The mapping data is input into the trained unmanned surface vessel trajectory generation model; the unmanned surface vessel trajectory generation model is trained by the method according to any one of claims 1-4; The unmanned surface vessel (USV) trajectory generation model outputs the Q-values ​​of various actions performed by the USV, and the USV selects the actions to be performed based on a greedy strategy.

6. A training system for an unmanned surface vessel trajectory generation model based on graph neural networks and deep reinforcement learning, characterized in that, The system includes: The scene building module is used to build a trajectory generation scene, which includes an unmanned surface vessel (USV), moving obstacles, and a target location; and sets the environment in the scene, the actions performed by the USV, the reward, and the environment after the USV performs the actions as parameters for a deep reinforcement learning method. The training data selection module is used to construct a memory pool with a fixed capacity to store experience and the sampling probability of the experience. Each experience includes the environment at a certain moment, the action performed by the unmanned surface vessel, the reward, and the environment after the unmanned surface vessel performs the action. The model building and training module is used to construct and train the unmanned surface vessel (USV) trajectory generation model. The USV trajectory generation model adopts a dual deep Q-network model structure, in which the experience in the memory pool is used as training data, the current value network and the target value network adopt a graph convolutional neural network, and the deviation between the current value network and the target value network is used as the loss function. Experience with high sampling probability is selected to train the USV trajectory generation model. The environment is mapped based on experience, and the graph convolutional neural network is used to fit the value Q of the unmanned surface vessel performing different actions in the environment. The mapping process specifically involves: A graph is constructed based on the environmental information at a given moment, treating the target location, obstacles, and unmanned surface vessel (USV) as nodes. The target location and obstacles are unidirectionally connected to the USV, while the obstacles themselves are bidirectionally connected to each other. The feature vector of each node represents its coordinates. The relationships between nodes are represented by an adjacency matrix A. Among them, the i OK j Column values This represents the existence of nodes. i Pointing to node j The edge, This indicates that the edge does not exist; the weight of the edge is defined as the inverse square of the distance multiplied by a coefficient, where the coefficient between the obstacle and the unmanned surface vessel is... for: Coefficient between target location and unmanned surface vessel for: Coefficient between obstacles for: in, Indicates obstacles o The coordinates; Represents the coordinates of the unmanned surface vessel; Represents the coordinates of the target location; Indicates obstacles The coordinates; Indicates obstacles The coordinates.

7. The system according to claim 6, characterized in that, The environment includes the unmanned surface vessel (USV), moving obstacles, and the target's position coordinates at various times; the actions performed by the USV include up, down, left, right, and stop; the reward is: in, Represents the reward at time t; ; These represent unmanned surface vessels in t Time Distance The distance of the moving obstacle; These represent unmanned surface vessels in t time, t- Distance from the target location at any given moment; This indicates that the unmanned surface vessel collided with a moving obstacle; This indicates that the unmanned surface vessel has arrived at the target location.

8. A trajectory generation system for unmanned surface vessels based on graph neural networks and deep reinforcement learning, characterized in that, The system includes: An environmental data acquisition unit is used to collect environmental data during the navigation of the unmanned surface vessel (USV). The environmental data includes the coordinates of the USV, the coordinates of obstacles, and the coordinates of the target point. The unit also constructs a map of the environmental data. A data input unit is used to input the mapping data into the trained unmanned surface vessel trajectory generation model; the unmanned surface vessel trajectory generation model is trained by the method according to any one of claims 1-4; The trajectory calculation unit is used to output the Q values ​​of various actions performed by the unmanned surface vessel (USV) from the trajectory generation model of the USV. The USV selects the actions to be performed based on a greedy strategy.

Images