Autonomous berthing method for a surface unmanned vehicle with twin water-jet propellers

Abstract

The application discloses a kind of double water-jet propeller water surface unmanned ship autonomous berthing methods, by defining state space, action space, reward function and neural network structure element, using reinforcement learning method, according to the current radar observation information of the unmanned ship, target berth position information, unmanned ship current speed information, obtain unmanned ship control output information, avoid the problems that traditional manual design berthing collision avoidance rules, physical motion model, machine learning method, environment model are greatly influenced by environment, many and complex intermediate steps, effectively improve the computing efficiency and one-time planning berthing success rate, to improve the berthing efficiency as a whole.

Description

Technical Field

[0001] This invention relates to the field of unmanned surface vessel technology, specifically to a method for autonomous berthing of a surface unmanned vessel with dual waterjet propulsion. Background Technology

[0002] Unmanned surface vessels (USVs) are being used more and more widely in surface operations. USVs with dual waterjet propulsion have significant advantages over other types of USVs, especially those using traditional propellers or single waterjet propulsion systems.

[0003] However, current traditional autonomous berthing methods for unmanned surface vessels (USVs) involve several important steps: environmental perception, which involves collecting information about the surrounding environment through sensors; berthing modeling, which involves establishing a mathematical model of the USV and the surrounding environment of the target berth based on the environmental perception data; berthing path planning, which involves designing a reasonable path using the modeling data, and the path planning is implemented through algorithms (such as the A* algorithm, dynamic window method, etc.); and USV control, which involves performing actual operations based on the path planning results and sending movement commands to the USV.

[0004] Such methods not only consume significant time and computational resources but also make timely task completion difficult. Furthermore, accurately constructing a model of the surrounding environment becomes increasingly challenging as the aquatic environment becomes more complex. Simultaneously, traditional path planning algorithms rely on manually set heuristic functions. While these functions can provide effective guidance in specific scenarios, their design is typically environment-specific, resulting in poor adaptability to new environments. This lack of generalization ability means that applying these algorithms to unfamiliar environments requires substantial effort to redesign the heuristic functions. Therefore, improving the efficiency of berthing systems and reducing reliance on manually set heuristic functions and accurate environmental modeling has become a current research hotspot. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a method for autonomous berthing of unmanned surface vessels with dual waterjet propulsion based on reinforcement learning by rationally constructing various elements of reinforcement learning, thereby achieving autonomous, rapid, efficient and stable berthing.

[0006] To achieve the above objectives, this invention designs an autonomous berthing method for a dual-jet propulsion unmanned surface vessel. This autonomous berthing method is based on reinforcement learning, defining elements such as state space, action space, reward function, and neural network structure (or neural network model). It utilizes reinforcement learning based on the unmanned vessel's current radar observation information. t Target berth location information g t Current speed information of the unmanned surface vessel (v) t Obtain unmanned surface vessel control output information F t(i.e., the output of the model method), where the distribution information of obstacles observed by radar and the location information of the target berth are also called environmental information, the velocity information includes linear velocity and angular velocity information, and the output information includes the thrust magnitude of the two thrusters;

[0007]

[0008] The unmanned surface vessel's state space state s t =[o t ||g t ||v t ], where "||" represents concatenation between vectors;

[0009] The unmanned surface vessel's (USV) action space includes a discrete action space or a continuous action space. The discrete action space includes at least seven actions, namely: (N, N), (N / 2, N / 2), (N / 2, N), (N, N / 2), (N, 0), (0, N), (0,0), where N is the maximum thrust of the thruster. The element on the left in the parentheses is the thrust of the left thruster, and the element on the right is the thrust of the right thruster. The different thrusts of the two thrusters can control the USV's movement and / or turning on the sea surface.

[0010] The reward function

[0011]

[0012] After the unmanned surface vessel (USV) performs the corresponding action, the reward it should receive is determined based on its status: when the USV safely reaches the target berth and docks safely, it receives a positive reward. The reward is set relatively high to inform the unmanned surface vessel (USV) of the mission's ultimate objective; a penalty is imposed when the USV collides with any obstacle after performing the corresponding action. The value is a negative number, which is used to make the unmanned surface vessel realize that it has just completed a "bad" action;

[0013] When the unmanned surface vessel (USV) fails to complete the docking task and does not collide with an obstacle, the reward it receives consists of three parts. The first part is the distance control item. :

[0014]

[0015] in, This indicates the distance between the unmanned surface vessel and the target berth at the current moment. This indicates the distance between the unmanned surface vessel and the target berth at the previous moment;

[0016] The second part is the speed control item. This is used to control the speed of the unmanned surface vessel, and its expression is as follows:

[0017]

[0018] in v It's the speed of the unmanned surface vessel. l This is a minimum distance threshold used to prevent the distance between the unmanned surface vessel and the target berth from becoming infinitely small;

[0019] The third part is the thrust control items. This represents the thrust difference between two consecutive moments, and its function is to control the stability of the unmanned surface vessel. The expression is as follows:

[0020]

[0021] in, and These represent the thrust magnitudes of the left and right thrusters at the current moment;

[0022] at last, , , This is a hyperparameter.

[0023] Furthermore, the input layer of the neural network structure is a 364-dimensional input vector, which contains 360-dimensional lidar information, 2-dimensional target berth location information, and 2-dimensional unmanned surface vessel current speed, followed by 3 fully connected layers.

[0024] Furthermore, the training strategy for the berthing method model is in rounds, with no less than 1000 rounds; in each round, the initial position of the unmanned surface vessel and the position of the target berth are randomly generated, and the distribution of obstacles is also randomly distributed.

[0025] The conditions for starting a new training round at the end of each round are as follows: the unmanned surface vessel (USV) safely berths, completes the berthing task, and does not collide with any obstacles; the USV collides with an obstacle; or the USV fails to complete the task within a certain number of timesteps.

[0026] Furthermore, the virtual environment for model training includes a water-based virtual environment. In each different task, the initial position of the unmanned surface vessel and the position of the target berth are randomly generated, and the distribution of obstacles is also randomly generated.

[0027] Furthermore, the goal of the reinforcement learning is to maximize the cumulative discount reward G in a single round. t :

[0028]

[0029] Here, γ is a discount factor, mainly used to reduce the value of future rewards and prevent G from being discounted. t Infinitely large.

[0030] Furthermore, the motion model of the unmanned surface vessel (USV) includes a three-degree-of-freedom model, where the state variable η represents the USV's position and orientation in the inertial coordinate system.

[0031]

[0032] Where x and y represent the position of the unmanned surface vessel on the sea surface, This indicates the heading direction of the unmanned surface vessel (USV), i.e., the direction the USV is facing.

[0033] Define the velocity variable of the unmanned surface vessel: V = [uvw] T (The superscript T here indicates transpose), where u is the longitudinal velocity of the unmanned surface vessel, that is, the speed at which the unmanned surface vessel moves along its own orientation; v is the lateral velocity of the unmanned surface vessel, that is, the speed at which the unmanned surface vessel moves along a direction perpendicular to its own orientation; w is the yaw rate of the unmanned surface vessel, that is, the rotational speed of the unmanned surface vessel, which represents the rotational speed of the unmanned surface vessel about its axis perpendicular to the sea surface.

[0034] Therefore, the kinematic equations can be defined as:

[0035]

[0036] in, J represents the state changes of the unmanned surface vessel (USV), specifically, the changes in its position and orientation relative to the sea surface over time. J is a matrix representing the transformation relationship between the body coordinate system and the inertial coordinate system, and its form is as follows:

[0037]

[0038] Furthermore, the reinforcement learning method includes model-based or model-free methods; the model-based method calculates the state value function and / or Q-value function based on the probability of transition between each state to predict the optimal action policy; the model-free method includes value-based methods, policy-based methods, or the actor-critic method.

[0039] The advantages and beneficial effects of this invention are as follows: Compared with traditional methods, this method has the following advantages: First, it avoids the use of manually designed rules (docking and collision avoidance rules) in traditional methods. These rules are often customized for specific environments and perform differently in different environments, requiring redesign. Second, the deep reinforcement learning-based model can learn from high-dimensional domains, whether from sensor observations or the physical model of the unmanned surface vessel (USV), which is very challenging for traditional heuristic methods. Third, compared with other machine learning algorithms, reinforcement learning is more suitable for problems with complex reward structures and a strong need for sequential interaction with the environment, such as the autonomous docking problem of USVs. Fourth, the reinforcement learning-based method can omit all intermediate steps in traditional methods, such as building environmental models and path planning, eliminating the need for manually set heuristic functions in traditional path planning, thus greatly improving task efficiency. This invention does not rely on a precise environmental model and allows the USV to directly select appropriate actions based on current sensor observations. Fifth, by defining variables reasonably, the computational dimensionality is reduced, thereby further improving computational efficiency. Attached Figure Description

[0040] Figure 1 The learning process of an agent based on reinforcement learning is shown in the figure.

[0041] Figure 2 This is a diagram of the neural network structure used in this invention;

[0042] Figure 3 It is the average reward curve obtained by the unmanned surface vessel during the training process. Detailed Implementation

[0043] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and examples. The following examples are only used to more clearly illustrate the technical solutions of the present invention and should not be construed as limiting the scope of protection of the present invention.

[0044] Reinforcement learning is an important branch of machine learning. It primarily studies how to learn relevant experiences by obtaining feedback from interactions. Inspired by observations of human learning methods, reinforcement learning is considered the machine learning algorithm closest to human learning. When humans learn certain behaviors or skills, they typically interact with the environment to obtain rewards, ultimately forming experience. In short, reinforcement learning can be viewed as a newborn "infant" with no knowledge or abilities about the world, learning relevant skills and understanding what constitutes "correct" behavior through exploration and feedback from the world (such as punishment or reward). In reinforcement learning, the entity that learns the relevant behaviors is called an agent.

[0045] During the learning process, an agent explores its environment by taking a series of actions and receiving feedback from these actions, forming experience. Reinforcement learning stores these actions and feedback for further learning. The agent's goal is to use the learned behaviors to obtain more rewards, thereby reinforcing its confidence in performing "good" behaviors. This process is the origin of the term "reinforcement." In the past, people used parameterized distributions to store the experiences learned by agents. With the development of deep learning, parameterized distributions have been replaced by deep neural networks, which is the origin of the term "deep reinforcement learning."

[0046] like Figure 1 As shown, the core idea of ​​reinforcement learning is that an agent learns from the environment, executes actions based on the state of the environment, and is guided to perform better actions based on the feedback from the environment. This leads to the five core elements of reinforcement learning:

[0047] An "agent" is the core component of a robot. It's the entity that interacts with and explores its environment. Through reinforcement learning, the entity that learns relevant capabilities becomes the agent. For example, in an autonomous docking mission, the unmanned surface vessel (USV) is the agent.

[0048] "Environment" refers to the environment in which an intelligent agent exists. There is no precise, rigorous definition of the environment; it primarily refers to the environment that the intelligent agent is expected to explore. For example, in an autonomous berthing mission, the objective marine environment in which the unmanned surface vessel (USV) operates, including obstacles and berths, constitutes the environment.

[0049] "State" refers to the state of an agent within its environment at any given moment. In many tasks, it's impossible to obtain the agent's global state. For example, in the autonomous docking task of an unmanned surface vessel (USV), the state of the USV within the entire ocean environment cannot be obtained. Therefore, in many reinforcement learning-based robotic tasks, "observation" from the robot's sensors is used instead of "state." The observation range of the sensors equipped on the agent is very limited; what the agent can perceive is considered observation.

[0050] "Action" refers to the action performed by the agent. This action is used to interact with the environment; it can be considered a "bridge" between the agent and the environment. In reinforcement learning, actions can be described in various ways. For example, discrete numerical values ​​can represent specific actions, such as "0" representing the robot moving forward and "1" representing moving backward. Actions can also be represented directly by agent control commands, such as the command [a, b] controlling the two thrusters of a dual-jet propulsion surface vessel, where "a" represents the thrust of the left thruster and "b" represents the thrust of the right thruster. The way actions are expressed depends on the type of agent.

[0051] It is worth noting that while actions serve as the medium for interaction between an agent and its environment, actions do not necessarily affect the agent's state in the next moment. For example, whether a robot opens or closes an umbrella will not affect whether it will rain in the next moment. However, in many cases, actions will affect the state in the next moment. For instance, the different thrusts of the two thrusters of an unmanned surface vessel will affect its movement and ultimately its state in the marine environment in the next moment.

[0052] "Reward" is sometimes translated as "reward value." It refers to the environmental feedback received by an agent after performing an action at a certain moment. This value is used to describe the "goodness" or "badness" of the action. If the agent performs an action "good," then the agent may receive a large reward value; if the agent's action is "bad," then it will receive a penalty, i.e., a very small reward.

[0053] The general process of reinforcement learning is as follows: at the current moment, the agent selects an appropriate action based on the current state. After the action is executed, the agent receives a corresponding reward from the environment and enters the next state. It is worth noting that the action performed at the current moment may affect all future states, actions, and rewards. Therefore, the reward obtained by the agent after taking the current action cannot be used as the sole criterion for judging whether the current action is "good" or "bad." See Deep Q Network (DQN) for details (CHUY, WU Z, ZHU X, YUE Y, LIM EG, PAOLETTI P. Self-supervised dock poseestimator for unmanned surface vehicles autonomous docking[C] / / 2024 10th International Conference on Mechatronics and Robotics Engineering (ICMRE). Milan, Italy, 2024: 189–194.).

[0054] Example 1:

[0055] This invention relates to a method for autonomous berthing of an unmanned surface vessel with dual waterjet propulsion, such as... Figure 1 As shown, the autonomous docking method is based on reinforcement learning. It defines elements such as state space, action space, reward function, and neural network structure (or neural network model), and utilizes reinforcement learning based on the current radar observation information of the unmanned surface vessel. t Target berth location information g t Current speed information of the unmanned surface vessel (v) t Obtain unmanned surface vessel control output information F t (i.e., the output of the model method), where the distribution information of obstacles observed by radar and the location information of the target berth are also called environmental information, the velocity information includes linear velocity and angular velocity information, and the output information includes the thrust magnitude of the two thrusters;

[0056]

[0057] The unmanned surface vessel's state space state s t =[o t ||g t ||v t ], where "||" represents concatenation between vectors; in this embodiment g t The coordinates used are polar coordinates relative to the target berth in the unmanned surface vessel coordinate system, i.e., g. t =[d t ,α t ], d t Let α be the polar radius in polar coordinates. t The polar angle is the polar coordinate; the position information of the target berth describes the attitude information of the unmanned surface vessel relative to the target berth; in actual use, the position of the unmanned surface vessel and the target berth are both based on the world coordinate system, so in this invention, it is necessary to obtain the position information of the target berth in the coordinate system of the unmanned surface vessel through coordinate system transformation.

[0058] Let {X} w , Y w} represents the coordinates of the unmanned surface vessel in the world coordinate system, {X R , Y R} represents the coordinate system of the unmanned surface vessel; where X R The current velocity direction of the unmanned surface vessel is shown in the Z-axis direction, which points outwards and is perpendicular to the water surface. R Determined by the right-hand rule; G(x) g , y g The coordinates () represent the location of the target berth in the world coordinate system, which needs to be converted to the unmanned surface vessel coordinate system.

[0059]

[0060] Where x and y are the coordinate components of the target berth in the world coordinate system, respectively. t y t Let θ be the coordinates of the unmanned surface vessel (USV) itself, and θ be the heading angle of the USV. The final solution is:

[0061]

[0062]

[0063] Convert to polar coordinates:

[0064]

[0065]

[0066] The unmanned surface vessel's (USV) action space includes a discrete action space or a continuous action space. In this embodiment, the discrete action space includes at least seven actions, namely: (N, N), (N / 2, N / 2), (N / 2, N), (N, N / 2), (N, 0), (0,N), (0,0), where N is the maximum thrust of the thruster. The element on the left in the parentheses is the thrust of the left thruster, and the element on the right is the thrust of the right thruster. The movement and / or turning of the USV on the sea surface can be controlled according to the different thrusts of the two thrusters. Generally, the turning of the USV is controlled by the magnitude of the two thrusters.

[0067] The reward function

[0068]

[0069] After the unmanned surface vessel (USV) performs the corresponding action, the reward it should receive is determined based on its status: when the USV safely reaches the target berth and docks safely, it receives a positive reward. The reward is set relatively high to inform the unmanned surface vessel (USV) of the mission's ultimate objective; a penalty is imposed when the USV collides with any obstacle after performing the corresponding action. The value is a negative number, which is used to make the unmanned surface vessel realize that it has just completed a "bad" action;

[0070] When the unmanned surface vessel (USV) fails to complete the docking task and does not collide with an obstacle, the reward it receives consists of three parts. The first part is the distance control item. :

[0071]

[0072] in, This indicates the distance between the unmanned surface vessel and the target berth at the current moment. This indicates the distance between the unmanned surface vessel and the target berth at the previous moment. Therefore, This represents the distance between the unmanned surface vessel (USV) and the target berth at the previous moment minus the distance between the USV and the target berth at the current moment. It serves two purposes: first, it encourages the USV to move towards the target berth; second, it provides a dense reward system, meaning the USV receives a corresponding reward for every action it takes at any given moment. If the rewards received by the agent are very sparse—meaning that the agent receives no reward or penalty for its actions most of the time—it will have many negative impacts on model training. These include slow learning processes that prevent the agent from promptly mastering the correct strategy, delayed reward signals that reduce learning efficiency, difficulty in finding a balance between "exploration" (trying new behaviors) and "exploitation" (executing known optimal behaviors) (overexploration may lead to low learning efficiency, while overexploitation may prevent the agent from discovering better strategies), gradient sparsity and update difficulties affecting the efficiency of policy optimization during agent training, and the risk of getting trapped in local optima and insufficient exploration.

[0073] The second part is the speed control item. This is used to control the speed of the unmanned surface vessel (USV). During berthing missions, to ensure mission safety, it is necessary to maintain an appropriate speed for the USV. Specifically, the closer the USV is to the target berth, the lower its speed should be. The expression is as follows:

[0074]

[0075] in v It's the speed of the unmanned surface vessel. l This is a minimum distance threshold used to prevent the distance between the unmanned surface vessel and the target berth from becoming infinitely small;

[0076] The third part is the thrust control items. This represents the thrust difference between two consecutive moments. Its function is to control the stability of the unmanned surface vessel (USV). If the thrust difference between the USV's thrusters is too large between two consecutive moments, the USV's motion will become unstable, which will reduce the safety of the berthing mission. The expression is as follows:

[0077]

[0078] in, and These represent the thrust magnitudes of the left and right thrusters at the current moment;

[0079] at last, , , These are hyperparameters used to balance the impact of each component on the reward function; it is worth noting that... If the value is positive, and the unmanned surface vessel moves closer to the target berth at the current moment than it did at the previous moment, then the unmanned surface vessel will receive a positive reward. If the unmanned surface vessel maintains a high speed when it is close to the target berth, it will receive a negative reward. It is also a negative number. If the difference in thrust between two consecutive moments is too large, causing the unmanned surface vessel to become unstable, then the unmanned surface vessel will also receive a negative reward.

[0080] The value of needs to be large enough to highlight the priority of approaching the target, and is usually a positive number in the range of 1 to 10;

[0081] The value of is to balance the intensity of the speed penalty, avoiding excessive speed without overly restricting movement. It is usually a negative number in the range of -0.1 to -1.

[0082] The value of is: to suppress thrust mutations but not to excessively limit maneuverability, and is usually a negative number in the range of -0.01 to -0.1.

[0083] Specific methods for determining and optimizing values ​​include: manual debugging, setting initial values ​​based on experience, and gradually adjusting until the model converges; such as fixing the initial values ​​first. Set a large value (such as 10 or 5) to ensure the unmanned surface vessel (USV) approaches the target first; then adjust. and The absolute value of the parameters is used to observe whether the velocity curve and thrust change are smooth; finally, by comparing the model performance under different weights (such as SPL index, trajectory smoothness, etc.), the parameter with the best overall performance is selected.

[0084] This embodiment , , .

[0085] Preferred, such as Figure 2 As shown, the input layer of the neural network structure in this embodiment is a 364-dimensional input vector, which contains 360-dimensional LiDAR information, 2-dimensional target berth location information, and 2-dimensional unmanned surface vessel (USV) current speed, followed by three fully connected layers. The USV in this embodiment is equipped with a 360-degree LiDAR, whose main function is to comprehensively reflect the obstacle situation around the USV.

[0086] Fully connected layers (FC layers) are the most common type of layer in neural networks, particularly widely used in multilayer perceptrons (MLPs) and convolutional neural networks (CNNs). The main characteristic of a fully connected layer is that each neuron is connected to all neurons in the previous layer. The basic structure of a fully connected layer is a two-dimensional weight matrix, where each row represents the input weight of a neuron, and the columns represent the output values ​​of neurons in the previous layer. Assuming the previous layer has n neurons and the current fully connected layer has m neurons, then the size of the weight matrix is ​​m*n, and the output of each neuron can be calculated by matrix multiplication and adding a bias. As shown in the figure, the three fully connected layers used in this embodiment each have 256 neurons.

[0087] The parameters obtained after each fully connected layer are processed by ReLU activation. The Rectified Linear Unit (ReLU) function is a commonly used activation function in neural networks. Its function is to introduce a non-linear transformation, helping the neural network learn more complex patterns and features. In this embodiment, the formula for the ReLU function is defined as follows:

[0088]

[0089] In short, when the input to the ReLU function is negative, the output is 0, and when the input is positive, the output is the input itself. The main functions of the ReLU function are: introducing non-linearity—although ReLU appears linear (linear for the positive part), it "suppresses" all inputs to zero in the negative region, thus introducing non-linearity. This is crucial for neural networks to learn complex data patterns and high-dimensional features. Reducing the vanishing gradient problem—in traditional activation functions, the gradient tends to become very small when the input value is large or small, leading to gradient vanishing and making training very difficult. ReLU's gradient is always 1 in the positive region, effectively avoiding the vanishing gradient problem. Improving computational efficiency—ReLU calculation is very simple, involving only comparing the magnitude of the input values, making it very fast and suitable for efficient training in deep networks.

[0090] The final output layer has a 7-dimensional dimension, representing the Q-values ​​of 7 discrete actions. The unmanned surface vessel (USV) will select the action with the highest Q-value to execute.

[0091] The Deep Q Network (DQN) algorithm used in this embodiment is a typical value-based deep reinforcement learning algorithm. The neural network parameters in the figure are... Updating the DQN (Deep Q-Network) network requires updating based on each transition sample, i.e., (s, a, r). t+1 , st+1 ), where s and a are the current state and action, respectively, and r t+1 For the reward obtained, s t+1 For the next time step, each transition is stored in the experience replay pool. Neural network parameters. The specific update steps involve first randomly sampling a batch of transfer samples from the experience replay pool to construct unbiased mini-batch training data. In this paper, 64 transfer samples are randomly sampled from the experience replay pool each time the neural network is trained. Subsequently, a loss function is calculated for each sample. For details, please refer to (see CHU Y, WU Z, ZHU X, YUE Y, LIME G, PAOLETTI P. Self-supervised dock pose estimator for unmanned surfacevehicles autonomous docking[C] / / 2024 10th International Conference on Mechatronics and Robotics Engineering (ICMRE). Milan, Italy, 2024: 189–194.), and Deep Deterministic Policy Gradient (DDPG) based on continuous action space (see PEREIRA MI, PINTO A M. Reinforcement learning based robot navigation using illegalactions for autonomous docking of surface vehicles in unknown environments[J]. Engineering Applications of Artificial Intelligence, 2024, 133:108506.). Finally, the gradient descent method (Adam optimizer) is used to minimize the loss function and update the network parameters.

[0092] Meanwhile, during the neural network update process, an ε-greedy policy is used for exploration. This means that during training, actions are randomly selected with probability ε to encourage exploration of the unknown policy space. The value of ε gradually decays from a high value to a low value to balance exploration and exploitation.

[0093] Preferably, the training strategy for the berthing method model is in rounds, with no less than 1,000 rounds, to ensure the reliability of the training results. In this embodiment, a total of 10,000 rounds are trained. In each round, the initial position of the unmanned surface vessel and the position of the target berth are randomly generated, and the distribution of obstacles is also randomly distributed.

[0094] The conditions for starting a new training round at the end of each round are as follows: the unmanned surface vessel (USV) safely docks, completes the docking task, and does not collide with any obstacles; the USV collides with an obstacle; or the USV fails to complete the task within a certain number of time steps. In this embodiment, 1000 time steps are set to prevent infinite loops.

[0095] During the training of the DQN model, the exploration strategy was set to ε-greedy. This strategy involves randomly selecting an action with a probability of "ε" and choosing the action with the largest Q-value output by the neural network with a probability of "1-ε". As shown above, in the berthing model training process of this embodiment, a total of 10,000 rounds were trained. In the first 8,000 training rounds, ε linearly decreased from 1 to 0.2. This indicates that in the first training round, the unmanned surface vessel (USV) selected actions completely randomly. In the 8,000th training round, the USV selected actions randomly with a probability of 20%. From the 8,001st to the 10,000th training round, ε remained at 0.2, ensuring that the USV maintained a certain level of exploration capability in subsequent training.

[0096] Preferably, the virtual environment for model training includes a water-based virtual environment. In each different task, the initial position of the unmanned surface vessel (USV) and the position of the target berth are randomly generated, and the distribution of obstacles is also randomly generated. The USV is equipped with a 360-degree lidar and is trained and tested in this random environment. Various obstacles in the environment, as well as the area around the berth, have collision attributes, and the USV needs to avoid these obstacles during docking. Collision detection is based on data observed by the radar. If any data point observed by the radar is lower than a pre-set threshold, then the USV is considered to have collided with an obstacle.

[0097] The unmanned surface vessel (USV) is equipped with a LiDAR-HDL-64E lidar sensor. The LiDAR-HDL-64E is a high-performance lidar sensor widely used in robot navigation, USV navigation, and 3D modeling. Specific parameters are as follows: a frame rate of 10 Hz, meaning the radar performs 10 complete laser scans per second; a measurement range of 120 meters, indicating the maximum distance at which the radar can detect obstacles is 120 meters; a ranging accuracy of 0.02 meters, referring to the radar's ranging error; and a horizontal field of view of 360 degrees, meaning the radar can scan the environment in all directions horizontally, forming a complete 360-degree circular field of view.

[0098] To facilitate the training of reinforcement learning models and the invocation of intelligent agents, the virtual environment in this embodiment is also equipped with a separate program interface. For example... Figure 1 As shown, the five elements in reinforcement learning are agent, environment, state, action, and reward. The established virtual water environment is the environment element. The program interface of the virtual environment can be extracted from the environment, which facilitates model training and code invocation.

[0099] In each training or testing round, the unmanned surface vessel (USV) performs actions in a virtual aquatic environment. The environment provides the USV with its next state based on these actions, i.e., the radar observation results at the next moment. The virtual environment also provides the agent with other signals, such as whether the task has failed or succeeded and the reward obtained after performing the action.

[0100] The virtual environment described is built using Gazebo. Gazebo is a powerful open-source robot simulation platform widely used for researching, developing, and testing robot algorithms, particularly in fields such as autonomous driving, robot control, vision, and sensor simulation. It provides a realistic 3D physics engine capable of simulating robot behavior in various environments, supporting multiple robot types, sensors, and physical phenomena such as collisions, friction, and gravity. Furthermore, Gazebo supports simulations of various sensors, such as the radar primarily used in this paper. In addition, Gazebo is tightly integrated with ROS (Robot Operating System), a popular robot operating system that provides a variety of robot development tools, libraries, and drivers. Through the ROS interface, Gazebo allows the simulation environment to seamlessly interface with actual robot systems on hardware.

[0101] In this embodiment, the total reward obtained by the unmanned surface vessel in each round is recorded during training, and the average reward over 100 training rounds is calculated. The average reward during training is as follows: Figure 3 As shown.

[0102] As shown in the figure, the average reward obtained by the unmanned surface vessel (USV) increases significantly with the number of training rounds. After the 2000th training round, the reward steadily increases, and after 6000 training rounds, it remains around 20. This indicates that the state space, action space, reward function, and training strategy proposed in this invention are all reasonable. The USV can perform the docking task well in the later stages of training.

[0103] After training, the docking model was tested by removing the exploration mechanism and selecting the action with the highest Q-value for each turn. A total of 500 rounds were tested, with the initial position of the UAV and the target berth randomly generated in each round, and obstacles also randomly distributed. The success rate over the 500 test rounds was 88.2%. This data demonstrates that the model designed in this paper performs excellently in complex docking tasks, and the randomization of the target berth position in each test round indicates that the trained docking model has very good generalization ability and performs well in different environments.

[0104] To further validate the performance of the trained docking model, the testing metric SPL (Success weighted by Path Length) proposed by Anderson et al. in 2018 was used to further evaluate the performance of the docking model (see SUTTONR S, MCALLESTER D, SINGH S, et al. Policy gradient methods for reinforcement learning with function approximation[C] / / Proceedings of the 13th International Conference on Neural Information Processing Systems. Denver, 1999: 1057-1063.). SPL is calculated as follows:

[0105]

[0106] in, S i S is a binary number indicating whether each round was successful. If the unmanned surface vessel successfully completed the docking mission in that round, then S... i It is 1 if it is true, otherwise it is 0. l i It is the actual shortest path from the initial position of the unmanned surface vessel to the target berth in each round. p i This is the actual distance traveled by the unmanned surface vessel in each round. Therefore, SPLThe larger the value, the closer the unmanned surface vessel's trajectory is to the shortest path, indicating that the unmanned surface vessel's movement is "better".

[0107] In 500 rounds of testing, the SPL reached 0.802, with a success rate of 88.2%. This indicates that the trained docking strategy not only controls the unmanned surface vessel (USV) to safely complete the docking task, but also ensures the optimality of the trajectory, making the USV's trajectory close to the shortest path.

[0108] Preferably, the objective of the reinforcement learning is to maximize the cumulative discounted reward G in a single round. t :

[0109]

[0110] Here, γ is a discount factor, mainly used to reduce the value of future rewards and prevent G from being discounted. t The above formula, which is infinitely large, indicates that the agent needs to infer the optimal sequence of actions based on the current state.

[0111] The following sections will introduce some important concepts and methods from reinforcement learning used in this invention. First, there is the policy π, which refers to the agent's policy for choosing an action based on its current state. There are two ways to express a policy: the first is by using a probability distribution function.

[0112]

[0113] This refers to the probability of choosing an action given the current state. For example, if a robot has four actions to choose from: "forward, backward, turn left, and backward", then at each moment, the policy will output the probability of choosing these four actions. For example, "forward" is 0.3, "backward" is 0.4, "turn left" is 0.2, and "turn right" is 0.1. The sum of the probabilities of choosing all actions is 1.

[0114] The second approach is to have the policy function output a specific action:

[0115]

[0116] The formula represents an agent choosing a specific action based on its current state. t Let A be the action space. Action spaces are mainly divided into two categories: discrete action spaces and continuous action spaces. Discrete action spaces refer to spaces where the number of actions an agent can choose at any given moment is finite; conversely, continuous action spaces have an infinite number of actions. In most cases, reinforcement learning models based on discrete action spaces are easier to train, but discrete action spaces are more suitable for complex and variable environments.

[0117] The state-value function is a very important concept in reinforcement learning; it refers to the expected cumulative discounted reward that can be obtained in the future.

[0118]

[0119] The establishment of a state-value function must be based on a defined state and policy.

[0120] The state-action value function, also known as the Q-value function, is another very important concept. It is similar to the state value function in that it expresses the same meaning; the only difference is that the Q-value function requires an additional parameter: the currently executing action.

[0121]

[0122] The above formula represents the expected cumulative discount reward in the future, given a state, an action, and a definite policy.

[0123] From this, we can deduce the mathematical relationship between the state value function and the Q-value function:

[0124]

[0125] The state value function can judge the "goodness" or "badness" of a policy in the current state, while the Q-value function judges the "goodness" or "badness" of performing a specific action given a state.

[0126] Preferably, the motion model of the unmanned surface vessel (USV) includes a three-degree-of-freedom (DOF) model, where the state variable η represents the USV's position and orientation in the inertial coordinate system.

[0127]

[0128] Where x and y represent the position of the unmanned surface vessel on the sea surface, This indicates the heading direction of the unmanned surface vessel (USV), i.e., the direction the USV is facing.

[0129] Define the velocity variable of the unmanned surface vessel: V = [uvw] T (The superscript T here indicates transpose), where u is the longitudinal velocity of the unmanned surface vessel, that is, the speed at which the unmanned surface vessel moves along its own orientation; v is the lateral velocity of the unmanned surface vessel, that is, the speed at which the unmanned surface vessel moves along a direction perpendicular to its own orientation; w is the yaw rate of the unmanned surface vessel, that is, the rotational speed of the unmanned surface vessel, which represents the rotational speed of the unmanned surface vessel about its axis perpendicular to the sea surface.

[0130] Note that the definitions of the state variable η and the velocity variable V (both are vectors) here are different from the traditional definitions of three-dimensional spatial translation x, y, z degrees of freedom and pitch, yaw, roll, and rotation three degrees of freedom. This is a concise and necessary variable form designed by the present invention based on the requirements and characteristics of the method. It simplifies the traditional design of two sets of six-dimensional variables for spatial translation and rotation, which greatly improves the computational efficiency.

[0131] Therefore, the kinematic equations can be defined as:

[0132]

[0133] in, J represents the state changes of the unmanned surface vessel (USV), specifically, the changes in its position and orientation relative to the sea surface over time. J is a matrix representing the transformation relationship between the body coordinate system and the inertial coordinate system, and its form is as follows:

[0134]

[0135] The function of J is essentially to transform the velocity of the unmanned surface vessel (USV) in the body coordinate system to the inertial coordinate system, thereby describing the motion of the USV in the world coordinate system.

[0136] Dual-jet propulsion surface unmanned surface vessels (USVs) typically have two independent propulsion systems, located on either side of the USV. These two propellers provide thrust along the direction the USV is heading. Assuming the thrust of the two propellers is T1 and T2, the combination of these two thrusts affects the USV's longitudinal, lateral, and rotational motion.

[0137] The thrust of each propeller is related to its rotational speed and the water flow velocity. The magnitude of the thrust can be determined based on the propeller's operating characteristics, rotational speed, and water flow conditions. Generally, there is a certain relationship between the propeller's thrust and the water flow velocity and propeller rotational speed. Because the two propellers are installed in different positions, their effects on the hull are different. Therefore, their combined effect needs to be calculated based on the direction and magnitude of the thrust.

[0138] The relationship between the thrust of the dual waterjet propulsion system and the motion of the hull can be described by dynamic equations. In a three-degree-of-freedom model, the dynamic equations of the unmanned surface vessel mainly consist of two parts: translational motion and rotational motion, and a clear theoretical relationship can also be established between thrust and velocity.

[0139] Preferably, the reinforcement learning method includes model-based or model-free methods; the model-based method calculates the state value function and / or Q-value function based on the probability of transition between each state to predict the optimal action policy; the model-free method includes value-based methods, policy-based methods, or the actor-critic method, as detailed in "Chen Zhanyang, Wang Zhenyu, Cui Haixin, et al. Visualization study of ship attitude prediction based on deep learning [J]. Journal of Huazhong University of Science and Technology (Natural Science Edition), 2025, 53(04): 132-137.", "TIAN X, ZHANG J, MA Z, et al. Frame stacking and retaining for recurrent neural network acoustic model [J]. arXiv preprint arXiv:1705.05992, 2017.", and "

[57] PEREIRA MI, PINTO A M. Reinforcement learning based robot navigation using illegal actions for autonomous docking of surface vehicles in unknown environments[J]. Engineering Applications of Artificial Intelligence, 2024, 133: 108506." and other documents.

[0140] Example 2:

[0141] The difference from Embodiment 1 is that the unmanned surface vessel's motion space in this embodiment is a continuous motion space, that is, the thrust of the left and right thrusters is continuously variable.

[0142] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for autonomous berthing of an unmanned surface vessel with dual waterjet propulsion, characterized in that, The autonomous docking method is based on reinforcement learning. It defines a state space, action space, reward function, and neural network structure elements, and utilizes reinforcement learning based on the unmanned surface vessel's current radar observation information. t Target berth location information g t Current speed information of the unmanned surface vessel (v) t Obtain unmanned surface vessel control output information F t The velocity information includes linear velocity and angular velocity information, and the output information includes the thrust magnitudes of the two thrusters; The unmanned surface vessel's state space state s t =[o t ||g t ||v t ], where "||" represents concatenation between vectors; The unmanned surface vessel's action space includes a discrete action space or a continuous action space. The discrete action space includes at least seven actions, namely: (N, N), (N / 2, N / 2), (N / 2, N), (N, N / 2), (N, 0), (0, N), (0,0), where N is the maximum thrust of the thruster, the element on the left in "()" is the thrust of the left thruster, and the element on the right is the thrust of the right thruster; The reward function After the unmanned surface vessel (USV) performs the corresponding action, the reward it should receive is determined based on its status: when the USV safely reaches the target berth and docks safely, it receives a positive reward. When an unmanned surface vessel collides with any obstacle after performing a certain action, it will be penalized. ; When the unmanned surface vessel (USV) fails to complete the docking task and does not collide with an obstacle, the reward it receives consists of three parts. The first part is the distance control item. : in, This indicates the distance between the unmanned surface vessel and the target berth at the current moment. This indicates the distance between the unmanned surface vessel and the target berth at the previous moment; The second part is the speed control item. The expression is as follows: in v It's the speed of the unmanned surface vessel. l Minimum distance threshold; The third part is the thrust control items. This represents the thrust difference between two consecutive moments, expressed as follows: in, and These represent the thrust magnitudes of the left and right thrusters at the current moment; , , For hyperparameters; The motion model of the unmanned surface vessel (USV) includes a three-degree-of-freedom (DOF) model, where the state variable η represents the USV's position and orientation in the inertial coordinate system. Where x and y represent the position of the unmanned surface vessel on the sea surface, This indicates the heading direction of the unmanned surface vessel (USV), i.e., the direction the USV is facing. Define the velocity variable of the unmanned surface vessel: V = [uvw] T Where u is the longitudinal velocity of the unmanned surface vessel (USV), i.e., the speed at which the USV moves along its own orientation; v is the lateral velocity of the USV, i.e., the speed at which the USV moves along a direction perpendicular to its own orientation; and w is the yaw rate of the USV, i.e., the rotational speed of the USV, which represents the rotational speed of the USV about its axis perpendicular to the sea surface. Therefore, the kinematic equations can be defined as: in, J represents the state changes of the unmanned surface vessel (USV), specifically, the changes in its position and orientation relative to the sea surface over time. J is a matrix representing the transformation relationship between the body coordinate system and the inertial coordinate system, and its form is as follows:

2. The method for autonomous berthing of a dual-waterjet propulsion unmanned surface vessel according to claim 1, characterized in that, The input layer of the neural network structure is a 364-dimensional input vector, which contains 360-dimensional lidar information, 2-dimensional target berth location information, and 2-dimensional unmanned surface vessel current speed, followed by 3 fully connected layers.

3. The method for autonomous berthing of a dual-waterjet propulsion unmanned surface vessel according to claim 1, characterized in that, The training strategy for the berthing method model is in rounds, with no less than 1000 rounds; in each round, the initial position of the unmanned surface vessel and the position of the target berth are randomly generated, and the distribution of obstacles is also randomly distributed. The conditions for starting a new training round at the end of each round are as follows: the unmanned surface vessel (USV) safely berths, completes the berthing task, and does not collide with any obstacles; the USV collides with an obstacle; or the USV fails to complete the task within a certain number of timesteps.

4. The autonomous berthing method for a dual-waterjet propulsion unmanned surface vessel according to claim 3, characterized in that, The virtual environment used for model training includes a water-based virtual environment. In each different task, the initial position of the unmanned surface vessel and the position of the target berth are randomly generated, and the distribution of obstacles is also randomly generated.

5. The autonomous berthing method for a dual-waterjet propulsion unmanned surface vessel according to claim 1, characterized in that, The goal of the reinforcement learning is to maximize the cumulative discount reward G over a single round. t : Here, γ is a discount factor used to reduce the value of future rewards and prevent G from being devalued. t Infinitely large.

6. The method for autonomous berthing of a dual-waterjet propulsion unmanned surface vessel according to claim 1, characterized in that, The reinforcement learning methods include model-based or non-model-based methods; the model-based methods calculate the state value function and / or Q-value function based on the probability of transitions between each state to predict the optimal action policy; the non-model-based methods include value-based methods, policy-based methods, or the "actor-critic" method.

Images