Unmanned ship intelligent navigation method and system based on deep reinforcement learning

Abstract

The invention relates to an unmanned ship intelligent navigation method and system based on deep reinforcement learning, and the method employs the construction of a virtual training environment to cooperate with the real-time sensing data collected by a real unmanned ship sensing system, thereby greatly reducing the training cost and risk. The autonomous navigation intelligent agent trained by using a deep reinforcement learning algorithm improves the adaptive and decision-making ability of the unmanned ship to a complex environment, reduces the dependence on a communication link, and improves the motion precision and stability of the unmanned ship.

Description

Technical Field

[0001] This invention relates to the field of unmanned vessel intelligent control technology, and more specifically, to an unmanned vessel intelligent navigation method and system based on deep reinforcement learning. Background Technology

[0002] With the development of artificial intelligence technology, unmanned surface vehicles (USVs) are increasingly being used in environmental monitoring, water patrol, surveying and mapping, and other fields. Intelligent navigation is the core technology of USVs, and its key lies in the ability to perceive the environment, plan paths, and achieve autonomous obstacle avoidance.

[0003] Currently, most mainstream intelligent navigation solutions are based on traditional path planning algorithms (such as A*, artificial potential field method, Dijkstra's algorithm, and D algorithm) or rely on a combination of high-precision Global Positioning System (GPS) and Inertial Measurement Unit (IMU). However, these methods often encounter the following technical problems when facing complex and variable water environments (such as wind, waves, and current interference) and dense dynamic obstacles: 1. Poor environmental adaptability: Traditional algorithms rely on accurate environmental models, which makes it difficult to effectively handle unstructured, dynamically changing obstacles and complex hydrodynamic disturbances in the water environment. The decision-making flexibility is insufficient, which can easily lead to navigation failure or collisions.

[0004] 2. Training is costly and risky. Machine learning-based navigation algorithms typically require training on large amounts of data collected in real-world environments. This process not only consumes significant human and material resources but also poses risks such as equipment damage, personnel safety, and high time costs, severely hindering the efficiency of algorithm iteration and optimization.

[0005] 3. They are highly dependent on communication links. Many solutions rely heavily on shore-based systems or remote control centers for data fusion and decision-making. In waters with poor or interrupted communication signals, the autonomy and reliability of unmanned vessels will be significantly reduced.

[0006] 4. The motion model of unmanned vessels is nonlinear and coupled, making it difficult for traditional control methods to achieve accurate and smooth trajectory tracking and obstacle avoidance. Summary of the Invention

[0007] The technical problem to be solved by this invention is how to solve the problems of poor environmental adaptability, high training cost and strong dependence on communication in the prior art.

[0008] This invention provides an intelligent navigation method for unmanned vessels based on deep reinforcement learning, comprising: Step 1: Construct a virtual training environment, build an unmanned vessel motion model in the virtual training environment, construct an autonomous navigation agent in the unmanned vessel motion model to control the operation of the unmanned vessel motion model in the virtual training environment, and set a target point in the virtual training environment. Step 2: Map the real-time perception data collected by the sensor system of the real unmanned vessel into the virtual training environment; the real-time perception data includes environmental data and unmanned vessel operation data; Step 3: Preprocess the real-time perception data mapped to the virtual training environment to form a state space; Step 4: Using the state space as input and the navigation strategy composed of the control commands of the motor and servo motor of the unmanned ship as the action space, a deep reinforcement learning algorithm is used to train the autonomous navigation agent. The training is based on a multi-objective composite reward function of travel distance, obstacle avoidance effect, path tracking and attitude stability, which guides the autonomous navigation agent to generate the optimal navigation strategy. Step 5: Deploy the trained autonomous navigation agent onto a real unmanned vessel, using real-time perception data provided by the sensor system as input to control the unmanned vessel to make autonomous navigation decisions.

[0009] Compared with existing technologies, the method of this application has the following advantages: by constructing a virtual training environment, the costly and high-risk autonomous navigation training of unmanned surface vessels is transferred to the virtual training environment, which greatly reduces the economic cost, time cost and safety risk of algorithm training; at the same time, by combining deep reinforcement learning to train the autonomous navigation agent, the adaptive decision-making ability of unmanned surface vessels in complex and dynamic water environments is improved, the dependence on high-precision environmental models and stable communication links is reduced, and the navigation autonomy and reliability in waters with poor or interrupted signals are significantly improved.

[0010] In one possible implementation, step 1 employs digital twin technology, uses WebGL and Three.js to construct a virtual training environment in the physics engine, introduces environmental interference factors that affect the operation of the unmanned vessel in the natural environment, uses a noise function to simulate these environmental interference factors and transforms them into forces in the virtual training environment. In step 1, 3D modeling software is used to construct an unmanned vessel motion model in a virtual training environment.

[0011] Compared with existing technologies, the use of noise functions to simulate environmental interference improves the realism and fidelity of the virtual training environment, which helps autonomous navigation agents trained in the virtual training environment to better adapt to and combat uncertain interference in the real environment.

[0012] In one possible implementation, the sensing system includes, but is not limited to, lidar, camera, GPS, inertial measurement equipment, ranging equipment, and attitude acceleration sensor; The unmanned surface vessel's operational data includes its coordinates, attitude, speed and acceleration, and distance traveled. The environmental data includes the coordinates of the target point and the coordinates of the obstacles.

[0013] In one possible implementation, step 3, which involves preprocessing the real-time perception data mapped to the virtual training environment, specifically includes: Dimensionality reduction: The LiDAR data collected by the LiDAR is reduced in dimension by segmenting the minimum values. Normalization processing: The coordinates, attitude, speed and acceleration, distance traveled and the coordinates of obstacles of the reduced-dimensional LiDAR data are normalized. Synthesis processing: Reconstructing the normalized data back to the original dimensions.

[0014] In one possible implementation, the expression for the multi-objective composite reward function is: ; In the formula, Indicates the weighting coefficient; Indicates a time penalty. This represents the reward for the distance the unmanned vessel moves towards the target point. This indicates the orientation reward for the unmanned vessel's bow pointing towards the target point. This indicates a reward for the unmanned surface vessel's proactive obstacle avoidance of obstacles. This represents the reward based on the distance-based nonlinear artificial potential field method. This represents a penalty for attitude stability.

[0015] Compared with existing technologies, by designing a multi-objective composite reward function, the complex autonomous navigation of unmanned vessels is decomposed into multiple quantifiable alphabets, providing the autonomous navigation agent with rich, continuous and clearly oriented learning signals, which greatly accelerates the training convergence process and guides the autonomous navigation agent to learn safe, efficient and stable autonomous navigation strategies.

[0016] In one possible implementation, the time penalty A constant negative reward is given at each time step to incentivize the unmanned surface vessel (USV) model to quickly reach the target point. The expression is as follows: ; The distance reward The reward or penalty is determined based on the cosine similarity threshold between the unmanned vessel's bow movement direction and the target direction, expressed as: ; In the formula, This represents the rate of change of distance at each step. This represents the cosine similarity.

[0017] Compared with existing technologies, the effect of dividing rewards into intervals by combining distance changes and heading alignment (cosine similarity) is to incentivize autonomous navigation agents to learn "efficient progress in the right direction" rather than simply shortening the distance, thereby avoiding inefficient and energy-intensive strategies such as learning to take detours or "zigzag" progress.

[0018] In one possible implementation, the orientation towards the reward Based on the actions selected by the unmanned vessel and the angle between the bow of the unmanned vessel and the target point as variables, the normalized actions are divided into center straight actions and turning actions. The ratio of turning actions to center straight actions is calculated as the turning intensity. Rewards are given based on the threshold range of turning intensity, the angle between the unmanned vessel and the target point, and the unmanned vessel's running distance. ; In the formula, This indicates the angle between the bow of the unmanned surface vessel and the target point. Indicates steering strength. Indicates the distance traveled by the unmanned surface vessel; The active obstacle avoidance reward Based on the angle between the nearest obstacle and the bow of the unmanned surface vessel (USV), and the USV's selected action as variables, a danger detection zone is defined with the USV's bow direction as the reference. The calculation formula is as follows: ; In the formula This indicates the angle between the nearest obstacle and the bow of the unmanned vessel. Indicates the actions selected by the unmanned surface vessel. This indicates the angle value of the unmanned vessel's bow direction.

[0019] Compared with existing technologies, by introducing steering intensity, the action selection of autonomous navigation agents is optimized, encouraging unmanned ships to maintain straight-line fine-tuning after aligning with the target point, suppressing unnecessary sharp turns and high-frequency jitter, making the navigation of unmanned ships smoother; Meanwhile, by setting up danger detection zones and rewarding the obstacle avoidance actions of unmanned vessels, active obstacle avoidance is achieved, enabling autonomous navigation agents to plan paths in advance, maintain a safe distance from obstacles, and improve navigation safety.

[0020] In one possible implementation, the nonlinear artificial potential field method reward Based on the fundamental principles of the artificial potential field method, with the target point as the center of attraction and the obstacle as the center of repulsion, and the distance between the unmanned surface vessel (USV) and the target point as the variable, a target attraction reward was designed; an obstacle repulsion reward was designed using the distance between the USV and the obstacle as the variable. The overall calculation formula is as follows: ; In the formula, Indicates that the target attracts rewards. , As the reward coefficient, Indicates the distance between the unmanned vessel and the target point; This indicates that obstacles exclude reward items. , Indicates the distance between the unmanned vessel and the obstacle. This indicates the preset safe distance threshold between the unmanned vessel and obstacles.

[0021] In one possible implementation, the attitude stability penalty Specifically, it includes: ; In the formula, This represents the continuous heading stability reward. When the angle between the unmanned vessel and the target point is less than a threshold, the heading stability step counter is incremented by 1; if it is greater than the threshold, the heading stability step counter is decremented by 1. , This indicates the value of the heading stability step counter; This represents the continuous movement stability reward. When the distance between the unmanned vessel and the target point is greater than a threshold, the movement stability step counter is incremented by 1; if it is less than the threshold, the movement stability step counter is decremented by 1. , This indicates the value of the moving stability step counter; This represents the basic stability reward. , This represents the average of the three most recent changes in heading angle; This indicates a small, stable reward. .

[0022] Compared with existing technologies, the use of square root and logarithmic functions to design the potential field avoids the problems of traditional artificial potential field methods, such as the inability to reach the target point due to the approaching zero gravity, and the oscillation caused by the infinite repulsive force near obstacles. This makes the reward change smoother and guides the autonomous navigation agent to approach the target point and stay away from obstacles more stably. Meanwhile, by introducing a stable step counter and a moving stable step counter, the autonomous navigation agent is encouraged to learn to maintain stable navigation for a long time and in a continuous manner, which effectively suppresses the violent swaying of the unmanned ship in wind and waves.

[0023] An intelligent navigation system for unmanned vessels based on deep reinforcement learning includes: An unmanned vessel, set in a real-world environment, is equipped with motors and servos to drive its operation; the unmanned vessel is also equipped with an onboard computing platform for decision-making and control. The sensing system module is installed on the unmanned boat in the real environment to collect real-time sensing data. The virtual training environment module, connected to the sensing system module, is used to map real-time perception data collected by the sensing system module onto the virtual training environment; the virtual training environment module is also used to construct the motion model of the unmanned vessel and the autonomous navigation agent. The reinforcement learning training module, connected to the virtual training environment module, is used to train the autonomous navigation agent in the virtual training environment and generate the optimal navigation strategy; and the trained autonomous navigation agent is deployed to the shipborne computing platform to realize real-time decision-making and control of the unmanned ship.

[0024] Compared with existing technologies, the system proposed in this application has the following advantages: it constructs an intelligent navigation system for unmanned vessels based on deep reinforcement learning, materializes the intelligent navigation method for unmanned vessels, and enables the sensor system module, reinforcement learning training module, virtual training environment module and unmanned vessel to work together to realize a closed loop of perception, decision-making, control and optimization for unmanned vessels, so that unmanned vessels can navigate stably and smoothly in complex and dynamic water environments. Attached Figure Description

[0025] Figure 1 This is a flowchart of the present invention; Figure 2 This is a diagram illustrating the different intervals of the distance reward. Figure 3 This is a schematic diagram of the bypass function method; Figure 4 This is a schematic diagram of a nonlinear artificial potential field. Detailed Implementation

[0026] First, those skilled in the art should understand that these embodiments are merely used to explain the technical principles of the embodiments of this application and are not intended to limit the scope of protection of the embodiments of this application. Those skilled in the art can make adjustments as needed to adapt to specific application scenarios.

[0027] In the description of the embodiments of this application, it should be noted that, unless otherwise explicitly specified and limited, the terms "connected" and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in the embodiments of this application based on the specific circumstances.

[0028] In the embodiments of this application, unless otherwise expressly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "on top of," and "over" the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0029] The present application will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0030] Example 1: See Figures 1-4 As shown in the figure, this application discloses an intelligent navigation method for unmanned vessels based on deep reinforcement learning, including: Step 1: Construct a virtual training environment, build an unmanned vessel motion model in the virtual training environment, construct an autonomous navigation agent in the unmanned vessel motion model to control the operation of the unmanned vessel motion model in the virtual training environment, and set a target point in the virtual training environment. Specifically, based on digital twin technology, a virtual training environment is constructed using WebGL and Three.js in the physics engine. Environmental interference factors affecting the operation of unmanned vessels, including but not limited to wind, current, and waves, are introduced from the natural environment. Noise functions are used to simulate these environmental interference factors and transform them into forces acting on the virtual training environment. By simulating complex interferences such as wind, waves, and currents in the virtual environment, the autonomous navigation agent undergoes millions of trial-and-error training iterations before being deployed to real waters. This fundamentally avoids the risks of frequent collisions, damage, or even loss of physical equipment in the early stages of training, reducing the high trial-and-error costs of traditional methods to an extremely low level. Next, a motion model of the unmanned vessel is constructed in a virtual training environment using 3D modeling software, such as SolidWorks or Rhino, as described in this application.

[0031] Step 2: Map the real-time perception data collected by the sensor system of the real unmanned vessel into the virtual training environment; the real-time perception data includes environmental data and unmanned vessel operation data; the unmanned vessel operation data includes the coordinate position, navigation attitude, navigation speed and acceleration, and navigation distance of the unmanned vessel; the environmental data includes the coordinate position of the target point and the coordinate position of the obstacle. Specifically, the sensing system includes, but is not limited to, lidar, cameras, GPS, inertial measurement equipment, ranging equipment, and attitude acceleration sensors; This application's embodiments introduce YOLO to map real-time perception data collected by the sensing system onto a virtual training environment, helping the unmanned surface vessel (USV) to actively perceive the virtual training environment. Mapping real-time data from real sensors (such as visual data processed by YOLO) back to the virtual training environment essentially provides a "real" anchor point for it. This dynamic mapping continuously calibrates the accuracy of the autonomous navigation agent, enabling the navigation strategy output by the agent trained in the virtual environment to be seamlessly and efficiently transferred to the real-world USV environment, thus solving a core bottleneck in the application of reinforcement learning in robotics.

[0032] Step 3: Preprocess the real-time perception data mapped to the virtual training environment to form a state space; the preprocessing includes dimensionality reduction and normalization, specifically: Dimensionality reduction: The 24-dimensional lidar data acquired by the lidar is reduced to 20-dimensional lidar data by segmenting and minimizing the values; for example... Figure 2 As shown; Normalization processing: The coordinates, attitude, speed and acceleration, distance and obstacle coordinates of the 20-dimensional LiDAR data, unmanned vessel, and other data after dimensionality reduction are normalized respectively. Synthesis processing: Reconstructing the normalized data back to the original dimensions.

[0033] By employing piecewise minimum dimensionality reduction on 24-dimensional LiDAR data, the dimensionality and redundancy of the state space are significantly reduced while retaining key obstacle distance information. This not only accelerates the processing speed of the neural network but also effectively avoids the "curse of dimensionality" and speeds up model convergence.

[0034] Step 4: Using the state space as input and the navigation strategy composed of the control commands of the motor and servo motor of the unmanned ship as the action space, a deep reinforcement learning algorithm is used to train the autonomous navigation agent. The training is based on a multi-objective composite reward function of travel distance, obstacle avoidance effect, path tracking and attitude stability, which guides the autonomous navigation agent to generate the optimal navigation strategy. Among them, the autonomous navigation agent adopts the Dueling DQN algorithm. As an innovative variant of DQN, Dueling DQN solves the shortcomings of DQN in sparse rewards, action redundancy, and policy stability by decoupling state value and action advantage. Its mathematical expression is: ; In the formula, Representing the state space, Represents the action space; The training definition of this application defines a multi-objective composite reward function, the expression of which is: ; In the formula, In this embodiment of the application, the weighting coefficient is represented. ; Indicates a time penalty. This represents the reward for the distance the unmanned vessel moves towards the target point. This indicates the orientation reward for the unmanned vessel's bow pointing towards the target point. This indicates a reward for the unmanned surface vessel's proactive obstacle avoidance of obstacles. This represents the reward based on the distance-based nonlinear artificial potential field method. This represents a penalty for attitude stability.

[0035] Wherein, the time penalty A constant negative reward is given at each time step to incentivize the unmanned surface vessel (USV) model to quickly reach the target point. The expression is as follows: ; The penalty value is 0.05 per step, which encourages the unmanned boat to reach the target point quickly.

[0036] The distance reward The reward or penalty is determined based on the cosine similarity threshold between the unmanned vessel's bow movement direction and the target direction, expressed as: ; In the formula, This represents the rate of change of distance at each step. Represents cosine similarity; such as Figure 2 As shown, where Figure 2 a is a schematic diagram of high cosine similarity. Figure 2 b is a schematic diagram of the cosine similarity. Figure 2 c is a schematic diagram of low cosine similarity. Figure 2d represents a false approach illustration. When the cosine similarity is greater than 0.7, the unmanned surface vessel (USV) is given the maximum reward value to maximize its movement towards the target point. When the cosine similarity is between 0.3 and 0.7, the USV is penalized to prevent deviation from its course when attempting to shorten the distance by oblique movement. In other words, when the cosine similarity is between 0 and 0.3, even if the USV's direction of movement has a small angle with the target direction, although it is not the optimal path, the overall trend is still towards the target. To encourage the USV's exploratory behavior and avoid overly rigid paths that lead to local optima, the USV is penalized... A moderate reward is given; therefore, when the cosine similarity is less than 0, it indicates that the unmanned vessel is far from the target point. To prevent this situation and avoid training instability caused by excessive penalties, the unmanned vessel is given a moderate reward. A small penalty is applied; by introducing a cosine similarity threshold range, "progress efficiency" is linked to "heading quality." This enables the autonomous navigation agent not only to learn to approach the target, but also to approach it efficiently by "aiming at the target," fundamentally eliminating inefficient strategies such as detours and hesitation, and guiding it to plan a near-optimal path.

[0037] The orientation reward Based on the actions selected by the unmanned vessel and the angle between the bow of the unmanned vessel and the target point as variables, the normalized actions are divided into center straight actions and turning actions. The ratio of turning actions to center straight actions is calculated as the turning intensity. Rewards are given based on the threshold range of turning intensity, the angle between the unmanned vessel and the target point, and the unmanned vessel's running distance. ; In the formula, This indicates the angle between the bow of the unmanned surface vessel and the target point. Indicates steering strength. This represents the distance traveled by the unmanned vessel. When the angle between the bow of the unmanned vessel and the target point is less than the diagonal threshold of 15°, if the turning intensity is less than the action deviation threshold of 0.2 and the distance traveled by the unmanned vessel to the target point is greater than 0.2, a turning reward is given; otherwise, no reward is given. When the angle between the bow of the unmanned vessel and the target point is greater than the diagonal threshold of 15°, an over-turning reward is given, where 0.083 is the threshold after normalization to 15°.

[0038] Moving from "state rewards" to "action rewards," it not only rewards the state of "alignment" but also the behavior of "maintaining alignment with minute steering movements." This directly suppresses high-frequency noise and severe jitter in the control output of the autonomous navigation agent, making the unmanned vessel stable in a straight line. It avoids the efficiency loss and equipment damage caused by frequent over-adjustment in real-world environments, and significantly improves the smoothness and anthropomorphism of the control.

[0039] The active obstacle avoidance reward Based on the angle between the nearest obstacle and the bow of the unmanned surface vessel (USV), and the USV's selected action, a hazard detection zone is defined with the USV's bow direction as the reference. In this embodiment, the hazard detection zone is defined as ±6.39° from the bow direction. Figure 3 As shown, the calculation formula is: ; In the formula This indicates the angle between the nearest obstacle and the bow of the unmanned vessel. Indicates the actions selected by the unmanned surface vessel. This indicates the angle value of the unmanned surface vessel's bow direction; by designating a small "hazard detection zone" at the bow, active obstacle avoidance rewards are provided. It endows autonomous navigation agents with forward-looking risk perception capabilities; active obstacle avoidance rewards Instead of punishing after a collision, it incentivizes correct avoidance actions when risks are imminent. This drives autonomous navigation agents to learn to actively observe the environment, anticipate risks, and maintain a safety margin with all obstacles, achieving a qualitative leap from "passive non-collision" to "active safety".

[0040] The nonlinear artificial potential field method reward Based on the fundamental principles of the artificial potential field method, with the target point as the center of attraction and the obstacle as the center of repulsion, and the distance between the unmanned surface vessel (USV) and the target point as the variable, a target attraction reward was designed; an obstacle repulsion reward was designed using the distance between the USV and the obstacle as the variable. Figure 4 As shown, specifically, the target attraction reward term is a reward function of the attraction potential field centered on the target point. Based on the design principle that the greater the actual distance between the unmanned vessel and the target point, the greater the penalty value, an exponential function is used as the calculation basis, and a smooth function is selected. The function, where y is the reward value and x is the distance between the unmanned vessel and the target point, effectively prompts the unmanned vessel to approach the target point; The obstacle repulsion reward is a reward function for the repulsive potential field centered on the obstacle. Based on the design principle that the closer the unmanned vessel is to the obstacle, the greater the penalty value, a logarithmic function is used. The design incorporates an obstacle repulsion reward component to prevent excessive fluctuations in reward values ​​due to an overly large exponent. It also sets a repulsion force influence range to prevent the unmanned surface vessel (USV) from moving excessively away from obstacles, effectively encouraging it to move away. The overall calculation formula is as follows: ; In the formula, Indicates that the target attracts rewards. , As the reward coefficient, Indicates the distance between the unmanned vessel and the target point; This indicates that obstacles exclude reward items. , Indicates the distance between the unmanned vessel and the obstacle. This indicates the preset safe distance threshold between the unmanned vessel and obstacles; The embodiments of this application are designed with target attraction rewards and obstacle repulsion rewards to encourage unmanned vessels to choose a movement route that is close to the target point and far away from obstacles.

[0041] The attitude stability penalty Specifically, it includes: ; In the formula, This represents the continuous heading stability reward. When the angle between the unmanned vessel and the target point is less than a threshold, the heading stability step counter is incremented by 1; if it is greater than the threshold, the heading stability step counter is decremented by 1. , This indicates the value of the heading stability step counter; This represents the continuous movement stability reward. When the distance between the unmanned vessel and the target point is greater than a threshold, the movement stability step counter is incremented by 1; if it is less than the threshold, the movement stability step counter is decremented by 1. , This indicates the value of the moving stability step counter; This represents the basic stability reward. , This represents the average of the three most recent changes in heading angle; This indicates a small, stable reward. .

[0042] By employing the square root function as the gravitational field and the logarithmic function as the repulsive field, this design effectively solves the classic problems of traditional artificial potential field methods, such as insufficient attraction near the target point (making it unreachable) and infinite repulsion near obstacles (leading to oscillations). This design ensures that the reward field changes smoothly globally, greatly improving training stability and guiding the agent to naturally find a smooth and safe path between target attraction and obstacle repulsion.

[0043] Step 5: Deploy the trained autonomous navigation agent onto a real unmanned surface vessel (USV). Using real-time sensing data provided by the sensor system as input, the USV makes autonomous navigation decisions. The speed and rudder angle data output by the trained autonomous navigation agent are sent serially to the ESP32 lower-level computer. The lower-level computer generates control commands for the USV's motors and servos, sending PWM signals to the motors and servos for precise control. This completely eliminates the reliance on continuous, high-speed wireless communication links, giving the USV true autonomy, enabling it to independently and reliably perform tasks in complex aquatic environments with weak or interrupted communication signals.

[0044] Step 6: Data generated in the real-world environment is fed back to the virtual training environment for further optimization and calibration of the autonomous navigation agent, forming a closed-loop optimization system of "virtual training - real-world verification - data feedback." This closed loop enables the entire system to become a continuously learning and self-optimizing entity. New data collected from the real world constantly feeds back into the virtual environment, making it increasingly realistic; new models trained on a more realistic environment also become increasingly robust.

[0045] Example 2: An intelligent navigation system for unmanned vessels based on deep reinforcement learning includes: An unmanned vessel, set in a real-world environment, is equipped with motors and servos to drive its operation; the unmanned vessel is also equipped with an onboard computing platform for decision-making and control. The sensing system module is installed on the unmanned boat in the real environment to collect real-time sensing data. The virtual training environment module, connected to the sensing system module, is used to map real-time perception data collected by the sensing system module onto the virtual training environment; the virtual training environment module is also used to construct the motion model of the unmanned vessel and the autonomous navigation agent. The reinforcement learning training module, connected to the virtual training environment module, is used to train the autonomous navigation agent in the virtual training environment and generate the optimal navigation strategy; and the trained autonomous navigation agent is deployed to the shipborne computing platform to realize real-time decision-making and control of the unmanned ship.

[0046] In the description of the embodiments of this application, it should be noted that the terms "inner" and "outer" and other terms indicating direction or positional relationship are based on the direction or positional relationship shown in the drawings. This is only for the convenience of description and does not indicate or imply that the device or component must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation of this application.

[0047] In the description of this application, the references to terms such as "an embodiment," "some embodiments," "in this embodiment," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0048] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An intelligent navigation method for unmanned surface vessels based on deep reinforcement learning, characterized in that, include: Step 1: Construct a virtual training environment, build an unmanned vessel motion model in the virtual training environment, construct an autonomous navigation agent in the unmanned vessel motion model to control the operation of the unmanned vessel motion model in the virtual training environment, and set a target point in the virtual training environment. Step 2: Map the real-time perception data collected by the sensor system of the real unmanned ship into the virtual training environment; The real-time sensing data includes environmental data and unmanned vessel operation data; Step 3: Preprocess the real-time perception data mapped to the virtual training environment to form a state space; Step 4: Using the state space as input and the navigation strategy composed of the control commands of the motor and servo motor of the unmanned ship as the action space, a deep reinforcement learning algorithm is used to train the autonomous navigation agent. The training is based on a multi-objective composite reward function of travel distance, obstacle avoidance effect, path tracking and attitude stability, which guides the autonomous navigation agent to generate the optimal navigation strategy. Step 5: Deploy the trained autonomous navigation agent onto a real unmanned vessel, using real-time perception data provided by the sensor system as input to control the unmanned vessel to make autonomous navigation decisions.

2. The intelligent navigation method for unmanned vessels based on deep reinforcement learning according to claim 1, characterized in that, In step 1, digital twin technology is used to construct a virtual training environment in the physics engine using WebGL and Three.js, and environmental interference factors that affect the operation of unmanned ships in the natural environment are introduced. Noise functions are used to simulate these environmental interference factors and transform them into forces in the virtual training environment. In step 1, 3D modeling software is used to construct an unmanned vessel motion model in a virtual training environment.

3. The intelligent navigation method for unmanned vessels based on deep reinforcement learning according to claim 1, characterized in that, The sensing system includes, but is not limited to, lidar, cameras, GPS, inertial measurement equipment, ranging equipment, and attitude acceleration sensors; The unmanned surface vessel's operational data includes its coordinates, attitude, speed and acceleration, and distance traveled. The environmental data includes the coordinates of the target point and the coordinates of the obstacles.

4. The intelligent navigation method for unmanned vessels based on deep reinforcement learning according to claim 1, characterized in that, Step 3, which involves preprocessing the real-time perception data mapped to the virtual training environment, specifically includes: Dimensionality reduction: The LiDAR data collected by the LiDAR is reduced in dimension by segmenting the minimum values. Normalization processing: The coordinates, attitude, speed and acceleration, distance traveled and the coordinates of obstacles of the reduced-dimensional LiDAR data are normalized. Synthesis processing: Reconstructing the normalized data back to the original dimensions.

5. The intelligent navigation method for unmanned vessels based on deep reinforcement learning according to claim 1, characterized in that, The expression for the multi-objective composite reward function is: ； In the formula, Indicates the weighting coefficient; Indicates a time penalty. This represents the reward for the distance the unmanned vessel moves towards the target point. This indicates the orientation reward for the unmanned vessel's bow pointing towards the target point. This indicates a reward for the unmanned surface vessel's proactive obstacle avoidance of obstacles. This represents the reward based on the distance-based nonlinear artificial potential field method. This represents a penalty for attitude stability.

6. The intelligent navigation method for unmanned vessels based on deep reinforcement learning according to claim 5, characterized in that, The time penalty A constant negative reward is given at each time step to incentivize the unmanned surface vessel (USV) model to quickly reach the target point. The expression is as follows: ； The distance reward The reward or penalty is determined based on the cosine similarity threshold between the unmanned vessel's heading direction and the target direction, expressed as: ； In the formula, This represents the rate of change of distance at each step. This represents the cosine similarity.

7. The intelligent navigation method for unmanned vessels based on deep reinforcement learning according to claim 5, characterized in that, The orientation reward Based on the actions selected by the unmanned vessel and the angle between the bow of the unmanned vessel and the target point as variables, the normalized actions are divided into center straight actions and turning actions. The ratio of turning actions to center straight actions is calculated as the turning intensity. Rewards are given based on the threshold range of turning intensity, the angle between the unmanned vessel and the target point, and the unmanned vessel's running distance. ； In the formula, This indicates the angle between the bow of the unmanned surface vessel and the target point. Indicates steering strength. Indicates the distance traveled by the unmanned surface vessel; The active obstacle avoidance reward Based on the angle between the nearest obstacle and the bow of the unmanned surface vessel (USV), and the USV's selected action as variables, a danger detection zone is defined with the USV's bow direction as the reference. The calculation formula is as follows: ； In the formula This indicates the angle between the nearest obstacle and the bow of the unmanned vessel. Indicates the actions selected by the unmanned surface vessel. This indicates the angle value of the unmanned vessel's bow direction.

8. The intelligent navigation method for unmanned vessels based on deep reinforcement learning according to claim 5, characterized in that, The nonlinear artificial potential field method reward Based on the basic principles of the artificial potential field method, with the target point as the center of gravity, the obstacle as the center of repulsion, and the distance between the unmanned vessel and the target point as the variable, a target attraction reward was designed. The obstacle rejection reward is designed using the distance between the unmanned surface vessel and the obstacle as a variable; the overall calculation formula is as follows: ； In the formula, Indicates that the goal attracts rewards. , As the reward coefficient, Indicates the distance between the unmanned vessel and the target point; This indicates that obstacles exclude reward items. , Indicates the distance between the unmanned vessel and the obstacle. This indicates the preset safe distance threshold between the unmanned vessel and obstacles.

9. The intelligent navigation method for unmanned vessels based on deep reinforcement learning according to claim 5, characterized in that, The attitude stability penalty Specifically, it includes: In the formula, This represents the continuous heading stability reward. When the angle between the unmanned vessel and the target point is less than a threshold, the heading stability step counter is incremented by 1; if it is greater than the threshold, the heading stability step counter is decremented by 1. , This indicates the value of the heading stability step counter; This represents the continuous movement stability reward. When the distance between the unmanned vessel and the target point is greater than a threshold, the movement stability step counter is incremented by 1; if it is less than the threshold, the movement stability step counter is decremented by 1. , This indicates the value of the moving stability step counter; This represents the basic stability reward. , This represents the average of the three most recent changes in heading angle; This indicates a small, stable reward. .

10. An intelligent navigation system for unmanned vessels based on deep reinforcement learning, characterized in that, include: An unmanned vessel, set in a real-world environment, is equipped with motors and servos to drive its operation; the unmanned vessel is also equipped with an onboard computing platform for decision-making and control. The sensing system module is installed on the unmanned boat in the real environment to collect real-time sensing data. The virtual training environment module is connected to the sensing system module and is used to map the real-time sensing data collected by the sensing system module in real time onto the virtual training environment. The virtual training environment module is also used to construct unmanned vessel motion models and autonomous navigation agents; The reinforcement learning training module, connected to the virtual training environment module, is used to train the autonomous navigation agent in the virtual training environment and generate the optimal navigation strategy; and the trained autonomous navigation agent is deployed to the shipborne computing platform to realize real-time decision-making and control of the unmanned ship.

Images