Unmanned ship maneuvering optimization method and device based on reinforcement learning excitation signal plasticity

Abstract

The present invention relates to an unmanned ship manipulation optimization method and device based on reinforcement learning excitation signal plasticity, including the following: S1, constructing a navigation situation analysis network and a sample storage network according to an adaptive evolutionary control algorithm, and constructing a two-dimensional plane coordinate system; S2, constructing an intelligent agent autonomously controlling the steering gear decision action space of the ship according to the maximum and minimum operating angles of the steering gear of the basic physical structure of the unmanned ship; S3, constructing a soft penalty function for exciting the core information of the unmanned ship's own physical characteristics and the navigation situation information, and incorporating it into the navigation situation analysis network to form a global environment excitation network; S4, performing repeated basic experience and experience reuse autonomous sampling reinforcement learning on the formed global environment excitation network; S5, based on the state random transfer matrix migration sample approximation strategy in the sample storage network, obtaining an updated ship manipulation intelligent agent model. The present invention enables the unmanned ship intelligent agent to successfully perform a steering operation similar to that of an actual ship.

Description

Technical Field

[0001] The present invention relates to the technical field related to ship autonomous driving, and in particular to an unmanned ship manipulation optimization method and device based on reinforcement learning excitation signal plasticity. Background Art

[0002] There are still many challenges in the application of unmanned ships in the fields of ocean exploration, environmental monitoring, military reconnaissance, etc., such as long-distance autonomous navigation, accurate grasp of posture information, and obstacle avoidance. To solve these problems, researchers have been committed to exploring new technologies and methods in recent years, among which unmanned ship automatic driving systems based on deep learning and sensor fusion technology have attracted much attention.

[0003] Autonomous ships based on reinforcement learning will inevitably have random steering behavior. For example, during smooth operation at a 15° rudder angle, the ship may suddenly switch to a -15° rudder angle. The resulting violent shaking may pose a great risk to the structure and stability of the ship, posing a serious safety hazard. Summary of the invention

[0004] The purpose of the present invention is to solve at least one of the deficiencies of the prior art and to provide an unmanned ship maneuvering optimization method and device based on reinforcement learning excitation signal plasticity.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] Specifically, an unmanned ship maneuvering optimization method based on reinforcement learning incentive signal plasticity is proposed, including the following:

[0007] S1. Based on the adaptive evolutionary control algorithm, a navigation situation analysis network and a sample storage network are constructed. With the center of the unmanned ship as the original center, a two-dimensional plane coordinate system is constructed. The navigation situation analysis network is used to obtain navigable waters and non-navigable waters under uncertain waters. The sample storage network is used to store the information samples taken by the n-level intelligent agents during the training process, which are called experience, and then used for experience reuse learning;

[0008] S2. Construct the decision-making action space of the steering gear of the intelligent agent to autonomously control the ship according to the maximum and minimum operating angles of the steering gear of the basic physical structure of the unmanned ship;

[0009] S3, construct a soft penalty function for motivating the core information of the unmanned ship's own physical characteristics and navigation situation information, and incorporate it into the input end of the navigation situation analysis network to form a global environment incentive network;

[0010] S4, autonomous sampling reinforcement learning of repeated basic experience and experience reuse on the formed global environment incentive network;

[0011] S5, based on the state random transfer matrix migration sample storage network navigation sample approximation strategy, to obtain an updated ship maneuvering intelligent agent model;

[0012] S6. Optimize the operation of the unmanned ship using the updated ship manipulation agent model.

[0013] Further, specifically, according to the maximum and minimum operating angles of the steering gear of the basic physical structure of the unmanned ship, the steering gear decision action space of the intelligent agent autonomously controlling the ship is constructed, including:

[0014] The steering gear decision action space constructed according to the unmanned ship's own information and its rudder angle information is used for the rudder angle decision when the intelligent agent actually operates the ship. The constructed steering gear decision action space ACTIONS is as follows:

[0015] ACTIONS={-35°, -30°...-5°, 0°...20°, 25°...35°}.

[0016] Furthermore, specifically, a soft penalty function is constructed to motivate the core information of the unmanned ship's own physical characteristics and navigation status information, including calculating the Euclidean distance between the unmanned ship and the target point, the step-by-step quadratic change in the unmanned ship's heading angle, and the difference in the actual rudder angle implemented by the step-by-step quadratic steering gear decision, and based on this, the final reward value is calculated. The specific design process is as follows:

[0017] S3.1. Generate distance reward r Length , the penalty value r of the second step change of the unmanned ship heading angle Angle , the difference penalty value r of the actual implementation of the rudder angle of the step-secondary steering gear decision Action ;in:

[0018] Distance Reward Length The calculation formula is as follows:

[0019]

[0020] In the above formula, x K ,y K are the actual horizontal and vertical coordinates of the unmanned ship itself compared to the initial coordinate system at time K; g ,y g are the horizontal and vertical coordinates of the center of the destination in the initial coordinate system; Argmax{Lengths} is the maximum distance between the initial position of the unmanned ship and the destination position in the navigable waters in the two-dimensional plane,

[0021] The penalty value r of the second step change of the heading angle of the unmanned ship Angle The calculation formula is as follows:

[0022] r Angle =0.001|φK-1 -φ K |

[0023] In the above formula, φ K-1 ,φ K is the longitudinal deviation of the unmanned ship from the initial heading angle of 0° at time K-1 and K, || represents the absolute value calculation symbol,

[0024] The difference penalty value r of the actual rudder angle implemented by the step-secondary steering gear decision Action The calculation formula is as follows:

[0025] r Action =|10-|a K-2 -a K-1 ||

[0026] In the above formula, a K-1 、a K-2 is the actual steering angle of the steering gear decision angle taken by the agent to manipulate the unmanned ship at time K-1 and K-2, || represents the absolute value calculation symbol;

[0027] S3.2 Sum the final reward value of the soft reward function:

[0028]

[0029] In the above formula, S K-1 , S K Extract the state value [(x, y), φ] of the unmanned ship for the agent at time K, where (x, y) is the current x and y coordinate values ​​of the unmanned ship in the two-dimensional plane coordinate system, and φ is the rudder angle value currently selected in the steering gear decision action space ACTIONS.

[0030] Further, specifically, based on the state random transfer matrix migration sample storage network navigation sample approximation strategy, an updated ship maneuvering agent model is obtained, including:

[0031] The collected experience in the sample storage network is reused for learning. After N strategy decisions, a random state transfer of M numbers is performed and the experience is reused to obtain an updated ship manipulation agent model.

[0032] Furthermore, specifically, non-navigable waters include but are not limited to dangerous waters, obstacle blocks, and restricted areas.

[0033] The present invention also proposes an unmanned ship maneuvering optimization device based on reinforcement learning excitation signal plasticity, comprising:

[0034] The initialization module is used to construct a navigation situation analysis network and a sample storage network based on the adaptive evolutionary control algorithm, and to construct a two-dimensional plane coordinate system with the center of the unmanned ship as the original center. The navigation situation analysis network is used to obtain navigable waters and non-navigable waters under uncertain waters, and the sample storage network is used to store the information samples taken by the n-level intelligent agents during the training process, which are called experience, and then used for experience reuse learning;

[0035] The steering gear decision action space construction module is used to construct the steering gear decision action space of the intelligent agent autonomously controlling the ship according to the maximum and minimum operating angles of the steering gear of the basic physical structure of the unmanned ship;

[0036] The global environment incentive network construction module is used to construct a soft penalty function for incentives based on the core information of the unmanned ship's own physical characteristics and navigation situation information, and is incorporated into the input end of the navigation situation analysis network to form a global environment incentive network;

[0037] The training module is used to conduct autonomous sampling reinforcement learning of repeated basic experience and experience reuse on the formed global environment incentive network;

[0038] The ship maneuvering agent model update module is used to obtain an updated ship maneuvering agent model based on the state random transfer matrix migration sample storage network navigation sample approximation strategy;

[0039] The operation optimization module is used to optimize the operation of the unmanned ship with the updated ship manipulation agent model.

[0040] The beneficial effects of the present invention are:

[0041] 1. Considering that the unmanned ship operated by the reinforcement learning agent directly manipulating the steering gear has significant operational uncertainty, the present invention significantly optimizes the operational uncertainty, making the steering gear operation of the agent more seaworthy, thereby improving navigation safety and hull protection.

[0042] 2. Traditional reinforcement learning has low sampling efficiency and is prone to falling into local optimality, resulting in the inability to find an effective path to the destination. The soft penalty function proposed in this invention significantly improves the efficiency of the agent in finding the destination through its guided distance reward mechanism.

[0043] 3. This invention proposes a new reward signal mechanism, which combines three different elementary function models, which not only improves the practical operability of the function, but also makes it compatible with simple function models, making it easier for the intelligent agent to learn.

[0044] 4. The present invention adopts a method in which the trained intelligent agent directly implements the steering gear operation, which further advances the application of reinforcement learning in the field of autonomous driving and also has higher practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The above and other features of the present disclosure will become more obvious by describing in detail the embodiments shown in the accompanying drawings. The same reference numerals in the accompanying drawings of the present disclosure represent the same or similar sampling monitoring points. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure. For those skilled in the art, other accompanying drawings can be obtained based on these accompanying drawings without creative work. In the accompanying drawings:

[0046] Figure 1 It is a flow chart of the unmanned ship maneuvering optimization method based on reinforcement learning excitation signal plasticity of the present invention;

[0047] Figure 2 The figure shows the overall architecture of the unmanned ship maneuvering optimization method based on reinforcement learning excitation signal plasticity of the present invention;

[0048] Figure 3 Shown is a schematic diagram of the strategy optimization method of the unmanned ship manipulation optimization method based on reinforcement learning excitation signal plasticity of the present invention. DETAILED DESCRIPTION

[0049] The following will be combined with the embodiments and drawings to clearly and completely describe the concept, specific structure and technical effects of the present invention, so as to fully understand the purpose, scheme and effect of the present invention. It should be noted that the embodiments in this application and the features in the embodiments can be combined with each other without conflict. The same reference numerals used throughout the drawings indicate the same or similar parts.

[0050] Example 1, reference Figure 1 as well as Figure 2 The present invention proposes an unmanned ship maneuvering optimization method based on reinforcement learning excitation signal plasticity, comprising the following:

[0051] S1. Based on the adaptive evolutionary control algorithm, a navigation situation analysis network and a sample storage network are constructed. With the center of the unmanned ship as the original center, a two-dimensional plane coordinate system is constructed. The navigation situation analysis network is used to obtain navigable waters and non-navigable waters under uncertain waters. The sample storage network is used to store the information samples taken by the n-level intelligent agents during the training process, which are called experience, and then used for experience reuse learning;

[0052] S2. Construct the decision-making action space of the steering gear of the intelligent agent to autonomously control the ship according to the maximum and minimum operating angles of the steering gear of the basic physical structure of the unmanned ship;

[0053] S3, construct a soft penalty function for motivating the core information of the unmanned ship's own physical characteristics and navigation situation information, and incorporate it into the input end of the navigation situation analysis network to form a global environment incentive network;

[0054] S4, autonomous sampling reinforcement learning of repeated basic experience and experience reuse on the formed global environment incentive network;

[0055] S5, based on the state random transfer matrix migration sample storage network navigation sample approximation strategy, to obtain an updated ship maneuvering intelligent agent model;

[0056] S6. Optimize the operation of the unmanned ship using the updated ship manipulation agent model.

[0057] In this embodiment 1, by adopting the above method, the navigation safety and stability of the unmanned ship are significantly improved, and the navigation efficiency is also effectively guaranteed. In addition, the method also helps to speed up the learning speed of the intelligent agent and enables the unmanned ship intelligent agent to successfully perform steering operations similar to those of an actual ship. Therefore, this method is of great significance in practical applications.

[0058] As a preferred embodiment of the present invention, specifically, the steering gear decision action space of the intelligent agent autonomously controlling the ship is constructed according to the maximum and minimum operating angles of the steering gear of the basic physical structure of the unmanned ship, including:

[0059] The steering gear decision action space constructed according to the unmanned ship's own information and its rudder angle information is used for the rudder angle decision when the intelligent agent actually operates the ship. The constructed steering gear decision action space ACTIONS is as follows:

[0060] ACTIONS={-35°, -30°...-5°, 0°...20°, 25°...35°}.

[0061] Reference Figure 3 As a preferred embodiment of the present invention, specifically, a soft penalty function is constructed to motivate the core information of the physical characteristics of the unmanned ship and the navigation situation information, including calculating the Euclidean distance between the unmanned ship and the target point, the step-by-step quadratic change in the heading angle of the unmanned ship, and the difference in the actual implementation of the rudder angle of the step-by-step quadratic steering gear decision, and the final reward value is calculated based on this. The specific design process is as follows.

[0062] S3.1. Generate distance reward r Length , the penalty value r of the second step change of the unmanned ship heading angle Angle , the difference penalty value r of the actual implementation of the rudder angle of the step-secondary steering gear decision Action ;in:

[0063] Distance Reward Length The calculation formula is as follows: distance reward r Length, the Euclidean distance formula is used to determine the distance between the center of the ship and the center of the destination, aiming to prevent the unmanned ship from staying around the destination, improve the clarity of the destination for the agent, and speed up the convergence of the agent's fitting of the soft penalty function:

[0064]

[0065] In the above formula, x K ,y K are the actual horizontal and vertical coordinates of the unmanned ship itself compared to the initial coordinate system at time K; g ,y g are the horizontal and vertical coordinates of the center of the destination in the initial coordinate system; Argmax{Lengths} is the maximum distance between the initial position of the unmanned ship and the destination position in the navigable waters in the two-dimensional plane,

[0066] The penalty value r of the second step change of the heading angle of the unmanned ship Angle The calculation formula is as follows, which mainly considers the change in the heading angle of the unmanned ship under the autonomous control of the intelligent agent:

[0067] r Angle =0.001|φ K-1 -φ K |

[0068] In the above formula, φ K-1 ,φ K is the longitudinal deviation of the unmanned ship from the initial heading angle of 0° at time K-1 and K, || represents the absolute value calculation symbol,

[0069] The difference penalty value r of the actual rudder angle implemented by the step-secondary steering gear decision Action The calculation formula is as follows, which mainly considers the steering gear decision action taken by the agent under the autonomous control of the unmanned ship, and the resulting rudder angle difference. It aims to improve the agent's consideration of the safety and maneuverability of the hull:

[0070] r Action =|10-|a K-3 -a K-1 ||

[0071] In the above formula, a K-1 、a K-2 is the actual rudder angle of the steering gear decision angle taken by the agent to manipulate the unmanned ship at time K-1 and K-2, || represents the absolute value calculation symbol. In addition, after the unmanned ship completes all initializations, the time before the first steering gear decision a1 in this voyage is recorded as "0";

[0072] S3.2 Sum the final reward value of the soft reward function:

[0073]

[0074] In the above formula, S K-1 , S K Extract the state value [(x, y), φ] of the unmanned ship for the agent at time K, where (x, y) is the current x and y coordinate values ​​of the unmanned ship in the two-dimensional plane coordinate system, and φ is the rudder angle value currently selected in the steering gear decision action space ACTIONS.

[0075] In this preferred embodiment, in order to reduce the instability of the hull, steering gear and heading caused by the intelligent agent manipulating the unmanned ship to take excessive steering gear decision-making actions, a penalty value r of the second step change of the unmanned ship heading angle is constructed and introduced. Angle , the difference penalty value r of the actual implementation of the rudder angle of the step-secondary steering gear decision Action , where r Angle 、r Action The penalty value will fluctuate as the agent makes different decisions. The more drastic the impact of the steering decision on navigation safety, the higher the penalty value will be.

[0076] In addition, the introduction of exponential elementary functions and advanced functions of piecewise function superposition system constitutes a soft penalty function. The introduction of exponential functions aims to improve the agent's awareness of the adverse effects of different steering decisions on the stability of the ship and navigation safety. Between two adjacent decisions, the greater the difference in the rudder angle taken by the steering decision, the greater the adverse effect, that is, the greater the penalty value given by the soft penalty function. If the difference in the rudder angle between the two decisions is less than 10°, no penalty will be imposed, but a distance reward will be given to guide the agent to the destination.

[0077] Specifically, the algorithm principle of the training process is as follows: Algorithm 1:

[0078]

[0079] As a preferred embodiment of the present invention, specifically, based on the state random transfer matrix migration sample storage network navigation sample approximation strategy, an updated ship maneuvering intelligent agent model is obtained, including:

[0080] The collected experience in the sample storage network is reused for learning. After N strategy decisions, a random state transfer of M numbers is performed and the experience is reused to obtain an updated ship manipulation agent model.

[0081] As a preferred embodiment of the present invention, specifically, the non-navigable waters include but are not limited to dangerous waters, obstacle blocks, and controlled areas.

[0082] The present invention also proposes an unmanned ship maneuvering optimization device based on reinforcement learning excitation signal plasticity, comprising:

[0083] The initialization module is used to construct a navigation situation analysis network and a sample storage network based on the adaptive evolutionary control algorithm, and to construct a two-dimensional plane coordinate system with the center of the unmanned ship as the original center. The navigation situation analysis network is used to obtain navigable waters and non-navigable waters under uncertain waters, and the sample storage network is used to store the information samples taken by the n-level intelligent agents during the training process, which are called experience, and then used for experience reuse learning;

[0084] The steering gear decision action space construction module is used to construct the steering gear decision action space of the intelligent agent autonomously controlling the ship according to the maximum and minimum operating angles of the steering gear of the basic physical structure of the unmanned ship;

[0085] The global environment incentive network construction module is used to construct a soft penalty function for incentives based on the core information of the unmanned ship's own physical characteristics and navigation situation information, and is incorporated into the input end of the navigation situation analysis network to form a global environment incentive network;

[0086] The training module is used to conduct autonomous sampling reinforcement learning of repeated basic experience and experience reuse on the formed global environment incentive network;

[0087] The ship maneuvering agent model update module is used to obtain an updated ship maneuvering agent model based on the state random transfer matrix migration sample storage network navigation sample approximation strategy;

[0088] The operation optimization module is used to optimize the operation of the unmanned ship with the updated ship manipulation agent model.

[0089] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of software functional modules.

[0090] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or system that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0091] Although the description of the present invention has been quite detailed and specifically describes several described embodiments, it is not intended to be limited to any of these details or embodiments or any particular embodiment, but should be regarded as providing a broad possible interpretation of these claims in view of the prior art by reference to the appended claims, thereby effectively covering the intended scope of the present invention. In addition, the above description of the present invention is based on the embodiments foreseeable by the inventor, and its purpose is to provide a useful description, and those non-substantial changes to the present invention that have not yet been foreseen may still represent equivalent changes to the present invention.

[0092] The above is only a preferred embodiment of the present invention. The present invention is not limited to the above implementation. As long as the technical effect of the present invention is achieved by the same means, it should belong to the protection scope of the present invention. Within the protection scope of the present invention, its technical scheme and / or implementation method can have various modifications and changes.

Claims

1. An unmanned ship maneuvering optimization method based on reinforcement learning excitation signal plasticity, characterized in that: Includes the following: S1. Based on the adaptive evolutionary control algorithm, a navigation situation analysis network and a sample storage network are constructed. With the center of the unmanned ship as the original center, a two-dimensional plane coordinate system is constructed. The navigation situation analysis network is used to obtain navigable waters and non-navigable waters under uncertain waters. The sample storage network is used to store the information samples taken by the n-level intelligent agents during the training process, which are called experience, and then used for experience reuse learning; S2. Construct the decision-making action space of the steering gear of the intelligent agent to autonomously control the ship according to the maximum and minimum operating angles of the steering gear of the basic physical structure of the unmanned ship; S3, construct a soft penalty function for motivating the core information of the unmanned ship's own physical characteristics and navigation situation information, and incorporate it into the input end of the navigation situation analysis network to form a global environment incentive network; S4, autonomous sampling reinforcement learning of repeated basic experience and experience reuse on the formed global environment incentive network; S5, based on the state random transfer matrix migration sample storage network navigation sample approximation strategy, to obtain an updated ship maneuvering intelligent agent model; S6. Optimizing the operation of the unmanned ship using the updated ship maneuvering agent model; Specifically, a soft penalty function is constructed to motivate the core information of the unmanned ship's own physical characteristics and navigation status information, including calculating the Euclidean distance between the unmanned ship and the target point, the step-by-step quadratic change in the unmanned ship's heading angle, and the difference in the actual rudder angle implemented by the step-by-step quadratic steering gear decision, and the final reward value is calculated based on this. The specific design process is as follows: S3.

1. Generation distance reward , penalty value of the second step change of the unmanned ship heading angle , the difference penalty value of the actual implementation of the rudder angle of the step-secondary steering gear decision ;in: Distance Rewards The calculation formula is as follows: In the above formula, , They are respectively the actual horizontal and vertical coordinates of the unmanned ship itself compared to the initial coordinate system at time K; , are the horizontal and vertical coordinates of the destination center in the initial coordinate system respectively; is the maximum distance between the initial position and the destination position of the unmanned ship in the navigable waters in the two-dimensional plane, Penalty value of the second step change of the heading angle of the unmanned ship The calculation formula is as follows: In the above formula, , is the longitudinal deviation of the unmanned ship from the initial heading angle of 0° at time K-1 and K, Indicates the absolute value calculation symbol, The difference penalty value of the actual rudder angle implemented by the step-by-step secondary steering gear decision The calculation formula is as follows: In the above formula, For K-1 time, is the actual rudder angle of the steering gear used by the agent to control the unmanned boat at time K-2, Indicates the absolute value calculation symbol; S3.2 Sum the final reward value of the soft reward function: In the above formula, For K-1 time, At time K, the agent extracts the state value of the unmanned ship [(x, y), φ], where (x, y) is the current x and y coordinate values ​​of the unmanned ship in the two-dimensional plane coordinate system, and φ is the current decision action space of the steering gear. The rudder angle value selected in .

2. The unmanned ship maneuvering optimization method based on reinforcement learning excitation signal plasticity according to claim 1 is characterized in that: Specifically, according to the maximum and minimum operating angles of the steering gear of the basic physical structure of the unmanned ship, the steering gear decision action space of the intelligent agent autonomously controlling the ship is constructed, including: The steering gear decision action space constructed according to the unmanned ship's own information and its rudder angle information is used for the rudder angle decision when the intelligent agent actually operates the ship. as follows, 。 3. The unmanned ship maneuvering optimization method based on reinforcement learning excitation signal plasticity according to claim 1 is characterized in that: Specifically, based on the state random transfer matrix migration sample storage network navigation sample approximation strategy, an updated ship maneuvering agent model is obtained, including: The collected experience in the sample storage network is reused for learning. After N strategy decisions, a random state transfer of M numbers is performed and the experience is reused to obtain an updated ship manipulation agent model.

4. The unmanned ship maneuvering optimization method based on reinforcement learning excitation signal plasticity according to claim 1 is characterized in that: Specifically, non-navigable waters include dangerous waters, obstacle areas, and restricted areas.

5. An unmanned ship maneuvering optimization device based on reinforcement learning excitation signal plasticity, characterized in that: The unmanned ship maneuvering optimization method based on reinforcement learning excitation signal plasticity according to any one of claims 1 to 4 is applied, comprising: The initialization module is used to construct a navigation situation analysis network and a sample storage network based on the adaptive evolutionary control algorithm, and to construct a two-dimensional plane coordinate system with the center of the unmanned ship as the original center. The navigation situation analysis network is used to obtain navigable waters and non-navigable waters under uncertain waters, and the sample storage network is used to store the information samples taken by the n-level intelligent agents during the training process, which are called experience, and then used for experience reuse learning; The steering gear decision action space construction module is used to construct the steering gear decision action space of the intelligent agent autonomously controlling the ship according to the maximum and minimum operating angles of the steering gear of the basic physical structure of the unmanned ship; The global environment incentive network construction module is used to construct a soft penalty function for incentives based on the core information of the unmanned ship's own physical characteristics and navigation situation information, and is incorporated into the input end of the navigation situation analysis network to form a global environment incentive network; The training module is used to conduct autonomous sampling reinforcement learning of repeated basic experience and experience reuse on the formed global environment incentive network; The ship maneuvering agent model update module is used to obtain an updated ship maneuvering agent model based on the state random transfer matrix migration sample storage network navigation sample approximation strategy; The operation optimization module is used to optimize the operation of the unmanned ship with the updated ship manipulation agent model.

Images