An unmanned ship collision avoidance method based on deep reinforcement learning of memory mechanism

Abstract

The present invention provides an unmanned ship collision avoidance method based on deep reinforcement learning of a memory mechanism. The method dynamically stores the historical navigation state sequence of the unmanned ship and constructs a memory space of fixed length. The state sequence includes the relative position of a target point, the relative position of an obstacle, and historical action data. The state sequence in the memory space is input into a reinforcement learning decision network, and timing features are extracted through a gated recurrent unit. The method generates collision avoidance action instructions in combination with a multi-layer perceptron. The navigation of the unmanned ship is controlled based on the collision avoidance action instructions, and the network parameters are optimized by calculating the instantaneous reward value according to the composite reward function to achieve autonomous collision avoidance under limited perception.

Description

Technical Field

[0001] The present invention belongs to the technical fields of unmanned ship collision avoidance and autonomous decision-making, and specifically relates to an unmanned ship collision avoidance method based on deep reinforcement learning of a memory mechanism. Background Art

[0002] With the rapid development of unmanned maritime systems, unmanned vessels are increasingly being used in fields such as marine resource exploration and maritime search and rescue. In actual navigation missions, unmanned vessels must contend with the dual threats of static obstacles such as buoys and islands, as well as dynamic obstacles such as ships with uncertain motion. This places stringent demands on autonomous collision avoidance algorithms. Particularly in complex ocean environments, due to limitations in sensor detection range (such as millimeter-wave radar field of view) and environmental interference (such as waves and tidal currents), unmanned vessels are often unable to obtain complete, real-time information on obstacles (including key parameters such as heading, speed, and size). This significantly reduces the decision-making reliability of existing collision avoidance algorithms in perception-limited scenarios.

[0003] Current solutions to the problem of limited perception fall into two main categories: First, improving environmental perception capabilities by adding sensing devices such as lidar and multispectral sensors. However, this approach significantly increases hardware cost and system complexity, and multi-source sensor data fusion presents technical bottlenecks. Second, improving decision-making algorithm design, however, traditional algorithms have inherent flaws. The dynamically improved A* algorithm, while incorporating the COLREGs rule, is highly dependent on the accuracy of the environmental model, resulting in inefficient path replanning in dynamic obstacle-intensive scenarios. The artificial potential field (APF) method performs well for static obstacle avoidance, but the superposition of multiple obstacle potential fields in dynamic environments can easily lead to local optimal traps. The relative velocity obstacle method (RVO), based on geometric prediction, struggles to cope with complex environmental perturbations and has a high risk of failure when obstacle information is missing. While deep reinforcement learning (DRL) has shown promise in decision-making tasks, traditional DRL models suffer from strong dependence on environmental information and inefficient learning with sparse rewards, leading to a sharp decline in collision avoidance performance when perception is limited.

[0004] Among the related prior art, authorized patent CN202411556502.2 (Unmanned Vessel Obstacle Avoidance Path Planning Method and System Based on Global Optimization) improves path planning accuracy but fails to address the issues of dynamic obstacle interaction and limited perception. Patent CN202010717418.X (A Unmanned Vessel Path Planning Method Based on Deep Reinforcement Learning and Taking into Account Marine Environmental Factors) focuses on marine environment modeling but does not address decision optimization in information-deficient scenarios. Patent CN201911043840.5 (A Swarm Unmanned Vessel Intelligent Collision Avoidance Method Based on Deep Reinforcement Learning) is designed for multi-agent collaborative scenarios, but its assumption of complete environmental information limits its applicability when a single vessel's perception is limited. This suggests that prior art has yet to effectively address the challenge of autonomous collision avoidance of unmanned vessels with dynamic obstacles under sensor-constrained conditions. Summary of the Invention

[0005] This paper discloses a collision avoidance decision-making scheme for unmanned vessels based on a memory mechanism and reinforcement learning. This solution addresses the issue of incomplete environmental information leading to decision-making failures in unmanned vessels under perception-constrained conditions. Its core design combines historical navigation data with the collaborative optimization of a reinforcement learning framework to achieve safe and compliant collision avoidance of dynamic obstacles.

[0006] Its core design is:

[0007] A collision avoidance method for an unmanned vessel based on a memory mechanism and reinforcement learning includes the following steps:

[0008] (1) Dynamically store the historical navigation state sequence of the unmanned ship and construct a fixed-length memory space. The state sequence includes the relative position of the target point, the relative position of the obstacle, and the historical action data;

[0009] (2) Inputting the state sequence in the memory space into the reinforcement learning decision network, extracting the temporal features through the gated recurrent unit (GRU), and combining it with the multi-layer perceptron (MLP) to generate collision avoidance action instructions;

[0010] (3) Based on the collision avoidance action instructions, the unmanned boat is controlled to navigate, and the network parameters are optimized by calculating the instant reward value according to the composite reward function to achieve autonomous collision avoidance under limited perception.

[0011] Furthermore, the reinforcement learning decision network adopts a soft actor-critic algorithm to optimize network parameters by minimizing the value network loss function and maximizing the policy entropy.

[0012] Furthermore, the memory space is implemented by a first-in-first-out queue, and the oldest state data is removed and the current state data is added each time it is updated.

[0013] The memory space uses a first-in, first-out (FIFO) queue for rolling updates. Each time a new state is added, the oldest historical data is automatically eliminated, ensuring that the stored n-step state sequence is strictly arranged in chronological order. This design solves the situation assessment bias caused by invalid historical data interference in traditional methods, allowing the unmanned vessel to plan a safe path even when the radar field of view is limited.

[0014] Furthermore, the gated recurrent unit (GRU) has a hidden state dimension of 128, the MLP layer dimension is 256, and the decision network outputs Gaussian distribution parameters for propulsion force and torque. This lightweight network structure reduces computing resource usage by 30% while maintaining decision accuracy, making it suitable for deployment in embedded devices.

[0015] Furthermore, the compound reward function includes:

[0016] Target proximity reward: Dynamically adjusted based on the Euclidean distance between the unmanned ship and the target point;

[0017] Collision avoidance safety reward: calculated segment by segment based on the ratio of the relative distance to the obstacle and the safety threshold, with a penalty applied when the distance falls below the minimum safety threshold;

[0018] COLREGs compliance reward: Dynamically adjusts the penalty weight by counting the number of violations in historical action sequences.

[0019] Among them, the target proximity reward is dynamically adjusted based on the Euclidean distance between the unmanned ship and the target point (step 6.1) to guide efficient navigation; the collision avoidance safety reward is calculated in segments based on the ratio of the relative distance to the obstacle (d_OT) to the safety threshold (d_safe) (step 6.2). When the distance is lower than the minimum safety threshold (d_min), a penalty of -6000 is imposed to force the maintenance of a safe distance; the COLREGs compliance reward dynamically adjusts the penalty weight by counting the number of violations in the historical action sequence (step 6.5). This method complies with the requirements of the rules in scenarios such as overtaking and encountering.

[0020] Furthermore, the network parameter update adopts continuous n-step historical data sampling and is updated through a KL divergence constraint strategy to avoid strategy oscillation caused by memory space data distribution offset.

[0021] This method constructs a fixed-length memory space to dynamically store the unmanned vessel's most recent n-step historical navigation state sequence, including the relative positions of target points and obstacles, as well as historical motion data. When the sensor detection range is limited (e.g., millimeter-wave radar only covers -60° to 60°), it uses a gated recurrent unit (GRU) to extract temporal features and infer obstacle motion trends (such as speed and heading changes). Combined with a multi-layer perceptron (MLP), this method generates propulsion and torque avoidance action commands, enabling autonomous collision avoidance in dynamic obstacle scenarios. Using a soft actor-critic (SAC) algorithm to optimize network parameters, this method has been validated in simulations and onboard experiments, increasing the collision avoidance success rate by 22% and reducing decision response time to 85ms.

[0022] And, corresponding to the method, an unmanned vessel collision avoidance decision system, comprising:

[0023] A memory module is used to store the historical navigation state sequence of the most recent n steps; the state sequence includes the relative position of the target point, the relative position of the obstacle and the historical action data;

[0024] A decision module, which integrates a reinforcement learning network of a gated recurrent unit and a multilayer perceptron, outputs collision avoidance action instructions based on the historical state sequence and optimizes network parameters through a composite reward function;

[0025] A control module controls the propulsion and steering of the unmanned vessel based on the motion instructions;

[0026] The training module optimizes network parameters through a compound reward function to achieve autonomous collision avoidance under limited perception.

[0027] The system successfully completed the collision avoidance task in a dynamic obstacle scenario.

[0028] Furthermore, the memory module implements rolling storage of the state sequence through a first-in-first-out queue, removing the oldest state data and adding the current state data each time it is updated.

[0029] Furthermore, the reinforcement learning network adopts a soft actor-critic algorithm to optimize network parameters by minimizing the value network loss function and maximizing the policy entropy; the GRU hidden layer dimension of the decision module is 128, the MLP layer dimension is 256, and the output is the Gaussian distribution parameters of the propulsion force and torque; the composite reward function of the training module includes target proximity reward, collision avoidance safety reward and COLREGs compliance reward, wherein the collision avoidance safety reward is calculated in segments based on the ratio of the relative distance of the obstacle to the safety threshold; the training module adopts continuous n-step historical data sampling and is updated through the KL divergence constraint strategy; it also includes millimeter wave radar and GNSS modules, the detection range of the millimeter wave radar is -60° to 60°, which is used to obtain the relative position of obstacles; the GNSS module is used to locate the coordinates of the unmanned ship in real time.

[0030] And, an electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.

[0031] A non-transitory computer-readable storage medium stores a computer program, which implements the steps of the method described above when executed by a processor.

[0032] Compared with the existing technology, the present invention and its preferred solution significantly improve the collision avoidance performance of unmanned vessels under perception-limited conditions through the coordinated optimization of the following core invention points:

[0033] 1. Situation completion by coordinating memory mechanism and GRU timing

[0034] By building a fixed-length memory space (FIFO queue) to store the most recent n steps of historical state (target location, obstacle location, and historical motion), and combining it with a GRU module to extract temporal features, the system can infer obstacle motion trends (speed change rate ≥ 0.3 m / s², heading angle deviation ≤ 5°) from historical data when the millimeter-wave radar's field of view is limited (-60° to 60°). This improves situation completion accuracy by 35%. The collision avoidance success rate has increased from 73% with traditional DRL to 95%, and path planning efficiency (flight time) has been improved by 15%.

[0035] 2. Multi-objective collaborative optimization of compound reward functions

[0036] The target proximity reward, collision avoidance safety reward, and COLREGs compliance reward are integrated and dynamically weighted. The collision avoidance safety reward enforces safe distance through segmented calculations, with a close distance penalty of -6000, reducing collision risk by 90%. The COLREGs compliance reward uses a sliding window (m=5) to count historical violations and dynamically adjust penalty weights, reducing violations by 90%. The target proximity reward guides path efficiency, reducing flight distance by 15%, and improving learning efficiency by 50% in reward-sparse scenarios.

[0037] 3. Lightweight Network and Stable Training Mechanism

[0038] This model uses a GRU (128-dimensional) + MLP (256-dimensional) network structure and updates with a KL divergence constraint strategy. This reduces network parameters by 30%. By sampling historical data for n consecutive steps (n=8) and applying a KL divergence constraint, training stability is improved by 40% and convergence speed is accelerated by 1.8 times.

[0039] 4. COLREGs rule-embedded reinforcement learning decision-making

[0040] By converting the COLREGs rule into a reward function constraint and combining it with the policy entropy maximization property of the SAC algorithm, the system achieved 100% compliance in key collision avoidance scenarios, including encounters and overtaking. Furthermore, the system reduced path tracking error by 42% in dynamic obstacle interaction scenarios.

[0041] Through the deep collaboration of memory mechanism and reinforcement learning, this invention overcomes the difficulties of real-time, safety and rule compatibility of unmanned ship collision avoidance decision-making under limited perception, providing an efficient and reliable technical solution for operations in complex marine environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0043] Figure 1 This is a flowchart of the unmanned ship collision avoidance according to an embodiment of the present invention;

[0044] Figure 2 This is a collision avoidance path planning diagram for a simulation scenario according to an embodiment of the present invention;

[0045] Figure 3 This is a diagram of the deep reinforcement learning network structure of the memory mechanism according to an embodiment of the present invention;

[0046] Figure 4 This is a diagram showing the construction of an unmanned ship model according to an embodiment of the present invention;

[0047] Figure 5 This is a collision avoidance scene division diagram according to an embodiment of the present invention;

[0048] Figure 6 This is a collision avoidance rule diagram according to an embodiment of the present invention;

[0049] Figure 7 This is an update diagram of the deep reinforcement learning network with a memory mechanism according to an embodiment of the present invention;

[0050] Figure 8 This is a comparison chart of the success rates of various algorithms in the embodiments of the present invention;

[0051] Figure 9 This is a comparison chart of rewards for various algorithms according to an embodiment of the present invention;

[0052] Figure 10 This is an obstacle-free navigation trajectory diagram according to an embodiment of the present invention;

[0053] Figure 11 This is a static obstacle avoidance trajectory diagram according to an embodiment of the present invention;

[0054] Figure 12 This is a collision avoidance trajectory diagram of the port side of an unmanned ship according to an embodiment of the present invention;

[0055] Figure 13 This is a collision avoidance trajectory diagram of the starboard side of the unmanned ship according to an embodiment of the present invention;

[0056] Figure 14 This is a trajectory diagram of an unmanned ship encounter according to an embodiment of the present invention;

[0057] Figure 15 This is a diagram of the unmanned ship overtaking trajectory according to an embodiment of the present invention;

[0058] Figure 16 This is an equipment diagram of an unmanned ship according to an embodiment of the present invention;

[0059] Figure 17 This is a diagram of a real ship experiment of the unmanned ship collision avoidance method with a memory mechanism according to an embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to make the features and advantages of the present invention more clearly understood, the following embodiments are given for detailed description:

[0061] It should be noted that the following detailed description is illustrative and is intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meaning as commonly understood by those skilled in the art to which this application belongs.

[0062] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0063] To address the issue of autonomous collision avoidance for unmanned vessels (UAVs) under limited perception conditions, this paper proposes an autonomous collision avoidance decision-making method based on a memory mechanism. This method utilizes historical navigation data to construct a situational understanding of the surrounding environment, thereby improving the accuracy and safety of collision avoidance decisions under limited perception. This method utilizes a deep reinforcement learning (MDRL) algorithm based on this memory mechanism to optimize the UAV's collision avoidance decisions. By constructing a memory space, the algorithm stores and processes historical navigation data, effectively assessing the surrounding situation in the absence of real-time environmental information. The core of this approach is to utilize historical navigation data to compensate for the shortcomings of immediate perception. Combined with an improved decision-making network, this approach ensures that UAVs make safe and compliant collision avoidance decisions in dynamic and complex marine environments. Furthermore, by combining historical information with short-term decision-making, this method enhances the UAV's adaptability to changing sea conditions, achieving more efficient autonomous collision avoidance. This provides safety assurance for UAVs performing various maritime missions and promotes the further development of UAV technology.

[0064] This embodiment of the present invention proposes a memory-based collision avoidance decision-making method for an unmanned vessel. Based on the soft actor-critic (SAC) algorithm from deep reinforcement learning, it incorporates the memory mechanism of a gated recurrent unit (GRU). This method constructs a decision-making model that leverages historical navigation data to address the autonomous collision avoidance problem for unmanned vessels under perception-constrained conditions. By optimizing the reward function and improving the design of the experience pool, the unmanned vessel's collision avoidance decision-making capability and learning efficiency are enhanced.

[0065] The process steps of the unmanned ship collision avoidance decision training method are as follows: Figure 1 As shown:

[0066] Step 1: Initialize the training environment;

[0067] Step 2: Obtain environmental information;

[0068] Step 3, update the memory space;

[0069] Step 4: Based on step 3, the decision network makes a decision;

[0070] Step 5: Execute the corresponding decision-making behavior according to step 4;

[0071] Step 6, according to step 5, calculate the reward value;

[0072] Step 7: Determine whether the navigation mission is completed or a collision occurs, update the decision network or end the navigation.

[0073] For the implementation of step 1, this embodiment establishes a simulated ocean environment and an intelligent agent containing dynamic and static obstacles on a simulation platform. Figure 2 shown.

[0074] As a preferred implementation, initializing the training environment is specifically divided into the following steps:

[0075] Step 1.1: Set environmental parameters, including water area, waves, currents and other environmental interference factors, as well as the number, location, size, speed and heading of obstacles;

[0076] Step 1.2, initialize the position information, speed, heading and other state parameters of the unmanned ship;

[0077] Step 1.3: Set the target point and navigation path of the unmanned ship and plan the navigation mission goal for the unmanned ship;

[0078] Step 1.4, build the network structure, such as Figure 3 As shown in the figure, the SAC algorithm decision network and value network are constructed. Both the policy network and the value network contain GRU and multi-layer perceptron (MLP) modules. The dimension of the GRU layer is 128, and the dimension of the MLP layer is 256. The policy network includes a mean layer and a standard deviation layer to output propulsion force and torque. The value network directly outputs the behavior value for network updates.

[0079] For step 2, this embodiment can obtain limited environmental information by simulating the real scene of the unmanned ship and collecting the current navigation status. ,like Figure 4 As shown, it contains the position coordinates of the target point relative to the unmanned ship , target azimuth , the obstacle position coordinates relative to the unmanned ship The action performed at the previous moment a .

[0080] As one of the core design points of the present invention, for step 3, this embodiment simulates the human short-term memory and forgetting mechanism by updating the memory space, retaining key historical information to assist decision making. t Store in memory space c t , the memory space stores the navigation state of the latest n steps, forming a state sequence .

[0081] For step 4, the state sequence C in the memory space obtained in step 3 is t , is input into the decision network. As a preferred implementation, the specific decision-making process of the network is divided into the following steps:

[0082] In step 4.1, the GRU module in the policy network processes the state sequence data to extract the dynamic characteristics and temporal correlation of the environment;

[0083] Step 4.2, based on the environmental dynamic features and time correlation extracted in step 4.1, such as Figure 5 As shown, it is determined whether the unmanned ship will perform collision avoidance actions. If collision avoidance actions are performed, then according to Figure 6 Decision making is based on the International Regulations for Preventing Collisions at Sea (COLREGs) as shown;

[0084] Step 4.3: After the extracted features are processed by the multi-layer perceptron (MLP), the action to be performed is output. ,in Indicates thrust, Indicates torque;

[0085] For step 5, in this embodiment, the unmanned ship takes actions according to the output of the strategy network. , performing corresponding propulsion and steering operations in the simulation environment. The environment simulates the state changes of the unmanned ship after executing the action, and updates the position information, speed, heading, etc. of the unmanned ship.

[0086] For step 6, calculate the immediate reward based on the action performed by the unmanned ship and the new state , the reward function consists of multiple indicators, For each reward weight, as a preferred implementation, the specific calculation steps are as follows:

[0087] Step 6.1, calculate the reward for approaching the target , used to encourage the unmanned ship to sail towards the target point. The calculation formula is as follows:

[0088]

[0089] Among them, K1 is the target proximity reward weight; (x o ,y o ) represents the current coordinates of the unmanned vessel, which can be obtained in real time through GNSS.

[0090] Step 6.2, calculate the collision avoidance reward , according to the distance between the unmanned ship and the obstacle , give rewards or penalties to encourage safe navigation, the calculation formula is as follows:

[0091]

[0092] Among them, K2 is the collision avoidance safety reward weight, d OT Represents the real-time distance between the unmanned ship and the obstacle, d safe is the safety threshold, d min It is the size of an unmanned ship.

[0093] Step 6.3, calculate the angle reward , control the heading change of the unmanned ship, avoid excessive turning, and ensure smooth navigation. The calculation formula is as follows:

[0094]

[0095]

[0096]

[0097] in, is the penalty item for heading deviation from the target; is the penalty term for heading angle change; is the heading angle change of the unmanned ship, which can be measured by a gyroscope; K3 is the angle reward weight.

[0098] Step 6.4, calculate the speed bonus , encourage the unmanned ship to sail at an appropriate speed and maintain a stable speed. The calculation formula is as follows:

[0099]

[0100]

[0101]

[0102] in, It is the speed error penalty term, which can be dynamically adjusted according to the mission type (such as overtaking / other); is the speed change penalty term; is the speed change of the unmanned ship, which can be measured by an accelerometer; K4 is the speed reward weight;

[0103] Step 6.5, calculate COLREGs reward r COLREGs , to ensure that the collision avoidance behavior of the unmanned vessel complies with COLREGs. The calculation formula is as follows:

[0104]

[0105] Among them, K5 is the violation penalty weight;

[0106] Step 6.6, calculate the total reward :

[0107]

[0108] For step 7, the policy network is optimized by continuously interacting with the environment. Determine whether the navigation task is completed or a collision occurs. If the task is completed or a collision occurs, calculate the total reward value and the result. Otherwise, update the network model parameters, such as Figure 7 The specific update steps are as follows:

[0109] Step 7.1: Sample data from the experience pool. Randomly sample a batch of n consecutive steps of navigation data from the experience pool to form training samples. , where C is the current state sequence, a is the action performed in state C, r is the total reward calculated in step 6, C' is the state sequence at the next moment after the action is performed, and d is the termination signal, indicating whether the task is completed or a collision occurs.

[0110] Step 7.2, calculate the state value target , the calculation formula is

[0111]

[0112] Among them, γ is the discount factor, α is the entropy weight used to adjust the randomness of the strategy, is the target value network, In state C', we follow the strategy π θ Sampling action.

[0113] Step 7.3, Update the Value Network , minimize the loss function, value network Used to evaluate the quality of the policy network, the update formula is:

[0114]

[0115] in .

[0116] Step 7.4, update the policy network to maximize the expected reward and entropy. The update formula is:

[0117]

[0118] in is obtained from the policy π by reparameterization technique θ The action of sampling.

[0119] Step 7.5, soft update target value network , the update formula is:

[0120]

[0121] Where ρ is the soft update coefficient.

[0122] This collision avoidance algorithm, based on a memory-based deep reinforcement learning (MDRL) algorithm, develops an improved strategy that combines memory mechanisms with GRUs. During autonomous collision avoidance, the UAV uses a memory space to store historical navigation data, while the GRU module processes time series information, integrating current and historical environmental information to assess collision risk. When an obstacle is detected within the detectable range, the UAV utilizes its memory mechanism and limited environmental perception, drawing on historical navigation data to determine the obstacle's motion trend and potential collision threat.

[0123] The algorithm executes collision avoidance actions based on the magnitude of the collision risk, using rules that conform to COLREGs (Colliders and Relational Entities) to achieve safe navigation along the optimal path. By introducing a memory mechanism and an improved reward function, the algorithm effectively addresses the sparse reward problem in reinforcement learning, improving network update speed and learning efficiency while overcoming the drawbacks of high randomness and low learning rates. This enhances the algorithm's collision avoidance capabilities and demonstrates good generalization, improving the safety and efficiency of unmanned vessels in complex marine environments.

[0124] The specific simulation implementation and actual ship verification process corresponding to the solution of this embodiment are further elaborated below.

[0125] 1. Simulation experiment part:

[0126] 1.1 Initialize the training environment

[0127] A simulated ocean environment containing both dynamic and static obstacles is established on the simulation platform. Environmental parameters are set, including the water area (e.g., a 300×300m area), wave currents (size and direction), the number of obstacles (set to a fixed value, such as one static obstacle and three dynamic obstacles), and their locations, sizes, speeds, and headings. These parameters are dynamically simulated using a random number generator to simulate the complexity of a real-world ocean environment.

[0128] 1.2 Initialize the unmanned ship status

[0129] Set the initial position of the unmanned vessel ,speed , heading angle As well as thrust and torque. The initial state of the unmanned ship starts from a preset starting point and the target point position is ,The dynamic behavior model of the unmanned ship is described by combining the dynamic equation and the kinematic equation.

[0130] 1.3 Obtaining Environmental Information

[0131] In the simulation environment, the only information about the surrounding environment known to the unmanned ship is the location information of obstacles.

[0132] 1.4 Update memory space

[0133] The environmental information and historical state obtained by the unmanned ship are stored in the memory space for input to the subsequent decision network. Specifically, the memory space stores the state sequence of the last n steps. This short-term memory mechanism simulates human memory behavior and helps unmanned ships make reasonable decisions under limited perception conditions.

[0134] 1.5 Decision-making Process of Decision Network

[0135] The state sequence in the memory space Input to the policy network. The GRU module first processes the input time series data, extracts the dynamic features and time correlation of the environment, and then converts the hidden state vector The input is fed into the MLP, which outputs force and torque. The force and torque outputs are computed through the mean and standard deviation layers to generate a Gaussian distribution, which the policy network then uses to select actions.

[0136] 1.6 Execution of Decision-Making Behaviors

[0137] The UAV propels and steers within the simulation environment based on the thrust and torque output by the policy network. The thrust controls the UAV's forward speed, while the torque controls its heading angle. After executing the action, the simulation platform updates the UAV's state and calculates its distance to obstacles and the target point to facilitate subsequent reward calculations.

[0138] 1.7 Calculating Reward Value

[0139] After each action is executed, the reward value of the current state of the unmanned ship is calculated , used to optimize the decision network.

[0140] 1.8 Update the decision network

[0141] Through continuous interaction with the environment, the policy network is continuously optimized to improve the collision avoidance capability of the unmanned ship.

[0142] 1.9 Circuit Training

[0143] The above process is repeated until the unmanned vessel can effectively perform collision avoidance maneuvers and reach the target point. Through multiple rounds of training, the model is continuously optimized, and ultimately a stable decision network is obtained for actual collision avoidance decisions of the unmanned vessel.

[0144] 2. Actual ship verification part:

[0145] 2.1 Experimental Equipment and Setup

[0146] 2.1.1 Unmanned Boat Platform

[0147] The unmanned vessel used in the experiment was 2 meters long, constructed from carbon fiber, and possessed excellent navigation performance. The vessel was equipped with a variety of sensors (such as cameras, millimeter-wave radar, and GNSS) to collect environmental information. However, since the vessel was equipped with a single millimeter-wave radar with a monitoring range between -60° and 60°, it was unable to obtain precise information on the speed and size of obstacles, resulting in limited perception. The actual vessel's control system consisted of an onboard computer and a shore-based computer. The onboard computer enabled the unmanned vessel to execute policy decisions in real time through the policy network, while the shore-based computer was primarily responsible for data monitoring, collection, and experimental recording.

[0148] 2.1.2 Experimental Environment

[0149] The experiment took place in a water area approximately 500 meters by 500 meters. Dynamic obstacles (small, manned boats) were placed within the area. During the experiment, the wind speed was approximately 16 km / h, and the water surface was slightly choppy.

[0150] 2.2 Experimental Procedure

[0151] 2.2.1 Unmanned Boat Initialization

[0152] Set the initial position of the unmanned vessel in the experimental waters and target point The navigation task is set to start from the starting point and reach the target point. When the unmanned boat starts, it first obtains the starting position information and the target position, and the initial speed is set to 0.5m / s.

[0153] 2.2.2 Obtaining Environmental Information

[0154] During navigation, the unmanned vessel uses its onboard millimeter-wave radar to detect the location of obstacles and uses the GNSS module to obtain its own position data, forming a current state vector for subsequent input into the decision-making network. The sensor information is updated at a frequency of 10 Hz, ensuring that the unmanned vessel obtains timely environmental data in dynamic environments.

[0155] 2.2.3 Real-time decision-making and collision avoidance control

[0156] The unmanned ship inputs the collected state information into the decision network. The strategy network outputs thrust and torque based on the time series features extracted by the GRU module and the feature calculations of the MLP module.

[0157] 2.3 Experimental scenario testing

[0158] 2.3.1 Encounter

[0159] During the experiment, the unmanned boat encountered a small, human-operated boat head-on. According to COLREGs, the unmanned boat should have steered to the right to avoid the oncoming vessel. The results showed that the unmanned boat steered right, avoided the oncoming vessel, and successfully resumed its original course.

[0160] 2.3.2 Starboard Crossing

[0161] The target vessel approached from the right side of the unmanned vessel. The decision network determined that it was crossing to starboard, and the unmanned vessel should turn right to avoid it. In the experiment, the unmanned vessel demonstrated a smooth collision avoidance path and successfully avoided the target vessel.

[0162] 2.3.3 Port crossing

[0163] The target vessel approached from the left side of the unmanned vessel, which maintained its course and speed as required. During the experiment, the target vessel failed to take evasive action, posing a threat to the unmanned vessel. The unmanned vessel then took evasive action to the right, maintaining a safe distance from the target vessel.

[0164] 2.3.4 Overtaking

[0165] The unmanned vessel approached the target vessel from behind and attempted to overtake it. The decision network determined the target vessel's speed and heading based on its location information, selected the optimal overtaking route, and successfully completed the overtaking while maintaining a safe distance from the target vessel.

[0166] like Figures 8-17 As shown in the example, this example experimentally demonstrates the effectiveness and feasibility of the proposed unmanned vessel collision avoidance decision-making method based on a memory mechanism. Even with limited perception, the unmanned vessel can rationally assess the situation and make collision avoidance decisions consistent with COLREGs. Compared to traditional reinforcement learning algorithms, this method offers improved safety, stability, and generalization capabilities.

[0167] Based on the same inventive concept, the present invention also provides a computer device, which includes: one or more processors, and a memory for storing one or more computer programs; the program includes program instructions, and the processor is used to execute the program instructions stored in the memory. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is used to implement one or more instructions, specifically for loading and executing one or more instructions in a computer storage medium to implement the above method.

[0168] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium having a computer program stored thereon, which, when executed by a processor, performs the above-described method. The storage medium may be any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electrical, magnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0169] It should be noted that, unless otherwise defined, the technical or scientific terms used in the present invention should have the usual meanings understood by people with ordinary skills in the field to which the present invention belongs. The "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0170] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other manner. Any person skilled in the art may utilize the above-disclosed technical content to modify or modify the present invention into equivalent embodiments. However, any simple modifications, equivalent variations, and modifications to the above embodiments that do not depart from the technical content of the present invention and are based on the technical essence of the present invention remain within the scope of protection of the present invention.

[0171] The present invention is not limited to the above-mentioned optimal implementation mode. Anyone can derive other forms of ship tracking and control methods with predefined time requirements based on the inspiration of the present invention. All equivalent changes and modifications made within the scope of the patent application of the present invention should fall within the scope of the present invention.

Claims

1. A collision avoidance method for unmanned vessels based on deep reinforcement learning with a memory mechanism, characterized by: Dynamically store the historical navigation state sequence of the unmanned ship and build a fixed-length memory space. The state sequence includes the relative position of the target point, the relative position of the obstacle and the historical action data. The state sequence in the memory space is input into a reinforcement learning decision network, temporal features are extracted through a gated recurrent unit, and collision avoidance action instructions are generated in combination with a multilayer perceptron; The unmanned vessel is controlled to navigate based on the collision avoidance action instructions, and the network parameters are optimized by calculating the instantaneous reward value according to the composite reward function to achieve autonomous collision avoidance under limited perception; The memory space is implemented by a first-in-first-out queue, and the oldest state data is removed and the current state data is added each time it is updated; The hidden state dimension of the gated recurrent unit is 128, the MLP layer dimension is 256, and the decision network outputs Gaussian distribution parameters of propulsion force and torque; The network parameters are updated by continuous n-step historical data sampling and updated through KL divergence constraint strategy to avoid strategy oscillation caused by memory space data distribution offset.

2. The unmanned vessel collision avoidance method based on deep reinforcement learning with a memory mechanism according to claim 1, characterized in that: The reinforcement learning decision network adopts the soft actor-critic algorithm to optimize network parameters by minimizing the value network loss function and maximizing the policy entropy.

3. The unmanned vessel collision avoidance method based on deep reinforcement learning with a memory mechanism according to claim 1, characterized in that: The compound reward function includes: Target proximity reward: Dynamically adjusted based on the Euclidean distance between the unmanned ship and the target point; Collision avoidance safety reward: calculated segment by segment based on the ratio of the relative distance to the obstacle and the safety threshold, with a penalty applied when the distance falls below the minimum safety threshold; COLREGs compliance reward: Dynamically adjusts the penalty weight by counting the number of violations in historical action sequences.

4. An unmanned vessel collision avoidance decision system, characterized in that: include: A memory module is used to store the historical navigation state sequence of the most recent n steps; the state sequence includes the relative position of the target point, the relative position of the obstacle and the historical action data; A decision module, which integrates a reinforcement learning network of a gated recurrent unit and a multilayer perceptron, outputs collision avoidance action instructions based on the state sequence and optimizes network parameters through a composite reward function; A control module controls the propulsion and steering of the unmanned vessel based on the motion instructions; The training module optimizes network parameters through a composite reward function to achieve autonomous collision avoidance under limited perception; The memory module implements rolling storage of the state sequence through a first-in-first-out queue, removing the oldest state data and adding the current state data each time it is updated; The decision module has a GRU hidden layer dimension of 128 and an MLP layer dimension of 256, and outputs Gaussian distribution parameters of propulsion force and torque; The training module adopts continuous n-step historical data sampling and updates through the KL divergence constraint strategy.

5. The unmanned vessel collision avoidance decision system according to claim 4, characterized in that: The reinforcement learning network uses a soft actor-critic algorithm to optimize network parameters by minimizing the value network loss function and maximizing the policy entropy. The training module's composite reward function includes a target proximity reward, a collision avoidance safety reward, and a COLREGs compliance reward. The collision avoidance safety reward is calculated in segments based on the ratio of the relative distance to the obstacle to the safety threshold. The system also includes a millimeter-wave radar and a GNSS module. The millimeter-wave radar has a detection range of -60° to 60° and is used to obtain the relative position of obstacles. The GNSS module is used to locate the coordinates of the unmanned ship in real time.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 3 are implemented.

Images