Multi-mode reinforcement learning unmanned ship intelligent obstacle avoidance method and system based on attention driving

Abstract

The invention discloses a multi-modal reinforcement learning unmanned ship intelligent obstacle avoidance method and system based on attention driving. The method comprises the following steps: acquiring and preprocessing multi-modal environment information; performing feature extraction on the preprocessed multi-modal environment information by using a multi-branch deep neural network; inputting the extracted multi-modal features into a hierarchical attention fusion module, refining each single-modal key feature through an intra-modal attention layer and filtering redundant noise, and dynamically allocating multi-modal weights according to an environment context through an inter-modal attention layer to generate environment state characterization; splicing the environment state representation and the state information of the unmanned ship to obtain a complete state representation; and inputting the state representation into a reinforcement learning agent based on an SAC algorithm, and directly mapping the state representation into a continuous course and speed control instruction so as to drive the unmanned ship to execute an obstacle avoidance action. According to the invention, the accuracy and environmental adaptability of the intelligent obstacle avoidance action output of the unmanned ship can be obviously improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and autonomous decision-making technology for unmanned vessels, and in particular to an intelligent obstacle avoidance method, system, terminal device, and computer-readable storage medium for unmanned vessels based on attention-driven multimodal reinforcement learning. Background Technology

[0002] With the widespread application of unmanned surface vessel (USV) technology in fields such as marine surveying, water transportation, and environmental monitoring, its autonomous navigation and obstacle avoidance capabilities on water have become key core technologies. Reinforcement learning, especially deep reinforcement learning, provides an effective approach for autonomous decision-making by enabling intelligent agents to learn optimal strategies through interaction with the environment.

[0003] Traditional obstacle avoidance methods for unmanned surface vessels (USVs) often rely on single sensors such as lidar, cameras, and sonar, or employ simple multi-sensor data fusion strategies (such as the Chinese patent application CN202310792017.4), which fuses camera data with millimeter-wave radar data. However, these methods are still susceptible to the complex and ever-changing aquatic environment. Visual sensors are easily affected by fog, rain, light reflection, and nighttime conditions. Radar has limited detection capabilities for obstacles such as small, non-metallic buoys. While sonar can detect underwater obstacles, it suffers from sparse data and high noise levels. Existing obstacle avoidance methods based on reinforcement learning directly concatenate preprocessed sensor data into a state vector and input it into the policy network. This approach has significant drawbacks: First, the fusion strategy is rigid, employing a fixed-weight fusion method that cannot dynamically adjust the confidence levels of each sensor's information according to the current environment. In calm, clear daytime conditions, camera information should dominate, while in heavy fog, at night, or when water surface reflections are strong, radar and sonar information should be given higher weights. Second, the state vector contains redundancy and noise. Simply concatenating the high-dimensional features of all sensors leads to an explosion in the dimensionality of the state space, which not only increases the training burden of the reinforcement learning algorithm but also makes it difficult for the network to extract effective information from a large amount of redundant or occasionally malfunctioning sensor data. Third, it lacks focus on key spatiotemporal information. Due to the significant spatiotemporal characteristics of obstacles on the water surface, traditional methods make it difficult for the agent to adaptively focus on sensor signals from specific obstacles or specific areas that are crucial to the current decision, resulting in sluggish or unsafe decision-making.

[0004] Therefore, there is an urgent need for a technical solution that can adaptively fuse multimodal information and dynamically focus on key environmental features, thereby improving the accuracy, robustness, and efficiency of obstacle avoidance decisions by unmanned vessels in complex aquatic environments. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides an attention-driven multimodal reinforcement learning-based intelligent obstacle avoidance method, system, terminal device, and computer-readable storage medium for unmanned surface vessels. By constructing a multimodal reinforcement learning network with an embedded hierarchical attention mechanism and deeply integrating it into the state representation of reinforcement learning, dynamic and adaptive information fusion is achieved. The stochastic policy and entropy regularization mechanism of the SAC algorithm encourage full exploration, and within its actor-critic network structure, the fused state is directly mapped to the optimal continuous control action, thereby significantly improving the accuracy and environmental adaptability of the unmanned surface vessel's obstacle avoidance action output.

[0006] The first objective of this invention is to provide an intelligent obstacle avoidance method for unmanned surface vessels based on attention-driven multimodal reinforcement learning.

[0007] The second objective of this invention is to provide an intelligent obstacle avoidance system for unmanned surface vessels based on attention-driven multimodal reinforcement learning.

[0008] The third objective of this invention is to provide a terminal device.

[0009] A fourth objective of this invention is to provide a computer-readable storage medium.

[0010] The first objective of this invention can be achieved by adopting the following technical solution:

[0011] An attention-driven, multimodal reinforcement learning-based intelligent obstacle avoidance method for unmanned surface vessels, the method comprising:

[0012] Acquire multimodal environment information and perform preprocessing;

[0013] Feature extraction is performed on preprocessed multimodal environmental information using a multi-branch deep neural network;

[0014] The extracted multimodal features are input into the hierarchical attention fusion module. The key features of each single modality are refined and redundant noise is filtered out through the intramodal attention layer. Then, the multimodal weights are dynamically allocated according to the environmental context through the intermodal attention layer to adaptively generate a unified and refined environmental state representation.

[0015] By splicing together the environmental state representation and the unmanned vessel's own state information, a complete state representation is obtained.

[0016] The state representation is input into the reinforcement learning agent based on the SAC algorithm, and the state representation is directly mapped into continuous heading and speed control commands to drive the unmanned surface vessel to perform obstacle avoidance actions. Based on the obstacle avoidance actions performed by the unmanned surface vessel, the reward function is calculated and fed back to the reinforcement learning agent. Based on the SAC algorithm, the random policy and entropy regularization mechanism are used to encourage full exploration, so as to effectively handle the high-dimensional, continuous state and action space.

[0017] Preferably, a destination is set, and the current position of the unmanned vessel is used as the starting point;

[0018] The acquisition of multimodal environment information includes:

[0019] During the unmanned vessel's journey to its destination, there will be static and dynamic obstacles on the water surface and underwater. The system acquires real-time visual images through cameras, obtains real-time 3D point cloud data through lidar, acquires vector information of target distance and velocity through millimeter-wave radar, and acquires underwater depth and contour information in real time through sonar.

[0020] Preferably, the preprocessing includes timestamp alignment, noise reduction, and format unification of the multimodal environment information.

[0021] Preferably, the feature extraction of the preprocessed multimodal environment information using a multi-branch deep neural network includes:

[0022] Spatial features of the visual image are extracted using a convolutional neural network;

[0023] The three-dimensional structural features of the three-dimensional point cloud data are extracted using PointNet or a similar network.

[0024] The target motion and position features of the vector information are extracted using a multilayer perceptron.

[0025] The underwater depth and contour features of the sonar time-series data were extracted using a convolutional recurrent network.

[0026] The feature vectors output by each branch have a uniform dimension.

[0027] Preferably, the reinforcement learning agent includes an Actor network and a dual Critic network; the Actor network adopts an architecture with embedded hierarchical attention fusion modules to map complete state representations into probability distributions of continuous actions; the Critic network is used to evaluate the state-action Q-value.

[0028] Preferably, the environmental state characterization includes the location of the navigation endpoint, the location and size of static obstacles, the real-time location and motion state of dynamic obstacles, the water surface topography and water depth, and the unmanned vessel's own state information includes the unmanned vessel's own position, heading and speed.

[0029] The reward function r is a weighted sum of multiple reward components:

[0030] r = ω nav ·R nav (s t )+ω col ·R col (s t)+ω smo ·R smo (a t )+ω conf ·R conf (s t )

[0031] In the formula, ω nav ω col ω smo ω conf R represents the weighting coefficient for each reward component. nav (s t ) is for navigation rewards; R col (s t ) is the collision penalty; R smo (a t R represents a smooth reward; conf (s t (This refers to a reward weighted by perceived confidence level.)

[0032] in:

[0033] R nav (s t )=d t-1 -d t

[0034] In the formula, d t d is the Euclidean distance between the unmanned vessel and the target point, calculated based on the unmanned vessel's own position at time step t and the position of the destination point. t-1 The Euclidean distance between the unmanned vessel and the target point is calculated based on the unmanned vessel's own position at time step t-1 and the position of the end point of the voyage.

[0035]

[0036] In the formula, C col min_dist(s) is a very large negative constant; k is the scaling factor; t The distance between the unmanned vessel and the obstacle is the minimum distance calculated based on the unmanned vessel's own position and the obstacle's position at time step t; the obstacle includes static obstacles and dynamic obstacles;

[0037] R smo (a t )=-(α·|Δψ t | 2 +β·|Δv t | 2 )

[0038] In the formula, Δψ t and Δv t These are the actions of the unmanned vessel based on the current time step t, respectively.t The calculated changes in heading angle and velocity, action a t The complete state representation s at time step t t The decision is derived; α and β are coefficients that control the smoothness weights;

[0039] Action a t The complete state representation s at time step t t The decision was made as follows:

[0040] State representation s is performed using an Actor network. t Perform mapping and decision-making, and output the control action 'a' at the current time step. t Action a t This includes heading angle adjustment commands and speed control commands, used to guide unmanned surface vessels in obstacle avoidance and navigation.

[0041] R conf (s t =Confidence(s t )·(R nav (s t )+R col (s t ))

[0042] In the formula, Confidence(s t ) is the confidence scalar calculated based on the multimodal weights corresponding to the environmental state representation at time step t.

[0043] Preferably, the reinforcement learning agent is trained before the state representation is input into the reinforcement learning agent based on the SAC algorithm; the Actor network parameters in the trained reinforcement learning agent are used as the optimal obstacle avoidance strategy.

[0044] The second objective of this invention can be achieved by adopting the following technical solution:

[0045] An attention-driven, multimodal reinforcement learning-based intelligent obstacle avoidance system for unmanned surface vessels, the system comprising:

[0046] The data acquisition module is used to acquire multimodal environment information and perform preprocessing.

[0047] The feature extraction module is used to extract features from the preprocessed multimodal environmental information using a multi-branch deep neural network.

[0048] The weight fusion module is used to input the extracted multimodal features into the hierarchical attention fusion module. The intramodal attention layer refines the key features of each single modality and filters out redundant noise. Then, the intermodal attention layer dynamically allocates multimodal weights according to the environmental context to adaptively generate a unified and refined environmental state representation.

[0049] The stitching module is used to stitch together the environmental state representation and the unmanned vessel's own state information to obtain a complete state representation.

[0050] The intelligent obstacle avoidance module is used to input the state representation into the reinforcement learning agent based on the SAC algorithm, and directly map the state representation into continuous heading and speed control commands to drive the unmanned surface vessel to perform obstacle avoidance actions. Based on the obstacle avoidance actions performed by the unmanned surface vessel, the reward function is calculated and fed back to the reinforcement learning agent. Based on the SAC algorithm, the module uses a stochastic policy and entropy regularization mechanism to encourage full exploration, so as to effectively handle the high-dimensional, continuous state and action space.

[0051] The third objective of this invention can be achieved by adopting the following technical solution:

[0052] A terminal device includes a processor and a memory for storing processor-executable programs. When the processor executes the program stored in the memory, it implements the above-described attention-driven multimodal reinforcement learning-based intelligent obstacle avoidance method for unmanned surface vessels.

[0053] The fourth objective of this invention can be achieved by adopting the following technical solution:

[0054] A computer-readable storage medium storing a program that, when executed by a processor, implements the above-described attention-driven multimodal reinforcement learning-based intelligent obstacle avoidance method for unmanned surface vessels.

[0055] The present invention has the following advantages over the prior art:

[0056] (1) A hierarchical attention fusion mechanism is adopted to realize dynamic adaptive fusion of multimodal information: intramodal attention refines key features of a single modality and filters out redundant noise, while intermodal attention dynamically allocates the weights of each sensor according to the environmental context, which solves the problem of rigidity of traditional fixed weight fusion strategy; in complex marine environments, such as fog, rain, night, water surface reflection, etc., the sensor confidence can be adaptively adjusted to improve the accuracy and robustness of environmental perception.

[0057] (2) High-quality reinforcement learning state representation is adopted to reduce the training burden: multi-branch deep neural networks extract heterogeneous modal features in a customized manner, and combine them with hierarchical attention fusion to generate refined environmental state representations. This avoids the explosion of state space dimensions caused by simple splicing of high-dimensional features, reduces redundancy and noise interference, makes it easier for the network to extract effective decision information, and significantly reduces the training complexity of reinforcement learning algorithms.

[0058] (3) High-precision continuous control is achieved by using the SAC algorithm to improve the smoothness of obstacle avoidance actions: The random strategy and entropy regularization mechanism of the SAC algorithm are used to encourage environmental exploration. The fused state is directly mapped to continuous control commands of heading angle and speed through the Actor-Critic network structure. Combined with a multi-objective reward function that includes navigation, collision penalty, smoothness and perception confidence, the unmanned ship is guided to learn a safe, efficient and actual navigation-compliant obstacle avoidance strategy, avoiding energy waste and navigation risks caused by violent actions.

[0059] (4) Construct an end-to-end closed-loop control architecture to enhance adaptability to complex environments: integrate multimodal perception, attention fusion, reinforcement learning decision-making and execution feedback to form a real-time closed-loop system from environmental perception to action execution. It can quickly respond to dynamically changing marine environments and effectively handle scenarios where traditional sensors are weak, such as small non-metallic buoys and underwater obstacles. It significantly improves the accuracy, timeliness and environmental adaptability of obstacle avoidance decisions for unmanned vessels. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0061] Figure 1 This is a simplified flowchart of the intelligent obstacle avoidance method for unmanned surface vessels based on attention-driven multimodal reinforcement learning, according to an embodiment of the present invention.

[0062] Figure 2 This is a detailed flowchart of the intelligent obstacle avoidance method for unmanned surface vessels based on attention-driven multimodal reinforcement learning, according to an embodiment of the present invention.

[0063] Figure 3 This is a diagram of the multimodal reinforcement learning network structure according to an embodiment of the present invention;

[0064] Figure 4 This is a schematic diagram of the structure of a reinforcement learning agent according to an embodiment of the present invention;

[0065] Figure 5This is a structural block diagram of an attention-driven, multimodal reinforcement learning-based intelligent obstacle avoidance system for unmanned surface vessels, according to an embodiment of the present invention.

[0066] Figure 6 This is a partial structural schematic diagram of the attention-driven multimodal reinforcement learning-based unmanned surface vessel intelligent obstacle avoidance system according to an embodiment of the present invention.

[0067] Figure 7 This is a structural block diagram of a terminal device according to an embodiment of the present invention. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. It should be understood that the specific embodiments described are merely used to explain this application and are not intended to limit this application.

[0069] This embodiment provides an attention-driven, multimodal reinforcement learning-based intelligent obstacle avoidance method for unmanned surface vessels (USVs). It utilizes a combination of visible light cameras, lidar, millimeter-wave radar, and sonar systems to collaboratively acquire environmental information, constructing a multimodal reinforcement learning network embedded with a hierarchical attention mechanism, and employing the SAC algorithm to build an obstacle avoidance agent. This algorithm leverages its stochastic policy and entropy regularization mechanism to encourage thorough exploration, effectively handling high-dimensional, continuous state and action spaces. The method directly maps the fused environmental state representation into continuous control commands such as heading and speed, driving the thrusters and servos through the USV's control system to execute actions, achieving intelligent and adaptive obstacle avoidance for the USV in complex marine environments.

[0070] Reference Figure 1 , 2 The intelligent obstacle avoidance method for unmanned surface vessels based on attention-driven multimodal reinforcement learning provided in this embodiment includes the following steps:

[0071] S101. Obtain multimodal environment information and perform preprocessing.

[0072] Further, step S101 includes:

[0073] (1) Obtain multimodal environment information.

[0074] The multimodal sensor array includes visible light cameras, lidar, millimeter-wave radar, and sonar systems. The multimodal sensor array carried by the unmanned vessel can collect marine environmental information synchronously or asynchronously.

[0075] In a simulation or experimental environment, a navigation endpoint is set, with the current position of the unmanned surface vessel (USV) serving as the starting point. During the USV's journey to the endpoint, there will be static and dynamic obstacles on the water surface and underwater. Real-time visual images can be acquired via cameras, 3D point cloud data can be obtained in real-time via lidar, vector information such as target distance and velocity can be obtained in real-time via millimeter-wave radar, and underwater depth and contour information can be obtained in real-time via sonar.

[0076] (2) Preprocess the multimodal environment information.

[0077] The sensor data preprocessing module performs timestamp alignment, denoising, and format unification on the acquired modal environment information. Due to differences in sensor acquisition frequencies and startup sequences, timestamp alignment aims to unify the time base of data from various sensors and avoid timing misalignments. This can be achieved through linear interpolation or time synchronization algorithms, mapping the raw data from different sensors to the same time point. Denoising specifically suppresses the inherent noise of each modal data, improving the signal-to-noise ratio. Format unification standardizes heterogeneous data into a unified format and dimension suitable for subsequent network input, preparing for feature extraction.

[0078] S102. Extract features from the preprocessed multimodal environment information.

[0079] The preprocessed heterogeneous data is fed into a proprietary multi-branch deep neural network for feature extraction, where:

[0080] Visual images extract spatial features using branches of convolutional neural networks (CNNs);

[0081] 3D point cloud data is used to extract 3D structural features from PointNet or similar network branches;

[0082] Vector information is used to extract target motion and position features from the multilayer perceptron (MLP) branch;

[0083] Sonar time-series data were used to extract underwater depth and contour features using branches of a convolutional recurrent network (CNN / RNN).

[0084] Each branch outputs a feature vector of the same dimension, forming an intermediate feature space.

[0085] S103. Input the extracted multimodal features into the hierarchical attention fusion module, and generate an environmental state representation based on the dynamic weighted fusion of hierarchical attention.

[0086] The multimodal feature vectors obtained in step S102 are input into the hierarchical attention fusion module: through the intramodal attention layer, the features of each modality are refined using the self-attention mechanism to highlight key information; the refined features are input into the intermodal attention layer, and a dynamic weighted network calculates the contribution weight of each modality according to the current environmental context and performs weighted fusion to finally generate a unified and refined environmental state representation.

[0087] A multi-branch deep neural network and a hierarchical attention fusion module constitute a multimodal reinforcement learning network, which can be referenced. Figure 3 .

[0088] S104. Construct a state vector based on the environmental state representation, input the state vector into the trained reinforcement learning agent based on the SAC algorithm, and output the optimal continuous action instructions through the Actor network in the agent.

[0089] The environmental state representation obtained in step S103 is concatenated with the unmanned vessel's own state information (including position, heading, and velocity) to form the complete state vector s of the reinforcement learning agent. t The state vector is input into a pre-trained reinforcement learning agent based on the SAC (Soft Actor-Critic) algorithm. The agent includes an Actor-Critic network, which directly outputs optimal continuous action commands, i.e., heading and speed control commands. These commands are then transmitted to the unmanned surface vessel (USV) control system to drive the USV to perform obstacle avoidance maneuvers. Simultaneously, the reward function evaluates the action's effectiveness and feeds back the feedback information to the SAC agent's Actor-Critic network, forming a closed-loop feedback loop that helps the agent optimize its decisions.

[0090] Reinforcement learning agents are centered around state, action, and reward. The agent's goal is to learn a policy that maximizes the accumulated expected reward through interaction with the environment. The SAC algorithm builds upon this by introducing an entropy term, encouraging the policy to maintain randomness while pursuing high rewards. Its objective function is:

[0091]

[0092] Where π is the strategy function, t is the discrete time step, T is the round termination time step, and s t Let J(π) be the complete state vector of the agent at time step t, and J(π) be the objective function value of policy π. Let ρ be the marginal distribution of state and action. π The expectation operator, r(s) t ,a tLet be the immediate reward function at time step t, and α be the temperature coefficient used to control the importance of the entropy term. t )) is the policy entropy, used to quantify the policy π in the current state s. t To address the randomness of actions, we construct an Actor network to approximate the policy function π(a|s) and a Critic network to approximate the state-action value function Q(s,a), thus solving the decision-making problem in a high-dimensional continuous state and action space.

[0093] The constructed agent neural network structure can be referenced. Figure 4 The complete state vector is input into the Actor network, which is embedded in a hierarchical attention fusion module. After network computation, the probability distributions of two continuous actions, heading angle and velocity, are output. The Critic network adopts a dual-Q network structure, with state and action vectors as inputs and corresponding Q values ​​as outputs to evaluate the value of the actions.

[0094] The steps for training an agent are as follows:

[0095] (1) Initialize network parameters: Set the Actor network parameter φ, the Critic network parameters θ1, θ2 and the Target network parameters θ′1, θ′2, and copy the Critic network parameters to the Target network (θ1→θ′1, θ2→θ′2); initialize the temperature coefficient α, and set the total number of iterations T. total The discount factor γ, the experience replay buffer capacity N, the number of samples per sampling m, and the target network soft update coefficient τ are defined. The experience replay buffer D is initialized to store the interaction samples (s,a,r,s′).

[0096] (2) For each training round within the total number of iterations, initialize the simulation or experimental environment, reset the unmanned ship to the initial state, obtain the initial state s, set the initial reward r = 0, and set the round end flag done = False, where done indicates that the destination has been reached or a collision has occurred.

[0097] (3) Perform the following loop for each step in a round:

[0098] ① Convert the complete state vector s at time step t. t Input the Actor network and sample the action a at time step t. t (Based on the probability distribution of the network output); to encourage exploration, small Gaussian noise can be added to the action output.

[0099] ② Perform action a t Drive the unmanned vessel to adjust its course and speed, and acquire new state s t 'The reward function r of environmental feedback (including the positive reward R for approaching the target) nav Penalty and reward for colliding with obstacles (R)col Regularization reward R for heading smoothness smo Weighted reward R based on multimodal perception confidence conf ), which is specifically represented as follows:

[0100] r = ω nav ·R nav (s t )+ω col ·R col (s t )+ω smo ·R smo (a t )+ω conf ·R conf (s t )

[0101] Where, ω nav ω col ω smo ω conf These are the weighting coefficients for each reward component, used to adjust the importance of different objectives in the total reward.

[0102] R nav (s t )=d t-1 -d t

[0103] Where, d t This represents the Euclidean distance between the unmanned vessel and the target point at time step t.

[0104]

[0105] Among them, C col It is a very large negative constant, serving as the final penalty when a collision occurs; to avoid collisions, a penalty based on the nearest obstacle distance min_dist(s) is introduced. t ) 2 A dense penalty is applied when the unmanned vessel is less than a preset safety threshold (danger distance) from any obstacle. The penalty value is inversely proportional to the square of the distance (k is a scaling factor), meaning that the closer to the obstacle, the greater the penalty, thus strongly encouraging the agent to stay away from danger.

[0106] R smo (a t )=-(α·|Δψ t | 2 +β·|Δv t | 2 )

[0107] Where, Δψ t and Δv tThese are the changes in heading angle and velocity at the current time step t, respectively; α and β are coefficients that control the smoothness weights.

[0108] This formula penalizes drastic and discontinuous control actions, and its quadratic form can significantly amplify the cost of large-amplitude actions, thereby guiding the agent to learn a smooth, energy-efficient control strategy that conforms to actual navigation standards.

[0109] R conf (s t =Confidence(s t )·(R nav (s t )+R col (s t ))

[0110] Among them, Confidence(s t ) is a scalar derived from the weight entropy or minimum weight of the intermodal attention layer output in the hierarchical attention fusion module; its implementation is Confidence(s t ) = 1 - NormalizedEntropy(W t ), W t It is the attention weight vector of all sensor modalities. A low entropy value indicates that the system is very confident in the current fusion result.

[0111] ③ Store the learning samples (s,a,r,s') into the experience replay buffer D.

[0112] ④ Once the number of samples in the buffer reaches a preset threshold, randomly sample m samples {(s) from D. i ,a i ,r i ,s′ i )}.

[0113] ⑤ For each sampled sample, calculate the target Q value: if done i If True, then the target value y i =r i Otherwise, sample the next action a′~πφ(s) from the Actor network. i '), Calculate the target value:

[0114]

[0115] The log term is the entropy regularization term.

[0116] ⑥ Minimize the loss function and To achieve this, update the dual-commenter network parameters θ1 and θ2.

[0117] ⑦ By maximizing the actor's objective function (where a~πφ(s)) Update the actor network parameter φ.

[0118] ⑧ Adjust the temperature coefficient α to satisfy the entropy target constraint (such as the preset target entropy value H_target), and minimize the loss L(α)=-E[α·(logπφ(a|s)+H_target)] through gradient descent.

[0119] ⑨ Update the target commentator network parameters using a soft update method: θ'1←τθ1+(1-τ)θ1', θ'2←τθ2+(1-τ)θ2'.

[0120] (4) Repeat step (3) until the current round ends (done = True).

[0121] (5) After completing one iteration, if the total number of iterations T has not been reached. total If the condition is met, return to step (2) to reinitialize the environment and start the next round of training; otherwise, end the training and save the final actor network parameters φ as the optimal obstacle avoidance strategy.

[0122] S105. Based on continuous action instructions, execute the corresponding actions to complete obstacle avoidance maneuvers; calculate the reward function based on the environmental state after the action is executed and return it to the agent to form closed-loop control.

[0123] The action commands generated in step S104 are sent to the unmanned surface vessel's underlying control system, driving the thrusters and servos to perform corresponding obstacle avoidance maneuvers. Simultaneously, the system calculates the reward function Rt based on the new state after execution, evaluates the action's effectiveness, and uses this as feedback for online learning or strategy optimization, forming a real-time closed-loop control system from perception and decision-making to execution.

[0124] This embodiment provides an attention-driven, multimodal reinforcement learning-based intelligent obstacle avoidance method for unmanned surface vessels (USVs). It acquires current environmental state information through the collaborative use of visible light cameras, LiDAR, millimeter-wave radar, and sonar systems, and constructs an obstacle avoidance agent using the SAC algorithm. This algorithm, based on the maximum entropy reinforcement learning framework, handles the continuous action space problem through its actor-critic network structure and utilizes entropy regularization to encourage the agent to explore fully, effectively improving the robustness and environmental adaptability of the strategy. Since different sensor data exhibit significant differences in dimension, structure, and physical meaning (e.g., images, point clouds, vectors, time-series signals), a hierarchical attention fusion module is constructed. Feature refinement is performed within each modality, followed by dynamic weighting between modalities, thus solving the heterogeneity and information redundancy problems of multimodal data and achieving adaptive, context-aware information fusion. Based on the continuous action commands (heading angle and velocity) output by the SAC algorithm, the USV's control system directly drives the thrusters and servos, achieving intelligent and smooth obstacle avoidance.

[0125] This embodiment also provides an attention-driven, multimodal reinforcement learning-based intelligent obstacle avoidance system for unmanned surface vessels, which can be referred to in the appendix. Figure 5 It includes a data acquisition module 501, a feature extraction module 502, a weight fusion module 503, a stitching module 504, and an intelligent obstacle avoidance module 505, wherein:

[0126] Data acquisition module 501 is used to acquire multimodal environment information and perform preprocessing;

[0127] The feature extraction module 502 is used to extract features from the preprocessed multimodal environmental information using a multi-branch deep neural network;

[0128] The weight fusion module 503 is used to input the extracted multimodal features into the hierarchical attention fusion module, refine the key features of each single modality and filter out redundant noise through the intramodal attention layer, and then dynamically allocate multimodal weights according to the environmental context through the intermodal attention layer to adaptively generate a unified and refined environmental state representation.

[0129] The splicing module 504 is used to splice the environmental state representation and the state information of the unmanned vessel itself to obtain a complete state representation.

[0130] The intelligent obstacle avoidance module 505 is used to input the state representation into the reinforcement learning agent based on the SAC algorithm, and directly map the state representation into continuous heading and speed control commands to drive the unmanned vessel to perform obstacle avoidance actions; based on the obstacle avoidance actions performed by the unmanned vessel, the reward function is calculated and fed back to the reinforcement learning agent; based on the SAC algorithm, the random policy and entropy regularization mechanism are used to encourage full exploration, so as to effectively handle the high-dimensional, continuous state and action space.

[0131] Specifically, the data acquisition module 501 includes a multimodal sensor array and a preprocessing unit; the multimodal sensor array is used to acquire multimodal environmental information, and the preprocessing unit is used to preprocess the multimodal environmental information. (See reference...) Figure 6 .

[0132] Specifically, modules 501 to 505 serve as decision-making modules, including embedded computing units, which are used to ultimately run a reinforcement learning agent based on the SAC algorithm to generate control commands based on the preprocessed multimodal environment information.

[0133] The system also includes an execution structure. The execution structure includes a control system, a thruster, and a servo motor, which are used to execute obstacle avoidance actions according to control commands. The result of executing obstacle avoidance actions will change the environmental state of the unmanned vessel. The changed environmental state is captured again by the data acquisition module and processed to form a closed-loop feedback.

[0134] This system constitutes a complete closed loop of acquisition, decision-making, and execution.

[0135] The system is centered around a controller. The system's information flow originates from environmental perception sensors (including but not limited to cameras, LiDAR, millimeter-wave radar, and sonar) and the unmanned surface vessel's (USV) positioning system. Sensor data and positioning signals collected by these sensors are transmitted to the controller; the controller runs a reinforcement learning agent algorithm to fuse multimodal perception data and make decisions, then outputs control commands. These commands are sent to the actuators, which generate physical actions to drive the USV's movement. The USV's movement constitutes an interaction with the mission environment, and changes in the environmental state are again sensed by the sensors, thus forming a real-time autonomous control closed loop.

[0136] The system can interact with shore-based or shipboard computers via a wireless communication module to enable status monitoring, command issuance, or manual intervention.

[0137] This embodiment constructs an end-to-end intelligent decision-making system that adaptively fuses multimodal sensor data through a hierarchical attention mechanism to generate a refined environmental state representation. It then directly outputs continuous and smooth control commands using the SAC algorithm. This not only effectively solves the fusion challenge caused by the heterogeneity of multimodal data but also guides the agent to learn safe, efficient, and practical navigation behaviors through a carefully designed reward function. In simulations or experiments, it significantly improves the obstacle avoidance intelligence, decision-making accuracy, and overall robustness of unmanned surface vessels (USVs) in complex and dynamic marine environments, promoting research on high-level autonomous navigation of USVs.

[0138] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware, and the corresponding program can be stored in a computer-readable storage medium.

[0139] It should be noted that although the method operations of the above embodiments are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. On the contrary, the order of execution of the described steps may be changed. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0140] This embodiment also provides a terminal device, which can be a computer, such as... Figure 7 As shown, the system bus 701 connects a processor 702, a memory, an input device 703, a display 704, and a network interface 705. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium 706 and an internal memory 707. The non-volatile storage medium 706 stores the operating system, computer programs, and a database. The internal memory 707 provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. When the processor 702 executes the computer programs stored in the memory, it implements the aforementioned attention-driven multimodal reinforcement learning-based intelligent obstacle avoidance method for unmanned surface vessels.

[0141] This embodiment also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described attention-driven multimodal reinforcement learning-based intelligent obstacle avoidance method for unmanned surface vessels.

[0142] It should be noted that the computer-readable storage medium in this embodiment can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0143] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope disclosed in the present invention, based on the technical solution and inventive concept of the present invention, shall fall within the scope of protection of the present invention.

Claims

1. A method for intelligent obstacle avoidance of unmanned surface vessels based on attention-driven multimodal reinforcement learning, characterized in that, The method includes: Acquire multimodal environment information and perform preprocessing; Feature extraction is performed on preprocessed multimodal environmental information using a multi-branch deep neural network; The extracted multimodal features are input into the hierarchical attention fusion module. The key features of each single modality are refined and redundant noise is filtered out through the intramodal attention layer. Then, the multimodal weights are dynamically allocated according to the environmental context through the intermodal attention layer to adaptively generate a unified and refined environmental state representation. By splicing together the environmental state representation and the unmanned vessel's own state information, a complete state representation is obtained. The state representation is input into the reinforcement learning agent based on the SAC algorithm, and the state representation is directly mapped into continuous heading and speed control commands to drive the unmanned surface vessel to perform obstacle avoidance actions. Based on the obstacle avoidance actions performed by the unmanned surface vessel, the reward function is calculated and fed back to the reinforcement learning agent. Based on the SAC algorithm, the random policy and entropy regularization mechanism are used to encourage full exploration, so as to effectively handle the high-dimensional, continuous state and action space.

2. The multimodal reinforcement learning-based intelligent obstacle avoidance method for unmanned surface vessels according to claim 1, characterized in that, Set the destination of the voyage, and use the current position of the unmanned vessel as the starting point; The acquisition of multimodal environment information includes: During the unmanned vessel's journey to its destination, there will be static and dynamic obstacles on the water surface and underwater. The system acquires real-time visual images through cameras, obtains real-time 3D point cloud data through lidar, acquires vector information of target distance and velocity through millimeter-wave radar, and acquires underwater depth and contour information in real time through sonar.

3. The multimodal reinforcement learning-based intelligent obstacle avoidance method for unmanned surface vessels according to any one of claims 1 and 2, characterized in that, The preprocessing includes timestamp alignment, noise reduction, and format unification of the multimodal environment information.

4. The multimodal reinforcement learning-based intelligent obstacle avoidance method for unmanned surface vessels according to claim 2, characterized in that, The feature extraction of the preprocessed multimodal environment information using a multi-branch deep neural network includes: Spatial features of the visual image are extracted using a convolutional neural network; The three-dimensional structural features of the three-dimensional point cloud data are extracted using PointNet or a similar network. The target motion and position features of the vector information are extracted using a multilayer perceptron. The underwater depth and contour features of the sonar time-series data were extracted using a convolutional recurrent network. The feature vectors output by each branch have a uniform dimension.

5. The multimodal reinforcement learning-based intelligent obstacle avoidance method for unmanned surface vessels according to claim 1, characterized in that, The reinforcement learning agent includes an Actor network and a dual Critic network; the Actor network adopts an architecture with embedded hierarchical attention fusion modules to map complete state representations into probability distributions of continuous actions; the Critic network is used to evaluate the state-action Q-value.

6. The multimodal reinforcement learning-based intelligent obstacle avoidance method for unmanned surface vessels according to any one of claims 1 and 5, characterized in that, The environmental state representation includes the location of the navigation endpoint, the location of static obstacles, and the real-time location of dynamic obstacles. The unmanned vessel's own state information includes its own position, heading, and speed. The reward function r is a weighted sum of multiple reward components: r=ω nav ·R nav (s t )+ω col ·R col (s t )+ω smo ·R smo (a t )+ω conf ·R conf (s t ) In the formula, ω nav ω col ω smo ω conf R represents the weighting coefficient for each reward component. nav (s t ) is for navigation rewards; R col (s t ) is the collision penalty; R smo (a t R represents a smooth reward; conf (s t (This refers to a reward weighted by perceived confidence level.) in: R nav (s t )=d t-1 -d t In the formula, d t d is the Euclidean distance between the unmanned vessel and the target point, calculated based on the unmanned vessel's own position at time step t and the position of the destination point. t-1 The Euclidean distance between the unmanned vessel and the target point is calculated based on the unmanned vessel's own position at time step t-1 and the position of the end point of the voyage. In the formula, C col min_dist(s) is a very large negative constant; k is the scaling factor; t The distance between the unmanned surface vessel (USV) and the obstacle is the minimum distance calculated based on the USV's own position and the obstacle's position at time step t; the obstacle is... Including static obstacles and dynamic obstacles; R smo (a t )=-(α·|Δψ t | 2 +β·|Δv t | 2 ) In the formula, Δψ t and Δv t These are the actions of the unmanned vessel based on the current time step t, respectively. t The calculated changes in heading angle and velocity, action a t The complete state representation s at time step t t The decision is derived; α and β are coefficients that control the smoothness weights; R conf (s t )=Confidence(s t )·(R nav (s t )+R col (s t )) In the formula, Confidence(s t ) is the confidence scalar calculated based on the multimodal weights corresponding to the environmental state representation at time step t.

7. The multimodal reinforcement learning-based intelligent obstacle avoidance method for unmanned surface vessels according to claim 5, characterized in that, Before inputting the state representation into the reinforcement learning agent based on the SAC algorithm, the reinforcement learning agent is trained; the Actor network parameters in the trained reinforcement learning agent are used as the optimal obstacle avoidance strategy.

8. An attention-driven, multimodal reinforcement learning-based intelligent obstacle avoidance system for unmanned surface vessels, characterized in that, The system includes: The data acquisition module is used to acquire multimodal environment information and perform preprocessing. The feature extraction module is used to extract features from the preprocessed multimodal environmental information using a multi-branch deep neural network. The weight fusion module is used to input the extracted multimodal features into the hierarchical attention fusion module. The intramodal attention layer refines the key features of each single modality and filters out redundant noise. Then, the intermodal attention layer dynamically allocates multimodal weights according to the environmental context to adaptively generate a unified and refined environmental state representation. The stitching module is used to stitch together the environmental state representation and the unmanned vessel's own state information to obtain a complete state representation. The intelligent obstacle avoidance module is used to input the state representation into the reinforcement learning agent based on the SAC algorithm, and directly map the state representation into continuous heading and speed control commands to drive the unmanned surface vessel to perform obstacle avoidance actions. Based on the obstacle avoidance actions performed by the unmanned surface vessel, the reward function is calculated and fed back to the reinforcement learning agent. Based on the SAC algorithm, the module uses a stochastic policy and entropy regularization mechanism to encourage full exploration, so as to effectively handle the high-dimensional, continuous state and action space.

9. A terminal device, comprising a processor and a memory for storing a processor-executable program, characterized in that, When the processor executes the program stored in the memory, it implements the multimodal reinforcement learning unmanned surface vessel intelligent obstacle avoidance method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multimodal reinforcement learning unmanned surface vessel intelligent obstacle avoidance method according to any one of claims 1 to 7.

Images