Intelligent identification method for motion mode of unmanned surface vehicle based on deep learning

Abstract

The invention discloses a deep learning-based intelligent recognition method for a motion mode of an unmanned surface vehicle. The method comprises the following steps: acquiring original sensing data in real time through a multi-source sensor system mounted on the unmanned surface vehicle, wherein the multi-source sensor system at least comprises an inertial measurement unit (IMU), a global navigation satellite system (GNSS) receiver, a laser radar and a visible light camera; performing time synchronization and coordinate system alignment processing on the original sensing data to generate a fused sensing data stream with a unified timestamp and a spatial reference system; dividing the fused sensing data stream into time window segments with fixed lengths, wherein each time window segment corresponds to one to-be-identified motion state sample; the IMU, the GNSS, the laser radar and the visible light / infrared camera are synchronously integrated, quadruple sensing dimensions are formed, the IMU captures instantaneous acceleration and angular velocity changes and reflects control instruction execution, the reliability of single-point observation is improved, and the system can still maintain the basic recognition capability when part of sensors fail.

Description

Technical Field

[0001] This invention belongs to the field of unmanned surface vessel (USV) motion technology, specifically a method for intelligent recognition of motion patterns of USVs based on deep learning. Background Technology

[0002] With the rapid development of intelligent shipping and unmanned systems technologies, unmanned surface vehicles (USVs) are widely used in marine surveying, environmental monitoring, port inspection, and emergency rescue scenarios. Motion pattern recognition, which determines whether an USV is currently in a typical behavior such as "straight-line cruising," "turning in place," "snake-like avoidance," "emergency braking," or "berthing," not only provides key inputs for path planning, behavioral decision-making, and human-machine collaboration, but also serves as an important basis for anomaly detection and safety control. However, in existing technologies, motion pattern recognition for USVs still faces the following technical challenges:

[0003] First, most studies rely solely on GNSS trajectory data or IMU signals for pattern discrimination. For example, some methods distinguish between "stationary" and "moving" states using velocity thresholds, or roughly differentiate between "straight ahead" and "turning" states using the rate of change of heading angle. GNSS signals often experience signal loss or multipath effects under bridges or in densely populated port areas, leading to position jumps; IMUs, on the other hand, generate high-frequency noise due to wave disturbances, making it difficult to distinguish between genuine control and passive swaying. Furthermore, a single sensor cannot capture the complete semantics of behavior. For instance, "berthing" requires multiple conditions to be met simultaneously, including "velocity approaching zero," "continuous approach to a fixed obstacle," and "no significant lateral displacement," which cannot be independently determined by GNSS or IMU alone.

[0004] Secondly, even when some studies attempt to integrate IMU, GNSS and lidar, they often overlook the accuracy requirements of time synchronization and coordinate alignment. Because each sensor has an independent sampling clock and different communication delays, the raw data naturally has a time offset of tens or even hundreds of milliseconds. Furthermore, IMU attitude, lidar point cloud and image pixels are based on their respective local coordinate systems. If they are not unified to the same geographic reference frame (such as the ENU system), the "same physical event" will be spatially misaligned in different modes.

[0005] Furthermore, some solutions use pure convolutional networks to process time-series data. While these can extract instantaneous features, they cannot model complex behavioral sequences spanning several seconds, such as "deceleration, then turning, and finally docking." Others directly use LSTM or Transformer, but these suffer from unstable training or poor generalization ability due to excessively long input sequences and significant noise interference. In addition, most models use downsampled low-frequency data (such as below 10Hz) during the training phase, losing crucial transient details such as acceleration spikes and angular velocity plateaus, resulting in severely insufficient sensitivity to recognize short-term, high-dynamic behaviors like "emergency braking." Summary of the Invention

[0006] In order to overcome the shortcomings of the prior art, this invention provides a method for intelligent recognition of motion patterns of unmanned surface vessels based on deep learning, so as to at least partially solve the above-mentioned technical problems.

[0007] The technical solution adopted in this invention is as follows:

[0008] This invention proposes a deep learning-based intelligent recognition method for the motion patterns of unmanned surface vessels, comprising the following steps:

[0009] Step S1: Collect raw sensor data in real time through a multi-source sensor system installed on the unmanned surface vessel. The multi-source sensor system includes at least an inertial measurement unit (IMU), a global navigation satellite system (GNSS) receiver, a lidar, and a visible light camera.

[0010] Step S2: Perform time synchronization and coordinate system alignment processing on the original sensing data to generate a fused sensing data stream with a unified timestamp and spatial reference system;

[0011] Step S3: Divide the fused sensing data stream into time window segments of fixed length, with each time window segment corresponding to a motion state sample to be identified;

[0012] Step S4: Construct a hybrid deep neural network model consisting of a cascaded one-dimensional convolutional neural network and a bidirectional long short-term memory network (BiLSTM). The one-dimensional convolutional layer is used to extract local temporal features, and the BiLSTM layer is used to model long-range dependencies.

[0013] Step S5: Use the historical unmanned surface vessel operation dataset labeled with motion mode tags to train the hybrid deep neural network model end-to-end. The motion mode tags include, but are not limited to, five categories: straight cruise, turning in place, serpentine avoidance, emergency braking, and docking.

[0014] Step S6: Input the real-time collected and preprocessed fused sensor data stream into the trained hybrid deep neural network model, and output the motion mode category and confidence score of the current unmanned surface vessel.

[0015] In one embodiment of the present invention, in step S2, the time synchronization processing adopts a hardware trigger signal or a software interpolation alignment method to ensure that the time deviation of each sensor data does not exceed 10 milliseconds; the coordinate system alignment processing includes converting the attitude angle output by the IMU to the Northeast ENU coordinate system with the GNSS positioning point as the origin, and projecting the lidar point cloud and the visible light image to the same world coordinate system through calibration parameters to ensure that the spatial consistency error is less than 0.1 meters.

[0016] In one embodiment of the present invention, in step S3, the length of the time window segment is set to be between 5 and 15 seconds, the sliding step is between 1 and 3 seconds, and a continuous sensing data sequence with an original sampling frequency of not less than 50 Hz is retained in each time window to maintain the integrity of motion dynamic details.

[0017] In one embodiment of the present invention, the one-dimensional convolutional neural network part of the hybrid deep neural network model includes at least three convolutional modules. Each convolutional module includes a one-dimensional convolutional layer, a batch normalization layer, a ReLU activation function, and a max pooling layer in sequence. The convolutional kernel sizes are 64, 32, and 16 in sequence, and the number of channels are 64, 128, and 256 in sequence. The BiLSTM layer includes two stacked BiLSTM units, each unit having a hidden state dimension of 256. The output of the BiLSTM is connected to a Dropout layer with a dropout rate of 0.5.

[0018] In one embodiment of the present invention, in step S5, the process of constructing the historical unmanned surface vessel operation dataset includes: operating the unmanned surface vessel to perform standardized motion tasks in various water environments, and simultaneously recording the corresponding multi-source sensor data and manually labeled motion patterns; at the same time, introducing data augmentation strategies to apply Gaussian noise, time axis stretching or compression, and sensor missing simulation perturbation to the original data to improve the robustness of the model.

[0019] In one embodiment of the present invention, an anomaly detection mechanism is further included after step S6: when the highest confidence score of the model output is lower than a preset threshold of 0.7, or when the prediction results of three consecutive time windows switch frequently and there is no dominant category, an anomaly status flag is triggered, and a backup rule engine is started to assist in the judgment. The rule engine sets threshold logic based on the rate of change of velocity, the rate of change of heading angle, and the physical quantity of obstacle distance.

[0020] In one embodiment of the present invention, the visible light camera automatically switches to infrared imaging mode at night or in low light conditions, and inputs the visible light image and infrared image into the model after pixel-level alignment by an image fusion algorithm to ensure all-weather recognition capability; the image fusion algorithm adopts a gradient domain-based multi-scale weighted averaging method to preserve edge details while suppressing noise.

[0021] In one embodiment of the present invention, the hybrid deep neural network model is deployed on a local embedded computing platform of the unmanned surface vessel. The platform uses NVIDIA Jetson AGX Orin or an AI acceleration chip with equivalent computing power, and uses TensorRT to quantize and optimize the model, so that the single inference latency is controlled within 50 milliseconds, which meets the real-time requirements.

[0022] In one embodiment of the present invention, an online model update mechanism is also included: when the unmanned surface vessel returns to port, the data samples marked as "uncertain" or "abnormal" during its operation are uploaded to the shore-based server. The server manually reviews and corrects the labels of the incremental data, and then the original model is fine-tuned using a transfer learning strategy. The updated model version is then pushed to the unmanned surface vessel through a secure communication protocol to achieve continuous evolution of recognition capabilities.

[0023] In one embodiment of the present invention, the motion pattern category expansion supports the addition of custom modes. The training of the new modes only requires providing no less than 200 time window samples and freezing the bottom convolution and BiLSTM parameters of the hybrid deep neural network model. Only the final fully connected classification layer and a few adaptation layers are fine-tuned, so as to quickly adapt to new task scenarios, such as formation cooperative navigation and autonomous search and rescue trajectory tracking, while maintaining the original recognition performance.

[0024] The beneficial effects of the technical solution of this invention are as follows:

[0025] This invention integrates an IMU (Induction Detection Unit), GNSS (Global Position Reference), LiDAR (Geometric Environment Constraints), and a visible / infrared camera (Visual Semantic Clues) to form a four-dimensional perception system. The IMU captures instantaneous acceleration and angular velocity changes to reflect the execution of control commands; GNSS provides macroscopic trajectory trends to determine the cruising or berthing phase; LiDAR quantifies obstacle distances and relative motions to support obstacle avoidance and berthing behavior recognition; and the vision system supplements the high-level semantics of dock markings, water surface textures, and lighting conditions, not only improving the reliability of single-point observations but also enabling the system to maintain basic recognition capabilities even when some sensors fail (such as GNSS lock-off or camera fogging).

[0026] This invention achieves sub-millisecond time alignment through hardware triggering or high-order interpolation, and based on the ENU geographic coordinate system—a spatial reference for all sensors—ensures that physical events at the same time and location correspond accurately at the data level. This enables subsequent deep networks to extract effective features from a consistent spatiotemporal context, improving the accuracy of modeling causal relationships of dynamic behaviors.

[0027] This invention retains an original sampling rate of no less than 50Hz, fully preserving key dynamic details such as acceleration spikes, angular velocity plateaus, and exponential distance decay, providing the model with sufficiently fine temporal cues. A sliding step size of 1-3 seconds ensures recognition frequency (multiple outputs per second) while avoiding redundant computation, meeting the state update rate requirements of the upper-level navigation system. The front-end 1D-CNN captures local dynamic primitives from macroscopic trends to microscopic mutations step by step using large convolutional kernels (64→32→16); the back-end BiLSTM utilizes a bidirectional memory mechanism to model long-range dependencies, understanding the complex behavioral logic of "deceleration → turning → approaching". The increasing number of channels (64→128→256) and the hidden state dimension (256) achieve a balance between expressive power and computational overhead. The Dropout layer (0.5) further suppresses overfitting, enabling the model to maintain good generalization performance even with limited real-world ship data.

[0028] This invention constructs a labeled dataset covering typical operating conditions by performing standardized tasks in diverse scenarios such as open waters, narrow channels, and ports. Gaussian noise, time stretching, and sensor-missing perturbations are then applied to simulate degradation conditions in real-world operation. Simultaneously, when the model confidence is low or predicts oscillations, the system automatically activates hard criteria based on physical laws (such as acceleration thresholds, rate of change of heading, and obstacle distances) to avoid decision-making errors caused by the unreliability of deep learning "black box" outputs.

[0029] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0030] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0031] Figure 1 This is a method framework diagram of the intelligent recognition method for motion patterns of unmanned surface vessels based on deep learning proposed in an embodiment of the present invention;

[0032] Figure 2 This is a first extended framework diagram of the intelligent recognition method for motion patterns of unmanned surface vessels based on deep learning proposed in an embodiment of the present invention;

[0033] Figure 3 This is a second extended framework diagram of the intelligent recognition method for motion patterns of unmanned surface vessels based on deep learning proposed in an embodiment of the present invention. Detailed Implementation

[0034] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0035] The following describes an embodiment of the present invention with reference to the accompanying drawings, a method for intelligent recognition of motion patterns of unmanned surface vessels based on deep learning.

[0036] like Figures 1 to 3 As shown, this embodiment of the invention provides a method for intelligent recognition of motion patterns of unmanned surface vessels based on deep learning, including the following steps:

[0037] Step S1: Collect raw sensor data in real time through a multi-source sensor system installed on the unmanned surface vessel. The multi-source sensor system includes at least an inertial measurement unit (IMU), a global navigation satellite system (GNSS) receiver, a lidar, and a visible light camera.

[0038] Step S2: Perform time synchronization and coordinate system alignment processing on the raw sensor data to generate a fused sensor data stream with a unified timestamp and spatial reference system;

[0039] Step S3: Divide the fused sensor data stream into fixed-length time window segments, with each time window segment corresponding to a motion state sample to be identified;

[0040] Step S4: Construct a hybrid deep neural network model consisting of a cascaded one-dimensional convolutional neural network and a bidirectional long short-term memory network (BiLSTM). The one-dimensional convolutional layer is used to extract local temporal features, and the BiLSTM layer is used to model long-range dependencies.

[0041] Step S5: Use the historical unmanned surface vessel operation dataset labeled with motion mode tags to train the hybrid deep neural network model end-to-end. The motion mode tags include, but are not limited to, five categories: straight cruise, stationary turn, serpentine avoidance, emergency braking, and docking.

[0042] Step S6: Input the real-time collected and preprocessed fused sensor data stream into the trained hybrid deep neural network model, and output the motion mode category and confidence score of the current unmanned surface vessel.

[0043] In practical applications of this invention, during system operation, a multi-source sensor system deployed on the unmanned surface vessel platform first synchronously acquires raw observation data of the environment and its own state. The system includes four key types of sensors: an inertial measurement unit (IMU), a global navigation satellite system (GNSS) receiver, a lidar, and a visible light camera. The IMU outputs triaxial acceleration and angular velocity at high frequencies (typically ≥100Hz) to capture the instantaneous dynamic response of the vessel; the GNSS provides global position and velocity information, forming a macroscopic reference for the motion trajectory; the lidar scans the distribution of surrounding obstacles and the shoreline contours, forming local environmental geometric constraints; and the visible light camera records the visual scene, assisting in determining berthing and obstacle avoidance semantic behaviors.

[0044] To this end, the system performs strict time synchronization and coordinate system alignment operations before the data enters the model. Time synchronization adopts hardware-level triggering or high-precision software interpolation strategies to ensure that the timestamp deviation of the data from each sensor is controlled within 10 milliseconds, thereby ensuring that the representation of the same physical event in different sensors is within the same time window. Spatial alignment is achieved by converting the attitude information of the IMU to the Northeast-Eastern Sky (ENU) geographic coordinate system with the GNSS positioning point as the origin through a pre-calibrated extrinsic parameter matrix, and by using the rigid body transformation parameters between the lidar and the camera to project the point cloud and image pixels onto a unified world coordinate framework. After completing the spatiotemporal alignment, the fused sensor data stream is divided into fixed-length time window segments, such as each segment lasting 10 seconds, with a sliding truncation step of 1 second. Each window retains the original high sampling rate (e.g., 50-100Hz) continuous sequence, fully preserving key dynamic details such as acceleration abrupt changes, continuous rotation of heading angle, and gradual changes in obstacle distance. The segmentation method satisfies the requirement of the deep learning model for fixed input size, while avoiding the loss of long-term behavioral context due to excessively short windows or the introduction of irrelevant state interference due to excessively long windows.

[0045] Subsequently, these time window samples are fed into a specially constructed hybrid deep neural network model. The front end of the model employs a 1D-CNN, whose convolutional kernels slide along the time axis, extracting local temporal patterns layer by layer. Examples include periodic fluctuations in IMU signals, step changes in GNSS velocity curves, and monotonically decreasing trends in lidar ranging sequences. After multiple layers of convolution and pooling, these local features form a high-dimensional, sparse, but semantically rich intermediate representation. This representation is then fed into a Bidirectional Long Short-Term Memory (BiLSTM) network. BiLSTM models temporal dependencies simultaneously in both forward and backward directions, enabling it to capture "deceleration followed by turning." " The "continuous left and right swing followed by stopping" is a complex motion pattern with a clear timing logic. Because BiLSTM has a memory gating mechanism, it can effectively suppress high-frequency jitter noise caused by water surface waves, focusing on low-frequency behavioral signals that truly reflect the control intention.

[0046] Furthermore, a large amount of historical data collected and manually labeled in real aquatic environments was used, with labels covering five typical motion modes: straight-line cruising, stationary turning, serpentine avoidance, emergency braking, and docking. During training, the loss function drove the network to automatically adjust the convolutional kernel weights and LSTM gating parameters, causing similar motions to cluster in the feature space and dissimilar motions to separate from each other.

[0047] In one specific implementation, in step S2, the time synchronization processing adopts a hardware trigger signal or software interpolation alignment method to ensure that the time deviation of each sensor data does not exceed 10 milliseconds; the coordinate system alignment processing includes converting the attitude angle output by the IMU to the Northeast ENU coordinate system with the GNSS positioning point as the origin, and projecting the lidar point cloud and the visible light image to the same world coordinate system through calibration parameters to ensure that the spatial consistency error is less than 0.1 meters.

[0048] In step S3, the length of the time window segment is set to be between 5 and 15 seconds, the sliding step is between 1 and 3 seconds, and a continuous sensing data sequence with an original sampling frequency of not less than 50 Hz is retained in each time window to maintain the integrity of motion dynamic details.

[0049] In practical applications of this invention, after system startup, multiple sensors, including an inertial measurement unit (IMU), a global navigation satellite system (GNSS) receiver, a lidar, and a visible light camera, continuously output raw observation data at their respective inherent sampling frequencies. Because these sensors differ in hardware architecture, communication protocols, and internal clock sources, the arrival time of their data at the main control unit naturally exhibits a shift in timing, ranging from microseconds to milliseconds. Without correction, this temporal misalignment will cause the representation of the same physical event (such as a turning maneuver) in different sensors to be recorded at different time points. For example, the IMU might detect a sudden increase in angular velocity at time t, while the GNSS might only reflect a change in heading at t+30ms, and the lidar might capture the ship's azimuth shift relative to the shoreline at t+50ms. If these misaligned data are directly concatenated and input into the model, the network will be forced to learn "false" temporal relationships, weakening its ability to model the true motion logic.

[0050] Specifically, the system introduces a high-precision time synchronization mechanism at the data acquisition layer. When hardware conditions permit, a unified external trigger signal (such as a PPS pulse or GPIO synchronization line) is used to simultaneously drive all sensors to begin sampling, fundamentally eliminating clock drift. On platforms that only support software synchronization, high-order interpolation algorithms (such as cubic spline interpolation or state estimation based on Kalman filtering) are used to resample the data from each sensor, mapping them uniformly onto a common time axis. This common time axis is based on the high-precision UTC time provided by GNSS, and all other sensor data are aligned accordingly. After this processing, the maximum time deviation between any two sensors is controlled within 10 milliseconds.

[0051] Meanwhile, the attitude angles (roll, pitch, yaw) output by the IMU are based on its own installed coordinate system by default, while GNSS provides latitude, longitude, and altitude in a geocentric coordinate system. If the raw attitude angles are used directly to determine "whether turning," directional errors will occur due to the inconsistency in coordinate systems. Therefore, the system first establishes a local northeast-sky (ENU) geographic coordinate system using the GNSS positioning results, with the current GNSS position as the origin, east as the x-axis, north as the y-axis, and sky as the z-axis. Subsequently, through pre-calibrated installation extrinsic parameters (rotation matrix and translation vector) between the IMU and the hull, the attitude measured by the IMU is converted to the ENU system in real time, so that the yaw angle directly corresponds to the geographic heading angle, thereby accurately reflecting the hull's orientation change relative to the real world.

[0052] For LiDAR and visible light cameras, LiDAR outputs a 3D point cloud centered on itself, while the camera outputs a 2D pixel image; their viewpoints, scales, and dimensions differ. The system obtains the rigid body transformation matrix (including rotation R and translation t) between the LiDAR and camera through offline calibration, and during runtime, it transforms the point P in each frame of the LiDAR point cloud... _ Lidar transformation to camera coordinate system: P _ cam = R × P _ lidar+t. Furthermore, by combining the camera's intrinsic parameter matrix, 3D points can be projected onto the image plane, achieving pixel-level alignment between the point cloud and the image. Simultaneously, the transformation chain ultimately maps all perceived data to the aforementioned ENU world coordinate system. For example, when the unmanned surface vessel approaches a dock, the lidar detects a vertical obstacle 2 meters ahead, the camera identifies the berth marker, and after coordinate system alignment, the system can confirm that "the obstacle is located 2 meters directly in front of the vessel, at ENU coordinates (0, 2, 0)," thus accurately linking the series of actions of "deceleration-alignment-berthing."

[0053] The fused data stream after spatiotemporal alignment is still a continuous time series and needs further structuring to adapt to the input requirements of deep learning models. The system uses a sliding window strategy to segment it: the window length is set between 5 and 15 seconds. A window shorter than 5 seconds is insufficient to fully encompass a typical maneuver (e.g., serpentine evasion typically requires more than 8 seconds), while a window longer than 15 seconds is prone to including multiple irrelevant motion phases, reducing sample purity. The sliding step size is set between 1 and 3 seconds to ensure timely recognition while avoiding redundant computation. Within a 10-second window, the IMU can provide over 500 acceleration / angular velocity samples, GNSS provides hundreds of position / velocity points, and LiDAR and images are also synchronously embedded according to their respective frequencies.

[0054] In one specific implementation, the one-dimensional convolutional neural network part of the hybrid deep neural network model includes at least three convolutional modules. Each convolutional module includes a one-dimensional convolutional layer, a batch normalization layer, a ReLU activation function, and a max pooling layer, wherein the convolutional kernel sizes are 64, 32, and 16, and the number of channels are 64, 128, and 256, respectively. The BiLSTM layer includes two stacked BiLSTM units, each unit having a hidden state dimension of 256, and a Dropout layer is connected after the BiLSTM output, with a dropout rate of 0.5.

[0055] In specific applications, after the multi-source sensor data (such as IMU acceleration / angular velocity, GNSS position / velocity, LiDAR distance sequence, and image feature vector) processed by spatiotemporal alignment and high-fidelity windowing are concatenated into a unified time series input, they are first fed into a 3D convolutional front-end composed of three convolutional modules. The first layer uses 64 convolutional kernels of length 64 to slide and scan the entire input window (e.g., 500 time steps) in the time dimension, capturing local dynamic patterns within a span of about 1 second, such as a short acceleration pulse, a continuous angular velocity shift, or a rapid decreasing trend in obstacle distance. Since the number of input channels is usually the dimension after multi-sensor fusion (e.g., 12-20 dimensions), the layer outputs a 64-channel high-dimensional temporal feature map, whose temporal resolution is still close to the original sampling rate.

[0056] Specifically, the batch normalization layer standardizes the feature distribution of each channel, eliminating internal covariate shifts caused by differences in sensor dimensions or environmental disturbances (such as IMU baseline drift caused by waves), thereby accelerating training convergence and improving model stability. The ReLU activation function introduces nonlinear expressive power here, setting negative responses to zero, preserving physically meaningful positive dynamic features (such as positive acceleration and right-turn angular velocity), while suppressing meaningless noise responses. The subsequent max-pooling layer downsamples the time axis with a stride of 2, compressing the number of time steps by half, which reduces the computational burden and enhances robustness to small time shifts.

[0057] The second convolutional module inherits the output of the first layer, using 128 convolutional kernels of length 32 to detect composite patterns at a mid-temporal scale (approximately 0.5 seconds) in a more abstract feature space. For example, it identifies the combined feature of a two-stage behavior of "decelerating first and then turning left," or the primary representation of serpentine avoidance corresponding to "periodic left and right swaying." The third layer further uses 256 convolutional kernels of length 16 to focus on more refined but semantically stronger local structures, such as the final stage of parking where the distance to the obstacle decays exponentially, or the sharp negative peak of acceleration accompanied by a sudden drop in speed during emergency braking. With each layer, the temporal dimension continues to be compressed by pooling, while the channel dimension doubles, allowing the network to gradually reduce the temporal granularity while improving the richness and discriminativeness of feature representation.

[0058] After processing by three convolutional layers, the original input of hundreds of time steps has been compressed into dozens of high-dimensional feature vector sequences, each of which fuses joint dynamic information from multiple sensors within a local time window. The sequences are then fed into the BiLSTM part. The BiLSTM consists of two stacked bidirectional LSTM units, each with a hidden state dimension of 256. The forward LSTM traverses from the starting point to the ending point, capturing the causal relationship of "how the past influences the present"; the backward LSTM traverses in reverse, capturing the inverse context of "how future intentions constrain current behavior". For example, when recognizing the "berthing" pattern, the first half appears as normal deceleration, but the BiLSTM can only fully understand the berthing intention by combining the latter half's behavior of continuously approaching the obstacle and approaching zero speed. Each time step hidden state (dimension 512, due to bidirectional concatenation) output by the BiLSTM contains full-window context information, but directly using it for classification can easily lead to overfitting, especially with limited data on unmanned surface vessels. Therefore, a Dropout layer is added after the BiLSTM to randomly mask half of the neuron outputs at a dropout rate of 0.5. Finally, the regularized context vector is fed into the fully connected classifier head, which outputs the probability distribution of the five motion modes.

[0059] In one specific implementation, the process of constructing the historical unmanned surface vessel (USV) operation dataset in step S5 includes: operating the USV to perform standardized motion tasks in various water environments and simultaneously recording the corresponding multi-source sensor data and manually labeled motion patterns; at the same time, data augmentation strategies are introduced to apply Gaussian noise, time axis stretching or compression, and sensor missing simulation perturbation to the original data to improve the robustness of the model.

[0060] After step S6, an anomaly detection mechanism is also included: when the highest confidence score of the model output is lower than the preset threshold of 0.7, or when the prediction results of three consecutive time windows switch frequently and there is no dominant category, an anomaly status flag is triggered, and a backup rule engine is started to assist in the judgment. The rule engine sets threshold logic based on the rate of change of velocity, the rate of change of heading angle, and the physical quantity of obstacle distance.

[0061] In specific applications of this invention, before model training, the system first constructs a high-quality, high-coverage historical operational dataset through systematic real-ship experiments. In specific operations, the unmanned surface vessel (USV) is deployed in various typical water scenarios, including open lakes, narrow inland waterways, densely packed port berths, and near-shore areas with strong currents or waves. In each environment, operators strictly follow preset procedures to execute a series of standardized motion tasks: for example, navigating in a straight line for 100 meters at a fixed speed, completing a 360-degree turn within a 20-meter diameter, performing serpentine avoidance along an S-shaped trajectory, simulating sudden obstacles to trigger emergency braking, and automatically berthing at a dock. While performing each task, the onboard multi-source sensor system simultaneously records the complete raw data stream, including outputs from IMU, GNSS, lidar, and visible light cameras. Professional personnel manually label each time window sample according to task instructions and video playback, clarifying its corresponding motion mode category (e.g., "straight-line cruising" or "turning in place").

[0062] To further enhance the model's adaptability to real-world disturbances, multi-dimensional data augmentation was implemented based on the original acquired data. On the one hand, Gaussian white noise, consistent with actual noise characteristics, was injected into the sensor signals to simulate IMU zero-bias drift, GNSS multipath effects, or low-light noise in image sensors. On the other hand, the time axis was nonlinearly stretched or compressed (e.g., randomly expanding a 10-second window to 11 seconds or compressing it to 9 seconds) to simulate the timescale differences of the same motion pattern under different boat types, loads, or sea states. In addition, scenarios of temporary sensor failure were actively simulated, such as randomly blocking lidar data or camera frames within a certain time period, forcing the model to learn to rely on other modalities to make judgments even under conditions of missing information.

[0063] Once the trained model is deployed in real-world operation, the system not only outputs the motion mode prediction result for the current time window but also simultaneously calculates the prediction confidence score (i.e., the maximum probability value output by softmax). If the score is lower than a preset threshold of 0.7, it indicates that the model lacks confidence in discriminating the current input—reasons include: the input data contains new behaviors not seen in the training set (such as following maneuvers in formation driving), severe sensor interference (such as position jumps caused by GNSS signal loss), or the current state is in a transitional phase between two modes (such as transitioning from cruising to turning). Simultaneously, the system continuously monitors the prediction sequence across multiple consecutive time windows. If, within the last three windows, the prediction category frequently switches (e.g., "cruising → turning → cruising → braking") and no single category dominates (i.e., occurrences ≥ 2), it is determined to be unstable or the model is oscillating. Meeting any of the above conditions triggers an abnormal state flag.

[0064] Once an anomaly flag is activated, the system immediately activates a backup rule engine as an auxiliary judgment mechanism. This rule engine does not rely on deep learning models but instead constructs hard logical criteria based on physically interpretable kinematics and environmental perception quantities. For example, it calculates whether the current rate of change of velocity (absolute value of acceleration) exceeds 0.5 m / s². 2 To determine whether emergency braking is underway; to detect whether the rate of change of heading angle is consistently higher than 10° / s and the displacement increment is less than 1 meter, in order to identify turning in place; or to combine the distance to the nearest obstacle measured by lidar and its changing trend to determine whether it is in the berthing approach phase.

[0065] In one specific implementation, the visible light camera automatically switches to infrared imaging mode at night or in low-light conditions. The visible light image and the infrared image are then pixel-level aligned using an image fusion algorithm before being input into the model to ensure all-weather recognition capability. The image fusion algorithm employs a gradient domain-based multi-scale weighted averaging method to preserve edge details while suppressing noise. The hybrid deep neural network model is deployed on a local embedded computing platform of the unmanned surface vessel. The platform uses an NVIDIA Jetson AGX Orin or an AI acceleration chip with equivalent computing power. TensorRT is used to quantize and optimize the model, keeping the single inference latency within 50 milliseconds to meet real-time requirements.

[0066] In specific applications of this invention, the visible light camera on the system does not operate independently, but is integrated with the infrared thermal imaging sensor in the same optical payload module and scheduled by a unified environmental perception controller. When the ambient light intensity is lower than a preset threshold (e.g., 10 lux, automatically determined by the built-in photosensor or image brightness histogram), the controller immediately triggers an imaging mode switch: shutting down the color image acquisition of the visible light camera, activating the infrared sensor, and starting to output a thermal radiation intensity map. This switching process is completed within milliseconds, avoiding perception gaps due to imaging interruptions. The fusion process first relies on pre-completed multimodal calibration: by acquiring synchronous visible light-infrared image pairs of a checkerboard pattern or a specific heat source target in a laboratory environment, the spatial extrinsic parameters (rotation matrix and translation vector) and pixel mapping relationship between the two sensors are calculated to establish an accurate geometric correspondence model. During operation, each frame of synchronously acquired visible light and infrared images is first aligned pixel-level by the mapping model to eliminate misalignment caused by differences in field of view or installation offset. Subsequently, a gradient domain-based multi-scale weighted average fusion algorithm is used to process the aligned image pairs. The algorithm calculates the texture gradient of the visible light image and the thermal contrast gradient of the infrared image at multiple scales of the Laplacian pyramid or wavelet decomposition, and dynamically assigns weights based on the local gradient magnitude. In areas with rich edges (such as dock outlines and obstacle boundaries), the visible light is given higher weights to preserve structural details, while in areas with low texture but significant thermal features (such as the temperature difference between the water surface and the ship hull at night), the infrared weights are increased to enhance target visibility.

[0067] The fused image is then encoded into a fixed-size tensor and stitched together with other sensor data (such as IMU, GNSS, and LiDAR) as input to a hybrid deep neural network. To ensure the entire recognition process meets the real-time response requirements of high-speed maneuvering of unmanned surface vessels, the entire model is deployed on an onboard embedded AI computing platform, typically with NVIDIA Jetson AGX Orin or a domestically produced AI acceleration chip with the same INT8 computing power (≥200 TOPS). Before deployment, the original training model undergoes full-process optimization using the TensorRT inference engine: first, quantization from FP32 to INT8 is performed, and the distribution of activation values ​​in each layer is statistically analyzed using a calibration dataset to generate optimal quantization parameters, significantly reducing computation and memory usage with almost no loss of accuracy; second, graph optimization is performed, including layer fusion (such as merging convolution, batch normalization, and ReLU into a single CUDNN layer), redundant operation elimination, and memory reuse planning; finally, operator scheduling is performed for the GPU architecture and DLA (deep learning accelerator) of the Orin platform, offloading computationally intensive convolution operations to dedicated hardware units.

[0068] After the above optimizations, the single forward inference latency of the model on the embedded platform is strictly controlled within 50 milliseconds. This means that the system can process at least 20 time window samples per second, perfectly matching the sampling rhythm of 1-3 second sliding step and 5-15 second window length set in step S3. The latency includes the complete link time from image acquisition, multimodal fusion, data stitching to neural network inference, ensuring that the output motion pattern category can be fed back to the upper navigation or decision module in a timely manner for path replanning, behavior warning, or human-machine collaborative intervention.

[0069] In one specific implementation, an online model update mechanism is also included: when the unmanned surface vessel returns to port, data samples marked as "uncertain" or "abnormal" during its operation are uploaded to a shore-based server. The server manually reviews and corrects the labels of the incremental data, and then fine-tunes the original model using a transfer learning strategy. The updated model version is then pushed to the unmanned surface vessel via a secure communication protocol, enabling continuous evolution of recognition capabilities. The motion mode category expansion supports the addition of custom modes. Training of new modes only requires providing no less than 200 time window samples, and freezing the bottom convolutional and BiLSTM parameters of the hybrid deep neural network model. Only the final fully connected classification layer and a few adaptation layers are fine-tuned, thereby maintaining the original recognition performance while quickly adapting to new task scenarios, such as formation cooperative navigation and autonomous search and rescue trajectory tracking.

[0070] In specific applications, during the mission of the unmanned surface vessel (USV), the system not only performs routine motion pattern recognition but also synchronously records all time window samples judged as "uncertain" or "abnormal." These samples include two categories: one is low-confidence predictions with a model output confidence level below 0.7; the other is unstable state segments that trigger anomaly detection mechanisms (such as frequent category switching or rule engine intervention). After the USV completes its mission and returns to port, this batch of incremental data is automatically uploaded to the shore-based data center via a wired or secure wireless link (such as the MQTT protocol based on TLS 1.3 encryption).

[0071] On the shore-based server, the received incremental data first enters a manual review process. Operators with ship handling experience, combining task logs, video playback, and environmental records, re-label each sample's true motion pattern. Samples confirmed as belonging to new categories are added to the pool of patterns to be expanded; those belonging to the original categories but incorrectly labeled are used to correct training set bias. After labeling, the system initiates a transfer learning fine-tuning process: if only existing categories are being corrected, a full-model fine-tuning strategy is used, continuing training with a smaller learning rate while retaining the original weights; if new motion patterns are involved (such as "formational navigation"), an efficient incremental learning mechanism is activated. The bottom-level one-dimensional convolutional modules and BiLSTM layers of the hybrid deep neural network are completely frozen because they have learned general temporal dynamic feature representation capabilities (such as acceleration abrupt changes, continuous changes in heading, and obstacle approach trends) through a large amount of historical data. The bottom-level features have high transferability to both new and old tasks. Only the final fully connected classification layer is unfrozen, and a lightweight adaptation layer (such as a 128-dimensional fully connected layer + ReLU) is inserted between the BiLSTM output and the classification head to map the general features to an expanded label space containing the new category. Since the new mode only requires no less than 200 high-quality time window samples (about 30-60 minutes of real ship data) and the training only involves a small number of parameters, the entire fine-tuning process can be completed in a few minutes on a regular GPU server, reducing the cost of data acquisition and training.

[0072] After fine-tuning, the system automatically verifies the new model: testing the recognition accuracy of the original five classes on the retained validation set to see if there is a significant decrease (setting a tolerance threshold such as ≤1% performance degradation), while simultaneously evaluating the recall and precision of the newly added classes. Once verification is successful, the new model is assigned a unique version number and pushed to the corresponding unmanned surface vessel (USV) via a secure communication protocol (supporting digital signatures and integrity verification). Upon receiving the update package, the USV first loads and runs a self-check program in an isolated environment to confirm normal inference and acceptable latency before atomically switching to the new model, ensuring uninterrupted service.

[0073] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0074] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the invention, such designs should fall within the protection scope of the present invention.

Claims

1. A method for intelligent recognition of motion patterns of a water unmanned surface vehicle based on deep learning, characterized in that, The method comprises the following steps: Step S1: Real-time acquisition of original sensor data by a multi-source sensor system installed on the water surface unmanned vehicle, the multi-source sensor system comprising at least an inertial measurement unit (IMU), a global navigation satellite system (GNSS) receiver, a laser radar and a visible light camera; Step S2: Time synchronization and coordinate system alignment processing of the original sensor data to generate a fused sensor data stream with a unified timestamp and spatial reference system; Step S3: Division of the fused sensor data stream into fixed-length time window segments, each time window segment corresponding to a motion state sample to be identified; Step S4: Construction of a hybrid deep neural network model composed of a one-dimensional convolutional neural network and a bidirectional long short-term memory (BiLSTM) network, wherein the one-dimensional convolutional layer is used to extract local time sequence features, and the BiLSTM layer is used to model long-range dependencies; Step S5: End-to-end training of the hybrid deep neural network model using a historical water surface unmanned vehicle operation data set labeled with motion mode labels, the motion mode labels including but not limited to straight cruising, in-place turning, snake-like evasion, emergency braking and parking; Step S6: Input of the real-time acquired and preprocessed fused sensor data stream into the trained hybrid deep neural network model to output the motion mode category to which the current water surface unmanned vehicle belongs and the confidence score thereof.

2. The deep learning-based intelligent identification method for the motion mode of the water surface unmanned ship according to claim 1, characterized in that, In step S2, the time synchronization processing adopts a hardware trigger signal or a software interpolation alignment method, so that the time deviation of each sensor data is not more than 10 milliseconds; the coordinate system alignment processing comprises conversion of the attitude angle output by the IMU to an east-north-up (ENU) coordinate system with a GNSS positioning point as the origin, and projection of the laser radar point cloud and the visible light image to the same world coordinate system through calibration parameters to ensure that the spatial consistency error is less than 0.1 meters.

3. The method of claim 1, wherein the method further comprises: In step S3, the length of the time window segment is set to be between 5 seconds and 15 seconds, the sliding step is 1 second to 3 seconds, and the continuous sensor data sequence with a sampling frequency not lower than 50 Hz is retained in each time window to maintain the integrity of the motion dynamic details.

4. The method of claim 1, wherein the method further comprises: The one-dimensional convolutional neural network part of the hybrid deep neural network model comprises at least three convolutional modules, each convolutional module comprising a one-dimensional convolutional layer, a batch normalization layer, a ReLU activation function and a max pooling layer in sequence, wherein the convolution kernel size is 64, 32 and 16 in sequence, and the channel number is 64, 128 and 256 in sequence; the BiLSTM layer comprises two stacked BiLSTM units, the hidden state dimension of each unit is 256, and a Dropout layer is connected after the BiLSTM output, with a dropout rate of 0.

5.

5. The method of claim 1, wherein the method further comprises: In step S5, the construction process of the historical water surface unmanned vehicle operation data set comprises: controlling the water surface unmanned vehicle to perform standardized motion tasks in various water environments, and synchronously recording the corresponding multi-source sensor data and the manually labeled motion mode labels; meanwhile, a data enhancement strategy is introduced to apply Gaussian noise, time axis stretching or compression and sensor missing simulation disturbance to the original data to improve the model robustness.

6. The method of claim 1, wherein the method further comprises: After step S6, an abnormality detection mechanism is further included: when the highest confidence score of the model output is lower than the preset threshold 0.7, or the prediction results of three consecutive time windows frequently switch and there is no dominant class, an abnormal state flag is triggered, and a backup rule engine is started for auxiliary judgment, the rule engine sets threshold logic based on the speed change rate, the heading angle change rate and the obstacle distance physical quantity.

7. The method of claim 1, wherein the method further comprises: The visible light camera automatically switches to an infrared imaging mode at night or in low light conditions, and inputs the visible light image and the infrared image into the model after pixel-level alignment through an image fusion algorithm to ensure all-weather recognition capability; the image fusion algorithm adopts a multi-scale weighted average method based on a gradient domain to retain edge details while suppressing noise. 8.The method of claim 1, wherein, The hybrid deep neural network model is deployed on a water surface unmanned ship local embedded computing platform, the platform adopts an NVIDIA Jetson AGX Orin or an AI acceleration chip with the same computing power, and the model is quantized and graph optimized through TensorRT, so that the single inference delay is controlled within 50 milliseconds, meeting the real-time requirement.

9. The method of claim 1, wherein the method further comprises: An online model updating mechanism is further included: when the water surface unmanned ship returns to the port, the data samples marked as "uncertain" or "abnormal" during the operation of the water surface unmanned ship are uploaded to a shore-based server, the server side manually reviews and corrects the labels of the incremental data, then fine-tunes the original model by using a transfer learning strategy, and pushes the updated model version to the unmanned ship end through a secure communication protocol, so as to realize continuous evolution of the recognition capability.

10. The method of claim 1, wherein the method further comprises: The motion mode category expansion supports custom addition of modes, the training of the added mode only needs to provide not less than 200 time window samples, and the bottom convolution and BiLSTM parameters of the hybrid deep neural network model are frozen, only the last fully connected classification layer and a small number of adaptive layers are fine-tuned, so that the original recognition performance is maintained while quickly adapting to new task scenarios such as formation cooperative navigation and autonomous search and rescue trajectory tracking.

Images