Ship maneuvering motion trajectory prediction method and system fusing shipboard radar wave measurement

Abstract

The present application relates to the technical field of ship motion, in particular to a ship maneuvering motion trajectory prediction method and system fusing shipborne radar wave measurement, the method comprising: acquiring multi-modal data, and preprocessing the acquired multi-modal data; performing multi-modal feature extraction on the preprocessed multi-modal data; performing cross-modal fusion based on the extracted multi-modal features to obtain a fusion feature vector; constructing a feedforward neural network with the fusion feature vector as input, and outputting a prediction value of a ship motion state; performing parameter optimization based on the constructed feedforward neural network, verifying the prediction result output by the feedforward neural network, and outputting a final prediction result; and the final output of the present application, the prediction value of the ship motion state, covers key parameters such as longitudinal velocity, transverse velocity, and yaw angle velocity, providing an accurate basis for ship maneuvering decision-making, and helping to improve the safety and precision of ship navigation.

Description

Technical Field

[0001] This invention relates to the field of ship motion technology, and in particular to a method and system for predicting ship maneuvering trajectories by integrating shipborne radar wave measurement. Background Technology

[0002] In the complex and ever-changing marine environment, a ship's maneuvering and collision avoidance capabilities are crucial for navigation safety. In complex sea states and busy waters, ships often face high-risk maneuvering tasks such as collision avoidance and anti-collision. However, due to factors such as wind and wave disturbances, traditional maneuvering methods rely on experience-based judgment, which involves considerable uncertainty. Therefore, accurately predicting the ship's trajectory and navigation status in advance based on environmental information about external waves measured by shipborne radar, combined with control signals from the propeller and rudder, is of great significance for improving the timeliness and accuracy of maneuvering decisions.

[0003] In recent years, artificial intelligence technology has been applied in the field of ship maneuvering motion prediction. Models such as support vector machines, recurrent neural networks, and long short-term memory neural networks have been used to extrapolate and predict ship maneuvers using historical motion data. However, most models rely solely on the ship's own historical motion data, such as speed and rudder angle, without considering the impact of external environmental factors such as waves on ship motion. This leads to a significant decrease in prediction accuracy under complex sea conditions and poor environmental adaptability. In addition, when using models such as one-way LSTM, only past time information can be utilized, resulting in insufficient modeling ability for long-distance time series dependencies and significant error accumulation in long-term prediction scenarios. Furthermore, the models exhibit poor robustness when facing non-stationary conditions such as abnormal waves or sudden maneuvers. Finally, for multi-source data such as environmental parameters and ship motion data, there is a lack of efficient cross-modal feature interaction mechanisms, which fails to fully explore the potential correlations between different data modes, limiting the improvement of prediction accuracy. At present, there is a need for ship maneuvering motion trajectory prediction methods and systems that integrate shipborne radar wave measurement. Summary of the Invention

[0004] To address the problems of poor environmental adaptability and low prediction accuracy in traditional ship maneuvering motion prediction, this invention provides a method and system for predicting ship maneuvering motion trajectories by integrating shipborne radar wave measurement.

[0005] Firstly, the ship maneuver trajectory prediction method based on shipborne radar wave measurement provided by the present invention adopts the following technical solution:

[0006] A method for predicting ship maneuvering trajectories by integrating shipborne radar wave measurement includes:

[0007] Acquire multimodal data including radar wave images, ship motion history data, and ship maneuvering signal data, and preprocess the acquired multimodal data;

[0008] Multimodal feature extraction is performed on the preprocessed multimodal data, including the extraction of environmental feature vectors, motion temporal feature vectors, and manipulation feature vectors;

[0009] Cross-modal fusion is performed based on the extracted multimodal features to obtain a fused feature vector. This includes introducing an attention mechanism, which inputs the environmental feature vector and the motion temporal feature vector into a multi-head attention layer for fusion to obtain an attention fused feature vector.

[0010] The fused feature vectors are used as input to construct a feedforward neural network, which outputs a predicted value of the ship's motion state.

[0011] The parameters are optimized based on the constructed feedforward neural network, including using gradient-based optimization methods to update the parameters and minimize the loss function;

[0012] The prediction results output by the feedforward neural network are verified, and the final prediction result is output.

[0013] Furthermore, the preprocessing of the acquired multimodal data includes blind zone masking of the acquired radar wave measurement image, calculating the polar radius from the pixel to the image center point with the image center point as the origin, constructing a mask matrix based on the preset blind zone radius and the maximum effective radius, generating a radar image after blind zone masking based on the mask matrix, cropping the radar image after blind zone masking, filtering the cropped image using a two-dimensional Gaussian filter, normalizing the effective area of ​​the mask, generating a preprocessed shipborne radar wave measurement image, and finally using linear interpolation to time-align the ship motion history data and ship maneuvering signal data, and filtering and standardizing the aligned data. The normalization formula is as follows:

[0014]

[0015] Among them, t k Let (x, y) be the sampling time, and (x, y) be the pixel coordinates of the radar image. filt (x,y,t k ) represents the pixel value of the radar image after two-dimensional Gaussian filtering, and σ represents the pixel value of the image. I μ is the standard deviation of all pixel values ​​in the radar image within the effective area of ​​the mask. I This is the average value of all pixel values ​​in the radar image within the effective area of ​​the mask.

[0016] Furthermore, the multimodal feature extraction of the preprocessed multimodal data includes inputting the preprocessed radar wave measurement image into a convolutional neural network. This convolutional neural network consists of multiple convolutional blocks, dynamic pooling layers, and batch normalization layers. First, single-frame image features are extracted through convolution operations. Then, a dynamic scaling factor is added to the batch normalization layer. The improved batch normalization layer processes the convolutional output features, concatenating the multi-frame features into a four-dimensional feature tensor along the time dimension. The spatial dimension is then compressed through a dynamic pooling layer, and finally, dimensionality is reduced through a fully connected layer with residual connections to obtain the environmental feature vector. The expression for the batch normalization layer is as follows:

[0017]

[0018] Where, μ B,i and These represent the mean and variance of all eigenvalues ​​within the current batch in the i-th layer of batch normalization, respectively, and α. i β is the dynamic scaling factor. i Z is the bias parameter, ε is a very small constant, and z i These are the original feature values, i.e., the original feature values ​​after the i-th layer convolution operation without any normalization processing. These are the eigenvalues ​​after normalization.

[0019] Furthermore, the multimodal feature extraction of the preprocessed multimodal data also includes inputting the standardized ship motion time-history data sequence into a bidirectional long short-term memory (LSTM) network. A dynamic gating mechanism and a temporal attention fusion module are introduced into the LTM network. The output sequences of the forward LTM network and the reverse LTM network are fused using temporal attention weights to obtain a bidirectional feature sequence. Temporal importance pooling is used to aggregate the bidirectional feature sequence, with pooling weights calculated based on the second derivative of the feature sequence. Finally, the motion temporal feature vector is obtained by dimensionality reduction through a fully connected layer with residual connections. The expression for the bidirectional feature sequence is:

[0020]

[0021] Where, α t Let be the temporal attention coefficient at time t. The output features of the positive long short-term memory network at time t are... For the output features of the inverse long short-term memory network at time t, w a This is a learnable attention weight vector.

[0022] Furthermore, the multimodal feature extraction of the preprocessed multimodal data also includes inputting the standardized ship maneuvering signal data sequence into an artificial neural network. The ship maneuvering signal data sequence contains multiple time steps, each time step containing the left and right rudder angles and the left and right propeller speeds. The artificial neural network is a multi-layer fully connected network, with each layer of neurons employing an activation function. After being flattened, the input ship maneuvering signal data is sequentially passed through each layer of the fully connected network. First, feature mapping is performed through the first layer of the network, and then feature compression is performed through the second layer of the network. The output features are then reduced in dimensionality by linear projection to obtain the maneuvering feature vector.

[0023] Furthermore, the cross-modal fusion based on extracted multimodal features includes introducing an attention mechanism. Environmental feature vectors and motion temporal feature vectors are input into a multi-head attention layer for fusion. The environmental feature vector is used as the query vector, and the motion temporal feature vector is used as the key vector and value vector, respectively. A trainable weight matrix maps the query vector, key vector, and value vector. The multi-head attention layer calculates the attention output through multiple parallel attention heads. The output features of each attention head are concatenated and then linearly transformed to obtain the attention fusion feature vector. The linear transformation expression is:

[0024] MultiHead(Q,K,V)=[head1;…;head h W O ,

[0025] Wherein, head1; ...; head h Let W represent the output features of the first to the h-th attention heads respectively, and [.] represent the high-dimensional feature matrix formed by concatenating the output features of the h attention heads according to their dimensions. O This represents the trainable output weight matrix.

[0026] Furthermore, the cross-modal fusion based on the extracted multimodal features also includes concatenating the attention fusion feature vector, motion temporal feature vector, and manipulation feature vector by dimension to obtain intermediate fusion features, and then inputting the intermediate fusion features into a fully connected network containing batch normalization and Dropout layers for nonlinear transformation to output a fusion feature vector.

[0027] Furthermore, the step of constructing a feedforward neural network by using the fused feature vector as input includes using the fused feature vector as input to construct a feedforward neural network consisting of several fully connected layers and activation functions. Each fully connected layer uses an activation function to strengthen the nonlinear correlation between features, and a Dropout operation is directly set between the fully connected layers. The feedforward neural network outputs a predicted value of the ship maneuvering motion at a future set time interval through feature mapping and nonlinear transformation of multiple fully connected layers.

[0028] Furthermore, the parameter optimization based on the constructed feedforward neural network includes using the actual sequence of ship motion states as the supervision signal, constructing an MSE loss function to measure the difference between the predicted value output by the feedforward neural network and the actual sequence, updating the parameters using a gradient-based optimization method, calculating the gradient of the loss function with respect to all trainable parameters of the network through the backpropagation algorithm, and finally updating the parameters using the Adam optimizer. The gradient expression of the trainable parameters is as follows:

[0029]

[0030] Among them, L out To predict the window length, d s As a dimension of ship motion state, loss function Predicted value of the i-th motion state parameter at time τ The partial derivatives, The predicted value of the i-th motion state parameter at time τ. The partial derivative with respect to the trainable parameter θ reflects the degree to which changes in parameter θ affect the predicted value.

[0031] Secondly, a ship maneuvering trajectory prediction system integrating shipborne radar wave measurement includes:

[0032] The data acquisition module is configured to acquire multimodal data, including radar wave images, ship motion history data, and ship maneuvering signal data, and to preprocess the acquired multimodal data.

[0033] The feature extraction module is configured to perform multimodal feature extraction on the preprocessed multimodal data, including extracting environmental feature vectors, motion temporal feature vectors, and manipulation feature vectors.

[0034] The fusion module is configured to perform cross-modal fusion based on the extracted multimodal features to obtain a fused feature vector, including introducing an attention mechanism to input environmental feature vectors and motion temporal feature vectors into a multi-head attention layer for fusion to obtain an attention fusion feature vector;

[0035] The prediction module is configured to: construct a feedforward neural network by taking the fused feature vector as input and output a predicted value of the ship's motion state;

[0036] The optimization module is configured to: perform parameter optimization based on the constructed feedforward neural network, including updating parameters using gradient-based optimization methods to minimize the loss function;

[0037] The output module is configured to verify the prediction results of the feedforward neural network and output the final prediction results.

[0038] In summary, the present invention has the following beneficial technical effects:

[0039] 1. This invention utilizes shipborne radar wave measurement images to acquire environmental information. Compared with existing technologies that rely on buoys or external observation systems, it can perceive the transient wave distribution around the ship in real time, avoiding the problems of delayed or insufficient environmental data acquisition. Through blind zone shielding, cropping, filtering, and normalization preprocessing methods, it lays the foundation for accurate extraction of environmental features, improves the availability and reliability of environmental information, and enables the model to better adapt to complex and changing sea conditions.

[0040] 2. This invention employs convolutional neural networks, bidirectional long short-term memory networks, and artificial neural networks to extract environmental, motion timing, and manipulation feature vectors, respectively. This multimodal feature extraction method can comprehensively capture various types of information related to ship motion. It introduces an attention mechanism for cross-modal fusion, especially by using environmental feature vectors as query vectors and motion timing feature vectors as key and value vectors. This allows the model to dynamically focus on key features, enhances the interaction between different modal features, and effectively improves the model's adaptability and prediction accuracy under complex sea conditions and different manipulation conditions.

[0041] 3. The predicted ship motion state output by this invention covers key parameters such as longitudinal speed, lateral speed and yaw rate, and can predict the motion situation at a set time interval in advance. This provides an accurate basis for ship maneuvering decisions, helps to improve the safety of ship navigation and the precision of control, and has important application value in actual scenarios such as maritime emergency avoidance and ship assisted driving. It can effectively reduce accident risks and improve operational efficiency. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the overall process of the ship maneuvering trajectory prediction method based on the fusion of shipborne radar wave measurement according to an embodiment of the present invention.

[0043] Figure 2 This is a schematic diagram of the model structure in the ship maneuvering trajectory prediction method that integrates shipborne radar wave measurement according to an embodiment of the present invention.

[0044] Figure 3 This is a graph showing the predicted longitudinal velocity change in the ship maneuver trajectory prediction method based on the fusion of shipborne radar wave measurement according to an embodiment of the present invention.

[0045] Figure 4 This is a curve showing the predicted lateral velocity change in the ship maneuvering trajectory prediction method based on the fusion of shipborne radar wave measurement according to an embodiment of the present invention.

[0046] Figure 5 This is a curve showing the predicted change in bow angular velocity in the ship maneuvering trajectory prediction method that integrates shipborne radar wave measurement according to an embodiment of the present invention.

[0047] Figure 6 This is a diagram showing the effect of trajectory comparison in the ship maneuvering trajectory prediction method that integrates shipborne radar wave measurement according to an embodiment of the present invention. Detailed Implementation

[0048] The present invention will be further described in detail below with reference to the accompanying drawings.

[0049] Example 1

[0050] Reference Figure 1 The ship maneuver trajectory prediction method based on fusion of shipborne radar wave measurement in this embodiment includes:

[0051] Acquire multimodal data and preprocess the acquired multimodal data, including acquiring radar wave measurement images, ship motion history data and ship maneuvering signal data;

[0052] Multimodal feature extraction is performed on the preprocessed multimodal data, including the extraction of environmental feature vectors, motion temporal feature vectors, and manipulation feature vectors;

[0053] Cross-modal fusion is performed based on the extracted multimodal features to obtain a fused feature vector. This includes introducing an attention mechanism, which inputs the environmental feature vector and the motion temporal feature vector into a multi-head attention layer for fusion to obtain an attention fused feature vector.

[0054] The fused feature vectors are used as input to construct a feedforward neural network, which outputs a predicted value of the ship's motion state.

[0055] The parameters are optimized based on the constructed feedforward neural network, including using gradient-based optimization methods to update the parameters and minimize the loss function;

[0056] The prediction results output by the feedforward neural network are verified, and the final prediction result is output.

[0057] Specifically, the method for predicting ship maneuvering trajectories by integrating shipborne radar wave measurement includes the following steps:

[0058] S1. Acquire multimodal data and preprocess the acquired multimodal data, including acquiring radar wave measurement images, ship motion history data and ship maneuvering signal data;

[0059] like Figure 1 As shown, firstly, three types of raw data during the ship's navigation process are acquired simultaneously: wave measurement image sequences collected by the shipborne radar, motion history data recorded by the ship's motion sensors, including longitudinal speed, lateral speed, and yaw rate, and control signal data output by the ship's control system, including port and starboard rudder angles and port and starboard propeller speeds.

[0060] For the preprocessing of radar wave measurement images, the center point of each radar image frame is taken as the origin, and the distance from each pixel (x, y) in the image to the center point (c) is calculated. x ,c y The extreme radius of a pixel is the Euclidean distance between the pixel and the center point. The extreme radius from the pixel to the center is calculated as follows:

[0061]

[0062] Where (x,y) are the horizontal and vertical coordinates of any pixel in the radar image, (c x ,c y The coordinates of the center point of the radar image are given. Based on the preset blind zone radius, a ring-shaped mask matrix is ​​constructed to shield the physical blind zone and strong reflection interference at the radar center, and to define the effective wave monitoring area, according to the maximum effective radius. The mask matrix has a value of 0 outside the blind zone and maximum effective radius, and a value of 1 within the effective ring region. The expression for the mask matrix is:

[0063]

[0064] Where, ρ min ρ is the minimum effective radius. max To obtain the maximum effective radius, the original radar image is multiplied pixel-by-pixel by the mask matrix to obtain the radar image after the blind zone is masked. The radar image after the blind zone and the image edges is I. mask (x,y,t k ) = I raw (x,y,t k )·M(x,y), where M(x,y) is the value of the mask matrix at pixel (x,y), I raw (x,y,t k ) represents the t-th k The raw radar image value of pixel (x, y) at time step (x, y) contains all unprocessed pixel information, setting the pixel values ​​of invalid regions to 0 and retaining only the pixel information of valid wave regions. Subsequently, at (c... x ,c y I is obtained by cropping within a square region S centered on ) crop (x,y,t k ), (x,y)∈S, retain the complete region containing the effective ring region, and ensure that the image size input to the subsequent network is fixed.

[0065] To reduce random noise in radar images, a two-dimensional Gaussian filter is used to process the cropped image. Noise is smoothed by convolving the image with a Gaussian kernel; the calculation formula is as follows:

[0066] I filt (x,y,t k)=G*I crop (x,y,t k ),

[0067] Where G is a two-dimensional Gaussian kernel, the value of which follows a Gaussian distribution, and I crop (x,y,t k ) represents the t-th k The image values ​​cropped within the square region S at any given time are filtered while preserving the edge features of the wave pattern to obtain the filtered radar image I. filt (x,y,t k Finally, the mean μ of all pixel values ​​is calculated within the effective region of the mask matrix (i.e., the region where the mask value is 1). I and standard deviation σ I And through the normalization formula:

[0068]

[0069] Among them, t k Let (x, y) be the sampling time, and (x, y) be the pixel coordinates of the radar image. filt (x,y,t k ) represents the pixel value of the radar image after two-dimensional Gaussian filtering, and σ represents the pixel value of the image. I μ is the standard deviation of all pixel values ​​in the radar image within the effective area of ​​the mask. I To obtain the mean value of all pixel values ​​in the radar image within the effective area of ​​the mask, the pixel values ​​in the effective area are normalized to standardize the distribution of image pixel values, eliminate the grayscale differences between different frames, and generate a preprocessed radar wave measurement image sequence.

[0070] For the preprocessing of ship motion history data and ship maneuvering signal data, due to the differences in sampling frequency and timestamps of different sensors, the ship motion state vector is first defined as... Where u is the longitudinal velocity, v is the lateral velocity, and r is the bow roll rate, linear interpolation is used to map both types of data to a fixed time step to achieve time alignment and ensure data synchronization in the time dimension. The interpolation method is as follows: After interpolation, the data is filtered to remove high-frequency noise interference. Finally, standardization (based on the mean and standard deviation of the dataset) is applied to normalize the two classes of data. The normalization calculation formula is as follows:

[0071]

[0072] Where x(t) k ) represents the target interpolation time t k The interpolation result, x(t) i+1 ) and x(t i) represent the original data at time t i and t i+1 The measured value, x'(t) k (This is the time t after standardization) k The normalized data value, μ x Let σ be the mean of this type of data in the training set. x The standard deviation of this type of data in the training set is used to normalize different physical quantities (such as speed, rudder angle, and rotational speed) to make them comparable on a numerical scale. Then, the ship control input is... Where, δ L ,δ R These are the left and right rudder angles, n L ,n R The left and right propeller speeds are respectively used. This signal is also aligned to a unified time series using interpolation methods, and then filtered and standardized. A sliding window method is then used to construct the model input and output samples from the above multi-source data. Let the input window length be L. in The prediction window length is L out The k-th input sample is:

[0073]

[0074] in, The input window contains a sequence of radar wave images. For the ship motion time history sequence in the input window, The input window contains a sequence of ship maneuvering signals, and the corresponding target output is the future L outside the window. out The sequence of ship motion states at each moment Wherein s(t) k+n ), n=1,2,…,Lout, represents the true value of the ship's motion state at the nth time in the future. The input and output samples constructed by the sliding window transform multi-source time series data into a structured form that the model can process.

[0075] S2. Perform multimodal feature extraction on the preprocessed multimodal data, including extracting environmental feature vectors, motion time-series feature vectors, and manipulation feature vectors;

[0076] like Figure 2 As shown, this embodiment performs multimodal feature extraction on the radar wave measurement image sequence, ship motion time history data sequence, and ship maneuvering signal data sequence after preprocessing in step S1. A specially designed neural network module is used to mine the core features of each type of data, resulting in environmental feature vectors, motion time series feature vectors, and maneuvering feature vectors. The specific process is as follows:

[0077] First, for the preprocessed radar wave measurement image sequence Convolutional Neural Networks (CNNs) are used to extract environmental feature vectors. The core working principle of CNNs is to capture spatial features through the correlation between convolutional kernels and the local image. Their structure consists of multiple sets of convolutional blocks (containing convolutional layers and activation functions), dynamic pooling layers, and improved batch normalization layers stacked together. Feature extraction of a single frame image begins with the convolution operation: [The text then describes a process involving input feature maps I...] in The i-th convolutional layer is passed through a convolutional kernel W. i The feature map is calculated by performing a sliding convolution on a local pixel region (m,n), with a size of 3×3.

[0078] F i (x,y)=σ(∑ m,n W i (m,n)·I in (xm·d i ,yn·d i )+b i ),

[0079] Where σ is the activation function, ∑ m,n This represents the summation operation over all elements of the convolution kernel, where m and n are the coordinate indices inside the kernel, and W... i (m,n) represents the weight parameters of the i-th convolutional kernel at position (m,n). in (xm·d i ,yn·d i ) is the input feature map I in At position I in (xm·d i ,yn·d i The pixel value at position ) where d i Let b be the stride of the i-th convolutional layer. i is the bias parameter of the i-th convolutional layer, a trainable parameter used to adjust the baseline value of the convolution operation, enhance the network's fitting ability, strengthen the discriminative power of features through nonlinear transformation, and extract low-level features such as edges and textures. To address the slow convergence problem caused by feature distribution fluctuations during training, the convolutional output features are processed by an improved batch normalization layer, the expression of which is:

[0080]

[0081] Where, μ B,i and These represent the mean and variance of all eigenvalues ​​within the current batch in the i-th layer of batch normalization, respectively, and α. i β is the dynamic scaling factor. i Z is the bias parameter, ε is a very small constant, and z i These are the original feature values, i.e., the original feature values ​​after the i-th layer convolution operation without any normalization processing. Here are the normalized eigenvalues, where... and Let N and N be the mean and variance of all feature values ​​in the i-th layer of the current batch B, respectively. Here, B represents the sample set of the current training batch, and N represents the total number of feature values ​​in batch B. After multiple convolutions and batch normalization, the multi-frame image features are concatenated into a four-dimensional tensor along the time dimension, with dimensions B×T×H×W, where B is the batch size, T is the time step, and H and W are the feature map sizes. Then, a dynamic pooling layer is used to compress the spatial dimension, preserving the key spatial distribution features of the wave morphology. Finally, dimensionality reduction is achieved through a fully connected layer with residual connections, where the expression for the fully connected layer is f. img =W fc ·F+b fc +F res Among them, W fc Let b be the weight matrix of the fully connected layer, F be the high-dimensional feature tensor obtained after processing by the dynamic pooling layer, and b be the weight matrix of the fully connected layer. fc F represents the bias parameter of the fully connected layer. res The residual term is passed through a fully connected layer to obtain the environmental feature vector f. img This vector integrates the spatial distribution and temporal dynamics of waves.

[0082] Secondly, regarding the standardized ship motion time history data sequence A bidirectional long short-term memory (BiLSTM) network is used to extract temporal feature vectors. The working principle of BiLSTM is to capture the bidirectional dependencies of temporal data through LSTM networks in both forward and backward directions, solving the long-distance dependency forgetting problem of traditional RNNs. The forward LSTM starts from the sequence origin. Towards the endpoint t k The data is processed, and the cell state is dynamically updated through a gating mechanism to output a positive hidden state sequence. The inverse LSTM processes from the end point to the beginning point, outputting the inverse hidden state sequence. To dynamically fuse bidirectional features, a temporal attention fusion module is introduced to calculate the attention coefficient at time t:

[0083]

[0084] Where, α t Let be the temporal attention coefficient at time t. The output features of the positive long short-term memory network at time t are... For the output features of the inverse long short-term memory network at time t, w a The attention weight vector is a learnable vector, and positive features are measured through the inner product. and reverse features The importance of, and then through Obtain the fused bidirectional feature sequence To make the model focus more on key moments in the prediction (such as turning and acceleration), temporal importance pooling is then used to aggregate sequential features: the pooling weights are based on the second derivative of the feature sequence. The larger the second derivative, the more drastic the feature change and the higher the weight, as indicated by H = ∑ t w t ·h t (∑ t w t =1) Compress the temporal dimension, and finally reduce the dimension through a fully connected layer with residual connections to obtain the motion temporal feature vector. Among them, w t Let be the pooling weights at time t. For bidirectional feature sequence h t The absolute value of the second derivative at time t. Let the feature vector be d dyn A real vector of dimension d dyn For feature dimensions.

[0085] Finally, regarding the standardized ship maneuvering signal data sequence Artificial neural networks (ANNs) are used to extract manipulation feature vectors. An ANN is a multi-layer fully connected network that works by mapping the input signal into a discriminative representation in a high-dimensional feature space through a combination of linear transformations and nonlinear activations. The input manipulation signal sequence is flattened into a one-dimensional vector of dimension L. in After ×4, the signal is passed through two fully connected layers: the first layer with 128 neurons performs feature mapping and calculates F1 = σ(W1·x1+b1), converting the original control signal into high-dimensional features to capture the synergistic effect of rudder angle and speed; the second layer with 64 neurons performs feature compression and calculates F2 = σ(W2·F1+b2), retaining control features that significantly affect ship motion; finally, the signal is reduced in dimensionality by a linear projection layer. ctrl =W p ·F2+b p Where W1 is the weight matrix of the first fully connected layer, x1 is the flattened ship maneuvering signal sequence, F1 is the high-dimensional feature vector output by the first fully connected layer, and F2 is the compressed feature vector output by the second fully connected layer. p Let b be the weight matrix of the linear projection layer. p The bias parameters of the linear projection layer are used to obtain the manipulated feature vector through the above operations. This vector characterizes the dynamic features of the manipulation command, such as its strength and rate of change.

[0086] S3. Based on the extracted multimodal features, cross-modal fusion is performed to obtain a fused feature vector, including the introduction of an attention mechanism, which inputs the environmental feature vector and the motion temporal feature vector into a multi-head attention layer for fusion to obtain an attention fused feature vector;

[0087] The environmental feature vector f extracted in step S2 img Motion time sequence feature vector f dyn and manipulate the eigenvector f ctrl Cross-modal fusion is performed by using an attention mechanism to achieve deep interaction of features from different modalities, ultimately yielding a fused feature vector. A multi-head attention mechanism is introduced to fuse environmental features and motion temporal features, using the environmental feature vector f as the final result. img As the query vector Q, the motion time-series feature vector f dyn Let K be the key vector and V be the value vector, respectively, and let Q = f be the trainable weight matrix. img (t k W Q K = f dyn (t k W K V=f dyn (t k W V Among them, W Q W k W V For trainable weights, d k The dimension of the key vector is represented by a mapping to Q, K, and V, transforming the features into a unified high-dimensional space to enhance interactivity. The multi-head attention layer contains h parallel attention heads, each independently computing its attention output. The output of the i-th attention head is:

[0088]

[0089] in, These are the trainable weight matrices specific to the i-th attention head, used to map the original query, key, and value vectors to a specific subspace. The calculation of a single-head attention is as follows: d k The dimension of the key vector is used to scale the inner product result to avoid gradient instability. The softmax function makes the sum of the attention weights equal to 1. This operation dynamically assigns the association weights between environmental features and motion features, highlighting feature combinations that are crucial for prediction (such as the association between specific wave patterns and ship turning motion). Subsequently, the outputs of h attention heads are concatenated according to their dimensions to form a high-dimensional feature matrix [head1; ...; head...]. h Then, through the trainable output weight matrix W O Perform a linear transformation to obtain the attention fusion feature vector:

[0090] f attn =MultiHead(Q,K,V)=[head1;…;head h W O ,

[0091] Wherein, head1; ...; head h Let W represent the output features of the first to the h-th attention heads respectively, and [.] represent the high-dimensional feature matrix formed by concatenating the output features of the h attention heads according to their dimensions. O This represents the trainable output weight matrix. Next, the attention is fused with the feature vector f. attn Motion time sequence feature vector f dyn and manipulating the eigenvector f ctrl By concatenating along the dimensions, the intermediate fused features are obtained:

[0092] f mid (t k )=[f attn (t k );f dyn (t k );f ctrl (t k )],

[0093] By concatenating the features, the original feature information of each modality is preserved, providing comprehensive input for subsequent nonlinear fusion. Finally, the intermediate fusion features f are... mid The input is processed through a fully connected network with batch normalization and Dropout layers for nonlinear transformation. First, batch normalization is applied to stabilize the feature distribution. The formula is as follows: Where, μ B and Let f be the batch mean and variance, and α and β be learnable parameters. Overfitting is then suppressed by a Dropout layer (randomly discarding some neurons, e.g., with a dropout probability of 0.5). Following this, feature mapping and compression are performed through multiple fully connected layers, ultimately outputting a fused feature vector f. fusion This vector integrates key information from three modalities: environment, motion, and manipulation, as well as the correlation features between the modalities.

[0094] S4. Construct a feedforward neural network using the fused feature vector as input, and output the predicted value of the ship's motion state;

[0095] The core working principle of a feedforward neural network is to progressively map the fused feature vector to the prediction space of the ship's motion state through the stacking of multiple fully connected layers. Activation functions are used to strengthen the nonlinear correlation between features, while Dropout operations suppress overfitting, ensuring the model's generalization ability under complex sea conditions. The network structure consists of several fully connected layers. The first fully connected layer receives the fused feature vector f. fusionThe intermediate features F are obtained by performing a linear transformation using the weight matrix and biases, followed by processing with the ReLU activation function. 11 =σ(W 11 ·f fusion +b 11 ).

[0096] Among them, F 11 The intermediate feature vector is the output of the first fully connected layer, σ is the activation function, and W is the intermediate feature vector. 11 This is the trainable weight matrix of the first fully connected layer, with dimensions (intermediate feature dimension × fused feature dimension), used to fuse the feature vector f. fusion Mapped to a high-dimensional feature space, b 11 The trainable bias parameters for the first fully connected layer are used to capture the complex interactions between environment, motion, and manipulation in the fused features through nonlinear transformations (such as the impact of rudder angle adjustment on the ship's lateral speed under specific wave conditions). To avoid overfitting due to excessive reliance on local features, a Dropout layer is set after the first fully connected layer, randomly discarding some neuron outputs with a preset probability of 0.5 in this embodiment, making the network learn more robust feature patterns. Subsequently, the intermediate feature F... 11 The input to the second fully connected layer is processed similarly to obtain F. 22 =σ(W 22 ·F dropout +b 22 ), where F dropout W represents the first layer output features after processing by the Dropout layer. 22 b is the trainable weight matrix for the second fully connected layer. 22 The trainable bias parameters for the second fully connected layer are used. Through layer-by-layer feature compression and transformation, the high-dimensional fused features are mapped to a feature space related to the ship's motion state dimension. Finally, the network's output layer is a fully connected layer without an activation function to ensure the continuity of the predicted values. The mapping relationship is S. k =F pred (f fusion ) = W out ·F last +b out Among them, W out and b out These represent the weights and biases of the output layer, F. last S is the feature output of the penultimate layer. k For the future F out The predicted sequence of ship motion states at each time step, with dimension [missing information]. Corresponding to longitudinal velocity u, lateral velocity v, and yaw rate r, specifically, Where F pred(·) represents a mapping consisting of several fully connected layers and activation functions. A dropout operation is set between layers to suppress overfitting, where the motion state at each prediction time is... in, This represents the predicted longitudinal velocity of the ship at the τ-th prediction time. This represents the predicted lateral velocity of the ship at the τ-th prediction time. This represents the predicted value of the ship's bow roll rate at the τ-th prediction time. This output layer directly enables simultaneous prediction of multiple time steps and multiple parameters without recursive generation, thus improving prediction efficiency. Through the design of the feedforward neural network, the environmental, motion, and maneuvering information in the fused features is transformed into specific predicted values ​​of the ship's motion state. The network's multi-layer nonlinear transformation capability can model the ship's strong nonlinear dynamic response under complex sea conditions.

[0097] S5. Optimize the parameters based on the constructed feedforward neural network, including using gradient-based optimization methods to update the parameters and minimize the loss function;

[0098] like Figure 3 , Figure 4 , Figure 5 As shown, using the actual sequence of ship motion states as the supervision signal, the parameters of the feedforward neural network are optimized by constructing a loss function, calculating the gradient, and using an optimizer to update the parameters. First, the MSE (mean squared error) loss function is constructed to measure the difference between the predicted and the actual values. The expression of the loss function is as follows:

[0099]

[0100] in, Let s be the motion state parameter at time τ, output by the feedforward neural network. i (t k+τ ) represents the corresponding true value, L out To predict the window length, d s As a dimension of ship motion state, this loss function quantifies the overall prediction bias by calculating the mean squared error of all motion parameters at all prediction times, providing a target for parameter optimization.

[0101] Next, a gradient-based optimization method is employed. The gradient of the loss function with respect to all trainable parameters θ of the network is calculated using the backpropagation algorithm. These trainable parameters θ encompass all parameters in the network that require optimization, including the kernel weights and biases of convolutional neural networks, the gating weights and biases of bidirectional long short-term memory networks, the mapping weight matrix of multi-head attention layers, and the weights and biases of fully connected networks. The gradient calculation follows the chain rule, and its core expression is:

[0102]

[0103] Among them, L out To predict the window length, d s As a dimension of ship motion state, loss function Predicted value of the i-th motion state parameter at time τ The partial derivatives can be obtained by differentiating the MSE loss function. The predicted value of the i-th motion state parameter at time τ. The partial derivative with respect to the trainable parameter θ reflects the degree of influence of the change in parameter θ on the predicted value. The double summation integrates the gradient contributions of all prediction times and all motion parameters to parameter θ, ensuring that the gradient calculation covers all dimensions and time range of the model output.

[0104] Finally, the Adam optimizer is used for parameter updates. The Adam optimizer combines the advantages of momentum and adaptive learning rates, dynamically adjusting the update step size for each parameter by maintaining the first and second moment estimates. Its update formula is as follows:

[0105] Through this update rule, the parameter θ is adjusted in the opposite direction of the gradient, gradually reducing the value of the loss function until the model converges on the validation set. Through the above parameter optimization process, the network can continuously learn the mapping relationship between the environment, motion, and maneuvering features and the future motion state of the ship, so that the predicted value gradually approaches the true value, and ultimately improves the prediction accuracy and stability of the model in complex sea conditions.

[0106] S6. Verify the prediction results output by the feedforward neural network and output the final prediction results;

[0107] like Figure 6 As shown, after completing the parameter optimization in step S5, the prediction results output by the feedforward neural network need to be verified to ensure the effectiveness of the model in the actual scenario, and the final ship maneuvering motion prediction result is output to form a closed loop of the technical solution. The specific process is as follows: First, an independent test dataset (without overlap with the training and validation sets) is input into the trained model. This test dataset contains preprocessed radar wave image sequences, ship motion time history data, and maneuvering signal data, and covers a variety of typical sea states (such as calm sea surface, medium and high sea state) and maneuvering conditions to comprehensively test the model's generalization ability. The model's prediction output for the test data is the future L out The sequence of ship motion states at each moment:

[0108]

[0109] in, Subsequently, quantitative evaluation indicators were used to verify the accuracy of the prediction results. The core indicators included root mean square error (RMSE) and mean absolute error (MAE). The formula for calculating MAE is as follows:

[0110]

[0111] Where N is the total number of samples in the test set. and s i,k (t k+τ Let and represent the predicted and actual values ​​of the i-th motion state parameter at the τ-th prediction time, respectively. The mean absolute error is used to measure the average deviation, providing a more intuitive reflection of prediction accuracy. If the verification results meet the preset accuracy requirements (RMSE and MAE are below a set threshold), the model passes verification and enters the inference phase. If not, the process returns to step S5 to adjust the optimization strategy, retrain, and verify until the target is met. During the inference phase, the standardized parameters (mean μ) saved during training are used. x and standard deviation σ x The normalized predicted values ​​output by the model are then processed through... Convert the data back to physical quantities (e.g., longitudinal velocity in m / s, rudder angle in degrees) to ensure the output dimensions are consistent with the actual navigation parameters. pred (t) represents the predicted ship motion state with actual physical meaning after restoration. σ is the normalized predicted value output by the model. x and μ x These are the standard deviation and mean of the training dataset calculated during the preprocessing stage, respectively. The final output prediction results are the sequences of the ship's longitudinal velocity, lateral velocity, and bow roll rate over a set future time interval, expressed as:

[0112]

[0113] in The longitudinal velocity, lateral velocity, and bow turning angular velocity are represented respectively, and T represents the prediction step size. This result can directly provide a basis for ship maneuvering decisions, such as collision avoidance path planning and collision warning, realizing a complete closed loop from data input to decision support. Through the above verification and output process, the reliability and practicality of the prediction results are ensured, enabling the intelligent prediction method of ship maneuvering motion based on shipborne radar wave measurement to be effectively applied to actual navigation scenarios, thereby improving the safety and intelligence level of ship maneuvering.

[0114] Example 2

[0115] The difference between this embodiment and Embodiment 1 is that this embodiment provides a ship maneuvering trajectory prediction system that integrates shipborne radar wave measurement, including:

[0116] The data acquisition module is configured to acquire multimodal data and preprocess the acquired multimodal data, including acquiring radar wave images, ship motion history data and ship maneuvering signal data.

[0117] The feature extraction module is configured to perform multimodal feature extraction on the preprocessed multimodal data, including extracting environmental feature vectors, motion temporal feature vectors, and manipulation feature vectors.

[0118] The fusion module is configured to perform cross-modal fusion based on the extracted multimodal features to obtain a fused feature vector, including introducing an attention mechanism to input environmental feature vectors and motion temporal feature vectors into a multi-head attention layer for fusion to obtain an attention fusion feature vector;

[0119] The prediction module is configured to: construct a feedforward neural network by taking the fused feature vector as input and output a predicted value of the ship's motion state;

[0120] The optimization module is configured to: perform parameter optimization based on the constructed feedforward neural network, including updating parameters using gradient-based optimization methods to minimize the loss function;

[0121] The output module is configured to verify the prediction results of the feedforward neural network and output the final prediction results.

[0122] The above are all preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, all equivalent changes made in accordance with the structure, shape and principle of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A ship maneuvering motion trajectory prediction method that fuses shipborne radar wave measurement, characterized by, The application relates to a ship motion state prediction method based on multi-modal data fusion, comprising the following steps: acquiring multi-modal data including a radar wave image, ship motion time history data and ship maneuvering signal data, and preprocessing the acquired multi-modal data; extracting multi-modal features from the preprocessed multi-modal data, including extracting an environmental feature vector, a motion time sequence feature vector and a maneuvering feature vector; the step of extracting multi-modal features from the preprocessed multi-modal data further comprises the following steps: inputting the normalized ship motion time history data sequence into a bidirectional long short-term memory network, introducing a dynamic gating mechanism and a time sequence attention fusion module into the long short-term memory network, fusing the output sequence of the forward long short-term memory network and the output sequence of the reverse long short-term memory network through a time sequence attention weight, obtaining a bidirectional feature sequence, aggregating the bidirectional feature sequence through a time sequence importance pooling, calculating a second derivative of the feature sequence as a pooling weight, finally reducing the dimension of the motion time sequence feature vector through a fully connected layer with a residual connection, and the expression of the bidirectional feature sequence is as follows: wherein, α t is the time sequence attention coefficient at the t th moment, is the output feature of the forward long short-term memory network at the t th moment, is the output feature of the reverse long short-term memory network at the t th moment, w a is a learnable attention weight vector; performing cross-modal fusion based on the extracted multi-modal features to obtain a fusion feature vector, including introducing an attention mechanism, inputting the environmental feature vector and the motion time sequence feature vector into a multi-head attention layer to obtain an attention fusion feature vector, wherein the environmental feature vector is used as a query vector, the motion time sequence feature vector is used as a key vector and a value vector, the query vector, the key vector and the value vector are mapped through a trainable weight matrix, the multi-head attention layer calculates the attention output through multiple parallel attention heads, the output features of the attention heads are spliced and then linearly transformed to obtain the attention fusion feature vector, and the expression of the linear transformation is as follows: MultiHead(Q, K, V) = [head1;... ; head h ]W O , wherein head1;... ; head h represent the output features of the 1st to hth attention heads, respectively, [.] represents a high-dimensional feature matrix formed by concatenating the output features of the h attention heads in the dimension, W O represents a trainable output weight matrix; constructing a feedforward neural network by taking the fusion feature vector as input, and outputting a predicted value of a ship motion state; performing parameter optimization based on the constructed feedforward neural network, including updating the parameters through a gradient-based optimization method to minimize a loss function; verifying the prediction result output by the feedforward neural network, and outputting a final prediction result.

2. The fusion shipboard radar wave- tracked ship maneuvering motion trajectory prediction method according to claim 1, characterized by, the preprocessing of the acquired multi-modal data comprises the following steps: shielding a blind area of the acquired radar wave image, taking a center point of the image as an origin, calculating a polar radius of a pixel point to the center point of the image, constructing a mask matrix according to a preset blind area radius and a maximum effective radius, generating a radar image after shielding the blind area according to the mask matrix, performing a clipping operation on the radar image after shielding the blind area, performing a filtering operation on the clipped image through two-dimensional Gaussian filtering, performing normalization processing in a mask effective area, generating a preprocessed shipborne radar wave image, finally performing time alignment on the ship motion time history data and the ship maneuvering signal data through a linear interpolation method, and performing filtering and standardization processing on the aligned data, and the normalization processing formula is as follows: where t k is the sampling time, (x, y) is the pixel coordinate of the radar image, I filt (x, y, t k ) is the pixel value of the radar image after two-dimensional Gaussian filtering, σ I is the standard deviation of all pixel values of the radar image within the effective area of the mask, μ I is the mean value of all pixel values of the radar image within the effective area of the mask.

3. The fusion shipborne radar wave-observed ship maneuvering motion trajectory prediction method according to claim 1, characterized by, The multi-modal feature extraction on the pre-processed multi-modal data comprises inputting the pre-processed radar wave image into a convolutional neural network, the convolutional neural network is composed of multiple groups of convolutional blocks, a dynamic pooling layer and a batch normalization layer, single-frame image features are extracted through a convolutional operation first, the improved batch normalization layer is used to process the convolutional output features based on the increase of the dynamic scaling factor in the batch normalization layer, after a plurality of frame features are spliced into a four-dimensional feature tensor in the time dimension, the spatial dimension is compressed through the dynamic pooling layer, and then the environment feature vector is obtained through the full connection layer with residual connection for dimension reduction, and an expression of the batch normalization layer is as follows: where μ B,i and are the mean and variance of all feature values in the current batch in the i-th layer batch normalization, α i is the dynamic scaling factor, β i is the bias parameter, and ε is a small constant, z i is the original feature value after the i-th layer convolution operation without any normalization, and is the normalized feature value.

4. The fusion shipborne radar wave-observed ship maneuvering motion trajectory prediction method according to claim 1, characterized by, The multi-modal feature extraction on the pre-processed multi-modal data also comprises inputting the normalized ship maneuvering signal data sequence into an artificial neural network, the ship maneuvering signal data sequence comprises a plurality of time steps, each time step comprises a rudder angle and a left and right propeller rotating speed, the artificial neural network is a multi-layer full connection network, each layer of neurons adopts an activation function, the input ship maneuvering signal data is processed through flattening and then sequentially passes through each layer of full connection network, is mapped to features through the first layer of network first, is compressed to features through the second layer of network, and the output features are dimensionally reduced through linear projection to obtain a maneuvering feature vector.

5. The fusion shipborne radar wave- tracked ship maneuvering motion trajectory prediction method according to claim 1, characterized by, The cross-modal fusion based on the extracted multi-modal features also comprises splicing the attention fusion feature vector, the motion time sequence feature vector and the maneuvering feature vector in the dimension to obtain an intermediate fusion feature, and then inputting the intermediate fusion feature into a full connection network with a batch normalization and a Dropout layer for nonlinear transformation to output a fusion feature vector.

6. The fusion shipborne radar wave- tracked ship maneuvering motion trajectory prediction method according to claim 1, characterized by, The fusion feature vector is taken as input to construct a feedforward neural network, which comprises taking the fusion feature vector as input to construct a feedforward neural network composed of a plurality of full connection layers and activation functions, each full connection layer adopts an activation function to strengthen the nonlinear correlation between features, and a Dropout operation is directly set between the full connection layers, the feedforward neural network outputs a ship maneuvering motion prediction value at a future set time interval through feature mapping and nonlinear transformation of a plurality of full connection layers.

7. The fusion shipborne radar wave- tracked ship maneuvering motion trajectory prediction method according to claim 1, characterized by, The parameter optimization based on the constructed feedforward neural network comprises taking a real sequence of ship motion states as a supervision signal, constructing an MSE loss function to measure the difference between the prediction value output by the feedforward neural network and the real sequence, updating parameters by using a gradient-based optimization method, calculating the gradient of all trainable parameters of the network by using a back propagation algorithm, and finally updating the parameters by using an Adam optimizer, and an expression of the gradient of the trainable parameters is as follows: Among them, L out To predict the window length, d s As a dimension of the ship's motion state, For loss function Predicted value of the i-th motion state parameter at time τ The partial derivatives, The predicted value of the i-th motion state parameter at time τ. The partial derivative with respect to the trainable parameter θ reflects the degree to which changes in parameter θ affect the predicted value.

8. A ship maneuvering motion trajectory prediction system fusing shipboard radar wave measurement, performing the method of claim 1, characterized by, The data acquisition module is configured to acquire multi-modal data comprising a radar wave image, ship motion time history data and ship maneuvering signal data, and pre-process the acquired multi-modal data; The feature extraction module is configured to perform multi-modal feature extraction on the pre-processed multi-modal data, comprising extracting an environment feature vector, a motion time sequence feature vector and a maneuvering feature vector; ​ The fusion module is configured to perform cross-modal fusion based on the extracted multi-modal features to obtain a fusion feature vector, including introducing an attention mechanism, inputting the environmental feature vector and the motion time sequence feature vector into a multi-head attention layer to obtain an attention fusion feature vector; The prediction module is configured to construct a feedforward neural network by taking the fusion feature vector as input and output a prediction value of the ship motion state; The optimization module is configured to perform parameter optimization based on the constructed feedforward neural network, including updating the parameters by using a gradient-based optimization method to minimize a loss function; The output module is configured to verify the prediction result output by the feedforward neural network and output a final prediction result.

Images