A Multimodal Deep Learning Fusion Method and System for Inverting Ocean Wave Parameters from Ocean Remote Sensing

Abstract

This invention belongs to the field of marine environmental monitoring technology and discloses a method and system for inverting ocean wave parameters using multimodal deep learning fusion. The method acquires three-degree-of-freedom motion history data of ships within a target area and spaceborne synthetic aperture radar (SAR) image data. It preprocesses the ship motion history data and the SAR image data to construct spatiotemporally aligned multimodal input data. The preprocessed multimodal input data is then input into a pre-trained wave parameter inversion model to calculate the inverted values ​​of the wave parameters. This method innovatively uses an attention mechanism to achieve adaptive alignment and complementarity of heterogeneous information in both spatial and temporal dimensions. The attention module automatically assigns weights based on task relevance. Through spatiotemporal collaborative feature enhancement, the model can stably extract representations strongly correlated with target parameters even under complex sea conditions, thereby significantly improving the inversion accuracy and robustness of wave parameters.

Description

Technical Field

[0001] This invention belongs to the field of marine environmental monitoring technology, and in particular relates to a method and system for inverting ocean wave parameters by multimodal deep learning fusion. Background Technology

[0002] Accurate wave parameter inversion is crucial for ensuring safe maritime operations. However, the highly nonlinear characteristics of ocean waves and sensor measurement errors make accurate wave parameter sensing challenging. Researchers in this field utilize various sensors for marine environmental sensing, such as shipborne radar, buoys, satellite sensors, and marine weather stations.

[0003] Existing patent CN202410859719.4 discloses a ship motion prediction method based on a cyclic Transformer model. This method preprocesses ship motion time-series data, segmenting and dividing the preprocessed data into blocks. A cyclic Transformer-based ship motion prediction model then outputs embedded vector sequences from these blocks. Finally, it obtains predicted values ​​for the ship's roll angle, three-degree-of-freedom angles, and heave height to predict the ship's motion. However, this approach uses a single ship motion sequence as input and employs a cyclic / Transformer model to predict future ship motion. Its goal is to extrapolate the time series of motion trajectories / attitudes, without processing SAR imagery or outputting wave statistics.

[0004] Existing patent CN202111604121.3 discloses a method for inverting ocean wave parameters. This method pre-trains an inversion ocean wave parameter model capable of retrieving ocean wave parameters. By inputting a first wind dataset, a first temperature dataset, and a first inertial navigation dataset representing ocean wave parameters in a sea area, obtained by a floating offshore wind-measuring lidar within a first preset time period, into the pre-trained inversion ocean wave parameter model, the ocean wave parameters for that sea area within the first preset time period can be obtained. However, this method relies on integrating a floating offshore wind-measuring lidar with an ocean wave buoy sensor and inverting ocean wave parameters using the collected wind, temperature, and inertial navigation datasets, without utilizing the spatiotemporal complementarity between measured ship motion and SAR (Survey and Radar Analysis).

[0005] Currently, in wave parameter inversion research using Synthetic Aperture Radar (SAR), deep learning methods are gradually becoming an important tool for wave field parameter inversion due to their efficient real-time processing capabilities, powerful nonlinear modeling capabilities, and feature extraction capabilities. This method, based on collected sample data, extracts wave texture features from spaceborne SAR images through extensive training of a deep neural network model, thereby achieving accurate wave field parameter inversion. The existing technologies have the following problems and limitations:

[0006] (1) Existing single-modal data-driven methods are not robust enough. Due to the lack of data features, large-scale models are prone to overfitting. They are not robust enough and have poor adaptability when not fully regularized.

[0007] (2) Existing data fusion methods lack real-time performance and high precision. They rely on the complexity of the environment settings, making it difficult to respond to dynamic sea conditions in real time and failing to meet the timeliness requirements in practical applications.

[0008] (3) After a preliminary search of published literature and patents, no technical solution was found that combines ship motion response data with spaceborne SAR images at the feature level to carry out wave field parameter inversion. Summary of the Invention

[0009] To overcome the problems existing in related technologies, the present invention discloses a method and system for inverting ocean remote sensing wave parameters using multimodal deep learning fusion. Specifically, it involves combining spaceborne SAR images and ship motion data, using convolutional neural networks (CNN) and bidirectional long short-term memory networks (BiLSTM) for feature extraction, and inverting meaningful wave height and characteristic periodicity through attention mechanisms and progressive fusion strategies. The technical solution is as follows:

[0010] This invention is implemented as follows: a multimodal deep learning fusion-based method for inverting ocean remote sensing wave parameters, comprising the following steps:

[0011] S1, read the collected three-degree-of-freedom motion time history data of the ship to obtain the motion time sequence. And acquire SAR image data under the corresponding working conditions; use attention mechanism to achieve adaptive alignment and complementarity of heterogeneous information in the spatial and temporal dimensions. The attention module automatically assigns weights according to the task relevance and uses ship motion and SAR images to perform wave parameter fusion and inversion.

[0012] S2, preprocess the three-degree-of-freedom motion history data of the ship and the SAR image data to construct spatiotemporally aligned multimodal input data;

[0013] S3, the preprocessed multimodal input data is input into the pre-trained wave parameter inversion model to calculate the inversion values ​​of the wave parameters; the wave parameter inversion model is a deep learning-based fusion network model used to extract temporal features from the ship's three-degree-of-freedom motion history data and spatial features from the SAR image sub-region, and to fuse the temporal and spatial features; wherein, the SAR image sub-region is a square region cropped based on the ship's position.

[0014] In step S2, the preprocessing includes: first, selecting a spaceborne SAR image with both imaging and sea state data, and recording the center imaging time of the SAR image as... Then, a time series segment with a length of 100 mm is extracted before and after that point in time. motion sequence window ,use Use the ship's position at any given time as the image cropping center to crop out one image. A square region is defined to ensure spatial alignment, thereby completing the construction of multimodal input data.

[0015] In step S3, the multimodal input data is calculated using the CNN-BiLSTM fusion framework. The CNN module automatically learns and extracts discriminative spatial or local hierarchical features from the high-dimensional radar image, and the BiLSTM module effectively captures the temporal dependencies and contextual information in the ship motion response. Multiple attention mechanisms are incorporated to optimize features and improve model focusing capabilities, and wave parameters are obtained by inversion.

[0016] In step S3, the wave parameter inversion model adopts a heterogeneous dual-stream parallel structure. This architecture fully extracts modal features in both the image and time-series domains, avoiding feature aliasing and modal interference, preserving the spatial texture and periodic structure of the radar image, and capturing key fluctuations and dynamic changes in the ship's response; including:

[0017] The radar image branch employs a two-dimensional convolutional neural network (Conv2D) structure to extract spatial visual features from the SAR image sub-regions.

[0018] The ship motion branch employs a stacked bidirectional long short-term memory network (BiLSTM) structure to extract temporal dynamic features from the ship's three-degree-of-freedom motion history data.

[0019] The feature fusion module employs a progressive multimodal fusion strategy (PMMFS) to fuse the spatial visual features output from the radar image branch and the temporal dynamic features output from the ship motion branch.

[0020] Furthermore, an attention module is also embedded in the radar image branch to enhance the feature map extracted by the CNN. The attention module includes a spatial attention (SA) submodule and a channel attention (CA) submodule.

[0021] The spatial attention (SA) submodule first generates a normalized attention mask in the spatial dimension, dynamically emphasizing high-energy wave packets and significant peak regions; the channel attention (CA) submodule explicitly models cross-channel dependencies through global pooling and lightweight MLP, assigning independent modulation weights to each channel.

[0022] Convolution output feature map The refined formalization of attention is as follows:

[0023] ;

[0024] ;

[0025] ;

[0026] ;

[0027] In the formula, For the Sigmoid function, for convolution, Global average pooling for empty dimensions. Global max pooling for empty dimensions For global average pooling across all dimensions, For global max pooling across the entire dimension, For element-wise multiplication, For ReLU function, The feature map after applying spatial attention. This is the output after weighting the feature map using spatial attention. The output after further weighting of channel attention, Here is the weight matrix in the channel attention CA. This is the weight matrix for the second stage of the attention mechanism. The feature map after applying spatial attention and channel attention. This is the original feature map output by the convolutional layer. ;

[0028] The image branch is global pooled and fully connected to project a 256-dimensional compact vector, which serves as the single-modal space representation for subsequent multimodal fusion.

[0029] Furthermore, the spatial attention (SA) submodule and the channel attention (CA) submodule operate in series, first performing spatial attention weighting and then performing channel attention weighting.

[0030] Furthermore, the ship motion branch is connected to a temporal attention (TA) module after the network BiLSTM structure, which is used to perform weighted aggregation of the hidden states of all time steps output by BiLSTM to obtain a fixed-length temporal feature vector.

[0031] At time step Input sequence The output of the BiLSTM network structure is:

[0032] ;

[0033] In the formula, For time step The output is hidden. To perform vector concatenation operations, The hidden state of the input sequence is processed by the network in the forward direction. The hidden state of the input sequence is processed in reverse for the network;

[0034] ;

[0035] In the formula, positive The calculation process, For the reverse The calculation process, For the first Input data at each time step, positive At time step The hidden state, For the reverse At time step The hidden state.

[0036] Furthermore, the progressive multimodal fusion strategy PMMFS includes:

[0037] Feature alignment projection: The spatial visual features and temporal dynamic features are projected onto a shared adaptive feature space through a fully connected layer to achieve intramodal alignment and scale matching;

[0038] Preliminary fusion and cross-modal attention: The projected features are stitched together and preliminarily fused. A bidirectional asymmetric cross-modal attention mechanism is introduced, which uses temporal dynamic features as the query and reweights spatial visual features, and / or uses spatial visual features as the query and reweights temporal dynamic features.

[0039] Deep integration and regression: The features enhanced by cross-modal attention and the preliminary fusion features are input into a shared fully connected layer for integration, and the final wave parameter inversion value is output through a regression layer.

[0040] Furthermore, the wave parameters include significant wave height and characteristic period.

[0041] Another objective of this invention is to provide a multimodal deep learning fusion-based ocean remote sensing wave parameter inversion system. This system is used to regulate the multimodal deep learning fusion-based ocean remote sensing wave parameter inversion method. The system includes:

[0042] The data acquisition module is used to acquire ship motion history data and spaceborne SAR image data;

[0043] The data preprocessing module is used to preprocess the ship motion history data and the SAR image data to construct spatiotemporally aligned multimodal input data;

[0044] The model calculation module has a built-in pre-trained wave parameter inversion model, which is used to receive the multimodal input data and output wave parameter inversion values.

[0045] Combining all the above technical solutions, the beneficial effects of this invention are as follows:

[0046] First, this invention proposes a network framework that integrates spaceborne SAR images and ship motion response data to perform high-precision wave parameter inversion. This method innovatively uses an attention mechanism to achieve adaptive alignment and complementarity of heterogeneous information in both spatial and temporal dimensions. The attention module automatically allocates weights based on task relevance: in the spatial dimension, it highlights significant regions related to wave texture, wave crests and ridges, etc., and in the temporal dimension, it focuses on key segments such as peaks and valleys, periodicity and trend inflection points, while suppressing spurious features and noise. Through this spatiotemporal collaborative feature enhancement, the model can stably extract representations strongly correlated with target parameters even under complex sea conditions, thereby significantly improving the inversion accuracy and robustness of wave parameters and possessing good cross-scene generalization ability.

[0047] Experimental results show that the ocean remote sensing wave parameter inversion method based on attention mechanism and multimodal deep learning fusion of the present invention outperforms other neural network models on all evaluation metrics, fully demonstrating the high accuracy and effectiveness of this method in wave field statistical parameter inversion. By fusing multimodal data, this neural network model can integrate information from different sources, thereby significantly improving the model's robustness and outperforming other methods in dealing with complex sea conditions. This method not only improves inversion accuracy and real-time performance, but also innovatively enhances adaptability to changing sea conditions by flexibly fusing multi-source heterogeneous data from spatiotemporal dimensions.

[0048] Secondly, this invention improves the accuracy of wave parameter inversion. Through a multimodal fusion method based on an attention mechanism, it effectively integrates multimodal information from ship motion response data and spaceborne SAR images from a spatiotemporal perspective, enabling accurate inversion of key statistical parameters of the wave field, such as characteristic period and meaningful wave height. Compared to traditional single-modal methods, fusing multiple data sources allows the model to capture more comprehensive wave features, thereby significantly improving inversion accuracy.

[0049] This invention enhances the robustness of the model; through the fusion of multimodal data, it can fully utilize the complementarity of data from different sources, overcoming the limitations of a single data source. When faced with environmental factors such as sea state changes and wind speed fluctuations, the model of this invention maintains stable performance, exhibiting strong adaptability and robustness, and avoiding the instability in inversion accuracy caused by data changes in traditional methods.

[0050] This invention is adaptable to complex sea conditions and has wide applicability. It can not only adapt to common sea conditions but also effectively handle wave inversion tasks in complex sea conditions. Compared to traditional methods that are mainly applicable to single or limited sea conditions, the multimodal fusion characteristics of this method give it broader adaptability and enable it to maintain high-precision inversion results under various dynamic sea conditions.

[0051] Third, this invention utilizes the edge deployment of a real-time wave parameter inversion system based on the fusion of spaceborne SAR and ship motion to provide real-time wave parameter inversion, directly supporting route optimization, offshore work window assessment, nearshore wind power maintenance scheduling, and maritime emergency command. It eliminates the need for continuous maintenance of numerous buoys and shore-based radars, and relies on on-orbit satellites and the large-scale production of wave products from the fleet of ships in operation, significantly reducing the cost of observation per unit sea area.

[0052] Fourth, this invention is the first to deeply fuse spaceborne SAR imagery with measured ship motion responses at the feature level for wave statistical parameter inversion. It proposes a heterogeneous dual-stream structure with Conv2D spatial branching and BiLSTM temporal branching, combined with PMMFS progressive multimodal fusion, and superimposes spatial, channel, and temporal attention to achieve spatiotemporal adaptive alignment and complementarity. Through the fusion of spaceborne SAR and ship motion, reliable inversion is achieved under weak synchronization conditions, eliminating dependence on fixed observation arrays. Empirical evidence demonstrates the advantages of feature-level cross-modal fusion in terms of accuracy and robustness. The progressive fusion of heterogeneous dual-stream and cross-modal attention achieves collaborative learning within a unified representation space, resulting in high-quality inversion results. Attached Figure Description

[0053] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure;

[0054] Figure 1 This is a flowchart of the multimodal deep learning fusion method for inverting ocean remote sensing wave parameters provided in this embodiment of the invention;

[0055] Figure 2 This is a schematic diagram of motion time history data provided in an embodiment of the present invention; wherein, (a) is a heave diagram, (b) is a three-degree-of-freedom diagram, and (c) is a roll diagram;

[0056] Figure 3These are spaceborne SAR images provided in embodiments of the present invention;

[0057] Figure 4 This is a schematic diagram of wave heights from wave time-history data provided in an embodiment of the present invention;

[0058] Figure 5 This is a schematic diagram of radar image cropping provided in an embodiment of the present invention;

[0059] Figure 6 This is a diagram of the fusion network structure provided in an embodiment of the present invention;

[0060] Figure 7 This is a visualization of the first CA module applied to 256 channels provided in this embodiment of the invention.

[0061] Figure 8 This is a radar chart of the characteristic periodicity evaluation index provided in the embodiments of the present invention;

[0062] Figure 9 This is a radar chart of the meaningful wave height evaluation index provided in the embodiments of the present invention;

[0063] Figure 10 This is the weight graph of the temporal attention module provided in the embodiments of the present invention;

[0064] Figure 11 This is a heatmap of the spatial attention module provided in an embodiment of the present invention. Detailed Implementation

[0065] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0066] The innovation of this invention lies in its pioneering use of ship motion and SAR images for wave parameter fusion and inversion. It innovatively designs a multimodal fusion framework for the extraction and fusion of spatiotemporal information, significantly improving inversion accuracy. This invention employs an attention-based multimodal fusion method, effectively integrating multi-source heterogeneous information from spaceborne SAR images and ship motion response data across spatiotemporal dimensions to achieve information complementarity and accurate wave parameter inversion. This method overcomes the limitations of traditional single-modal data processing, utilizing spatial attention, channel attention, and temporal attention mechanisms to weightedly fuse different modal data, thereby improving the model's robustness and accuracy. Through this innovative data fusion approach, real-time high-resolution wave parameter inversion is achieved, exhibiting excellent adaptability and stability, especially under complex sea conditions.

[0067] This invention integrates spaceborne SAR and ship motion data for wave parameter inversion, fusing multi-source heterogeneous data in both temporal and spatial dimensions to achieve information complementarity. It designs a multimodal fusion network based on an attention mechanism, utilizing spatial, channel, and temporal attention mechanisms to adaptively weight and optimize different modal data. Through a dual-branch network structure (combining Conv2D and BiLSTM) and a progressive multimodal fusion strategy (PMMFS), it achieves efficient wave parameter inversion, making it particularly suitable for real-time applications under dynamic sea conditions.

[0068] Example 1, such as Figures 1-6 As shown in the figure, the multimodal deep learning fusion method for inverting ocean remote sensing wave parameters provided in this embodiment of the invention includes the following steps:

[0069] S1, read the collected three-degree-of-freedom motion time history data of the ship to obtain the motion time sequence. And acquire SAR image data under the corresponding working conditions; use attention mechanism to achieve adaptive alignment and complementarity of heterogeneous information in the spatial and temporal dimensions. The attention module automatically assigns weights according to the task relevance and uses ship motion and SAR images to perform wave parameter fusion and inversion.

[0070] S2, preprocess the three-degree-of-freedom motion history data of the ship and the SAR image data to construct spatiotemporally aligned multimodal input data;

[0071] S3, the preprocessed multimodal input data is input into the pre-trained wave parameter inversion model to calculate the inversion values ​​of the wave parameters; the wave parameter inversion model is a deep learning-based fusion network model used to extract temporal features from the ship's three-degree-of-freedom motion history data and spatial features from the SAR image sub-region, and to fuse the temporal and spatial features; wherein, the SAR image sub-region is a square region cropped based on the ship's position.

[0072] In step S2, the preprocessing includes: first, selecting a spaceborne SAR image with both imaging and sea state data, and recording the center imaging time of the SAR image as... Then, a time series segment with a length of 100 mm is extracted before and after that point in time. motion sequence window ,use Use the ship's position at any given time as the image cropping center to crop out one image. A square region is defined to ensure spatial alignment, thereby completing the construction of multimodal input data.

[0073] The image branch uses a two-dimensional convolutional neural network (Conv2D, a variant of CNN) to learn representations from spaceborne SAR images, extracting visual patterns such as texture, shape, and hierarchical spatial structure. This branch exhibits equivariance to visual elements such as translation, enabling the construction of multi-level representations ranging from low-order statistics to high-order semantics. Its goal is to decode latent spatial structures strongly correlated with wave statistical / dynamic properties in sea clutter images containing multiplicative speckle and nonlinear components.

[0074] Following a hierarchical abstraction approach: shallow convolutional units (with smaller receptive fields) prioritize capturing high-frequency visual primitives (convolutional layers use smaller receptive fields to process local regions of the image, allowing the network to focus on subtle changes in the image, such as ripples, brightness gradients of peaks and troughs. Since these high-frequency features typically manifest as edges, textures, and small-scale fluctuations in the image, shallow convolutions can extract discriminative visual primitives through their sensitive response to these local features). As the network deepens, deep convolutions exponentially expand the receptive field through cascaded convolutions and downsampling, thereby encoding larger-scale spatial information and representing more abstract, macroscopic wave field patterns (such as global periodicity and anisotropy). With increasing network layers, deep convolutions, through multiple convolutions and pooling operations, gradually fuse local information into higher-order abstract features, capturing macroscopic features such as global wave periodicity and anisotropy. This gradually expanding receptive field enables the network to effectively learn different scale features from local ripples to global wave structures, thereby improving the accuracy and stability of wave parameter inversion.

[0075] To improve feature extraction performance, this embodiment starts with a lightweight baseline and empirically optimizes the network depth and convolutional kernel settings. Considering that "bright ridges" in sea clutter images often correspond to coherent structures related to wave peaks, spatial attention (SA) and channel attention (CA) modules are cascaded after each convolutional block to form a robust feature post-processing unit. SA first generates a normalized attention mask in the spatial dimension to dynamically emphasize high-energy wave packets and significant wave peak regions; then CA explicitly models cross-channel dependencies through global pooling and a lightweight MLP, assigning independent modulation weights to each channel.

[0076] Convolution output feature map Attention refinement can be formalized as:

[0077] ;

[0078] ;

[0079] ;

[0080] ;

[0081] In the formula, For the Sigmoid function, for convolution, Global average pooling for empty dimensions. Global max pooling for empty dimensions For global average pooling across all dimensions, For global max pooling across the entire dimension, For element-wise multiplication, For ReLU function, The feature map after applying spatial attention. This is the output after weighting the feature map using spatial attention. The output after further weighting of channel attention, This is the weight matrix in Channel Attention (CA). This is the weight matrix for the second stage of the attention mechanism. The feature map after applying spatial attention and channel attention. This is the original feature map output by the convolutional layer. ;

[0082] Finally, the image branch is processed by global pooling and fully connected projection to output a 256-dimensional compact vector, which serves as the unimodal space representation for subsequent multimodal fusion. The attention visualization is shown below. Figure 7 As shown.

[0083] Long Short-Term Memory (LSTM) networks overcome the gradient vanishing / exploding problem in traditional RNNs when processing long sequences through input / forgetting / output gates. In ship motion response, there is a bidirectional time dependency between force and response. To make full use of the context, this embodiment uses a stacked bidirectional LSTM (BiLSTM) as the core time feature extractor, simultaneously modeling short-term and long-term dynamics.

[0084] The network consists of a hierarchical structure of two consecutive BiLSTM layers: the shallow layer focuses on high-frequency local motion patterns (such as short-wavelength induced rapid oscillations), while the deep layer aggregates and identifies broader, long-term dynamic trends. At time steps... Input sequence The output of BiLSTM is:

[0085] ;

[0086] In the formula, For time step The output is hidden. To perform vector concatenation operations, The hidden state of the input sequence is processed by the network in the forward direction. The hidden state of the input sequence is processed in reverse for the network;

[0087] ;

[0088] In the formula, positive The calculation process, For the reverse The calculation process, For the first Input data at each time step, positive At time step The hidden state, For the reverse At time step The hidden state.

[0089] Considering that the measured time series can be as long as 1200 steps, and key dynamics (such as peaks, troughs, and trend inflection points) need to be explicitly emphasized, this embodiment adds a temporal attention (TA) module after BiLSTM to perform weighted aggregation of the hidden state sequence, obtaining a fixed-length context vector; then, through fully connected layers and Dropout, a compact temporal representation with the same dimension (256 dimensions) as the image branches is obtained. The channel attention mechanism can explicitly model the dependencies between the channels of the feature map, learning the importance weight of each channel during the process, adaptively enhancing feature channels with high information content and suppressing channels with low contribution.

[0090] To achieve effective fusion of heterogeneous spatiotemporal data, this invention proposes a Progressive Multimodal Fusion Strategy (PMMFS) to systematically extract complementary features from spatial imagery and time-series data. Given that heterogeneous modalities are often distributed across different feature manifolds, simple feature stitching is insufficient to characterize their inherent nonlinear correlations. Therefore, this invention constructs an architecture using Conv2D and BiLSTM as parallel feature encoders and designs an interactive fusion mechanism that gradually transitions feature representation from low-level representation to high-level semantic integration, thereby achieving cross-modal collaboration and dynamic adaptation.

[0091] The PMMFS architecture consists of three stages: Stage 1 (Alignment Projection): Independent fully connected subnetworks with non-linear activation and Dropout are input to each modality's features and projected onto a shared adaptive feature space, achieving intra-modal alignment and scale matching. Stage 2 (Initial Fusion and Cross-Modal Attention): Spatial and temporal feature vectors are concatenated and fused through a fully connected layer. Then, a bidirectional asymmetric cross-modal attention mechanism inspired by Transformer self-attention is introduced. Specifically, the temporal context extracted by BiLSTM is used as the query to reweight the spatial features (Key / Value) of Conv2D, enabling the model to dynamically highlight spatial responses relevant to the current temporal dynamics. Simultaneously, spatial features are used to enhance temporal representations, establishing bidirectional intermodal perception. This cross-modal attention interaction explicitly models the interdependencies of heterogeneous contexts. The third stage (deep integration and regression) involves concatenating the initially fused features with the cross-attention output and inputting them into a shared fully connected layer for deep integration and dimensionality reduction. Finally, the output layer performs regression prediction on the target wave parameters, specifically including the feature period and meaningful wave height. Based on the aforementioned multimodal semantic representation, the inversion results are obtained, as shown in Tables 1 and 2. Figure 10 , Figure 11 As shown.

[0092] Table 1 Comparison of characteristic periodicity inversion results

[0093]

[0094] Table 2 Comparison of inversion results for meaningful waves

[0095]

[0096] The complete formula for this multimodal fusion network is expressed as follows:

[0097] S1, Data preprocessing and input definition;

[0098] S1.1 SAR Image Input: This is a single-polarization spaceborne SAR sub-image cropped with the ship's position as the center, normalized to 0-1.

[0099] S1.2, Motion Timing Input: , is the sequence of three-degree-of-freedom ship motion after outlier processing.

[0100] Target vector: (Yoshibo Takashi) With characteristic wave period ).

[0101] S2, Radar Image Branch (Conv2D + Spatial / Channel Attention)

[0102] S2.1 Extracting image features through three-segment convolution:

[0103] ;

[0104] ;

[0105] In the formula, For the first Feature maps of convolutional blocks For convolution and batch normalization combination, Input SAR image;

[0106] S2.2, Channel Attention (CA):

[0107] ;

[0108] In the formula, Channel attention weights (same dimension as the number of channels). For the Sigmoid function, It is a fully connected layer. For global average pooling, For global max pooling, This is a characteristic of the current stage.

[0109] S2.3 Spatial Attention (SA):

[0110] ;

[0111] In the formula, Spatial attention weights (corresponding to spatial dimensions). For convolution, For tensor splicing, For averaging operations on the channel dimension, For the maximum operation on the channel dimension;

[0112] S2.4, Enhanced attention features:

[0113] ;

[0114] In the formula, The enhanced features, weighted by channel and spatial attention, are used for subsequent branch outputs or fusion. Element-wise multiplication;

[0115] S3, Temporal Data Branch (Bidirectional LSTM + Temporal Attention)

[0116] S3.1, Bidirectional LSTM Processing

[0117] ;

[0118] In the formula, It is a bidirectional LSTM (hidden_size=128×bidirectional→output dimension 256). Output the sequence for each time step. These represent the final hidden state and the cell state, respectively.

[0119] S3.2, Temporal Attention (TA)

[0120] ;

[0121] In the formula, Importance weights for each time step (along) (normalization) ; To utilize Normalize the function. For linear layers, Activated for hyperbolic tangent;

[0122] S3.3 Weighted Feature Aggregation

[0123] ;

[0124] In the formula, This is the aggregated temporal feature vector (256 dimensions). For the first Weight of each time step For the first The hidden state is set at each time step.

[0125] S4, Progressive Fusion Module

[0126] S4.1 Independent Feature Processing (Feature Alignment + Regularization)

[0127] ;

[0128] ;

[0129] In the formula, The image enhancement features are represented in one dimension (256) after GAP+FC. For the two-modal features mapped to the shared space, For random inactivation.

[0130] S4.2, Preliminary Fusion

[0131] ;

[0132] ;

[0133] In the formula, For splicing features, Preliminary fusion results (linear layer output dimension 384).

[0134] S4.3, Cross-modal attention enhancement

[0135] ;

[0136] ;

[0137] In the formula, , , , , , , For learnable projection matrices, For key / query dimensions (e.g., 64); As learnable scaling factors, the residual terms employ their respective projection characteristics to maintain numerically stable outputs. .

[0138] S4.4, Fusion and Output

[0139] ;

[0140] ;

[0141] ;

[0142] ;

[0143] In the formula, , For the final predicted output, corresponding .

[0144] The formulas for calculating the evaluation indicators are as follows:

[0145] ;

[0146] ;

[0147] ;

[0148] ;

[0149] ;

[0150] ;

[0151] In the formula, It is the actual value. This is a predicted value.

[0152] Example 2: The multimodal deep learning fusion ocean remote sensing wave parameter inversion system provided in this embodiment of the invention includes:

[0153] The data acquisition module is used to acquire ship motion history data and spaceborne SAR image data;

[0154] The data preprocessing module is used to preprocess the ship motion history data and the SAR image data to construct spatiotemporally aligned multimodal input data;

[0155] The model calculation module has a built-in pre-trained wave parameter inversion model, which is used to receive the multimodal input data and output wave parameter inversion values.

[0156] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0157] To further demonstrate the positive effects of the above embodiments, the present invention conducts the following experiments based on the above technical solutions.

[0158] Tables 3 and 4 show a comparison of the performance of different models in retrieving characteristic periods and meaningful wave heights.

[0159] Table 3 Comparison of performance of different models in retrieving characteristic periods

[0160]

[0161] Table 4 Comparison of performance of different models in inversion meaningful waves

[0162]

[0163] The fusion model was compared with other single-modal intelligent models and common fusion models in other fields to obtain a radar chart of performance indicators, as shown below. Figure 8 , Figure 9 As shown.

[0164] The results above demonstrate that the ocean remote sensing wave parameter inversion method based on attention mechanism multimodal deep learning fusion outperforms other neural network models across all evaluation metrics, fully proving its high accuracy and effectiveness in wave field statistical parameter inversion. By fusing multimodal data, this neural network model can integrate information from different sources, significantly improving its robustness and outperforming other methods in handling complex sea conditions.

[0165] This method not only improves the accuracy and real-time performance of inversion, but also innovatively enhances the adaptability to changing sea conditions by flexibly fusing multi-source heterogeneous data from a spatiotemporal dimension, providing a new technical route for high-resolution wave parameter inversion.

[0166] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention and within the spirit and principles of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A method for retrieving ocean wave parameters from marine remote sensing data by multi-modal deep learning fusion, characterized in that, The method comprises the following steps: S1, read the collected ship three-degree-of-freedom motion time history data, obtain motion time sequence T n ={T1, T2,...T n-1 , T n} and obtain SAR image data under corresponding working conditions; use attention mechanism to realize adaptive alignment and complementarity of heterogeneous information in space-time two dimensions, attention module automatically allocates weight according to task correlation, and uses ship motion and SAR image to carry out sea wave parameter fusion inversion; S2, pre-processing the three-degree-of-freedom motion time history data of the ship and the SAR image data, and constructing spatio-temporally aligned multi-modal input data; S3, inputting the pre-processed multi-modal input data into a pre-trained sea wave parameter inversion model to calculate the inversion value of the sea wave parameter; the sea wave parameter inversion model is a deep learning fusion network model, which is used to extract time sequence features from the three-degree-of-freedom motion time history data of the ship and spatial features from the SAR image sub-region, and fuse the time sequence features and the spatial features; wherein the SAR image sub-region is a square region obtained by cutting based on the ship position; In step S3, the multi-modal input data is calculated by using a CNN-BiLSTM fusion framework, the CNN module is used to automatically learn and extract discriminative spatial or local hierarchical features from the high-dimensional radar image, the BiLSTM module is used to effectively capture the time sequence dependence and context information in the ship motion response, and a plurality of attention mechanisms are used to optimize the features and improve the focusing ability of the model, and the wave parameter is inverted; In step S3, the sea wave parameter inversion model adopts a heterogeneous dual-flow parallel structure, which fully extracts modal features in the image domain and the time sequence domain respectively, avoids feature aliasing and modal interference, retains the spatial texture and periodic structure in the radar image, and captures the key fluctuations and dynamic changes in the ship response; comprising: a radar image branch adopting a two-dimensional convolutional neural network Conv2D structure for extracting spatial visual features from the SAR image sub-region; a ship motion branch adopting a stacked bidirectional long short-term memory network BiLSTM structure for extracting time sequence dynamic features from the three-degree-of-freedom motion time history data of the ship; a feature fusion module adopting a progressive multi-modal fusion strategy PMMFS for fusing the spatial visual features output by the radar image branch and the time sequence dynamic features output by the ship motion branch; The progressive multi-modal fusion strategy PMMFS comprises: feature alignment projection: projecting the spatial visual features and the time sequence dynamic features into a shared adaptive feature space through a fully connected layer respectively to realize intra-modal alignment and scale matching; preliminary fusion and cross-modal attention: concatenating and preliminarily fusing the projected features, introducing a bidirectional asymmetric cross-modal attention mechanism, taking the time sequence dynamic features as a query Query to re-weight the spatial visual features, and / or taking the spatial visual features as a query Query to re-weight the time sequence dynamic features; deep integration and regression: inputting the cross-modal attention enhanced features and the preliminarily fused features into a shared fully connected layer for integration, and outputting the final sea wave parameter inversion value through a regression layer.

2. The method according to claim 1, wherein, In step S2, the preprocessing includes: first, selecting a spaceborne SAR image imaging with sea conditions, taking the center imaging time of the SAR image as t sar Then, a motion sequence window with a time length of L = 1200 s is intercepted before and after the time point The ship position at t sar The center of the image is cut out as a 600m×600m square region, ensuring spatial alignment, thereby completing the construction of multi-modal input data.

3. The method according to claim 1, wherein, The radar image branch also embeds an attention module for enhancing the feature map extracted by the CNN, and the attention module comprises a spatial attention SA submodule and a channel attention CA submodule; The spatial attention SA submodule generates a normalized attention mask in the spatial dimension, dynamically emphasizing high-energy wave packets and significant peak regions. The channel attention CA submodule explicitly models cross-channel dependencies through global pooling and a lightweight MLP, assigning independent modulation weights to each channel. The attention refinement of the convolution output feature map F is formalized as: M s (F) = σ(f k×k [AvgPool s (F), MaxPool s (F)]) M c (F sa ) = σ(W2δ(W1[AvgPool c (F sa ), MaxPool c (F sa )])) where σ() is a Sigmoid function, f k×k is a k x k convolution, AvgPool s () is a global average pooling with empty dimension, MaxPool s () is a global max pooling with empty dimension, AvgPool c () is a global average pooling with full dimension, MaxPool c () is a global max pooling with full dimension, is element-wise multiplication, δ() is a ReLU function, F sa is a feature map after applying spatial attention, M s (F) is an output after spatial attention weights the feature map, M c (F sa ) is an output after channel attention further weights, W1 is a weight matrix in channel attention CA, W2 is a weight matrix in the second stage of attention mechanism, F ca+sa is a feature map after applying spatial attention and channel attention, F is an original feature map output by a convolutional layer, F ∈ R C×H×W ; The image branch outputs a 256-dimensional compact vector through global pooling and fully connected projection, serving as a single-modal spatial representation for subsequent multi-modal fusion.

4. The method according to claim 3, wherein, The spatial attention SA submodule and the channel attention CA submodule work in series, first performing spatial attention weighting and then channel attention weighting.

5. The method according to claim 1, wherein, The ship motion branch is also connected with a time attention TA module after the network BiLSTM structure, which is used to weight and aggregate the hidden states of all time steps output by the BiLSTM, obtaining a fixed-length time sequence feature vector. At time step t, the input sequence S = [s1,..., S T ], the output of the network BiLSTM structure is: where h t is the output hidden state at time step t, concat() is a concatenation operation on vectors, is the hidden state of the network processing the input sequence forward, is the hidden state of the network processing the input sequence backward. where LSTM fwd () is the forward LSTM computation process, LSTM bwd () is the backward LSTM computation process, s t is the input data at the t-th time step, is the hidden state of the forward LSTM at time step t-1, is the hidden state of the backward LSTM at time step t+1.

6. The method according to claim 1, wherein, The sea wave parameters include significant wave height and characteristic period.

7. A multi-modal deep learning fusion ocean remote sensing sea wave parameter inversion system, characterized in that, The system is used to regulate the multi-modal deep learning fusion ocean remote sensing sea wave parameter inversion method according to any one of claims 1-6, and the system comprises: a data acquisition module for acquiring ship motion time history data and satellite-borne SAR image data; a data preprocessing module for preprocessing the ship motion time history data and the SAR image data to construct multi-modal input data with spatio-temporal alignment; a model calculation module with a pre-trained sea wave parameter inversion model for receiving the multi-modal input data and outputting the sea wave parameter inversion value.

Images