Multi-modal deep learning fusion ocean remote sensing sea wave parameter inversion method and system

Abstract

The invention belongs to the technical field of marine environment monitoring, and discloses a multi-mode deep learning fused marine remote sensing sea wave parameter inversion method and system, and the method comprises the steps: obtaining three-degree-of-freedom motion time calendar data of a ship in a target region and spaceborne synthetic aperture radar (SAR) image data; preprocessing ship motion time calendar data and the SAR image data, and constructing time-space aligned multi-modal input data; and inputting the preprocessed multi-modal input data into a pre-trained sea wave parameter inversion model, and calculating to obtain an inversion value of the sea wave parameter. According to the method, an attention mechanism is innovatively used to realize adaptive alignment and complementation of heterogeneous information in two dimensions of space and time, and an attention module automatically allocates weights according to task correlation; through time-space collaborative feature enhancement, the model can stably extract characterization strongly related to target parameters even under complex sea conditions, so that the inversion precision and robustness of sea wave parameters are remarkably improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of marine environment monitoring, and particularly relates to a multi-modal deep learning fusion marine remote sensing sea wave parameter inversion method and system. BACKGROUND

[0002] Precise wave parameter inversion is of great significance to ensure offshore operations. Due to the strong nonlinear characteristics of sea waves and sensor measurement errors, precise wave parameter perception is a challenge. Researchers in the field use various sensors for marine environment perception, such as ship-borne radars, buoys, satellite sensors, and marine weather stations.

[0003] The existing patent CN202410859719.4 discloses a ship motion prediction method based on a recurrent Transformer model. The ship motion time series data is preprocessed, and the preprocessed ship motion time series data is segmented and blocked. Based on the ship motion prediction model of the recurrent Transformer, the block data output embedding vector sequence is obtained. Finally, the prediction values of the ship motion roll angle, the prediction values of the three-degree-of-freedom angle, and the prediction values of the heave height are obtained to predict the ship motion. However, this scheme uses a single ship motion sequence as input to predict future ship motion using a recurrent / Transformer. The goal is to extrapolate the time series of motion trajectories / poses, and it does not process SAR images or output sea wave statistical parameters.

[0004] The existing patent CN202111604121.3 discloses a sea wave parameter inversion method. In this scheme, an inversion sea wave parameter model capable of inverting sea wave parameters is pre-trained. The first wind data set, the first temperature, and the first inertial navigation data set representing the sea wave parameters of the sea area measured by the offshore floating wind laser radar within the first preset time length are input into the pre-trained inversion sea wave parameter model as input items, and the sea wave parameters of the sea area within the first preset time length are obtained. However, this scheme relies on the integration of the offshore floating wind laser radar and the sea wave buoy sensor, and uses the collected wind data set, temperature data set, and inertial navigation data set to invert the sea wave parameters. It does not use real-time ship motion and SAR for spatio-temporal complementation.

[0005] Currently, in the study of wave parameter inversion using synthetic aperture radar (SAR), deep learning methods have gradually become an important means of wave field parameter inversion due to their efficient real-time processing capability, powerful nonlinear modeling capability, and feature extraction capability. Based on the collected sample data, the deep neural network model is fully trained to extract the wave texture features in the spaceborne SAR image, thereby realizing accurate inversion of the wave field parameters. The existing technology has the following problems and defects: (1) The robustness of the existing single-modal data-driven method is insufficient, the large-scale model is prone to overfitting due to the lack of data features, and the robustness is poor when not fully regularized, and the adaptability is poor.

[0006] (2) The existing data fusion method lacks real-time and high precision, relies on the complexity of the environment setting, is difficult to respond to dynamic sea state changes in real time, and cannot meet the timeliness requirements in actual applications.

[0007] (3) After preliminary search of public literature and patents, no technical solution is found for feature-level fusion of ship motion response data and space-borne SAR image to carry out wave field parameter inversion. SUMMARY

[0008] To overcome the problems in the related art, the present application discloses a multi-modal deep learning fusion ocean remote sensing sea wave parameter inversion method and system, specifically relates to combining space-borne SAR image and ship motion data, using convolutional neural network (CNN) and bidirectional long short-term memory network (BiLSTM) for feature extraction, and using attention mechanism and progressive fusion strategy to invert significant wave height and characteristic period. The technical solution is as follows: The present application is implemented as follows: a multi-modal deep learning fusion ocean remote sensing sea wave parameter inversion method, comprising the following steps: S1, reading the collected ship three-degree-of-freedom motion time history data to obtain the motion time sequence , and obtaining the SAR image data under the corresponding working condition; using the attention mechanism to realize the adaptive alignment and complementation of heterogeneous information in space-time two dimensions, the attention module automatically allocates weights according to the task correlation, and uses the ship motion and SAR image to perform sea wave parameter fusion inversion; S2, preprocessing the three-degree-of-freedom motion time history data of the ship and the SAR image data to construct multi-modal input data aligned in time and space; S3, inputting the preprocessed multi-modal input data into a pre-trained sea wave parameter inversion model to calculate the inversion value of the sea wave parameter; the sea wave parameter inversion model is a fusion network model based on deep learning, which is used to extract time sequence features from the ship three-degree-of-freedom motion time history data, and extract spatial features from SAR image sub-regions, and fuse the time sequence features and spatial features; wherein the SAR image sub-region is a square region obtained by cropping based on the ship position.

[0009] In step S2, the preprocessing includes: first, selecting a space-borne SAR image of imaging and sea state, taking the center imaging time of the SAR image as , then taking a motion sequence window with a time sequence length of before and after the time point , the ship position at the moment is taken as the image cropping center to crop a square region of , and spatial alignment is ensured to complete the construction of the multi-modal input data.

[0010] In step S3, the multi-modal input data is calculated by using a CNN-BiLSTM fusion framework, the CNN module is used to automatically learn and extract discriminative spatial or local level features from high-dimensional radar images, the BiLSTM module is used to effectively capture the time sequence dependence and context information in the ship motion response, and a plurality of attention mechanisms are used to optimize the features and improve the focusing ability of the model, and the wave parameters are obtained by inversion.

[0011] In step S3, the sea wave parameter inversion model adopts a heterogeneous dual-flow parallel structure, which fully extracts modal features in the image domain and the time sequence domain respectively, avoids feature aliasing and modal interference, retains the spatial texture and periodic structure in the radar image, and captures the key fluctuations and dynamic changes in the ship response; including: The radar image branch adopts a two-dimensional convolutional neural network Conv2D structure, which is used to extract spatial visual features from the SAR image sub-region; The ship motion branch adopts a stacked bidirectional long short-term memory network BiLSTM structure, which is used to extract time sequence dynamic features from the ship three-degree-of-freedom motion time history data; The feature fusion module adopts a progressive multi-modal fusion strategy PMMFS, which is used to fuse the spatial visual features output by the radar image branch and the time sequence dynamic features output by the ship motion branch.

[0012] Further, the radar image branch also embeds an attention module for enhancing the feature map extracted by CNN, and the attention module includes a spatial attention SA submodule and a channel attention CA submodule; The spatial attention SA submodule generates a normalized attention mask in the spatial dimension first, and dynamically emphasizes the high-energy wave packet and significant wave peak region; the channel attention CA submodule explicitly models the cross-channel dependence through global pooling and lightweight MLP, and assigns independent modulation weights to each channel; The attention refinement of the convolution output feature map is formalized as: ; ; ; ; wherein, is a Sigmoid function,​ For Convolution, Global average pooling for empty dimension, Global max pooling for empty dimension, Global average pooling for channel dimension, Global max pooling for channel dimension, Element-wise multiplication, ReLU function, Feature map after applying spatial attention, Output after weighting feature map by spatial attention, Output after further weighting by channel attention, Weight matrix in channel attention CA, Weight matrix in the second stage of attention mechanism, Feature map after applying spatial attention and channel attention, Original feature map output by convolution layer, ; The image branch is globally pooled and projected by fully connected layer to output a 256-dimensional compact vector as the single-modal spatial representation for subsequent multi-modal fusion.

[0013] Further, the spatial attention SA submodule and the channel attention CA submodule work in series, first performing spatial attention weighting, and then performing channel attention weighting.

[0014] Further, the ship motion branch is also connected with a time attention TA module after the network BiLSTM structure, which is used to weight and aggregate the hidden states of all time steps output by BiLSTM to obtain a fixed-length time sequence feature vector; At time step , the input sequence , the output of the network BiLSTM structure is: ; In the formula, is the output hidden state of time step , is the concatenation operation on vectors, is the hidden state of the network for processing the input sequence forwardly, is the hidden state of the network for processing the input sequence reversely; ; In the formula, is the calculation process of forward , is the calculation process of reverse , is the first , is the second , is the third​ input data of a time step, is a forward hidden state of a time step, hidden state of a time step, is a reverse hidden state of a time step.

[0015] Further, the progressive multi-modal fusion strategy PMMFS comprises: feature alignment projection: projecting the spatial visual features and the temporal dynamic features into a shared adaptive feature space through full connection layers respectively, realizing intra-modal alignment and scale matching; preliminary fusion and cross-modal attention: splicing and preliminarily fusing the projected features, introducing a bidirectional asymmetric cross-modal attention mechanism, taking the temporal dynamic features as a query Query to re-weight the spatial visual features, and / or taking the spatial visual features as a query Query to re-weight the temporal dynamic features; deep integration and regression: inputting the cross-modal attention enhanced features and the preliminarily fused features into a shared full connection layer for integration, and outputting the final sea wave parameter inversion value through a regression layer.

[0016] Further, the sea wave parameters comprise significant wave height and characteristic period.

[0017] Another purpose of the present application is to provide a multi-modal deep learning fusion ocean remote sensing sea wave parameter inversion system, which is used for regulating and controlling the multi-modal deep learning fusion ocean remote sensing sea wave parameter inversion method, and comprises: a data acquisition module for acquiring ship motion time history data and satellite-borne SAR image data; a data preprocessing module for preprocessing the ship motion time history data and the SAR image data, and constructing multi-modal input data with space-time alignment; a model calculation module with a pre-trained sea wave parameter inversion model built in, for receiving the multi-modal input data and outputting sea wave parameter inversion values.

[0018] In combination with all the above technical solutions, the present application has the following beneficial effects: ​Firstly, the application proposes a network framework for fusing satellite-borne SAR images and ship motion response data and performing high-precision inversion of wave parameters. The method innovatively uses an attention mechanism to realize adaptive alignment and complementarity of heterogeneous information in the space-time two dimensions. The attention module automatically allocates weights according to task relevance: in the spatial dimension, it highlights the significant areas related to wave texture, wave ridge lines, etc.; in the time dimension, it focuses on key segments such as peaks and valleys, periodicity, and trend inflection points, while suppressing false features and noise. Through this spatio-temporal collaborative feature enhancement, the model can stably extract representations strongly related to target parameters even in complex sea conditions, thereby significantly improving the inversion accuracy and robustness of sea wave parameters and having good cross-scene generalization ability.

[0019] The experimental results show that the multi-modal deep learning fusion ocean remote sensing sea wave parameter inversion method based on the attention mechanism is superior to other neural network models in all evaluation indicators, fully proving the high precision and effectiveness of the method in wave field statistical parameter inversion; by fusing multi-modal data, the neural network model can integrate information from different sources, thereby significantly improving the robustness of the model and its performance in complex sea conditions, far exceeding other methods. This method not only improves the inversion accuracy and real-time performance, but also innovatively integrates multi-source heterogeneous data from the time and space dimensions, improving the adaptability to variable sea conditions.

[0020] Secondly, the application improves the wave parameter inversion accuracy; through the multi-modal fusion method based on the attention mechanism, it effectively integrates multi-modal information from ship motion response data and satellite-borne SAR images in the time and space dimensions, and can accurately invert key statistical parameters of the sea wave field, such as characteristic period and significant wave height. Compared with traditional single-modal methods, the fusion of multiple data sources enables the model to capture more comprehensive wave features, thereby significantly improving the inversion accuracy.

[0021] The application enhances the robustness of the model; through the fusion of multi-modal data, the application can fully utilize the complementarity of data from different sources and overcome the limitations of a single data source. In the face of environmental factors such as changes in sea conditions and fluctuations in wind speed, the model of the application can maintain stable performance, has strong adaptability and robustness, and avoids the problem of unstable inversion accuracy caused by changes in data in traditional methods.

[0022] The application is suitable for complex sea conditions and has wide applicability; the application not only adapts to common sea conditions, but also effectively handles wave inversion tasks in complex sea conditions. Compared with traditional methods that are mainly applicable to single sea conditions or limited sea conditions, the multi-modal fusion characteristics of this method make it more adaptable and can maintain high-precision inversion results in various dynamic sea conditions.

[0023] Third, based on the edge deployment of the sea wave parameter real-time inversion system of the fusion of the satellite-borne SAR and the ship motion, the sea wave parameter real-time inversion is provided, which directly supports the route optimization, the sea work window evaluation, the offshore wind power maintenance scheduling and the maritime emergency command and the like. The continuous maintenance of a large number of buoys and shore-based radars is got rid of, and the sea wave products are produced relying on the on-orbit satellite and the in-sea ship fleet, so that the unit sea area observation cost is significantly reduced.

[0024] Fourth, the satellite-borne SAR image and the measured ship motion response are first fused at the feature level for wave statistical parameter inversion. The heterogeneous dual-flow structure of Conv2D spatial branch and BiLSTM time sequence branch and the PMMFS progressive multi-modal fusion are proposed, and the spatial, channel and time attention is superimposed to realize the spatio-temporal adaptive alignment and complementation. Through the fusion of the satellite-borne SAR and the ship motion, reliable inversion is realized under weak synchronization conditions, and the dependence on fixed observation arrays is got rid of. The experiments show that the feature-level cross-modal fusion has advantages in precision and robustness. The progressive fusion of heterogeneous dual-flow and cross-modal attention realizes collaborative learning in a unified representation space and brings high-quality inversion effect. BRIEF DESCRIPTION OF DRAWINGS

[0025] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure; Figure 1 is a multi-modal deep learning fusion ocean remote sensing sea wave parameter inversion method flowchart provided by the embodiment of the present application; Figure 2 is a motion time history data schematic diagram provided by the embodiment of the present application; wherein (a) is a heave diagram, (b) is a three-degree-of-freedom diagram, and (c) is a roll diagram; Figure 3 is a satellite-borne SAR image provided by the embodiment of the present application; Figure 4 is a wave height schematic diagram of wave time history data provided by the embodiment of the present application; Figure 5 is a radar image cutting schematic diagram provided by the embodiment of the present application; Figure 6 is a fusion network structure diagram provided by the embodiment of the present application; Figure 7 is a visualization effect diagram of the first CA module acting on 256 channels provided by the embodiment of the present application; Figure 8 is a feature period evaluation index radar diagram provided by the embodiment of the present application; Figure 9 is a significant wave height evaluation index radar diagram provided by the embodiment of the present application; Figure 10This is the weight graph of the temporal attention module provided in the embodiments of the present invention; Figure 11 This is a heatmap of the spatial attention module provided in an embodiment of the present invention. Detailed Implementation

[0026] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0027] The innovation of this invention lies in its pioneering use of ship motion and SAR images for wave parameter fusion and inversion. It innovatively designs a multimodal fusion framework for the extraction and fusion of spatiotemporal information, significantly improving inversion accuracy. This invention employs an attention-based multimodal fusion method, effectively integrating multi-source heterogeneous information from spaceborne SAR images and ship motion response data across spatiotemporal dimensions to achieve information complementarity and accurate wave parameter inversion. This method overcomes the limitations of traditional single-modal data processing, utilizing spatial attention, channel attention, and temporal attention mechanisms to weightedly fuse different modal data, thereby improving the model's robustness and accuracy. Through this innovative data fusion approach, real-time high-resolution wave parameter inversion is achieved, exhibiting excellent adaptability and stability, especially under complex sea conditions.

[0028] This invention integrates spaceborne SAR and ship motion data for wave parameter inversion, fusing multi-source heterogeneous data in both temporal and spatial dimensions to achieve information complementarity. It designs a multimodal fusion network based on an attention mechanism, utilizing spatial, channel, and temporal attention mechanisms to adaptively weight and optimize different modal data. Through a dual-branch network structure (combining Conv2D and BiLSTM) and a progressive multimodal fusion strategy (PMMFS), it achieves efficient wave parameter inversion, making it particularly suitable for real-time applications under dynamic sea conditions.

[0029] Example 1, such as Figures 1-6 As shown in the figure, the multimodal deep learning fusion method for inverting ocean remote sensing wave parameters provided in this embodiment of the invention includes the following steps: S1, read the collected three-degree-of-freedom motion time history data of the ship to obtain the motion time sequence. And acquire SAR image data under the corresponding working conditions; use attention mechanism to achieve adaptive alignment and complementarity of heterogeneous information in the spatial and temporal dimensions. The attention module automatically assigns weights according to the task relevance and uses ship motion and SAR images to perform wave parameter fusion and inversion. S2, preprocessing the three-degree-of-freedom motion time history data of the ship and the SAR image data, and constructing spatio-temporally aligned multi-modal input data; S3, inputting the preprocessed multi-modal input data into a pre-trained sea wave parameter inversion model to calculate the inversion value of the sea wave parameter; the sea wave parameter inversion model is a deep learning-based fusion network model for extracting time sequence features from the three-degree-of-freedom motion time history data of the ship and spatial features from the SAR image sub-region, and fusing the time sequence features and the spatial features; wherein the SAR image sub-region is a square region obtained by cutting based on the ship position.

[0030] In step S2, the preprocessing includes: first selecting a spaceborne SAR image of sea state imaging, taking the center imaging time of the SAR image as Then, a motion sequence window with a time sequence length of is intercepted before and after the time point The ship position at time is used as the image cropping center to crop a square region with a size of to ensure spatial alignment, thereby completing the construction of multi-modal input data.

[0031] The image branch uses a two-dimensional convolutional neural network (Conv2D, a variant of CNN) to learn the representation of spaceborne SAR images, extracting texture, shape, and hierarchical spatial structure, and other visual patterns. This branch has equivariance to translation and other visual elements, and can construct multi-level representations from low-order statistics to high-order semantics. The goal is to decode the potential spatial structure that is strongly related to wave statistical / dynamic characteristics in sea clutter images containing multiplicative speckle and nonlinear components.

[0032] Follow the hierarchical abstraction idea: shallow convolutional units (smaller receptive field) prefer to capture high-frequency visual primitives (convolutional layers use smaller receptive fields to process local regions of the image, which allows the network to focus on changes in detail in the image, such as the luminance gradient of ripples, wave crests and troughs. Since these high-frequency features usually manifest as edges, textures and small-scale fluctuations in the image, shallow convolution can extract discriminative visual primitives by responding sensitively to these local features); as the network deepens, deep convolution expands the receptive field exponentially through cascading convolution and downsampling, thus encoding larger-scale spatial information and being able to represent more abstract and macroscopic wave field patterns (such as global periodicity and anisotropy). With the increase of network levels, deep convolution gradually fuses local information into higher-order abstract features through multiple convolution and pooling operations, and can capture global wave periodicity and anisotropy and other macroscopic features. This gradually expanding receptive field allows the network to effectively learn features of different scales from local ripples to global wave structures, thereby improving the accuracy and stability of wave parameter inversion.

[0033] To improve the feature extraction effect, the embodiment starts from a lightweight baseline and empirically optimizes the network depth and convolution kernel settings. Considering that the "bright ridge" in the sea clutter image often corresponds to the coherent structure related to the wave crest, a spatial attention (SA) and channel attention (CA) module is concatenated after each convolution block to form a robust feature post-processing unit. SA first generates a normalized attention mask in the spatial dimension, dynamically emphasizing high-energy wave packets and significant wave crest regions; then CA explicitly models cross-channel dependencies through global pooling and lightweight MLP, assigning independent modulation weights to each channel.

[0034] The attention refinement of the convolution output feature map can be formalized as: ; ; ; ; In the formula, is a Sigmoid function, is a convolution, is a global average pooling in spatial dimension, is a global max pooling in spatial dimension, is a global average pooling in channel dimension, is a global max pooling in channel dimension, is element-wise multiplication, is a ReLU function, is a feature map after applying spatial attention, the output after weighting the feature map by spatial attention, the output after further weighting by channel attention, the weight matrix in channel attention (CA), the weight matrix in the second stage of attention mechanism, the feature map after applying spatial attention and channel attention, the original feature map output by the convolutional layer, ; Finally, the image branch is globally pooled and projected by fully connected layers to output a 256-dimensional compact vector as the single-modal spatial representation for subsequent multi-modal fusion. The attention visualization effect is shown in Figure 7 .

[0035] The long short-term memory network (LSTM) overcomes the gradient vanishing / explosion problem of traditional RNN when processing long sequences through input / forgetting / output gates. In ship motion response, there is a bidirectional time dependence between force and response. To make full use of the context, the embodiment uses a stacked bidirectional LSTM (BiLSTM) as the core time feature extractor, which models both short-term and long-term dynamics.

[0036] The network consists of a hierarchical structure composed of two consecutive BiLSTMs: the shallow layer focuses on high-frequency local motion patterns (such as short-wave-induced rapid oscillations), and the deep layer aggregates and identifies more wide-area, long-term dynamic trends. At time step , the input sequence , the output of the BiLSTM is: ; wherein, is the output hidden state at time step , is the concatenation operation on vectors, is the hidden state of the network processing the input sequence in the forward direction, is the hidden state of the network processing the input sequence in the reverse direction; ; wherein, is the calculation process in the forward direction, is the calculation process in the reverse direction, is the input data at the th time step, is the hidden state of the forward direction at time step , is the hidden state of the reverse direction at time step .

[0037] ​​​​​​Considering that the measured time series can be as long as 1200 steps, and the key dynamics (such as peaks, troughs, and trend inflection points) need to be explicitly emphasized, this embodiment adds a time attention (TA) module after the BiLSTM to weight and converge the hidden state sequence to obtain a fixed-length context vector; then, a full connection and Dropout are performed to obtain a compact time representation with the same dimension (256 dimensions) as the image branch. The channel attention mechanism can explicitly model the dependency between each channel of the feature map, and learn the importance weight of each channel during the process, to adaptively enhance the feature channels with large information and suppress the channels with less contribution.

[0038] To realize effective fusion of heterogeneous spatio-temporal data, the present application proposes a progressive multi-modal fusion strategy (PMMFS) for systematically extracting complementary features from spatial images and time series data. Since heterogeneous modalities are often distributed in different feature manifolds, simple feature concatenation cannot capture their inherent nonlinear correlation. Therefore, the present application constructs an architecture with Conv2D and BiLSTM as parallel feature encoders, and designs an interactive fusion mechanism to gradually transition from low-level representation to high-level semantic integration, thereby realizing cross-modal collaboration and dynamic adaptation.

[0039] The architecture of PMMFS consists of three stages: the first stage (alignment projection): the features of each modality are input into independent fully connected subnetworks with nonlinear activation and Dropout, projected into a shared adaptive feature space to realize intra-modal alignment and scale matching. The second stage (preliminary fusion and cross-modal attention): the spatial and temporal feature vectors are first concatenated and preliminarily fused through a fully connected layer, and then a bidirectional asymmetric cross-modal attention inspired by the Transformer self-attention mechanism is introduced. Among them, the temporal context extracted by BiLSTM is taken as Query, and the spatial features (Key / Value) of Conv2D are reweighted, so that the model can dynamically highlight the spatial response related to the current temporal dynamics; at the same time, the spatial features are used to enhance the temporal representation in reverse, establishing a bidirectional perception between modalities. This cross-modal attention interaction explicitly models the mutual dependence of heterogeneous contexts. The third stage (deep integration and regression): the preliminary fusion features and cross-attention outputs are concatenated again and input into a shared fully connected layer for deep integration and dimension reduction; finally, the output layer is used to regress the target wave parameters, including feature period and significant wave height, and the inversion results are obtained based on the above multi-modal semantic representation, as shown in Tables 1, 2, Figure 10 、 Figure 11

[0040] Table 1 Comparison of feature period inversion results

[0041] Table 2 Comparison of significant wave height inversion results​

[0042] The complete formula of the multi-modal fusion network is as follows: S1, data preprocessing and input definition; S1.1, SAR image input: , which is a monopolar satellite SAR subgraph centered on the ship position and normalized to 0-1.

[0043] S1.2, motion time sequence input: , which is a sequence of three-degree ship motion after abnormal value processing.

[0044] Target vector: (significant wave height and characteristic wave period ).

[0045] S2, radar image branch (Conv2D+ spatial / channel attention) S2.1, extract image features through three convolutional layers: ; ; In the formula, is the feature map of the th convolutional block, is the combination of convolution and batch normalization , is the input SAR image; S2.2, channel attention (CA): ; In the formula, is the channel attention weight (same dimension as the channel number), is the Sigmoid function, is the full connection layer, is the global average pooling, is the global maximum pooling, is the current stage feature.

[0046] S2.3, spatial attention (SA): ; In the formula, is the spatial attention weight (same shape as the spatial dimension), is the convolution, is the tensor concatenation, is the average operation in the channel dimension, is the maximum operation in the channel dimension; S2.4, attention feature enhancement: ; where, are the enhanced features weighted by channel and spatial attention, used for subsequent branch output or fusion; is element-wise multiplication; S3, Temporal data branch (Bi-LSTM + temporal attention) S3.1, Bi-LSTM processing ; where, is the Bi-LSTM (hidden_size=128x bidirectional -> output dimension 256), is the output sequence of time steps, are the final hidden / cell states, respectively.

[0047] S3.2, Temporal attention (TA) ; where, is the importance weight of each time step (along normalized), ; is normalized using function, is a linear layer, is a hyperbolic tangent activation; S3.3, Weighted feature aggregation ; where, is the aggregated temporal feature vector (256 dimensions), is the weight of the time step, is the hidden state of the time step.

[0048] S4, Progressive fusion module S4.1, Independent feature processing (feature alignment + regularization) ; ; where, is the one-dimensional representation of the image enhanced feature after GAP + FC (dimension 256), is the two-modal feature mapped to the shared space, is random dropout.

[0049] S4.2, Preliminary fusion ; ; wherein, is the concatenation feature, is the preliminary fusion result (linear layer output dimension 384).

[0050] S4.3, Cross-modal attention enhancement ; ; wherein, , , , , , , is a learnable projection matrix, is the key / query dimension (e.g. 64); is a learnable scaling coefficient, the residual term adopts the respective projection feature to maintain numerical stability output .

[0051] S4.4, fusion and output ; ; ; ; wherein, , is the final prediction output, corresponding to .

[0052] The evaluation index calculation formula is as follows: ; ; ; ; ; ; wherein, is the true value, is the predicted value.

[0053] Embodiment 2, the multi-modal deep learning fusion ocean remote sensing sea wave parameter inversion system provided by the embodiment of the application comprises: a data acquisition module configured to acquire ship motion time history data and satellite-borne SAR image data; a data preprocessing module, configured to preprocess the ship motion time history data and the SAR image data, and construct spatio-temporally aligned multi-modal input data; a model calculation module, configured to receive the multi-modal input data and output sea wave parameter inversion values.

[0054] In the above embodiments, the description of each embodiment focuses on different aspects. The parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0055] To further prove the positive effects of the above embodiments, the present application based on the above technical solutions carries out the following experiments.

[0056] The performance comparison of different models in inversion of characteristic period and significant wave height is shown in Tables 3 and 4.

[0057] Table 3 Performance comparison of different models in inversion of characteristic period

[0058] Table 4 Performance comparison of different models in inversion of significant wave height

[0059] Using the fusion model, compared with other single-modal intelligent models and other common fusion models in other fields, the index comparison radar chart is as shown in Figure 8 、 Figure 9 .

[0060] As can be seen from the above results, the multi-modal deep learning fusion method of ocean remote sensing sea wave parameter inversion based on attention mechanism is superior to other neural network models in all evaluation indexes, fully proving the high precision and effectiveness of the method in wave field statistical parameter inversion. By fusing multi-modal data, the neural network model can integrate information from different sources, thereby significantly improving the robustness of the model and the performance in complex sea conditions, far exceeding other methods.

[0061] The method not only improves the inversion accuracy and real-time performance, but also innovatively fuses multi-source heterogeneous data from the time and space dimensions, improves the adaptability to variable sea conditions, and provides a new technical route for high-resolution sea wave parameter inversion.

[0062] The above description is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any modification, equivalent replacement and improvement made by those skilled in the art within the technical range disclosed by the present application, as long as it is within the spirit and principles of the present application, should be covered within the protection scope of the present application.

Claims

1. A method for retrieving ocean wave parameters from marine remote sensing data by multi-modal deep learning fusion, characterized in that, The method comprises the following steps: S1, read the collected ship three-degree-of-freedom motion time history data to obtain motion time sequence and obtain SAR image data under corresponding working conditions; use attention mechanism to realize adaptive alignment and complement of heterogeneous information in space-time two dimensions, attention module automatically allocates weight according to task correlation, and uses ship motion and SAR image to perform sea wave parameter fusion inversion; S2, pre-processing the three-degree-of-freedom motion time history data of the ship and the SAR image data, and constructing multi-modal input data aligned in space and time; S3, inputting the pre-processed multi-modal input data into a pre-trained sea wave parameter inversion model to calculate the inversion value of the sea wave parameter; the sea wave parameter inversion model is a fusion network model based on deep learning, which is used to extract time sequence features from the three-degree-of-freedom motion time history data of the ship and spatial features from the SAR image sub-region, and fuse the time sequence features and the spatial features; wherein the SAR image sub-region is a square region obtained by cutting based on the position of the ship.

2. The method according to claim 1, wherein, In step S2, the preprocessing includes: first, selecting a spaceborne SAR image imaging with sea conditions, taking the center imaging time of the SAR image as , then taking a time sequence window with a length of before and after the time point , taking the ship position at time as the image cropping center, cropping a square region with a size of to ensure spatial alignment, thereby completing the construction of multi-modal input data.

3. The method according to claim 1, wherein, In step S3, the multi-modal input data is calculated by using a CNN-BiLSTM fusion framework, the CNN module is used to automatically learn and extract discriminative spatial or local hierarchical features from high-dimensional radar images, the BiLSTM module is used to effectively capture the time sequence dependence and context information in the ship motion response, and a plurality of attention mechanisms are used for feature optimization and improvement of model focusing ability, and the wave parameter is inverted.

4. The method according to claim 3, wherein, In step S3, the sea wave parameter inversion model adopts a heterogeneous double-flow parallel structure, which fully extracts modal features in the image domain and the time sequence domain respectively, avoids feature aliasing and modal interference, retains the spatial texture and periodic structure in the radar image, and captures the key fluctuations and dynamic changes in the ship response; comprising: a radar image branch adopting a two-dimensional convolutional neural network Conv2D structure for extracting spatial visual features from the SAR image sub-region; a ship motion branch adopting a stacked bidirectional long short-term memory network BiLSTM structure for extracting time sequence dynamic features from the three-degree-of-freedom motion time history data of the ship; a feature fusion module adopting a progressive multi-modal fusion strategy PMMFS for fusing the spatial visual features output by the radar image branch and the time sequence dynamic features output by the ship motion branch.

5. The method according to claim 4, wherein, The radar image branch further embeds an attention module for enhancing the feature map extracted by CNN, and the attention module comprises a spatial attention SA submodule and a channel attention CA submodule; The spatial attention SA submodule generates a normalized attention mask in the spatial dimension first, and dynamically emphasizes the high-energy wave packet and the significant wave peak region; The channel attention CA submodule explicitly models the cross-channel dependence through global pooling and lightweight MLP, and assigns independent modulation weights to each channel; Attention refinement of the convolution output feature map is formalized as: ； ； ； ； wherein, sigmoid function, convolution, global average pooling for empty dimension, global max pooling for empty dimension, global average pooling for channel dimension, global max pooling for channel dimension, element-wise multiplication, ReLU function, feature map after applying spatial attention, output after weighting the feature map by spatial attention, output after further weighting by channel attention, weight matrix in channel attention CA, weight matrix in the second stage of attention mechanism, feature map after applying spatial attention and channel attention, original feature map output by the convolutional layer, ;​ The image branch outputs a 256-dimensional compact vector through global pooling and fully connected projection as a single-modal spatial representation for subsequent multi-modal fusion.

6. The method according to claim 5, wherein, The spatial attention SA submodule and the channel attention CA submodule work in a series manner, first performing spatial attention weighting and then performing channel attention weighting.

7. The method according to claim 4, wherein, The ship motion branch is further connected with a time attention TA module after the network BiLSTM structure, which is used to weight and aggregate the hidden states of all time steps output by BiLSTM to obtain a fixed-length time sequence feature vector; At time step , the input sequence , the output of the network BiLSTM structure is: ； wherein is the output hidden state at time step is a concatenation operation on vectors, is the hidden state of the network processing the input sequence forward, is the hidden state of the network processing the input sequence backward;​ ； wherein is the forward computation process, is the backward computation process, is the input data at the time step, is the hidden state at time step in the forward direction, is the hidden state at time step in the backward direction.

8. The method according to claim 4, wherein, The progressive multi-modal fusion strategy PMMFS comprises: Feature alignment projection: the spatial visual features and the temporal dynamic features are projected to a shared adaptive feature space through fully connected layers respectively, to realize intra-modal alignment and scale matching; Preliminary fusion and cross-modal attention: the projected features are spliced and preliminarily fused, and a bidirectional asymmetric cross-modal attention mechanism is introduced, taking the temporal dynamic features as a query (Query) to re-weight the spatial visual features, and / or taking the spatial visual features as a query (Query) to re-weight the temporal dynamic features; Deep integration and regression: the cross-modal attention enhanced features and the preliminarily fused features are input into a shared fully connected layer for integration, and the final sea wave parameter inversion value is output through a regression layer.

9. The method according to claim 8, wherein, The sea wave parameters include significant wave height and characteristic period.

10. A multi-modal deep learning fusion ocean remote sensing sea wave parameter inversion system, characterized in that, The system is used for regulating and controlling the multi-modal deep learning fusion ocean remote sensing sea wave parameter inversion method of any one of claims 1-9, and the system comprises: a data acquisition module for acquiring ship motion time history data and satellite-borne SAR image data; a data preprocessing module for preprocessing the ship motion time history data and the SAR image data, and constructing multi-modal input data for space-time alignment; a model calculation module, which is built-in with a pre-trained sea wave parameter inversion model, is used for receiving the multi-modal input data and outputting sea wave parameter inversion values.

Images