A Multimodal Fusion Prediction Method and System for Marine Environment Based on Digital Twin

Abstract

The present invention relates to the technical field of marine environment prediction, and particularly to a digital-twin-based multi-modal fusion prediction method and system for marine environment. The method includes capturing spatio-temporal correlation features of multi-modal data of the marine environment based on a dynamic multi-modal graph neural network; performing multi-scale feature fusion on the spatio-temporal correlation features by using a multi-scale gated unit to obtain a comprehensive feature representation; predicting the comprehensive feature representation by using a hybrid time series prediction framework to obtain preliminary marine environment prediction data, including short-term dynamic modeling and long-term trend modeling; and fitting noise to the preliminary marine environment prediction data by using a generative adversarial network to generate final marine environment prediction data. Through an innovative combination of dynamic multi-modal data fusion, a hybrid time series prediction framework, and a multi-scale gated Tanh unit, the present invention effectively solves the problems of insufficient data fusion, limited prediction accuracy, and poor dynamic response ability existing in traditional marine environment monitoring and prediction technologies.

Description

Technical Field

[0001] The present invention relates to the technical field of marine environment prediction, and in particular to a multi-modal fusion prediction method and system for marine environment based on digital twin. Background Art

[0002] With the increasing demands for marine resource development and environmental protection, real-time monitoring, dynamic prediction, and emergency response of the marine environment have become the focus of current research. However, traditional marine environment monitoring methods have many deficiencies in dealing with the dynamic characteristics of complex marine environments. First, traditional methods usually rely on a single data source, such as buoys, meteorological stations, or satellite remote sensing. This single mode is difficult to comprehensively capture the multi-dimensional dynamic characteristics of the marine environment, resulting in limited monitoring scope and temporal continuity. Second, in the face of data with significant spatio-temporal variations such as temperature, salinity, flow velocity, and sea waves, existing methods lack in-depth fusion of multi-modal data during data processing and analysis, making it difficult to capture the spatio-temporal correlations between different modal data, resulting in insufficient accuracy in predicting the change trends across time scales. Third, traditional systems often rely on rule models in emergency response. These models are usually based on fixed empirical rules and static data, making it difficult to handle complex non-linear dynamic characteristics and unable to adapt to complex and changing marine environments in real time and efficiently, thus severely restricting the monitoring and decision-making efficiency.

[0003] In recent years, digital twin technology, as an emerging intelligent technology, has provided new possibilities for marine environment monitoring and prediction. Digital twin technology integrates various data sources such as satellite remote sensing, buoys, and underwater sensors into a unified virtual model through multi-modal data fusion to reconstruct and simulate the real marine environment with high fidelity. This technology can not only perceive the marine state in real time but also predict future environmental change trends, effectively reducing the high costs and high risks of physical experiments. However, current digital twin systems still have significant deficiencies in high-precision data fusion, intelligent prediction, and complex ecosystem simulation. Especially in the face of the heterogeneity, real-time nature, and accuracy requirements of multi-source data, existing technologies have not yet formed a unified and efficient solution. For example, when processing multi-modal data, existing technologies lack an effective framework to capture the cross-modal spatio-temporal correlations of multi-source heterogeneous data; when modeling the dynamic characteristics of complex ecosystems, existing prediction models usually cannot take into account both short-term dynamic changes and long-term trend characteristics at the same time. Especially in extreme weather scenarios, the reliability and accuracy of prediction results are still relatively low. In addition, current digital twin technology is still insufficient in result display and interactivity. Existing visualization means are difficult to intuitively present the dynamic changes of complex environments and also difficult to meet the flexible analysis requirements in practical applications.

[0004] Meanwhile, when traditional methods deal with multi-source heterogeneous data (such as satellite remote sensing, buoy sensor, and weather station data), due to differences in data formats, resolutions, and time synchronization, it is difficult to achieve efficient integration and spatio-temporal correlation modeling, resulting in limited monitoring accuracy and insufficient ability to capture dynamic changes. In addition, existing methods lack the ability to jointly model short-term dynamics and long-term trends when dealing with multi-scale changes in complex marine ecosystems. Especially in the prediction of extreme events (such as typhoons and tsunamis), it is difficult to provide reliable and accurate results.

[0005] At the data fusion level, current methods mostly adopt simple weighted average or linear integration when dealing with the heterogeneity and spatio-temporal dynamic characteristics of multi-modal data, and cannot effectively capture complex cross-modal and cross-spatio-temporal dependencies, restricting the comprehensive expression of multi-dimensional dynamic features. At the same time, in multi-scale modeling, traditional time series models (such as ARIMA or single deep learning models) only focus on single-time scale features, ignoring the coupling relationship between short-term drastic changes and long-term stable trends, resulting in low prediction accuracy. Especially in the simulation of extreme weather scenarios, existing methods are difficult to generate realistic abnormal data for scenario deduction. In addition, existing systems mostly use two-dimensional static charts or simple indicators for result display, lacking dynamic interaction capabilities and the intuitiveness of visualization, and it is difficult to support users' efficient analysis and understanding of complex marine environments. Summary of the Invention

[0006] To solve the above-mentioned problems, the present invention provides a digital-twin-based multi-modal fusion prediction method and system for marine environments.

[0007] In a first aspect, a digital-twin-based multi-modal fusion prediction method for marine environments provided by the present invention adopts the following technical solutions:

[0008] A digital-twin-based multi-modal fusion prediction method for marine environments includes:

[0009] Obtain multi-modal data of the marine environment, including satellite remote sensing, buoy sensor, and weather station data;

[0010] Perform data preprocessing on the obtained multi-modal data;

[0011] Capture spatio-temporal correlation features of multi-modal data of the marine environment based on a dynamic multi-modal graph neural network;

[0012] Use a multi-scale gated unit to perform multi-scale feature fusion on the spatio-temporal correlation features to obtain a comprehensive feature representation;

[0013] Use a hybrid time series prediction framework to predict the comprehensive feature representation to obtain preliminary marine environment prediction data, including short-term dynamic modeling and long-term trend modeling;

[0014] The preliminary marine environment prediction data is subjected to noise fitting using a generative adversarial network to generate the final marine environment prediction data.

[0015] Furthermore, the data preprocessing of the obtained multimodal data includes data cleaning, missing value processing, and normalization processing of the obtained multimodal data. Among them, the missing value processing uses the average method to fill in the missing values, and the filling method is set as:

[0016]

[0017] Among them, the missing value is the average value of the past k moments, which is used as the estimated value of the missing data at time t.

[0018] Furthermore, the spatio-temporal correlation features of the multimodal data of the marine environment are captured based on the dynamic multimodal graph neural network, including modeling the multimodal data of the marine environment as a dynamic heterogeneous graph , where the node set represents each modal data source, and the edge set represents the dynamic relationship between the data sources. The adjacency matrix defines the association weights between nodes. After the construction of the dynamic heterogeneous graph is completed, the propagation and update of node features are performed through the graph convolutional network GCN. The update formula for each layer of graph convolutional operation is:

[0019]

[0020] Among them, is the node feature matrix of the l-th layer, is the weight matrix for linear transformation, is the bias vector, and σ(·) is the non-linear activation function.

[0021] Furthermore, the multi-scale feature fusion of the spatio-temporal correlation features is performed using the multi-scale gated unit, including using the multi-scale gated Tanh unit to extract short-term dynamics and long-term trend features from the generated features. Among them, the local features within different time windows are captured through convolutional operations, the gating mechanism is used to generate feature selection weights through the Sigmoid function, and the Tanh activation and non-linear transformation are introduced to enhance the model's representation ability for complex features. The features of different time scales are weighted and fused to generate a comprehensive feature representation:

[0022]

[0023] Among them, K is the number of time scales. α k is the fusion weight of time scale k, which is dynamically learned by the model through training.

[0024] Further, the short-term dynamic modeling includes using dilated convolutions to capture local time series features. For a given input feature Z t , the calculation formula of the dilated convolution is:

[0025]

[0026] where h t is the convolution output at time t, K is the convolution kernel size, d is the dilation factor that controls the time jump step, and w k is the weight of the convolution kernel.

[0027] Further, the long-term trend modeling includes using a Transformer based on the multi-head attention mechanism to model the non-linear dependencies over long time spans. Among them, first, query, key, and value matrices are generated through linear transformations, the weights between feature time steps are calculated based on the attention mechanism, and the results of different attention heads are concatenated through multi-head attention and then linearly transformed to generate long-term feature representations. Finally, the long-term branch maps the long-term feature representation Flong to the target prediction space through a linear layer.

[0028] Further, using a generative adversarial network to perform noise fitting on the preliminary ocean environment prediction data to generate the final ocean environment prediction data includes using the generator of the generative adversarial network to generate abnormal scenario data and learning the distribution of real data through adversarial training to generate abnormal data close to the real environment.

[0029] In a second aspect, a digital-twin-based multi-modal fusion prediction system for ocean environment includes:

[0030] A data acquisition module, configured to acquire multi-modal data of the ocean environment, including satellite remote sensing, buoy sensor, and weather station data;

[0031] A preprocessing module, configured to perform data preprocessing on the acquired multi-modal data;

[0032] A spatio-temporal correlation module, configured to capture spatio-temporal correlation features of the multi-modal data of the ocean environment based on a dynamic multi-modal graph neural network;

[0033] A feature representation module, configured to perform multi-scale feature fusion on the spatio-temporal correlation features using a multi-scale gated unit to obtain a comprehensive feature representation;

[0034] A prediction module, configured to perform prediction on the comprehensive feature representation using a hybrid time series prediction framework to obtain preliminary ocean environment prediction data, including short-term dynamic modeling and long-term trend modeling;

[0035] A fitting module, configured to perform noise fitting on preliminary ocean environment prediction data by using a generative adversarial network to generate final ocean environment prediction data.

[0036] In a third aspect, the present invention provides a computer-readable storage medium, in which multiple instructions are stored, and the instructions are adapted to be loaded and executed by a processor of a terminal device for the described method for digital-twin-based multi-modal fusion prediction of ocean environment.

[0037] In a fourth aspect, the present invention provides a terminal device, including a processor and a computer-readable storage medium. The processor is used to implement each instruction; the computer-readable storage medium is used to store multiple instructions, and the instructions are adapted to be loaded and executed by the processor for the described method for digital-twin-based multi-modal fusion prediction of ocean environment.

[0038] In summary, the present invention has the following beneficial technical effects:

[0039] Through the innovative combination of dynamic multi-modal data fusion, a hybrid time series prediction framework (HTPF), and multi-scale gated Tanh units (M-GTU), the present invention effectively solves the problems of insufficient data fusion, limited prediction accuracy, and poor dynamic response ability existing in traditional ocean environment monitoring and prediction technologies. Based on the real-time acquisition and deep fusion technology of multi-source heterogeneous data of dynamic multi-modal graph neural networks (DM-GNN), it comprehensively integrates multi-modal data sources such as satellite remote sensing, buoy sensors, and weather stations, overcoming the problems of data isolation, heterogeneity, and complex correlation processing in traditional methods.

[0040] Combined with the multi-time scale feature extraction mechanism of M-GTU, the system can accurately capture short-term dynamic changes and long-term trends in the ocean environment, especially showing excellent sensitivity and robustness in the prediction of extreme events such as storms and tides. Through the synergistic effect of the short-term dynamic modeling branch (based on a temporal convolutional network) and the long-term trend modeling branch (based on a multi-head attention mechanism), the system shows excellent performance in both fast-changing event modeling and long-time-span ecological evolution modeling. The introduction of a generative adversarial network (GAN) enables the system to simulate the ocean environment state under extreme scenarios, providing richer scenario analysis support for scientific decision-making. The prediction results achieve the optimal combination of short-term and long-term features through dynamic fusion, significantly improving the prediction accuracy and reliability.

[0041] This system integrates an interactive 3D visualization platform, which intuitively displays the real-time status and future change trends of the marine environment. Users can flexibly adjust the region, time range or input scenario simulation parameters through the interactive interface, and dynamically view the simulation of specific scenarios under complex environments. This intuitive and efficient visualization ability not only supports users to analyze the changes in the marine environment from multiple dimensions, but also provides a scientific basis for optimizing resource management, ecological protection and emergency response strategies. Compared with traditional methods, the present invention has achieved improvements in data processing efficiency, prediction accuracy, scenario simulation ability and visualization interaction experience, providing key technical support for the construction of the smart ocean and having wide application value in the fields of marine resource development, environmental protection and disaster prevention and control.

[0042] The digital twin-based multi-modal fusion prediction system for marine environment proposed by the present invention can be widely applied to the real-time monitoring and future state prediction of complex marine dynamic environments through the deep integration and intelligent analysis of multi-source data. In marine resource management, the present invention can predict the spatial distribution and temporal changes of temperature and salinity by integrating satellite remote sensing, buoy sensor and weather station data, providing data support for fishery resource assessment and planning. In terms of marine disaster management, the present invention can simulate the impact effects of extreme events such as typhoons and tsunamis on specific sea areas in real time, assisting the government and relevant departments to formulate accurate evacuation and disaster emergency plans. In addition, this system combines three-dimensional dynamic visualization technology to support the real-time display and trend analysis of environmental monitoring data, and has important value in public science popularization and scientific research applications. Especially when dealing with complex changes across time scales and analyzing the interactive effects between marine ecosystems and climate, the present invention demonstrates excellent performance and adaptability. Brief Description of the Drawings

[0043] Figure 1 It is a schematic diagram of a digital twin-based multi-modal fusion prediction method for marine environment according to Embodiment 1 of the present invention;

[0044] Figure 2 It is another schematic diagram of a digital twin-based multi-modal fusion prediction method for marine environment according to Embodiment 1 of the present invention. Detailed Description of the Invention

[0045] The present invention will be further described in detail below with reference to the accompanying drawings.

[0046] Embodiment 1

[0047] Refer to Figure 1 and Figure 2 A digital twin-based multi-modal fusion prediction method for marine environment in this embodiment includes:

[0048] Obtain multi-modal data of the marine environment, including satellite remote sensing, buoy sensor and weather station data;

[0049] Preprocess the acquired multimodal data;

[0050] Capture the spatio-temporal correlation features of the multimodal data of the ocean environment based on a dynamic multimodal graph neural network;

[0051] Use a multi-scale gated unit to perform multi-scale feature fusion on the spatio-temporal correlation features to obtain a comprehensive feature representation;

[0052] Use a hybrid time series prediction framework to predict the comprehensive feature representation to obtain preliminary ocean environment prediction data, including short-term dynamic modeling and long-term trend modeling;

[0053] Use a generative adversarial network to perform noise fitting on the preliminary ocean environment prediction data to generate the final ocean environment prediction data.

[0054] Specifically, it includes the following steps:

[0055] S1. Acquire multimodal data of the ocean environment, including satellite remote sensing, buoy sensor, and weather station data,

[0056] Among them, the acquisition of ocean environment data relies on a multi-source sensor network. Through distributed sensing devices, real-time monitoring of physical parameters such as ocean temperature, salinity, flow velocity, and sea waves, as well as chemical parameters such as dissolved oxygen, pH value, and nutrient concentration, and meteorological parameters such as wind speed, wind direction, and atmospheric pressure is realized. After being collected by the sensors, the data is uploaded to the data center through wireless communication and uniformly calibrated by timestamps to ensure time consistency.

[0057] S2. Preprocess the acquired multimodal data,

[0058] Among them, in the data preprocessing, to make the acquired multimodal data adapt to the input requirements of the graph neural network (DM-GNN), the preprocessing operations include the following parts:

[0059] (1) Data cleaning: Remove noise and outliers, and eliminate abnormal data points with high deviation from the mean. Set the cleaning condition as:

[0060]

[0061] Among them, and are the mean and standard deviation calculated for the i-th data source within the historical time window, respectively.

[0062] (2) Missing value processing: Use the moving average method to fill in the missing values. Set the filling method as:

[0063]

[0064] Among them, the missing value is the average of the past k moments, which is used as the estimated value of the missing data at time t.

[0065] (3)Normalization processing: The eigenvalue ranges of different modalities may vary greatly. To eliminate the influence of dimensions, the normalization method is adopted:

[0066]

[0067] Among them, and are the mean and standard deviation calculated within the historical time window for the i-th data source, respectively.

[0068] S3. Dynamic multimodal fusion,

[0069] Dynamic multimodal fusion is a key step in realizing the efficient integration of multi-source heterogeneous data. With the support of dynamic multimodal fusion, the system can not only achieve in-depth modeling of spatio-temporal features between different modality data sources, but also provide accurate support for multiple practical application scenarios. For example:

[0070] Extreme event scenario prediction: The system generates realistic typhoon and tsunami scenario data through the generative adversarial network (GAN) module. Combining with the dynamic prediction results, it can accurately simulate the ocean current changes and wave amplification under extreme weather, providing a decision-making basis for disaster prevention and control.

[0071] Pollution diffusion simulation and emergency response: Through the intelligent analysis module, it simulates the diffusion path and influence range of pollutants (such as oil spills or chemical wastes). Combining with the dynamically updated prediction data, it quickly identifies high-risk areas and assists in the efficient deployment of emergency resources.

[0072] Ecological evolution trend analysis: With the support of the long-term trend modeling module, the system can model and analyze the impact of seasonal climate change on ocean temperature and salinity, evaluate the long-term impact of environmental changes on the distribution of fish habitats, and optimize the development of fishery resources.

[0073] To solve the problems of spatio-temporal resolution, physical properties, and data format differences between different data sources, the system adopts a dynamic heterogeneous graph modeling method. With the help of the dynamic multimodal graph neural network (DM-GNN), it captures the spatio-temporal correlation characteristics between multimodal data. Specifically, the multimodal data of the marine environment (including satellite remote sensing, buoy sensor, and weather station data) are modeled as a dynamic heterogeneous graph , where the node set represents each modality data source, the edge set represents the dynamic relationship between data sources, and the adjacency matrix defines the association weights between nodes.

[0074] The node set Include the feature vectors of multimodal data sources, each node represents the i-th data source, and its feature is , that is, the observed value at time t, and the specific form is:

[0075]

[0076] where , is the feature dimension of data source i. Through the node feature matrix , the system makes a unified representation of multimodal data. The dynamic adjacency matrix defines the similarity between nodes, and the specific calculation is as follows:

[0077]

[0078] where is the embedding function, which is used to map node features to a unified high-dimensional representation space to eliminate the scale differences between different modalities. By calculating the adjacency matrix through cosine similarity, the correlation between nodes can be accurately captured in the high-dimensional space.

[0079] To enhance the stability of the graph structure and prevent the problems of gradient explosion or disappearance, the adjacency matrix is further normalized to the normalized adjacency matrix :

[0080]

[0081] where is the degree matrix.

[0082] After the dynamic heterogeneous graph is constructed, the propagation and update of node features are carried out through the graph convolutional network (GCN). The update formula for each layer of graph convolutional operation is:

[0083]

[0084] where is the node feature matrix of the l-th layer, is the weight matrix for linear transformation, is the bias vector, and σ(·) is the non-linear activation function.

[0085] Multiple layers of graph convolutional operations gradually capture the high-order features of the global graph structure by stacking local neighborhood information. Finally, after L layers of convolution, the global spatio-temporal feature matrix output by the system is:

[0086]

[0087] This feature matrix Z tIt includes complex spatio-temporal associations among multi-modal data sources, providing a unified input for subsequent dynamic modeling and prediction.

[0088] S4. Multi-scale Gated Tanh Unit (M-GTU),

[0089] The multi-scale gated Tanh unit (M-GTU) is an important module for multi-time-scale modeling of spatio-temporal features, aiming to extract short-term dynamic and long-term trend features from the dynamically fused generated feature Z t . The M-GTU learns the priority of features from different time scales by combining convolutional operations, gating mechanisms, and non-linear activation functions.

[0090] Given the input feature matrix Z t , multi-time-scale features are extracted through convolutional kernels K k of different sizes:

[0091]

[0092] Among them, is the convolutional kernel of time scale k, and K k represents the convolutional kernel size. The convolutional operation slides along the time dimension of the input features to capture local features within different time windows.

[0093] The gating mechanism generates feature selection weights through the Sigmoid function to select which time-scale features are more important:

[0094]

[0095] Among them, g k is the gating weight matrix of time scale k. W g and b g are the weight and bias of the gating mechanism respectively.

[0096] The features after gating processing are:

[0097]

[0098] Among them, tanh() is the Tanh activation function, and ⊙ represents element-wise multiplication. The Tanh activation is used to introduce non-linear transformation and enhance the model's representation ability for complex features.

[0099] Features of different time scales are weighted and fused to generate a comprehensive feature representation:

[0100]

[0101] Among them, K is the number of time scales. α kis the fusion weight of time scale k, dynamically learned by the model through training.

[0102] After dynamic multi-modal fusion and feature extraction are completed, the system enters the intelligent prediction and anomaly simulation module. This module aims to achieve accurate prediction of future ocean environmental states based on high-dimensional spatio-temporal features Z t while combining the generative adversarial network (GAN) and the hybrid time series prediction framework (HTPF) to generate possible extreme event scenarios. The intelligent prediction module adopts the hybrid time series prediction framework (HTPF), which captures both rapidly changing local characteristics and global trends across time spans through the combination of short-term dynamic modeling and long-term trend modeling.

[0103] S5. Hybrid Time Series Prediction Framework (HTPF),

[0104] (1) Short-term dynamic modeling,

[0105] The short-term dynamic modeling branch is based on the Temporal Convolutional Network (TCN) and uses dilated convolutions to capture local time series features. Dilated convolutions can expand the receptive field in the time dimension, thus modeling long-term dependencies while maintaining computational efficiency. Given the input feature Z t , the formula for dilated convolution is:

[0106]

[0107] where h t is the convolutional output at time t, K is the kernel size, d is the dilation factor that controls the time skip step, and w k is the weight of the convolutional kernel.

[0108] The temporal convolutional feature h t of the last layer is mapped to the target prediction value space:

[0109]

[0110] where W short is the mapping weight matrix that maps the feature dimension d to the output target dimension. b short is the bias term, and Y short is the short-term prediction result at time t, representing the response to rapid dynamic changes (such as swells or changes in ocean currents within a short period).

[0111] By stacking dilated convolutions layer by layer, TCN can capture multi-level dependencies in the time series while avoiding information loss. The goal of the short-term dynamic branch is to quickly respond to short-term drastic changes in the ocean environment, such as rapid changes in swells or typhoon intensity.

[0112] (2) Long-term trend modeling,

[0113] The long-term trend modeling branch is based on the Transformer with multi-head attention mechanism, which can model non-linear dependencies over long time spans. Given the input feature F t , first generate query, key, and value matrices through linear transformation:

[0114]

[0115] where Q, K, and V are the query, key, and value matrices respectively; W Q , W K , W V are weight matrices for feature transformation.

[0116] The attention mechanism calculates the weights between feature time steps:

[0117]

[0118] where d k is the dimension of the key vector, used to scale the dot product result. Multi-head attention concatenates the results of different attention heads and then performs a linear transformation to generate the long-term feature representation:

[0119]

[0120] where, W O is the weight matrix of the output layer.

[0121] Finally, the long-term branch maps the long-term feature representation F long to the target prediction space through a linear layer:

[0122]

[0123] where, W long is the mapping weight matrix, b long is the bias term, and Y long represents the modeling result of long-term dependencies.

[0124] Through the multi-layer stacked Transformer structure, the long-term branch can effectively capture trend information over long time spans, such as seasonal temperature changes or slow changes in ocean surface salinity.

[0125] The prediction results of the short-term and long-term branches are weighted and fused to generate the final prediction value:

[0126]

[0127] where β is the dynamically adjusted fusion weight, representing the contribution ratio of the short-term and long-term branches to the final prediction.

[0128] S6. Generative Adversarial Network (GAN)

[0129] Generative adversarial networks are used to simulate extreme event scenarios, such as the ocean environmental state under extreme weather conditions like typhoons and tsunamis. The GAN consists of a generator G and a discriminator D. The generator G receives random noise z∽p z and reference features Z t , Y t as inputs to generate abnormal scenario data x′:

[0130]

[0131] where z∽p z is a random noise vector that simulates potential unobservable factors; Z t is a feature matrix for multimodal fusion; Y t is the fusion value of short-term and long-term predictions; θ G are the parameters of the generator.

[0132] The discriminator D receives real data x∽p data or generated data x′ and outputs its authenticity probability D(x).

[0133]

[0134] where σ is the Sigmoid activation function; f(x; θ D ) is the feature mapping of the discriminator; θ D are the parameters of the discriminator.

[0135] The goal of the GAN is to make the data generated by the generator indistinguishable from the real data through adversarial training. Its optimization goal is:

[0136]

[0137] where z∽p z is a random noise vector used to introduce diversity in the generated data; Z t is a multimodal feature matrix that provides environmental basic features; Y t is the fusion value of short-term and long-term predictions used to provide future trend references;

[0138] The generator learns the distribution of real data through adversarial training, thereby generating abnormal data close to the real environment.

[0139] S7. Interactive visualization

[0140] The interactive visualization module visually presents the prediction results and simulation scenarios of the ocean environment through 3D GIS technology. Users can adjust the time range, geographical area, and simulation parameters through the interactive interface to observe the dynamic changes and future trends of the environmental state.

[0141] Through the combination of 3D GIS technology and virtual reality (VR) technology, this system realizes immersive visual analysis of the dynamic marine environment. Users can not only intuitively view real-time monitoring data such as wave height and ocean current direction, but also adjust observation parameters according to specific needs. For example, in extreme event scenarios, users can simulate the impact of different typhoon paths on a specific area and observe in real time the increase in waves caused by the typhoon and the affected situation in the coastal area. At the same time, the system supports visual analysis of pollution events. Users can input the location of the pollution source and diffusion parameters to dynamically simulate the diffusion range of pollutants under different meteorological conditions.

[0142] Three-dimensional dynamic display:

[0143] Based on a three-dimensional geographic coordinate system, the system maps the predicted value Y t and the real-time observation value F t to three-dimensional space and visualizes them through methods such as color gradients and vector fields. The visual mapping function is defined as::

[0144]

[0145] where x and y are geographic coordinates, and f is the mapping function that converts numerical features into visual features.

[0146] Data output and report generation:

[0147] The system supports exporting the prediction results and simulation data as a standardized report, including screenshots of the three-dimensional display of dynamic changes; key indicators of the simulation results (such as wave height, flow velocity, temperature); trend analysis and error assessment. The interactive visualization module provides users with flexible analysis tools, enabling the intuitive presentation of the dynamic changes in the complex marine environment, thus supporting scientific research and decision-making.

[0148] Example 2

[0149] This example provides a digital-twin-based multi-modal fusion prediction system for the marine environment, including:

[0150] A data acquisition module, configured to:

[0151] A computer-readable storage medium storing multiple instructions, which are adapted to be loaded and executed by a processor of a terminal device for the digital-twin-based multi-modal fusion prediction method for the marine environment.

[0152] A terminal device, including a processor and a computer-readable storage medium, where the processor is used to implement each instruction; the computer-readable storage medium is used to store multiple instructions, which are adapted to be loaded and executed by the processor for the digital-twin-based multi-modal fusion prediction method for the marine environment.

[0153] The above are all preferred embodiments of the present invention, and the protection scope of the present invention is not limited thereby. Therefore, all equivalent changes made according to the structure, shape, and principle of the present invention shall be covered within the protection scope of the present invention.

Claims

1. A multimodal fusion prediction method for marine environment based on digital twin, characterized in that: include: Acquire multimodal data of the marine environment, including satellite remote sensing, buoy sensors, and weather station data; Perform data preprocessing on the acquired multimodal data; Capturing the spatiotemporal correlation characteristics of multimodal data of the marine environment based on dynamic multimodal graph neural networks; Multi-scale gating units are used to fuse the spatiotemporal correlation features at multiple scales to obtain comprehensive feature representation. The hybrid time series prediction framework is used to predict the comprehensive feature representation and obtain preliminary marine environmental prediction data, which includes short-term dynamic modeling and long-term trend modeling. Generate final ocean environment prediction data by using generative adversarial networks to perform noise fitting on preliminary ocean environment prediction data. The data preprocessing of the acquired multimodal data includes data cleaning, missing value processing and normalization processing of the acquired multimodal data, wherein the missing value processing adopts the average method to fill the missing value, and the filling method is set as: Among them, the missing value is the average value of the past k moments as the estimated value of the missing data at time t; The dynamic multimodal graph neural network captures the spatiotemporal correlation characteristics of the multimodal data of the marine environment, including modeling the multimodal data of the marine environment as a dynamic heterogeneous graph , where the node set Represents each modal data source, edge set Represents the dynamic relationship between data sources, adjacency matrix Then define the association weights between nodes. After the dynamic heterogeneous graph is constructed, the node features are propagated and updated through the graph convolution network GCN. The update formula for each layer of graph convolution operation is: in, is the node feature matrix of the lth layer, is the weight matrix for linear transformation, is the bias vector, σ(.) is the nonlinear activation function; The multi-scale gating unit is used to perform multi-scale feature fusion on the spatiotemporal correlation features, including using a multi-scale gated Tanh unit to extract short-term dynamic and long-term trend features from the generated features, wherein the local features in different time windows are captured by convolution operations, the gating mechanism is used to generate feature selection weights through the Sigmoid function, and the Tanh activation is used to introduce nonlinear transformations to enhance the model's ability to represent complex features, and the features of different time scales are weightedly fused to generate a comprehensive feature representation: Where K1 is the number of time scales, α k1 is the fusion weight of time scale k1, which is dynamically learned by the model through training; The short-term dynamic modeling includes using dilated convolution to capture local time series features. For a given input feature Z t , the calculation formula of the extended convolution is: Among them, h t is the convolution output at time t, K2 is the convolution kernel size, d is the expansion factor, which controls the time jump step, and w k2 is the weight of the convolution kernel; The long-term trend modeling includes using a transformer based on a multi-head attention mechanism to model nonlinear dependencies over a long time span, wherein firstly, query, key and value matrices are generated through linear transformation, weights between feature time steps are calculated based on the attention mechanism, and the results of different attention heads are concatenated through multi-head attention and then linearly transformed to generate long-term feature representations, and finally the long-term branch maps the long-term feature representation Flong to the target prediction space through a linear layer; The method uses a generative adversarial network to perform noise fitting on preliminary ocean environment prediction data to generate final ocean environment prediction data, including using a generator of the generative adversarial network to generate abnormal scene data and learning the distribution of real data through adversarial training to generate abnormal data close to the real environment.

2. A marine environment multimodal fusion prediction system based on digital twin, executing a marine environment multimodal fusion prediction method based on digital twin as claimed in claim 1, characterized in that: include: The data acquisition module is configured to acquire multimodal data of the marine environment, including satellite remote sensing, buoy sensor and weather station data; A preprocessing module is configured to perform data preprocessing on the acquired multimodal data; The spatiotemporal correlation module is configured to capture the spatiotemporal correlation characteristics of multimodal data of the marine environment based on a dynamic multimodal graph neural network; The feature representation module is configured to perform multi-scale feature fusion on the spatiotemporal correlation features using a multi-scale gating unit to obtain a comprehensive feature representation; The prediction module is configured to predict the comprehensive feature representation using a hybrid time series prediction framework to obtain preliminary marine environment prediction data, which includes short-term dynamic modeling and long-term trend modeling; The fitting module is configured to use a generative adversarial network to perform noise fitting on preliminary ocean environment prediction data to generate final ocean environment prediction data.

3. A computer-readable storage medium storing a plurality of instructions, characterized in that: The instructions are suitable for being loaded by a processor of a terminal device and executing the method according to claim 1 .

4. A terminal device, comprising a processor and a computer-readable storage medium, wherein the processor is used to implement each instruction; and the computer-readable storage medium is used to store multiple instructions, characterized in that: The instructions are suitable for being loaded by a processor and executing the method as claimed in claim 1 .

Images