Marine environment multi-modal fusion prediction method and system based on digital twinning

Abstract

The invention relates to the technical field of marine environment prediction, in particular to a marine environment multi-modal fusion prediction method and system based on digital twinning. The method comprises the following steps: capturing time-space correlation characteristics of multi-modal data of a marine environment based on a dynamic multi-modal graph neural network; performing multi-scale feature fusion on the space-time correlation features by using a multi-scale gating unit to obtain comprehensive feature representation; predicting the comprehensive feature representation by using a mixed time sequence prediction framework to obtain preliminary marine environment prediction data, including short-term dynamic modeling and long-term trend modeling; and performing noise fitting on the preliminary marine environment prediction data by using a generative adversarial network to generate final marine environment prediction data. According to the invention, through innovative combination of dynamic multi-modal data fusion, a mixed time sequence prediction framework and a multi-scale gating Tanh unit, the problems of insufficient data fusion, limited prediction precision and poor dynamic response capability in the traditional marine environment monitoring and prediction technology are effectively solved.

Description

Technical Field

[0001] The present invention relates to the technical field of marine environment prediction, and in particular to a marine environment multimodal fusion prediction method and system based on digital twins. Background Art

[0002] With the growing demand for marine resource development and environmental protection, real-time monitoring, dynamic prediction and emergency response of the marine environment have become the focus of current research. However, traditional marine environmental monitoring methods have many shortcomings in dealing with the dynamic characteristics of complex marine environments. First, traditional methods usually rely on a single data source, such as buoys, weather stations or satellite remote sensing. This single mode is difficult to fully capture the multi-dimensional dynamic characteristics of the marine environment, resulting in limited monitoring range and time continuity; second, in the face of data with significant spatiotemporal changes such as temperature, salinity, flow rate, and waves, the existing methods lack deep fusion of multimodal data during data processing and analysis, and it is difficult to capture the spatiotemporal correlation between different modal data, resulting in insufficient accuracy in predicting cross-time scale change trends; third, traditional systems often rely on rule models in emergency response. These models are usually based on fixed empirical rules and static data, and it is difficult to handle complex nonlinear dynamic characteristics. They cannot adapt to the complex and changing marine environment in real time and efficiently, which seriously restricts the efficiency of monitoring and decision-making.

[0003] In recent years, digital twin technology, as an emerging intelligent technology, has provided new possibilities for marine environmental monitoring and prediction. Through multimodal data fusion, digital twin technology integrates multiple data sources such as satellite remote sensing, buoys, and underwater sensors into a unified virtual model to reconstruct and simulate the real marine environment with high fidelity. This technology can not only perceive the state of the ocean in real time, but also predict future environmental change trends, effectively reducing the high cost and high risk of physical experiments. However, the current digital twin system still has significant deficiencies in high-precision data fusion, intelligent prediction, and complex ecosystem simulation. In particular, when facing the heterogeneity, real-time and accuracy requirements of multi-source data, the existing technology has not yet formed a unified and efficient solution. For example, when processing multimodal data, the existing technology lacks an effective framework that can capture the cross-modal spatiotemporal correlation of multi-source heterogeneous data; when modeling the dynamic characteristics of complex ecosystems, the existing prediction models usually cannot take into account both short-term dynamic changes and long-term trend characteristics at the same time, especially in extreme weather scenarios, the reliability and accuracy of the prediction results are still low. In addition, the current digital twin technology is still insufficient in result display and interactivity. Existing visualization methods are difficult to intuitively present the dynamic changes of complex environments, and it is difficult to meet the flexible analysis needs in practical applications.

[0004] At the same time, when traditional methods process multi-source heterogeneous data (such as satellite remote sensing, buoy sensors and weather station data), due to differences in data format, resolution and time synchronization, it is difficult to achieve efficient integration and spatiotemporal correlation modeling, resulting in limited monitoring accuracy and insufficient ability to capture dynamic changes. In addition, existing methods lack the ability to jointly model short-term dynamics and long-term trends when dealing with multi-scale changes in complex marine ecosystems, especially in the prediction of extreme events (such as typhoons and tsunamis), and it is difficult to provide reliable and accurate results.

[0005] At the data fusion level, current methods mostly use simple weighted average or linear integration methods to deal with the heterogeneity and spatiotemporal dynamic characteristics of multimodal data, which cannot effectively capture the complex dependencies across modalities and time and space, limiting the comprehensive expression of multidimensional dynamic features. At the same time, in multiscale modeling, traditional time series models (such as ARIMA or a single deep learning model) only focus on single time scale characteristics, ignoring the coupling relationship between short-term drastic changes and long-term stable trends, resulting in low prediction accuracy. Especially in the simulation of extreme weather scenarios, existing methods find it difficult to generate realistic abnormal data for scenario deduction. In addition, existing systems mostly use two-dimensional static charts or simple indicator displays in the display of results, lack dynamic interaction capabilities and intuitive visualization, and are difficult to support users in the efficient analysis and understanding of complex marine environments. Summary of the invention

[0006] In order to solve the above-mentioned problems, the present invention provides a multimodal fusion prediction method and system for marine environment based on digital twins.

[0007] In the first aspect, the present invention provides a method for multimodal fusion prediction of marine environment based on digital twin, which adopts the following technical solution: A multimodal fusion prediction method for marine environment based on digital twins, including: Acquire multimodal data of the marine environment, including satellite remote sensing, buoy sensors, and weather station data; Perform data preprocessing on the acquired multimodal data; Capturing the spatiotemporal correlation characteristics of multimodal data of the marine environment based on dynamic multimodal graph neural networks; Multi-scale gating units are used to fuse the spatiotemporal correlation features at multiple scales to obtain comprehensive feature representation. The hybrid time series prediction framework is used to predict the comprehensive feature representation and obtain preliminary marine environmental prediction data, which includes short-term dynamic modeling and long-term trend modeling. The generative adversarial network is used to perform noise fitting on the preliminary ocean environment prediction data to generate the final ocean environment prediction data.

[0008] Furthermore, the data preprocessing of the acquired multimodal data includes data cleaning, missing value processing and normalization processing of the acquired multimodal data, wherein the missing value processing adopts the average method to fill the missing values, and the filling method is set as: The missing value is the average value of the past k moments as the estimated value of the missing data at time t.

[0009] Furthermore, the dynamic multimodal graph neural network captures the spatiotemporal correlation characteristics of the multimodal data of the marine environment, including modeling the multimodal data of the marine environment as a dynamic heterogeneous graph , where the node set Represents each modal data source, edge set Represents the dynamic relationship between data sources, adjacency matrix Then define the association weights between nodes. After the dynamic heterogeneous graph is constructed, the node features are propagated and updated through the graph convolution network GCN. The update formula for each layer of graph convolution operation is: in, is the node feature matrix of the lth layer, is the weight matrix for linear transformation, is the bias vector and σ(·) is the nonlinear activation function.

[0010] Furthermore, the multi-scale gating unit is used to perform multi-scale feature fusion on the spatiotemporal correlation features, including extracting short-term dynamic and long-term trend features from the generated features using a multi-scale gating Tanh unit, wherein local features in different time windows are captured by convolution operations, feature selection weights are generated by a gating mechanism through a Sigmoid function, Tanh activation is used and nonlinear transformation is introduced to enhance the model's representation ability for complex features, and features of different time scales are weightedly fused to generate a comprehensive feature representation: Where K is the number of time scales. k is the fusion weight of time scale k, which is dynamically learned by the model through training.

[0011] Furthermore, the short-term dynamic modeling includes using dilated convolution to capture local time series features. For a given input feature Z t , the calculation formula of the extended convolution is: Among them, h t is the convolution output at time t, K is the convolution kernel size, d is the expansion factor, which controls the time jump step, and wk is the weight of the convolution kernel.

[0012] Furthermore, the long-term trend modeling includes using a transformer based on a multi-head attention mechanism to model nonlinear dependencies over a long time span, wherein query, key and value matrices are first generated through linear transformation, weights between feature time steps are calculated based on the attention mechanism, and the results of different attention heads are concatenated and linearly transformed through multi-head attention to generate long-term feature representations, and finally the long-term branch maps the long-term feature representation Flong to the target prediction space through a linear layer.

[0013] Furthermore, the use of a generative adversarial network to perform noise fitting on preliminary ocean environment prediction data to generate final ocean environment prediction data includes using a generator of the generative adversarial network to generate abnormal scene data and learning the distribution of real data through adversarial training to generate abnormal data close to the real environment.

[0014] The second aspect is a multi-modal fusion prediction system for marine environment based on digital twins, including: The data acquisition module is configured to acquire multimodal data of the marine environment, including satellite remote sensing, buoy sensor and weather station data; A preprocessing module is configured to perform data preprocessing on the acquired multimodal data; The spatiotemporal correlation module is configured to capture the spatiotemporal correlation characteristics of multimodal data of the marine environment based on a dynamic multimodal graph neural network; The feature representation module is configured to perform multi-scale feature fusion on the spatiotemporal correlation features using a multi-scale gating unit to obtain a comprehensive feature representation; The prediction module is configured to predict the comprehensive feature representation using a hybrid time series prediction framework to obtain preliminary marine environment prediction data, which includes short-term dynamic modeling and long-term trend modeling; The fitting module is configured to use a generative adversarial network to perform noise fitting on preliminary ocean environment prediction data to generate final ocean environment prediction data.

[0015] In a third aspect, the present invention provides a computer-readable storage medium storing a plurality of instructions, wherein the instructions are suitable for being loaded and executed by a processor of a terminal device, a multimodal fusion prediction method for an ocean environment based on digital twins.

[0016] In a fourth aspect, the present invention provides a terminal device comprising a processor and a computer-readable storage medium, the processor being used to implement various instructions; the computer-readable storage medium being used to store multiple instructions, the instructions being suitable for being loaded by the processor and executing the multimodal fusion prediction method for an ocean environment based on digital twins.

[0017] In summary, the present invention has the following beneficial technical effects: This invention effectively solves the problems of insufficient data fusion, limited prediction accuracy and poor dynamic response capability in traditional marine environment monitoring and prediction technologies through the innovative combination of dynamic multimodal data fusion, hybrid time series prediction framework (HTPF) and multi-scale gated Tanh unit (M-GTU). The real-time acquisition and deep fusion technology of multi-source heterogeneous data based on dynamic multimodal graph neural network (DM-GNN) fully integrates multimodal data sources such as satellite remote sensing, buoy sensors, and meteorological stations, overcoming the difficulties of traditional methods in data isolation, heterogeneity and complex association processing.

[0018] Combined with the multi-time scale feature extraction mechanism of M-GTU, the system can accurately capture the short-term dynamic changes and long-term trends in the marine environment, especially in the prediction of extreme events such as storms and tides, showing excellent sensitivity and robustness. Through the synergy of the short-term dynamic modeling branch (based on the temporal convolutional network) and the long-term trend modeling branch (based on the multi-head attention mechanism), the system has shown excellent performance in both rapid-changing events and long-span ecological evolution modeling. The introduction of the generative adversarial network (GAN) enables the system to simulate the state of the marine environment under extreme scenarios, providing richer scenario analysis support for scientific decision-making. The prediction results achieve the optimal combination of short-term and long-term features through dynamic fusion, significantly improving the accuracy and reliability of the prediction.

[0019] This system integrates an interactive 3D visualization platform to intuitively display the real-time status and future change trends of the marine environment. Users can adjust the area, time range or input scenario simulation parameters in real time through a flexible interactive interface to dynamically view specific scenario simulations under complex environments. This intuitive and efficient visualization capability not only supports users to analyze marine environmental changes in multiple dimensions, but also provides a scientific basis for optimizing resource management, ecological protection and emergency response strategies. Compared with traditional methods, the present invention has achieved improvements in data processing efficiency, prediction accuracy, scenario simulation capabilities and visual interactive experience, providing key technical support for the construction of a smart ocean, and has broad application value in the fields of marine resource development, environmental protection and disaster prevention and control.

[0020] The multimodal fusion prediction system for the marine environment based on digital twins proposed in the present invention can be widely used in real-time monitoring and future state prediction of complex marine dynamic environments through deep integration and intelligent analysis of multi-source data. In marine resource management, the present invention can predict the spatial distribution and temporal changes of temperature and salinity by integrating satellite remote sensing, buoy sensors and meteorological station data, and provide data support for fishery resource assessment and planning. In terms of marine disaster management, the present invention can simulate the impact of extreme events such as typhoons and tsunamis on specific sea areas in real time, and assist the government and relevant departments in formulating accurate evacuation and disaster emergency plans. In addition, the system combines three-dimensional dynamic visualization technology to support real-time display and trend analysis of environmental monitoring data, which is of great value in public science popularization and scientific research applications. In particular, the present invention demonstrates excellent performance and adaptability in dealing with complex changes across time scales and analyzing the interactive effects of marine ecosystems and climate. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a schematic diagram of a multimodal fusion prediction method for marine environment based on digital twins according to Example 1 of the present invention; Figure 2 It is another schematic diagram of the multimodal fusion prediction method of the marine environment based on digital twins in Example 1 of the present invention. DETAILED DESCRIPTION

[0022] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0023] Example 1 Reference Figure 1 and Figure 2 , a multi-modal fusion prediction method for ocean environment based on digital twins in this embodiment includes: Acquire multimodal data of the marine environment, including satellite remote sensing, buoy sensors, and weather station data; Perform data preprocessing on the acquired multimodal data; Capturing the spatiotemporal correlation characteristics of multimodal data of the marine environment based on dynamic multimodal graph neural networks; Multi-scale gating units are used to fuse the spatiotemporal correlation features at multiple scales to obtain comprehensive feature representation. The hybrid time series prediction framework is used to predict the comprehensive feature representation and obtain preliminary marine environmental prediction data, which includes short-term dynamic modeling and long-term trend modeling. The generative adversarial network is used to perform noise fitting on the preliminary ocean environment prediction data to generate the final ocean environment prediction data.

[0024] Specifically, the following steps are included: S1. Obtain multimodal data of the marine environment, including satellite remote sensing, buoy sensors and weather station data, The acquisition of marine environment data relies on a multi-source sensor network, which uses distributed sensor equipment to achieve real-time monitoring of physical parameters such as ocean temperature, salinity, flow rate, and waves, as well as chemical parameters such as dissolved oxygen, pH value, and nutrient concentration, and meteorological parameters such as wind speed, wind direction, and atmospheric pressure. After being collected by the sensor, the data is uploaded to the data center through wireless communication and uniformly calibrated through timestamps to ensure time consistency.

[0025] S2. Perform data preprocessing on the acquired multimodal data. Among them, in data preprocessing, in order to make the collected multimodal data adapt to the input requirements of the graph neural network (DM-GNN), the preprocessing operation includes the following parts: (1) Data cleaning: remove noise and outliers, and remove abnormal data points with high deviation from the mean. Set the cleaning conditions as follows: in, and are the mean and standard deviation calculated for the ith data source in the historical time window.

[0026] (2) Missing value processing: The sliding average method is used to fill the missing values, and the filling method is set as follows: The missing value is the average value of the past k moments as the estimated value of the missing data at time t.

[0027] (3) Normalization: The range of eigenvalues ​​of different modes may vary greatly. In order to eliminate the influence of dimension, the normalization method is used: in, and are the mean and standard deviation calculated for the ith data source in the historical time window.

[0028] S3. Dynamic multimodal fusion, Dynamic multimodal fusion is a key step to achieve efficient integration of multi-source heterogeneous data. With the support of dynamic multimodal fusion, the system can not only achieve deep modeling of spatiotemporal features between different modal data sources, but also provide accurate support for multiple practical application scenarios. For example: Extreme event scenario prediction: The system generates realistic typhoon and tsunami scenario data through the generative adversarial network (GAN) module. Combined with dynamic prediction results, it can accurately simulate ocean current changes and wave increases under extreme weather conditions, providing a decision-making basis for disaster prevention and control.

[0029] Pollution diffusion simulation and emergency response: Through the intelligent analysis module, the diffusion path and impact range of pollutants (such as oil spills or chemical waste) are simulated, combined with dynamically updated prediction data, high-risk areas can be quickly identified to assist in the efficient deployment of emergency resources.

[0030] Ecological evolution trend analysis: With the support of the long-term trend modeling module, the system can model and analyze the impact of seasonal climate change on ocean temperature and salinity, evaluate the long-term impact of environmental changes on the distribution of fish habitats, and optimize the development of fishery resources.

[0031] In order to solve the problems of differences in spatiotemporal resolution, physical properties and data formats between different data sources, the system adopts a dynamic heterogeneous graph modeling method, using a dynamic multimodal graph neural network (DM-GNN) to capture the spatiotemporal correlation characteristics between multimodal data. Specifically, the multimodal data of the marine environment (including satellite remote sensing, buoy sensors and weather station data) is modeled as a dynamic heterogeneous graph. , where the node set Represents each modal data source, edge set Represents the dynamic relationship between data sources, adjacency matrix The association weights between nodes are defined.

[0032] Node Collection Includes feature vectors of multimodal data sources, each node represents the i-th data source, whose characteristics are , that is, the observed value at time t, the specific form is: in , is the feature dimension of data source i. Through the node feature matrix , the system uniformly represents multimodal data. Dynamic adjacency matrix The similarity between nodes is defined, and the specific calculation is as follows: in, It is an embedding function that maps node features to a unified high-dimensional representation space and eliminates scale differences between different modalities. The adjacency matrix is ​​calculated by cosine similarity, which can accurately capture the correlation between nodes in high-dimensional space.

[0033] To enhance the stability of the graph structure and prevent gradient explosion or vanishing problems, the adjacency matrix Further normalized to the normalized adjacency matrix : in is the degree matrix.

[0034] After the dynamic heterogeneous graph is constructed, the node features are propagated and updated through the graph convolution network (GCN). The update formula for each layer of graph convolution operation is: in, is the node feature matrix of the lth layer, is the weight matrix for linear transformation, is the bias vector and σ(·) is the nonlinear activation function.

[0035] Multi-layer graph convolution operations gradually capture high-order features of the global graph structure by superimposing local neighborhood information. Finally, after L layers of convolution, the global spatiotemporal feature matrix output by the system is: The feature matrix Z t It includes complex spatiotemporal correlations between multimodal data sources, providing a unified input for subsequent dynamic modeling and prediction.

[0036] S4. Multi-scale Gated Tanh Unit (M-GTU), The Multi-scale Gated Tanh Unit (M-GTU) is an important module for multi-time scale modeling of spatiotemporal features. t Extract short-term dynamic and long-term trend features from the network. M-GTU learns the priority of features from different time scales by combining convolution operations, gating mechanisms, and nonlinear activation functions.

[0037] Given the input feature matrix Z t , through convolution kernels K of different sizes k Extract features at multiple time scales: in, is the convolution kernel of time scale k, K k Represents the convolution kernel size. The convolution operation slides on the time dimension of the input feature to capture local features in different time windows.

[0038] The gating mechanism generates feature selection weights through the Sigmoid function to select which time scale features are more important: Among them, g k is the gating weight matrix for time scale k. g and b g are the weight and bias of the gating mechanism respectively.

[0039] The features after gating are: Among them, tanh() is the Tanh activation function, and ⊙ represents element-by-element multiplication. Tanh activation is used to introduce nonlinear transformations and enhance the model's ability to represent complex features.

[0040] Features at different time scales are weighted and fused to generate a comprehensive feature representation: Where K is the number of time scales. k is the fusion weight of time scale k, which is dynamically learned by the model through training.

[0041] After dynamic multimodal fusion and feature extraction are completed, the system enters the intelligent prediction and abnormal simulation module. This module is designed to t Achieve accurate prediction of the future state of the ocean environment, and combine the Generative Adversarial Network (GAN) and the Hybrid Time Series Forecasting Framework (HTPF) to generate possible extreme event scenarios. The intelligent prediction module adopts the Hybrid Time Series Forecasting Framework (HTPF), which combines short-term dynamic modeling with long-term trend modeling to capture both rapidly changing local characteristics and global trends across time spans.

[0042] S5. Hybrid Temporal Prediction Framework (HTPF), (1) Short-term dynamic modeling, The short-term dynamic modeling branch is based on the temporal convolutional network (TCN) and uses dilated convolution to capture local time series features. Dilated convolution can expand the receptive field in the time dimension, thereby modeling longer time dependencies while maintaining computational efficiency. Given an input feature Z t , the calculation formula of the extended convolution is: Among them, h t is the convolution output at time t, K is the convolution kernel size, d is the expansion factor, which controls the time jump step, and w k is the weight of the convolution kernel.

[0043] The last layer of temporal convolution features h t Mapping to the target prediction value space: Among them, W short is the mapping weight matrix, mapping the feature dimension d to the output target dimension. short is the bias term, Y short It is a short-term prediction result for time t, which represents the response to rapid dynamic changes (such as swells or ocean current changes in a short period of time).

[0044] By stacking dilated convolutions layer by layer, TCN is able to capture multi-level dependencies in time series while avoiding information loss. The goal of the short-term dynamic branch is to quickly respond to short-term drastic changes in the marine environment, such as rapid changes in swells or typhoon intensity.

[0045] (2) Long-term trend modeling, The long-term trend modeling branch is based on the Transformer with multi-head attention mechanism, which can model nonlinear dependencies over long time spans. Given the input feature F t , first generate the query, key, and value matrices through linear transformation: Where Q, K, V are query, key, and value matrices respectively; W Q ,W K ,W V is the weight matrix used for feature transformation.

[0046] The attention mechanism calculates the weights between feature time steps: where d k is the dimension of the key vector, which is used to scale the dot product result. Multi-head attention concatenates the results of different attention heads and performs linear transformation to generate long-term feature representation: Among them, W O is the weight matrix of the output layer.

[0047] Finally, the long-term branch transforms the long-term feature representation F through a linear layer. long Mapping to the target prediction space: Among them, W long is the mapping weight matrix, b long is the bias term, Y long Represents the modeling results for long-term dependence.

[0048] Through a multi-layer stacked transformer structure, the long-term branch can effectively capture trend information over long time spans, such as seasonal temperature changes or slow changes in ocean surface salinity.

[0049] The prediction results of short-term and long-term branches are weighted and fused to generate the final prediction value: Where β is the dynamically adjusted fusion weight, which indicates the contribution ratio of short-term and long-term branches to the final prediction.

[0050] S6. Generative Adversarial Network (GAN), Generative adversarial networks are used to simulate extreme event scenarios, such as typhoons, tsunamis and other extreme weather conditions in the ocean environment. GAN consists of a generator G and a discriminator D. The generator G receives random noise z∽p z and reference feature Z t , Y t As input, generate abnormal scene data x′: Among them, z∽p z is a random noise vector, simulating potential unobservable factors; Z t is the feature matrix of multimodal fusion; Y t is the fusion value of short-term and long-term prediction; θ G The parameters for the generator.

[0051] The discriminator D receives the real data x∽p data Or generate data x′ and output its authenticity probability D(x).

[0052] Among them, σ is the Sigmoid activation function; f(x; θ D ) is the feature map of the discriminator; θ D are the parameters of the discriminator.

[0053] The goal of GAN is to make the data generated by the generator indistinguishable from the real data through adversarial training. Its optimization goal is: Among them, z∽p z is a random noise vector used to introduce diversity in generated data; Z t Y is a multimodal feature matrix that provides basic environmental features; t It is the fusion value of short-term and long-term forecasts, used to provide reference for future trends; The generator learns the distribution of real data through adversarial training, thereby generating abnormal data that is close to the real environment.

[0054] S7. Interactive Visualization, The interactive visualization module uses 3D GIS technology to intuitively present the prediction results and simulation scenarios of the marine environment. Users can adjust the time range, geographical area and simulation parameters through the interactive interface to observe the dynamic changes and future trends of the environmental status.

[0055] By combining 3D GIS technology with virtual reality (VR) technology, this system realizes immersive visual analysis of dynamic marine environment. Users can not only intuitively view real-time monitoring data such as wave height and ocean current direction, but also adjust observation parameters according to specific needs. For example, in extreme event scenarios, users can simulate the impact of different typhoon paths on specific areas, and observe in real time the increase in typhoon-induced waves and the impact on coastal areas. At the same time, the system supports visual analysis of pollution events. Users can input the location of pollution sources and diffusion parameters to dynamically simulate the diffusion range of pollutants under different meteorological conditions.

[0056] 3D dynamic display: The system uses a three-dimensional geographic coordinate system to calculate the predicted value Y t and real-time observation value F t Mapped to three-dimensional space and visualized by color gradient, vector field, etc. The visualization mapping function is defined as: Where x and y are geographic coordinates, and f is a mapping function that converts numerical features into visual features.

[0057] Data export and report generation: The system supports exporting prediction results and simulation data into standardized reports, including dynamic 3D display screenshots; key indicators of simulation results (such as wave height, current velocity, temperature); trend analysis and error assessment. The interactive visualization module provides users with flexible analysis tools to intuitively present the dynamic changes of complex marine environments, thereby supporting scientific research and decision-making.

[0058] Example 2 This embodiment provides a marine environment multimodal fusion prediction system based on digital twins, including: The data acquisition module is configured as follows: A computer-readable storage medium having a plurality of instructions stored therein, wherein the instructions are suitable for being loaded and executed by a processor of a terminal device, a method for multimodal fusion prediction of an ocean environment based on digital twins.

[0059] A terminal device includes a processor and a computer-readable storage medium, the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, and the instructions are suitable for being loaded and executed by the processor to describe a multimodal fusion prediction method for an ocean environment based on digital twins.

[0060] The above are all preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Therefore, any equivalent changes made based on the structure, shape, and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multimodal fusion prediction method for marine environment based on digital twin, characterized in that: include: Acquire multimodal data of the marine environment, including satellite remote sensing, buoy sensors, and weather station data; Perform data preprocessing on the acquired multimodal data; Capturing the spatiotemporal correlation characteristics of multimodal data of the marine environment based on dynamic multimodal graph neural networks; Multi-scale gating units are used to fuse the spatiotemporal correlation features at multiple scales to obtain comprehensive feature representation. The hybrid time series prediction framework is used to predict the comprehensive feature representation and obtain preliminary marine environmental prediction data, which includes short-term dynamic modeling and long-term trend modeling. The generative adversarial network is used to perform noise fitting on the preliminary ocean environment prediction data to generate the final ocean environment prediction data.

2. According to claim 1, a method for multimodal fusion prediction of marine environment based on digital twins is characterized in that: The data preprocessing of the acquired multimodal data includes data cleaning, missing value processing and normalization processing of the acquired multimodal data, wherein the missing value processing adopts the average method to fill the missing value, and the filling method is set as: The missing value is the average value of the past k moments as the estimated value of the missing data at time t.

3. According to claim 2, a method for multimodal fusion prediction of marine environment based on digital twins is characterized in that: The dynamic multimodal graph neural network captures the spatiotemporal correlation characteristics of the multimodal data of the marine environment, including modeling the multimodal data of the marine environment as a dynamic heterogeneous graph , where the node set Represents each modal data source, edge set Represents the dynamic relationship between data sources, adjacency matrix Then define the association weights between nodes. After the dynamic heterogeneous graph is constructed, the node features are propagated and updated through the graph convolution network GCN. The update formula for each layer of graph convolution operation is: in, is the node feature matrix of the lth layer, is the weight matrix for linear transformation, is the bias vector and σ(·) is the nonlinear activation function.

4. According to claim 3, a method for multimodal fusion prediction of marine environment based on digital twins is characterized in that: The multi-scale gating unit is used to perform multi-scale feature fusion on the spatiotemporal correlation features, including using a multi-scale gated Tanh unit to extract short-term dynamic and long-term trend features from the generated features, wherein the local features in different time windows are captured by convolution operations, the gating mechanism is used to generate feature selection weights through the Sigmoid function, and the Tanh activation is used to introduce nonlinear transformations to enhance the model's ability to represent complex features, and the features of different time scales are weightedly fused to generate a comprehensive feature representation: Where K is the number of time scales, α k is the fusion weight of time scale k, which is dynamically learned by the model through training.

5. The method for multimodal fusion prediction of marine environment based on digital twin according to claim 4 is characterized in that: The short-term dynamic modeling includes using dilated convolution to capture local time series features. For a given input feature Z t , the calculation formula of the extended convolution is: Among them, h t is the convolution output at time t, K is the convolution kernel size, d is the expansion factor, which controls the time jump step, and w k is the weight of the convolution kernel.

6. The method for multimodal fusion prediction of marine environment based on digital twin according to claim 5 is characterized in that: The long-term trend modeling includes using a transformer based on a multi-head attention mechanism to model nonlinear dependencies over a long time span, wherein firstly, query, key and value matrices are generated through linear transformation, weights between feature time steps are calculated based on the attention mechanism, and the results of different attention heads are concatenated through multi-head attention and then linearly transformed to generate long-term feature representation, and finally the long-term branch maps the long-term feature representation Flong to the target prediction space through a linear layer.

7. The method for multimodal fusion prediction of marine environment based on digital twin according to claim 6 is characterized in that: The method uses a generative adversarial network to perform noise fitting on preliminary ocean environment prediction data to generate final ocean environment prediction data, including using a generator of the generative adversarial network to generate abnormal scene data and learning the distribution of real data through adversarial training to generate abnormal data close to the real environment.

8. A multi-modal fusion prediction system for marine environment based on digital twins, characterized in that: include: The data acquisition module is configured to acquire multimodal data of the marine environment, including satellite remote sensing, buoy sensor and weather station data; A preprocessing module is configured to perform data preprocessing on the acquired multimodal data; The spatiotemporal correlation module is configured to capture the spatiotemporal correlation characteristics of multimodal data of the marine environment based on a dynamic multimodal graph neural network; The feature representation module is configured to perform multi-scale feature fusion on the spatiotemporal correlation features using a multi-scale gating unit to obtain a comprehensive feature representation; The prediction module is configured to predict the comprehensive feature representation using a hybrid time series prediction framework to obtain preliminary marine environment prediction data, which includes short-term dynamic modeling and long-term trend modeling; The fitting module is configured to use a generative adversarial network to perform noise fitting on preliminary ocean environment prediction data to generate final ocean environment prediction data.

9. A computer-readable storage medium storing a plurality of instructions, characterized in that: The instructions are suitable for being loaded by a processor of a terminal device and executing the method according to claim 1 .

10. A terminal device, comprising a processor and a computer-readable storage medium, wherein the processor is used to implement each instruction; and the computer-readable storage medium is used to store multiple instructions, characterized in that: The instructions are suitable for being loaded by a processor and executing the method as claimed in claim 1 .

Images