Typhoon probability prediction intelligent method and device based on multi-modal data fusion

Abstract

The application relates to a typhoon probability prediction intelligent method and device based on multi-modal data fusion, which comprises the following steps: A, constructing a typhoon intelligent prediction model; B, training the typhoon prediction intelligent model; C, using the trained typhoon prediction intelligent model to make inference prediction. Unlike the method for predicting typhoon by using a dynamic numerical mode in the meteorological field, the method provided by the application is completely based on a data-driven deep learning technology, and in the modeling process, the influence of high-dimensional remote sensing data, reanalysis data and dynamic-statistical factor data related to typhoon prediction and the cross-modal dependence among the data on the development of typhoon is considered, and the probability prediction of the typhoon path and intensity is provided. Compared with the prior art, the application has the advantages of low cost, easy improvement, high prediction precision and high confidence, and can be used for improving the business prediction of typhoon.

Description

Technical Field

[0001] This invention relates to the field of interdisciplinary technology of computer science and meteorological science, and in particular to a method and device for intelligent typhoon probability forecasting based on multimodal data fusion. Background Technology

[0002] A typhoon is the name given to a strong tropical cyclone in the Northwest Pacific region. As an extreme weather event, it typically brings strong winds and torrential rains as it passes through, causing severe casualties and direct economic losses of up to tens of billions of dollars. Operational typhoon forecasts mainly include the typhoon's path, intensity, and precipitation (including the affected area and rainfall amount).

[0003] Currently, 24-hour typhoon track forecasts generally meet the needs of practical operational forecasting, but longer-term track forecasts are less than ideal. Furthermore, research on typhoon intensity forecasting has progressed slowly internationally over the past few decades, making it difficult to further improve forecast accuracy. At the same time, due to the complexity of the underlying topography and the interaction of multi-scale systems, the key factors influencing precipitation in landfall typhoons exhibit significant individual differences, and their causes and mechanisms still need to be investigated, making landfall typhoon precipitation forecasts far from meeting practical needs.

[0004] Although significant progress has been made in typhoon forecasting in recent years, both numerical models and statistical methods have not been without their limitations. While dynamical numerical models remain the dominant method for typhoon forecasting both domestically and internationally, the uncertainties inherent in their modeling process (initial values, boundary conditions, parameterization schemes, etc.) inevitably reduce forecasting accuracy. In contrast, data-driven statistical typhoon forecasting, which learns typhoon dynamics directly from vast amounts of ocean-atmosphere data, has become a research hotspot in recent years. However, existing statistical typhoon forecasting models also have drawbacks, such as reliance on manually designed forecast factors, failure to utilize typhoon structure information from high-dimensional remote sensing data and ocean-atmosphere coupling data, lack of ensemble probability forecasts, and limitations in interpreting physical mechanisms. Improving deep learning models by extracting features from data across different dimensions through multimodal data fusion has become a research hotspot in the interdisciplinary field of meteorology and computer science. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a typhoon probability intelligent forecasting method based on multimodal data fusion.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] A typhoon probability intelligent forecasting method based on multimodal data fusion, comprising the following steps:

[0008] A. Construct a smart typhoon forecasting model;

[0009] B. Training intelligent models for typhoon forecasting;

[0010] C. Use a trained intelligent typhoon forecasting model to make inferences and predictions.

[0011] Step A specifically includes the following steps:

[0012] A.1. Select and collect multimodal and multi-element meteorological data related to typhoons based on atmospheric dynamics and thermodynamics models, including data from three modes: satellite and radar remote sensing images, reanalysis gridded data, and statistical dynamic factors. Construct a multimodal data preprocessing module. Using records in the typhoon optimal path data as an index, match multimodal and multivariate input factors, and perform preprocessing processes such as interpolation and normalization, spatiotemporal resolution adjustment, and autoregressive sequence construction on the multimodal and multivariate data.

[0013] A.2. For data of different modalities and different physical variables, determine the guidance time and coverage area, construct a multimodal multivariate feature coding network, and extract the spatiotemporal features of remote sensing images, ocean-atmosphere grids and statistical-dynamic factors through remote sensing image coding network, ocean-atmosphere grid coding network and entity embedding coding network respectively;

[0014] A.3. Based on the Transformer fusion network, capture the coupling and interaction process of multimodal and multi-factor variables, determine cross-modal feature dependencies, and fuse multimodal features;

[0015] A.4. Construct a probabilistic forecasting network for typhoon intensity and path, using the fused multimodal features as input to complete typhoon path and intensity forecasts. Construct a rainfall forecasting network for typhoon rainfall, using the fused multimodal features and multi-scale rainfall feature maps extracted by the ocean-atmosphere grid coding network as input to complete typhoon rainfall forecasts.

[0016] In step A.1, the multimodal and multi-element meteorological data includes data from three modes: satellite and radar remote sensing images, reanalysis gridded data, and statistical dynamic factors.

[0017] In step A.2, the encoding expression of the multimodal multivariate feature encoding network is:

[0018] Z IMG Z SDF Z ENV Z RAIN =encoder(X) IMG X SDF X ENV )

[0019] Among them, XIMG X SDF X ENV The data consist of remote sensing imagery, statistical-dynamic factor data, and environmental-atmosphere gridded data collected within the 24 hours prior to the reporting time. Each modality was encoded using a separate feature encoder (·). IMG Z SDF Z ENV The spatiotemporal characteristics of remote sensing images, ocean-atmosphere grids, and statistical-dynamic factors are respectively, Z. RAIN This is a multi-scale rainfall feature map.

[0020] In step A.3, for typhoon track and intensity forecasting, the feature data of remote sensing images and statistical-dynamic factors are used as the primary mode, and the feature data of ocean-atmosphere grid features are used as the secondary mode. For typhoon rainfall forecasting, the feature data of ocean-atmosphere grid data and remote sensing images are used as the primary mode, and the feature data of statistical-dynamic factors are used as the secondary mode. By constructing a Transformer fusion network with D layers, and using a multi-head self-attention feature fusion method, the features of all modes are aggregated onto the primary mode features, and the primary mode features are updated. The features of the secondary modes only complete their own updates and do not aggregate information from other modes. Therefore, the calculation expression based on the Transformer fusion network is:

[0021]

[0022] in, Represents the initial dominant modality features. This represents the initial auxiliary modal features, with the superscript D indicating the D-th layer. This refers to the features of the Dth layer after all modal information is aggregated into the main modality.

[0023] In step A.4, the calculation expression for the rainfall forecasting network is as follows:

[0024]

[0025] Where Rain_decoder is the feature decoder for rainfall forecasting. To forecast the rainfall field output over h time steps, Z IMG，SDF，ENV→SDF For the fused multimodal features, Z RAIN This is a multi-scale rainfall feature map.

[0026] In step A.4, the calculation expression for the probabilistic prediction network is as follows:

[0027]

[0028]

[0029] in, The intensity and path forecast values ​​are from the previous time step, and their distribution is determined by the Gaussian mixture model. The fitting is achieved by decomposing the variable distribution into a linear combination of multiple Gaussian distribution functions, a t Z is the encoding vector for the sequence position. t The fused features at time t are from the Transformer fusion network, where RNN represents a recurrent neural network. Represents the normal distribution function, with weighting coefficients π and parameters μ and σ. 2 Fitted by a neural network, P represents different The probability density of the selected value is the probability prediction result, where k represents the k-th Gaussian distribution function, and K is the number of Gaussian distribution functions in the Gaussian mixture model.

[0030] Step B specifically includes the following steps:

[0031] B.1. Select multimodal and multi-factor data related to typhoon forecasting and construct a training dataset;

[0032] B.2. Select appropriate temporal and spatial resolutions for each variable and perform feature engineering preprocessing on the data;

[0033] B.3. Using individual typhoon cases as an index, randomly divide the training set, validation set, and test set in a ratio of 8:1:1. Use the training set for thorough training, use the validation set to supervise the training effect, and use the test set to calculate the model's forecasting skills.

[0034] Step C specifically includes the following steps:

[0035] C.1. Based on the multimodal and multivariate initial meteorological data at the reporting time, determine the initial forecast field and perform the same data preprocessing as in step B.2, which serves as the input for the typhoon forecasting method;

[0036] C.2. Input the preprocessed forecast initial field into the feature encoding network of each modality data for feature extraction and dimensionality reduction, and encode the spatiotemporal features of the multimodal data;

[0037] C.3. Input multimodal features into the Transformer fusion network, determine cross-modal feature dependencies, and aggregate multimodal features;

[0038] C.4. Input the aggregated multimodal features into the probabilistic forecasting network to perform sequence forecasts of typhoon path, intensity, and precipitation, and obtain the probability distribution of the forecast values ​​after the superposition of mixed Gaussian distributions; combine the different scale feature codes output by the ocean-atmosphere grid coding network with the aggregated multimodal features, input them into the precipitation forecasting network, and obtain the forecast results of the precipitation field;

[0039] C.5. Process typhoon forecast results and calculate evaluation indicators.

[0040] A typhoon probability intelligent forecasting device based on multimodal data fusion, the device comprising:

[0041] The first determination module is used to determine the initial forecast factors based on multimodal and multivariate meteorological data from the reporting time.

[0042] Data preprocessing module: used to convert initial forecast factors into the input format specified by the method;

[0043] Inference and prediction module: used to deploy and run trained intelligent typhoon forecasting methods on selected hardware to obtain prediction results;

[0044] Results collection module: Used for post-processing of output, including: collecting field data, calculating various indices and evaluation indicators.

[0045] Compared with the prior art, the present invention has the following advantages:

[0046] The method provided by this invention is entirely based on data-driven deep learning technology. In the modeling process, it considers the impact of high-dimensional remote sensing data, reanalysis data, and dynamic-statistical factor data related to typhoon forecasting, as well as the cross-modal dependence between them, on typhoon development. It also provides probabilistic forecasts of typhoon track and intensity. Compared with existing technologies, this invention has the advantages of low cost, easy improvement, high forecast accuracy and confidence, and can be used to improve operational typhoon forecasts. Attached Figure Description

[0047] To more clearly illustrate the specific implementation methods of this embodiment or the technical solutions in the prior art, the drawings used in the description of the specific implementation methods or the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of this embodiment. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a flowchart illustrating a typhoon probability intelligent forecasting method based on multimodal data fusion, provided in an embodiment of the present invention.

[0049] Figure 2 This is a schematic diagram of the overall architecture of a typhoon probability intelligent forecasting model based on multimodal data fusion, provided in an embodiment of the present invention.

[0050] Figure 3 This is a schematic diagram of the probability forecasting network in a typhoon probability intelligent forecasting model based on multimodal data fusion, provided as an embodiment of the present invention.

[0051] Figure 4 This is a schematic diagram of a typhoon probability intelligent forecasting device based on multimodal data fusion, provided in an embodiment of the present invention.

[0052] Figure 5 This is a schematic diagram of a computer structure provided for an embodiment of the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this embodiment clearer, the technical solutions of this embodiment will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this embodiment, and not all embodiments. The components of this embodiment described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0054] Therefore, the following detailed description of the embodiments provided in the accompanying drawings is not intended to limit the scope of the claimed embodiments, but merely to illustrate selected embodiments of the present embodiments. All other embodiments obtained by those skilled in the art based on the embodiments in this work without inventive effort are within the scope of protection of this work.

[0055] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0056] Example

[0057] Investigations and analyses reveal that mainstream typhoon forecasting models based on dynamic numerical models comprehensively consider environmental airflow, landform distribution, and underlying surface factors. They utilize functions of typhoon movement patterns and apply numerical model analysis results to simulate typhoon paths, intensity changes, and associated rainfall. However, due to uncertainties in initial environmental conditions and numerical model equation errors, actual forecasts are prone to issues such as lost predictability and poor fault tolerance. Furthermore, while ensemble forecasting methods have shown advantages in improving typhoon path, intensity, and precipitation forecasts in recent years, many problems remain to be solved, such as the selection of ensemble members and the trade-off between computational cost and efficiency. Especially in practical operational forecasting systems, users often desire faster, more accurate, and more stable forecast results. Further exploration is needed to improve the adaptability, scalability, and stability of methods to meet application requirements.

[0058] Compared to numerical models that are complex to construct, require huge computing resources, and have long running times, current statistical typhoon forecasting has the advantages of flexibility, lightness, and real-time forecasting. In particular, in recent years, traditional machine learning models and shallow neural networks have achieved certain results in some specific forecasting tasks. However, they can only handle "point-to-point" forecasting tasks in form, and their ability to represent complex functions is limited with limited sample size and computing units. For "surface-to-point" and "surface-to-surface" forecasting tasks such as satellite cloud images and atmospheric and oceanic grids, they cannot directly utilize the interrelated structural information contained in high-dimensional remote sensing data and ocean-atmosphere coupled model data, which limits the forecast accuracy and generalization ability.

[0059] Therefore, this embodiment provides a data-driven intelligent typhoon probability forecasting method based on multimodal data fusion, and constructs an intelligent typhoon forecasting device based on this method.

[0060] Classical typhoon dynamics and thermodynamic mechanisms indicate that typhoon movement and intensity variations are influenced by multiple factors, including its own structure, surrounding weather systems, and large-scale environmental steering flows. In addition, typhoon activity exhibits climatological patterns over long timescales, and several climatological statistical factors are also significant in typhoon evolution. Multimodal and multivariate data can provide more comprehensive decision variables for typhoon lifecycle forecasting. For example, satellite infrared cloud images and water vapor cloud images can provide information about the cyclone's own structure; ocean-atmosphere gridded data can provide information about surrounding weather systems, large-scale circulation, and sea surface temperature, among other factors influencing typhoon evolution; and existing statistical models and dynamic-statistical models can provide statistical variables related to environmental information, climatology, and persistence from another perspective, improving typhoon track, intensity, and precipitation forecasts.

[0061] Therefore, this embodiment comprehensively utilizes the variables of the three modalities mentioned above, including but not limited to multi-channel infrared satellite images, passive microwave images, sea surface temperature, meridional wind, zonal wind, temperature, geopotential height, precipitation, water vapor content in the air, relative humidity, and several statistical-dynamic factors, to construct a deep learning-based multimodal variable fusion model for forecasting typhoon intensity, path, and precipitation field. Furthermore, probability distribution estimation physics is incorporated into the autoregressive training to describe the typhoon's uncertainty distribution and capture the expected development and extreme conditions of the typhoon. This will help to further improve the forecasting errors of typhoon path, intensity, and precipitation, providing more reliable suggestions for typhoon forecasting.

[0062] The following detailed description of some embodiments of this invention is provided in conjunction with the accompanying drawings. The multimodal variables described herein are only a portion, not all, of the embodiments. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0063] Figure 1 This embodiment provides a flowchart illustrating a method for constructing a typhoon probability intelligent forecasting method based on multimodal data fusion. The process includes the following three steps:

[0064] A. The construction of the typhoon intelligent forecasting model includes the following steps:

[0065] A.1. Based on typhoon-related atmospheric dynamics and thermodynamics models, select and collect multimodal and multi-element meteorological data related to typhoons, including data from at least three modes such as satellite and radar remote sensing images, reanalysis gridded data, and statistical dynamic factors. Construct a multimodal data prediction and processing method. Use records from the typhoon's optimal path data as an index to match multimodal and multivariate input factors; the multimodal and multivariate data undergoes preprocessing processes such as interpolation and normalization, spatiotemporal resolution adjustment, and autoregressive sequence construction.

[0066] A.2. For data with different modalities and different physical variables, determine the appropriate guidance time and appropriate coverage area, and use a feature coding network to extract multimodal and multivariate spatiotemporal features, such as... Figure 2 As shown in module A.2, this module is a set of multimodal multivariate feature coding networks provided in this embodiment. The calculation formulas for this module are as follows:

[0067]

[0068] in, This represents remote sensing images of the same typhoon from multiple detection methods within 24 hours prior to the reporting time. These images are overlaid on the channel. CNN_encoder represents a remote sensing image encoding network based on convolutional neural networks, such as Resnet_50, used to extract and encode features in the remote sensing images.

[0069]

[0070] in, There are j statistical-dynamic forecasting factors, which are derived from existing statistical-dynamic models such as SHIPS and STIPS. SDF_encoder is an entity embedding encoding network used to embed one-dimensional structured data into a high-dimensional continuous space.

[0071]

[0072] in, The environmental variables are composed of k physical quantities, forming an ocean-atmosphere grid. The ENV_encoder is the ocean-atmosphere grid encoding network. In this embodiment, a U-net structure is used for the encoding part, which is a convolutional neural network. The output is a one-dimensional feature vector formed by expanding the intermediate bottleneck vector, serving as the spatiotemporal feature encoding of the ocean-atmosphere grid. It also outputs a multi-scale precipitation feature map Z.RAIN .

[0073] A.3. Based on the Transformer fusion network, capture the coupling and interaction process of multimodal and multi-factor variables, determine cross-modal feature dependencies, and aggregate multimodal features;

[0074] Input features from multiple modalities are first categorized into primary and secondary modalities. For different downstream tasks, the classification of primary and secondary modalities depends on the degree of correlation between the input features and the regression target. In this embodiment, for typhoon track and intensity forecasting, the primary modal data consists of remote sensing images and statistical-dynamic factors, while the secondary modal data consists of ocean-atmosphere gridded data. For typhoon rainfall forecasting, the primary modal data consists of ocean-atmosphere gridded data and remote sensing images, while the secondary modal data consists of statistical-dynamic factors. Here, the primary modal data is collectively referred to as Z. α The auxiliary modal data is Z β .

[0075] For example, regarding typhoon intensity and track forecasts: Z α =Concat(Z) IMG Z SDF Z β =Z ENV Concat(·) is the feature concatenation function.

[0076] Preferably, this embodiment designs a multi-head self-attention feature fusion method that distinguishes between the primary modality and the auxiliary modality. For example... Figure 2 As shown in module A.3, the primary mode can receive information from all modes, such as... Figure 3 As shown, the Transformer fusion network A.3 receives feature inputs from the primary and auxiliary modalities. The transformers from layers 1 to D are defined by the following formula:

[0077]

[0078]

[0079]

[0080]

[0081] Where LN is the Layer Norm function and MHSA is the multi-head self-attention function. Here, `Concat` is the channel mapping function for a multilayer perceptron, and `Concat` is the vector concatenation function. This means that features from all modalities are aggregated onto the main modality features and updated accordingly. Features from auxiliary modalities only update themselves and do not aggregate information from other modalities. and Let be the transformation matrices of Q, V, and K in a self-attention network.

[0082] A.4. Construct probabilistic forecasting networks and precipitation forecasting networks for typhoon intensity, track, and precipitation forecasts respectively, and perform autoregressive forecasting using the fused features.

[0083] The Rain_decoder(·) network used for typhoon rainfall forecasting is the decoding part of the U-net structure of the ocean-atmosphere grid coding network in A.2, and the calculation formula is as follows:

[0084]

[0085] This network uses fused multimodal features and multi-scale rainfall feature maps extracted by an ocean-atmosphere grid coding network as input to forecast typhoon rainfall. The multi-scale rainfall feature maps are generated by the ocean-atmosphere grid coding network, and high-resolution rainfall features are directly passed to the rainfall forecasting network using skip connections. This approach maintains feature consistency, ensuring that features can be reliably converted into pixel-dense prediction results.

[0086] Probabilistic forecasting networks used for predicting typhoon intensity and track, such as Figure 3 As shown in A.4.1. Preferably, the probabilistic prediction network outputs prediction results with a probability distribution based on a recurrent neural network (RNN, such as a long short-term memory network LSTM) and a hybrid density network (MDN). The input of this network is the output of the previous time step. The encoding vector a of the sequence position t There is also the hidden feature Z from the Transformer fusion network. t This is used to predict the intensity and path of the next time step.

[0087]

[0088] This is a non-probabilistic prediction result. This output can serve as a supervisory signal during the early stages of model training, replacing the loss function of the MDN network and accelerating the model training process. The MDN network is then enabled during the later stages of training and when the model is making inference predictions.

[0089] Based on the theory that a weighted sum of multiple Gaussian distributions can approximate any conditional probability distribution, the model output can be expressed as the weighted sum of K Gaussian distributions, each with a different mean, variance, and weight. τ, as a temperature variable, is used to control the uncertainty of the model output. The formula is as follows:

[0090]

[0091] in, Represents the normal distribution function. P represents different... The probability density of the values. The mean, variance, and weights of different Gaussian members require three additional neural networks for fitting. The loss function of the MDN network during training is determined by the maximum likelihood estimation principle, which ensures that the probability of the label value in the observed samples is as large as possible:

[0092]

[0093] Where N is the number of samples, and ω is the parameter of the MDN network. During model inference and prediction, MDN can use a sampling function to obtain a specific predicted value, i.e.:

[0094]

[0095] B. Training the typhoon intelligent forecasting model includes the following steps:

[0096] B.1. Select multimodal and multi-factor data related to typhoon forecasting and construct a training dataset;

[0097] Multimodal and multi-element meteorological data can be obtained through publicly available third-party services online. For example, ocean-atmosphere gridded data comes from the ERA5 dataset of the European Centre for Medium-Range Weather Forecasts. Similarly, remote sensing satellite images of typhoons can be obtained from various observational data sources such as the digital Typhoon Network and TCIR (Dataset of Tropical Cyclone for Image-to-intensity Regression). Statistical-dynamic factors can be calculated independently or obtained from development data publicly available from the SHIPS (Statistical Hurricane Intensity Prediction Scheme) project.

[0098] B.2. Select appropriate temporal and spatial resolutions for each variable and perform feature engineering preprocessing on the data;

[0099] After preprocessing in step A.1, the multimodal and multi-factor variables are standardized using the following formula for both high-dimensional ocean-atmosphere gridded data and remote sensing satellite image data:

[0100]

[0101] in, and σ x These are the mean and standard deviation of the input data, x, respectively.* This is the result after data preprocessing.

[0102] This embodiment standardizes data for different physical variables and even the same physical quantity on different isobaric surfaces. Specifically, for total precipitation (TP) data, due to its significant long-tailed distribution, this embodiment processes it as follows:

[0103] TP * =log(1+TP / ∈)

[0104] Where ∈=1×10 -5 This processing can make the model predict rainfall data to tend to be 0 and weaken the impact of the long-tailed distribution.

[0105] B.3. Using individual typhoon cases as an index, randomly divide the training set, validation set, and test set in a ratio of 8:1:1. Use the training set for sufficient model training, use the validation set to supervise the training effect, and use the test set to calculate the model's forecasting skills.

[0106] During model training, this embodiment employs techniques including residual connections, early termination, network optimal solution ensemble, parameter regularization, random neuron deactivation, and random depth to avoid problems such as overfitting, gradient explosion, network degradation, and slow convergence.

[0107] In addition to the probability prediction error Loss described in step A.4 MDN It also incorporates traditional mean squared error to guide the model through thorough training and accelerate convergence.

[0108]

[0109] Where l represents the final objective function, and σ1 and σ2 are hyperparameters for adjusting the proportion of different loss terms.

[0110] C. The inference and prediction of the typhoon intelligent forecasting model specifically includes the following steps:

[0111] C.1. Based on the multimodal and multivariate initial meteorological data at the reporting time, determine the initial forecast field and perform the same data preprocessing as in step B.2, which will then serve as the input for the typhoon forecast model;

[0112] C.2. Input the preprocessed forecast initial field into the feature encoding network A.2 of each modality data for feature extraction and dimensionality reduction, and encode the spatiotemporal features of the multimodal data;

[0113] C.3. Input the multimodal features into the Transformer fusion network A.3 to determine cross-modal feature dependencies and aggregate the multimodal features;

[0114] C.4. Input the aggregated multimodal features into the probabilistic forecasting network A.4 to perform sequence forecasts of typhoon path, intensity, and precipitation, and obtain the probability distribution of the forecast values ​​after the superposition of mixed Gaussian distributions; in addition, combine the different scale feature codes output by the ocean-atmosphere grid coding network in A.2 with the aggregated multimodal features, input them into the precipitation forecasting network, and obtain the forecast results of the precipitation field.

[0115] C.5. Post-process typhoon forecast results and calculate evaluation indicators.

[0116] Preferably, this embodiment uses RMSE (Root Mean Square Error), MAE (Mean Absolute Error), and bias (Overall Deviation) to evaluate the forecasting skills for typhoon track and intensity; for rainfall forecasting, a correlation coefficient r is introduced to measure the closeness of the linear correlation between the measured rainfall and the forecasted rainfall, and a critical success index (CSI) is used to evaluate the accuracy of rainfall location forecasts at different rainfall rates.

[0117] Figure 4 This embodiment provides a schematic diagram of a typhoon probability intelligent forecasting device based on multimodal data fusion, the contents of which include:

[0118] D.1. The first determining module is used to determine the initial forecast factors based on multimodal and multivariate meteorological data from the reporting time.

[0119] D.2. Data preprocessing module, used to convert initial forecast factors into the input format specified by the model;

[0120] D.3. Inference and Prediction Module: This module is used to deploy a trained typhoon intelligent forecasting model on selected hardware, run the model, and obtain prediction results.

[0121] D.4. Results Collection Module: Used for post-processing of output, collecting field data and evaluating the model's forecasting ability.

[0122] The typhoon intelligent forecasting device provided in this embodiment has the same technical features as the typhoon probability intelligent forecasting method based on multimodal data fusion provided in the above embodiment, so it can also solve the same technical problems and achieve the same technical effects.

[0123] Figure 5The computer device E provided in this embodiment can be used to run a typhoon probability intelligent forecasting method based on multimodal data fusion. It includes: a memory E.1, a processor E.2, a graphics processing card E.3, and a bus. The memory E.1 stores machine-readable instructions executable by the processor E.2 and the graphics processing card E.3. When the computer device E is running, the processor E.2, the graphics processing card E.3, and the memory E.1 communicate via the bus. The processor E.2 and the graphics processing card E.3 jointly execute the machine-readable instructions to perform the steps of the typhoon probability forecasting based on multimodal data fusion as described above.

[0124] Specifically, the aforementioned memory E.1, processor E.2, and graphics processing card E.3 can be general-purpose memory, processor, and graphics processing card; no specific limitations are made here. When processor E.2 and graphics processing card E.3 run the computer program stored in memory E.1, they can execute the aforementioned typhoon probability intelligent forecasting method based on multimodal data fusion. Graphics processing card E.3 executes instructions related to the typhoon intelligent forecasting model, and processor E.2 executes other control instructions, such as the control of input and output flows.

[0125] Corresponding to the aforementioned intelligent typhoon probability forecasting method based on multimodal data fusion, this embodiment also provides a computer-readable storage medium that can be used to store the intelligent typhoon probability forecasting method based on multimodal data fusion. The computer-readable storage medium stores machine-executable instructions. When these machine-executable instructions are invoked and executed by a processor, they cause the processor to perform the steps of the aforementioned typhoon probability forecasting model based on multimodal data fusion.

[0126] The typhoon probability forecasting device based on multimodal data fusion provided in this embodiment can be specific hardware on the device or software or firmware installed on the device. The implementation principle and technical effects of the device provided in this embodiment are the same as those in the aforementioned method embodiments. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiments. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can all be referred to the corresponding processes in the above method embodiments, and will not be repeated here.

[0127] In the embodiments provided in this example, it should be understood that the disclosed apparatus and method can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and there may be other division methods in actual implementation. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the coupling or direct coupling or communication connection shown or discussed may be through some communication interface; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0128] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0129] In the several embodiments provided in this embodiment, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this embodiment. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0130] In addition, the functional units in the embodiments provided in this embodiment can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0131] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the mobile control methods of various embodiments of this embodiment. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0132] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the term "first" is used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0133] Finally, it should be noted that the above embodiments are merely specific implementations of this embodiment, used to illustrate the technical solution of this embodiment, and not to limit it. The protection scope of this embodiment is not limited thereto. Although this embodiment has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the technical scope disclosed in this embodiment; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of this embodiment. All should be covered within the protection scope of this embodiment.

Claims

1. A method for intelligent typhoon probability forecasting based on multimodal data fusion, characterized in that, The method includes the following steps: A. Construct a smart typhoon forecasting model; B. Training intelligent models for typhoon forecasting; C. Use the trained intelligent typhoon forecasting model to make inferences and predictions; Step A specifically includes the following steps: A.

1. Select and collect multimodal and multi-element meteorological data related to typhoons based on atmospheric dynamics and thermodynamics models, including data from three modes: satellite and radar remote sensing images, reanalysis gridded data, and statistical dynamic factors. Construct a multimodal data preprocessing module. Using records in the typhoon optimal path data as an index, match multimodal and multivariate input factors, and perform preprocessing processes such as interpolation and normalization, spatiotemporal resolution adjustment, and autoregressive sequence construction on the multimodal and multivariate data. A.

2. For data of different modalities and different physical variables, determine the guidance time and coverage area, construct a multimodal multivariate feature coding network, and extract the spatiotemporal features of remote sensing images, ocean-atmosphere grids and statistical-dynamic factors through remote sensing image coding network, ocean-atmosphere grid coding network and entity embedding coding network respectively; A.

3. Based on the Transformer fusion network, capture the coupling and interaction process of multimodal and multi-factor variables, determine cross-modal feature dependencies, and fuse multimodal features; A.

4. Construct a probabilistic forecasting network for typhoon intensity and path, using the fused multimodal features as input to complete typhoon path and intensity forecasts. Construct a rainfall forecasting network for typhoon rainfall, using the fused multimodal features and multi-scale rainfall feature maps extracted by the ocean-atmosphere grid coding network as input to complete typhoon rainfall forecasts.

2. The intelligent typhoon probability forecasting method based on multimodal data fusion according to claim 1, characterized in that, In step A.2, the encoding expression of the multimodal multivariate feature encoding network is: ) in, The data consisted of remote sensing imagery, statistical-dynamic factor data, and environmental-atmosphere gridded data collected within the 24 hours prior to the reporting time. Each modality was encoded using a separate feature encoder. , These are remote sensing images, spatiotemporal characteristics of statistical-dynamic factors, and ocean-atmosphere grid points, respectively. This is a multi-scale rainfall feature map.

3. The intelligent typhoon probability forecasting method based on multimodal data fusion according to claim 1, characterized in that, In step A.4, for typhoon track and intensity forecasting, the feature data of remote sensing images and statistical-dynamic factors are used as the primary mode, and the feature data of ocean-atmosphere gridded features are used as the auxiliary mode; for typhoon rainfall forecasting, the feature data of ocean-atmosphere gridded data and remote sensing images are used as the primary mode, and the feature data of statistical-dynamic factors are used as the auxiliary mode, by constructing... The fusion network employs a multi-head self-attention feature fusion approach, aggregating features from all modalities onto the main modality feature and updating the main modality feature. Auxiliary modal features only update themselves and do not aggregate information from other modalities. Therefore, the computational expression based on the Transformer fusion network is: in, Indicates the initial dominant modality features. This represents the initial auxiliary modal features, with the superscript D indicating the D-th layer. To aggregate all modal information into the first modality after the main modality Characteristics of the layer.

4. The intelligent typhoon probability forecasting method based on multimodal data fusion according to claim 1, characterized in that, In step A.4, the calculation expression for the rainfall forecasting network is as follows: in, For rainfall forecast feature decoder, For forecast The output of the rainfall field contained in each time step For the fused multimodal features, This is a multi-scale rainfall feature map.

5. The intelligent typhoon probability forecasting method based on multimodal data fusion according to claim 1, characterized in that, In step A.4, the calculation expression for the probabilistic prediction network is as follows: in, The intensity and path forecast values ​​are from the previous time step, and their distribution is determined by the Gaussian mixture model. It is obtained by fitting, that is, decomposing the variable distribution into a linear combination of multiple Gaussian distribution functions. The encoding vector for the sequence position. for The key features come from the fusion characteristics of the Transformer fusion network, where RNN stands for Recurrent Neural Network. Represents the normal distribution function, and its weighting coefficients and parameters Fitted by a neural network, For different The probability density of the values ​​is the probability prediction result. k Indicates the first k Gaussian distribution function, K The number of Gaussian distribution functions in the Gaussian mixture model; Temperature is used as a variable to control the uncertainty of the model output.

6. The intelligent typhoon probability forecasting method based on multimodal data fusion according to claim 5, characterized in that, Step B specifically includes the following steps: B.1 Select multimodal and multi-factor data related to typhoon forecasting and construct a training dataset; B.

2. Select appropriate time and spatial resolutions for each variable and perform feature engineering data preprocessing; B.

3. Using individual typhoon cases as an index, randomly divide the training set, validation set, and test set in a ratio of 8:1:

1. Use the training set for thorough training, use the validation set to supervise the training effect, and use the test set to calculate the model's forecasting skills.

7. The typhoon probability intelligent forecasting method based on multimodal data fusion according to claim 6, characterized in that, Step C specifically includes the following steps: C.

1. Based on the multimodal and multivariate initial meteorological data at the reporting time, determine the initial forecast field and perform the same data preprocessing as in step B.2, which serves as the input for the typhoon forecasting method; C.

2. Input the preprocessed prediction initial field into the feature encoding network of each modality data for feature extraction and dimensionality reduction, and encode the spatiotemporal features of the multimodal data; C.

3. Input multimodal features into the Transformer fusion network, determine cross-modal feature dependencies, and aggregate multimodal features; C.

4. Input the aggregated multimodal features into the probabilistic forecasting network to perform sequence forecasts of typhoon path, intensity, and precipitation, and obtain the probability distribution of the forecast values ​​after the superposition of mixed Gaussian distributions; combine the different scale feature codes output by the ocean-atmosphere grid coding network with the aggregated multimodal features, input them into the precipitation forecasting network, and obtain the forecast results of the precipitation field; C.

5. Process typhoon forecast results and calculate evaluation indicators.

8. A typhoon probability intelligent forecasting device based on multimodal data fusion, characterized in that, The apparatus performs the method according to any one of claims 1 to 7, the apparatus comprising: The first determination module is used to determine the initial forecast factors based on multimodal and multivariate meteorological data from the reporting time. Data preprocessing module: used to convert initial forecast factors into the input format specified by the method; Inference and prediction module: used to deploy and run trained intelligent typhoon forecasting methods on selected hardware to obtain prediction results; Results collection module: Used for post-processing of output, including: collecting field data, calculating various indices and evaluation indicators.

Images