Data fusion method and device, equipment and medium
By preprocessing meteorological observation datasets and enhancing image features, combined with a multimodal data fusion model, the problem of poor meteorological data fusion effect in existing technologies has been solved, and efficient fusion and accurate analysis of complex meteorological data have been achieved.
Patent Information
- Application Number
- CN202511053947.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-29
- Publication Date
- 2025-11-07
AI Technical Summary
In existing technologies, meteorological data fusion methods are difficult to effectively handle complex meteorological data, resulting in poor data fusion effects and failing to meet the needs of new energy power generation stations for high-quality meteorological early warning.
By acquiring meteorological observation datasets, preprocessing text data and enhancing the features of image data, and using a multimodal data fusion model to fuse data, a meteorological data fusion matrix is generated, thereby improving the semantic expressiveness of the data and the discriminative power of image modalities.
It significantly improves the expressive power and learning efficiency of multimodal fusion models, realizes collaborative understanding and deep representation of multi-source meteorological information, and enhances the accuracy and generalization ability of downstream meteorological analysis tasks.
Smart Images
Figure CN120910795A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of data processing, and in particular to a data fusion method, device, equipment and medium. BACKGROUND
[0002] In recent years, new energy power generation has developed rapidly. New energy power generation stations are usually widely distributed, have weak resistance to meteorological disasters, and have stronger demand for fine, industrialized, and indexed high-quality meteorological warning than traditional power generation stations. For meteorological data, the data sources are diverse, the data volume is large, the data structure is complex, there are time and space deviations, and it is often difficult to obtain data with unified sources and structures.
[0003] In the prior art, a commonly used data fusion method is usually based on statistical analysis. Specifically, first, meteorological data from different channels are preprocessed, including missing value filling, format standardization, time alignment, and the like. Then, statistical methods such as weighted average, least variance estimation, and Kalman filtering are used to fuse multi-source data to obtain a data fusion result.
[0004] However, the data fusion method in the prior art is difficult to effectively deal with complex meteorological data, and the data fusion effect is poor. SUMMARY
[0005] Embodiments of the present application provide a data fusion method, device, equipment and medium to solve the problem that the data fusion method in the prior art is difficult to effectively deal with complex meteorological data and the data fusion effect is poor.
[0006] In a first aspect, an embodiment of the present application provides a data fusion method, comprising:
[0007] Obtaining a meteorological observation data set, the meteorological observation data set comprising a meteorological observation text data set and a meteorological observation image data set;
[0008] Preprocessing the meteorological observation text data set to obtain a first meteorological data set;
[0009] Performing image feature enhancement processing on the meteorological observation image data set to obtain a second meteorological data set;
[0010] Inputting the first meteorological data set and the second meteorological data set into a multi-modal data fusion model for data fusion to obtain a meteorological data fusion matrix; the multi-modal data fusion model is obtained by pre-training according to historical meteorological observation data sets; and the meteorological data fusion matrix is used to input into other task models for data analysis and processing.
[0011] In a possible implementation, the meteorological observation text dataset includes single-source data and multi-source data, and the single-source data and the multi-source data include a plurality of meteorological features;
[0012] The preprocessing of the meteorological observation text dataset to obtain first meteorological data includes:
[0013] For each meteorological feature in the single-source data, a probability distribution of an observation value sequence of the meteorological feature is obtained through probabilistic statistics of the observation value sequence;
[0014] According to the probability distribution of each observation value sequence, a difference coefficient of the meteorological feature is calculated;
[0015] For each meteorological feature in the multi-source data, a Mahalanobis distance correlation coefficient of an observation value sequence of the meteorological feature is calculated;
[0016] According to the difference coefficient of each meteorological feature and the Mahalanobis distance correlation coefficient of the meteorological feature, the meteorological observation text dataset is screened to obtain a first meteorological dataset.
[0017] In a possible implementation, the screening of the meteorological observation text dataset according to the difference coefficient of each meteorological feature and the Mahalanobis distance correlation coefficient of the meteorological feature to obtain a first meteorological dataset includes:
[0018] It is judged whether the difference coefficient of each meteorological feature is greater than a preset threshold value;
[0019] If yes, the single-source data corresponding to the meteorological feature is determined as first valid data;
[0020] It is judged whether the Mahalanobis distance correlation coefficient of each meteorological feature is less than a preset coefficient;
[0021] If yes, the multi-source data corresponding to the meteorological feature is determined as second valid data;
[0022] According to the first valid data and the second valid data, a first meteorological dataset is obtained.
[0023] In a possible implementation, the meteorological observation image dataset is subjected to image feature enhancement processing to obtain a second meteorological dataset, which includes:
[0024] For each meteorological observation image data, a feature enhancement strategy of the meteorological observation image data is determined through a pre-constructed differentiable automatic data enhancement model, and the feature enhancement strategy includes at least one of image reversal, color processing, and image flipping;
[0025] According to the feature enhancement strategy, the meteorological observation image data is subjected to image feature enhancement processing to obtain enhanced meteorological observation image data.
[0026] According to each of the enhanced meteorological observation image data, a second meteorological data set is obtained.
[0027] In a possible implementation, the training process of the differentiable automatic data enhancement model comprises:
[0028] A sample meteorological observation image data set is obtained.
[0029] For each sample meteorological observation image data, a sample feature enhancement strategy is randomly generated by the differentiable automatic data enhancement model.
[0030] At least one feature enhancement sub-strategy and a probability weight of the feature enhancement sub-strategy are determined from the sample feature enhancement strategy by using a classification distribution;
[0031] The application intensity of the feature enhancement sub-strategy is determined by using a Bernoulli distribution;
[0032] According to the probability weight and the application intensity, a loss function of the sample meteorological observation image data is calculated;
[0033] The loss function is subjected to expectation estimation by using a Monte Carlo method to train the differentiable automatic data enhancement model.
[0034] In a possible implementation, the inputting of the first meteorological data set and the second meteorological data set into the multi-modal data fusion model for data fusion to obtain a meteorological data fusion matrix comprises:
[0035] A first digital sequence vector is extracted from the first meteorological data set by using a text modal compiler of the multi-modal data fusion model;
[0036] A second digital sequence vector is extracted from the second meteorological data set by using an image modal compiler of the multi-modal data fusion model;
[0037] The first digital sequence vector and the second digital sequence vector are subjected to data fusion by using an orthogonal sequence fusion mechanism to obtain a meteorological data fusion matrix.
[0038] In a possible implementation, the training process of the multi-modal data fusion model comprises:
[0039] A sample first meteorological data set, a sample second meteorological data set and a historical meteorological data fusion matrix are obtained.
[0040] extracting, by a text modality compiler of the multi-modal data fusion model, a sample first digital sequence vector from the sample first meteorological data set;
[0041] extracting, by an image modality compiler of the multi-modal data fusion model, a sample second digital sequence vector from the sample second meteorological data set;
[0042] fusing, by using an orthogonal sequence fusion mechanism, the first digital sequence vector and the second digital sequence vector to obtain a sample meteorological data fusion matrix;
[0043] training the multi-modal data fusion model based on an orthogonal sequence fusion loss calculated based on the sample meteorological data fusion matrix and the historical meteorological data fusion matrix.
[0044] In a second aspect, an embodiment of the present application provides a data fusion device, comprising:
[0045] an acquisition module configured to acquire a meteorological observation data set, the meteorological observation data set comprising a meteorological observation text data set and a meteorological observation image data set;
[0046] a first processing module configured to pre-process the meteorological observation text data set to obtain a first meteorological data set;
[0047] a second processing module configured to perform image feature enhancement processing on the meteorological observation image data set to obtain a second meteorological data set;
[0048] a fusion module configured to input the first meteorological data set and the second meteorological data set into a multi-modal data fusion model to perform data fusion, and obtain a meteorological data fusion matrix; the multi-modal data fusion model is pre-trained based on a historical meteorological observation data set; and the meteorological data fusion matrix is used to input into other task models for data analysis and processing.
[0049] In a possible implementation, the meteorological observation text data set comprises single-source data and multi-source data, and the single-source data and the multi-source data contain a plurality of meteorological features; and the first processing module is specifically configured to:
[0050] for each meteorological feature in the single-source data, performing probability statistics on an observation value sequence of the meteorological feature to obtain a probability distribution of the observation value sequence;
[0051] calculating a difference coefficient of the meteorological feature according to the probability distribution of each observation value sequence;
[0052] for each meteorological feature in the multi-source data, calculating a Mahalanobis distance correlation coefficient of an observation value sequence of the meteorological feature;
[0053] According to the difference coefficient of each meteorological feature and the Mahalanobis distance correlation coefficient of the meteorological feature, the meteorological observation text data set is screened to obtain a first meteorological data set.
[0054] In a possible implementation, the first processing module is specifically configured to:
[0055] determine whether the difference coefficient of each meteorological feature is greater than a preset threshold value;
[0056] If yes, it is determined that the single-source data corresponding to the meteorological feature is first effective data.
[0057] determine whether the Mahalanobis distance correlation coefficient of each meteorological feature is less than a preset coefficient;
[0058] If yes, it is determined that the multi-source data corresponding to the meteorological feature is second effective data.
[0059] According to the first effective data and the second effective data, a first meteorological data set is obtained.
[0060] In a possible implementation, the second processing module is specifically configured to:
[0061] For each meteorological observation image data, a feature enhancement strategy of the meteorological observation image data is determined through a pre-constructed differentiable automatic data enhancement model, and the feature enhancement strategy includes at least one of image reversal, color processing, and image flipping.
[0062] According to the feature enhancement strategy, image feature enhancement processing is performed on the meteorological observation image data to obtain enhanced meteorological observation image data.
[0063] According to each of the enhanced meteorological observation image data, a second meteorological data set is obtained.
[0064] In a possible implementation, the second processing module is specifically configured to:
[0065] obtain a sample meteorological observation image data set;
[0066] For each sample meteorological observation image data, a sample feature enhancement strategy is randomly generated through the differentiable automatic data enhancement model;
[0067] At least one feature enhancement sub-strategy and a probability weight of the feature enhancement sub-strategy are determined from the sample feature enhancement strategy by using a classification distribution;
[0068] The application intensity of the feature enhancement sub-strategy is determined by using a Bernoulli distribution.
[0069] According to the probability weight and the application intensity, a loss function of the sample meteorological observation image data is calculated;
[0070] A Monte Carlo method is used to perform expectation estimation on the loss function to train the differentiable automatic data enhancement model.
[0071] In a possible implementation, the fusion module is specifically configured to:
[0072] A first digital sequence vector is extracted from the first meteorological data set by a text modality compiler of the multi-modal data fusion model;
[0073] A second digital sequence vector is extracted from the second meteorological data set by an image modality compiler of the multi-modal data fusion model;
[0074] The first digital sequence vector and the second digital sequence vector are fused by using an orthogonal sequence fusion mechanism to obtain a meteorological data fusion matrix.
[0075] In a possible implementation, the fusion module is specifically configured to:
[0076] A sample first meteorological data set, a sample second meteorological data set, and a historical meteorological data fusion matrix are acquired;
[0077] A sample first digital sequence vector is extracted from the sample first meteorological data set by a text modality compiler of the multi-modal data fusion model;
[0078] A sample second digital sequence vector is extracted from the sample second meteorological data set by an image modality compiler of the multi-modal data fusion model;
[0079] The first digital sequence vector and the second digital sequence vector are fused by using an orthogonal sequence fusion mechanism to obtain a sample meteorological data fusion matrix;
[0080] The multi-modal data fusion model is trained based on an orthogonal sequence fusion loss calculated from the sample meteorological data fusion matrix and the historical meteorological data fusion matrix.
[0081] In a third aspect, an electronic device is provided, including a memory and a processor.
[0082] The memory stores computer execution instructions.
[0083] The processor executes the computer execution instructions stored in the memory, so that the processor performs the first aspect and / or various possible implementations of the first aspect.
[0084] In a fourth aspect, the embodiments of the present application provide a computer readable storage medium, the computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by a processor to implement the first aspect and / or various possible implementation manners of the first aspect.
[0085] The data fusion method, device, equipment and medium provided by the embodiments of the present application lay a data foundation for subsequent multi-modal fusion by first acquiring a meteorological observation dataset; then, the meteorological observation text dataset is preprocessed to obtain a first meteorological dataset; then, the meteorological observation image dataset is subjected to image feature enhancement processing to obtain a second meteorological dataset. In this way, the semantic expression ability and computability of the text data are improved, and the key visual information in the image data is strengthened, thereby improving the discriminability and robustness of the image modal, significantly improving the expression ability and learning efficiency of the subsequent multi-modal fusion model, laying a foundation for the final data fusion and analysis. Finally, the first meteorological dataset and the second meteorological dataset are input into a multi-modal data fusion model for data fusion to obtain a meteorological data fusion matrix. The data fusion method fuses meteorological observation data of two modalities of text and image, realizes collaborative understanding and deep representation of multi-source meteorological information, and thereby improves the accuracy and generalization ability of downstream meteorological analysis tasks. BRIEF DESCRIPTION OF DRAWINGS
[0086] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0087] Figure 1 Flowchart of the data fusion method provided by the embodiments of the present application Figure 1 ;
[0088] Figure 2 Flowchart of the data fusion method provided by the embodiments of the present application Figure 2 ;
[0089] Figure 3 Structure diagram of the data fusion device provided by the embodiments of the present application;
[0090] Figure 4 Structure diagram of the electronic device provided by the embodiments of the present application.
[0091] The above drawings have shown the specific embodiments of the present application, and more detailed descriptions will be given hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of the present application by any means, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION
[0092] The exemplary embodiments will be described in detail below with reference to the drawings. In the following description, like reference numerals refer to like elements throughout the description. The following exemplary embodiments described below are not representative of all embodiments consistent with the present application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.
[0093] In recent years, new energy power generation has developed rapidly. New energy power generation stations are usually widely distributed, have weak resistance to meteorological disasters, and have stronger demand for fine, industrialized, and indexed high-quality meteorological warning than traditional power generation stations. For meteorological data, the data sources are diverse, the data volume is large, the data structure is complex, there are time and space deviations, and it is often difficult to obtain data with unified sources and structures.
[0094] In the prior art, a commonly used data fusion method is usually based on statistical analysis. Specifically, first, meteorological data from different channels are preprocessed, including missing value filling, format standardization, time alignment, etc. Then, statistical methods such as weighted average, minimum variance estimation, and Kalman filtering are used to fuse multi-source data to obtain a data fusion result.
[0095] However, the data fusion method in the prior art relies on the distribution assumption and linear correlation of the data, and it is difficult to effectively deal with complex meteorological data that is nonlinear, non-Gaussian, and multi-scale. More importantly, the statistical method is difficult to fully mine the potential semantic association between multi-modal data, and the fusion result lacks deep feature expression, which limits its application effect in practical applications.
[0096] Therefore, in view of the problems of the data fusion method in the prior art, such as simple structure, weak feature extraction capability, and insufficient modality cooperation, when dealing with complex meteorological data, it is difficult to fully mine the potential association between multi-source meteorological information, resulting in poor fusion effect. The present application proposes a data fusion method. The inventors found that meteorological text data (such as observation records and sensor indicators) and meteorological image data (such as radar charts and satellite cloud charts) have information complementarity. The text expresses quantitative trends, and the image reflects spatial structure. By inputting the structured text data after preprocessing and the image data after image enhancement into a multi-modal data fusion model, the deep feature extraction network and the attention mechanism are used to align and fuse different modal features, thereby constructing a meteorological data fusion matrix with stronger expression capability. The fusion model is trained based on historical meteorological observation data, so that it has good generalization ability based on understanding the distribution pattern of meteorological data, realizes accurate information integration and representation in complex meteorological scenarios, and significantly improves the fusion quality and the accuracy and robustness of data analysis.
[0097] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.
[0098] Figure 1 Flowchart of the data fusion method provided in the embodiments of this application Figure 1 ;like Figure 1 As shown, the method includes:
[0099] S101. Obtain meteorological observation dataset.
[0100] The meteorological observation dataset includes meteorological observation text datasets and meteorological observation image datasets. The meteorological observation text dataset includes single-source data (such as data from a single observation station) and multi-source data (such as data from multiple stations and multiple systems). Both single-source and multi-source data contain multiple meteorological features (such as temperature, humidity, air pressure, wind speed, etc.).
[0101] Understandably, multi-source data can improve data coverage. Image data introduces spatial distribution and meteorological features, laying the data foundation for subsequent multimodal fusion and enhancing the perception dimension and expressive richness.
[0102] S102. Preprocess the meteorological observation text dataset to obtain the first meteorological dataset.
[0103] In one feasible approach, for each meteorological feature in single-source data, probability statistics are performed on the observation sequence of the meteorological feature to obtain the probability distribution of the observation sequence; then, based on the probability distribution of each observation sequence, the dissimilarity coefficient of the meteorological feature is calculated; for each meteorological feature in multi-source data, the Mahalanobis distance correlation coefficient of the observation sequence of the meteorological feature is calculated; finally, the meteorological observation text dataset is filtered based on the dissimilarity coefficient of each meteorological feature and the Mahalanobis distance correlation coefficient of the meteorological feature to obtain the first meteorological dataset.
[0104] It should be noted that for single-source data, i.e., data from the same observation equipment or in the same data format (such as messages, logs, etc.), a consistent data structure is required. In this case, whether a text sample is "redundant" or "representative" can be determined based on the degree of content difference between it and other meteorological features.
[0105] For multi-source data, there are often differences in dimensionality and statistical distribution. Mahalanobis distance can eliminate the influence between different dimensions and scales. By analyzing the linear correlation and redundancy between various modalities / data sources, a multi-source screening strategy can be implemented to maximize information complementarity and minimize redundancy.
[0106] It should be understood that the observation value sequence of each meteorological feature is subjected to probability statistics, and then the difference coefficient (such as the coefficient of variation, information entropy, skewness / kurtosis, etc.) is calculated according to the distribution to measure the volatility and expressiveness of the meteorological feature in different time periods; for the same meteorological feature in multiple observation sources, the Mahalanobis distance correlation coefficient between the observation value sequences is calculated to measure the correlation between the multi-source data, so as to avoid redundant or contradictory information.
[0107] Optionally, it is judged whether the difference coefficient of each meteorological feature is greater than a preset threshold; if yes, the single-source data corresponding to the meteorological feature is determined as the first effective data; then it is judged whether the Mahalanobis distance correlation coefficient of each meteorological feature is less than a preset coefficient; if yes, the multi-source data corresponding to the meteorological feature is determined as the second effective data; finally, the first meteorological data set is obtained according to the first effective data and the second effective data.
[0108] It should be understood that single-source data with a relatively average probability distribution is generally considered to have a large amount of information and information value and is worth retaining, while single-source data with a relatively concentrated probability distribution is considered to have poor variability and can be directly removed from the data set; when the Mahalanobis distance correlation coefficient is large (generally greater than or equal to 0.7), it is considered that the data is outlying or has poor correlation and needs to be merged and simplified; on the contrary, it indicates that the data has high correlation and this data is retained.
[0109] S103, performing image feature enhancement processing on the meteorological observation image data set to obtain a second meteorological data set.
[0110] It can be understood that due to meteorological images (such as radar images, cloud images, satellite images, etc.) may be caused by observation noise, low resolution, shielding, light difference, etc., it is difficult for the model to learn stable features, therefore, the meteorological observation image data set needs to be subjected to image feature enhancement processing, so as to enhance the image discriminability and generalization ability. The specific image feature enhancement processing mode is described in detail later, and this embodiment will not be described here.
[0111] S104, inputting the first meteorological data set and the second meteorological data set into a multi-modal data fusion model to perform data fusion and obtain a meteorological data fusion matrix.
[0112] The multi-modal data fusion model is trained in advance according to a historical meteorological observation data set; and the meteorological data fusion matrix is used to input into other task models for data analysis and processing.
[0113] It can be understood that by jointly learning the complementary information of the text and image modalities, the representation ability of the model for complex meteorological phenomena is improved, and the generated fusion matrix can be used for multi-task learning, such as classification (whether it is a heavy rain), regression (predicting rainfall), detection (identifying a thunderstorm center), and the like, which can improve the intelligent identification and prediction accuracy of sudden and extreme meteorological events.
[0114] The data fusion method provided by the embodiments of the present application first acquires a meteorological observation dataset, laying a data foundation for subsequent multi-modal fusion; then, the meteorological observation text dataset is preprocessed to obtain a first meteorological dataset; then, the meteorological observation image dataset is subjected to image feature enhancement processing to obtain a second meteorological dataset. In this way, the semantic expression ability and computability of the text data are improved, and the key visual information in the image data is strengthened, thereby improving the discriminability and robustness of the image modality and significantly improving the expression ability and learning efficiency of the subsequent multi-modal fusion model, laying a foundation for the final data fusion and analysis. Finally, the first meteorological dataset and the second meteorological dataset are input into a multi-modal data fusion model for data fusion to obtain a meteorological data fusion matrix. The data fusion method realizes collaborative understanding and deep representation of multi-source meteorological information by fusing meteorological observation data of two modalities, thereby improving the accuracy and generalization ability of downstream meteorological analysis tasks.
[0115] In an implementable manner, first, for each meteorological observation image data, a feature enhancement strategy of the meteorological observation image data is determined through a pre-constructed differentiable automatic data enhancement model; then, the meteorological observation image data is subjected to image feature enhancement processing according to the feature enhancement strategy to obtain enhanced meteorological observation image data; finally, the second meteorological dataset is obtained according to each enhanced meteorological observation image data.
[0116] The feature enhancement strategy includes at least one of image reversal, color processing, and image flipping.
[0117] It can be understood that firstly, for each meteorological observation image data, a pre-constructed differentiable automatic data augmentation (DADA) model is utilized to determine the optimal feature enhancement strategy based on a gradient-derivable policy search method, which automatically optimizes the enhancement operation and its parameters through back propagation during the training process to improve the sensitivity and generalization ability of the model to image features; then, according to the determined enhancement strategy, corresponding enhancement operations are performed on the original meteorological image data, so that the key cloud layer morphology, weather phenomenon, edge feature and the like in the image are highlighted or retained, thereby improving the robustness and expression ability of the model in extracting key weather image features. Finally, all the enhanced image samples are combined to form an enhanced image dataset, i.e., a second meteorological dataset, which can be input together with the processed text data into a subsequent multi-modal fusion model to enhance the accuracy and diversity of the overall meteorological data fusion.
[0118] It should be noted that the training process of the differentiable automatic data augmentation model includes: first, obtaining a sample meteorological observation image dataset; then, for each sample meteorological observation image data, randomly generating a sample feature enhancement strategy by the differentiable automatic data augmentation model; determining at least one feature enhancement sub-strategy and the probability weight of the feature enhancement sub-strategy from the sample feature enhancement strategy using a categorical distribution; then determining the application intensity of the feature enhancement sub-strategy using a Bernoulli distribution; further, calculating the loss function of the sample meteorological observation image data according to the probability weight and the application intensity; finally, using the Monte Carlo method to estimate the expectation of the loss function to train the differentiable automatic data augmentation model.
[0119] It should be understood that in the present embodiment, a sample meteorological observation image dataset is first obtained as a training input; then, for each sample image data, a differentiable automatic data augmentation model is used to generate a set of random sample feature enhancement strategies, which are composed of multiple enhancement sub-strategies such as image reversal, color processing, image flipping, etc.; then, a categorical distribution is used to probabilistically model these enhancement sub-strategies, from which at least one sub-strategy is selected and assigned a corresponding probability weight; further, a Bernoulli distribution is used to determine the actual application intensity of each sub-strategy in the image, thereby forming a specific enhancement combination scheme.
[0120] Subsequently, after applying the enhancement operation to the sample image, the corresponding loss function is calculated by forward propagation of the model, and then the influence of the enhancement strategy on the model performance is measured; in order to make the loss function optimizable in the strategy space with randomness, the expectation of the overall loss is estimated by the Monte Carlo method, and the parameters of the enhancement model are updated by back propagation, so that they tend to select the most beneficial enhancement strategy for model training.
[0121] It can be understood that by automatically learning and optimizing the enhancement strategy, the generated image enhancement operation has differentiability and efficiency, significantly improves the quality and diversity of the enhanced data, thereby enhancing the robustness and generalization ability of the main model to meteorological observation images, and improving the performance of the finally constructed second meteorological data set in subsequent model training.
[0122] Figure 2 Process diagram of the data fusion method provided by the embodiments of the present application Figure 2 As shown in the Figure 1 , the embodiments on the basis of the embodiments, the process of data fusion is described in detail, and the method comprises: Figure 3
[0123] S201, a text modality compiler of a multi-modal data fusion model extracts a first digital sequence vector from the first meteorological data set.
[0124] It can be understood that the text modality compiler based on a natural language processing (NLP) pre-training model (such as BERT, RoBERTa) can convert meteorological text information into an embedded vector sequence with semantic representation ability, and output a first digital sequence vector with unified dimensions, which retains the characteristics of time sequence, semantic relationship and key meteorological factors, realizes the mapping of text information to vector space, and lays a foundation for subsequent cross-modal fusion.
[0125] S202, an image modality compiler of the multi-modal data fusion model extracts a second digital sequence vector from the second meteorological data set.
[0126] It can be understood that the image modality compiler of the multi-modal data fusion model extracts features from the image based on a visual-sequential long short-term memory network (V-sLSTM), outputs a second digital sequence vector, forms a feature vector sequence compatible with the text modality structure, and contains spatial structure information and local texture features.
[0127] S203, using an orthogonal sequence fusion mechanism, the first digital sequence vector and the second digital sequence vector are fused to obtain a meteorological data fusion matrix.
[0128] The orthogonal sequence fusion mechanism (OSF) forces the feature expressions of different modalities to be as little overlapped as possible in the fusion space, thereby improving the information density and discriminability. Compared with simple concatenation or weighted average, orthogonal fusion can reduce redundancy and improve the learning ability of the model to modal specific features.
[0129] It can be understood that by feeding the sequence vectors of the two modalities into an orthogonal sequence fusion mechanism, the mechanism ensures that the features of the two modalities maintain information independence and complementarity when fused, prevents modal redundancy, and thus generates a meteorological data fusion matrix with high expressiveness and discriminability, providing multi-source support for subsequent meteorological prediction, classification or early warning and the like.
[0130] In an implementable manner, the training process of the multi-modal data fusion model is as follows:
[0131] First, a sample first meteorological data set, a sample second meteorological data set and a historical meteorological data fusion matrix are obtained. Then, a sample first digital sequence vector is extracted from the sample first meteorological data set by a text modality compiler of the multi-modal data fusion model. Then, a sample second digital sequence vector is extracted from the sample second meteorological data set by an image modality compiler of the multi-modal data fusion model. Further, the first digital sequence vector and the second digital sequence vector are fused by an orthogonal sequence fusion mechanism to obtain a sample meteorological data fusion matrix. Finally, the multi-modal data fusion model is trained based on the orthogonal sequential fusion loss calculated from the sample meteorological data fusion matrix and the historical meteorological data fusion matrix.
[0132] The historical meteorological data fusion matrix is a data sample accumulated in the long-term operation of the system, which is derived from a historical database, log records, historical sampling, long-term system records and the like.
[0133] Specifically, in the training process, the multi-modal fusion model needs to first construct training samples, including a sample first meteorological data set, a sample second meteorological data set and a corresponding historical meteorological data fusion matrix (as a label or target for supervised learning). First, the sample first data set is converted into a sample first digital sequence vector by a text modality compiler, and the sample second data set is converted into a sample second digital sequence vector by an image modality compiler. Subsequently, the two modality vectors are input into an orthogonal sequence fusion mechanism to generate a fusion feature representation, i.e. a sample meteorological data fusion matrix. The key of the fusion mechanism is to utilize orthogonal constraints (such as orthogonal projection, orthogonal attention and the like), so that the information from different modalities presents an orthogonal distribution in the feature space, thereby improving the diversity and generalization ability of the fused feature representation. Finally, the system compares the difference between the generated fusion matrix and the historical true fusion matrix, calculates the orthogonal sequential fusion loss function (Orthogonal Sequential Fusion Loss), which is used for back propagation optimization of network parameters, and gradually improves the fusion expression ability of the model for different modal data.
[0134] Figure 3A structural schematic diagram of a data fusion device provided by an embodiment of the present application is shown in Figure 4 The device includes:
[0135] An acquisition module 301 is configured to acquire a meteorological observation dataset, the meteorological observation dataset including a meteorological observation text dataset and a meteorological observation image dataset.
[0136] A first processing module 302 is configured to pre-process the meteorological observation text dataset to obtain a first meteorological dataset.
[0137] A second processing module 303 is configured to perform image feature enhancement processing on the meteorological observation image dataset to obtain a second meteorological dataset.
[0138] A fusion module 304 is configured to input the first meteorological dataset and the second meteorological dataset into a multi-modal data fusion model to perform data fusion, to obtain a meteorological data fusion matrix; the multi-modal data fusion model is pre-trained according to historical meteorological observation datasets; and the meteorological data fusion matrix is used to input into other task models for data analysis and processing.
[0139] In a possible implementation, the meteorological observation text dataset includes single-source data and multi-source data, and the single-source data and the multi-source data include a plurality of meteorological features; and the first processing module 302 is specifically configured to:
[0140] For each meteorological feature in the single-source data, perform probability statistics on an observation value sequence of the meteorological feature to obtain a probability distribution of the observation value sequence.
[0141] Calculate a difference coefficient of the meteorological feature according to the probability distribution of each observation value sequence.
[0142] For each meteorological feature in the multi-source data, calculate a Mahalanobis distance correlation coefficient of an observation value sequence of the meteorological feature.
[0143] Filter the meteorological observation text dataset according to the difference coefficient of each meteorological feature and the Mahalanobis distance correlation coefficient of the meteorological feature to obtain the first meteorological dataset.
[0144] In a possible implementation, the first processing module 302 is specifically configured to:
[0145] Determine whether the difference coefficient of each meteorological feature is greater than a preset threshold value.
[0146] If yes, determine that single-source data corresponding to the meteorological feature is first valid data.
[0147] Determine whether the Mahalanobis distance correlation coefficient of each meteorological feature is less than a preset coefficient.
[0148] If yes, the multi-source data corresponding to the weather feature is determined as second valid data;
[0149] According to the first valid data and the second valid data, a first weather data set is obtained.
[0150] In a possible implementation, the second processing module 303 is specifically configured to:
[0151] For each weather observation image data, a feature enhancement strategy of the weather observation image data is determined through a pre-constructed differentiable automatic data enhancement model, and the feature enhancement strategy includes at least one of image reversal, color processing, and image flipping;
[0152] According to the feature enhancement strategy, image feature enhancement processing is performed on the weather observation image data to obtain enhanced weather observation image data;
[0153] According to each enhanced weather observation image data, a second weather data set is obtained.
[0154] In a possible implementation, the second processing module 303 is specifically configured to:
[0155] A sample weather observation image data set is obtained;
[0156] For each sample weather observation image data, a sample feature enhancement strategy is randomly generated through the differentiable automatic data enhancement model;
[0157] At least one feature enhancement sub-strategy and a probability weight of the feature enhancement sub-strategy are determined from the sample feature enhancement strategy by using a classification distribution;
[0158] The application intensity of the feature enhancement sub-strategy is determined by using a Bernoulli distribution;
[0159] According to the probability weight and the application intensity, a loss function of the sample weather observation image data is calculated;
[0160] The loss function is expectedly estimated by using a Monte Carlo method to train the differentiable automatic data enhancement model.
[0161] In a possible implementation, the fusion module 304 is specifically configured to:
[0162] A first digital sequence vector is extracted from the first weather data set by using a text modality compiler of the multi-modal data fusion model;
[0163] A second digital sequence vector is extracted from the second weather data set by using an image modality compiler of the multi-modal data fusion model;
[0164] The first digital sequence vector and the second digital sequence vector are fused by using an orthogonal sequence fusion mechanism to obtain a meteorological data fusion matrix.
[0165] In a possible implementation, the fusion module 304 is specifically configured to:
[0166] obtain a sample first meteorological data set, a sample second meteorological data set, and a historical meteorological data fusion matrix;
[0167] extract a sample first digital sequence vector from the sample first meteorological data set by using a text modality compiler of the multi-modal data fusion model;
[0168] extract a sample second digital sequence vector from the sample second meteorological data set by using an image modality compiler of the multi-modal data fusion model;
[0169] fuse the first digital sequence vector and the second digital sequence vector by using an orthogonal sequence fusion mechanism to obtain a sample meteorological data fusion matrix;
[0170] train the multi-modal data fusion model based on an orthogonal sequence fusion loss calculated from the sample meteorological data fusion matrix and the historical meteorological data fusion matrix.
[0171] The data fusion apparatus provided in the embodiments of the present application can execute the method provided in the method embodiments, and has similar implementation principles and technical effects, which will not be described here again.
[0172] Figure 4 A structural schematic diagram of an electronic device provided in the embodiments of the present application is shown in FIG. 4. As shown in FIG. 4, the electronic device 40 provided in the embodiments of the present application includes at least one processor 401 and a memory 402. Optionally, the device 40 further includes a communication component 403. The processor 401, the memory 402, and the communication component 403 are connected through a bus 404.
[0173] In the implementation process, the at least one processor 401 executes the computer execution instructions stored in the memory 402, so that the at least one processor 401 executes the method described above.
[0174] The specific implementation process of the processor 401 can refer to the method embodiments described above, and has similar implementation principles and technical effects, which will not be described here again.
[0175] In the above embodiments, it should be understood that the processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or can also be any conventional processor. The steps of the method disclosed in combination with the application can be directly embodied as hardware processor execution, or executed by a combination of hardware and software modules in the processor.
[0176] The memory can include a random access memory (RAM), and can also include a non-volatile memory (NVM), such as at least one disk memory.
[0177] The bus can be an industry standard architecture (ISA) bus, a peripheral component (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application does not limit to only one bus or one type of bus.
[0178] The present application also provides a computer readable storage medium, and the computer readable storage medium stores computer execution instructions, when the processor executes the computer execution instructions, the above method is realized.
[0179] The above readable storage medium can be realized by any type of volatile or non-volatile storage device or their combination, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special purpose computer.
[0180] An example readable storage medium is coupled to the processor such that the processor can read information from the readable storage medium and can write information to the readable storage medium. Of course, the readable storage medium can also be a part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in the device.
[0181] The division of units is only a logical functional division, and in actual implementation, there can be another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.
[0182] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.
[0183] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.
[0184] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the part of the present application that essentially contributes to the prior art or the part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. The foregoing storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.
[0185] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware. The foregoing program can be stored in a computer readable storage medium. The program executes to perform the steps of the above-mentioned method embodiments; and the foregoing storage medium includes various media capable of storing program codes, such as ROM, RAM, magnetic disk, or optical disk.
[0186] Finally, it should be noted that other embodiments of the present application will readily occur to those skilled in the art upon consideration of the specification and practice of the present application disclosed herein. The present application is intended to include all such variations, uses, or adaptations of the application in which the general principles of the application are used to best advantage and encompassed within its scope. The present application is not limited to the precise structures described and shown in the accompanying drawings and figures, and can be practiced with variation of modifications and alterations without departing from the scope of the present application. The scope of the present application is limited only by the claims appended hereto.
Claims
1. A data fusion method characterized by, The method comprises the following steps: obtaining a meteorological observation data set, the meteorological observation data set comprising a meteorological observation text data set and a meteorological observation image data set; preprocessing the meteorological observation text data set to obtain a first meteorological data set; performing image feature enhancement processing on the meteorological observation image data set to obtain a second meteorological data set; inputting the first meteorological data set and the second meteorological data set into a multi-modal data fusion model to perform data fusion, thereby obtaining a meteorological data fusion matrix; the multi-modal data fusion model is obtained by pre-training according to historical meteorological observation data sets; and the meteorological data fusion matrix is used to input into other task models for data analysis and processing.
2. The method of claim 1, wherein, The meteorological observation text data set comprises single-source data and multi-source data, and the single-source data and the multi-source data contain multiple meteorological features; The preprocessing of the meteorological observation text data set to obtain the first meteorological data comprises the following steps: for each meteorological feature in the single-source data, performing probability statistics on the observation value sequence of the meteorological feature to obtain the probability distribution of the observation value sequence; calculating the difference coefficient of the meteorological feature according to the probability distribution of each observation value sequence; for each meteorological feature in the multi-source data, calculating the Mahalanobis distance correlation coefficient of the observation value sequence of the meteorological feature; screening the meteorological observation text data set according to the difference coefficient of each meteorological feature and the Mahalanobis distance correlation coefficient of the meteorological feature to obtain the first meteorological data set.
3. The method of claim 2, wherein, The screening of the meteorological observation text data set according to the difference coefficient of each meteorological feature and the Mahalanobis distance correlation coefficient to obtain the first meteorological data set comprises the following steps: determining whether the difference coefficient of each meteorological feature is greater than a preset threshold value; if yes, determining that the single-source data corresponding to the meteorological feature is first effective data; determining whether the Mahalanobis distance correlation coefficient of each meteorological feature is less than a preset coefficient; if yes, determining that the multi-source data corresponding to the meteorological feature is second effective data; obtaining the first meteorological data set according to the first effective data and the second effective data.
4. The method of claim 1, wherein, The image feature enhancement processing of the meteorological observation image data set to obtain the second meteorological data set comprises the following steps: for each meteorological observation image data, determining a feature enhancement strategy of the meteorological observation image data through a pre-constructed differentiable automatic data enhancement model, the feature enhancement strategy comprising at least one of image reversal, color processing, and image flipping; performing image feature enhancement processing on the meteorological observation image data according to the feature enhancement strategy to obtain enhanced meteorological observation image data; obtaining the second meteorological data set according to each enhanced meteorological observation image data.
5. The method of claim 4, wherein, The training process of the differentiable automatic data enhancement model comprises the following steps: obtaining a sample meteorological observation image data set; for each sample meteorological observation image data, randomly generating a sample feature enhancement strategy through the differentiable automatic data enhancement model; determining at least one feature enhancement sub-strategy and the probability weight of the feature enhancement sub-strategy from the sample feature enhancement strategy by using a classification distribution; The application intensity of the feature enhancer strategy is determined by using a Bernoulli distribution; According to the probability weight and the application intensity, a loss function of the sample meteorological observation image data is calculated; A Monte Carlo method is used to perform expectation estimation on the loss function to train the differentiable automatic data enhancement model.
6. The method according to any one of claims 1 to 5, characterized in that, The data fusion is performed by inputting the first meteorological data set and the second meteorological data set into a multi-modal data fusion model to obtain a meteorological data fusion matrix, including: A first digital sequence vector is extracted from the first meteorological data set by a text modal compiler of the multi-modal data fusion model; A second digital sequence vector is extracted from the second meteorological data set by an image modal compiler of the multi-modal data fusion model; The first digital sequence vector and the second digital sequence vector are fused by using an orthogonal sequence fusion mechanism to obtain a meteorological data fusion matrix.
7. The method of claim 6, wherein, The training process of the multi-modal data fusion model includes: A sample first meteorological data set, a sample second meteorological data set, and a historical meteorological data fusion matrix are obtained; A sample first digital sequence vector is extracted from the sample first meteorological data set by a text modal compiler of the multi-modal data fusion model; A sample second digital sequence vector is extracted from the sample second meteorological data set by an image modal compiler of the multi-modal data fusion model; The first digital sequence vector and the second digital sequence vector are fused by using an orthogonal sequence fusion mechanism to obtain a sample meteorological data fusion matrix; The multi-modal data fusion model is trained based on an orthogonal sequence fusion loss calculated from the sample meteorological data fusion matrix and the historical meteorological data fusion matrix.
8. A data fusion apparatus characterized by comprising: It includes: An acquisition module is configured to acquire a meteorological observation data set, the meteorological observation data set including a meteorological observation text data set and a meteorological observation image data set; A first processing module is configured to preprocess the meteorological observation text data set to obtain a first meteorological data set; A second processing module is configured to perform image feature enhancement processing on the meteorological observation image data set to obtain a second meteorological data set; A fusion module is configured to perform data fusion by inputting the first meteorological data set and the second meteorological data set into a multi-modal data fusion model to obtain a meteorological data fusion matrix; the multi-modal data fusion model is pre-trained based on historical meteorological observation data sets; and the meteorological data fusion matrix is used to input into other task models for data analysis and processing.
9. An electronic device, comprising: It includes: A memory and a processor; The memory stores computer execution instructions; The processor executes the computer execution instructions stored in the memory, so that the processor executes the method of any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to implement the method of any one of claims 1-7.