Multi-source data fusion method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a multi-source data fusion method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring multi-source heterogeneous data, and preprocessing the multi-source heterogeneous data to obtain preprocessed multi-source data; performing feature selection on the preprocessed multi-source data to obtain target fusion features; and performing data association mining and data matching on the preprocessed multi-source data according to the target fusion features, and performing data fusion based on association and matching results to obtain fused data. According to the embodiment of the invention, the information loss of the fused data is reduced, and the accuracy of data fusion is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to information processing technology, and more particularly to a multi-source data fusion method, device, electronic device and storage medium. Background Art

[0002] As the level of intelligence and informationization of power systems continues to improve, the amount of data generated in power systems is growing exponentially. These data come from different types of equipment and systems, are highly heterogeneous and complex, and bring huge challenges to data management, analysis and application.

[0003] In the prior art, data fusion is usually achieved by simply performing weighted averaging on multiple data sources for multi-source data.

[0004] However, multi-source data fusion by weighted averaging may lead to information loss and low accuracy of data fusion. Summary of the invention

[0005] The present application provides a multi-source data fusion method, device, electronic device and storage medium to reduce information loss of fused data and improve the accuracy of data fusion.

[0006] In a first aspect, an embodiment of the present application provides a multi-source data fusion method, the multi-source data fusion method comprising:

[0007] Acquire multi-source heterogeneous data, and preprocess the multi-source heterogeneous data to obtain preprocessed multi-source data;

[0008] Perform feature selection on the preprocessed multi-source data to obtain the target fusion features;

[0009] According to the target fusion features, data association mining and data matching are performed on the preprocessed multi-source data, and data fusion is performed based on the association and matching results to obtain fused data.

[0010] In a second aspect, an embodiment of the present application further provides a multi-source data fusion device, the multi-source data fusion device comprising:

[0011] A data preprocessing module is used to obtain multi-source heterogeneous data and preprocess the multi-source heterogeneous data to obtain preprocessed multi-source data;

[0012] The feature selection module is used to perform feature selection on the preprocessed multi-source data to obtain the target fusion features;

[0013] The data fusion module is used to perform data association mining and data matching on the pre-processed multi-source data according to the target fusion features, and to perform data fusion based on the association and matching results to obtain fused data.

[0014] In a third aspect, an embodiment of the present application further provides an electronic device, the electronic device comprising:

[0015] one or more processors;

[0016] A storage device for storing one or more programs;

[0017] When one or more programs are executed by one or more processors, the one or more processors implement any one of the multi-source data fusion methods provided in the embodiments of the present application.

[0018] In a fourth aspect, an embodiment of the present application further provides a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to execute any multi-source data fusion method provided in the embodiment of the present application.

[0019] The present application obtains multi-source heterogeneous data and pre-processes the multi-source heterogeneous data to obtain pre-processed multi-source data; performs feature selection on the pre-processed multi-source data to obtain target fusion features. By performing feature selection, features with greater contributions can be selected to provide a basis for data fusion; data association mining and data matching are performed on the pre-processed multi-source data according to the target fusion features, and data fusion is performed based on the association and matching results to obtain fused data. By performing data association mining and data matching on the pre-processed multi-source data, the association relationship between the data can be discovered, and the information loss of the fused data can be reduced. By matching, data with the same characteristics can be matched to improve the accuracy of data fusion. Therefore, the technical solution of the present application solves the problem that multi-source data fusion through weighted averaging may cause information loss and low accuracy of data fusion, and achieves the effect of reducing the information loss of the fused data and improving the accuracy of data fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flow chart of a multi-source data fusion method in Embodiment 1 of the present application;

[0021] Figure 2 is a flow chart of a multi-source data fusion method in Embodiment 2 of the present application;

[0022] Figure 3 This is a flow chart of a multi-source data fusion method in Embodiment 3 of the present application;

[0023] Figure 4 It is a structural schematic diagram of a multi-source data fusion device in Embodiment 4 of the present application;

[0024] Figure 5 It is a structural diagram of an electronic device in Example 5 of the present application. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first" and "second" etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] Embodiment 1

[0028] Figure 1 A flowchart of a multi-source data fusion method provided in Example 1 of the present application is provided. This embodiment is applicable to the situation of fusing multi-source heterogeneous data. The method can be executed by a multi-source data fusion device, which can be implemented in software and / or hardware and specifically configured in a data processing device, for example, a data processing platform.

[0029] See also Figure 1 The multi-source data fusion method shown in the figure specifically includes the following steps:

[0030] S110 , acquiring multi-source heterogeneous data, and preprocessing the multi-source heterogeneous data to obtain preprocessed multi-source data.

[0031] Multi-source heterogeneous data can be data with different sources and heterogeneity, which is used for data fusion. For example, in the power system, multi-source heterogeneous data can be data from different types of equipment and systems. For example, different types of equipment and systems in the power system include: substations, distribution networks, user terminals, meteorological systems, and market trading systems, etc., covering all aspects of power production, transmission, distribution, and consumption. Multi-source heterogeneous data is highly heterogeneous and complex in type, format, and spatiotemporal distribution, which brings huge challenges to data management, analysis, and application.

[0032] Preprocessing can be done by aligning, denoising, and supplementing missing values ​​to make the data conform to certain specifications, so as to facilitate data comparison and analysis, and then data fusion. Preprocessed multi-source data is also multi-source heterogeneous data after data preprocessing.

[0033] In the power system, multi-source heterogeneous data often have time and space asynchrony problems. The data collection frequency and delay differences may make it impossible to directly compare and analyze the data before fusion. In addition, multi-source heterogeneous data may also contain a large amount of noise and missing values, further increasing the complexity of data fusion processing. Therefore, it is necessary to effectively pre-process the multi-source heterogeneous data in the power system.

[0034] In an optional embodiment, preprocessing of multi-source heterogeneous data includes time alignment, space alignment, noise filtering and missing value filling.

[0035] In an optional embodiment, time alignment includes: timestamp correction, time interpolation and time window synchronization.

[0036] Timestamp correction can be to correct the timestamp of each data source according to the selected time reference. Exemplarily, the time reference can usually select the most commonly used or most accurate data source as the time reference. For example, the time reference can be the Global Positioning System (GPS) timing signal; specifically, timestamp correction can be implemented by the following formula:

[0037] T i ′=T i +ΔT i ;

[0038]

[0039] Where: ΔT i represents the time deviation; N represents the number of times the event occurs; T z,j and T i,j are the base time of the jth event and the timestamp of data source i, respectively; the time deviation can be obtained by collecting the timestamps of all data sources within a certain time range and calculating the average difference between the timestamp of each data source and the base timestamp.

[0040] Time interpolation can be used to interpolate data sources with inconsistent sampling frequencies using linear interpolation. Time interpolation can be achieved using the following formula:

[0041]

[0042] Where: V(t1) and V(t2) represent the time values ​​of two known time points t1 and t2; V(t) represents the interpolation result at time point t.

[0043] Time window synchronization can be used to divide the data of all data sources into a unified time window. The time window can be defined as a fixed length Δt, such as 1 second or 1 minute. The data of all data sources are aggregated within the corresponding time window to ensure the temporal consistency of the data.

[0044] In an optional embodiment, the spatial alignment includes: spatial coordinate conversion, spatial interpolation and spatial partition alignment.

[0045] The spatial coordinate conversion can be based on a predetermined spatial reference, and the location coordinates of the data source can be converted into a unified spatial reference system. The spatial reference can use the geographic information system (GIS) coordinate system of the power system as the spatial reference. For example, assuming that the original coordinates of a data source are (X i ,Y i ,Z i ), the coordinates after spatial coordinate transformation are as follows:

[0046]

[0047] Where: R represents the rotation matrix, which is used to adjust the differences between different coordinate systems; Indicates spatial reference; Indicates the coordinates after spatial alignment.

[0048] Spatial interpolation can be performed by using a bilinear interpolation method to interpolate a data source with discrete spatial positions. Specifically, bilinear interpolation can be implemented by the following formula:

[0049]

[0050] Among them, V(X1,Y1)+other items are the positions and values ​​of the known points; V(X,Y) represents the interpolation result of the target point; subscript 1 and subscript 2 represent the lower left corner and upper right corner coordinates of the known point, respectively. Other items include values ​​adjacent to the target point, usually from other known points in the interpolation area. By weighting these values, the interpolation result of the target point can be effectively calculated to ensure the comparison and fusion of spatial data under a unified benchmark.

[0051] Spatial partition alignment can be used to allocate data from all data sources to corresponding spatial partitions. Spatial partitioning can be based on the actual structure of the power system to ensure that spatial data from different data sources can be compared and integrated in the same geographical area.

[0052] Noise filtering can be to remove noise in multi-source heterogeneous data by using a filtering algorithm. For example, the filtering algorithm can be a Kalman filter. Missing value filling can be to fill missing values ​​in data by using an interpolation method or a mean filling method.

[0053] S120, performing feature selection on the preprocessed multi-source data to obtain target fusion features.

[0054] Feature selection can be to select the preprocessed multi-source data features according to certain rules to determine the target fusion features. Exemplarily, feature selection can be based on the similarity between different types of features and the preprocessed multi-source data, and a preset similarity threshold, to perform feature selection on the multi-source data. For example, when the similarity between feature X and data A is greater than the preset similarity threshold, feature X is used as the target fusion feature of data A. The preset similarity threshold can be determined by professional technicians based on experiments or experience, and this application does not specifically limit this.

[0055] The target fusion feature may be a data feature used as a basis for data matching. For example, the relevance of multi-source data may be determined based on the target fusion feature, and then the matching of the data sources of the multi-source data may be determined.

[0056] S130, performing data association mining and data matching on the preprocessed multi-source data according to the target fusion features, and performing data fusion based on the association and matching results to obtain fused data.

[0057] Data association mining can be used to mine the association relationship between different data sources. The association rules of preprocessed multi-source data can be determined based on the target fusion features. Discretize the continuous data and convert the numerical data into categorical data to adapt to the association rule mining algorithm. Exemplarily, the discretization process can choose equal-frequency discretization or equal-width discretization, etc., which is not specifically limited in this application. For example, for voltage data V, the discretization interval can be defined as [low, medium, high]. Frequent item sets, that is, variable combinations that frequently appear in multiple data sources, can be mined through the Apriori (professional term, association rule algorithm) algorithm or the FP-Growth (professional term, association rule algorithm) algorithm. Specifically, the support of the item set can be determined by the following formula:

[0058]

[0059] Where: Support(I) is the support of item set I; Count(I) is the number of times item set I appears; N is the total number of records in the data set. When , the item set I can be considered as a frequent set, where min support is the minimum support threshold.

[0060] Generate association rules from frequent item sets. The rule form is Where X and Y are frequent item sets.

[0061] Confidence of association rules It can be obtained by the following formula:

[0062]

[0063] The lift of association rules It can be obtained by the following formula:

[0064]

[0065] Association rules are selected based on predefined confidence and lift thresholds, and rules with high execution and lift greater than 1 can be selected as association rules.

[0066] Data matching can be performed based on predefined matching rules. Exemplarily, the predefined matching rules may include matching based on primary keys, matching based on time or space windows, and matching based on content similarity.

[0067] Primary key matching can be used to match data sources that share a unique identifier directly based on the primary key. Time or space window matching can be used to match data that cannot be matched directly based on the primary key, using a time or space window to match data from two data sources in the same time window or space window. The time window matching formula is as follows:

[0068]

[0069] Where: T A and T B are the timestamps of data sources A and B respectively; Δt is the time window;

[0070] The spatial window matching formula is as follows:

[0071]

[0072] Where: X A ,Y A ,X B ,Y B are the spatial coordinates of data sources A and B respectively; Δd represents the spatial window; and D represents the spatial distance calculation function.

[0073] For data with inconsistent content, content similarity matching can be used. For example, content similarity matching can be performed using cosine similarity.

[0074] Data fusion can be performed by weighted summing of pre-processed multi-source heterogeneous data based on association and matching results to obtain fused data.

[0075] The technical solution of this embodiment obtains multi-source heterogeneous data and pre-processes the multi-source heterogeneous data to obtain pre-processed multi-source data; performs feature selection on the pre-processed multi-source data to obtain target fusion features. By performing feature selection, features with greater contributions can be selected to provide a basis for data fusion; data association mining and data matching are performed on the pre-processed multi-source data according to the target fusion features, and data fusion is performed based on the association and matching results to obtain fused data. By performing data association mining and data matching on the pre-processed multi-source data, the association relationship between the data can be discovered, and the information loss of the fused data can be reduced. By matching, data with the same characteristics can be matched to improve the accuracy of data fusion. Therefore, the technical solution of this application solves the problem that multi-source data fusion through weighted averaging may cause information loss and low accuracy of data fusion, and achieves the effect of reducing the information loss of the fused data and improving the accuracy of data fusion.

[0076] Embodiment 2

[0077] Figure 2 This is a flow chart of a multi-source data fusion method provided in Example 2 of the present application. The technical solution of this embodiment is further refined on the basis of the above technical solution.

[0078] Furthermore, “performing feature selection on the preprocessed multi-source data to obtain target fusion features” is refined as: “performing feature extraction on the preprocessed multi-source data to obtain at least two multi-source data features; determining the correlation between the multi-source data features, and determining candidate fusion features based on the fusion threshold and the correlation between the multi-source data features; performing principal component analysis on the candidate fusion features to obtain target fusion features” to obtain the target fusion features.

[0079] See also Figure 2 A multi-source data fusion method shown includes:

[0080] S210: Acquire multi-source heterogeneous data, and preprocess the multi-source heterogeneous data to obtain preprocessed multi-source data.

[0081] S220 , extracting features from the preprocessed multi-source data to obtain at least two multi-source data features.

[0082] The multi-source data features may be feature values ​​obtained by extracting features from the pre-processed multi-source data, and are used to determine the correlation between the multi-source data features.

[0083] In an optional embodiment, feature extraction is performed on the preprocessed multi-source data, including: statistical feature extraction, time domain feature extraction, frequency domain feature extraction and power feature extraction on the preprocessed multi-source data.

[0084] Statistical feature extraction can be to extract statistical features of pre-processed multi-source data. Exemplarily, the statistical features may include mean, variance, standard deviation, kurtosis and skewness, etc., which are not specifically limited in this application.

[0085] Time domain feature extraction can be to extract the features of the preprocessed multi-source data in the time domain. Exemplarily, the time domain features may include maximum value, minimum value, rise time, fall time, stability and periodicity, etc., which are not specifically limited in this application.

[0086] The time domain data is converted into frequency domain data through Fourier transform, and accordingly, the frequency domain feature extraction can be to extract the features of the pre-processed multi-source data in the frequency domain. Exemplarily, the spectrum features may include the main frequency components and the frequency band energy, etc., which are not specifically limited in this application.

[0087] The power feature extraction may be to extract the power features specific to the power system. For example, the power features may include active power, reactive power, apparent power, power factor, etc., which are not specifically limited in this application.

[0088] S230: Determine the correlation between multi-source data features, and determine candidate fusion features according to the fusion threshold and the correlation between multi-source data features.

[0089] There is correlation between the features of multi-source data, which can be determined by the following formula:

[0090]

[0091] Where: r X,Y Represents the correlation coefficient between feature X and target Y; Represent the means of X and Y respectively; X i represents the value of the i-th data point; Y i Describes the value of the target Y at data point i.

[0092] The fusion threshold may be a pre-set correlation threshold between multi-source data features that meet the fusion condition. Multi-source data features with correlation higher than the fusion threshold are selected as candidate fusion features.

[0093] S240: Perform principal component analysis on the candidate fusion features to obtain target fusion features.

[0094] Principal component analysis can be used to remove redundant features from candidate fusion features and obtain target fusion features.

[0095] In an optional embodiment, principal component analysis is performed on the candidate fusion features to obtain the target fusion features, including: determining the principal component vector according to the feature weights and mean vectors of each candidate fusion feature; and performing feature dimensionality reduction according to the principal component vector and the corresponding feature weights to obtain the target fusion features.

[0096] The importance of candidate fusion features is preliminarily judged based on expert experience or domain knowledge to obtain initial weights. Based on historical data, the stability and contribution of candidate fusion features are calculated, and the initial weights are dynamically adjusted. The formula for dynamic adjustment of initial weights is as follows:

[0097]

[0098] Where: Var(X i ) represents feature X i Variance of Corr(X i ,Y) represents feature X i Correlation coefficient with variable Y.

[0099] Based on the weighted values ​​of the candidate fusion features, the weighted covariance matrix is ​​calculated:

[0100]

[0101] Where: μ is the mean vector of the feature; w i is the weight of the ith feature.

[0102] The weighted covariance matrix ∈ w Perform eigenvalue decomposition to obtain eigenvalues ​​and corresponding eigenvectors; sort the eigenvectors from large to small according to the eigenvalues, and select the first few eigenvectors with larger explained variance as principal components. Explained variance refers to the variance that can be explained by the principal component. The goal of principal component analysis is to reduce high-dimensional data to a low-dimensional space while retaining the variability (variance) of the data as much as possible. Each principal component has a corresponding eigenvalue, which indicates the variance explained by the principal component.

[0103] In an optional embodiment, feature dimensionality reduction is performed based on the principal component vector and the corresponding feature weights to obtain the target fusion feature, including: determining the principal component score of the principal component vector based on the candidate fusion features and feature weights corresponding to the principal component vector; and reducing the dimensionality of the principal component vector with a larger principal component score to obtain the target fusion feature.

[0104] The principal component score can be the projection of each candidate fusion feature in the new principal component space, which is used to perform weighted summation of the principal component scores in combination with the feature weights:

[0105] Z = W*X;

[0106] Among them, Z is the principal component score matrix, W is the weight matrix composed of weighted feature vectors, and X is the candidate fusion feature.

[0107] According to the weighted principal component scores, the first few principal components with the largest explained variance are selected for dimensionality reduction. The cumulative variance contribution rate can be used as a criterion to ensure that the selected principal components can explain enough variance of the candidate fusion features.

[0108] Based on the selected principal components, the candidate fusion features are reconstructed to obtain the feature data after dimensionality reduction. Different from the traditional principal component analysis method, the principal component analysis method of this application introduces feature weights in the feature selection and dimensionality reduction process, so that the influence of important features can be enhanced in the fusion process of multi-source heterogeneous data in the power system, thereby improving the reliability and accuracy of the data fusion results.

[0109] By introducing weights for each feature, the importance of different features in the data fusion process can be better reflected. The dynamic adjustment of weights combines the stability of features and the correlation with the target variable, making the weights more adaptable. The introduction of feature weights improves the robustness of principal component analysis in dealing with noise and outliers, especially in complex power system scenarios.

[0110] S250, performing data association mining and data matching on the preprocessed multi-source data according to the target fusion features, and performing data fusion based on the association and matching results to obtain fused data.

[0111] The technical solution of this embodiment obtains at least two multi-source data features by performing feature extraction on the preprocessed multi-source data; determines the correlation between the multi-source data features, and determines candidate fusion features based on the fusion threshold and the correlation between the multi-source data features, and determines the candidate fusion features based on the correlation between the features, which can ensure that the fusion result is more accurate when the data is fused subsequently; performs principal component analysis on the candidate fusion features to obtain target fusion features, and based on the principal component analysis, reduces the interference of noise, removes redundant features, improves the accuracy of the target fusion features, and thereby improves the accuracy of the fusion results.

[0112] Embodiment 3

[0113] Figure 3 This is a flow chart of a multi-source data fusion method provided in Example 3 of the present application. The technical solution of this embodiment is further refined on the basis of the above technical solution.

[0114] Furthermore, “performing data fusion based on association and matching results to obtain fused data” is refined into: “performing standardization on the preprocessed multi-source data based on the association and matching results to obtain the data to be fused; determining the prior probability and likelihood value of the data to be fused, and determining the posterior probability of the data to be fused based on the prior probability and the likelihood value; performing weighted summation on the data to be fused based on the posterior probability to obtain the fused data” to achieve data fusion.

[0115] See also Figure 3 A multi-source data fusion method shown includes:

[0116] S310: Acquire multi-source heterogeneous data, and preprocess the multi-source heterogeneous data to obtain preprocessed multi-source data.

[0117] S320: performing feature selection on the preprocessed multi-source data to obtain target fusion features.

[0118] S330: performing data association mining and data matching on the preprocessed multi-source data according to the target fusion features.

[0119] S340 , standardize the preprocessed multi-source data based on the association and matching results to obtain data to be fused.

[0120] Standardization processing can be to convert the preprocessed multi-source data to the same scale to reduce data fusion errors. Multi-source data may have large differences in properties, dimensions and magnitudes, so it is necessary to standardize the preprocessed multi-source data. Exemplarily, based on the association and matching results, the preprocessed multi-source data can be converted according to a certain ratio through a certain mathematical transformation method so that it falls into a specific interval. The data to be fused can be the result obtained after the preprocessed multi-source data is standardized, which is used for subsequent data fusion.

[0121] The accuracy of matching and association rules can improve the accuracy of data standardization. For example, if the time windows or spatial windows of two data sources do not match well, it may cause errors in the standardization process, thus affecting subsequent data processing. The matching results can also affect the statistical parameters of each data source, such as the mean and standard deviation, thereby affecting the calculation of subsequent likelihood values. The mined association rules can help identify the potential relationships between data sources and affect how to calculate the historical errors of each data source. For example, through the results of time window matching, the relevant errors can be calculated more accurately, thus affecting the setting of prior probabilities. If the data sources are correctly associated through primary key matching, time or space window matching, the calculated mean and standard deviation will be more accurate.

[0122] S350: Determine the prior probability and likelihood value of the data to be fused, and determine the posterior probability of the data to be fused according to the prior probability and the likelihood value.

[0123] Dynamically adjust the prior probability of each data to be fused based on historical data. Calculate the historical error of each data to be fused. The smaller the error, the greater the prior probability of the data to be fused. The historical error can be obtained by the following formula:

[0124]

[0125] Among them, x′ ik is the predicted value after the data to be fused is standardized; is the actual observed value of the data to be fused; E i Represents the historical error of the data i to be fused.

[0126] The prior probability can be obtained by the following formula:

[0127]

[0128] Among them, E j represents the historical error of data source j.

[0129] The likelihood value is calculated by combining Gaussian distribution and Student's t distribution. The likelihood value can be obtained by the following formula:

[0130] P(x k |D i ) = α·Gaussian(x k |μ i ,σ i )+(1-α)·Studemt-t(x k |v);

[0131] Among them, α represents the weight factor; Gaussian(x k |μ i ,σ i ) represents the data to be fused D i The Gaussian normal distribution density function value calculated by the parameters; μ i Represents the data to be fused D i The mean of i Represents the data to be fused D i The standard deviation of Student-t(x k |v) represents the data point x k The probability of following the Student's t distribution with given degrees of freedom; v represents the degrees of freedom parameter of the Student's t distribution.

[0132] S360, performing weighted summation on the data to be fused based on the posterior probability to obtain fused data.

[0133] Optionally, the posterior probability is sampled by a Markov chain Monte Carlo method; the sampled posterior probability is smoothed by kernel density estimation; and the fused data is obtained by weighted summing the smoothed posterior probabilities on the data to be fused.

[0134] The Markov Chain Monte Carlo method (MCMC) is used to calculate the posterior probability P(D i |x k ) for sampling:

[0135]

[0136] Among them, P(D i |x k ) represents a given observation x k In the case of i The posterior probability of k |D i ) represents the data point x k In the data to be fused D i The likelihood value under P(D i ) represents the data to be fused D i The prior probability of .

[0137] The kernel density estimate is used to smooth the sampled posterior probability. The smoothing is performed by the following formula:

[0138]

[0139] in, represents the posterior probability estimate after smoothing; L represents the number of samples obtained from MCMC; h represents the bandwidth parameter of the kernel function; K(·) represents the kernel function; x kl Represents the observation value of the lth sample point.

[0140] Based on the posterior probability of the observed value x k Perform weighted summation to obtain fused data.

[0141] By adopting the MCMC method to sample the posterior probability and combining it with kernel density estimation to smooth the result, the estimation of the posterior probability is made more accurate and smoother, avoiding the overfitting problem and improving the generalization ability of data fusion.

[0142] The technical solution of this embodiment obtains data to be fused by standardizing the pre-processed multi-source data based on the association and matching results; through the standardization, the data of each data source is placed on the same scale for comparison, eliminating the dimensional differences between different data sources, significantly reducing the errors in the data fusion process, and improving the accuracy of the fusion results; determining the prior probability and likelihood value of the data to be fused, and determining the posterior probability of the data to be fused based on the prior probability and likelihood value, and dynamically adjusting the prior probability of each data source based on historical data so that the data to be fused with smaller historical errors occupy a greater weight in the fusion, thereby increasing the influence of high-quality data to be fused and enhancing the reliability of the fusion results; combining Gaussian distribution and Student's t distribution to calculate the likelihood value, effectively dealing with the outliers and noise problems in the data, and ensuring the robustness and stability of the model when facing complex and heterogeneous data; weighted summing the data to be fused based on the posterior probability to obtain the fused data, and fusion of the observations by the weighted summing method based on the posterior probability, so that the final fusion result can fully reflect the contribution of each data to be fused, ensuring the high accuracy and consistency of the fused data.

[0143] Embodiment 4

[0144] Figure 4 The figure shows a schematic diagram of the structure of a multi-source data fusion device provided in the fourth embodiment of the present application. This embodiment is applicable to the situation of fusing multi-source heterogeneous data. The specific structure of the multi-source data fusion device is as follows:

[0145] The data preprocessing module 410 is used to obtain multi-source heterogeneous data and preprocess the multi-source heterogeneous data to obtain preprocessed multi-source data;

[0146] A feature selection module 420 is used to perform feature selection on the preprocessed multi-source data to obtain target fusion features;

[0147] The data fusion module 430 is used to perform data association mining and data matching on the pre-processed multi-source data according to the target fusion features, and perform data fusion based on the association and matching results to obtain fused data.

[0148] The technical solution of this embodiment obtains multi-source heterogeneous data and pre-processes the multi-source heterogeneous data to obtain pre-processed multi-source data; performs feature selection on the pre-processed multi-source data to obtain target fusion features. By performing feature selection, features with greater contributions can be selected to provide a basis for data fusion; data association mining and data matching are performed on the pre-processed multi-source data according to the target fusion features, and data fusion is performed based on the association and matching results to obtain fused data. By performing data association mining and data matching on the pre-processed multi-source data, the association relationship between the data can be discovered, and the information loss of the fused data can be reduced. By matching, data with the same characteristics can be matched to improve the accuracy of data fusion. Therefore, the technical solution of this application solves the problem that multi-source data fusion through weighted averaging may cause information loss and low accuracy of data fusion, and achieves the effect of reducing the information loss of the fused data and improving the accuracy of data fusion.

[0149] Optionally, the feature selection module 420 includes:

[0150] A feature extraction unit, used to extract features from the preprocessed multi-source data to obtain at least two multi-source data features;

[0151] A candidate fusion feature determination unit is used to determine the correlation between multi-source data features, and determine the candidate fusion feature according to the fusion threshold and the correlation between the multi-source data features;

[0152] The target fusion feature determination unit is used to perform principal component analysis on the candidate fusion features to obtain the target fusion features.

[0153] Optionally, the target fusion feature determination unit includes:

[0154] A principal component vector determination subunit, used to determine the principal component vector according to the feature weight and mean vector of each candidate fusion feature;

[0155] The feature dimension reduction subunit is used to perform feature dimension reduction according to the principal component vector and the corresponding feature weight to obtain the target fusion feature.

[0156] Optional, feature dimension reduction subunit, specifically used for:

[0157] Determine the principal component score of the principal component vector according to the candidate fusion features and feature weights corresponding to the principal component vector;

[0158] The principal component vector with a larger principal component score is reduced in dimension to obtain the target fusion feature.

[0159] Optionally, a feature extraction unit, comprising:

[0160] The feature extraction subunit is used to perform statistical feature extraction, time domain feature extraction, frequency domain feature extraction and power feature extraction on the preprocessed multi-source data.

[0161] Optionally, the data fusion module 430 includes:

[0162] A data standardization unit is used to standardize the pre-processed multi-source data based on the association and matching results to obtain the data to be fused;

[0163] A posterior probability determination unit, used to determine the prior probability and likelihood value of the data to be fused, and determine the posterior probability of the data to be fused according to the prior probability and likelihood value;

[0164] The fused data determination unit is used to perform weighted summation on the data to be fused based on the posterior probability to obtain the fused data.

[0165] The multi-source data fusion device provided in the embodiment of the present application can execute the multi-source data fusion method provided in any embodiment of the present application, and has the corresponding functional modules and beneficial effects for executing the multi-source data fusion method.

[0166] Embodiment 5

[0167] Figure 5 A schematic diagram of the structure of an electronic device provided in Embodiment 5 of the present application is shown in FIG. Figure 5 As shown, the electronic device includes a processor 510, a memory 520, an input device 530 and an output device 540; the number of the processor 510 in the electronic device can be one or more. Figure 5 A processor 510 is taken as an example; the processor 510, the memory 520, the input device 530 and the output device 540 in the electronic device can be connected via a bus or other means. Figure 5 The example of connecting through bus is taken in the following.

[0168] The memory 520, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the multi-source data fusion method in the embodiment of the present application (for example, the data preprocessing module 410, the feature selection module 420 and the data fusion module 430). The processor 510 executes various functional applications and data processing of the electronic device by running the software programs, instructions and modules stored in the memory 520, that is, realizes the above-mentioned multi-source data fusion method.

[0169] The memory 520 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 520 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 520 may further include a memory remotely arranged relative to the processor 510, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0170] The input device 530 may be used to receive input character information and generate key signal input related to user settings and function control of the electronic device. The output device 540 may include a display device such as a display screen.

[0171] According to an embodiment of the present invention, the present invention also provides an electronic device, a readable storage medium and a computer program product.

[0172] Embodiment 6

[0173] Embodiment 6 of the present application also provides a storage medium containing computer executable instructions. When the computer executable instructions are executed by a computer processor, they are used to execute a multi-source data fusion method. The method includes: acquiring multi-source heterogeneous data, and preprocessing the multi-source heterogeneous data to obtain preprocessed multi-source data; performing feature selection on the preprocessed multi-source data to obtain target fusion features; performing data association mining and data matching on the preprocessed multi-source data according to the target fusion features, and performing data fusion based on the association and matching results to obtain fused data.

[0174] Of course, the storage medium containing computer executable instructions provided in the embodiment of the present application is not limited to the method operations described above, and the computer executable instructions can also execute related operations in the multi-source data fusion method provided in any embodiment of the present application.

[0175] Through the above description of the implementation method, the technicians in the relevant field can clearly understand that the present application can be implemented with the help of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application can be essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for an electronic device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0176] It is worth noting that in the embodiment of the above-mentioned multi-source data fusion device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application.

[0177] Note that the above are only preferred embodiments of the present application and the technical principles used. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. A multi-source data fusion method, characterized in that: include: Acquire multi-source heterogeneous data, and preprocess the multi-source heterogeneous data to obtain preprocessed multi-source data; Performing feature selection on the preprocessed multi-source data to obtain target fusion features; Data association mining and data matching are performed on the preprocessed multi-source data according to the target fusion features, and data fusion is performed based on the association and matching results to obtain fused data.

2. The method according to claim 1, characterized in that The performing feature selection on the preprocessed multi-source data to obtain target fusion features includes: Performing feature extraction on the preprocessed multi-source data to obtain at least two multi-source data features; Determine the correlation between the multi-source data features, and determine candidate fusion features according to the fusion threshold and the correlation between the multi-source data features; Perform principal component analysis on the candidate fusion features to obtain target fusion features.

3. The method according to claim 2, characterized in that The performing principal component analysis on the candidate fusion features to obtain target fusion features includes: Determine a principal component vector according to the feature weights and mean vectors of each of the candidate fusion features; According to the principal component vector and the corresponding feature weight, feature dimension reduction is performed to obtain the target fusion feature.

4. The method according to claim 3, characterized in that The step of performing feature dimension reduction according to the principal component vector and the corresponding feature weight to obtain the target fusion feature includes: Determining a principal component score of the principal component vector according to the candidate fusion feature and the feature weight corresponding to the principal component vector; The principal component vector with a larger principal component score is reduced in dimension to obtain the target fusion feature.

5. The method according to claim 2, characterized in that: The extracting features from the preprocessed multi-source data includes: Statistical feature extraction, time domain feature extraction, frequency domain feature extraction and power feature extraction are performed on the preprocessed multi-source data.

6. The method according to claim 1, characterized in that The data fusion is performed based on the association and matching results to obtain fused data, including: Based on the association and matching results, the preprocessed multi-source data is standardized to obtain data to be fused; Determine a priori probability and likelihood value of the data to be fused, and determine a posterior probability of the data to be fused based on the priori probability and the likelihood value; The to-be-fused data are weightedly summed based on the posterior probability to obtain the fused data.

7. A multi-source data fusion device, characterized in that: include: A data preprocessing module is used to obtain multi-source heterogeneous data and preprocess the multi-source heterogeneous data to obtain preprocessed multi-source data; A feature selection module is used to perform feature selection on the preprocessed multi-source data to obtain target fusion features; The data fusion module is used to perform data association mining and data matching on the preprocessed multi-source data according to the target fusion feature, and perform data fusion based on the association and matching results to obtain fused data.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the multi-source data fusion method as described in any one of claims 1-6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the multi-source data fusion method as described in any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the computer program implements the multi-source data fusion method according to any one of claims 1 to 6.

Images