Multi-source heterogeneous data source fusion processing method

Abstract

The invention provides a multi-source heterogeneous data source fusion processing method, and belongs to the field of data fusion. The problem of low multi-source heterogeneous data fusion efficiency is solved; the method specifically comprises the following steps: acquiring and storing numeric data and non-numeric data; dividing the numerical data into mixed data, one-dimensional data and multi-dimensional data; the mixed data are integrated, dimensions are unified, and dominant fusion data are obtained; normalizing the one-dimensional data to obtain primary fusion data, and constructing a correlation function; extracting feature values and feature vectors of the multi-dimensional data to obtain dominant fusion data of the multi-dimensional data; carrying out secondary fusion on the multi-dimensional data with correlation; in the non-numeric data, screening and fusing the non-numeric data corresponding to the extraction target to obtain non-numeric fused data; according to the method, various different structural data or non-structural data are analyzed, extracted and integrated, so that the interpretability of the data is improved.

Description

Technical Field

[0001] The present invention discloses a multi-source heterogeneous data source fusion processing method, which relates to the field of data fusion. Background Art

[0002] Existing methods for data source fusion processing have the following shortcomings:

[0003] Difficulty in data standardization: Different data sources often use different data models, coding standards, and terminology systems, which complicates data integration. For example, different hospitals may use different medical record coding systems, which makes it difficult to integrate medical data. The lack of unified standards also increases the cost and complexity of data conversion.

[0004] High computational complexity: With the increase in data volume and the number of data sources, the computational overhead of data fusion grows exponentially. Although distributed computing and parallel processing technologies can complete the fusion processing of large-scale data within a limited time, this also brings about system complexity and consistency issues.

[0005] Data interference: Multi-source heterogeneous data fusion also faces problems such as semantic heterogeneity, differences in data timeliness, and inconsistent data quality. These factors may affect the quality and usability of the fusion results and need to be considered and addressed during the fusion process. Summary of the Invention

[0006] In view of the shortcomings of the existing technology, the purpose of the present invention is to provide a multi-source heterogeneous data source fusion processing method to solve the problem of low efficiency of multi-source heterogeneous data fusion.

[0007] In order to achieve the above object, the present invention is implemented through the following technical solutions: A multi-source heterogeneous data source fusion processing method includes:

[0008] Obtain the raw data provided by the data source, create a storage structure based on the number of data sources, and save the raw data; divide the raw data into numerical data and non-numerical data based on the data type;

[0009] Extract the data dimensions corresponding to all numerical data in the data source, and divide the numerical data into mixed data and non-mixed data based on whether the data dimensions of each numerical data in the data source are consistent; then divide the non-mixed data into one-dimensional data and multi-dimensional data based on the data dimensions; integrate each data in the mixed data, perform feature decomposition, and then unify the data dimensions of each data in the mixed data to obtain explicit fused data;

[0010] Normalize each one-dimensional data to obtain the primary fusion data of each one-dimensional data; determine whether there is correlation between multiple one-dimensional data with the same dimension; if so, construct a correlation function between different one-dimensional data; if not, extract the eigenvalues ​​and eigenvectors corresponding to each multidimensional data to obtain the explicit fusion data corresponding to each multidimensional data; based on nonlinear testing, determine whether there is correlation between multiple multidimensional data with the same dimension; if so, perform secondary fusion on the multidimensional data with correlation; if not, do not process it;

[0011] Obtain the extraction target corresponding to each non-numeric data; filter and fuse the non-numeric data corresponding to the extraction target in the non-numeric data through a neural network to obtain non-numeric fused data; summarize and encrypt the explicit fusion data and correlation function of hybrid data and multidimensional data, and synchronize them to the storage structure.

[0012] Furthermore, the specific steps of creating the storage structure are as follows:

[0013] Get the number of data sources da and calculate the depth bh of the complete binary tree: construct a complete binary tree with a depth of bh and a number of nodes na as tree (1);

[0014] Create storage space in the cloud server, and then create, deploy, and set up the blockchain through the cloud platform console;

[0015] Decentralize and encrypt the original data corresponding to the 1st to dath data sources to obtain the encrypted data corresponding to the 1st to dath data sources; store the encrypted data corresponding to the 1st to dath data sources into the nodes corresponding to the tree (1) to obtain the tree (2);

[0016] Generate a root hash, and in the storage space of each non-leaf node in the tree (2), store the hash value of the non-leaf node corresponding to the leaf node; in the storage space of the two leaf nodes corresponding to the non-leaf node, store the hash value of each other's node to obtain a Merkle tree, and use the root hash as the unique fingerprint of the entire Merkle tree;

[0017] Deploy the Merkle tree in the blockchain and return the blockchain access interface to all data sources.

[0018] Furthermore, the specific steps of processing mixed data are as follows:

[0019] Take the data in the mixed data as Class A data; get the dimension of the first Class A data and get di (1) ~di (mi) , and extract the maximum value di (max) ;

[0020] Based on zero padding, the dimensions of the 1st to mith class A data are expanded to di (max) Dimension, get data Aa (1) ~Aa (mi) ;

[0021] will (max) As dim, define a zero matrix of (mi×dim) and put the data Aa (1) ~Aa ( mi ) Fill in the zero matrix and get the matrix Ba (1) ;

[0022] Calculate the matrix Ba (1) The decentralized matrix of (2) , calculate the covariance matrix Ba(3);

[0023] The matrix Ba (3) The data in is denoted as va (1,1) ~va (dim,dim) ;

[0024] Assume that the eigenvalue of type A data is λ, construct the matrix Ba (4) :

[0025]

[0026] Expand the matrix Ba based on Newton's identity (4) , we get the dim function F(λ) about λ:

[0027] Among them, ck (i) represents the (dim-i)-order coefficient of λ;

[0028] ck (i) The expression is:

[0029] Among them, ck (i-j) represents the (i-j)-order coefficient of λ;

[0030] ti (j) Represents the matrix Ba (4) The trace of the jth power of .

[0031] Furthermore, the step of processing mixed data further includes:

[0032] According to the function F(λ), the eigenvalues ​​of the 1st dimension to the dim dimension of the A-type data are calculated to obtain λ (1) ~λ (dim) ;

[0033] According to λ (1) ~λ (dim)Construct matrix Ia (1) ~Ia (dim) ;

[0034] Using Gaussian elimination, calculate the matrix Ia (1) , Ia (2) ~Ia (dim) The non-zero vector in the null space is normalized to obtain the dim-dimensional (i.e. (1×dim)) eigenvector va (1) 、va (2) ~va (dim) ;

[0035] Extracting Lambda (1) ,λ (2) ~λ (dim) The maximum value λ in (max) and the minimum value λ (min) ;

[0036] Assume that the eigenvalue of dimension a of type A data is λ (a) ; The value range of a is: 1~dim; calculate λ (1) ,λ (2) ~λ (dim) The sum average value aλ:

[0037]

[0038] The eigenvalue greater than or equal to aλ is regarded as the dominant eigenvalue, and the dominant eigenvalue is recorded as dλ (1) 、dλ (2) ~dλ (od) ; Where od represents the number of dominant eigenvalues; 0≤od<dim;

[0039] Fusion of dominant features: Construct a (dim×od) zero matrix and fill it with dλ (1) ~dλ (od) The corresponding eigenvectors are obtained as matrix XW;

[0040] Calculate the fusion matrix corresponding to class A data as matrix ZA: ZA = Ba (1) *XW;

[0041] Extract the elements in the first and second rows up to the mi-th row in the matrix ZA as the explicit fusion data of category A data.

[0042] Furthermore, the specific steps of normalizing the one-dimensional data are as follows:

[0043] Assume that the values ​​of the 1st to dmth numbers in the one-dimensional data are: oe (1) ~oe (dm) ; Set oe (1) ~oe ( dm) The maximum value in is oe (max) , the minimum value is oe (min) ;

[0044] Let the normalized value of the oth number in the one-dimensional data be uoe (o) , define formula B-1:

[0045]

[0046] Normalize the 1st to dth one-dimensional data, and use the normalized data of the 1st to dth one-dimensional data as the primary fusion data of the 1st to dth one-dimensional data.

[0047] Furthermore, the specific steps of determining whether the one-dimensional data is correlated are as follows:

[0048] Get the number of values ​​dn corresponding to the 1st to deth one-dimensional data (1) ~dn (de) ;

[0049] Determine whether the first one-dimensional data has a correlation with other one-dimensional data, and construct a correlation function corresponding to the first one-dimensional data;

[0050] The number of marked values ​​in the 2nd to deth one-dimensional data is equal to dn (1) The one-dimensional data is regarded as type B data;

[0051] If there is no Class B data in the second to the deth one-dimensional data, it means that the first one-dimensional data is independent data and no other one-dimensional data is correlated with it. The eigenvalues ​​and eigenvectors corresponding to each multidimensional data are extracted to obtain the explicit fusion data corresponding to each multidimensional data.

[0052] If there is type B data in the second to de-th one-dimensional data, then count the number of type B data ub; (1) As dnl, analyze and judge the correlation between the first one-dimensional data and the first to ub-th B-type data, and construct the correlation function corresponding to the first one-dimensional data.

[0053] Furthermore, the specific steps of constructing the correlation function are as follows:

[0054] Calculate the average value go of the first one-dimensional data; calculate the average value ga of the first to ubth B-type data (1) ~ga (ub) ;

[0055] Let the value in the first one-dimensional data be xb (1) ~xb (dnl) ;

[0056] The values ​​of the 1st to ubth B-type data are denoted as xbb (1,1) ~xbb (ub,dnl) ;

[0057] Let the value of the qth number in the first one-dimensional data be xb (q) ; The value of the qth number in the pth B-type data is xbb (p,q) ; The average value of the p-th B-type data is ga (p) ;Define formula B-2-1:

[0058]

[0059] Among them, R (1，p) Represents the linear correlation coefficient of the first one-dimensional data with respect to the p-th B-type data;

[0060] Calculate the linear correlation coefficient R of the first one-dimensional data with respect to the first to ubth B-type data (1，1) ~R (1，ub) ;

[0061] From the 1st to ubth B-type data, extract the B-type data whose linear correlation coefficient is within ε of 0.05 as Bb-type data; where ε represents the error determination coefficient;

[0062] Count the number of Bb type data ld and define the linear correlation function corresponding to the first one-dimensional data.

[0063] Furthermore, the specific steps of defining the linear correlation function are as follows:

[0064] Extract the values ​​xbx from the 1st to ldth Bb class data (1,1) ~~xbx (ld,dnl) ;

[0065] Define matrix BX, matrix BY;

[0066] Let the coefficient of the first one-dimensional matrix for the first to ldth Bc class data be β (1) ~β (ld) ;Define matrix BB;

[0067] Calculating β (1) ~β (ld) Value: BB=BY*((BX T *BX) -1 *BX T ).

[0068] Furthermore, the step of constructing the correlation function further includes:

[0069] Count ul Bc class data; extract the average value gc(1)~gc of the 1st to ulth Bc class data (ul) ;

[0070] Extract the linear correlation coefficient Rc corresponding to the 1st to ulth Bc class data (1) ~Rc (ul) ;

[0071] Extract the value xbc from the 1st to ulth Bc class data (1,1) ~xbc (ul,dnl) ;

[0072] The value of the qth number in the tth Bc type data is xbc (t,q) ; The average value of the t-th Bc class data is gc (t) ; The linear correlation coefficient Rc corresponding to the t-th Bc class data (t) ;

[0073] Define formula B-2-2:

[0074]

[0075] Among them, ρ (1，t) Represents the nonlinear correlation coefficient of the first one-dimensional data with respect to the t-th Bc class data;

[0076] f (1，t) Represents the antilinear correlation coefficient of the first one-dimensional data with respect to the t-th Bc class data;

[0077] Calculate the nonlinear correlation coefficient ρ of the 1st to ulth Bc class data (1，1) ~ρ (1，ul) ;

[0078] Extract the Bc type data whose nonlinear correlation coefficient is within ε of 0.05 as Bd type data;

[0079] The number of Bd class data ud is counted, the first one-dimensional data is used as the dependent variable, and the first to udth Bd class data are used as independent variables; the local weighted regression algorithm and neural network are used to fit the nonlinear function of the first one-dimensional data with respect to the first to udth Bd class data.

[0080] Compared with the prior art, the present invention has the following beneficial effects:

[0081] Efficient hybrid data processing capabilities The present invention demonstrates excellent efficiency and quality in hybrid data processing. Through dimensional alignment and feature decomposition techniques, the present invention can effectively handle complex situations with inconsistent data dimensions. The specific process includes: identifying mixed data, determining the maximum data dimension, aligning data of different dimensions through zero padding, and then performing feature decomposition to extract key information. This processing method ensures the comparability and fusion quality of data of different dimensions.

[0082] Intelligent data correlation analysis: For one-dimensional data, the system adopts normalization processing, and then accurately identifies the correlation between data through linear correlation coefficient calculation and nonlinear function fitting, automatically distinguishes linear correlation and nonlinear correlation, and constructs corresponding function models respectively, which greatly improves the accuracy of analysis.

[0083] Unstructured data processing capabilities: The present invention demonstrates significant technical advantages in processing unstructured data. For non-numeric data such as images, text, audio, and video, the present invention uses neural network technology for intelligent processing. Based on user-defined extraction targets, the neural network can automatically filter and fuse relevant data, greatly improving the processing efficiency and accuracy of unstructured data. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0085] Figure 1 Schematic diagram of the method of the present invention;

[0086] Figure 2 It is a schematic diagram of the process of the present invention;

[0087] Figure 3 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0088] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0089] See also Figure 1 A multi-source heterogeneous data source fusion processing method includes:

[0090] Step S1: Obtain the raw data provided by the data source, and create a storage structure according to the number of data sources to save the raw data; according to the data type (in the raw data), the raw data is divided into numerical data and non-numerical data;

[0091] It should be noted that the data source in the present invention refers to the individual, unit or department that uses the present invention (a target quantity statistics method) to perform mathematical statistics;

[0092] The specific steps of step S1 are as follows:

[0093] Get the number of data sources da and calculate the depth bh of the complete binary tree:

[0094] Among them, log2 (da) is a positive integer, rounded down;

[0095] Use the build_binary_tree function to construct a complete binary tree with a depth of bh and a number of nodes na as tree (1);

[0096] Create storage space in the cloud server, and then create, deploy, and set up the blockchain through the cloud platform console;

[0097] Decentralize and encrypt the original data corresponding to the 1st to dath data sources to obtain the encrypted data corresponding to the 1st to dath data sources;

[0098] Taking the root node of tree (1) as the starting node, store the encrypted data corresponding to the 1st to dath data sources into the corresponding nodes of tree (1) in order from top to bottom and from left to right, and obtain tree (2);

[0099] Starting from the first leaf node at the bottom of the tree (2), hash upward layer by layer to generate a root hash (MerkleRoot);

[0100] In the storage space of each non-leaf node of tree (2), the hash value of the leaf node corresponding to the non-leaf node is additionally stored in the format of "left child hash + right child hash → parent hash"; in the storage space of the two leaf nodes corresponding to the non-leaf node, the hash values ​​of each other's nodes are stored; (For example: the two leaf nodes corresponding to the non-leaf node are leaf node (1) and leaf node (2); in leaf node (1), the hash value of leaf node (2) is stored; in leaf node (2), the hash value of leaf node (1) is stored;)

[0101] Get the Merkle tree and use the root hash as the unique fingerprint of the entire Merkle tree;

[0102] Deploy the Merkle tree in the blockchain and return the blockchain access interface to all data sources.

[0103] Step S2: (Processing Numerical Data) Extract the data dimensions corresponding to all numerical data in the data source, and divide the numerical data into mixed data (inconsistent) and non-mixed data (consistent) based on whether the data dimensions of each numerical data in the data source are consistent; then divide the non-mixed data into one-dimensional data and multi-dimensional data based on the data dimensions; integrate each data in the mixed data, perform feature decomposition, and then unify the data dimensions of each data in the mixed data to obtain explicit fused data;

[0104] See also Figure 2 , the specific steps of step S2 are as follows:

[0105] Extract the data dimensions corresponding to all numerical data in the data source. Based on whether the data dimensions of each numerical data in the data source are consistent, divide the numerical data into mixed data (inconsistent) and non-mixed data (consistent). Then, divide the non-mixed data into one-dimensional data and multi-dimensional data based on the data dimensions.

[0106] Handling mixed data:

[0107] Treat the data in the mixed data as Class A data;

[0108] Count the number of type A data mi; get the dimensions of the first, second, and mi-th type A data, and get di (1) 、di (2) ~di (mi) ; Extract di (1) ~di (mi) The maximum value di (max) ;

[0109] Define the filling format of type A data:

[0110] Let the data dimension of the a-th type A data be di (a) ; Let the 1st, 2nd, and dth data in the ath category A be (a) The value of the dimension is xa (1) ,xa (2) ~xa (da(a)) ; Among them, the value range of a is: 1~mi;

[0111] Definition (1×di (a) ) represents the a-th type A data; the matrix XA is:

[0112] (1 line di (a) Column, that is, di (a) dimension);

[0113] Definition (1×di (max) )’s matrix YA represents the a-th A-type data after alignment;

[0114] Comparison (a) With di (max) The size of , determines the format of matrix YA:

[0115] If (a) =di (max) , then the a-th type A data after alignment is itself, that is, the format of matrix YA is the same as that of matrix XA;

[0116] If (a) <di (max) , then the format of matrix YA is:

[0117]

[0118] (1 line di (max) Column, that is, di (max) Dimension; in xa( di(a) ) and then fill (di (max) -di (a) )0);

[0119] Based on the filling format of the A-type data (i.e., zero filling), the dimensions of the first, second, and mi-th A-type data are expanded to di (max) Dimension, get data Aa (1) 、Aa (2) ~Aa (mi) ; (where data Aa (1) Indicates the first A-type data after alignment, data Aa (2) Indicates the second type A data after alignment; and so on, data Aa (mi) represents the mi-th A-type data after alignment);

[0120] will (max) As dim, define a zero matrix of (mi×dim) and put the data Aa (1) ~Aa ( mi ) Fill in the zero matrix and get the matrix Ba (1) ;

[0121] Calculate the matrix Ba (1) The decentralized matrix of (2) ;

[0122] Calculate the covariance matrix Ba (3) : Where * represents matrix multiplication, T represents the transpose of the matrix; the covariance matrix Ba (3) The dimension is (dim×dim);

[0123] The matrix Ba(3) The data in is denoted as va (1,1) 、va (1,2) ~va (1,dim) 、va (2,1) ~va (dim,dim) Matrix Ba (3) for:

[0124]

[0125] Assume that the eigenvalue of type A data is λ, construct the matrix Ba (4) :

[0126]

[0127] Expand the matrix Ba based on Newton's identity (4) , we get the dim function F(λ) about λ:

[0128] Among them, the value range of i is: 0~dim;ck (i) represents the (dim-i)-order coefficient of λ;

[0129] ck (i) The expression is:

[0130] Among them, the value range of j is: 1~i;ck (i-j) represents the (i-j)-order coefficient of λ;

[0131] ti (j) Represents the matrix Ba (4) The trace of the jth power of ; (the trace of a matrix is ​​the sum of the diagonal elements of the matrix);

[0132] According to the function F(λ), calculate the eigenvalues ​​of the 1-dimensional, 2-dimensional, and dim-dimensional data of type A to obtain λ (1) ,λ (2) ~λ (dim) ;

[0133] According to λ (1) ,λ (2) ~λ (dim) Construct matrix Ia (1) , Ia (2) ~Ia (dim) Among them, Ia (1) The calculation formula is: Ia (1) =Ba (4) -(λ (1) *Ii); where Ii represents the (dim×dim) identity matrix;

[0134] Ia (2) The calculation formula is: Ia(2) =Ba (4) -(λ (2) *Ii);

[0135] By analogy, Ia (dim) The calculation formula is: Ia (dim) =Ba (4) -(λ (dim) *Ii);

[0136] Using Gaussian elimination, calculate the matrix Ia (1) , Ia (2) ~Ia (dim) The non-zero vector in the null space is normalized to obtain the dim-dimensional (i.e. (1×dim)) eigenvector va (1) 、va (2) ~va (dim) ;

[0137] Extracting Lambda (1) ,λ (2) ~λ (dim) The maximum value λ in (max) and the minimum value λ (min) ;

[0138] Assume that the eigenvalue of dimension a of type A data is λ (a) ; The value range of a is: 1~dim; calculate λ (1) ,λ (2) ~λ (dim) The sum average value aλ:

[0139]

[0140] The eigenvalue greater than or equal to aλ is regarded as the dominant eigenvalue, and the dominant eigenvalue is recorded as dλ (1) 、dλ (2) ~dλ (od) ; Where od represents the number of dominant eigenvalues; 0≤od<dim;

[0141] Fusion of dominant features: Construct a (dim×od) zero matrix and fill it with dλ (1) ~dλ (od) The corresponding eigenvectors are obtained as matrix XW;

[0142] Calculate the fusion matrix corresponding to class A data as matrix ZA: ZA = Ba (1) *XW;

[0143] Extract the elements in the first and second rows up to the mi-th row in the matrix ZA as the explicit fusion data of category A data.

[0144] Step S3: normalize each one-dimensional data to obtain primary fusion data of each one-dimensional data; determine whether there is correlation between multiple one-dimensional data with the same dimension; if so, construct a correlation function between different one-dimensional data; if not, extract the eigenvalues ​​and eigenvectors corresponding to each multidimensional data to obtain explicit fusion data corresponding to each multidimensional data; based on the nonlinear test (manifold learning), determine whether there is correlation between multiple multidimensional data with the same dimension; if so, perform secondary fusion on the multidimensional data with correlation; if not, do not process;

[0145] The specific steps of step S3 are as follows:

[0146] See also Figure 3 , step S31: counting the number of one-dimensional data de;

[0147] Define the calculation formula B-1 for one-dimensional data normalization;

[0148] Suppose the values ​​of the 1st, 2nd, and dmth numbers in the one-dimensional data are: oe (1) 、oe (1) ~oe (dm) ; Where dm represents the number of values ​​in one-dimensional data;

[0149] Set oe (1) ~oe (dm) The maximum value in is oe (max) , the minimum value is oe (min) ;

[0150] Let the normalized value of the oth number in the one-dimensional data be uoe (o) ; The value range of o is: 1 ~ dm; Definition formula B-1:

[0151]

[0152] According to formula B-1, the 1st to deth one-dimensional data are normalized, and the normalized data of the 1st to deth one-dimensional data are used as the primary fusion data of the 1st to deth one-dimensional data;

[0153] Step S32: Get the number of values ​​corresponding to the first, second, and deth one-dimensional data, and get dn (1) 、dn (2) ~dn (de) ;

[0154] Determine whether the first one-dimensional data has a correlation with other one-dimensional data, and construct a correlation function corresponding to the first one-dimensional data;

[0155] The number of marked values ​​in the 2nd to deth one-dimensional data is equal to dn (1)The one-dimensional data of is taken as the type B data (i.e. the alternative one-dimensional data for the first one-dimensional data);

[0156] If there is no Class B data in the second to deth one-dimensional data, it means that the first one-dimensional data is independent data and no other one-dimensional data is correlated with it. Step S33 is skipped and step S35 is executed (the eigenvalues ​​and eigenvectors corresponding to each multidimensional data are extracted to obtain the explicit fusion data corresponding to each multidimensional data).

[0157] If there is type B data in the second to de-th one-dimensional data, then count the number of type B data ub; (1) As dnl, analyze and determine the correlation between the first one-dimensional data and the first to ub-th B-type data, construct a correlation function corresponding to the first one-dimensional data, and execute step S33;

[0158] Step S33: Calculate the average value go of the first one-dimensional data; calculate the average value ga of the first, second, and up to ub-th B-type data (1) ,ga (2) ~ga (ub) ;

[0159] Let the value in the first one-dimensional data be xb (1) 、xb (2) ~xb (dnl) ;

[0160] The value in the first type B data is denoted as xbb (1,1) 、xbb (1,2) ~xbb (1,dnl) ;

[0161] The value in the second type B data is denoted as xbb (2,1) 、xbb (2,2) ~xbb (2,dnl) ;

[0162] Similarly, the value of the ubth B-type data is recorded as xbb (ub,1) 、xbb (ub,2) ~xbb(ub,dnl);

[0163] Let the value of the qth number in the first one-dimensional data be xb (q) ;The value range of q is: 1~dnl;

[0164] The value of the qth number in the pth B-type data is xbb (p,q) ; The average value of the p-th B-type data is ga (p) ;The value range of p is: 1~ub;

[0165] Define formula B-2-1:

[0166]

[0167] Among them, R (1，p) Represents the linear correlation coefficient of the first one-dimensional data with respect to the p-th B-type data;

[0168] According to formula B-2-1, calculate the linear correlation coefficient R of the first one-dimensional data with respect to the first, second, and up to the ubth B-type data. (1，1) 、R (1，2) ~R (1，ub) ;

[0169] From the 1st to the ubth B-type data, extract the B-type data whose linear correlation coefficient is within ε of 0.05 as Bb-type data (i.e., linear correlation data);

[0170] Wherein, ε represents the error determination coefficient; (the value of ε is 0.01; users or relevant technical personnel can adjust the value of ε according to actual needs);

[0171] Step S331: Count the number of Bb-type data ld and define the linear correlation function corresponding to the first one-dimensional data;

[0172] Extract the value xbx from the first Bb type data (1,1) 、xbx (1,2) ~xbx (1,dnl) ;

[0173] The value xbx in the second Bb type data (2,1) 、xbx (2,2) ~xbx (2,dnl) ;

[0174] Similarly, the value xbx in the ldth Bc class data (ld,1) 、xbx (ld,2) ~xbx(ld,dnl);

[0175] in,

[0176] Define the zero matrix (ld×dnl) and fill in xbx (1,1) ~xbx (ld,dnl) , get the matrix BX;

[0177] Define the zero matrix (1×dnl) and fill it with xb (1) ~xb (dnl) , get the matrix BY;

[0178] Let the coefficient of the first one-dimensional matrix with respect to the first, second, and ldth Bc class data be β (1) , β (2) ~β(ld) ;

[0179] Define a zero matrix (1×ld) and fill it with β (1) ~β (ld) , get the matrix BB;

[0180] Calculating β (1) ~β (ld) Value: BB=BY*((BX T *BX) -1 *BX T );

[0181] Step S332: Eliminate the Bb-type data from the 1st to ubth B-type data to obtain Bc-type data; and count the number of Bc-type data ul;

[0182] Extract the average value gc of the 1st, 2nd, and ulth Bc class data (1) 、gc (2) ~gc (ul) ;in,

[0183] Extract the linear correlation coefficient Rc corresponding to the 1st, 2nd, and ulth Bc class data (1) 、Rc (2) ~Rc (ul) ;

[0184] in,

[0185] Extract the value xbc from the first Bc type data (1,1) 、xbc (1,2) ~xbc (1,dnl) ;

[0186] The value xbc in the second Bc type data (2,1) 、xbc (2,2) ~xbc (2,dnl) ;

[0187] Similarly, the value xbc in the ul-th Bc class data (ul,1) 、xbc (ul,2) ~xbc(ul,dnl);

[0188] in,

[0189] The value of the qth number in the tth Bc type data is xbc (t,q) ; The average value of the t-th Bc class data is gc (t) ; The value range of t is: 1~ul;

[0190] The linear correlation coefficient Rc corresponding to the t-th Bc class data(t) ;

[0191] Define formula B-2-2:

[0192]

[0193] Among them, ρ (1，t) Represents the nonlinear correlation coefficient of the first one-dimensional data with respect to the t-th Bc class data;

[0194] f (1，t) Represents the antilinear correlation coefficient of the first one-dimensional data with respect to the t-th Bc class data;

[0195] According to formula B-2-2, calculate the nonlinear correlation coefficient ρ of the first, second, and up to the ulth Bc class data (1，1) , ρ (1，2) ~ρ (1，ul) ;

[0196] From the 1st to ulth Bc class data, extract the Bc class data whose nonlinear correlation coefficient is within ε of 0.05 as Bd class data (i.e., linear correlation class data);

[0197] Count the number of Bd-class data ud, with the first one-dimensional data as the dependent variable and the first to ud-th Bd-class data as independent variables; use the polynomial regression or local weighted regression algorithm in numpy.polyfit / statsmodels.api and the generalized additive model or neural network in pygam / sklearn.neural_network to fit the nonlinear functional relationship between the first one-dimensional data and the first to ud-th Bd-class data;

[0198] It should be noted that when multiple sets of data satisfy a nonlinear relationship, the relationship between the corresponding data points of these data is not a simple straight line or plane, and may involve curves, surfaces, or other complex forms. Common nonlinear models include exponential functions, logarithmic functions, polynomial functions, trigonometric functions, etc. Therefore, the present invention can only determine the existence of a nonlinear relationship between certain one-dimensional data, and cannot provide a general analytical expression for a nonlinear relationship.

[0199] Step S34: Repeat the same steps of constructing the correlation function of the first one-dimensional data to construct the correlations corresponding to the second to deth one-dimensional data;

[0200] Step S35: Repeat the calculation steps of extracting the eigenvalues ​​and eigenvectors of the mixed data, extract the eigenvalues ​​and eigenvectors corresponding to each multidimensional data, and obtain the explicit fusion data corresponding to each multidimensional data;

[0201] Based on the nonlinear test (manifold learning), it is determined whether there is correlation between multiple multidimensional data with the same dimension; if so, the multidimensional data with correlation is secondary fused; if not, no processing is done;

[0202] Step S4: (Processing non-numeric data, i.e., unstructured data such as images, text, audio, and video) Obtain the extraction target corresponding to each non-numeric data (i.e., the unstructured data that the user needs to fuse or extract); Through the neural network, the non-numeric data corresponding to the extraction target is screened and fused in the non-numeric data to obtain non-numeric fused data;

[0203] The explicit fusion data of the hybrid data and multi-dimensional data and the correlation function (of the one-dimensional data) are aggregated and encrypted, and synchronized to the storage structure (created in step S1).

[0204] The above formulas are all dimensionless and calculated by taking their numerical values. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions. For example, if there are weight coefficients and proportional coefficients, the size of the settings is to quantify each parameter to obtain a specific value, which is convenient for subsequent comparison. Regarding the size of the weight coefficient and the proportional coefficient, as long as it does not affect the proportional relationship between the parameter and the quantized value, it is fine.

[0205] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A method for fusion processing of multi-source heterogeneous data sources, characterized in that: The method comprises: Obtain the raw data provided by the data source, create a storage structure based on the number of data sources, and save the raw data; divide the raw data into numerical data and non-numerical data based on the data type; Extract the data dimensions corresponding to all numerical data in the data source, and divide the numerical data into mixed data and non-mixed data based on whether the data dimensions of each numerical data in the data source are consistent; then divide the non-mixed data into one-dimensional data and multi-dimensional data based on the data dimensions; integrate each data in the mixed data, perform feature decomposition, and then unify the data dimensions of each data in the mixed data to obtain explicit fused data; Normalize each one-dimensional data to obtain the primary fusion data of each one-dimensional data; determine whether there is correlation between multiple one-dimensional data with the same dimension; if so, construct a correlation function between different one-dimensional data; if not, extract the eigenvalues ​​and eigenvectors corresponding to each multidimensional data to obtain the explicit fusion data corresponding to each multidimensional data; based on nonlinear testing, determine whether there is correlation between multiple multidimensional data with the same dimension; if so, perform secondary fusion on the multidimensional data with correlation; if not, do not process it; Obtain the extraction target corresponding to each non-numeric data; filter and fuse the non-numeric data corresponding to the extraction target in the non-numeric data through a neural network to obtain non-numeric fused data; summarize and encrypt the explicit fusion data and correlation function of hybrid data and multidimensional data, and synchronize them to the storage structure.

2. A multi-source heterogeneous data source fusion processing method according to claim 1, characterized in that: The specific steps to create a storage structure are as follows: Get the number of data sources da and calculate the depth bh of the complete binary tree: construct a complete binary tree with a depth of bh and a number of nodes na as tree (1); Create storage space in the cloud server, and then create, deploy, and set up the blockchain through the cloud platform console; Decentralize and encrypt the original data corresponding to the 1st to dath data sources to obtain the encrypted data corresponding to the 1st to dath data sources; store the encrypted data corresponding to the 1st to dath data sources into the nodes corresponding to the tree (1) to obtain the tree (2); Generate a root hash, and in the storage space of each non-leaf node in the tree (2), store the hash value of the non-leaf node corresponding to the leaf node; in the storage space of the two leaf nodes corresponding to the non-leaf node, store the hash value of each other's node to obtain a Merkle tree, and use the root hash as the unique fingerprint of the entire Merkle tree; Deploy the Merkle tree in the blockchain and return the blockchain access interface to all data sources.

3. The method for fusion processing of multi-source heterogeneous data sources according to claim 1, characterized in that: The specific steps for processing mixed data are as follows: Take the data in the mixed data as Class A data; get the dimension of the first Class A data and get di (1) ~di (mi) , and extract the maximum value di (max) ; Based on zero padding, the dimensions of the 1st to mith class A data are expanded to di (max) Dimension, get data Aa (1) ~Aa (mi) ; will (max) As dim, define the matrix Ba (1) ; Calculate the matrix Ba (1) The decentralized matrix of (2) , calculate the covariance matrix Ba(3); The matrix Ba (3) The data in is denoted as va (1,1) ~va (dim,dim) ; Assume that the eigenvalue of type A data is λ, construct the matrix Ba (4) : Expand matrix Ba (4) , we get the dim function F(λ) about λ: Among them, ck (i) Represents the (dim-i)-order coefficient of λ.

4. The method for fusion processing of multi-source heterogeneous data sources according to claim 3, characterized in that: The steps for processing mixed data also include: According to the function F(λ), the eigenvalues ​​of the 1st dimension to the dim dimension of the A-type data are calculated to obtain λ (1) ~λ (dim) ; According to λ (1) ~λ (dim) Construct matrix Ia (1) ~Ia (dim) ; Using Gaussian elimination, calculate the matrix Ia (1) ~Ia (dim) The non-zero vector in the null space is normalized to obtain the dim-dimensional eigenvector va (1) ~va (dim) ; Extracting Lambda (1) ~λ (dim) The maximum value λ in (max) and the minimum value λ (min) ; Assume that the eigenvalue of dimension a of type A data is λ (a) ; The value range of a is: 1~dim; calculate λ (1) ~λ (dim) The sum average value aλ: The eigenvalue greater than or equal to aλ is regarded as the dominant eigenvalue, and the dominant eigenvalue is recorded as dλ (1) ~dλ (od) ; Where od represents the number of dominant eigenvalues; Fusion of dominant features: Construct a (dim×od) zero matrix and fill it with dλ (1) ~dλ (od) The corresponding eigenvectors are obtained as matrix XW; Calculate the fusion matrix corresponding to class A data as matrix ZA: ZA = Ba (1) *XW; Extract the elements from row 1 to row mi in the matrix ZA as the explicit fusion data of category A data.

5. The method for fusion processing of multi-source heterogeneous data sources according to claim 1, characterized in that: The specific steps to normalize one-dimensional data are as follows: Assume that the values ​​of the 1st to dmth numbers in the one-dimensional data are: oe (1) ~oe (dm) ; Set oe (1) ~oe ( dm ) The maximum value in is oe (max) , the minimum value is oe (min) ; Let the normalized value of the oth number in the one-dimensional data be uoe (o) , define formula B-1: Normalize the 1st to dth one-dimensional data, and use the normalized data of the 1st to dth one-dimensional data as the primary fusion data of the 1st to dth one-dimensional data.

6. The method for fusion processing of multi-source heterogeneous data sources according to claim 1, characterized in that: The specific steps to determine whether one-dimensional data is correlated are as follows: Get the number of values ​​dn corresponding to the 1st to deth one-dimensional data (1) ~dn (de) ; Determine whether the first one-dimensional data has a correlation with other one-dimensional data, and construct a correlation function corresponding to the first one-dimensional data; The number of marked values ​​in the 2nd to deth one-dimensional data is equal to dn (1) The one-dimensional data is regarded as type B data; If there is no Class B data in the second to deth one-dimensional data, it means that the first one-dimensional data is independent data. The eigenvalue and eigenvector corresponding to each multidimensional data are extracted to obtain the explicit fusion data corresponding to each multidimensional data. If there is type B data in the second to de-th one-dimensional data, then count the number of type B data ub; (1) As dnl, analyze and judge the correlation between the first one-dimensional data and the first to ub-th B-type data, and construct the correlation function corresponding to the first one-dimensional data.

7. The method for fusion processing of multi-source heterogeneous data sources according to claim 6, characterized in that: The specific steps to construct the correlation function are as follows: Calculate the average value go of the first one-dimensional data; calculate the average value ga of the first to ubth B-type data (1) ~ga (ub) ; Let the value in the first one-dimensional data be xb (1) ~xb (dnl) ; The values ​​of the 1st to ubth B-type data are denoted as xbb (1,1) ~xbb (ub,dnl) ; Let the value of the qth number in the first one-dimensional data be xb (q) ; The value of the qth number in the pth B-type data is xbb (p,q) ; The average value of the p-th B-type data is ga (p) ;Define formula B-2-1: R (1，p) Represents the linear correlation coefficient of the first one-dimensional data with respect to the p-th B-type data; Calculate the linear correlation coefficient R of the first one-dimensional data with respect to the first to ubth B-type data (1，1) ~R (1，ub) ; From the 1st to ubth B-type data, extract the B-type data whose linear correlation coefficient is within ε of 0.05 as Bb-type data; where ε represents the error determination coefficient; Count the number of Bb type data ld and define the linear correlation function corresponding to the first one-dimensional data.

8. The method for fusion processing of multi-source heterogeneous data sources according to claim 7, characterized in that: The specific steps to define the linear correlation function are as follows: Extract the values ​​xbx from the 1st to ldth Bb class data (1,1) ~~xbx (ld,dnl) ; Define matrix BX, matrix BY; Let the coefficient of the first one-dimensional matrix for the first to ldth Bc class data be β (1) ~β (ld) ;Define matrix BB; Calculating β (1) ~β (ld) The value of BB=BY*((BX T *BX) -1 *BX T ).

9. The method for fusion processing of multi-source heterogeneous data sources according to claim 7, characterized in that: The steps of constructing the correlation function also include: Count ul Bc class data; extract the average value gc of the 1st to ulth Bc class data (1) ~gc (ul) ; Extract the linear correlation coefficient Rc corresponding to the 1st to ulth Bc class data (1) ~Rc (ul) ; Extract the value xbc from the 1st to ulth Bc class data (1,1) ~xbc (ul,dnl) ; The value of the qth number in the tth Bc type data is xbc (t,q) ; The average value of the t-th Bc class data is gc (t) ; The linear correlation coefficient Rc corresponding to the t-th Bc class data (t) ; Define formula B-2-2: Among them, ρ (1，t) Represents the nonlinear correlation coefficient of the first one-dimensional data with respect to the t-th Bc class data; f (1，t) Represents the antilinear correlation coefficient of the first one-dimensional data with respect to the t-th Bc class data; Calculate the nonlinear correlation coefficient ρ of the 1st to ulth Bc class data (1，1) ~ρ (1，ul) ; Extract the Bc type data whose nonlinear correlation coefficient is within ε of 0.05 as Bd type data; The number of Bd class data ud is counted, the first one-dimensional data is used as the dependent variable, and the first to udth Bd class data are used as independent variables; the local weighted regression algorithm and neural network are used to fit the nonlinear function of the first one-dimensional data with respect to the first to udth Bd class data.

Images