A multi-source heterogeneous data source fusion processing method

Abstract

The application provides a multi-source heterogeneous data source fusion processing method, and belongs to the field of data fusion; solves the problem of low multi-source heterogeneous data fusion efficiency; specifically as follows: acquiring and saving numerical data and non-numerical data; dividing the numerical data into mixed data, one-dimensional data and multi-dimensional data; integrating the mixed data, unifying the dimension, and obtaining explicit fusion data; normalizing the one-dimensional data, obtaining primary fusion data, and constructing a correlation function; extracting eigenvalues and eigenvectors of the multi-dimensional data, and obtaining explicit fusion data of the multi-dimensional data; performing secondary fusion on the multi-dimensional data with correlation; in the non-numerical data, screening and fusing the non-numerical data corresponding to the extraction target, and obtaining non-numerical fusion data; the application improves the interpretability of data by analyzing, extracting and integrating various different structural data or non-structural data.

Description

Technical Field

[0001] This invention relates to a method for fusion processing of multi-source heterogeneous data sources, and pertains to the field of data fusion. Background Technology

[0002] Existing methods for data source fusion processing have the following shortcomings:

[0003] Data standardization is difficult: different data sources often use different data models, coding standards, and terminology systems, which makes data integration complex; for example, different hospitals may use different medical record coding systems, which makes medical data integration difficult; the lack of unified standards also increases the cost and complexity of data conversion.

[0004] High computational complexity: As the amount of data increases and the number of data sources increases, the computational overhead of data fusion grows exponentially. Although distributed computing and parallel processing technologies can complete the fusion processing of large-scale data in a limited time, this also brings system complexity and consistency issues.

[0005] Data interference: Multi-source heterogeneous data fusion also faces problems such as semantic heterogeneity, differences in data timeliness, and inconsistent data quality; these factors may affect the quality and usability of the fusion results and need to be considered and handled during the fusion process. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the present invention aims to provide a method for fusion processing of multi-source heterogeneous data sources, thereby solving the problem of low fusion efficiency of multi-source heterogeneous data.

[0007] To achieve the above objectives, the present invention provides a method for fusion processing of multi-source heterogeneous data sources, comprising:

[0008] Obtain the raw data provided by the data source, create a storage structure based on the number of data sources, and save the raw data; classify the raw data into numerical data and non-numerical data according to the data type.

[0009] Extract all data dimensions corresponding to the numerical data in the data source. Based on whether the data dimensions of each numerical data in the data source are consistent, divide the numerical data into mixed data and non-mixed data. Then, based on the data dimensions, divide the non-mixed data into one-dimensional data and multi-dimensional data. Integrate each data in the mixed data and perform feature decomposition. Then, unify the data dimensions of each data in the mixed data to obtain explicit fused data.

[0010] Normalize each one-dimensional data point to obtain the initial fused data for each one-dimensional data point; determine whether there is a correlation between multiple one-dimensional data points with the same dimension; if so, construct a correlation function between different one-dimensional data points; if not, extract the feature values ​​and feature vectors corresponding to each multi-dimensional data point to obtain the explicit fused data for each multi-dimensional data point; based on a nonlinearity test, determine whether there is a correlation between multiple multi-dimensional data points with the same dimension; if so, perform secondary fusion on the multi-dimensional data points with correlation; if not, do not process them.

[0011] Obtain the extraction target corresponding to each non-numerical data; through a neural network, filter and fuse the non-numerical data that corresponds to the extraction target to obtain non-numerical fused data; summarize and encrypt the explicit fused data of mixed data and multidimensional data, as well as the correlation function, and synchronize them to the storage structure.

[0012] Furthermore, the specific steps for creating the storage structure are as follows:

[0013] Get the number of data sources da, calculate the depth bh of the complete binary tree: construct a complete binary tree with depth bh and number of nodes na, as the tree (1);

[0014] Create storage space on a cloud server, and then create, deploy, and configure the blockchain through the cloud platform console;

[0015] The original data corresponding to the first to the dath data sources are encrypted in a decentralized manner to obtain the encrypted data corresponding to the first to the dath data sources; the encrypted data corresponding to the first to the dath data sources are stored in the nodes corresponding to the tree (1) to obtain the tree (2);

[0016] Generate a root hash. In the storage space of each non-leaf node in tree (2), additionally store the hash value of the leaf node corresponding to the non-leaf node. In the storage space of the two leaf nodes corresponding to the non-leaf node, store the hash value of the other node to obtain the Merkle tree, and use the root hash as the unique fingerprint of the entire Merkle tree.

[0017] Deploy the Merkle tree in the blockchain and return an access interface to the blockchain to all data sources.

[0018] Furthermore, the specific steps for processing mixed-type data are as follows:

[0019] Treat the data in the mixed dataset as category A; obtain the dimension of the first category A data, and get di. (1) ~di (mi) And extract the maximum value di (max) ;

[0020] Based on zero-padding, the dimensions of the first to mi-th A-class data are expanded to di. (max) Dimension, to obtain data Aa (1) ~Aa (mi) ;

[0021] di (max) As dim, define a zero matrix of (mi × dim) and put the data Aa (1) ~Aa ( mi ) Filling it into the zero matrix yields matrix Ba. (1) ;

[0022] Calculate matrix Ba (1) The decentralized matrix is ​​obtained as matrix Ba. (2) Calculate the covariance matrix Ba(3);

[0023] matrix Ba (3) The data in the middle is denoted as va (1,1) ~va (dim,dim) ;

[0024] Let the eigenvalues ​​of class A data be λ, and construct matrix Ba. (4) :

[0025]

[0026] Based on the expansion matrix Ba of Newton's identity (4) We obtain a dim function F(λ) with respect to λ:

[0027] Among them, ck (i) Let represent the coefficient of the (dim-i)th term of λ;

[0028] ck (i) The expression is:

[0029] Among them, ck (i-j) Let represent the coefficient of the (i-j)th term of λ;

[0030] ti (j) Represents matrix Ba (4) The trace of the j-th power.

[0031] Furthermore, the step of processing mixed-type data also includes:

[0032] Based on the function F(λ), calculate the eigenvalues ​​of class A data from 1 to dim dimensions, and obtain λ. (1) ~λ (dim) ;

[0033] According to λ (1) ~λ (dim)Construct matrix Ia (1) ~Ia (dim) ;

[0034] Calculate matrix Ia using Gaussian elimination. (1) Ia (2) ~Ia (dim) The non-zero vectors in the null space are then normalized to obtain the eigenvectors va of dimension dim (i.e., (1×dim)). (1) va (2) ~va (dim) ;

[0035] Extracting λ (1) , λ (2) ~λ (dim) The maximum value λ in (max) and minimum value λ (min) ;

[0036] Let the a-dimensional eigenvalue of class A data be λ. (a) The value of 'a' ranges from 1 to 'dim'. Calculate λ. (1) , λ (2) ~λ (dim) The combined mean aλ:

[0037]

[0038] Eigenvalues ​​greater than or equal to aλ are considered dominant eigenvalues, and are denoted as dλ. (1) , dλ (2) ~dλ (od) Where od represents the number of dominant eigenvalues; 0 ≤ od < dim;

[0039] Fusion of dominant features: Construct a (dim×od) zero matrix and fill it with dλ. (1) ~dλ (od) The corresponding eigenvectors yield the matrix XW;

[0040] Calculate the fusion matrix corresponding to class A data, as matrix ZA: ZA = Ba (1) *XW;

[0041] Extract the elements from the first row, the second row, up to the mi row of matrix ZA, as explicit fused data for class A data.

[0042] Furthermore, the specific steps for normalizing the one-dimensional data are as follows:

[0043] Let the values ​​of the 1st to the dmth elements in the one-dimensional data be: oe (1) ~oe (dm) Let oe (1) ~oe ( dm) The maximum value in is oe (max) The minimum value is oe (min) ;

[0044] Let the normalized value of the 0th number in one-dimensional data be uoe. (o) Define formula B-1:

[0045]

[0046] The first to the de one-dimensional data are normalized, and the normalized data of the first to the de one-dimensional data are used as the primary fusion data of the first to the de one-dimensional data.

[0047] Furthermore, the specific steps for determining whether one-dimensional data has a correlation are as follows:

[0048] Get the number of values ​​dn corresponding to the first to the de one-dimensional data. (1) ~dn (de) ;

[0049] Determine whether there is a correlation between the first one-dimensional data and other one-dimensional data, and construct the correlation function corresponding to the first one-dimensional data;

[0050] The number of labeled values ​​in the 2nd to the deth one-dimensional data is equal to dn. (1) One-dimensional data is used as category B data;

[0051] If there is no B-type data in the second to the third one-dimensional data, it means that the first one-dimensional data is independent data and there is no correlation between it and other one-dimensional data. Extract the feature value and feature vector corresponding to each multidimensional data to obtain the explicit fusion data corresponding to each multidimensional data.

[0052] If there are B-type data in the 2nd to de-th one-dimensional data, then count the number of B-type data, ub; dn (1) As a DNL, ​​we analyze and determine the correlation between the first one-dimensional data and the first to ub-th B-class data, and construct the correlation function corresponding to the first one-dimensional data.

[0053] Furthermore, the specific steps for constructing the correlation function are as follows:

[0054] Calculate the mean go of the first one-dimensional data point; calculate the mean ga of the first to ubth B-class data points. (1) ~ga (ub) ;

[0055] Let the value in the first one-dimensional data be denoted as xb. (1) ~xb (dnl) ;

[0056] The values ​​in the first to the ubth B-class data are denoted as xbb. (1,1) ~xbb (ub,dnl) ;

[0057] Let the value of the q-th number in the first one-dimensional data be xb. (q) The value of the q-th number in the p-th B-class data is xbb. (p,q) The average value of the p-th B-class data is ga. (p) Define formula B-2-1:

[0058]

[0059] Among them, R (1，p) This represents the linear correlation coefficient between the first one-dimensional data point and the p-th B-class data point;

[0060] Calculate the linear correlation coefficient R of the first one-dimensional data point with respect to the first to the ubth B-class data points. (1，1) ~R (1，ub) ;

[0061] From the first to the ubth B-class data, extract the B-class data whose linear correlation coefficient differs from 0.05 by ε, and use them as Bb-class data; where ε represents the error determination coefficient.

[0062] Count the number of data points of class Bb, ld, and define the linear correlation function corresponding to the first one-dimensional data point.

[0063] Furthermore, the specific steps for defining the linear correlation function are as follows:

[0064] Extract the values ​​xbx from the 1st to the ldth Bb class data. (1,1) ~~xbx (ld,dnl) ;

[0065] Define matrices BX and BY;

[0066] Let β be the coefficient of the first one-dimensional matrix with respect to the first to the ldth Bc class data. (1) ~β (ld) Define matrix BB;

[0067] Calculate β (1) ~β (ld) Value: BB = BY * ((BX) T *BX) -1 *BX T ).

[0068] Furthermore, the step of constructing the correlation function further includes:

[0069] Count ul Bc class data; extract the average value gc(1)~gc of the 1st to ulth Bc class data. (ul) ;

[0070] Extract the linear correlation coefficient Rc corresponding to the 1st to the ulth Bc class data. (1) ~Rc (ul) ;

[0071] Extract the values ​​xbc from the first to the ulth Bc class data. (1,1) ~xbc (ul,dnl) ;

[0072] The value of the q-th number in the t-th Bc-class data is xbc. (t,q) The average value of the t-th Bc class data is gc. (t) The linear correlation coefficient Rc corresponding to the t-th Bc class data. (t) ;

[0073] Define formula B-2-2:

[0074]

[0075] Where, ρ (1，t) This represents the nonlinear correlation coefficient of the first one-dimensional data point with respect to the t-th Bc-class data point;

[0076] fR (1，t) This represents the inverse linear correlation coefficient of the first one-dimensional data point with respect to the t-th Bc-class data point;

[0077] Calculate the nonlinear correlation coefficient ρ of the first to the ulth Bc class data. (1，1) ~ρ (1，ul) ;

[0078] Data in class Bc with a nonlinear correlation coefficient within ε of 0.05 are extracted and used as data in class Bd.

[0079] The number of Bd-class data points ud is counted, with the first one-dimensional data point as the dependent variable and the first to ud-th Bd-class data points as independent variables. A local weighted regression algorithm and a neural network are used to fit a nonlinear function of the first one-dimensional data point with respect to the first to ud-th Bd-class data points.

[0080] Compared with the prior art, the beneficial effects of the present invention are:

[0081] This invention demonstrates superior efficiency and quality in processing mixed data. Through dimensional alignment and feature decomposition techniques, it effectively handles complex situations with inconsistent data dimensions. The specific process includes: identifying mixed data, determining the largest data dimension, aligning data of different dimensions using zero-padding, and then performing feature decomposition to extract key information. This processing method ensures the comparability and fusion quality between data of different dimensions.

[0082] Intelligent data correlation analysis: For one-dimensional data, the system adopts normalization processing, and accurately identifies the correlation between data through linear correlation coefficient calculation and nonlinear function fitting. It automatically distinguishes between linear and nonlinear correlation and constructs corresponding function models for each, which greatly improves the accuracy of the analysis.

[0083] Unstructured data processing capability: This invention demonstrates significant technical advantages in processing unstructured data; for non-numerical data such as images, text, audio, and video, this invention employs neural network technology for intelligent processing; based on user-defined extraction targets, the neural network can automatically filter and fuse relevant data, greatly improving the processing efficiency and accuracy of unstructured data. Attached Figure Description

[0084] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0085] Figure 1 This is a schematic diagram of the method of the present invention;

[0086] Figure 2 This is a schematic diagram of the process of the present invention;

[0087] Figure 3 This is a schematic diagram of the process of the present invention. Detailed Implementation

[0088] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0089] Please see Figure 1 A method for fusion processing of multi-source heterogeneous data sources includes:

[0090] Step S1: Obtain the raw data provided by the data source, and create a storage structure according to the number of data sources to save the raw data; according to the data type (in the raw data), divide the raw data into numerical data and non-numerical data;

[0091] It should be noted that the data source in this invention refers to the individual, unit, or department that uses this invention (a method for statistical analysis of target quantities) to perform mathematical statistics;

[0092] The specific steps of step S1 are as follows:

[0093] Get the number of data sources da, and calculate the depth bh of the complete binary tree:

[0094] Wherein log2 (da) If it is a positive integer, round down.

[0095] The `build_binary_tree` function is used to construct a complete binary tree of depth `bh` and number of nodes `na`, which is then used as the tree (1).

[0096] Create storage space on a cloud server, and then create, deploy, and configure the blockchain through the cloud platform console;

[0097] The original data corresponding to the first to the da data sources are encrypted in a decentralized manner to obtain the encrypted data corresponding to the first to the da data sources.

[0098] Starting from the root node of tree (1), the encrypted data corresponding to the first to the dath data sources are stored in the corresponding nodes of tree (1) in the order from top to bottom and from left to right, to obtain tree (2).

[0099] Starting from the first leaf node at the bottom of tree (2), hash upwards layer by layer to generate a root hash (MerkleRoot);

[0100] In the storage space of each non-leaf node in tree (2), the hash value of the leaf node corresponding to the non-leaf node is additionally stored in the format of "left child hash + right child hash → parent hash"; in the storage space of the two leaf nodes corresponding to the non-leaf node, the hash value of the other node is stored in each other; (for example: the two leaf nodes corresponding to the non-leaf node are leaf node (1) and leaf node (2); in leaf node (1), the hash value of leaf node (2) is stored; in leaf node (2), the hash value of leaf node (1) is stored;)

[0101] Obtain the Merkle tree and use the root hash as the unique fingerprint of the entire Merkle tree;

[0102] Deploy the Merkle tree in the blockchain and return an access interface to the blockchain to all data sources.

[0103] Step S2: (Processing numerical data) Extract the data dimensions corresponding to all numerical data in the data source. Based on whether the data dimensions of each numerical data in the data source are consistent, divide the numerical data into mixed data (inconsistent) and non-mixed data (consistent). Then, based on the data dimensions, divide the non-mixed data into one-dimensional data and multi-dimensional data. Integrate each data in the mixed data and perform feature decomposition. Then, unify the data dimensions of each data in the mixed data to obtain explicit fused data.

[0104] Please see Figure 2 The specific steps of step S2 are as follows:

[0105] Extract all data dimensions corresponding to the numerical data in the data source. Based on whether the data dimensions of each numerical data in the data source are consistent, divide the numerical data into mixed data (inconsistent) and non-mixed data (consistent). Then, based on the data dimensions, divide the non-mixed data into one-dimensional data and multi-dimensional data.

[0106] Handling mixed data:

[0107] Treat the data in the mixed data as category A data;

[0108] Count the number mi of class A data; obtain the dimensions of the 1st, 2nd, and so on, up to the mi-th class A data, and get di. (1) di (2) ~di (mi) Extract di (1) ~di (mi) The maximum value di (max) ;

[0109] Define the fill format for class A data:

[0110] Let the data dimension of the a-th data point in class A be di. (a) Let the first and second data points in the a-th data type A be, up to the di-th data point. (a) The value of the dimension is xa (1) xa (2) ~xa (da(a)) Where, the value of a ranges from 1 to mi;

[0111] Define (1×di) (a) Let matrix XA represent the a-th data point in class A; matrix XA is:

[0112] (1 line di) (a) column, i.e., di (a) (dimension);

[0113] Define (1×di) (max) Let YA be the matrix representing the a-th data point of class A after alignment.

[0114] Compare di (a) with di (max) The size of the matrix determines the format of matrix YA:

[0115] If di (a) =di (max) If the a-th data of class A after alignment is itself, then the format of matrix YA is the same as that of matrix XA.

[0116] If di (a) <di (max) Then the format of matrix YA is:

[0117]

[0118] (1 line di) (max) column, i.e., di (max) Dimension; in xa( di(a) Fill (di) after (max) -di (a) ) zeros);

[0119] Based on the padding format of Class A data (i.e., zero padding), the dimensions of the 1st, 2nd, and up to the mith Class A data are expanded to di. (max) Dimension, to obtain data Aa (1) Aa (2) ~Aa (mi) ;(where data Aa) (1) This represents the first aligned data of class A, data Aa. (2) This represents the second A-class data after alignment; and so on, data Aa (mi) (This represents the mi-th A-class data after alignment);

[0120] di (max) As dim, define a zero matrix of (mi × dim) and put the data Aa (1) ~Aa ( mi ) Filling it into the zero matrix yields matrix Ba. (1) ;

[0121] Calculate matrix Ba (1) The decentralized matrix is ​​obtained as matrix Ba. (2) ;

[0122] Calculate the covariance matrix Ba (3) : Where * denotes matrix multiplication, T denotes matrix transpose; covariance matrix Ba (3) The dimension is (dim×dim);

[0123] matrix Ba(3) The data in the middle is denoted as va (1,1) va (1,2) ~va (1,dim) va (2,1) ~va (dim,dim) ; matrix Ba (3) for:

[0124]

[0125] Let the eigenvalues ​​of class A data be λ, and construct matrix Ba. (4) :

[0126]

[0127] Based on the expansion matrix Ba of Newton's identity (4) We obtain a dim function F(λ) with respect to λ:

[0128] Where i ranges from 0 to dim; ck (i) Let represent the coefficient of the (dim-i)th term of λ;

[0129] ck (i) The expression is:

[0130] Where j takes values ​​from 1 to i; ck (i-j) Let represent the coefficient of the (i-j)th term of λ;

[0131] ti (j) Represents matrix Ba (4) The trace of a power of j; (the trace of a matrix is ​​the sum of the elements on the diagonal of the matrix);

[0132] Based on the function F(λ), calculate the 1-dimensional, 2-dimensional, and even dim-dimensional eigenvalues ​​of class A data to obtain λ. (1) , λ (2) ~λ (dim) ;

[0133] According to λ (1) , λ (2) ~λ (dim) Construct matrix Ia (1) Ia (2) ~Ia (dim) Among them, Ia (1) The formula for calculating Ia is: (1) =Ba (4) -(λ (1) *Ii); where Ii represents the (dim×dim) identity matrix;

[0134] Ia (2) The formula for calculating Ia is:(2) =Ba (4) -(λ (2) *Ii);

[0135] And so on, Ia (dim) The formula for calculating Ia is: (dim) =Ba (4) -(λ (dim) *Ii);

[0136] Calculate matrix Ia using Gaussian elimination. (1) Ia (2) ~Ia (dim) The non-zero vectors in the null space are then normalized to obtain the eigenvectors va of dimension dim (i.e., (1×dim)). (1) va (2) ~va (dim) ;

[0137] Extracting λ (1) , λ (2) ~λ (dim) The maximum value λ in (max) and minimum value λ (min) ;

[0138] Let the a-dimensional eigenvalue of class A data be λ. (a) The value of 'a' ranges from 1 to 'dim'. Calculate λ. (1) , λ (2) ~λ (dim) The combined mean aλ:

[0139]

[0140] Eigenvalues ​​greater than or equal to aλ are considered dominant eigenvalues, and are denoted as dλ. (1) , dλ (2) ~dλ (od) Where od represents the number of dominant eigenvalues; 0 ≤ od < dim;

[0141] Fusion of dominant features: Construct a (dim×od) zero matrix and fill it with dλ. (1) ~dλ (od) The corresponding eigenvectors yield the matrix XW;

[0142] Calculate the fusion matrix corresponding to class A data, as matrix ZA: ZA = Ba (1) *XW;

[0143] Extract the elements from the first row, the second row, up to the mi row of matrix ZA, as explicit fused data for class A data.

[0144] Step S3: Normalize each one-dimensional data point to obtain the initial fused data for each one-dimensional data point; determine whether there is a correlation between multiple one-dimensional data points with the same dimension; if there is, construct a correlation function between different one-dimensional data points; if not, extract the feature values ​​and feature vectors corresponding to each multi-dimensional data point to obtain the explicit fused data corresponding to each multi-dimensional data point; based on the nonlinearity test (manifold learning), determine whether there is a correlation between multiple multi-dimensional data points with the same dimension; if there is, perform secondary fusion on the multi-dimensional data points with correlation; if not, do not process them.

[0145] The specific steps of step S3 are as follows:

[0146] Please see Figure 3 Step S31: Count the number of one-dimensional data points;

[0147] Define formula B-1 for normalizing one-dimensional data;

[0148] Let the value of the 1st, 2nd, up to the dmth number in the one-dimensional data be: oe (1) ,oe (1) ~oe (dm) Where dm represents the number of values ​​in the one-dimensional data;

[0149] Let oe (1) ~oe (dm) The maximum value in is oe (max) The minimum value is oe (min) ;

[0150] Let the normalized value of the 0th number in one-dimensional data be uoe. (o) The range of values ​​for o is 1 to dm; Formula B-1 is defined as follows:

[0151]

[0152] According to formula B-1, the first to the de one-dimensional data are normalized, and the normalized data of the first to the de one-dimensional data are used as the primary fusion data of the first to the de one-dimensional data.

[0153] Step S32: Obtain the number of values ​​corresponding to the 1st, 2nd, and up to the deth one-dimensional data, and get dn. (1) dn (2) ~dn (de) ;

[0154] Determine whether there is a correlation between the first one-dimensional data and other one-dimensional data, and construct the correlation function corresponding to the first one-dimensional data;

[0155] The number of labeled values ​​in the 2nd to the deth one-dimensional data is equal to dn. (1)The one-dimensional data is used as the B-type data (i.e., the alternative one-dimensional data for the first one-dimensional data);

[0156] If there is no B-type data in the second to the third one-dimensional data, it means that the first one-dimensional data is independent data and there is no other one-dimensional data related to it. Skip step S33 and execute step S35 (extract the feature value and feature vector corresponding to each multidimensional data to obtain the explicit fusion data corresponding to each multidimensional data).

[0157] If there are B-type data in the 2nd to de-th one-dimensional data, then count the number of B-type data, ub; dn (1) As a dnl, analyze and determine the correlation between the first one-dimensional data and the first to ubth B-class data, construct the correlation function corresponding to the first one-dimensional data, and execute step S33;

[0158] Step S33: Calculate the mean go of the first one-dimensional data; calculate the mean ga of the first, second, and up to the ubth B-class data. (1) ga (2) ~ga (ub) ;

[0159] Let the value in the first one-dimensional data be denoted as xb. (1) 、xb (2) ~xb (dnl) ;

[0160] The value in the first B-class data is denoted as xbb. (1,1) 、xbb (1,2) ~xbb (1,dnl) ;

[0161] The value in the second B-class data is denoted as xbb. (2,1) 、xbb (2,2) ~xbb (2,dnl) ;

[0162] And so on, the value in the ub-th B-class data is denoted as xbb. (ub,1) 、xbb (ub,2) ~xbb(ub,dnl);

[0163] Let the value of the q-th number in the first one-dimensional data be xb. (q) The value range of q is 1 to dnl.

[0164] The value of the q-th number in the p-th B-class data is xbb. (p,q) The average value of the p-th B-class data is ga. (p) The value of p ranges from 1 to ub.

[0165] Define formula B-2-1:

[0166]

[0167] Among them, R (1，p) This represents the linear correlation coefficient between the first one-dimensional data point and the p-th B-class data point;

[0168] According to formula B-2-1, calculate the linear correlation coefficient R of the first one-dimensional data with respect to the first, second, and up to the ubth B-class data. (1，1) R (1，2) ~R (1，ub) ;

[0169] From the first to the ubth B-class data, extract the B-class data whose linear correlation coefficient differs from 0.05 by ε, and use them as Bb-class data (i.e. linearly correlated data).

[0170] Wherein, ε represents the error determination coefficient; (ε takes the value of 0.01; users or relevant technical personnel can adjust the value of ε according to actual needs);

[0171] Step S331: Count the number of data points ld in class Bb, and define the linear correlation function corresponding to the first one-dimensional data point;

[0172] Extract the value xbx from the first Bb class data. (1,1) xbx (1,2) ~xbx (1,dnl) ;

[0173] The value xbx in the second Bb class data (2,1) xbx (2,2) ~xbx (2,dnl) ;

[0174] And so on, the value xbx in the ld-th Bc class data (ld,1) xbx (ld,2) ~xbx(ld,dnl);

[0175] in,

[0176] Define a zero matrix of size (ld × dnl) and fill it with xbx. (1,1) ~xbx (ld,dnl) This yields matrix BX;

[0177] Define a zero matrix of size (1×dnl) and fill it with xb. (1) ~xb (dnl) , thus obtaining matrix BY;

[0178] Let β be the coefficient of the first one-dimensional matrix with respect to the first, second, and up to the ld-th Bc class data. (1) β (2) ~β(ld) ;

[0179] Define a zero matrix of size (1×ld) and fill it with β. (1) ~β (ld) We obtain matrix BB;

[0180] Calculate β (1) ~β (ld) Value: BB = BY * ((BX) T *BX) -1 *BX T );

[0181] Step S332: Remove Bb data from the first to the ubth B-class data to obtain Bc data; count the number of Bc data ul.

[0182] Extract the average value gc of the 1st, 2nd, up to the ulth Bc class data. (1) , gc (2) ~gc (ul) ;in,

[0183] Extract the linear correlation coefficient Rc corresponding to the 1st, 2nd, up to the ulth Bc class data. (1) 、Rc (2) ~Rc (ul) ;

[0184] in,

[0185] Extract the value xbc from the first Bc class data. (1,1) xbc (1,2) ~xbc (1,dnl) ;

[0186] The value xbc in the second Bc category data. (2,1) xbc (2,2) ~xbc (2,dnl) ;

[0187] And so on, the value xbc in the ul-th Bc-class data. (ul,1) xbc (ul,2) ~xbc(ul,dnl);

[0188] in,

[0189] The value of the q-th number in the t-th Bc-class data is xbc. (t,q) The average value of the t-th Bc class data is gc. (t) The value range of t is 1 to ul.

[0190] The linear correlation coefficient Rc corresponding to the t-th Bc class data is(t) ;

[0191] Define formula B-2-2:

[0192]

[0193] Where, ρ (1，t) This represents the nonlinear correlation coefficient of the first one-dimensional data point with respect to the t-th Bc-class data point;

[0194] fR (1，t) This represents the inverse linear correlation coefficient of the first one-dimensional data point with respect to the t-th Bc-class data point;

[0195] According to formula B-2-2, calculate the nonlinear correlation coefficient ρ of the 1st, 2nd, and up to the ulth Bc class data. (1，1) ρ (1，2) ~ρ (1，ul) ;

[0196] In the first to the ulth Bc class data, the Bc class data with a non-linear correlation coefficient difference of less than 0.05 within ε are extracted and used as Bd class data (i.e. linear correlation class data);

[0197] The number of data points in class Bd, ud, is counted. The first one-dimensional data point is used as the dependent variable, and the first to udth data points in class Bd are used as independent variables. Multinomial regression or locally weighted regression algorithms in numpy.polyfit / statsmodels.api and generalized additive models or neural networks in pygam / sklearn.neural_network are used to fit the nonlinear function relationship between the first one-dimensional data point and the first to udth data points in class Bd.

[0198] It should be noted that when multiple sets of data satisfy a nonlinear relationship, the relationship between the corresponding data points is not a simple straight line or plane, but may involve curves, surfaces or other complex forms; common nonlinear models include exponential functions, logarithmic functions, polynomial functions, trigonometric functions, etc.; therefore, this invention can only determine that there is a nonlinear relationship between certain one-dimensional data, and cannot give a general analytical expression for a certain nonlinear relationship.

[0199] Step S34: Repeat the same steps as constructing the first one-dimensional data correlation function to construct the correlations corresponding to the second to the de one-dimensional data.

[0200] Step S35: Repeat the calculation steps of extracting feature values ​​and feature vectors of mixed data, extract the feature values ​​and feature vectors corresponding to each multidimensional data, and obtain the explicit fusion data corresponding to each multidimensional data;

[0201] Based on the nonlinear test (using manifold learning), it is determined whether there is a correlation between multiple multidimensional data with the same dimension; if there is, the multidimensional data with correlation are fused again; if not, no processing is performed.

[0202] Step S4: (Processing non-numerical data, i.e., unstructured data such as images, text, audio, and video) Obtain the extraction target corresponding to each non-numerical data (i.e., the unstructured data that the user needs to fuse or extract); Through a neural network, filter and fuse the non-numerical data that corresponds to the extraction target to obtain non-numerical fused data.

[0203] The explicit fusion data of mixed-type data and multidimensional data, as well as the correlation function (of one-dimensional data), are aggregated and encrypted, and synchronized to the storage structure (created in step S1).

[0204] The above formulas are all dimensionless calculations. The formulas are derived from software simulations using a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation. For example, there are weighting coefficients and proportional coefficients. The values ​​set are to quantify each parameter to obtain a specific value, which is convenient for subsequent comparison. The values ​​of the weighting coefficients and proportional coefficients are only required to not affect the proportional relationship between the parameters and the quantified values.

[0205] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for fusion processing of multi-source heterogeneous data sources, characterized in that, The method includes: Obtain the raw data provided by the data source, create a storage structure based on the number of data sources, and save the raw data; classify the raw data into numerical data and non-numerical data according to the data type. Extract all data dimensions corresponding to the numerical data in the data source. Based on whether the data dimensions of each numerical data in the data source are consistent, divide the numerical data into mixed data and non-mixed data. Then, based on the data dimensions, divide the non-mixed data into one-dimensional data and multi-dimensional data. Integrate each data in the mixed data and perform feature decomposition. Then, unify the data dimensions of each data in the mixed data to obtain explicit fused data. The specific steps for processing mixed-type data are as follows: Treat the data in the mixed dataset as category A; obtain the dimension of the first category A data, and get di. （1） ~di （mi） And extract the maximum value di （max） ; Based on zero-padding, the dimensions of the first to mi-th A-class data are expanded to di. （max） Dimension, to obtain data Aa （1） ~Aa （mi） ; di （max） As dim, define matrix Ba （1） ; Calculate matrix Ba （1） The decentralized matrix is ​​obtained as matrix Ba. （2） Calculate the covariance matrix Ba （3） ; matrix Ba （3） The data in the middle is denoted as va （1,1） ~va （dim,dim） ; Let the eigenvalues ​​of class A data be λ, and construct matrix Ba. （4） : ； Expanded matrix Ba （4） We obtain the dim function F(λ) with respect to λ: Among them, ck （i） Let represent the coefficient of the (dim-i)th term of λ; Based on the function F(λ), calculate the eigenvalues ​​of class A data from 1 dimension to dim dimension, and obtain λ. （1） ~λ （dim） ; According to λ （1） ~λ （dim） Construct matrix Ia （1） ~Ia （dim） ; Calculate matrix Ia using Gaussian elimination. （1） ~Ia （dim） The non-zero vectors in the null space are then normalized to obtain the dim-dimensional eigenvectors va. （1） ~va （dim） ; Extracting λ （1） ~λ （dim） The maximum value λ in （max） and minimum value λ （min） ; Let the a-dimensional eigenvalue of class A data be λ. （a） The value of 'a' ranges from 1 to 'dim'. Calculate λ. （1） ~λ （dim） The combined mean aλ: ； Eigenvalues ​​greater than or equal to aλ are considered dominant eigenvalues, and are denoted as dλ. （1） ~dλ （od） Where od represents the number of dominant eigenvalues; Fusion of dominant features: Construct a dim×od zero matrix and fill it with dλ. （1） ~dλ （od） The corresponding eigenvectors yield the matrix XW; Calculate the fusion matrix corresponding to class A data, as matrix ZA: ; Extract the elements from row 1 to row mi in matrix ZA as explicit fused data for class A data; Normalize each one-dimensional data point to obtain the initial fused data for each one-dimensional data point; determine whether there is a correlation between multiple one-dimensional data points with the same dimension; if so, construct a correlation function between different one-dimensional data points; if not, extract the feature values ​​and feature vectors corresponding to each multi-dimensional data point to obtain the explicit fused data for each multi-dimensional data point; based on a nonlinearity test, determine whether there is a correlation between multiple multi-dimensional data points with the same dimension; if so, perform secondary fusion on the multi-dimensional data points with correlation; if not, do not process them. Obtain the extraction target corresponding to each non-numerical data; through a neural network, filter and fuse the non-numerical data that corresponds to the extraction target to obtain non-numerical fused data; summarize and encrypt the explicit fused data of mixed data and multidimensional data, as well as the correlation function, and synchronize them to the storage structure.

2. The method for fusion processing of multi-source heterogeneous data sources according to claim 1, characterized in that, The specific steps for creating a storage structure are as follows: Get the number of data sources da, calculate the depth bh of the complete binary tree: construct a complete binary tree with depth bh and number of nodes na, as tree 1; Create storage space on a cloud server, and then create, deploy, and configure the blockchain through the cloud platform console; The original data corresponding to the first to the dath data sources are decentralized and encrypted to obtain the encrypted data corresponding to the first to the dath data sources; the encrypted data corresponding to the first to the dath data sources is stored in the node corresponding to tree 1 to obtain tree 2; Generate a root hash. In the storage space of each non-leaf node in tree 2, additionally store the hash value of the leaf node corresponding to the non-leaf node. In the storage space of the two leaf nodes corresponding to the non-leaf node, store the hash value of the other node to obtain the Merkle tree, and use the root hash as the unique fingerprint of the entire Merkle tree. Deploy the Merkle tree in the blockchain and return an access interface to the blockchain to all data sources.

3. The method for fusion processing of multi-source heterogeneous data sources according to claim 1, characterized in that, The specific steps for normalizing one-dimensional data are as follows: Let the values ​​of the 1st to the dmth elements in the one-dimensional data be: oe （1） ~oe （dm） Let oe （1） ~oe （dm） The maximum value in is oe （max） The minimum value is oe （min） ; Let the normalized value of the 0th number in one-dimensional data be uoe. （o） Define the formula: ； The first to the de one-dimensional data are normalized, and the normalized data of the first to the de one-dimensional data are used as the primary fusion data of the first to the de one-dimensional data.

4. The method for fusion processing of multi-source heterogeneous data sources according to claim 1, characterized in that, The specific steps to determine whether there is a correlation in one-dimensional data are as follows: Get the number of values ​​dn corresponding to the first to the de one-dimensional data. （1） ~dn （de） ; Determine whether there is a correlation between the first one-dimensional data and other one-dimensional data, and construct the correlation function corresponding to the first one-dimensional data; The number of labeled values ​​in the 2nd to the deth one-dimensional data is equal to dn. （1） One-dimensional data is used as category B data; If there is no B-class data in the second to the third one-dimensional data, it means that the first one-dimensional data is independent data. Extract the feature value and feature vector corresponding to each multidimensional data to obtain the explicit fusion data corresponding to each multidimensional data. If there are B-type data in the 2nd to de-th one-dimensional data, then count the number of B-type data, ub; dn （1） As a DNL, ​​we analyze and determine the correlation between the first one-dimensional data and the first to ub-th B-class data, and construct the correlation function corresponding to the first one-dimensional data.

5. The method for fusion processing of multi-source heterogeneous data sources according to claim 4, characterized in that, The specific steps for constructing the correlation function are as follows: Calculate the mean go of the first one-dimensional data point; calculate the mean ga of the first to ubth B-class data points. （1） ~ga （ub） ; Let the value in the first one-dimensional data be denoted as xb. （1） ~xb （dnl） ; The values ​​in the first to the ubth B-class data are denoted as xbb. （1,1） ~xbb （ub,dnl） ; Let the value of the q-th number in the first one-dimensional data be xb. （q） The value of the q-th number in the p-th B-class data is xbb. （p,q） The average value of the p-th B-class data is ga. （p） ; Define the formula: ； R （1，p） This represents the linear correlation coefficient between the first one-dimensional data point and the p-th B-class data point; Calculate the linear correlation coefficient R of the first one-dimensional data point with respect to the first to the ubth B-class data points. （1，1） ~R （1，ub） ; From the first to the ubth B-class data, extract the B-class data whose linear correlation coefficient differs from 0.05 by ε, and use them as Bb-class data; where ε represents the error determination coefficient. Count the number of data points of class Bb, ld, and define the linear correlation function corresponding to the first one-dimensional data point.

6. The method for fusion processing of multi-source heterogeneous data sources according to claim 5, characterized in that, The specific steps for defining a linear correlation function are as follows: Extract the values ​​xbx from the 1st to the ldth Bb class data. （1,1） ~~xbx （ld,dnl） ; Define matrices BX and BY; Let β be the coefficient of the first one-dimensional matrix with respect to the first to the ldth Bc class data. （1） ~β （ld） Define matrix BB; Calculate β （1） ~β （ld） Value: .

7. The method for fusion processing of multi-source heterogeneous data sources according to claim 5, characterized in that, The steps for constructing the correlation function also include: Count ul Bc class data; extract the average value gc of the 1st to ulth Bc class data. （1） ~gc （ul） ; Extract the linear correlation coefficient Rc corresponding to the 1st to the ulth Bc class data. （1） ~Rc （ul） ; Extract the values ​​xbc from the first to the ulth Bc class data. （1,1） ~xbc （ul,dnl） ; The value of the q-th number in the t-th Bc-class data is xbc. （t,q） The average value of the t-th Bc class data is gc. （t） The linear correlation coefficient Rc corresponding to the t-th Bc class data. （t） ; Define the formula: ； Where, ρ （1，t） This represents the nonlinear correlation coefficient of the first one-dimensional data point with respect to the t-th Bc-class data point; fR （1，t） This represents the inverse linear correlation coefficient of the first one-dimensional data point with respect to the t-th Bc-class data point; Calculate the nonlinear correlation coefficient ρ of the first to the ulth Bc class data. （1，1） ~ρ （1，ul） ; Data in class Bc with a nonlinear correlation coefficient within ε of 0.05 are extracted and used as data in class Bd. The number of Bd-class data points ud is counted, with the first one-dimensional data point as the dependent variable and the first to ud-th Bd-class data points as independent variables. A local weighted regression algorithm and a neural network are used to fit a nonlinear function of the first one-dimensional data point with respect to the first to ud-th Bd-class data points.

Images