A Multi-Source Data Integration Method and System Based on Data Mining

Abstract

This invention discloses a multi-source data integration method and system based on data mining, relating to the field of data management technology. It involves extracting data features from data uploaded from front-end data sources and dividing the uploaded data into different data subsets based on these features. After optimizing the data within each subset, a Bayesian network model is used to determine the data relationships between data in different subsets. These relationships are then used to construct an integrated database for the data with associated relationships. Different levels of index sequences are used to bind and associate the associated data, thereby achieving the classification and organization of data uploaded from different front-end data sources, making data management more efficient.

Description

Technical Field

[0001] This invention relates to the field of data management technology, specifically a multi-source data integration method and system based on data mining. Background Technology

[0002] With the advent of the digital age, data has become a core resource across all industries. Enterprises, research institutions, and government departments accumulate massive amounts of data in their daily operations. These data come from a wide range of sources, including sensors, social media, enterprise databases, and public data platforms. The heterogeneity, dispersion, and complexity of multi-source data pose significant challenges to data integration and utilization. Traditional data integration methods often rely on manual rules or fixed patterns, making it difficult to adapt to dynamically changing data environments and fully explore the potential relationships between data. At the same time, the rapid development of data mining technology has provided new possibilities for multi-source data integration. By automating the analysis of patterns, trends, and relationships in data, data mining can help achieve more efficient and intelligent data integration.

[0003] How to classify multi-source data according to corresponding features, and integrate and manage multi-source data based on the correlation between the data after feature classification, so as to achieve more efficient multi-source data integration, is the problem we need to solve. To this end, we now provide a multi-source data integration method and system based on data mining. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for multi-source data integration based on data mining.

[0005] The objective of this invention can be achieved through the following technical solution: a multi-source data integration method based on data mining, comprising the following steps:

[0006] Step S1: Extract data feature items from the data uploaded by the front-end data source, classify the extracted data feature items, construct corresponding data subsets based on the classification results, and import the uploaded data into each data subset;

[0007] Step S2: Optimize the data in each data subset, and import the optimized data subsets into the Bayesian network model to output the data relationship between the data in different datasets;

[0008] Step S3: Based on the data relationships between the data in each of the output data subsets, perform integrated management of the data uploaded from each front-end data source.

[0009] Furthermore, the process of extracting data feature items from the data uploaded by the front-end data source includes:

[0010] Generate a corresponding data guidance sequence based on the front-end data source of the uploaded data and the data upload time;

[0011] The generated data guide sequence is associated with the data uploaded by the front-end data source to obtain the original data;

[0012] Construct a corresponding data feature set based on the generated data guide sequence;

[0013] Iterate through the obtained raw data and input the obtained raw data into the completed training decision tree model;

[0014] The decision tree model is used to extract features from the input raw data to obtain the corresponding data feature items.

[0015] Furthermore, the training process of the decision tree model is as follows:

[0016] Collect sample data and summarize the collected sample data to obtain the corresponding sample dataset;

[0017] Classify the sample data within the sample dataset to obtain the corresponding sample types;

[0018] Obtain the proportion of each sample type in the sample data within the sample dataset, and obtain the information entropy of the sample dataset;

[0019] Each obtained sample type is used as the corresponding sample feature, and the number of each sample feature in the sample dataset is obtained, and several sample feature subsets corresponding to the sample features are obtained.

[0020] And construct the corresponding root node according to the obtained sample type, and obtain the information gain of each sample feature corresponding to the sample type;

[0021] The sample feature corresponding to the maximum value in the obtained information gain is recorded as the root node of the decision tree;

[0022] Set corresponding internal nodes for each root node, and set corresponding node partitioning conditions for each internal node;

[0023] Obtain the sample features in the sample dataset that meet the corresponding node partitioning conditions, and obtain the information gain of the corresponding sample features, and then generate the child nodes corresponding to the sample features;

[0024] Then set the corresponding internal nodes for the child node, and set the corresponding node partitioning conditions for each internal node, and so on;

[0025] Set a corresponding quantity threshold for each sample feature;

[0026] When the number of sample features that meet the corresponding node splitting conditions is less than the set threshold, the splitting stops, and the corresponding child node is marked as a leaf node, thus completing the construction of the decision tree model.

[0027] The collected sample data is divided into training, testing, and validation sets to train the decision tree model.

[0028] Furthermore, the process of classifying the extracted data features, constructing corresponding data subsets based on the classification results, and importing the uploaded data into each data subset includes:

[0029] The obtained data features are classified, and the classification results include attribute features, parameter features, and association features.

[0030] Within the data feature set, establish attribute subsets, parameter subsets, and association subsets respectively;

[0031] Import the obtained attribute features into the attribute subset, import the obtained parameter features into the parameter subset, and import the obtained association features into the association subset.

[0032] Furthermore, the process of optimizing the data within each subset includes:

[0033] The data range in the original data for each data feature item within the obtained data feature set is marked;

[0034] Set up several redundant comparison fields, and set a corresponding sequence code for each redundant comparison field;

[0035] Match the unmarked data range within the original data with the redundancy comparison field, and mark the data content that is the same as the redundancy comparison field as redundant data;

[0036] The original data is converted into a data stream, and the data stream segments corresponding to the redundant data are replaced with the corresponding sequence codes;

[0037] After replacing all redundant data stream segments in the original data, the optimization of the original data is completed.

[0038] Furthermore, the process of importing the optimized data subsets into the Bayesian network model and outputting the data relationships between data in different datasets includes:

[0039] Construct a Bayesian network model and train the constructed Bayesian network model;

[0040] Import the optimized raw data from the attribute subset, parameter subset, and association subset into the trained Bayesian network model.

[0041] The probability that there is a correlation between attribute feature items within the output attribute subset, parameter feature items within the parameter subset, and correlation feature items within the correlation subset;

[0042] Set a probability estimation threshold and compare the probability of correlation between the output feature items with the set probability estimation threshold.

[0043] If the probability of a correlation between different feature items is higher than the set probability estimation threshold, it indicates that there is a data correlation between the corresponding feature items; otherwise, there is no data correlation.

[0044] Traverse each attribute feature item within the attribute subset. If there is a corresponding feature item in the parameter subset and the associated subset that has a data relationship with the attribute feature item, then record the attribute feature item as the base feature item.

[0045] Construct a benchmark subset corresponding to the benchmark feature item, and import other feature items that have data association with the benchmark feature item into the benchmark subset;

[0046] Based on the probability that there is a correlation between a feature term within the benchmark subset and the benchmark feature term, obtain the correlation coefficient Gx between the benchmark feature term and the benchmark subset;

[0047] Set the threshold ranges for association relationships: [G1], [G2], [G3];

[0048] When the correlation coefficient Gx∈[G1] between the obtained benchmark feature and each benchmark subset, it indicates that the benchmark feature is correlated with the corresponding benchmark subset;

[0049] When the correlation coefficient Gx∈[G2] between the obtained benchmark feature and each benchmark subset indicates that the benchmark feature is correlated with the corresponding benchmark subset, a correlation ratio coefficient is generated.

[0050] When the correlation coefficient Gx∈[G3] between the obtained benchmark feature and each benchmark subset indicates that there is no correlation between the benchmark feature and the corresponding benchmark subset, no operation is performed.

[0051] Furthermore, when there is a correlation between the benchmark feature and the benchmark subset, a correlation index sequence between the benchmark feature and all feature items within the benchmark subset is generated.

[0052] When there is a partial correlation between the benchmark feature and the benchmark subset, the feature items in the benchmark subset are sorted from high to low according to the probability of being correlated with the benchmark feature. Then, the corresponding features are selected from high to low according to the correlation ratio coefficient, and the correlation index sequence between the benchmark feature and the selected features in the benchmark subset is generated.

[0053] The data association between the benchmark feature and each feature in the benchmark subset is determined by associating the index sequence.

[0054] Furthermore, based on the data relationships between the data within each output subset, the process of integrating and managing the data uploaded from each front-end data source includes:

[0055] Read the associated index sequence corresponding to each benchmark feature item, and build an integrated database of the corresponding type according to the type of feature corresponding to the associated index sequence;

[0056] Import the original data corresponding to each feature and the corresponding associated index sequence into the integrated database of the corresponding type;

[0057] The associated index sequence is marked as the first index sequence;

[0058] Generate a corresponding second index sequence for the integrated database, and associate the second index sequence with each first index sequence in the corresponding integrated database;

[0059] Each second index sequence is associated with its corresponding baseline feature item, thereby completing the corresponding data integration management.

[0060] Furthermore, a multi-source data integration system applied to a multi-source data integration method based on data mining includes a data management center, which is communicatively connected to a front-end data transmission module, a multi-source data processing module, an association model output module, and a data integration management module.

[0061] The front-end data transmission module is used to upload data from the front-end data source;

[0062] The multi-source data processing module is used to extract data feature items from the data uploaded by the front-end data source, classify the extracted data feature items, construct corresponding data subsets based on the classification results, and import the uploaded data into each data subset.

[0063] The association model output module is used to optimize the data in each data subset, and import the optimized data subsets into the Bayesian network model to output the data association relationship between data in different datasets.

[0064] The data integration management module is used to integrate and manage the data uploaded from each front-end data source based on the data relationships between the data in each of the output data subsets.

[0065] Compared with the prior art, the beneficial effects of the present invention are:

[0066] By extracting data features from the data uploaded from the front-end data source and dividing the uploaded data into different data subsets based on the extracted features, and optimizing the data within each data subset, the data relationships between the data in different data subsets are determined through a Bayesian network model. Then, based on the data relationships, an integrated database is built for the data with data relationships. By using index sequences at different levels to bind and associate the data with data relationships, the data uploaded from different front-end data sources can be classified and organized, making data management more efficient. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0068] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0069] like Figure 1 As shown, a multi-source data integration method based on data mining includes the following steps:

[0070] Step S1: Extract data feature items from the data uploaded by the front-end data source, classify the extracted data feature items, construct corresponding data subsets based on the classification results, and import the uploaded data into each data subset;

[0071] Step S2: Optimize the data in each data subset, and import the optimized data subsets into the Bayesian network model to output the data relationship between the data in different datasets;

[0072] Step S3: Based on the data relationships between the data in each of the output data subsets, perform integrated management of the data uploaded from each front-end data source.

[0073] It should be further explained that, in the specific implementation process, the process of extracting data feature items from the data uploaded by the front-end data source includes:

[0074] Generate a corresponding data guidance sequence based on the front-end data source of the uploaded data and the data upload time;

[0075] The generated data guide sequence is associated with the data uploaded by the front-end data source to obtain the original data;

[0076] Construct a corresponding data feature set based on the generated data guide sequence;

[0077] Iterate through the obtained raw data and input the obtained raw data into the completed training decision tree model;

[0078] The decision tree model is used to extract features from the input raw data to obtain the corresponding data feature items.

[0079] It should be further explained that, in specific implementation, the training process of the decision tree model is as follows:

[0080] Collect sample data and summarize the collected sample data to obtain the corresponding sample dataset, which is denoted as D;

[0081] Classify the sample data within the sample dataset to obtain the corresponding sample types, and label each sample type as i, i = 1, 2, ..., k;

[0082] The proportion of each sample type in the sample dataset is denoted as p. i ;

[0083] Then obtain the information entropy of the sample dataset, denoted as H(D), where:

[0084]

[0085] Each obtained sample type is taken as the corresponding sample feature, denoted as A, and the number of each sample feature in the sample dataset is obtained, denoted as V;

[0086] The obtained sample features are divided into several subsets of sample features corresponding to sample feature A. Each subset of sample features is labeled and denoted as v, where v = 1, 2, ..., V;

[0087] The feature subset of the sample labeled v is then denoted as D. v ;

[0088] Based on the obtained sample types, a corresponding root node is constructed, and the information gain of each sample feature corresponding to that sample type is obtained. The information gain corresponding to sample feature A is denoted as Gain(D,A), where:

[0089]

[0090] The sample features corresponding to the maximum value in the obtained information gain are labeled, and the labeled sample features are recorded as the root node of the decision tree;

[0091] Set corresponding internal nodes for each root node, and set corresponding node partitioning conditions for each internal node;

[0092] Based on the node partitioning conditions of each internal node, obtain the sample features in the sample dataset that meet the corresponding node partitioning conditions, obtain the information gain of the corresponding sample features, and then generate the child nodes corresponding to the sample features.

[0093] Then set the corresponding internal nodes for the child node, and set the corresponding node partitioning conditions for each internal node, and so on;

[0094] Set a corresponding quantity threshold for each sample feature;

[0095] When the number of sample features that meet the corresponding node splitting conditions is less than the set threshold, the splitting stops, and the corresponding child node is marked as a leaf node, thus completing the construction of the decision tree model.

[0096] The collected sample data is divided into training set, test set and validation set. The training set is input into the constructed decision tree model to train the decision tree model. After the decision tree model is trained, the data recognition accuracy of the decision tree model is tested through the test set and validation set. When the data recognition accuracy reaches the expected level, the construction and training of the decision tree model is completed.

[0097] It should be further explained that, in the specific implementation process, the process of classifying the extracted data feature items, constructing corresponding data subsets based on the classification results, and importing the uploaded data into each data subset includes:

[0098] The obtained data features are classified, and the classification results include attribute features, parameter features, and association features.

[0099] It should be further explained that, in the specific implementation process, the number of attribute feature items and parameter feature items is the same, and each attribute feature item corresponds to one parameter feature item; the association feature item is used to associate attribute feature items that have a relationship, that is, when multiple attribute feature items are in the same factor, an association feature item corresponding to these attribute feature items is generated. The factor can be a table, document, text, etc. in the specific implementation process.

[0100] Within the data feature set, establish attribute subsets, parameter subsets, and association subsets respectively;

[0101] Import the obtained attribute features into the attribute subset, import the obtained parameter features into the parameter subset, and import the obtained association features into the association subset.

[0102] It should be further explained that, in the specific implementation process, the data optimization process within each data subset includes:

[0103] The data range in the original data for each data feature item within the obtained data feature set is marked;

[0104] Set up several redundant comparison fields, and set a corresponding sequence code for each redundant comparison field;

[0105] Match the unmarked data range within the original data with the redundancy comparison field, and mark the data content that is the same as the redundancy comparison field as redundant data;

[0106] The original data is converted into a data stream, and the data stream segments corresponding to the redundant data are replaced with the corresponding sequence codes;

[0107] After replacing all redundant data stream segments in the original data, the optimization of the original data is completed.

[0108] It should be further explained that, in the specific implementation process, the process of importing the various data subsets that have undergone data optimization into the Bayesian network model and outputting the data relationships between data in different datasets includes:

[0109] Construct a Bayesian network model and train the constructed Bayesian network model;

[0110] Import the optimized raw data from the attribute subset, parameter subset, and association subset into the trained Bayesian network model.

[0111] The probability that there is a correlation between attribute feature items within the output attribute subset, parameter feature items within the parameter subset, and correlation feature items within the correlation subset;

[0112] Set a probability estimation threshold and compare the probability of correlation between the output feature items with the set probability estimation threshold.

[0113] If the probability of a correlation between different feature items is higher than the set probability estimation threshold, it indicates that there is a data correlation between the corresponding feature items; otherwise, there is no data correlation.

[0114] Traverse each attribute feature item within the attribute subset. If there is a corresponding feature item in the parameter subset and the associated subset that has a data relationship with the attribute feature item, then record the attribute feature item as the base feature item.

[0115] Construct a benchmark subset corresponding to the benchmark feature item, and import other feature items that have data association with the benchmark feature item into the benchmark subset;

[0116] The feature terms within the benchmark subset are labeled and denoted as j, j = 1, 2, ..., m;

[0117] The probability that there is a correlation between the feature term labeled j and the benchmark feature term is denoted as P. j ;

[0118] Obtain the correlation coefficient between the benchmark feature term and the benchmark subset, denoted as Gx, where:

[0119]

[0120] Where α is a constant, and 1 > α > 0, P max P is the maximum probability among the feature terms that are associated with the baseline feature term. min It is the minimum value among the probabilities that a feature term is associated with a baseline feature term;

[0121] Set the threshold ranges for association relationships: [G1], [G2], [G3];

[0122] When the correlation coefficient Gx∈[G1] between the obtained benchmark feature and each benchmark subset, it indicates that the benchmark feature is correlated with the corresponding benchmark subset;

[0123] When the correlation coefficient Gx∈[G2] between the obtained benchmark feature and each benchmark subset indicates that the benchmark feature is correlated with the corresponding benchmark subset, a correlation ratio coefficient K1 is generated, where K1 is a linear coefficient and is related to the position of Gx in the range of [G2].

[0124] When the correlation coefficient Gx∈[G3] between the obtained benchmark feature and each benchmark subset, it means that there is no correlation between the benchmark feature and the corresponding benchmark subset, and no operation is performed.

[0125] When there is a correlation between the benchmark feature and the benchmark subset, a correlation index sequence between the benchmark feature and all feature items within the benchmark subset is generated.

[0126] When there is a partial correlation between the benchmark feature and the benchmark subset, the feature items in the benchmark subset are sorted from high to low according to the probability of being correlated with the benchmark feature. Then, the corresponding features are selected from high to low according to the correlation ratio coefficient, and the correlation index sequence between the benchmark feature and the selected features in the benchmark subset is generated.

[0127] The data association between the benchmark feature and each feature in the benchmark subset is determined by associating the index sequence.

[0128] It should be further explained that, in the specific implementation process, the process of integrating and managing the data uploaded from each front-end data source based on the data relationships between the data within each of the output data subsets includes:

[0129] Read the associated index sequence corresponding to each benchmark feature item, and build an integrated database of the corresponding type according to the type of feature corresponding to the associated index sequence;

[0130] Import the original data corresponding to each feature and the corresponding associated index sequence into the integrated database of the corresponding type;

[0131] The associated index sequence is marked as the first index sequence;

[0132] Generate a corresponding second index sequence for the integrated database, and associate the second index sequence with each first index sequence in the corresponding integrated database;

[0133] Each second index sequence is associated with its corresponding baseline feature item, thereby completing the corresponding data integration management.

[0134] In another embodiment of the present invention, a multi-source data integration system based on data mining is also disclosed, including a data management center, wherein the data management center is communicatively connected to a front-end data transmission module, a multi-source data processing module, an association model output module, and a data integration management module;

[0135] The front-end data transmission module is used to upload data from the front-end data source;

[0136] The multi-source data processing module is used to extract data feature items from the data uploaded by the front-end data source, classify the extracted data feature items, construct corresponding data subsets based on the classification results, and import the uploaded data into each data subset.

[0137] The association model output module is used to optimize the data in each data subset, and import the optimized data subsets into the Bayesian network model to output the data association relationship between data in different datasets.

[0138] The data integration management module is used to integrate and manage the data uploaded from each front-end data source based on the data relationships between the data in each of the output data subsets.

[0139] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any modifications or equivalent substitutions made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A multi-source data integration method based on data mining, characterized in that, Includes the following steps: Step S1: Extract data feature items from the data uploaded by the front-end data source, classify the extracted data feature items, construct corresponding data subsets based on the classification results, and import the uploaded data into each data subset; Step S2: Optimize the data in each data subset, and import the optimized data subsets into the Bayesian network model to output the data relationship between the data in different datasets; Step S3: Based on the data relationships between the data in each of the output data subsets, perform integrated management of the data uploaded from each front-end data source; The process of classifying the extracted data features, constructing corresponding data subsets based on the classification results, and importing the uploaded data into each data subset includes: The obtained data features are classified, and the classification results include attribute features, parameter features, and association features. Within the data feature set, establish attribute subsets, parameter subsets, and association subsets respectively; Import the obtained attribute features into the attribute subset, import the obtained parameter features into the parameter subset, and import the obtained association features into the association subset; Construct a Bayesian network model and train the constructed Bayesian network model; Import the optimized raw data from the attribute subset, parameter subset, and association subset into the trained Bayesian network model. The probability that there is a correlation between attribute feature items within the output attribute subset, parameter feature items within the parameter subset, and correlation feature items within the correlation subset; Set a probability estimation threshold and compare the probability of correlation between the output feature items with the set probability estimation threshold. If the probability of a correlation between different feature items is higher than the set probability estimation threshold, it indicates that there is a data correlation between the corresponding feature items; otherwise, there is no data correlation. Traverse each attribute feature item within the attribute subset. If there is a corresponding feature item in the parameter subset and the associated subset that has a data relationship with the attribute feature item, then record the attribute feature item as the base feature item. Construct a benchmark subset corresponding to the benchmark feature item, and import other feature items that have data association with the benchmark feature item into the benchmark subset.

2. The multi-source data integration method based on data mining according to claim 1, characterized in that, The process of extracting data features from the data uploaded by the front-end data source includes: Generate a corresponding data guidance sequence based on the front-end data source of the uploaded data and the data upload time; The generated data guide sequence is associated with the data uploaded by the front-end data source to obtain the original data; Construct a corresponding data feature set based on the generated data guide sequence; Iterate through the obtained raw data and input the obtained raw data into the completed training decision tree model; The decision tree model is used to extract features from the input raw data to obtain the corresponding data feature items.

3. The multi-source data integration method based on data mining according to claim 2, characterized in that, The training process of the decision tree model is as follows: Collect sample data and summarize the collected sample data to obtain the corresponding sample dataset; Classify the sample data within the sample dataset to obtain the corresponding sample types; Obtain the proportion of each sample type in the sample data within the sample dataset, and obtain the information entropy of the sample dataset; Each obtained sample type is used as the corresponding sample feature, and the number of each sample feature in the sample dataset is obtained, and several sample feature subsets corresponding to the sample features are obtained. And construct the corresponding root node according to the obtained sample type, and obtain the information gain of each sample feature corresponding to the sample type; The sample feature corresponding to the maximum value in the obtained information gain is recorded as the root node of the decision tree; Set corresponding internal nodes for each root node, and set corresponding node partitioning conditions for each internal node; Obtain the sample features in the sample dataset that meet the corresponding node partitioning conditions, and obtain the information gain of the corresponding sample features, and then generate the child nodes corresponding to the sample features; Then set the corresponding internal nodes for the child node, and set the corresponding node partitioning conditions for each internal node, and so on; Set a corresponding quantity threshold for each sample feature; When the number of sample features that meet the corresponding node splitting conditions is less than the set threshold, the splitting stops, and the corresponding child node is marked as a leaf node, thus completing the construction of the decision tree model. The collected sample data is divided into training, testing, and validation sets to train the decision tree model.

4. The multi-source data integration method based on data mining according to claim 3, characterized in that, The process of optimizing data within each subset of data includes: The data range in the original data for each data feature item within the obtained data feature set is marked; Set up several redundant comparison fields, and set a corresponding sequence code for each redundant comparison field; Match the unmarked data range within the original data with the redundancy comparison field, and mark the data content that is the same as the redundancy comparison field as redundant data; The original data is converted into a data stream, and the data stream segments corresponding to the redundant data are replaced with the corresponding sequence codes; After replacing all redundant data stream segments in the original data, the optimization of the original data is completed.

5. The multi-source data integration method based on data mining according to claim 4, characterized in that, The process of importing optimized subsets of data into a Bayesian network model and outputting the data relationships between different datasets includes: Based on the probability that there is a correlation between a feature term within the benchmark subset and the benchmark feature term, obtain the correlation coefficient Gx between the benchmark feature term and the benchmark subset; Set the threshold ranges for association relationships: [G1], [G2], [G3]; When the correlation coefficient Gx∈[G1] between the obtained benchmark feature and each benchmark subset, it indicates that the benchmark feature is correlated with the corresponding benchmark subset; When the correlation coefficient Gx∈[G2] between the obtained benchmark feature and each benchmark subset indicates that the benchmark feature is correlated with the corresponding benchmark subset, a correlation ratio coefficient is generated. When the correlation coefficient Gx∈[G3] between the obtained benchmark feature and each benchmark subset indicates that there is no correlation between the benchmark feature and the corresponding benchmark subset, no operation is performed.

6. The multi-source data integration method based on data mining according to claim 5, characterized in that, When there is a correlation between the benchmark feature and the benchmark subset, a correlation index sequence between the benchmark feature and all feature items within the benchmark subset is generated. When there is a partial correlation between the benchmark feature and the benchmark subset, the feature items in the benchmark subset are sorted from high to low according to the probability of being correlated with the benchmark feature. Then, the corresponding features are selected from high to low according to the correlation ratio coefficient, and the correlation index sequence between the benchmark feature and the selected features in the benchmark subset is generated. The data association between the benchmark feature and each feature in the benchmark subset is determined by associating the index sequence.

7. The multi-source data integration method based on data mining according to claim 6, characterized in that, The process of integrating and managing data uploaded from various front-end data sources, based on the data relationships between the data within each output subset, includes: Read the associated index sequence corresponding to each benchmark feature item, and build an integrated database of the corresponding type according to the type of feature corresponding to the associated index sequence; Import the original data corresponding to each feature and the corresponding associated index sequence into the integrated database of the corresponding type; The associated index sequence is marked as the first index sequence, a corresponding second index sequence is generated for the integrated database, and the second index sequence is associated with each first index sequence in the corresponding integrated database; each second index sequence is associated with the corresponding benchmark feature item, thereby completing the corresponding data integration management.

8. A multi-source data integration system applied to the multi-source data integration method based on data mining as described in any one of claims 1 to 7, comprising a data management center, characterized in that, The data management center is communicatively connected to a front-end data transmission module, a multi-source data processing module, an association model output module, and a data integration management module. The front-end data transmission module is used to upload data from the front-end data source; The multi-source data processing module is used to extract data feature items from the data uploaded by the front-end data source, classify the extracted data feature items, construct corresponding data subsets based on the classification results, and import the uploaded data into each data subset. The association model output module is used to optimize the data in each data subset, and import the optimized data subsets into the Bayesian network model to output the data association relationship between data in different datasets. The data integration management module is used to integrate and manage the data uploaded from each front-end data source based on the data relationships between the data in each of the output data subsets.

Images