A data quality assessment method based on random forest and fuzzy hierarchical analysis

Abstract

The present invention discloses a data quality assessment method based on random forest and fuzzy hierarchical analysis. The basic idea of ​​the method is: first, the historical data of the data source to be evaluated and the frequency of use of the data are obtained, the data quality assessment standard is defined, and the scores of its various quantitative indicators are calculated. Then, the correlation between the various data indicators and the frequency of use of the data is analyzed using random forest, and the relative importance of the various data indicators on the data quality is obtained. On this basis, a fuzzy complementary judgment matrix is ​​constructed, and then the influence weights of the various data indicators on the data quality are calculated. Finally, for new data, the comprehensive score of the data quality can be calculated according to its score in each indicator. By combining the two methods of random forest and fuzzy hierarchical analysis, the present invention can further improve the accuracy and efficiency of data quality assessment while ensuring objectivity.

Description

Technical Field

[0001] The invention proposes a data quality assessment method based on random forest and fuzzy hierarchical analysis, which belongs to the technical field of data governance. Background Art

[0002] In the context of today's digital transformation, smart parks and enterprise management are facing unprecedented challenges and opportunities. With the rapid development of technologies such as the Internet of Things, cloud computing, and big data, smart parks, as an important part of modern enterprise management, are becoming an important foundation for efficient enterprise operations, resource optimization, and intelligent decision-making. However, with the surge in data volume and the diversification of data types, a core issue facing enterprises is the management and governance of data quality.

[0003] Smart parks process a large amount of structured and unstructured data every day, which involves equipment, personnel, processes and other aspects within the park. The data sources are diverse and real-time, and these data may be lost, erroneous or delayed during transmission, storage and processing. Data quality not only affects daily operations, but also has a profound impact on strategic decision-making, compliance and customer experience. Therefore, how to accurately and efficiently evaluate the quality of data has become the key to enterprise development.

[0004] Traditional data quality assessment methods such as the analytic hierarchy process (AHP) usually rely on expert evaluation. Although they can discover problems in the data to a certain extent, they have a low degree of automation and are somewhat subjective, making it difficult to cope with large-scale and complex data quality assessment needs. Summary of the invention

[0005] In order to solve the problem that low-quality data will directly affect the accuracy of analysis results and the scientific nature of decision-making in big data analysis and mining, the present invention proposes a data quality assessment method based on random forest and fuzzy hierarchical analysis. This method obtains historical data and data usage frequency in advance, uses random forest to analyze the correlation between various data indicators and data usage frequency, and obtains the relative importance of various data indicators on data quality. Then, the fuzzy hierarchical analysis method is used to construct a fuzzy complementary judgment matrix, calculate the influence weight of various data indicators on data quality, and finally calculate the comprehensive score of data quality to judge the quality of data.

[0006] The technical solution adopted by the present invention to solve the above technical problems is: a data quality assessment method based on random forest and fuzzy hierarchical analysis, the main steps of the method are as follows:

[0007] Step 1: Obtain the historical data set of the data to be evaluated and the usage frequency set of each data, and determine the set of quantitative indicators for data evaluation.

[0008] Step 2: Determine the calculation method for all quantitative indicator scores in the quantitative indicator set.

[0009] Step 3: Based on the set of quantitative indicators, calculate the score of each quantitative indicator corresponding to each data in the historical data set to form a data matrix.

[0010] Step 4: Randomly divide the data matrix and usage frequency set into training set and validation set by row.

[0011] Step 5: Build a random forest and set model parameters. Input the training set and validation set into the random forest for training.

[0012] Step 6: Evaluate the performance of the random forest prediction results on the training set and validation set.

[0013] Step 7: Use random forest to calculate the relative importance vector between the elements in the data evaluation quantitative indicator set.

[0014] Step 8: Calculate the judgment matrix based on the relative importance vector.

[0015] Step 9: Calculate the weight of the elements in the quantitative indicator set for the comprehensive score of data quality based on the judgment matrix.

[0016] Step 10: Get the data set to be evaluated, calculate the score of each data in the data set corresponding to each quantitative standard according to the quantitative indicator set, construct a data matrix, and calculate the comprehensive score of the data according to the weight vector.

[0017] Preferably, the data evaluation quantitative indicator set in step 1 includes: completeness, richness, immediacy, accuracy, security, consistency, and reliability. The data quality evaluation standard is that the higher the usage frequency of the data, the better the data quality. Preferably, the calculation methods of all quantitative indicator scores in the quantitative indicator set in step 2 are respectively:

[0018] (1) Completeness score

[0019]

[0020] (2) Richness score

[0021] S(x2)=data s i number of fields

[0022] (3) Immediacy score

[0023]

[0024] (4) Accuracy score

[0025]

[0026] (5) Safety score

[0027] S(x5)=data s i Access level

[0028] (6) Consistency score

[0029]

[0030] (7) Reliability score

[0031] S(x7)=data s i Credibility of the source.

[0032] Preferably, the parameters of the random forest in step 5 are set as the number of trees n_estimators=81, the maximum depth of the tree max_depth=100, the minimum number of leaf samples min_samples_leaf=1, and the minimum number of branch node samples min_samples_split=2.

[0033] Preferably, the calculation method of the judgment matrix in step 8 is:

[0034]

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] 1. The random forest method used in the present invention is a commonly used ensemble learning method with powerful classification and regression capabilities, and can effectively analyze various aspects of data characteristics. At the same time, the fuzzy analytic hierarchy process (FAHP) is also used, which is an improvement and extension of AHP. It introduces fuzzy mathematical theory into AHP, improves the problems existing in AHP, makes the judgment more accurate, and has a wider scope of application.

[0037] 2. By combining the two methods of random forest and fuzzy hierarchical analysis method, the present invention can further improve the accuracy and efficiency of data quality assessment while ensuring objectivity. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic diagram of a specific application scenario of the present invention.

[0039] Figure 2 It is an execution flow chart of the present invention. DETAILED DESCRIPTION

[0040] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0041] like Figure 1 and Figure 2 As shown, the present invention proposes a data quality assessment method based on random forest and fuzzy hierarchical analysis, which includes the following steps:

[0042] Step 1: Obtain the historical data set S = {s1, s2, s3, ..., s n} and the usage frequency set of each data R = {r1, r2, r3, ..., r n}, determine the data quality assessment standard as data s i The corresponding r i The higher the value, the better the data quality. The set of quantitative indicators for data evaluation is selected as X = {x1, x2, x3, ..., x7}, where the element x i The corresponding specific indicators are as follows:

[0043] X = {completeness, richness, immediacy, accuracy, security, consistency, reliability}

[0044] Step 2: Determine all quantitative indicators x in the quantitative indicator set X i The calculation method of the score, the calculation formula of all quantitative indicators is as follows:

[0045] (1) Completeness score

[0046]

[0047] (2) Richness score

[0048] S(x2)=data s i number of fields

[0049] (3) Immediacy score

[0050]

[0051] (4) Accuracy score

[0052]

[0053] (5) Safety score

[0054] S(x5)=data s i Access level

[0055] (6) Consistency score

[0056]

[0057] (7) Reliability score

[0058] S(x7)=data s i Credibility of the source

[0059] Where Norm represents the normalization process, the specific process is as follows:

[0060]

[0061] Step 3: Calculate the value of each data item s in the historical data set S according to the quantitative indicator set X i Corresponding to each quantitative indicator x j The score is recorded as v ij After calculating the scores of each data item in the data set S, the data matrix V is constructed.

[0062]

[0063] Step 4: Randomly divide the data matrix V into two data sub-matrices V1 and V2 with an element ratio of 7:3 by row. Use data sub-matrix V1 as the training set and data sub-matrix V2 as the validation set. Divide the usage frequency set R into subsets R1 and R2 in the same way, so that the data in R1 and R2 correspond to the data in V1 and V2.

[0064] Step 5: Use the sklearn library to build a random forest, set the number of trees n_estimators = 81, the maximum depth of the tree max_depth = 100, the minimum number of leaf samples min_samples_leaf = 1, and the minimum number of branch nodes min_samples_split = 2. Input the training set V1, R1 and the validation set V2, R2 into the random forest for training.

[0065] Step 6: Calculate the determination coefficient K of the random forest prediction results on the training set and validation set 2 , the formula is as follows:

[0066]

[0067] where r i is the true value, is the predicted value, is the mean of the true value. If K is satisfied on both the training set and the validation set 2 >0.7, the training is complete, otherwise jump to step 5 and modify the parameters.

[0068] Step 7: Use random forest to calculate the relative importance vector H = {h1, h2, h3, ..., h7} between the elements in the data evaluation quantitative indicator set X.

[0069] Step 8: For the feature x in the quantitative indicator set X j Its relative and characteristic x i The importance of

[0070]

[0071] The judgment matrix I is obtained by calculating all elements in the quantitative index set X.

[0072] Step 9: Calculate the element x in the quantitative index set X according to the following formula: i The weight w for the comprehensive score of the data i

[0073]

[0074] The weight vector W = {w1, w2, w3, ..., w7} is obtained.

[0075] Step 10: Get the dataset to be evaluated P = {p1, p2, p3, ..., p k}, according to the quantitative indicator set X, calculate each data p in the data set P i For each quantitative standard x j The scores form the data matrix The comprehensive score of the data calculated according to the weight vector W is E = G*W = {e1, e2, e3, ..., e k}.

[0076] Example:

[0077] 1. Preparation of training data

[0078] For the specific scenario of data quality assessment, determine the quantitative standards for data quality assessment according to specific needs, and define the score calculation method for each quantitative standard. Then select the data source to be evaluated, retrieve historical data and its usage frequency from the database, and calculate the scores of historical data on each quantitative standard as training data. See steps 1, 2, and 3 for details.

[0079] 2. Random Forest Training and Calculation of Quantization Standard Weights

[0080] Divide the training data into a test set and a validation set, initialize the random forest, input the training data into the random forest training, and continuously adjust the model parameters until the determination coefficient of the prediction result meets the standard. According to the relative importance vector of the quantitative standard output by the model, a fuzzy complementary judgment matrix is ​​constructed to calculate the weight of the quantitative standard. See steps 4-9 for details.

[0081] 3. Calculation of the final data quality score

[0082] For the newly added data in the future, first calculate its scores on each quantitative standard, and then calculate the final score of its data quality based on the weights. See step 10 for details.

[0083] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A data quality assessment method based on random forest and fuzzy hierarchical analysis, characterized in that: The following steps are involved: Step 1: Obtain the historical data set of the data to be evaluated and the usage frequency set of each data, and determine the set of quantitative indicators for data evaluation; Step 2: Determine the calculation method for all quantitative indicator scores in the quantitative indicator set; Step 3: According to the quantitative indicator set, calculate the score of each quantitative indicator corresponding to each data in the historical data set to form a data matrix; Step 4: Randomly divide the data matrix and usage frequency set into training set and validation set by row; Step 5: Build a random forest, set model parameters, and input the training set and validation set into the random forest for training; Step 6: Evaluate the prediction performance of random forest on the training set and validation set; Step 7: Use random forest to calculate the relative importance vector between the elements in the data evaluation quantitative indicator set; Step 8: Calculate the judgment matrix based on the relative importance vector; Step 9: Calculate the weight of the elements in the quantitative indicator set for the comprehensive score of data quality according to the judgment matrix; Step 10: Get the data set to be evaluated, calculate the score of each data in the data set corresponding to each quantitative standard according to the quantitative indicator set, construct a data matrix, and calculate the comprehensive score of the data according to the weight vector.

2. A data quality assessment method based on random forest and fuzzy hierarchical analysis according to claim 1, characterized in that: The data evaluation quantitative indicator set in step 1 includes: completeness, richness, immediacy, accuracy, security, consistency, and reliability. The data quality evaluation standard is that the higher the usage frequency of the data, the better the data quality.

3. The data quality assessment method based on random forest and fuzzy hierarchical analysis according to claim 1 is characterized in that: The calculation methods of all quantitative indicator scores in the quantitative indicator set in step 2 are respectively: (1) Completeness score (2) Richness score S(x2)=data s i number of fields (3) Immediacy score (4) Accuracy score (5) Safety score S(x5)=data s i Access level (6) Consistency score (7) Reliability score S(x7)=data s i Credibility of the source.

4. The data quality assessment method based on random forest and fuzzy hierarchical analysis according to claim 1 is characterized in that: The parameters of the random forest in step 5 are set as the number of trees n_estimators=81, the maximum depth of the tree max_depth=100, the minimum number of leaf samples min_samples_leaf=1, and the minimum number of branch node samples min_samples_split=2.

5. The data quality assessment method based on random forest and fuzzy hierarchical analysis according to claim 1 is characterized in that: The calculation method of the judgment matrix in step 8 is:

Images