Big data quality assessment method, device, equipment and storage medium
By constructing a big data quality model to evaluate indicators such as the completeness and accuracy of datasets, the problem of the lack of standardized methods for big data quality assessment is solved, enabling efficient data quality assessment and report generation, and supporting better business decisions.
Patent Information
- Application Number
- CN202410580880.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-11
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-05-11
AI Technical Summary
Existing technologies lack effective standard methods to measure and evaluate the quality of big data, leading to business losses, resource waste, and decision-making errors caused by low-quality data, making it difficult to ensure data quality control from the source.
By constructing a big data quality model, including models for completeness, accuracy, timeliness, uniqueness, validity, consistency, reliability, and availability, and combining weighted coefficients, the quality standard score of the dataset to be evaluated is assessed, and a data quality score is generated.
It enables accurate assessment of big data quality, generates detailed data quality reports, supports better business decisions, and ensures business success.
Smart Images

Figure CN118152693B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of Internet big data technology, and in particular to big data quality assessment methods, devices, equipment and storage media. Background Art
[0002] Due to the advancement of technologies such as cloud computing, the Internet of Things, social networking devices, and the use of mobile applications, the amount of data generated is now greater than ever before. However, due to the huge amount of data generated, the speed at which new data arrives, and the variety of heterogeneous data. Low-quality data often brings countless negative effects. First of all, one negative effect of low-quality data is missed business. Poor data quality may cause many tangible and intangible losses to the enterprise. For a typical organization, the estimated cost may be as high as 8% to 12% of revenue and may generate 40% to 60% of the service organization's expenses. Obviously, bad data may hinder revenue goals. Low-quality data may also lead to communication errors, resulting in dissatisfied customers. Another negative effect of low-quality data is increased resource consumption. However, since most companies usually don't know why data quality is important, they will wait until there is a problem with the data before seeking a solution. In this way, a lot of labor and time are wasted. Finally, poor service based on wrong data leads to wrong decisions, resulting in poor product quality. As a result, the service will not meet the expected quality standards, so all the hard work, time and labor invested may be of little value.
[0003] There are often the following reasons for the poor quality of big data: The five variables of big data (variety, volume, value, velocity, and accuracy), while important, also lead to problems in measuring the quality of big data. Due to the high volume of data, it is challenging to maintain data quality within a given time. It is also difficult to integrate data due to the presence of multiple data formats. In addition, different organizations have different needs for data, so they all need their own data processing technologies. They also need to have their own big data management and quality assurance methods. Poor management in any of these areas will lead to substandard data quality. At the same time, in the processing of big data, there are factors such as data collection, data conversion, data service scalability, and data conversion. Due to its inherent high volume, big data often faces challenges in collection, conversion, and transformation. Ultimately, this leads to poor quality of data organization. Therefore, it is difficult to control the poor quality of big data from the source, so it is urgent to conduct timely evaluation of the quality of the collected big data in order to manage big data.
[0004] But currently, there is no standard way to measure data quality, so there is still a need to develop completely reliable benchmarks. Summary of the invention
[0005] The present invention provides a big data quality assessment method, apparatus, device and storage medium.
[0006] According to a first aspect of the present disclosure, a method for evaluating the quality of big data is provided. The method comprises:
[0007] Acquire a data set to be evaluated and an expected data set; the data set to be evaluated is a plurality of data collected by sensors of a network site; the expected data set is a preset data set corresponding to the network site;
[0008] Input the data set to be evaluated and the expected data set into a pre-built big data quality model, and output the quality standard score corresponding to the data set to be evaluated;
[0009] Determine the corresponding weight coefficient according to the quality standard score;
[0010] The data quality score of the data set to be evaluated is determined according to the quality standard score and the corresponding weight coefficient.
[0011] According to the above aspects and any possible implementation, an implementation is further provided, wherein the data set to be evaluated includes: a plurality of transactions collected by a sensor of a network site, and a parameter corresponding to each transaction, as shown in the following R matrix:
[0012]
[0013] Where mr is the total number of transactions collected, and nr is the total number of parameters for each transaction; Represents the value of the i-th transaction and the j-th parameter;
[0014] The expected data set includes: a plurality of transactions that are expected to be collected by the website, and parameters corresponding to each transaction, as shown in the following E matrix:
[0015]
[0016] Where m represents the total number of expected transactions, n is the total number of parameters for each expected transaction; E ij Represents the value of the i-th transaction and the j-th parameter.
[0017] According to the aspects described above and any possible implementation methods, an implementation method is further provided, and the big data quality model includes: a big data integrity model, a big data accuracy model, a big data timeliness model, a big data uniqueness model, a big data validity model, a big data consistency model, a big data reliability model and a big data availability model.
[0018] According to the above aspects and any possible implementation, an implementation is further provided, wherein the quality standard score includes: a completeness score, an accuracy score, a timeliness score, a uniqueness score, a validity score, a consistency score, a reliability score, and a usability score;
[0019] The data set to be evaluated also includes: an actual timestamp consisting of the time of each transaction collected;
[0020] The expected data set further includes: an expected timestamp consisting of a preset expected arrival time of each transaction, a preset validity rule and a preset total number of sensors of the network site;
[0021] The step of inputting the to-be-evaluated data set and the expected data set into a pre-built big data quality model and outputting a quality standard score corresponding to the to-be-evaluated data set includes:
[0022] Inputting the data set to be evaluated and the expected data set into a big data integrity model, and outputting the integrity score of the data set to be evaluated;
[0023] Inputting the data set to be evaluated and the expected data set into a big data accuracy model, and outputting the accuracy score of the data set to be evaluated;
[0024] Generate a timestamp array according to the actual timestamp and the expected timestamp; input the timestamp array into a big data timeliness model, and output a timeliness score of the data set to be evaluated;
[0025] Inputting the data set to be evaluated into a big data uniqueness model, and outputting a uniqueness score of the data set to be evaluated;
[0026] Inputting the data set to be evaluated and the validity rule into a big data validity model, and outputting a validity score of the data set to be evaluated;
[0027] Inputting the data set to be evaluated and the actual timestamp into a big data consistency model, and outputting a consistency score of the data set to be evaluated;
[0028] The number of data sets of the data set to be evaluated is used as the number of sensors that are normally working at the network site; the number of sensors that are normally working and the total number of sensors at the network site are input into a big data reliability model, and a reliability score of the data set to be evaluated is output;
[0029] The completeness score, accuracy score and timeliness score are input into a big data availability model, and the availability score of the data set to be evaluated is output.
[0030] According to the above aspects and any possible implementation, there is further provided an implementation, wherein the big data integrity model includes a transaction integrity model and a parameter integrity model;
[0031] The step of inputting the to-be-evaluated data set and the expected data set into a big data integrity model and outputting the integrity score of the to-be-evaluated data set includes:
[0032] Input the to-be-evaluated data set and the expected data set into a transaction integrity model, and output an average transaction completion rate;
[0033] Inputting the data set to be evaluated and the expected data set into a parameter integrity model, and outputting average parameter integrity;
[0034] The preset weight value is used as a coefficient of the average transaction completion rate and the average parameter integrity, and the sum is calculated to obtain the integrity score of the data set to be evaluated.
[0035] According to the above aspects and any possible implementation, an implementation is further provided, wherein the data set to be evaluated and the expected data set are input into a transaction integrity model, and the average transaction completion rate is output, which is implemented by the following formula:
[0036] Formula 1: ;
[0037] ;
[0038] ;
[0039] ;
[0040] ;
[0041] ;
[0042] ;
[0043] in, is the completeness of transaction i in the R matrix; is the total number of missing data in the R matrix with mr transactions and nr parameters; is the total number of null values in the R matrix; is the data set in the E matrix corresponding to transaction i: E i1 ×E in ; is the data set in the R matrix corresponding to transaction i: R i1 ×R inr ; is the completion percentage of transaction i; is the time difference between two transactions, is the total number of hours in a day in which these two transactions occur; is the affairs of day j; represents the average transaction completion rate of transaction i on day j; is the number of days in month j when the event occurs; Derived from formula 5; is the average transaction completion rate of transactions in the jth month; represents the number of months in which transactions occurred in year j; represents the average transaction completion rate in year j; Derived from Equation 6.
[0044] According to the above aspects and any possible implementation manner, an implementation manner is further provided, wherein determining the data quality score of the to-be-evaluated data set according to the quality standard score and the corresponding weight coefficient comprises:
[0045] Multiplying the completeness score, accuracy score, timeliness score, uniqueness score, validity score, consistency score, reliability score and availability score by corresponding weight coefficients respectively to obtain completeness quality score, accuracy quality score, timeliness quality score, uniqueness quality score, validity quality score, consistency quality score, reliability quality score and availability quality score as data quality scores of the data set to be evaluated;
[0046] And / or, the sum of the integrity quality score, accuracy quality score, timeliness quality score, uniqueness quality score, validity quality score, consistency quality score, reliability quality score and availability quality score is used as the data quality score of the data set to be evaluated.
[0047] According to a second aspect of the present disclosure, a large data quality assessment device is provided. The device comprises:
[0048] A data acquisition module, used to acquire a data set to be evaluated and an expected data set; the data set to be evaluated is a plurality of data collected by sensors of a network site; the expected data set is a preset data set corresponding to the network site;
[0049] A quality score determination module is used to input the data set to be evaluated and the expected data set into a pre-built big data quality model, and output a quality standard score corresponding to the data set to be evaluated;
[0050] A weight determination module, used to determine a corresponding weight coefficient according to the quality standard score;
[0051] The quality score calculation module is used to determine the data quality score of the data set to be evaluated according to the quality standard score and the corresponding weight coefficient.
[0052] According to a third aspect of the present disclosure, an electronic device is provided, which includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the program, the method described above is implemented.
[0053] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method according to the first aspect of the present disclosure is implemented.
[0054] The big data quality assessment method, apparatus, device and storage medium provided in the embodiments of the present disclosure are intended to model eight big data parameters to more accurately assess data quality and generate more detailed data quality reports, thereby providing more scientific data support for making better business decisions and ensuring business success.
[0055] It should be understood that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, among which:
[0057] Figure 1 A flowchart of a big data quality assessment method according to an embodiment of the present disclosure is shown;
[0058] Figure 2 A block diagram of a large data quality assessment device according to an embodiment of the present disclosure is shown;
[0059] Figure 3 A block diagram of an exemplary electronic device capable of implementing embodiments of the present disclosure is shown. DETAILED DESCRIPTION
[0060] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.
[0061] In addition, the term "and / or" in this article is only a description of the association relationship between the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.
[0062] This application proposes an eight-parameter model for big data quality assurance to assess data quality and ensure that it is of high quality. Data is of high quality if it is suitable for its intended use in operations, decision-making, and planning. High-quality data is accurate, available, complete, consistent, trustworthy, processable, relevant, and timely. This application relies on eight quality parameters: (1) Completeness: whether all required values are in the data set; (2) Accuracy: whether the data accurately describes an event or object; (3) Timeliness: whether the data arrives at the expected time; (4) Uniqueness: whether there is redundancy in the data set; (5) Validity: whether the data follows specific rules; (6) Consistency: whether there are any contradictions in the data; (7) Reliability of instruments / sensors: reliability of the state of machine-collected data; (8) Availability: whether the data meets given requirements to check the quality standards of big data.
[0063] 1) Big data completeness is a measure of the amount of available data versus the amount of data required for the intended use. Completeness is used to verify whether defects in the data affect its usability. Big data completeness can be defined as the ratio of stored data relative to the potential of 100% complete data. To measure completeness, this application takes the number of available values in a given data set and calculates its ratio to the total expected values. The unit of measurement is a percentage.
[0064] 2) Big data accuracy can be defined as the extent to which the data correctly describes the "real world" object or event being considered. In order to measure the accuracy of a data set or data item, the data is compared to the truth of the "real world". Third-party reference data is usually used, which is generally considered to be credible and homogeneous data. The unit of measurement is the percentage of data entries that meet the data accuracy requirements. In some cases, accuracy is easy to measure, for example, distinguishing between genders (i.e., male or female). Other cases may not have such a clear distinction, which makes accuracy more difficult to measure. Accuracy helps answer questions such as whether the data provided is accurate, whether it causes ambiguity, and whether it reflects the true state of the data source.
[0065] 3) The timeliness of big data is an important factor in the quality assessment of big data because data changes every second. The timeliness of big data is measured by the degree to which the data represents reality at the desired point in time. To measure timeliness, the time difference between the occurrence of an event and its recording can be marked. In other words, it is the difference between the expected time data and the readily available time data. The unit of measurement is the percentage of the time difference. Timeliness helps determine whether the data arrives on time and whether the data is updated regularly.
[0066] 4) Big data uniqueness is defined as the measure of a data item relative to itself or its counterpart in another dataset or database. The unit of measure is percentage. This parameter is used to confirm that the dataset has no duplicate values. In big data, checking this factor helps to eliminate redundancy.
[0067] 5) Big data validity is also known as data correctness. Data is valid if it complies with its defined syntax (format, type, and scope). To measure validity, data needs to be compared with the validity rules defined for it. The unit of measurement is a percentage. It helps to understand whether the data is valid for its intended use. This application models validity at both transaction and parameter levels.
[0068] 6) Big data consistency refers to the degree to which the logical relationship between related data is correct and complete. That is, there is no difference when comparing two or more representations of the same thing. To measure consistency, a data item can be measured based on a data item in another data set or its counterpart. To maintain consistency, two data sets should have the same value and the same meaning. Therefore, it is necessary to check the consistency between them. This application models the value and time consistency of data.
[0069] 7) The big data reliability of a system is defined as the ability of a network to ensure reliable data transmission under the condition of a constantly changing network structure. To measure the reliability of a system, it is necessary to describe whether a component or system works normally according to its specifications within a specific time. Check the sensor to determine whether it is reliable.
[0070] 8) Big data availability can be defined as whether the data is useful and meets user needs. To measure availability, timeliness, accuracy, and completeness need to be calculated because the values of these three quality parameters define whether the data is usable. The unit of measurement is percentage.
[0071] Figure 1 A flowchart of a big data quality assessment method 100 according to an embodiment of the present disclosure is shown. The method 100 includes:
[0072] Step 110: Obtain the data set to be evaluated and the expected data set.
[0073] The data set to be evaluated is a plurality of data collected by sensors of a network site; the expected data set is a preset data set corresponding to the network site.
[0074] In some embodiments, the data set to be evaluated includes: multiple transactions collected by sensors of network sites, and parameters corresponding to each transaction; the expected data set includes: multiple transactions expected to be collected by the network sites, and parameters corresponding to each transaction; let S represent k sites in the network, so that , where S i represents the i-th sensor in the site. Assume that in sensor S i In general, one expects the dataset to arrive with m transactions, each consisting of n parameters. In addition, the sensor S i Receive a data set with mr transactions, each transaction has nr parameters. Let the E matrix be the expected data set and the R matrix represent the data set to be evaluated, as shown below:
[0075] R matrix:
[0076]
[0077] Where mr is the total number of transactions collected, and nr is the total number of parameters for each transaction; Represents the value of the i-th transaction and the j-th parameter;
[0078] E Matrix:
[0079]
[0080] Where m represents the total number of expected transactions, n is the total number of parameters for each expected transaction; E ij Represents the value of the i-th transaction and the j-th parameter.
[0081] Of course, in some embodiments, the expected data set may also be preset, and the corresponding expected data set may be determined directly based on which sensor of which network site the data set to be evaluated is collected by.
[0082] Step 120: input the dataset to be evaluated and the expected dataset into a pre-built big data quality model, and output a quality standard score corresponding to the dataset to be evaluated.
[0083] In some embodiments, the big data quality model includes: a big data integrity model, a big data precision model, a big data timeliness model, a big data uniqueness model, a big data validity model, a big data consistency model, a big data reliability model and a big data availability model. The quality standard score includes: a completeness score, an accuracy score, a timeliness score, a uniqueness score, a validity score, a consistency score, a reliability score and an availability score. The data set to be evaluated also includes: an actual timestamp consisting of the time of each transaction collected; the expected data set also includes: an expected timestamp consisting of the preset expected arrival time of each transaction, a preset validity rule and a preset total number of sensors of the network site; the data set to be evaluated and the expected data set are input into the pre-built big data quality model, and the quality standard score corresponding to the data set to be evaluated is output, including: inputting the data set to be evaluated and the expected data set into the big data integrity model, and outputting the integrity score of the data set to be evaluated; inputting the data set to be evaluated and the expected data set into the big data precision model, and outputting the accuracy score of the data set to be evaluated; generating a timestamp array according to the actual timestamp and the expected timestamp; inputting the timestamp array into the big data timeliness model, and outputting The timeliness score of the data set to be evaluated; input the data set to be evaluated into the big data uniqueness model, and output the uniqueness score of the data set to be evaluated; input the data set to be evaluated and the validity rule into the big data validity model, and output the validity score of the data set to be evaluated; input the data set to be evaluated and the actual timestamp into the big data consistency model, and output the consistency score of the data set to be evaluated; use the data set number of the data set to be evaluated as the number of sensors working normally at the network site; input the number of sensors working normally and the total number of sensors at the network site into the big data reliability model, and output the reliability score of the data set to be evaluated; input the integrity score, accuracy score and timeliness score into the big data availability model, and output the availability score of the data set to be evaluated. The following is a detailed description of the process of evaluating data by the eight models in combination with the formulas.
[0084] 1. Big Data Integrity Model
[0085] In some embodiments, the big data integrity model includes a transaction integrity model and a parameter integrity model; the step of inputting the data set to be evaluated and the expected data set into the big data integrity model and outputting the integrity score of the data set to be evaluated includes: inputting the data set to be evaluated and the expected data set into the transaction integrity model and outputting the average transaction completion rate; inputting the data set to be evaluated and the expected data set into the parameter integrity model and outputting the average parameter integrity; using preset weight values as coefficients of the average transaction completion rate and the average parameter integrity, and summing them up to obtain the integrity score of the data set to be evaluated.
[0086] In some embodiments, the inputting of the to-be-evaluated data set and the expected data set into the transaction integrity model and outputting the average transaction completion rate is implemented by the following formula:
[0087] Formula 1: ;
[0088] ;
[0089] ;
[0090] ;
[0091] ;
[0092] ;
[0093] ;
[0094] in, is the completeness of transaction i in the R matrix; is the total number of missing data in the R matrix with mr transactions and nr parameters; is the total number of null values in the R matrix; is the data set in the E matrix corresponding to transaction i: E i1 ×E in ; is the data set in the R matrix corresponding to transaction i: R i1 ×R inr ; is the completion percentage of transaction i; is the time difference between two transactions, is the total number of hours in a day in which these two transactions occur; is the affairs of day j; represents the average transaction completion rate of transaction i on day j; is the number of days in month j when the event occurs; Derived from formula 5; is the average transaction completion rate of transactions in the jth month; represents the number of months in which transactions occurred in year j; represents the average transaction completion rate in year j; Derived from Equation 6.
[0095] In some embodiments, the inputting of the to-be-evaluated data set and the expected data set into a parameter integrity model and outputting average parameter integrity is achieved by the following formula:
[0096] When defining big data, let X be the parameter for computing completeness. Here, the matrix R constitutes all the values in the parameter X. , calculate the total number of missing data in the matrix R using the number of transactions mr. In order to calculate completeness, it is necessary to know the expected amount of data to consider the received data set complete. , find the total number of expected values in the data set, is the total number of missing data in the R matrix with mr transactions and nr parameters; is the total number of null values in the R matrix; since the requirement is the completeness of the parameters, is the data set in the E matrix corresponding to parameter i: E 1i ×E mi ; is the data set in the R matrix corresponding to parameter i: R 1i ×R mri .
[0097] Assumptions represents the integrity of parameter i. The subscript 𝑝𝑎𝑟𝑎𝑚 indicates that the integrity is measured based on the parameter. It can be defined as:
[0098]
[0099] is the integrity percentage of parameter i. It can be defined as:
[0100]
[0101] Assumptions represents the average integrity of the jth day, which can be defined as Formula 8:
[0102]
[0103] in is the transaction on day j derived from Formula 4, the sum applied to all values of i (i value ranges from 1 to ). In order to calculate the completeness of all transactions that occurred on day j. The sum is divided by Produces the average parameter integrity for day j.
[0104] Assumptions is the average completeness of the parameters in the jth month, so It can be defined as formula 9:
[0105]
[0106] here is the number of days in j months when the event occurs, From Equation 8. The sum is applied to all values of i (i value ranges from 1 to ). In order to calculate the j month The completeness of all transactions that occurred during the day, this sum is divided by , and obtain the average parameter completeness for each month.
[0107] Assumptions represents the average integrity of the parameter in year j. Therefore, 𝐶𝑜𝑚𝑝l𝑒𝑡𝑒𝑛𝑒𝑠𝑠𝑗 can be given by the following formula:
[0108]
[0109] here 𝑗 represents the month number in which the transaction occurs in year 𝑗, and From Equation 9, the sum is applied to all values of i (i value ranges from 1 to ). In order to calculate the j year The total value of all transactions that occurred in the month is divided by , and the average parameter completeness for each year is obtained.
[0110] In some embodiments, the data quality report can accurately reflect the daily transaction integrity, monthly transaction integrity, and annual transaction integrity obtained according to the above-mentioned big data integrity model, as well as the daily parameter integrity, monthly parameter integrity, and annual parameter integrity. Of course, according to user needs, the total integrity value of the data set to be evaluated can be output, that is, all transaction integrity and parameter integrity are added and summed. Among them, according to user needs, different coefficients can be set for transactions and parameters that need to be paid attention to, and corresponding multiplication operations can be performed and then added and summed, etc., which can be flexibly set, and the results are finally output to the data quality report.
[0111] 2. Big Data Precision Model
[0112] Here are the accuracy per transaction and accuracy per parameter models. The accuracy per transaction model checks the accuracy of each element in a single transaction. The transaction accuracy parameter model checks each element in the parameter during all transactions at a given time. Both use percentages as the unit of measurement.
[0113] 1) Accuracy of each transaction. In order to calculate the accuracy, a reference dataset is required. The expected dataset described in the previous section is the reference dataset for all calculations. The distance between the two selected datasets gives their accuracy.
[0114] The accuracy of each transaction is defined as For transaction i, j is the difference between the reference dataset and the dataset to be evaluated for transaction i and parameter j:
[0115]
[0116] Where i represents the number of transactions and j represents the number of parameters.
[0117] Assumptions is the accuracy of transaction i. It can be defined as:
[0118]
[0119] is the accuracy percentage of transaction i. It can be defined as:
[0120]
[0121] Assumptions represents the average accuracy of transactions on day j, which can be defined as Formula 10:
[0122]
[0123] in is the transaction for day j derived from Equation 4, the sum is applied to 𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦𝑡𝑟𝑎𝑛𝑖 % for all values of i, i equals 1 to . Calculate the accuracy of all transactions that occurred on day j. This sum is divided by , and the average transaction accuracy on day j is obtained.
[0124] Assumptions is the average accuracy of transactions in month j. Therefore It can be defined as formula 11:
[0125]
[0126] here is the number of days in the jth month when the transaction occurs. It is derived from Equation 10 that the sum is applied to all values of i , i value is equal to 1 to Calculate j month The accuracy of all transactions that occurred during the day, this sum divided by , and the average transaction accuracy per month is obtained.
[0127] Assumptions represents the average accuracy of each transaction in year j. It can be given by the following formula:
[0128]
[0129] here represents the month number of the transaction in year j, where Derived from Equation 11. The summation is applied to all values of i , i value is equal to 1 to . Calculate year j The accuracy of all transactions that occurred during the month, this sum divided by , and obtain the average transaction accuracy per year.
[0130] 2) Accuracy of each parameter. For the accuracy of each parameter, here, the number of transactions will be displayed as m, which represents the maximum number of transactions for each parameter between the reference and received datasets.
[0131] The precision of each parameter is defined as . It is defined as:
[0132]
[0133] in , m is the number of transactions for each parameter.
[0134] Assumptions is the accuracy percentage of parameter i, which is defined as:
[0135]
[0136] The parameter represents the average accuracy of the jth day. It can be defined as formula 12:
[0137]
[0138] in is the transaction on day j derived from Equation 4, and the sum is applied to all values of i , i value is equal to 1 to . To calculate the accuracy of all transactions occurring on day j. The sum is divided by , and the average parameter accuracy on day j is obtained.
[0139] Assumptions is the average accuracy of parameter j in each month. It can be defined as formula 13:
[0140]
[0141] in is the number of days in the jth month when the transaction occurs, This summation is applied to all values of i. , i value ranges from 1 to To calculate j month The accuracy of all transactions that occurred during the day, divided by the total , and the average parameter accuracy for each month is obtained.
[0142] Assumptions represents the average accuracy of the parameters in year j, so It can be given by the following formula:
[0143]
[0144] in Indicates the month number of the event in year j, is derived from Equation 13. The summation is applied to all values of i , i value ranges from 1 to . To calculate year j The accuracy of all transactions that occurred in the month, the sum of which is divided by , and the average parameter precision for each year is obtained.
[0145] In some embodiments, the data quality report can accurately reflect the daily transaction accuracy, monthly transaction accuracy, and annual transaction accuracy obtained according to the above-mentioned big data accuracy model, as well as the daily parameter accuracy, monthly parameter accuracy, and annual parameter accuracy. Of course, according to user needs, the total accuracy value of the data set to be evaluated can be output, that is, all transaction accuracy and parameter accuracy are added and summed. Among them, according to user needs, different coefficients can be set for the transactions and parameters that need to be paid attention to, and corresponding multiplication operations can be performed and then added and summed, etc., which can be flexibly set, and the results are finally output to the data quality report.
[0146] 3. Big Data Timeliness Model
[0147] To measure timeliness, we need to store a timestamp for each transaction. An array of timestamps representing the start and end time of each record. Thus, ,in represents the expected time of arrival of transaction i, that is, the expected timestamp, represents the actual receiving time of transaction i, that is, the actual timestamp. is the timeliness of transaction i. It can be defined as:
[0148]
[0149] Assumptions is the timeliness percentage of transaction i. It can be defined as:
[0150]
[0151] Assumptions represents the average timeliness of transactions on day j. It can be defined as formula 14:
[0152]
[0153] in is the transaction on day j derived from Equation 4, and the sum is applied to all values of i , i value is equal to 1 to . To calculate the timeliness of all transactions occurring on day j. The sum is divided by , and obtain the average transaction timeliness on the jth day.
[0154] Assumptions is the average timeliness of transactions in month j. So It can be defined as formula 15:
[0155]
[0156] in is the number of days in month j when the transaction occurs, and Derived from Equation 14. The summation is applied to For all values of i, i is equal to 1 to To calculate j month The timeliness of all transactions that occurred during the day. This sum is divided by , and the average transaction timeliness per month is obtained.
[0157] Assumptions represents the average timeliness of transactions in year j. It can be given by the following formula:
[0158]
[0159] in represents the month number in which the transaction occurred in year j, and is derived from Equation 15. The summation is applied to all values of i , i value is equal to 1 to . To calculate year j The accuracy of all transactions that occurred during the month. This sum is divided by , and the average transaction timeliness is obtained each year.
[0160] In some embodiments, the data quality report can accurately reflect the daily transaction timeliness, monthly transaction timeliness, and annual transaction timeliness obtained according to the above big data timeliness model. Of course, according to user needs, the total timeliness value of the data set to be evaluated can be output, that is, the timeliness of all transactions can be added and summed. Among them, according to user needs, different coefficients can be set for transactions that need attention, and corresponding multiplication operations can be performed and then added and summed, etc., which can be flexibly set, and the results can be finally output to the data quality report.
[0161] 4. Big Data Uniqueness Model
[0162] Big data uniqueness is measured by comparing data with corresponding data in the same dataset to check for redundancy. The following describes the uniqueness of each transaction conducted in a day, and then the uniqueness of transactions per month and per year. Suppose there is a transaction; to calculate its uniqueness, compare it with other transactions.
[0163] Assumptions is the uniqueness of transaction i. To define the uniqueness of a transaction, compare the transaction with the rest of the transactions in the dataset. It can be defined as:
[0164]
[0165] is the uniqueness percentage of transaction i. It can be defined as:
[0166]
[0167] let represents the average uniqueness of transactions on day j. It can be defined as Formula 16:
[0168]
[0169] in is the transaction on day j derived from Formula 4, summed for all values of i , i value is equal to 1 to . To calculate the uniqueness of all transactions occurring on day j. The sum is divided by , and the average transaction uniqueness on day j is obtained.
[0170] Assumptions is the average uniqueness of transactions in month j. So It can be defined as formula 17:
[0171]
[0172] in is the day of the jth month in which the transaction occurs, and is derived from Equation 16. The summation is applied to all values of i , i value is equal to 1 to To calculate j month The uniqueness of all transactions that occurred on that day. This sum is divided by , and obtain the average transaction uniqueness for each month.
[0173] Assumptions represents the average unique transactions in year j, so It can be given by the following formula:
[0174]
[0175] in represents the month number in year j in which the transaction occurs, and is derived from Equation 17. The sum is applied to all values of i , i value is equal to 1 to . To calculate year j The uniqueness of all transactions that occurred in the month, this sum divided by , and the average transaction uniqueness per year is obtained.
[0176] In some embodiments, the data quality report can accurately reflect the daily transaction uniqueness, monthly transaction uniqueness, and annual transaction uniqueness obtained according to the above big data uniqueness model. Of course, the total uniqueness of the data set to be evaluated can also be output according to user needs, that is, all transaction uniquenesses are added and summed. Among them, according to user needs, different coefficients can be set for transactions that need attention, and corresponding multiplication operations can be performed and then added and summed, etc., which can be flexibly set, and the results are finally output to the data quality report.
[0177] 5. Big Data Validity Model
[0178] The definition of Big Data validity correctly states that it involves a measure of validity. It is important to have rules or grammars to assess accuracy. This application proposes validity at both transaction and parameter levels.
[0179] 1) Validity of each parameter. To validate the data, there should be specific defined rules, assuming that the data is considered valid.
[0180] Assume that the received data set is R, in which there are mr transactions, and each transaction has nr parameters. For the data set R, assume that the verification standard V = {v1, v2...vk}. To define the validity of each parameter, keep nr = 1. Check the validity of each value item in the parameter to determine which values need to be calculated for validity. To verify the parameter, measure each value of the parameter according to its rule to check the validity.
[0181] The validity of each value in the dataset R is defined as For a value i, it is defined as:
[0182]
[0183] Now, all the parameters in the application Therefore, for parameter i It can be defined as follows:
[0184]
[0185] Where mr is the total number of transactions, assuming is the integrity percentage of parameter i. It can be defined as:
[0186]
[0187] The parameter represents the average effectiveness on day j. It can be defined as formula 18:
[0188]
[0189] in is the transaction on day j derived from Equation 4, and the sum is applied to all values of i , i value is equal to 1 to . To calculate the effectiveness of all parameters occurring on day j. The sum is divided by , and the average parameter effectiveness on the jth day is obtained.
[0190] Assumptions is the average effectiveness of the parameter in j months. Then It can be defined as formula 19:
[0191]
[0192] in is the number of days in month j when the transaction occurs, and is derived from Equation 18. The summation is applied to all values of i , i value is equal to 1 to To calculate j month The total value of all parameters occurring on the day is divided by , and the average parameter effectiveness for each month is obtained.
[0193] Assumptions represents the average effectiveness of the parameters in year j. Then, It can be given by the following formula:
[0194]
[0195] in represents the month number in year j in which the transaction occurs, and is derived from Equation 19. The summation is applied to all values of i , i value is equal to 1 to . To calculate year j The validity of all parameters that occur in the month. The sum is divided by , and the average annual parameter effectiveness is obtained.
[0196] 2) Validity of each transaction. In order to measure the validity of each transaction, validity rules or grammars must be developed for each value in the transaction. This means that each value needs to be compared with its rules. Assume that the received data set is R, with mr transactions, and each transaction has nr parameters. For each value in the transaction, check its validity according to the following formula:
[0197] Now apply all Therefore, for transaction i It can be defined as follows:
[0198]
[0199] Where nr is the total number of parameters per transaction.
[0200] Assumptions is the validity percentage of transaction i. It can be defined as:
[0201]
[0202] Assumptions represents the average effectiveness of the transaction on day j. It can be determined as formula 20:
[0203]
[0204] in is the transaction on day j derived from Equation 4, and the sum is applied to all values of i , i value is equal to 1 to . To calculate the validity of all transactions that occurred on day j. The sum is divided by , and the average transaction effectiveness on day j is obtained.
[0205] Assumptions is the average effectiveness of transactions in month j. Therefore, It can be defined as formula 21:
[0206]
[0207] in is the number of days in month j on which the event occurs, and is derived from Equation 20. The summation is applied to all values of i , i value is equal to 1 to To calculate j month The effectiveness of all transactions that occurred on that day. This sum is divided by , and obtain the average monthly transaction effectiveness.
[0208] Assumptions represents the average effectiveness of transactions in year j. It can be given by the following formula:
[0209]
[0210] in represents the month number in year j in which the transaction occurs, and is derived from Equation 21. The summation is applied to all values of i , i value is equal to 1 to . To calculate year j The effectiveness of all transactions that occurred in the month. This sum is divided by , and obtain the average transaction effectiveness per year.
[0211] In some embodiments, the data quality report can accurately reflect the daily transaction effectiveness, monthly transaction effectiveness, and annual transaction effectiveness, as well as the daily parameter effectiveness, monthly parameter effectiveness, and annual parameter effectiveness obtained according to the above-mentioned big data effectiveness model. Of course, according to user needs, the total effectiveness value of the data set to be evaluated can be output, that is, all transaction effectiveness and parameter effectiveness are added and summed. Among them, according to user needs, different coefficients can be set for the transactions and parameters that need to be paid attention to, and corresponding multiplication operations can be performed and then added and summed, etc., which can be flexibly set, and the results are finally output to the data quality report.
[0212] 6. Big Data Consistency Model
[0213] This application proposes two kinds of consistency; one based on parameters and the other based on time. In the case of parameter consistency, each value is compared with the value of a different data set. Whereas in the case of time-based, the timestamp is compared with two data sets.
[0214] 1) Consistency of each parameter. The consistency of each parameter i of sensor X dataset is defined, and the dataset of sensor Y is used as the reference dataset.
[0215]
[0216] The dimensions of the dataset of X should be equal to the dataset of Y. If not, then nulls in the missing dimensions are replaced with equality so that mr is the total number of transactions and is equal to the maximum of the number of transactions of the two datasets.
[0217] The consistency of sensor X relative to Y is determined as follows:
[0218]
[0219] The consistency of each parameter is defined as , which is defined as:
[0220]
[0221] Where mr is the total number of transactions.
[0222] Assumptions is the consistency percentage of parameter i. It can be defined as:
[0223]
[0224] let represents the average consistency of the parameter on day j, which can be defined as Formula 22:
[0225]
[0226] in is the transaction on day j derived from Equation 4, and the sum is applied to all values of i , i value is equal to 1 to . To calculate the consistency of all parameters occurring on day j. This sum is divided by , yielding the average parameter consistency for day j.
[0227] Assumptions is the average consistency of the parameters in the jth month, so, It can be defined as formula 23:
[0228]
[0229] in is the number of days in month j when the transaction occurs, and is derived from Equation 22. The summation is applied to all values of i , i value is equal to 1 to To calculate j month The consistency of all parameters occurring on the day. This sum is divided by , and the average parameter consistency for each month was obtained.
[0230] Assumptions represents the average consistency of the parameters in the jth year, so, It can be given by the following formula:
[0231]
[0232] in represents the month number in which the transaction occurred in year j, and is derived from Equation 23. The summation is applied to all values of i , i value is equal to 1 to . To calculate year j The consistency of all parameters occurring in the month. This sum is divided by , and the average parameter consistency for each year was obtained.
[0233] 2) Temporal consistency. Temporal consistency is measured to show the temporal consistency between two data sets. For the two sensors explained above, X and Y measure the received temporal transactions to see if they maintain temporal consistency between the same transactions. Let Defined as an array of received timestamps for sensor X. For sensor X, It can be given as follows:
[0234] , in Indicates The timestamp of each transaction, mr represents the total number of transactions.
[0235] Assumptions represents the temporal consistency of transaction i. To define the temporal consistency of a transaction, compare the timestamp of that transaction with the timestamp of its reference dataset. Use sensor X and sensor Y to check the temporal consistency between them. Therefore For sensor X and sensor Y, it can be defined as:
[0236]
[0237] in represents the sensor y The timestamp of the transaction, is the temporal consistency percentage of transaction i. It can be defined as:
[0238]
[0239] Assumptions represents the average time consistency of the transaction on day j. It can be defined as formula 24:
[0240]
[0241] in is the transaction on day j derived from Equation 4, and the sum is applied to all values of i , i value is equal to 1 to . To calculate the time consistency of all transactions occurring on day j. The sum is divided by , and the average transaction time consistency of day j is obtained.
[0242] Assumptions is the average temporal consistency of transactions in month j. Therefore, It can be defined as formula 25:
[0243]
[0244] in is the number of days in month j when the transaction occurs, and is derived from Equation 24. The summation is applied to all values of i , i value is equal to 1 to To calculate j month The time consistency of all transactions that occurred on the day is the sum of the values divided by , and the average transaction time consistency for each month is obtained.
[0245] Assumptions represents the average time consistency of transactions in j years, so, It can be given by the following formula:
[0246]
[0247] in represents the month number in year j in which the transaction occurs, and is derived from Equation 25. The summation is applied to all values of i , i value is equal to 1 to . To calculate year j The time consistency of all transactions that occurred in the month. The sum is divided by , and the average transaction time consistency per year is obtained.
[0248] In some embodiments, the data quality report can accurately reflect the daily time consistency, monthly time consistency and annual time consistency obtained according to the above-mentioned big data consistency model, as well as the daily parameter consistency, monthly parameter consistency and annual parameter consistency. Of course, according to user needs, the total consistency value of the data set to be evaluated can be output, that is, all time consistency and parameter consistency are added and summed. Among them, according to user needs, the parameters that need to be paid attention to can be flexibly set by setting different coefficients, performing corresponding multiplication operations, and then adding and summing, etc., and finally output the results to the data quality report.
[0249] 7. Big Data Reliability Model
[0250] Reliability is indirectly related to Big Data quality. This is important because data quality may degrade if the system acquiring the data itself is faulty. Assume that station S has sensors S = {S1, S2, Sn} in a time interval, with the help of finding the reliability of the sensors. All other parameters defined in this application are at the sensor level. Big Data reliability is defined at the site level.
[0251] Assumptions is the reliability of site S. To define the reliability of site S with k unreliable sensors and n as the total number of sensors:
[0252]
[0253] Where k is the number of unreliable sensors and n is the total number of sensors.
[0254] is the reliability percentage of site S, which can be defined as:
[0255]
[0256] 8. Big Data Availability Model
[0257] Big data availability can be modeled by simply measuring three different quality parameters such as completeness, accuracy, and timeliness.
[0258] Assumptions is the availability of transaction i, which can be defined as:
[0259]
[0260] is the availability percentage of transaction i. It can be defined as:
[0261]
[0262] Assumptions In terms of transactions, represents the average availability on day j. It can be defined as Formula 26:
[0263]
[0264] in is the transaction on day j derived from Equation 4, and the sum is applied to all values of i , i value is equal to 1 to To calculate the availability of all transactions occurring on day j, this sum is divided by , and the average parameter availability on day j is obtained.
[0265] Assumptions is the average availability of transactions in month j. Therefore, It can be defined as formula 27:
[0266]
[0267] in is the number of days in month j when the event occurs, and is derived from Equation 26. The summation is applied to all values of i , i value is equal to 1 to , to calculate j month The availability of all transactions that occurred on that day. This sum is divided by , and obtain the average monthly transaction availability.
[0268] Assume that 𝑈𝑠𝑎𝑏𝑖l𝑖𝑡𝑦𝑗 represents the average transaction availability in year j, therefore, It can be given by the following formula:
[0269]
[0270] in represents the month number of the transaction occurring in year j, and is derived from Equation 27. The summation is applied to all values of i , i value is equal to 1 to . To calculate year j The availability of all transactions that occurred in the month. This sum is divided by , and the average transaction availability per year is obtained.
[0271] In some embodiments, the data quality report can accurately reflect the daily transaction availability, monthly transaction availability, and annual transaction availability obtained according to the above big data availability model. Of course, the total availability value of the data set to be evaluated can also be output according to user needs, that is, all transaction availabilities are added and summed. Among them, according to user needs, different coefficients can be set for transactions that need attention, and corresponding multiplication operations can be performed and then added and summed, etc., which can be flexibly set, and the results are finally output to the data quality report.
[0272] Step 130: determining a corresponding weight coefficient according to the quality standard score.
[0273] In some embodiments, weight coefficients corresponding to the above eight models are set respectively, which can be flexibly adjusted by the user according to needs.
[0274] Step 140: Determine the data quality score of the data set to be evaluated according to the quality standard score and the corresponding weight coefficient.
[0275] In some embodiments, the completeness score, accuracy score, timeliness score, uniqueness score, validity score, consistency score, reliability score and availability score are respectively multiplied by corresponding weight coefficients to obtain a completeness quality score, an accuracy quality score, a timeliness quality score, a uniqueness quality score, a validity quality score, a consistency quality score, a reliability quality score and an availability quality score as data quality scores of the data set to be evaluated;
[0276] And / or, the sum of the integrity quality score, accuracy quality score, timeliness quality score, uniqueness quality score, validity quality score, consistency quality score, reliability quality score and availability quality score is used as the data quality score of the data set to be evaluated.
[0277] In some embodiments, the quality scores obtained by each of the above models can be output to the data quality report respectively, or the sum of the quality scores obtained by each of the above models and the corresponding weight coefficients can be output to the data quality report.
[0278] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present disclosure is not limited by the order of the actions described, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present disclosure.
[0279] The above is an introduction to the method embodiment. The following is a further explanation of the scheme disclosed in the present invention through an apparatus embodiment.
[0280] Figure 2 FIG. 2 shows a block diagram of a large data quality assessment device 200 according to an embodiment of the present disclosure. Figure 2 As shown, the device 200 includes:
[0281] The data acquisition module 210 is used to acquire a data set to be evaluated and an expected data set; the data set to be evaluated is a plurality of data collected by sensors of a network site; the expected data set is a preset data set corresponding to the network site;
[0282] The quality score determination module 220 is used to input the data set to be evaluated and the expected data set into a pre-built big data quality model, and output the quality standard score corresponding to the data set to be evaluated;
[0283] A weight determination module 230, configured to determine a corresponding weight coefficient according to the quality standard score;
[0284] The quality score calculation module 240 is used to determine the data quality score of the to-be-evaluated data set according to the quality standard score and the corresponding weight coefficient.
[0285] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0286] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.
[0287] Figure 3A schematic block diagram of an electronic device 300 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0288] The electronic device 300 includes a computing unit 301, which can perform various appropriate actions and processes according to a computer program stored in a ROM 302 or a computer program loaded from a storage unit 308 into a RAM 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 can also be stored. The computing unit 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An I / O interface 305 is also connected to the bus 304.
[0289] A number of components in the electronic device 300 are connected to the I / O interface 305, including: an input unit 306, such as a keyboard, a mouse, etc.; an output unit 307, such as various types of displays, speakers, etc.; a storage unit 308, such as a disk, an optical disk, etc.; and a communication unit 309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 309 allows the electronic device 300 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0290] The computing unit 301 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 301 performs the various methods and processes described above, such as the big data quality assessment method. For example, in some embodiments, the big data quality assessment method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 308. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 300 via the ROM 302 and / or the communication unit 309. When the computer program is loaded into the RAM 303 and executed by the computing unit 301, one or more steps of the big data quality assessment method described above may be performed. Alternatively, in other embodiments, the computing unit 301 may be configured to perform the big data quality assessment method in any other appropriate manner (e.g., by means of firmware).
[0291] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0292] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0293] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0294] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0295] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0296] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.
[0297] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0298] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A method for evaluating the quality of big data. It is characterized in that include: Obtain the data set to be evaluated and the expected data set; The data set to be evaluated is a plurality of data collected by sensors at network sites; The expected data set is a plurality of transactions that are expected to be collected by the network site, and parameters corresponding to each transaction; Input the data set to be evaluated and the expected data set into a pre-built big data quality model, and output the quality standard score corresponding to the data set to be evaluated; Determine the corresponding weight coefficient according to the quality standard score; Determining a data quality score of the data set to be evaluated according to the quality standard score and the corresponding weight coefficient; The big data quality model includes: a big data integrity model, a big data accuracy model, a big data timeliness model, a big data uniqueness model, a big data validity model, a big data consistency model, a big data reliability model and a big data availability model; The quality standard scores include: completeness score, accuracy score, timeliness score, uniqueness score, validity score, consistency score, reliability score and availability score; The data set to be evaluated also includes: an actual timestamp consisting of the time of each transaction collected; The expected data set also includes: an expected timestamp consisting of a preset expected arrival time of each transaction, a preset validity rule and a preset total number of sensors of the network site; The step of inputting the to-be-evaluated data set and the expected data set into a pre-built big data quality model and outputting a quality standard score corresponding to the to-be-evaluated data set includes: Inputting the data set to be evaluated and the expected data set into a big data integrity model, and outputting the integrity score of the data set to be evaluated; Inputting the data set to be evaluated and the expected data set into a big data accuracy model, and outputting the accuracy score of the data set to be evaluated; Generate a timestamp array according to the actual timestamp and the expected timestamp; input the timestamp array into a big data timeliness model, and output a timeliness score of the data set to be evaluated; Inputting the data set to be evaluated into a big data uniqueness model, and outputting a uniqueness score of the data set to be evaluated; Inputting the data set to be evaluated and the validity rule into a big data validity model, and outputting a validity score of the data set to be evaluated; Inputting the data set to be evaluated and the actual timestamp into a big data consistency model, and outputting a consistency score of the data set to be evaluated; The number of data sets of the data set to be evaluated is used as the number of sensors that are normally working at the network site; the number of sensors that are normally working and the total number of sensors at the network site are input into a big data reliability model, and a reliability score of the data set to be evaluated is output; Input the completeness score, accuracy score and timeliness score into a big data availability model, and output the availability score of the data set to be evaluated; The big data integrity is a measure of the amount of available data and the amount of data required for the intended use, and is used to verify whether defects in the data will affect its usability; The definition of big data completeness is the ratio of stored data relative to the potential of 100% complete data, taking the number of available values in a given data set and calculating its ratio to the total expected values, measured in percentage; The big data consistency refers to the degree to which the logical relationship between related data is correct and complete, and the data items are measured according to the data items in another data set or their corresponding items. The two data sets have the same value and the same meaning; The consistency of the big data includes the consistency of the value and time of the big data.
2. The method according to claim 1, It is characterized in that The data set to be evaluated includes: multiple transactions collected by sensors of network sites, and parameters corresponding to each transaction, as shown in the following R matrix: Where mr is the total number of transactions collected, nr is the total number of parameters for each transaction; R ij Represents the value of the i-th transaction and the j-th parameter; The expected data set is represented by the following E matrix: Where m represents the total number of expected transactions, n is the total number of parameters for each expected transaction; E ij Represents the value of the i-th transaction and the j-th parameter.
3. The method according to claim 1, It is characterized in that The big data integrity model includes a transaction integrity model and a parameter integrity model; The step of inputting the to-be-evaluated data set and the expected data set into a big data integrity model and outputting the integrity score of the to-be-evaluated data set includes: Input the to-be-evaluated data set and the expected data set into a transaction integrity model, and output an average transaction completion rate; Inputting the data set to be evaluated and the expected data set into a parameter integrity model, and outputting average parameter integrity; The preset weight value is used as a coefficient of the average transaction completion rate and the average parameter integrity, and the sum is calculated to obtain the integrity score of the data set to be evaluated.
4. The method according to claim 3, It is characterized in that The data set to be evaluated and the expected data set are input into the transaction integrity model, and the average transaction completion rate is output, which is achieved by the following formula: Formula 1: M total =E total -R total +Null value ; Formula 2: Formula 3: Formula 4: Formula 5: Formula 6: Formula 7: in, is the completeness of transaction i in the R matrix; M total is the total number of missing data in the R matrix with mr transactions and nr parameters; Null value is the total number of null values in the R matrix; E total is the data set in the E matrix corresponding to transaction i: E i1 ×E in ; R total is the data set in the R matrix corresponding to transaction i: R i1 ×R inr ; is the completion percentage of transaction i; Interval hr is the time difference between two transactions, Total hr is the total number of hours in a day in which these two transactions occur; is the affairs of day j; represents the average transaction completion rate of transaction i on day j; is the number of days in month j when the event occurs; Derived from formula 5; is the average transaction completion rate of transactions in the jth month; represents the number of months in which transactions occurred in year j; represents the average transaction completion rate in year j; Derived from Equation 6.
5. The method according to claim 1, It is characterized in that Determining the data quality score of the to-be-evaluated data set according to the quality standard score and the corresponding weight coefficient includes: Multiplying the completeness score, accuracy score, timeliness score, uniqueness score, validity score, consistency score, reliability score and availability score by corresponding weight coefficients respectively to obtain completeness quality score, accuracy quality score, timeliness quality score, uniqueness quality score, validity quality score, consistency quality score, reliability quality score and availability quality score as data quality scores of the data set to be evaluated; And / or, the sum of the integrity quality score, accuracy quality score, timeliness quality score, uniqueness quality score, validity quality score, consistency quality score, reliability quality score and availability quality score is used as the data quality score of the data set to be evaluated.
6. A big data quality assessment device, It is characterized in that include: A data acquisition module is used to obtain the data set to be evaluated and the expected data set; The data set to be evaluated is a plurality of data collected by sensors at network sites; The expected data set is a plurality of transactions that are expected to be collected by the network site, and parameters corresponding to each transaction; A quality score determination module is used to input the data set to be evaluated and the expected data set into a pre-built big data quality model, and output a quality standard score corresponding to the data set to be evaluated; A weight determination module, used to determine a corresponding weight coefficient according to the quality standard score; A quality score calculation module, used to determine the data quality score of the data set to be evaluated according to the quality standard score and the corresponding weight coefficient; The big data quality model includes: a big data integrity model, a big data accuracy model, a big data timeliness model, a big data uniqueness model, a big data validity model, a big data consistency model, a big data reliability model and a big data availability model; The quality standard scores include: completeness score, accuracy score, timeliness score, uniqueness score, validity score, consistency score, reliability score and availability score; The data set to be evaluated also includes: an actual timestamp consisting of the time of each transaction collected; The expected data set also includes: an expected timestamp consisting of a preset expected arrival time of each transaction, a preset validity rule and a preset total number of sensors of the network site; The step of inputting the to-be-evaluated data set and the expected data set into a pre-built big data quality model and outputting a quality standard score corresponding to the to-be-evaluated data set includes: Inputting the data set to be evaluated and the expected data set into a big data integrity model, and outputting the integrity score of the data set to be evaluated; Inputting the data set to be evaluated and the expected data set into a big data accuracy model, and outputting the accuracy score of the data set to be evaluated; Generate a timestamp array according to the actual timestamp and the expected timestamp; input the timestamp array into a big data timeliness model, and output a timeliness score of the data set to be evaluated; Inputting the data set to be evaluated into a big data uniqueness model, and outputting a uniqueness score of the data set to be evaluated; Inputting the data set to be evaluated and the validity rule into a big data validity model, and outputting a validity score of the data set to be evaluated; Inputting the data set to be evaluated and the actual timestamp into a big data consistency model, and outputting a consistency score of the data set to be evaluated; The number of data sets of the data set to be evaluated is used as the number of sensors that are normally working at the network site; the number of sensors that are normally working and the total number of sensors at the network site are input into a big data reliability model, and a reliability score of the data set to be evaluated is output; Input the completeness score, accuracy score and timeliness score into a big data availability model, and output the availability score of the data set to be evaluated; The big data integrity is the balance between the amount of available data and the amount of data required for the intended use. Measurement, used to verify whether defects in the data affect its usability; The definition of big data completeness is the ratio of stored data relative to the potential of 100% complete data, taking the number of available values in a given data set and calculating its ratio to the total expected values, measured in percentage; The big data consistency refers to the degree to which the logical relationship between related data is correct and complete, and the data items are measured according to the data items in another data set or their corresponding items. The two data sets have the same value and the same meaning; The consistency of the big data includes the consistency of the value and time of the big data.
7. An electronic device, It is characterized in that include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 5.
8. A non-transitory computer-readable storage medium storing computer instructions, It is characterized in that The computer instructions are used to cause the computer to execute the method according to any one of claims 1-5.
