Data quality evaluation method and system based on real-sequence database

Abstract

The invention discloses a data quality evaluation method and system based on a real-sequence database, and the method comprises the steps: collecting factory bit number time sequence data from at least one data source, and storing the time sequence data in a time sequence data storage layer; based on preset multi-dimensional data quality evaluation indexes, the time series data are evaluated, a corresponding quality evaluation result is generated, and the multi-dimensional data quality evaluation indexes at least comprise data integrity, data consistency, data accuracy, data credibility and data stability; according to the quality evaluation result, a factory data quality space is constructed and maintained, and the data quality space is used for centralized management and visual display of the data quality state of the factory position number; and providing an access interface for the data quality space for a user through the client. A statistical analysis technology is utilized to research factory data quality, and a user is helped to better understand and solve a data quality problem.

Description

Technical Field

[0001] This invention belongs to the field of time-series database technology, and in particular relates to a data quality evaluation method and system based on real-series databases. Background Technology

[0002] The process manufacturing industry generates massive amounts of diverse and heterogeneous data during production. The inherent complexity of the processes and the variability of the environment bring about problems such as difficulties in integrating heterogeneous data, unstable data quality, and difficulty in processing large amounts of data, thus posing challenges to the evaluation, analysis, and assessment of data quality.

[0003] Currently, commonly used time-series databases in the industrial sector include open-source types (InfluxDB, OpenTSDB, etc.) and closed-source types (ISYS, PI, InfoPlus21, PHD, etc.). All of them use quality codes to determine data quality, and each time-series database has a different definition of the quality code. At present, time-series database products lack standardized definitions and comprehensive evaluation systems for factory data quality, making it impossible to effectively evaluate and assess data quality. Summary of the Invention

[0004] To address the aforementioned issues, the present invention aims to provide a data quality evaluation method and system based on a real-sequence database. Utilizing statistical analysis techniques, it offers a set of methods for evaluating the quality of factory tag data, researching the quantitative assessment and analysis of factory data quality, and helping users better understand and resolve data quality problems.

[0005] The technical solution provided by this invention is: a data quality evaluation method based on real-series databases, applied to a distributed system including a client, a data quality spatial layer, and a time-series data storage layer, comprising the following steps: Factory location time-series data are collected from at least one data source, and the time-series data is stored in a time-series data storage layer; The time-series data is evaluated based on preset multi-dimensional data quality evaluation indicators to generate corresponding quality evaluation results. The multi-dimensional data quality evaluation indicators include at least data integrity, data consistency, data accuracy, data reliability, and data stability. Based on the quality evaluation results, a factory data quality space is constructed and maintained. The data quality space is used for centralized management and visual display of the data quality status of the factory tag number. The client provides users with an access interface to the data quality space.

[0006] Preferably, the evaluation of the stability of the data includes at least one of the following methods: Maximum-minimum difference method: The stability is judged by comparing the absolute value of the difference between the maximum and minimum values ​​of time series data within a preset time period with a preset comparison value. Statistical methods: Calculate the variance or standard deviation of the time series data within the preset time period, and determine the stability based on the calculation results; Percentage measurement method: Calculate the percentage of the difference between the maximum and minimum values ​​of the time series data within the preset time period relative to the average value of the data within that time period, and compare it with a preset percentage threshold to determine stability.

[0007] Preferably, the stability determination further includes comparing the absolute value of the difference between the maximum and minimum values ​​of the time-series data within a preset time period with a preset comparison value. According to the calculation formula V max -V min ≤σ, where: V max For the maximum value of the parameter during this period, V min The minimum value of the parameter is given, and σ is a fixed value.

[0008] Preferably, according to the calculation formula n*∑V 2 -(∑V) 2 ≤(n*σ) 2 , where n represents the total number of observations, V represents a random variable, and σ represents the population standard deviation.

[0009] Preferably, according to the calculation formula 100*(V) max -V min ) / V avg ≤μ,V max V represents the upper bound of the data. min V represents the lower bound of the data. avg This represents the arithmetic mean of all observations.

[0010] Preferably, the data quality space stores and associates the following information: tag identifier, original time-series data, quality evaluation results for each dimension, quality evaluation timestamp, and data source information.

[0011] Preferably, the method further includes: Based on the quality evaluation results, a data quality report or alarm information is generated and pushed to the user through the client.

[0012] Preferably, the time-series data storage layer supports multiple time-series databases, including at least one of InfluxDB, OpenTSDB, ISYS, PI, InfoPlus21, and PHD, and the method is compatible with the quality code definition differences of different time-series databases.

[0013] Based on the same concept, the present invention also provides a data quality evaluation system based on a real-sequence database, comprising: The data acquisition module is used to acquire time-series data of factory location numbers from at least one data source, and the time-series data is stored in the time-series data storage layer; The quality evaluation module is used to evaluate the time series data based on preset multi-dimensional data quality evaluation indicators and generate corresponding quality evaluation results. The multi-dimensional data quality evaluation indicators include at least data integrity, data consistency, data accuracy, data reliability, and data stability. The quality space construction module is used to construct and maintain a factory data quality space based on the quality evaluation results. The data quality space is used for centralized management and visual display of the data quality status of the factory tag number. The client interface module is used to provide users with an access interface to the data quality space through the client.

[0014] Based on the same concept, the present invention also provides an electronic device, characterized in that it comprises: The memory is used to store the processing program; A processor, which, when executing the processing program, implements any one of the above-described data quality evaluation methods based on a real-sequence database.

[0015] Based on the same concept, the present invention also provides a readable storage medium storing a processing program, which, when executed by a processor, implements the data quality evaluation method based on a real-sequence database as described above.

[0016] Because the present invention adopts the above technical solution, it has the following advantages and positive effects compared with the prior art: By systematically and quantitatively evaluating factory tag number time-series data through pre-set multi-dimensional evaluation indicators and constructing a centrally managed "data quality space," this solves the problem that existing technologies rely on single, poorly defined "quality codes," making it impossible to effectively evaluate and assess data quality. The result is the establishment of a standardized and unified factory data quality evaluation system, integrating and visualizing the dispersed and heterogeneous data quality status. This helps users comprehensively and intuitively understand and monitor data quality, providing a reliable basis for data governance and decision-making. Attached Figure Description

[0017] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings, wherein: Figure 1 A diagram illustrating aspects of data governance; Figure 2This is a dimensional diagram of the data quality evaluation system based on a real-sequence database according to the present invention; Figure 3 This is a schematic diagram illustrating the process of the data quality evaluation method based on real-sequence database of the present invention. Detailed Implementation

[0018] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The advantages and features of the present invention will become clearer from the following description and claims. It should be noted that the drawings are all in a very simplified form and use non-precise ratios, and are only used to facilitate and clarify the illustration of the embodiments of the present invention.

[0019] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indication will also change accordingly.

[0020] The challenge of building a data warehouse lies in data governance as business grows and business lines expand. Data governance encompasses a wide range of aspects, including data management, data security, data quality, and data costs. Among these, data quality governance is paramount, as it forms the foundation for the validity and accuracy of data analysis conclusions and is a prerequisite for everything. Therefore, ensuring data quality and availability is an indispensable part of data warehouse construction. Data governance, such as… Figure 1 As shown.

[0021] Data quality involves a wide range of aspects, spanning the entire lifecycle of the data space, from data generation -> data access -> data storage -> data processing -> data output -> data display, and quality governance is required at each stage.

[0022] There are different standards for evaluating data quality, such as... Figure 2 This invention is evaluated from the following six dimensions: completeness, standardization, consistency, accuracy, uniqueness, and timeliness.

[0023] Data integrity refers to whether there are any missing data information. Data missing may be due to the entire data record being missing, or it may be due to the missing information in a certain field of the data.

[0024] Data standardization refers to the degree to which data follows predetermined grammatical rules and conforms to its definition, such as data type, format, and value range.

[0025] Data consistency refers to whether data follows a unified standard and whether the dataset maintains a consistent format. Data quality consistency is mainly reflected in the standardization of data records and whether the data is logically sound. Consistency does not mean absolute numerical identicalness, but rather consistency in the methods and standards used for data collection and processing. Common consistency indicators include: ID overlap, attribute consistency, value consistency, data collection method consistency, and transformation step consistency.

[0026] Data accuracy refers to whether the information recorded in the data is abnormal or erroneous. Unlike consistency, data with accuracy problems is not just inconsistent with the rules; more common data accuracy errors include garbled text, and abnormally large or small data points also indicate data that does not meet the criteria. Common accuracy indicators include: percentage of missing values, percentage of erroneous values, percentage of outliers, sampling bias, and data noise.

[0027] Data uniqueness refers to the absence of duplicate data in a database. For example, if there are 10,000 actual transactions, but 3,000 of them are duplicates in the data table, resulting in 13,000 transaction records, this does not meet the requirement of data uniqueness.

[0028] Data timeliness refers to the time interval between data generation and its availability for viewing; it is also called data latency. For example, if a dataset is used to calculate offline data for today, and the results are not available until the next day or even the third day, this data does not meet the timeliness requirement.

[0029] First Embodiment like Figure 3 As shown, a data quality evaluation method based on real-series databases is applied to a distributed system comprising a client, a data quality spatial layer, and a time-series data storage layer, and includes the following steps: Factory location time-series data are collected from at least one data source, and the time-series data is stored in a time-series data storage layer; The time-series data is evaluated based on preset multi-dimensional data quality evaluation indicators to generate corresponding quality evaluation results. The multi-dimensional data quality evaluation indicators include at least data integrity, data consistency, data accuracy, data reliability, and data stability. Based on the quality evaluation results, a factory data quality space is constructed and maintained. The data quality space is used for centralized management and visual display of the data quality status of the factory tag number. The client provides users with an access interface to the data quality space.

[0030] This solution systematically and quantitatively evaluates factory tag number time-series data through pre-defined multi-dimensional evaluation indicators and constructs a centrally managed "data quality space." This solves the problem that existing technologies rely on single, poorly defined "quality codes," making it impossible to effectively evaluate and assess data quality. The result is the establishment of a standardized and unified factory data quality evaluation system, integrating and visualizing scattered and heterogeneous data quality status. This helps users comprehensively and intuitively understand and monitor data quality, providing a reliable basis for data governance and decision-making.

[0031] Data quality assessment is a crucial process for ensuring data reliability and validity, involving comprehensive consideration of multiple dimensions.

[0032] Data integrity assessment: Identify missing values ​​through statistical summaries or programming scripts, and check field-level integrity (such as the completeness of units, descriptions, and ranges of measurement data). Formula: $X = \frac{A}{B}$ (number of elements assigned / expected number of elements assigned).

[0033] Data consistency verification: Check the consistency of multi-source data models (such as measurement point names, units, etc.) and entities (such as DCS, etc.) to ensure that logical relationships are correct. For example, ensure that the data queried and displayed in the distributed industrial time series database is consistent with the data from the underlying data source (such as DCS, PLC, etc.).

[0034] Data accuracy verification: Ensure that the provided tag data type can represent the precision range of all production data and will not cause data overflow due to exceeding the data precision range.

[0035] Data credibility analysis: Based on data quality code attributes, correctly determine the credibility of the data.

[0036] Data stability test: Data stability is an indicator that measures the volatility and dispersion of data. The smaller the data volatility and the smaller the dispersion, the higher the stability.

[0037] Preferably, the evaluation of the stability of the data includes at least one of the following methods: Maximum-minimum difference method: The stability is judged by comparing the absolute value of the difference between the maximum and minimum values ​​of time series data within a preset time period with a preset comparison value. Statistical methods: Calculate the variance or standard deviation of the time series data within the preset time period, and determine the stability based on the calculation results; Percentage measurement method: Calculate the percentage of the difference between the maximum and minimum values ​​of the time series data within the preset time period relative to the average value of the data within that time period, and compare it with a preset percentage threshold to determine stability.

[0038] This preferred solution provides multiple stability assessment methods, including the extreme value difference method, statistical methods, and percentage measurement methods. Its effect is to enhance the flexibility and adaptability of the evaluation system. Users can select the most suitable stability assessment algorithm based on specific business scenarios, data characteristics, and different requirements for computational accuracy and efficiency, achieving a balance between computational complexity and evaluation accuracy, and meeting the diverse needs of data volatility analysis in industrial production.

[0039] The extreme value difference method assesses stability by finding the maximum and minimum values ​​of a parameter within a fixed time period and comparing the absolute value of their difference. Empirical averages or similar metrics can be used as a reference. This method is the simplest, but considering only the maximum and minimum values ​​can lead to significant errors. Statistical methods utilize mathematical indicators such as variance or standard deviation to compare the variance and mean squared error of the parameter data, thus analyzing its stability range. While straightforward, this method becomes computationally burdensome when dealing with large values ​​or a large number of values. The percentage method assesses stability by analyzing the percentage of the difference between the maximum and minimum values ​​relative to the parameter's mean. While similar in approach and simplicity, it can also have significant errors.

[0040] Preferably, the stability determination further includes comparing the absolute value of the difference between the maximum and minimum values ​​of the time-series data within a preset time period with a preset comparison value. According to the calculation formula V max -V min ≤σ, where: V max For the maximum value of the parameter during this period, V min The minimum value of the parameter is given, and σ is a fixed value.

[0041] This solution provides a specific calculation formula V. max -V min The value ≤σ establishes a clear and quantifiable judgment rule for the "maximum-minimum difference method". Its effect is to transform the judgment of stability from a qualitative description to an objective numerical comparison, making the evaluation process standardized and repeatable, easy to automate in the system, and improving the evaluation efficiency and consistency. It is especially suitable for scenarios with high requirements for computational efficiency and the need to quickly judge abnormal data fluctuations.

[0042] Preferably, according to the calculation formula n*∑V 2 -(∑V) 2 ≤(n*σ) 2 , where n represents the total number of observations, V represents a random variable, and σ represents the population standard deviation.

[0043] This solution provides a specific calculation formula based on variance / standard deviation: n*∑V2 -(∑V) 2 ≤(n*σ) 2 This provides a rigorous mathematical definition for "statistical methods." Its advantage lies in utilizing complete sample statistical information for stability assessment, avoiding the errors that may arise from relying solely on extreme values. This makes the evaluation results more statistically significant and reliable, suitable for production processes that require high precision in handling data fluctuations and in-depth statistical analysis.

[0044] Preferably, according to the calculation formula 100*(V) max -V min ) / V avg ≤μ,V max V represents the upper bound of the data. min V represents the lower bound of the data. avg This represents the arithmetic mean of all observations.

[0045] This solution provides a specific percentage calculation formula: 100*(V) max -V min ) / V avg ≤μ sets a clear threshold for the "percentage measurement method". Its effect is to correlate the data fluctuation range with its average level for relative evaluation, making the stability judgment results more comparable, adaptable to index data of different dimensions or benchmark levels, and providing an intuitive and easy-to-understand stability measurement method.

[0046] Preferably, the data quality space stores and associates the following information: tag identifier, original time-series data, quality evaluation results for each dimension, quality evaluation timestamp, and data source information.

[0047] This solution clearly defines the core information elements associated with the data quality space, including tag identifiers, raw data, evaluation results, timestamps, and data sources. Its effect is to construct a structured "digital twin" view of data quality, ensuring that quality evaluation results are traceable and auditable. By closely linking multi-dimensional evaluation results with the raw data and its sources, it greatly facilitates the root cause identification and accountability for data quality problems.

[0048] Preferably, the method further includes: Based on the quality evaluation results, a data quality report or alarm information is generated and pushed to the user through the client.

[0049] This solution adds a step to automatically generate reports and alerts based on evaluation results. Its effect is to transform static data quality assessment into a dynamic, actionable data governance-driven mechanism. It proactively and promptly pushes quality insights (reports) and anomalies (alerts) to users, shifting from passive inspection to proactive monitoring, significantly improving the timeliness and effectiveness of data quality management, and supporting preventative maintenance and rapid response.

[0050] Preferably, the time-series data storage layer supports multiple time-series databases, including at least one of InfluxDB, OpenTSDB, ISYS, PI, InfoPlus21, and PHD, and the method is compatible with the quality code definition differences of different time-series databases.

[0051] This solution explicitly supports and is compatible with various open-source and commercial time-series databases, including InfluxDB, OpenTSDB, ISYS, and PI. Its effectiveness lies in overcoming the data quality assessment barriers caused by the proprietary and inconsistent quality code definitions of different database products, achieving unified coverage of heterogeneous time-series data environments with a single methodology. This provides industrial enterprises with hybrid IT architectures and multiple data sources with cross-platform, integrated data quality assessment capabilities, promoting global data quality governance.

[0052] Second Embodiment Based on the same concept, the present invention also provides a data quality evaluation system based on a real-sequence database, comprising: The data acquisition module is used to acquire time-series data of factory location numbers from at least one data source, and the time-series data is stored in the time-series data storage layer; The quality evaluation module is used to evaluate the time series data based on preset multi-dimensional data quality evaluation indicators and generate corresponding quality evaluation results. The multi-dimensional data quality evaluation indicators include at least data integrity, data consistency, data accuracy, data reliability, and data stability. The quality space construction module is used to construct and maintain a factory data quality space based on the quality evaluation results. The data quality space is used for centralized management and visual display of the data quality status of the factory tag number. The client interface module is used to provide users with an access interface to the data quality space through the client.

[0053] By systematically and quantitatively evaluating factory tag number time-series data through pre-set multi-dimensional evaluation indicators and constructing a centrally managed "data quality space," this invention solves the problem of existing technologies relying on single, poorly defined "quality codes" that cannot effectively evaluate and assess data quality. Its effect lies in establishing a standardized and unified factory data quality evaluation system, integrating and visualizing scattered and heterogeneous data quality status, helping users to comprehensively and intuitively understand and monitor data quality, and providing a reliable basis for data governance and decision-making. The system construction of the factory data quality space in this invention facilitates users' better understanding and resolution of data quality issues.

[0054] Based on the same concept, the present invention also provides an electronic device, comprising: a memory for storing a processing program; and a processor, wherein the processor, when executing the processing program, implements the data quality evaluation method based on a real-sequence database as described above.

[0055] Based on the same concept, the present invention also provides a readable storage medium storing a processing program, which, when executed by a processor, implements the data quality evaluation method based on a real-sequence database as described above.

[0056] If a data quality evaluation method based on a real-sequence database is implemented as a program instruction and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in software. This computer software is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0057] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific identification content executed by the system and device described above can be referred to the corresponding process in the foregoing method embodiments.

[0058] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the above embodiments. Even if various changes are made to the present invention, if these changes fall within the scope of the claims of the present invention and their equivalents, they shall still fall within the protection scope of the present invention.

Claims

1. A data quality evaluation method based on real-series databases, applied to a distributed system comprising a client, a data quality spatial layer, and a time-series data storage layer, characterized in that, Includes the following steps: Factory location time-series data are collected from at least one data source, and the time-series data is stored in a time-series data storage layer; The time-series data is evaluated based on preset multi-dimensional data quality evaluation indicators to generate corresponding quality evaluation results. The multi-dimensional data quality evaluation indicators include at least data integrity, data consistency, data accuracy, data reliability, and data stability. Based on the quality evaluation results, a factory data quality space is constructed and maintained. The data quality space is used for centralized management and visual display of the data quality status of the factory tag number. The client provides users with an access interface to the data quality space.

2. The data quality evaluation method based on a real-sequence database according to claim 1, characterized in that, The evaluation of the stability of the data includes at least one of the following methods: Maximum-minimum difference method: The stability is judged by comparing the absolute value of the difference between the maximum and minimum values ​​of time series data within a preset time period with a preset comparison value. Statistical methods: Calculate the variance or standard deviation of the time series data within the preset time period, and determine the stability based on the calculation results; Percentage measurement method: Calculate the percentage of the difference between the maximum and minimum values ​​of the time series data within the preset time period relative to the average value of the data within that time period, and compare it with a preset percentage threshold to determine stability.

3. The data quality evaluation method based on a real-sequence database according to claim 2, characterized in that, The stability is determined by comparing the absolute value of the difference between the maximum and minimum values ​​of time-series data within a preset time period with a preset comparison value, which further includes: According to the calculation formula V max -V min ≤σ, where: V max For the maximum value of the parameter during this period, V min The minimum value of the parameter is given, and σ is a fixed value.

4. The data quality evaluation method based on a real-sequence database according to claim 2, characterized in that, According to the calculation formula n*∑V 2 -(∑V) 2 ≤(n*σ) 2 , where n represents the total number of observations, V represents a random variable, and σ represents the population standard deviation.

5. The data quality evaluation method based on a real-sequence database according to claim 2, characterized in that, According to the calculation formula 100*(V) max -V min ) / V avg ≤μ,V max V represents the upper bound of the data. min V represents the lower bound of the data. avg This represents the arithmetic mean of all observations.

6. The data quality evaluation method based on a real-sequence database according to claim 1, characterized in that, The data quality space stores and associates the following information: tag identifier, original time-series data, quality evaluation results for each dimension, quality evaluation timestamp, and data source information.

7. The data quality evaluation method based on a real-sequence database according to claim 1, characterized in that, The method further includes: Based on the quality evaluation results, a data quality report or alarm information is generated and pushed to the user through the client.

8. The data quality evaluation method based on a real-sequence database according to claim 1, characterized in that, The time-series data storage layer supports multiple time-series databases, including at least one of InfluxDB, OpenTSDB, ISYS, PI, InfoPlus21, and PHD, and the method is compatible with the quality code definition differences of different time-series databases.

9. A data quality evaluation system based on a real-sequence database, characterized in that, include: The data acquisition module is used to acquire time-series data of factory location numbers from at least one data source, and the time-series data is stored in the time-series data storage layer; The quality evaluation module is used to evaluate the time series data based on preset multi-dimensional data quality evaluation indicators and generate corresponding quality evaluation results. The multi-dimensional data quality evaluation indicators include at least data integrity, data consistency, data accuracy, data reliability, and data stability. The quality space construction module is used to construct and maintain a factory data quality space based on the quality evaluation results. The data quality space is used for centralized management and visual display of the data quality status of the factory tag number. The client interface module is used to provide users with an access interface to the data quality space through the client.

10. An electronic device, characterized in that, include: The memory is used to store the processing program; A processor, which, when executing the processing program, implements the data quality evaluation method based on a real-sequence database as described in any one of claims 1 to 8.

Images