Data quality evaluation method and system based on big data analysis

Abstract

The invention provides a data quality evaluation method and system based on big data analysis. The method comprises the following steps: receiving real-time data from different data sources, the real-time data comprising structured data and unstructured data; performing significance analysis processing on the unstructured data to obtain a business context of the real-time data; determining the integrity and consistency of the real-time data through an intelligent algorithm to obtain an integrity and consistency evaluation result of the real-time data; based on the evaluation result of the logical relationship, combining the integrity and consistency evaluation results to construct a data quality evaluation model, the data quality evaluation model being used for quantifying the accuracy and timeliness of the real-time data; generating a data quality report, and proposing service improvement measures according to the data quality report; and optimizing the data acquisition and processing flow according to business improvement measures. According to the technical scheme provided by the invention, the flexibility and efficiency of data quality evaluation can be improved, and the real-time data analysis demand of an enterprise can be better supported.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data analysis technology, and in particular, to a data quality assessment method and system based on big data analysis. Background Art

[0002] In today's big data era, enterprises are faced with the challenge of processing massive amounts of data. These data not only come from internal systems, but may also include multiple data sources such as external partners, social media, and IoT devices. In order to ensure the effectiveness and accuracy of business decisions, enterprises need to conduct real-time quality assessments on the collected data. This quality assessment should not only cover structured data, such as database records, but also be able to parse unstructured data, such as text, images, and videos, to understand the business meaning behind them. In addition, as the business environment changes, data validation rules also need to have the ability to dynamically adjust to adapt to the changing data characteristics.

[0003] There are many methods and technical tools for data quality assessment on the market, which usually provide preset data verification rules to check the integrity, consistency and timeliness of data. Some advanced solutions have begun to use machine learning algorithms to automatically detect outliers or pattern deviations, and try to understand and classify unstructured data through natural language processing (NLP) technology.

[0004] Although existing data quality assessment tools are quite mature, they still have some limitations in practical applications. First, most traditional methods rely on statically defined data quality rules, which makes it difficult for them to cope with rapidly changing data types and business logic. Second, when processing unstructured data, existing technologies can often only achieve surface-level understanding and lack in-depth semantic analysis capabilities, which affects the overall effect of data quality assessment. Finally, although some systems have begun to introduce intelligent algorithms to improve the degree of automation, these algorithms are not accurate enough for the identification and processing of complex logical relationships, resulting in the final generated data quality evaluation model may not be comprehensive or accurate enough. Summary of the invention

[0005] The embodiments of the present application provide a data quality assessment method and system based on big data analysis to solve the problems of low data quality and availability in the prior art.

[0006] In a first aspect, an embodiment of the present application provides a data quality assessment method based on big data analysis, comprising: Receiving real-time data from different data sources, wherein the real-time data includes structured data and unstructured data; Performing meaning analysis on the unstructured data to obtain the business context of the real-time data; According to the business context, the data validation rules are dynamically adjusted, and the integrity and consistency of the real-time data are determined by an intelligent algorithm to obtain the integrity and consistency evaluation results of the real-time data; Evaluate the logical relationship between the real-time data, and build a data quality evaluation model based on the evaluation results of the logical relationship and the integrity and consistency evaluation results, wherein the data quality evaluation model is used to quantify the accuracy and timeliness of the real-time data; A data quality report is generated based on the data quality evaluation model, and business improvement measures are proposed based on the data quality report.

[0007] In a second aspect, an embodiment of the present application provides a data quality assessment system based on big data analysis, comprising: A receiving module, used for receiving real-time data from different data sources, wherein the real-time data includes structured data and unstructured data; A parsing module, used to perform meaning parsing on the unstructured data to obtain the business context of the real-time data; An adjustment module is used to dynamically adjust the data validation rules according to the business context, and determine the integrity and consistency of the real-time data through an intelligent algorithm to obtain the integrity and consistency evaluation results of the real-time data; A construction module is used to evaluate the logical relationship between the real-time data, and based on the evaluation results of the logical relationship, in combination with the integrity and consistency evaluation results, to construct a data quality evaluation model, wherein the data quality evaluation model is used to quantify the accuracy and timeliness of the real-time data; The generating module is used to generate a data quality report according to the data quality evaluation model, and propose business improvement measures according to the data quality report.

[0008] In a third aspect, an embodiment of the present application provides a computing device, comprising a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement a data quality assessment method based on big data analysis as described in any one of the first aspects above.

[0009] In a fourth aspect, an embodiment of the present application provides a computer storage medium storing a computer program. When the computer program is executed by a computer, it implements a data quality assessment method based on big data analysis as described in any one of the first aspects above.

[0010] In an embodiment of the present application, real-time data from different data sources is received, and the real-time data includes structured data and unstructured data; the unstructured data is processed for meaning parsing to obtain the business context of the real-time data; the integrity and consistency of the real-time data are determined and processed by an intelligent algorithm to obtain the integrity and consistency evaluation results of the real-time data; based on the evaluation results of logical relationships, a data quality evaluation model is constructed in combination with the integrity and consistency evaluation results, and the data quality evaluation model is used to quantify the accuracy and timeliness of real-time data; a data quality report is generated, and business improvement measures are proposed based on the data quality report; based on the business improvement measures, the data collection and processing process is optimized. The technical solution provided by this application can improve the flexibility and efficiency of data quality evaluation, and can better support the real-time data analysis needs of enterprises.

[0011] These and other aspects of the present application will become more clearly understood in the description of the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0013] Figure 1 A flowchart of a data quality assessment method based on big data analysis provided in an embodiment of the present application; Figure 2 A schematic diagram of the structure of a data quality assessment system based on big data analysis provided in an embodiment of the present application; Figure 3 A schematic diagram of the structure of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0014] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0015] In some of the processes described in the specification and claims of this application and the above-mentioned figures, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0016] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0017] Figure 1 A flowchart of a data quality assessment method based on big data analysis is provided for an embodiment of the present application, such as Figure 1 As shown, the method includes: 101. Receive real-time data from different data sources, where the real-time data includes structured data and unstructured data; In this step, real-time data is collected from various data sources (such as sensors, devices, applications, etc.), which can be structured (such as database records) or unstructured (such as documents, pictures, audio, etc.).

[0018] Taking the power grid as an example, real-time data refers to the instant data generated during the operation of the power system, which reflects the current status of the power grid. For example, the voltage and current values ​​of the substation, the output power of the generator set, the status information of the transmission line, etc. These data are crucial for the monitoring, fault detection and rapid response of the power system.

[0019] In the power grid, structured data usually includes power grid operation parameters, equipment status information, power transaction data, etc. For example, the voltage, current, active power, reactive power and other values ​​reported by the substation every minute, as well as the operation instructions issued by the power grid dispatching center. This type of data is usually stored in a relational database for easy query and analysis.

[0020] Unstructured data includes maintenance records of power equipment, photos and videos of on-site inspections, and reports of power grid failures. Although these data are in various forms, they also contain a lot of valuable information that can be used for equipment status assessment, fault analysis, and preventive maintenance.

[0021] In an embodiment of the present application, in a power grid monitoring system, structured data such as voltage, current, and temperature are collected from multiple locations such as substations, power plants, and user terminals, and unstructured data such as photos and videos of on-site inspections are collected to predict power demand.

[0022] 102. The unstructured data is processed for meaning analysis to obtain the business context of the real-time data; In this step, business context refers to combining the information parsed from unstructured data with actual application scenarios based on specific business needs to provide support for decision-making. For example, in power grid maintenance, business context involves equipment health status assessment, fault location, and preventive maintenance plan formulation.

[0023] In the embodiment of the present application, first, it is necessary to collect unstructured data from different channels, such as social media platforms, customer service hotline recordings, equipment photos taken by inspectors, etc.

[0024] Secondly, the collected data is preliminarily processed, including but not limited to text cleaning (removing irrelevant characters, punctuation marks, etc.), image denoising, audio-to-text conversion, etc., to improve the efficiency and accuracy of subsequent processing.

[0025] Furthermore, natural language processing technology is used to extract keywords and topics from text data; computer vision technology is used to analyze information such as objects and damage levels in images. This step aims to convert unstructured raw information into quantifiable feature vectors. Based on the extracted features, supervised learning or unsupervised learning methods are used to train the model. For example, a classifier can be trained with historical data to automatically identify the main problem categories in customer complaints; or a regression model can be trained to predict the changing trend of power grid load under specific conditions.

[0026] Finally, the trained model is applied to actual scenarios to assist power grid operators in making more accurate judgments and decisions. At the same time, the algorithm parameters are continuously adjusted and optimized according to the performance of the model in actual applications to improve the accuracy and practicality of the analysis.

[0027] Through the above process, power grid companies can not only monitor and maintain the power system more effectively, but also improve customer service quality and satisfaction and realize intelligent management.

[0028] Furthermore, the embodiment of the present invention proposes a more sophisticated meaning parsing solution, which aims to obtain more accurate and rich business context through in-depth analysis of unstructured data.

[0029] The options are as follows: Optionally, the "performing meaning analysis processing on the unstructured data to obtain the business context of the real-time data" in step 102 includes: converting the words in the unstructured data into vector representations to capture the semantic relationship between words; identifying entity information from the vectorized unstructured data; parsing the sentence structure to determine the grammatical dependencies between words to understand the semantic structure within the sentence; performing context information processing on the unstructured data to capture context information at the sentence and paragraph levels; evaluating the emotional tendencies expressed in the unstructured data, identifying positive, negative or neutral emotions, and obtaining sentiment analysis results; extracting the main discussion topics from the unstructured data; and integrating the entity information, the semantic structure, the context information, the sentiment analysis results and the topics to generate the business context in the unstructured data and use it as the business context of the real-time data.

[0030] In this step, entity information refers to specific things or concepts mentioned in the document, such as names of people, places, and organizations. Grammatical dependencies refer to linguistic features that describe how words interact with each other within a sentence, helping to understand sentence structure. Contextual information refers to information surrounding a specific element, which is crucial for correctly understanding that element. Sentiment analysis results refer to the process of determining whether the author's attitude is positive, negative, or neutral based on the content of the text. Topic extraction refers to automatically discovering and summarizing the main topics or focuses of discussion from a large number of documents.

[0031] In the embodiment of the present application, a pre-trained language model (such as Word2Vec, BERT) is first used to convert the words in the text into a vector of fixed length, so that the semantic connection between words can be better captured. Secondly, named entity recognition (NER) technology is used to extract key entity information from the text to help quickly locate important information points. The internal structure of the sentence is further analyzed by means of dependency syntactic analysis, and the relationship between the components is clarified, thereby deepening the understanding of the meaning of the sentence. Then a long short-term memory network (LSTM) or a Transformer architecture is used to process longer sequences of text in order to capture information flow across sentences and even paragraphs. Then, the text is classified with the help of a deep learning model (such as CNN, RNN) to distinguish between positive, negative or neutral attitudes, which is particularly important for understanding the public's views on an event. Secondly, a topic model (such as LDA) is used to identify the core topics in the text, which helps to extract the main discussion direction of the text. Finally, all the above information is integrated to form a detailed business context description to provide comprehensive and in-depth data support for decision makers.

[0032] Suppose a power grid company receives a complaint about a power outage on social media: "My home suddenly had a power outage today, and it has lasted for 3 hours. I am very dissatisfied!" First, keywords such as "power outage" and "3 hours" are converted into vectors, and "my home" is identified as the user location and "3 hours" is identified as the time length.

[0033] Secondly, determine that "power outage" is the subject and "has lasted for 3 hours" is the predicate, which expresses the time span of the power outage.

[0034] Furthermore, combined with the user's historical electricity consumption records and the power grid conditions in the area, it is inferred that the power outage may have been caused by power grid maintenance. By analyzing the words and tone in the text, it is determined that the user's emotions are relatively negative. Hot issues such as "sudden power outage" and "slow service response" are extracted from similar complaints.

[0035] Finally, based on the above information, the following business context is generated: "A user in District XX had a power outage for more than 3 hours due to power grid maintenance, and the user was very dissatisfied. This situation reflects the company's inadequate emergency response mechanism in sudden power outages." Generating business context through the above solution can help power grid companies quickly locate problems, take timely measures to solve problems, and improve future service processes.

[0036] 103. Dynamically adjust the data validation rules according to the business context, and determine the integrity and consistency of the real-time data through an intelligent algorithm to obtain the integrity and consistency evaluation result of the real-time data; In this step, data validation rules refer to a series of pre-set inspection standards and conditions to ensure the quality of data. In the field of power grids, these rules may include whether the timestamp of the data is reasonable, whether the value is outside the normal range, whether the geographic location information is accurate, etc. Integrity refers to whether the data is complete and no important parts are missing. In the power grid, integrity means that all necessary measurements and status information have been collected. Consistency refers to whether there are contradictions between data and whether they can verify each other. In the power grid, consistency requires that data from different sources be logically consistent, such as the power flow between two adjacent substations should match.

[0037] In the embodiment of the present application, first, based on the business context generated in the previous steps, understand the challenges or opportunities currently facing the power grid, such as large-scale power outages caused by extreme weather. According to the current business context, adjust the data verification rule base. For example, if it is found that thunderstorm weather affects the accuracy of certain sensors, temporarily relax the verification standards for these sensor data, and increase the monitoring requirements for other key indicators (such as the status of backup power supply).

[0038] Secondly, conduct integrity checks, using machine learning algorithms to automatically detect whether there are missing values ​​or outliers in the data stream, especially for those data points that are particularly important in the current business context; conduct consistency checks, using graph databases or other association analysis technologies to check whether the logical relationships between different data sources are correct, such as confirming the consistency of current flow in the power grid topology.

[0039] Finally, the integrity check results and consistency check results obtained by the intelligent algorithm are summarized to form a final evaluation report. This report not only points out which data has problems, but also provides possible cause analysis and improvement suggestions.

[0040] Suppose a power grid company faces a large-scale power outage caused by a thunderstorm and needs to quickly assess the power restoration status of the affected area.

[0041] First, conduct a business context analysis: determine that the power outage was caused by thunderstorms, with a wide impact area, and the power grid infrastructure in some areas was severely damaged.

[0042] Next, update the rule base: Relax the verification standards for voltage fluctuation data in areas that are more affected by lightning strikes.

[0043] Strengthen the monitoring requirements for the operating status of backup generators and emergency power supply lines.

[0044] Furthermore, integrity checks are performed: using anomaly detection algorithms to check the voltage, current and other data uploaded in real time to ensure that no critical information is lost due to communication interruptions. Consistency checks are performed: through grid topology analysis, the power balance between substations is verified to ensure that the actual operating status reflected by the data is consistent with physical laws.

[0045] Finally, a detailed assessment report is generated to indicate which areas have seen smooth power restoration and where obstacles still exist (such as unrepaired damaged equipment), and to propose targeted solutions, such as giving priority to dispatching maintenance teams to key nodes for emergency repairs.

[0046] Through the above steps, power grid companies can respond to emergencies more efficiently and ensure the stable operation of the power system.

[0047] Furthermore, an embodiment of the present invention proposes a solution for dynamically adjusting data validation rules based on business context and performing integrity and consistency checks through intelligent algorithms, aiming to improve the flexibility and accuracy of data validation.

[0048] The options are as follows: Optionally, in step 103, "dynamically adjusting the data validation rules according to the business context, and determining the integrity and consistency of the real-time data through an intelligent algorithm to obtain the integrity and consistency evaluation results of the real-time data" includes: designing the data validation rules according to the business context to generate a dynamic data validation rule set, wherein the dynamic data validation rule set can adjust the corresponding verification standards according to different business scenarios; applying an intelligent algorithm to perform an integrity check on the real-time data, identifying whether there are missing values ​​or null values ​​in the real-time data, and counting integrity indicators to obtain integrity check results; performing a consistency check on the real-time data through the intelligent algorithm to ensure consistency between different fields of the real-time data and obtain consistency check results; adjusting the thresholds and standards of the integrity check and consistency check according to the dynamic data validation rule set to obtain inspection standards that meet different business needs; based on the inspection standards, combined with the integrity check results and consistency check results, generating an integrity and consistency evaluation report for the real-time data.

[0049] In this step, the integrity index is used to measure the quantitative standard of data integrity, such as missing rate, etc. The consistency index is used to measure the quantitative standard of data consistency, such as the proportion of inconsistent records, etc.

[0050] In the embodiment of this application, it is assumed that a power company is promoting a smart grid project and needs to monitor power consumption in real time to optimize scheduling. The company collects data from millions of smart meters every day. The following are specific application examples: Due to the different electricity consumption patterns in summer and winter, seasonal dynamic verification rules are designed. For example, during the peak hours in summer, the system will set a higher warning line for abnormally high electricity consumption. A machine learning-based anomaly detection model is used to check whether the electricity consumption data reported every hour is complete. If it is found that the data missing for a certain period exceeds a certain proportion (such as 10%), the alarm mechanism is triggered. The consistency is checked by comparing the meter readings of the same user with other related data (such as weather data). For example, when the temperature rises, the power consumption of air conditioners should increase in theory. If the data does not meet this expectation, it is marked as inconsistent. Under hot weather conditions, the upper limit constraint on electricity consumption is appropriately relaxed to adapt to the actual increase in electricity demand. A data quality report is generated once a day, which contains the integrity indicators of the data for that day (such as a missing rate of 3%) and a list of consistency issues for the operation and maintenance team to refer to.

[0051] Through the above solution, power companies can more flexibly respond to data changes in different seasons and weather conditions, and promptly discover and correct problems in the data. This not only improves the overall quality of the data, but also enhances the reliability of the decision support system. In addition, dynamically adjusted data validation rules help reduce false positives and false negatives, ensure the authenticity and validity of the data, and ultimately achieve more accurate power grid management and resource optimization configuration.

[0052] Optionally, the present application proposes a real-time data integrity check processing solution based on an intelligent algorithm, which aims to improve the efficiency and accuracy of data processing and ensure the integrity and reliability of real-time data in an automated and intelligent manner.

[0053] The options are as follows: Optionally, the application of an intelligent algorithm performs an integrity check on the real-time data, identifies whether there are missing values ​​or null values ​​in the real-time data, and calculates integrity indicators to obtain integrity check results, including: scanning the real-time data to identify missing values ​​or null values ​​in the real-time data, and calculating the proportion of missing values ​​or null values ​​in the real-time data to obtain integrity indicators; filling the identified missing values ​​or null values ​​to generate complete data; performing secondary verification on the filled data to confirm data integrity, and recording the integrity check results.

[0054] Scan processing: Perform a comprehensive inspection of the data set to identify missing or null values.

[0055] Completeness metric: A quantitative measure of the proportion of missing or null values ​​in a dataset.

[0056] Filling processing: Use appropriate algorithms or methods to fill missing values ​​or empty values ​​in the data set.

[0057] Secondary verification: Check the integrity of the data again after the filling process to ensure that the filled data meets the expected quality standards.

[0058] First, identify missing or null values ​​by performing a comprehensive scan of the real-time data. This can be achieved by simply traversing the data set. Secondly, calculate the proportion of missing or null values ​​in the total data volume. For example, if a field has 100 records and 10 of them are null, the missing rate of the field is 10%. Then use intelligent algorithms (such as machine learning-based prediction models) or statistical methods (such as mean filling, median filling, K nearest neighbor interpolation, etc.) to reasonably fill in the identified missing or null values. The selection of an appropriate filling method depends on the characteristics of the data and business needs. Finally, check the filled data again to confirm whether the data integrity meets the expected standards. This includes recalculating integrity indicators and performing other related verifications.

[0059] In the embodiment of the present application, it is assumed that a power company needs to perform integrity check and processing on real-time data from each substation in its smart grid monitoring system. The system collects data on multiple key indicators such as voltage (V), current (I), and power (P).

[0060] First, use the Pandas library to scan the real-time data and identify missing values. Assume that the scan results show that there are missing values ​​in the voltage data. The records are missing. Next, calculate the integrity index of the voltage data: integrity index = total data number missing value number = 0.05. Further select the linear interpolation method to fill in the missing voltage data. Assume that the original data is as follows: Time point 1: , time point , time point 3: , time point , time point 5: , the voltage value at time point 2 is obtained by linear interpolation: .

[0061] Finally, rescan the filled data to ensure that there are no new missing values. Assume that the secondary verification results show that the data integrity index after filling is 0.00, that is, there are no new missing values.

[0062] Through the above solution, power companies can effectively handle missing values ​​in real-time data and ensure data integrity and reliability. This not only improves the operating efficiency of the system, but also improves the accuracy of data analysis and decision-making. In addition, the intelligent processing method reduces manual intervention and reduces the error rate, making data processing more efficient and accurate. This method is also applicable to other fields that require high data integrity, such as industrial informatization, transportation, etc., which will help promote the digital transformation and development of the entire industry.

[0063] Optionally, the real-time data is subjected to a consistency check by the intelligent algorithm to ensure consistency between different fields of the real-time data and obtain a consistency check result, including: performing a logical consistency check on different fields in the real-time data to identify whether the logical relationship between fields is consistent, and recording the inconsistency to obtain a logical consistency check result between fields; performing a comparison on different fields in the real-time data to ensure data consistency between fields, and recording inconsistent data records to obtain a data consistency check result between fields; performing a consistency analysis on field values ​​in the real-time data to identify any data that does not comply with a predetermined consistency rule, and recording inconsistent data items to obtain a field value consistency check result; applying a machine learning model to perform pattern recognition on the real-time data, learning the consistency pattern in the real-time data, and dynamically adjusting the consistency check standard according to the learning result to obtain a dynamically adjusted consistency check standard; using a consistency verification algorithm, combined with the logical consistency check result between fields, the data consistency check result between fields, the field value consistency check result and the dynamically adjusted consistency check standard, to perform consistency verification on the real-time data to ensure data records are consistent between different fields, and recording the consistency verification result to obtain a consistency check result.

[0064] Logical consistency check: Check whether the logical relationship between different fields is as expected.

[0065] Comparison processing: Compare the data in different fields to ensure their consistency.

[0066] Consistency analysis: Identify data items that do not conform to predetermined consistency rules.

[0067] Pattern recognition: Using machine learning models to learn consistent patterns from data.

[0068] Consistency verification algorithm: Combines multiple check results to ensure that data records are consistent across different fields.

[0069] First, the logical consistency check process: check whether the logical relationship between different fields is consistent and record the inconsistency. Logical relationship refers to the expected relationship between fields, such as the Ohm's law relationship between voltage and current. Check whether the logical relationship between fields meets expectations through predefined business rules or natural laws. For example, the relationship between voltage and current should satisfy Ohm's law. Second, the comparison process: ensure that the data between different fields is consistent and record inconsistent data records. Comparison refers to comparing the data of different fields to ensure that they are consistent in content. Use data comparison tools or algorithms to check data records one by one and record inconsistent data records.

[0070] Further consistency analysis and processing: Identify any data that does not conform to the predetermined consistency rules and record the inconsistent data items. Consistency rules refer to predefined data consistency standards. Through programming logic or rule engines, check whether the data conforms to predefined consistency rules, such as data range, format, etc. Re-pattern recognition processing: Apply machine learning models to learn consistency patterns in data and dynamically adjust consistency check standards based on learning results. Pattern recognition refers to identifying consistency patterns in data through machine learning models. Using supervised learning or unsupervised learning algorithms, train models to identify consistency patterns in data and adjust consistency check standards accordingly. Final consistency verification: Based on the results of the above steps, verify the consistency of the data and record the consistency verification results. Consistency verification refers to the integration of various consistency check results to ensure that data records are consistent between different fields. Combine the logical consistency check results, comparison processing results, consistency analysis results, and dynamically adjusted consistency check standards to perform final consistency verification.

[0071] In the embodiment of the present application, it is assumed that a power company needs to perform consistency check on the real-time data from each substation in its smart grid monitoring system. , current (I), power (P) and other key indicators.

[0072] First check whether the logical relationship between voltage and current conforms to Ohm's law. Suppose there is a data record: voltage , current ,power According to Ohm's law , calculate the theoretical power , which is consistent with the actual power, is recorded as consistent. Next, compare the voltage data reported by different substations to ensure that they are consistent within a reasonable range. and The voltage data are: Substation A: , Substation If the substation and B are very close, the voltage difference should not exceed a certain range (such as ), then record it as consistent. Further check whether the voltage data is within a reasonable range (such as 100V to 150V). Suppose there is a data record: voltage This value is outside the reasonable range and is recorded as inconsistent. Then apply the machine learning model to learn the relationship between voltage, current, and power. Assume that the model learns from historical data that the relationship between voltage and current fluctuates slightly during certain time periods. Based on the learning results, dynamically adjust the consistency check standard to allow for a small range of deviations between voltage and current. Finally, combine the results of the above steps to perform a final consistency verification on the data. Assume that in the logical consistency check, comparison processing, and consistency analysis, most of the data records are in line with expectations, and only a few records are inconsistent. Finally, record the consistency verification results and further process the inconsistent data.

[0073] Through the above solution, the power company can effectively ensure the consistency of real-time data between different fields, avoiding system failures and decision-making deviations caused by inconsistent data. The intelligent consistency check and dynamic adjustment mechanism not only improves the efficiency and accuracy of data processing, but also reduces manual intervention and reduces the error rate. This method is also applicable to other fields that require high data consistency, such as industrial informatization, transportation, etc., which will help promote the digital transformation and development of the entire industry.

[0074] 104. Evaluate the logical relationship between the real-time data, and build a data quality evaluation model based on the evaluation result of the logical relationship and the integrity and consistency evaluation result, wherein the data quality evaluation model is used to quantify the accuracy and timeliness of the real-time data; In this step, the logical relationship between real-time data refers to the inherent connection and interaction between different data points in the power grid. For example, the input power and output power of the substation should be equal (ignoring losses), and the voltage levels of different monitoring points should vary within a reasonable range.

[0075] A data quality evaluation model refers to a mathematical model that comprehensively evaluates data quality. It quantifies the evaluation results of multiple dimensions (such as integrity, consistency, logical relationships, etc.) to form an overall data quality score.

[0076] Furthermore, an embodiment of the present invention proposes a comprehensive evaluation scheme based on the logical relationship between real-time data. Through logical relationship analysis, relationship strength mining, and combining integrity and consistency evaluation results, a comprehensive data quality evaluation model is constructed to improve the overall quality and credibility of the data.

[0077] The options are as follows: Optionally, the step 104 of "evaluating the logical relationship between the real-time data, and constructing a data quality evaluation model based on the evaluation results of the logical relationship and combining the integrity and consistency evaluation results" includes: performing logical relationship analysis on different fields in the real-time data, identifying the logical relationship between the fields, and recording the logical relationship information to obtain the logical relationship analysis results; mining the relationship between different fields in the real-time data, identifying the relationship strength between the fields, and recording the relationship strength information to obtain the association rule mining results; combining the logical relationship analysis results and the association rule mining results, comprehensively evaluating the logical relationship between the real-time data to obtain the logical relationship evaluation results; based on the comprehensive evaluation results of the logical relationship and the integrity and consistency evaluation results, quantifying the accuracy and timeliness of the real-time data to obtain the comprehensive analysis results; and constructing a data quality evaluation model based on the comprehensive analysis results.

[0078] In this step, in the data set, each specific attribute or variable is called a field. For example, in power grid data, fields may include voltage, current, power, timestamp, etc. Logical relationship analysis refers to analyzing the intrinsic connections between different fields in the data set, such as cause-effect relationships, functional relationships, etc. In the power grid, the logical relationship may be manifested as a balanced relationship between the input power and output power of a substation. Relationship strength refers to the degree of correlation between two or more fields. The higher the relationship strength, the closer the connection between the fields. Association rule mining refers to discovering frequently occurring patterns or rules from data, and is usually used to reveal implicit relationships between different fields. For example, when the power grid load exceeds a certain threshold, the voltage level may drop. Logical relationship evaluation refers to combining the results of logical relationship analysis and association rule mining to comprehensively evaluate the logical relationship between data to ensure the rationality and reliability of the data.

[0079] In the embodiment of the present application, firstly, different fields in the real-time data are analyzed, the logical relationships between them are identified, and the information of these relationships is recorded. For example, the relationship between the input power and the output power of the substation is analyzed to ensure that the two are equal.

[0080] Secondly, through data mining technology, the relationship strength between fields is identified and relevant information is recorded. For example, it is found that when the grid load exceeds 80%, the voltage level in some areas will drop significantly.

[0081] Furthermore, by combining the results of logical relationship analysis and association rule mining, a comprehensive evaluation of the logical relationship between the data is performed to generate a logical relationship evaluation result. This step helps to discover anomalies and inconsistencies in the data.

[0082] Furthermore, based on the comprehensive evaluation results of logical relationships, integrity and consistency, the accuracy and timeliness of real-time data are quantified to generate comprehensive analysis results. The evaluation of accuracy and timeliness can help power grid companies to promptly identify and resolve data quality issues.

[0083] Finally, based on the comprehensive analysis results, a data quality evaluation model is constructed, which can comprehensively evaluate all aspects of the data and provide decision support for power grid operations. It should be noted that in the context of data quality assessment, "constructing a data quality evaluation model" refers to defining an evaluation framework or scoring system, rather than traditional machine learning model training. This evaluation framework or scoring system is used to integrate the evaluation results of multiple dimensions to generate an overall data quality score.

[0084] Suppose a power grid company needs to assess the data quality of power supply in a certain area, especially after a natural disaster.

[0085] First, analyze the relationship between the input power and output power of the substation to ensure that they are equal (ignore losses). For example, the total input power of substation A is 1000MW and the output power is 998MW (the loss of 2MW is within a reasonable range). Check whether the voltage levels at different monitoring points fluctuate within the standard range. For example, the voltage levels at all monitoring points are within the range of 110kV±5%.

[0086] Secondly, it was found that when the grid load exceeded 80%, the voltage level in some areas would drop significantly. For example, when the load in a certain area reached 85%, the voltage level dropped by 3%.

[0087] Furthermore, it is identified that under thunderstorm conditions, the readings of some sensors may be abnormal. For example, during a thunderstorm, the readings of some current sensors fluctuate greatly, which may be due to signal interference.

[0088] Furthermore, the rationality of the data is evaluated by combining the results of logical relationship analysis and association rule mining. For example, the input and output power of substation A is basically balanced, but the voltage levels of some monitoring points fluctuate abnormally during thunderstorms, which requires further investigation. These evaluation results are recorded and a logical relationship evaluation report is generated.

[0089] Further, quantify accuracy and timeliness, including: Completeness score: 90 / 100 (a few data points were not reported) Consistency score: 85 / 100 (found inconsistencies between a few data points) Logical relationship score: 92 / 100 (most of the data conforms to the preset logical relationship) Accuracy score: 88 / 100 (small errors in data collection and transmission) Timeliness score: 90 / 100 (data is updated frequently with little delay) Finally, define the evaluation framework: Assuming that the weights of each dimension are the same, the comprehensive score is (90+85+92+88+90) / 5=89 / 100. Then, based on the evaluation framework, generate a comprehensive data quality score. The generated data quality score can be used to evaluate the overall quality of the data and provide a basis for business improvement.

[0090] Through the above steps, the power grid company can fully understand the quality of the data, discover and solve potential problems in a timely manner, and thus improve the reliability and efficiency of power grid operation. This model can also be used for daily operation and maintenance to help power grid companies continuously optimize data management processes.

[0091] Furthermore, the embodiment of the present invention proposes a quantification scheme based on the comprehensive evaluation results of logical relationships, integrity and consistency evaluation results. By setting a comprehensive quantitative indicator system, the accuracy and timeliness of the data are systematically evaluated, thereby improving the scientificity and practicality of data quality evaluation.

[0092] The options are as follows: Optionally, based on the comprehensive evaluation results of the logical relationships and the integrity and consistency evaluation results, the accuracy and timeliness of the real-time data are quantified to obtain a comprehensive analysis result, including: setting a quantitative indicator system for the accuracy and timeliness of the real-time data, including accuracy rate, error rate, update frequency, and delay time; determining the logical relationship indicators that affect the accuracy of the real-time data based on the comprehensive evaluation results of the logical relationships; determining the integrity and consistency indicators that affect the accuracy and timeliness of the real-time data based on the integrity and consistency evaluation results; calculating the specific values ​​of the logical relationship indicators and the integrity and consistency indicators, and combining them with the quantitative indicator system to obtain a comprehensive analysis result.

[0093] In this step, accuracy refers to the proportion of correct information to total information. Error rate refers to the proportion of erroneous information to total information. Update frequency refers to how often data is updated. Latency refers to the time difference from data generation to data processing. Logical relationship indicators refer to the metrics that reflect the strength of logical associations between different fields. Completeness and consistency indicators refer to the metrics that measure data missingness and logical consistency.

[0094] In the embodiment of the present application, a comprehensive set of quantitative index systems is first defined, including accuracy, error rate, update frequency and delay time. These indicators can fully reflect the accuracy and timeliness of the data. Secondly, according to the comprehensive evaluation results of the logical relationship, the logical relationship indicators that affect the accuracy of the data are determined. For example, if the logical relationship between certain fields is found to be unreasonable, the accuracy of the data may be reduced. Further, according to the integrity and consistency evaluation results, the integrity and consistency indicators that affect the accuracy and timeliness of the data are determined. For example, the missing rate and duplication rate of the data. Calculate the specific values ​​of the logical relationship indicators, integrity and consistency indicators. For example, the missing rate can be calculated by statistical methods, and the error rate can be predicted by machine learning models. Finally, combined with the quantitative index system, the specific values ​​of the logical relationship indicators and the integrity and consistency indicators are comprehensively analyzed to obtain the accuracy and timeliness scores of the data.

[0095] Specifically, the quantitative indicator system for setting the accuracy and timeliness of the real-time data includes: The corresponding formulas for defining accuracy, error rate, update frequency, and delay time are: ; in, Indicates the accuracy, It is The accuracy score of each field, It is The importance weight of each field, is the accuracy enhancement factor, Indicates the total number of evaluated fields; ; in, represents the error rate; ; in, Indicates the update frequency, It is The frequency score of the updates, is the update frequency enhancement factor, is the total number of updates or the total number of time units in the evaluation period; ; in, Indicates the delay time. It is The delay time of the update, is the delay time enhancement factor, is the total number of updates or the total number of time units in the evaluation period; The step of determining the logical relationship index that affects the accuracy of the real-time data according to the comprehensive evaluation result of the logical relationship, wherein the logical relationship index includes a logical relationship strength score and a logical relationship consistency score, including: The logical relationship strength score and logical relationship consistency score are calculated using the following calculation formula: ; in, represents the logical relationship strength score, It is Score the logical relationship strength of the field. It is Importance weights for fields, is the logical relationship enhancement factor, Indicates the total number of evaluated fields; ; in, represents the logical relationship consistency score, It is The number of inconsistent records in the field, It is The total number of records for the field, It is Importance weights for fields, is the consistency index, is the logical relationship consistency enhancement factor, Indicates the total number of evaluated fields; Determining, based on the integrity and consistency evaluation results, integrity and consistency indicators that affect the accuracy and timeliness of the real-time data, wherein the integrity and consistency indicators include integrity scores and consistency scores, including: The completeness score and consistency score are calculated using the following formula: ; in, represents the completeness score, It is The number of missing values ​​in the fields, It is The total number of records in the field, It is The importance weight of each field, is the completeness index, is the integrity enhancement factor, Indicates the total number of evaluated fields; ; in, represents the consistency score, It is The number of inconsistent records in the fields, It is The total number of records in the field, It is The importance weight of each field, is the consistency enhancement factor, Indicates the total number of evaluated fields; The specific values ​​of the logical relationship index and the integrity and consistency index are calculated, and combined with the quantitative index system to obtain a comprehensive analysis result, including: The comprehensive analysis results are calculated using the following formula: ; in, The comprehensive analysis results are shown in The logical relationship strength scores are , logical relationship consistency score , completeness score , consistency score , accuracy score , error rate score , update frequency score , delay time score The weight coefficient of is the corresponding exponential coefficient; It is an enhancement factor of the comprehensive score.

[0096] In the above formula, the accuracy enhancement factor ( ): Used to adjust the influence of different fields on accuracy scores. Update frequency enhancement factor ( ): Used to adjust the influence of update frequency scores at different time points. Delay time enhancement factor ( ): Used to adjust the influence of delay time scores at different time points. Logical relationship enhancement factor ( ): Used to adjust the influence of the strength score of the logical relationship between different fields. ): Measures the degree of consistency between or within fields. Completeness Index ( ): Measures the impact of missing values ​​in a field. Enhancement factor ( ): Used to adjust the weight of each indicator in the comprehensive score.

[0097] In the above accuracy And the error rate In the calculation formula, the accuracy score : Calculates the accuracy score for each field by comparing the actual value with the theoretical value or standard value. For example, the degree of agreement between the actual reading of a voltage sensor and the standard reading. Importance weight : Assign weights to fields based on their importance in the business. For example, voltage and current may be more important in a power grid than other fields. Accuracy Enhancement Factor : Adjust according to specific circumstances, usually 1.

[0098] The accuracy rate of the application The accuracy of the overall data is reflected by weighted average accuracy scores of each field. As the complement of accuracy, it directly reflects the degree of error in the data.

[0099] In the update frequency In the calculation formula, the frequency score : The score is calculated based on the time interval between each data update. The shorter the interval, the higher the score. Update frequency enhancement factor : Adjust according to specific circumstances, usually 1.

[0100] How often this application is updated The exponential function calculation reflects the timeliness of data update. The combination of logarithmic function and exponential function can better capture the impact of frequency changes on the score.

[0101] In delay time In the calculation formula, the delay time : Record the delay time of each data update. Delay time enhancement factor : Adjust according to specific circumstances, usually 1.

[0102] Delay time for this application The square root function is used to calculate the delay of data transmission. The square root function can smooth the impact of large delay times.

[0103] In the logical relationship strength score In the calculation formula, the logical relationship strength score : Calculate the strength of the logical relationship between each pair of fields through methods such as correlation analysis or logistic regression. Importance weight : Assign weights to the Ning segment pairs according to their importance in the business. Logical relationship enhancement factor : Adjust according to specific circumstances, usually 1.

[0104] The logical relationship strength score of this application The strength of the logical relationship between the data is reflected by weighted averaging the strength of the logical relationship between each pair of fields.

[0105] In the logical relationship consistency score In the calculation formula, the number of inconsistent records : Record the number of inconsistent records in each pair of fields. Total number of records : Total number of records for each pair of segments. Importance weight : Assign weights to field pairs based on their importance in the business. : Adjust according to the specific situation, usually set to 1. Logical relationship consistency enhancement factor : Adjust according to specific circumstances, usually 1.

[0106] The logical consistency score of this application The calculation in product form reflects the consistency between the data, which can better capture the impact of inconsistent records.

[0107] Integrity score In the calculation formula of : Record the number of missing values ​​in each field. Total number of records : The total number of records for each field. Importance weight : Assign weights to fields based on their importance in the business. Completeness Index : Adjust according to the specific situation, usually 1. Integrity enhancement factor : Adjust according to specific circumstances, usually 1.

[0108] Completeness score of this application The completeness of the data is reflected by weighted average of the completeness of each field. The exponential form can better capture the impact of missing values.

[0109] In consistency score In the calculation formula, the number of inconsistent records : Record the number of inconsistent records in each field. Total number of records : The total number of records for each field. Importance weight : Assign weights to fields based on their importance in the business. Consistency Enhancement Factor : Adjust according to specific circumstances, usually 1.

[0110] Consistency score for this application The exponential function reflects the consistency between the data. The combination of logarithmic and exponential functions can better capture the impact of inconsistent records.

[0111] In the comprehensive analysis results In the calculation formula, the weight coefficient : Assign weights to each indicator based on its importance in the business. Index coefficient : Adjust according to the specific situation, usually 1. Enhancement factor of comprehensive score : Adjust according to specific circumstances, usually 1.

[0112] Comprehensive analysis results of this application The weighted average of the scores of each indicator reflects the overall quality of the data. The index form and enhancement factor can better adjust the impact of each indicator.

[0113] By substituting the obtained values ​​into the above formula, the comprehensive analysis results can be finally calculated, so as to understand the quality status of the data based on the comprehensive analysis results, timely discover and solve potential problems, and thus improve the reliability and efficiency of power grid operation. This comprehensive scoring model provides a scientific basis for business improvement. The acquisition of each parameter and the design algorithm of the formula are to ensure the accuracy and reliability of the evaluation results and help decision makers make more informed decisions. For example, by comparing the comprehensive analysis results with the preset results, the overall quality of the data can be determined. If the preset results are not met, the analysis is further refined to find out which specific indicators have low scores, so as to make targeted improvements.

[0114] 105. Generate a data quality report based on the data quality evaluation model, and propose business improvement measures based on the data quality report; In this step, the data quality report is a detailed document generated based on the data quality evaluation model, which contains the scoring results of each dimension, problem analysis and improvement suggestions. Business improvement measures are specific action plans formulated based on the problem analysis and improvement suggestions in the data quality report, aiming to improve data quality and business efficiency.

[0115] In the embodiment of the present application, the method of generating a data quality report includes but is not limited to: generating a chart based on the comprehensive analysis results of the evaluation results of each dimension obtained in the above step 104 to intuitively display the scoring results of each dimension.

[0116] Improvement measures are proposed for low-scoring items. For example, if the accuracy is low, it may be due to low sensor accuracy or problems with the data processing algorithm. The corresponding business improvement measures may include: replacing or upgrading sensors to improve the accuracy of data collection; or optimizing data processing algorithms to reduce errors in data processing. The specific settings can be based on demand, and the embodiments of this application do not limit this.

[0117] Figure 2 A structural diagram of a data quality assessment system based on big data analysis is provided for an embodiment of the present application, such as Figure 2 As shown, the device comprises: A receiving module 21, configured to receive real-time data from different data sources, wherein the real-time data includes structured data and unstructured data; The parsing module 22 is used to perform meaning parsing on the unstructured data to obtain the business context of the real-time data; The adjustment module 23 is used to dynamically adjust the data validation rules according to the business context, and determine the integrity and consistency of the real-time data through an intelligent algorithm to obtain the integrity and consistency evaluation results of the real-time data; A construction module 24 is used to evaluate the logical relationship between the real-time data, and to construct a data quality evaluation model based on the evaluation results of the logical relationship and the integrity and consistency evaluation results, wherein the data quality evaluation model is used to quantify the accuracy and timeliness of the real-time data; The generating module 25 is used to generate a data quality report according to the data quality evaluation model, and propose business improvement measures according to the data quality report.

[0118] Figure 2 The data quality assessment system based on big data analysis can be performed Figure 1 The implementation principle and technical effect of the data quality assessment method based on big data analysis described in the embodiment shown are not repeated here. The specific way in which each module and unit performs operations in the data quality assessment system based on big data analysis in the above embodiment has been described in detail in the embodiment of the method, and will not be elaborated here.

[0119] In one possible design, Figure 2 A data quality assessment system based on big data analysis in the illustrated embodiment can be implemented as a computing device, such as Figure 3 As shown, the computing device may include a storage component 31 and a processing component 32; The storage component 31 stores one or more computer instructions, wherein the one or more computer instructions are called and executed by the processing component 32 .

[0120] The processing component 32 is used to: receive real-time data from different data sources, the real-time data including structured data and unstructured data; perform meaning analysis on unstructured data to obtain the business context of real-time data; determine the integrity and consistency of real-time data through intelligent algorithms to obtain integrity and consistency evaluation results of real-time data; construct a data quality evaluation model based on the evaluation results of logical relationships and in combination with the integrity and consistency evaluation results, the data quality evaluation model is used to quantify the accuracy and timeliness of real-time data; generate a data quality report, and propose business improvement measures based on the data quality report; optimize data collection and processing processes based on the business improvement measures.

[0121] The processing component 32 may include one or more processors to execute computer instructions to complete all or part of the steps in the above method. Of course, the processing component may also be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0122] The storage component 31 is configured to store various types of data to support operations at the terminal. The storage component can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0123] Of course, the computing device may also include other components, such as input / output interfaces, display components, communication components, etc.

[0124] The input / output interface provides an interface between the processing component and the peripheral interface module, which may be an output device, an input device, etc.

[0125] The communication component is configured to facilitate, among other things, wired or wireless communications between the computing device and other devices.

[0126] Among them, the computing device can be a physical device or an elastic computing host provided by a cloud computing platform, etc. In this case, the computing device can refer to a cloud server, and the above-mentioned processing components, storage components, etc. can be basic server resources rented or purchased from the cloud computing platform.

[0127] The present application also provides a computer storage medium storing a computer program, wherein the computer program can achieve the above-mentioned Figure 1 The illustrated embodiment is a data quality assessment method based on big data analysis.

[0128] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0129] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0130] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data quality assessment method based on big data analysis, characterized in that: include: Receiving real-time data from different data sources, wherein the real-time data includes structured data and unstructured data; Performing meaning analysis on the unstructured data to obtain the business context of the real-time data; According to the business context, the data validation rules are dynamically adjusted, and the integrity and consistency of the real-time data are determined by an intelligent algorithm to obtain the integrity and consistency evaluation results of the real-time data; Evaluate the logical relationship between the real-time data, and build a data quality evaluation model based on the evaluation results of the logical relationship and the integrity and consistency evaluation results, wherein the data quality evaluation model is used to quantify the accuracy and timeliness of the real-time data; A data quality report is generated based on the data quality evaluation model, and business improvement measures are proposed based on the data quality report.

2. The method according to claim 1, characterized in that: The evaluating and processing the logical relationship between the real-time data, and constructing a data quality evaluation model based on the evaluation result of the logical relationship and the integrity and consistency evaluation result, includes: Performing logical relationship analysis on different fields in the real-time data, identifying the logical relationship between the fields, and recording the logical relationship information to obtain a logical relationship analysis result; Mining the relationship between different fields in the real-time data, identifying the relationship strength between the fields, and recording the relationship strength information to obtain association rule mining results; Combining the logical relationship analysis result and the association rule mining result, a comprehensive evaluation process is performed on the logical relationship between the real-time data to obtain a logical relationship evaluation result; Based on the comprehensive evaluation results of the logical relationship and the integrity and consistency evaluation results, the accuracy and timeliness of the real-time data are quantified to obtain a comprehensive analysis result; Based on the comprehensive analysis results, a data quality evaluation model is constructed.

3. The method according to claim 2, characterized in that The comprehensive evaluation results based on the logical relationship and the integrity and consistency evaluation results are used to quantify the accuracy and timeliness of the real-time data and obtain comprehensive analysis results, including: Setting a quantitative indicator system for the accuracy and timeliness of the real-time data, including accuracy, error rate, update frequency, and delay time; Determining the logical relationship index that affects the accuracy of the real-time data according to the comprehensive evaluation result of the logical relationship; Determining integrity and consistency indicators that affect the accuracy and timeliness of the real-time data based on the integrity and consistency assessment results; Calculate the specific values ​​of the logical relationship index and the integrity and consistency index, and obtain a comprehensive analysis result by combining the quantitative index system; The quantitative indicator system for setting the accuracy and timeliness of the real-time data includes: The corresponding formulas for defining accuracy, error rate, update frequency, and delay time are: ； in, Indicates the accuracy, It is The accuracy score of each field, It is The importance weight of each field, is the accuracy enhancement factor, Indicates the total number of evaluated fields; ； in, represents the error rate; ； in, Indicates the update frequency, It is The frequency score of the updates, is the update frequency enhancement factor, is the total number of updates or the total number of time units in the evaluation period; ； in, Indicates the delay time. It is The delay time of the update, is the delay time enhancement factor, is the total number of updates or the total number of time units in the evaluation period; The step of determining the logical relationship index that affects the accuracy of the real-time data according to the comprehensive evaluation result of the logical relationship, wherein the logical relationship index includes a logical relationship strength score and a logical relationship consistency score, including: The logical relationship strength score and logical relationship consistency score are calculated using the following calculation formula: ； in, represents the logical relationship strength score, It is The logical relationship strength score of the field, It is Importance weights for fields, is the logical relationship enhancement factor, Indicates the total number of evaluated fields; ； in, represents the logical relationship consistency score, It is The number of inconsistent records in the field, It is The total number of records for the field, It is Importance weights for fields, is the consistency index, is the logical relationship consistency enhancement factor, Indicates the total number of evaluated fields; Determining, based on the integrity and consistency evaluation results, integrity and consistency indicators that affect the accuracy and timeliness of the real-time data, wherein the integrity and consistency indicators include integrity scores and consistency scores, including: The completeness score and consistency score are calculated using the following formula: ； in, represents the completeness score, It is The number of missing values ​​in the fields, It is The total number of records in the field, It is The importance weight of each field, is the completeness index, is the integrity enhancement factor, Indicates the total number of evaluated fields; ； in, represents the consistency score, It is The number of inconsistent records in the fields, It is The total number of records in the field, It is The importance weight of each field, is the consistency enhancement factor, Indicates the total number of evaluated fields; The specific values ​​of the logical relationship index and the integrity and consistency index are calculated, and combined with the quantitative index system to obtain a comprehensive analysis result, including: The comprehensive analysis results are calculated using the following formula: ； in, The comprehensive analysis results are shown in The logical relationship strength scores are , logical relationship consistency score , completeness score , consistency score , accuracy score , error rate score , update frequency score , delay time score The weight coefficient of is the corresponding exponential coefficient; It is an enhancement factor of the comprehensive score.

4. The method according to claim 1, characterized in that: The data validation rules are dynamically adjusted according to the business context, and the integrity and consistency of the real-time data are determined by an intelligent algorithm to obtain the integrity and consistency evaluation results of the real-time data, including: According to the business context, the data validation rules are designed and processed to generate a dynamic data validation rule set, wherein the dynamic data validation rule set can adjust the corresponding validation standards according to different business scenarios; Applying an intelligent algorithm to perform integrity check processing on the real-time data, identifying whether there are missing values ​​or null values ​​in the real-time data, and calculating integrity indicators to obtain integrity check results; By using the intelligent algorithm, the real-time data is subjected to consistency check processing to ensure consistency between different fields of the real-time data, and obtain a consistency check result; According to the dynamic data validation rule set, the thresholds and standards of integrity check and consistency check are adjusted to obtain the inspection standards that meet different business needs; Based on the inspection standard and in combination with the integrity inspection result and the consistency inspection result, an integrity and consistency assessment report of the real-time data is generated.

5. The method according to claim 4, characterized in that The application intelligent algorithm performs integrity check processing on the real-time data, identifies whether there are missing values ​​or null values ​​in the real-time data, and counts integrity indicators to obtain integrity check results, including: Scanning and processing the real-time data, identifying missing values ​​or null values ​​in the real-time data, and calculating the proportion of missing values ​​or null values ​​in the real-time data to obtain an integrity index; Fill in the identified missing or empty values ​​to generate complete data; Perform secondary verification on the filled data to confirm data integrity and record the integrity check results.

6. The method according to claim 4, characterized in that The real-time data is subjected to a consistency check by the intelligent algorithm to ensure consistency between different fields of the real-time data, and a consistency check result is obtained, including: Performing a logical consistency check on different fields in the real-time data to identify whether the logical relationships between the fields are consistent, and recording inconsistencies to obtain a logical consistency check result between the fields; Compare different fields in the real-time data to ensure the consistency of data between fields, record inconsistent data records, and obtain the consistency check result of data between fields; Performing consistency analysis on the field values ​​in the real-time data, identifying any data that does not conform to a predetermined consistency rule, and recording inconsistent data items to obtain a field value consistency check result; Applying a machine learning model to perform pattern recognition processing on the real-time data, learning the consistency pattern in the real-time data, and dynamically adjusting the consistency check standard according to the learning result to obtain a dynamically adjusted consistency check standard; Using the consistency verification algorithm, combined with the inter-field logical consistency check results, inter-field data consistency check results, field value consistency check results and the dynamically adjusted consistency check standard, the real-time data is subjected to consistency verification processing to ensure that data records are consistent between different fields, and the consistency verification results are recorded to obtain the consistency check results.

7. The method according to claim 1, characterized in that The performing meaning parsing processing on the unstructured data to obtain the business context of the real-time data includes: Converting words in the unstructured data into vector representations to capture semantic relationships between words; Identify entity information from vectorized unstructured data; Parse sentence structure and determine the grammatical dependencies between words to understand the semantic structure within the sentence; Performing context information processing on the unstructured data to capture context information at sentence and paragraph levels; Evaluate the sentiment tendency expressed in the unstructured data, identify positive, negative or neutral sentiment, and obtain sentiment analysis results; extracting main discussion topics from the unstructured data; The entity information, the semantic structure, the context information, the sentiment analysis result and the topic are integrated to generate a business context in the unstructured data and use it as the business context of the real-time data.

8. A data quality assessment system based on big data analysis, characterized in that: include: A receiving module, used for receiving real-time data from different data sources, wherein the real-time data includes structured data and unstructured data; A parsing module, used to perform meaning parsing on the unstructured data to obtain the business context of the real-time data; An adjustment module is used to dynamically adjust the data validation rules according to the business context, and determine the integrity and consistency of the real-time data through an intelligent algorithm to obtain the integrity and consistency evaluation results of the real-time data; A construction module is used to evaluate the logical relationship between the real-time data, and based on the evaluation results of the logical relationship, in combination with the integrity and consistency evaluation results, to construct a data quality evaluation model, wherein the data quality evaluation model is used to quantify the accuracy and timeliness of the real-time data; The generating module is used to generate a data quality report according to the data quality evaluation model, and propose business improvement measures according to the data quality report.

9. A computing device, characterized in that It comprises a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement a data quality assessment method based on big data analysis as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a computer, a data quality assessment method based on big data analysis as described in any one of claims 1 to 7 is implemented.

Images