Hive-based analysis data quality evaluation method and system
By using a Hive-based data quality assessment method, the problems of low efficiency, poor scalability, and security in large-scale data assessment are solved, achieving efficient, flexible, and secure data quality assessment.
Patent Information
- Application Number
- CN202511546601.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-28
- Publication Date
- 2026-01-13
AI Technical Summary
Existing data quality assessment methods are inefficient, lack scalability, have poor compatibility and flexibility, have coarse-grained access control, are disconnected from logs and data analysis, lack interactive visualization, and are difficult to handle large-scale data analysis at the TB level.
The Hive-based data quality assessment method leverages the Hive library to be compatible with data from multiple protocols and carriers, supports multi-threaded asynchronous queries, loads pre-defined and custom rules, performs distributed aggregation calculations, and combines a three-level permission model and visualization to generate assessment logs for rapid location.
It enables rapid quality assessment of TB-level data, improves assessment efficiency and flexibility, reduces the risk of data leakage, shortens fault location time, and enhances interactivity and security.
Smart Images
Figure CN121333979A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of this application relate to the field of data quality assessment technology, and in particular to a method and system for assessing the quality of parsed data based on Hive. Background Technology
[0002] Data quality assessment refers to the process of using systematic methods, indicators, and tools to detect, measure, and analyze key characteristics of data, such as accuracy, completeness, and consistency, to determine whether the data can meet the needs of a specific business scenario. Its core objective is to identify problems in the data (such as errors, missing data, and redundancy), quantify the level of data quality, and provide a basis for subsequent data cleaning, optimization, and management.
[0003] Traditional data quality assessments largely rely on manual testing, using scripts or tools to extract sample data for verification. For example, they might use Python or shell scripts to perform sample queries on a database and count the missing or formatting errors of fields. Such methods are inefficient, have limited coverage, and are ill-suited for large-scale data scenarios.
[0004] In addition, there are some commonly used data quality assessment techniques.
[0005] Validation techniques based on rule engines. Some tools (Apache Griffin, Great Expectations) perform batch validation on data using predefined rules. For example, they use SQL rules to check the integrity and range of values in fields. However, this technique has poor rule extensibility and cannot dynamically support user-defined rules.
[0006] Distributed computing framework technologies. Current solutions (such as Spark and Flink) accelerate data analysis through distributed computing. However, this technology requires complex environment deployment, lacks compatibility with the Hive library, and makes it difficult to directly reuse existing data warehouse structures.
[0007] User access control technology. Current systems (such as RBAC-based access control) restrict function access through role division, but this technology lacks fine-grained access control for data quality analysis scenarios, such as custom rule permissions.
[0008] Log management technology. Current solutions (such as ELK Stack) achieve operational monitoring through centralized log collection, but they are not deeply integrated with the data quality analysis process, making it difficult to quickly locate the cause of data anomalies.
[0009] Therefore, the existing data quality assessment methods have the following shortcomings.
[0010] First, it is inefficient and lacks scalability. Traditional manual testing and stand-alone tools cannot handle terabyte-level data, and the response time far exceeds business requirements (e.g., daily data analysis takes more than 30 minutes). In addition, the rules are fixed and cannot dynamically support private fields and custom rules.
[0011] Second, compatibility and flexibility are poor. Current distributed frameworks (such as Spark) require additional adaptation to Hive metadata and cannot directly utilize Hive's partitioning mechanism, resulting in poor query performance. Furthermore, they lack pre-built rule bases for multiple protocols (HTTP, FTP, etc.), requiring redundant development.
[0012] Third, the granularity of access control is too coarse. The RBAC model only controls access to functional modules and is not specifically designed for data quality analysis scenarios (such as restricting users to only analyze data from specific operators or protocols), which poses a risk of data leakage.
[0013] Fourth, data visualization is weak. Traditional tools mostly output results in the form of command lines or tables, lacking interactive visualizations (such as trend charts, heat maps, etc.), making it difficult to intuitively show the trend of data quality changes.
[0014] Fifth, logs are disconnected from data analysis. The log system is independent of the analysis process, making it impossible to quickly trace back data anomalies (such as missing analysis results due to a failed Hive query on a certain day), resulting in low troubleshooting efficiency. Summary of the Invention
[0015] To address the aforementioned technical issues, embodiments of this application propose a Hive-based data quality assessment method and system, which can perform quality assessment on large-scale data (TB level), effectively improving the efficiency, scalability, flexibility, and interactivity of data quality assessment, refining the granularity of permission management, reducing the risk of data leakage, and ultimately achieving efficient and secure data quality assessment.
[0016] To achieve the above objectives, embodiments of this application propose a Hive-based data quality assessment method. The method includes: efficiently extracting the data to be analyzed for this assessment from a Hive library; wherein the Hive library is compatible with data from multiple protocols and operators, and supports multi-threaded asynchronous queries; loading general rules from a pre-set rule base, and simultaneously acquiring user-inputted custom rules, generating a rule base for this assessment based on all general rules and all custom rules; performing distributed aggregation calculations on the data to be analyzed for this assessment, and combining this with the rule base for this assessment to perform data quality assessment, obtaining the assessment result; visualizing the assessment result, and determining access permissions for different users based on a pre-set three-level permission model, allowing only users to access within their permissions; and generating assessment logs that support rapid location based on the entire assessment process.
[0017] To achieve the above objectives, embodiments of this application also propose a Hive-based data quality assessment system. The system includes: a data acquisition unit for efficiently extracting the data to be analyzed in this assessment from a Hive library, wherein the Hive library is compatible with data from multiple protocols and operators and supports multi-threaded asynchronous queries; a rule management unit for loading general rules from a pre-set rule base and acquiring user-inputted custom rules, generating a rule base for this assessment based on all general and custom rules; a distributed computing unit for performing distributed aggregation calculations on the data to be analyzed in this assessment, and combining this with the rule base for this assessment to perform data quality assessment and obtain the assessment result; a visualization and permission control unit for visually displaying the assessment result and determining the access permissions for different users based on a preset three-level permission model, allowing only users to access within their permissions; and a log integration unit for generating assessment logs that support rapid location based on the entire assessment process.
[0018] To achieve the above objectives, embodiments of this application also propose an electronic device, including a processor and a memory, wherein the memory stores instructions executable by the processor, and the processor is configured to execute the instructions such that the electronic device can implement a Hive-based parsing data quality assessment method as described above.
[0019] To achieve the above objectives, embodiments of this application also propose a computer-readable storage medium storing a computer program that, when executed by a processor, enables a Hive-based data quality assessment method as described above.
[0020] This application proposes a Hive-based data quality assessment method. First, it efficiently extracts the data to be analyzed from the Hive library. The Hive library is compatible with data from multiple protocols and carriers and supports multi-threaded asynchronous queries, significantly improving assessment efficiency. The response time for analyzing TB-level data daily is less than 5 minutes, a 6-fold improvement compared to traditional methods. Next, it loads general rules from a pre-built rule base and simultaneously acquires user-inputted custom rules. Based on all general and custom rules, it generates a rule base for this assessment, effectively enhancing the flexibility of data quality assessment. Then, it performs distributed aggregation calculations on the data to be analyzed and combines this with the rule base to perform data quality assessment, yielding the assessment results. Distributed aggregation calculations effectively accelerate the analysis process. Following this, the assessment results are visualized, and a pre-defined three-level permission model determines the access permissions for different users, allowing only those permissions. This design effectively improves the security of data quality assessment, and the three-level permission model significantly reduces the risk of data leakage. Finally, it generates assessment logs supporting rapid fault location based on the entire assessment process, reducing fault location time by 80%. In summary, this method enables rapid quality assessment of terabyte-scale data, effectively improving the efficiency, scalability, flexibility, and interactivity of data quality assessment, refining the granularity of access control, reducing the risk of data leakage, and ultimately achieving efficient, flexible, and secure data quality assessment.
[0021] Optionally, the data to be analyzed in this evaluation can be efficiently extracted from the Hive library, including: A connection pool supporting multi-threaded asynchronous queries is established through the Hive JDBC driver to interact with the Hive library; Based on the start and end time parameters input by the user, Hive partition filtering conditions are automatically generated, and data scanning of the Hive library is performed based on the Hive partition filtering conditions. Dynamically construct HQL statements based on protocol type and field rules; The JVM is started using Jpype, and the pre-compiled Java JAR package is called to execute the concatenated HQL statements, thereby obtaining the data to be analyzed in this evaluation.
[0022] Optionally, the pre-built rule base contains a variety of general rules, including IMSI format verification rules and MSISDN length verification rules, which are stored in the MySQL rule table according to protocol type. Whitelist filtering of user-inputted custom rules, allowing only letters, numbers, and specific symbols; The custom rules filtered by the whitelist are parsed into an abstract syntax tree, and key parameters are extracted, including field names, operators, and thresholds. The parsed abstract syntax tree is converted into a parameterized HQL template, and then combined with all the general rules in the MySQL rule table to form the rule base for this evaluation.
[0023] Optionally, when performing distributed aggregation computation, data is stored in Hive ORC format to reduce I / O overhead, filtering conditions are pushed to the storage layer to reduce data reading, and computation tasks are distributed through Celery task queues to support horizontal scaling of Worker nodes; Regarding intermediate result caching, Hive query results are stored in Redis with a TTL of 1 hour for reuse in subsequent multi-dimensional analysis.
[0024] Optionally, when visualizing the evaluation results, it supports dynamically generating charts including bar charts, line charts, and heatmaps. The charts contain clickable elements, and when a clickable element is triggered, the corresponding detailed data is queried.
[0025] Optionally, the preset three-level permission model includes protocol level, carrier level, and rule level; Protocol-level restrictions are used to limit users to access only authorized protocols; Carrier-grade equipment is used to isolate data access between different carriers; Rule-level rules are used to restrict whether users are allowed to customize rules.
[0026] Optionally, an evaluation log supporting rapid location is generated based on the entire evaluation process, including: Throughout the entire evaluation process, Logstash was used to collect Django logs and Hive query logs. Use Elasticsearch to build a log index and generate evaluation logs, supporting fast retrieval and quick location based on timestamps, protocol types, and error codes; An email alert will be automatically triggered when the same type of error appears three times consecutively in the evaluation log. Attached Figure Description
[0027] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies of this application will be briefly introduced below. Obviously, the following drawings are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. The drawings described herein are only used to explain this application and are not intended to limit this application.
[0028] Figure 1This is a flowchart of a Hive-based data parsing quality assessment method provided in one embodiment of this application; Figure 2 This is a schematic diagram of the structure of a Hive-based parsing data quality assessment system provided in another embodiment of this application; Figure 3 This is a schematic diagram of the structure of an electronic device provided in another embodiment of this application. Detailed Implementation
[0029] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. Those skilled in the art will understand that many technical details have been presented in the embodiments of this application to facilitate better understanding. However, the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments. The division of the following embodiments is for ease of description and should not constitute any limitation on the specific implementation of this application. The following embodiments can be combined with and referenced by each other without contradiction.
[0030] One embodiment of this application proposes a Hive-based method for evaluating the quality of parsed data, applied to an electronic device, wherein the electronic device can be a terminal or a server. This embodiment and the following embodiments will use a server as an example for illustration. The implementation details of the Hive-based method for evaluating the quality of parsed data proposed in this embodiment are described below. The following implementation details are provided for ease of understanding only and are not essential for implementing this solution.
[0031] The specific process of the Hive-based parsed data quality assessment method proposed in this embodiment can be described as follows: Figure 1 As shown, it includes: Step 11: Efficiently extract the data to be analyzed for this evaluation from the Hive library. The Hive library is compatible with data from multiple protocols and multiple carriers and supports multi-threaded asynchronous queries.
[0032] In the implementation, the data to be analyzed is stored in a Hive library. When conducting data quality assessment, the first step is to efficiently extract the data to be analyzed from the Hive library. The Hive library is compatible with data from multiple protocols and carriers, and supports multi-threaded asynchronous queries.
[0033] In one example, the server interacts with the Hive library by establishing a multi-threaded asynchronous query connection pool through the Hive JDBC driver, thereby reducing connection overhead. The Hive JDBC driver is specifically designed for the Hive library, allowing Java applications to interact with it via JDBC. Features of the Hive JDBC driver include support for HiveQL, support for the Hadoop file system, and support for Hive-specific data types. While adhering to the JDBC specification, the Hive JDBC driver is optimized for the specific architecture of the Hive library, such as management of MapReduce jobs and optimizations for reading and writing Hive tables.
[0034] In one example, the server automatically generates Hive partition filtering conditions based on the start and end time parameters input by the user. The Hive database is then scanned based on these filtering conditions to narrow down the data scan scope. A typical filtering condition is: `WHERE dt BETWEEN '20230101' AND '20230107'`.
[0035] In one example, after completing a data scan, the server dynamically constructs an HQL statement based on the protocol type (such as HTTP, FTP, etc.) and field rules. For the "null check" rule, it generates `SELECT COUNT(*) FROM tableWHERE field IS NULL`.
[0036] In one example, for the concatenated HQL statement, the server starts the JVM via JPype, calls the pre-compiled Java JAR package to execute the concatenated HQL statement, avoids the performance bottleneck of the native Python Hive library, and thus obtains the data to be analyzed in this evaluation.
[0037] Step 12: Load the general rules in the preset rule base, and at the same time obtain the user-inputted custom rules. Generate the rule base for this evaluation based on all general rules and all custom rules.
[0038] In the specific implementation, after obtaining the data to be analyzed for this evaluation, it is necessary to continue building the rule base for this evaluation. The server loads the general rules in the pre-built rule base, and also needs to obtain the user-inputted custom rules. Finally, the rule base for this evaluation is generated based on all the general rules and all the custom rules.
[0039] It is important to note that not all users have permission to customize rules; users without this permission can only use the general rules in the pre-built rule library.
[0040] In one example, the pre-built rule base contains 38 general rules, including IMSI format verification rules and MSISDN length verification rules. These general rules are categorized and stored in the MySQL rule table according to protocol type (such as HTTP, FTP, etc.).
[0041] In one example, this embodiment designs SQL injection prevention for user-inputted custom rules, that is, it performs whitelist filtering on user-inputted custom rules, allowing only letters, numbers and specific symbols.
[0042] In one example, the server parses the whitelisted custom rules into an abstract syntax tree (AST), extracts key parameters including field names, operators, and thresholds, and finally converts the AST into a parameterized HQL template. This template is then combined with all common rules from the MySQL `rule` table to form the rule base for this evaluation. For example: ```sql SELECT COUNT(*) FROM {table} WHERE {field} {operator} {value}```.
[0043] Step 13: Perform distributed aggregation calculation on the data to be analyzed in this evaluation, and combine it with the rule base of this evaluation to conduct data quality evaluation, and obtain the evaluation result of this evaluation.
[0044] In practice, to improve evaluation efficiency, this embodiment designs a distributed aggregation computing architecture. The server performs distributed aggregation computing on the data to be analyzed in this evaluation, and combines it with the rule base of this evaluation to perform data quality evaluation and obtain the evaluation result of this evaluation.
[0045] In one example, the distributed aggregation computing architecture incorporates Hive MapReduce optimizations, including columnar storage and predicate pushdown. Columnar storage utilizes the Hive ORC format to store data, reducing I / O overhead. Predicate pushdown pushes filtering conditions down to the storage layer, reducing data reads.
[0046] In one example, for intermediate result caching, the server stores Hive query results in Redis with a TTL of 1 hour for reuse in subsequent multi-dimensional analysis. During parallel computing, computation tasks are distributed through a Celery task queue, supporting horizontal scaling of worker nodes.
[0047] Step 14: Visualize the evaluation results and determine the access permissions for different users based on the preset three-level permission model, allowing users to access only within their permissions.
[0048] In the specific implementation, after obtaining the evaluation results, the evaluation results can be visualized through an interactive visual interface, and the access permissions of different users can be determined based on the preset three-level permission model, allowing users to only access within their permissions, thus achieving fine-grained permission management.
[0049] In one example, when visualizing the evaluation results, it supports dynamically generating charts including bar charts, line charts, and heatmaps. The charts contain clickable elements, and when a clickable element (such as an anomaly point on a certain day) is triggered, the corresponding detailed data is queried.
[0050] In one example, the pre-defined three-tier permission model includes protocol level, carrier level, and rule level. Protocol level restricts users to access only authorized protocols (e.g., only HTTP is supported). Carrier level isolates data access between different carriers. Rule level restricts whether users are allowed to define custom rules; authorized users can define custom rules, while unauthorized users can only use general rules from a pre-defined rule base.
[0051] Step 15: Generate an assessment log that supports rapid location based on the entire assessment process.
[0052] In practice, for the entire evaluation process, the server will generate evaluation logs that support rapid location based on the entire evaluation process.
[0053] In one example, throughout the entire evaluation process, the server will collect Django logs and Hive query logs via Logstash, then use Elasticsearch to build a log index and generate evaluation logs, supporting fast retrieval and location based on timestamps, protocol types, and error codes. When three consecutive errors of the same type appear in the evaluation logs, an email alert will be automatically triggered to prompt operations personnel to handle the issue.
[0054] This embodiment proposes a Hive-based data quality assessment method. It efficiently extracts the data to be analyzed from the Hive library. The Hive library is compatible with data from multiple protocols and carriers and supports multi-threaded asynchronous queries, which significantly improves assessment efficiency. The response time for analyzing TB-level data daily is less than 5 minutes, a 6-fold improvement compared to traditional methods. Subsequently, it loads general rules from a pre-built rule base and simultaneously acquires user-inputted custom rules. A rule base for this assessment is generated based on all general and custom rules, effectively enhancing the flexibility of data quality assessment. Next, distributed aggregation computation is performed on the data to be analyzed, and the data quality is assessed using the rule base, yielding the assessment results. Distributed aggregation computation effectively accelerates the analysis process. The assessment results are then visualized, and a pre-defined three-level permission model determines access permissions for different users, allowing only those permissions. This design effectively improves the security of data quality assessment, and the three-level permission model significantly reduces the risk of data leakage. Finally, an assessment log supporting rapid fault location is generated based on the entire assessment process, reducing fault location time by 80%. In summary, this method enables rapid quality assessment of terabyte-scale data, effectively improving the efficiency, scalability, flexibility, and interactivity of data quality assessment, refining the granularity of access control, reducing the risk of data leakage, and ultimately achieving efficient, flexible, and secure data quality assessment.
[0055] The steps described above are for clarity only. In implementation, they can be combined into one step, or some steps can be broken down into multiple steps, as long as they involve the same logical relationship, they are all within the scope of protection of this application. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, without changing the core design of the algorithm and process, are also within the scope of protection of this application.
[0056] Another embodiment of this application proposes a Hive-based parsed data quality assessment system. The details of this Hive-based parsed data quality assessment system are described below. The following content is for ease of understanding and is not essential for implementing this solution. The specific structure of the Hive-based parsed data quality assessment system proposed in this embodiment can be as follows: Figure 2 As shown, it includes: a data acquisition unit 21, a rule management unit 22, a distributed computing unit 23, a visualization and permission control unit 24, and a log integration unit 25.
[0057] The data acquisition unit 21 is used to efficiently extract the data to be analyzed in this evaluation from the Hive library. The Hive library is compatible with data from multiple protocols and multiple operators and supports multi-threaded asynchronous queries.
[0058] The rule management unit 22 is used to load general rules from the pre-set rule base, and at the same time obtain user-inputted custom rules, and generate the rule base for this evaluation based on all general rules and all custom rules.
[0059] The distributed computing unit 23 is used to perform distributed aggregation computing on the data to be analyzed in this evaluation, and to conduct data quality evaluation in combination with the rule base of this evaluation to obtain the evaluation result of this evaluation.
[0060] The visualization and access control unit 24 is used to visualize the evaluation results of this assessment and determine the access permissions of different users based on a preset three-level access model, allowing users to access only those permissions.
[0061] Log integration unit 25 is used to generate evaluation logs that support rapid location based on the entire process of this evaluation.
[0062] It is worth mentioning that all modules and units involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed in this application; however, this does not mean that other units do not exist in this embodiment.
[0063] It is not difficult to see that this embodiment is a system embodiment corresponding to the above method embodiments, and this embodiment can be implemented in conjunction with the above method embodiments. The relevant technical details and technical effects mentioned in the above method embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above method embodiments.
[0064] Another embodiment of this application provides an electronic device, such as Figure 3 As shown, it includes a processor 31 and a memory 32. The memory 32 stores instructions that the processor 31 can execute. When the processor 31 is configured to execute the instructions, the electronic device can implement a Hive-based parsing data quality assessment method as described in the above method embodiment.
[0065] The memory and processor are connected via a bus, which includes any number of interconnecting buses and bridges, connecting various circuits of one or more processors and the memory. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.
[0066] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.
[0067] Another embodiment of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, can implement a Hive-based data parsing quality assessment method as described in the above method embodiments.
[0068] That is, those skilled in the art will understand that all or part of the steps in the above method embodiments can be implemented by a program instructing related hardware. The program is stored in a storage medium and includes several instructions to cause a device (such as a microcontroller, chip, etc.) or processor to execute all or part of the steps of the method described in the method embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.
[0069] Those skilled in the art will understand that the above embodiments are specific implementations of this application, and in practical applications, various changes can be made in form and detail without departing from the spirit and scope of this application. For those skilled in the art, several improvements and modifications can be made without departing from the principles of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.
Claims
1. A method for evaluating the quality of parsed data based on Hive, characterized in that, include: The data to be analyzed in this evaluation is extracted efficiently from the Hive library; the Hive library is compatible with data from multiple protocols and multiple carriers and supports multi-threaded asynchronous queries. Load the general rules from the pre-built rule base, and at the same time obtain the user-inputted custom rules. Generate the rule base for this evaluation based on all general rules and all custom rules. Distributed aggregation calculations were performed on the data to be analyzed in this assessment, and data quality was assessed in conjunction with the rule base of this assessment to obtain the assessment results. The evaluation results are visualized, and the access permissions for different users are determined based on a preset three-level permission model, allowing users to access only what they are authorized to do. Based on this assessment, an assessment log that supports rapid location is generated throughout the entire process.
2. The method for evaluating the quality of parsed data based on Hive according to claim 1, characterized in that, Efficiently extract the data to be analyzed in this evaluation from the Hive library, including: A connection pool supporting multi-threaded asynchronous queries is established through the Hive JDBC driver to interact with the Hive library; Based on the start and end time parameters input by the user, Hive partition filtering conditions are automatically generated, and data scanning of the Hive library is performed based on the Hive partition filtering conditions. Dynamically construct HQL statements based on protocol type and field rules; The JVM is started using Jpype, and the pre-compiled Java JAR package is called to execute the concatenated HQL statements, thereby obtaining the data to be analyzed in this evaluation.
3. The method for evaluating the quality of parsed data based on Hive according to claim 1, characterized in that, The pre-built rule base contains a variety of general rules, including IMSI format verification rules and MSISDN length verification rules, which are stored in the MySQL rule table according to protocol type. Whitelist filtering of user-inputted custom rules, allowing only letters, numbers, and specific symbols; The custom rules filtered by the whitelist are parsed into an abstract syntax tree, and key parameters are extracted, including field names, operators, and thresholds. The parsed abstract syntax tree is converted into a parameterized HQL template, and then combined with all the general rules in the MySQL rule table to form the rule base for this evaluation.
4. The method for evaluating the quality of parsed data based on Hive according to claim 1, characterized in that, When performing distributed aggregation computing, data is stored in Hive ORC format to reduce I / O overhead, filtering conditions are pushed to the storage layer to reduce data reading volume, and computing tasks are distributed through Celery task queues to support horizontal scaling of Worker nodes. Regarding intermediate result caching, Hive query results are stored in Redis with a TTL of 1 hour for reuse in subsequent multi-dimensional analysis.
5. The method for evaluating the quality of parsed data based on Hive according to claim 1, characterized in that, When visualizing the evaluation results, the system supports the dynamic generation of charts including bar charts, line charts, and heatmaps. The charts contain clickable elements, and when a clickable element is triggered, the corresponding detailed data is retrieved.
6. The method for evaluating the quality of parsed data based on Hive according to claim 5, characterized in that, The preset three-level permission model includes protocol level, carrier level, and rule level; Protocol-level restrictions are used to limit users to access only authorized protocols; Carrier-grade equipment is used to isolate data access between different carriers; Rule-level rules are used to restrict whether users are allowed to customize rules.
7. A method for assessing the quality of parsed data based on Hive according to any one of claims 1 to 6, characterized in that, Based on the entire evaluation process, evaluation logs supporting rapid location were generated, including: Throughout the entire evaluation process, Logstash was used to collect Django logs and Hive query logs. Use Elasticsearch to build a log index and generate evaluation logs, supporting fast retrieval and quick location based on timestamps, protocol types, and error codes; An email alert will be automatically triggered when the same type of error appears three times consecutively in the evaluation log.
8. A Hive-based data parsing quality assessment system, characterized in that, include: The data acquisition unit is used to efficiently extract the data to be analyzed in this evaluation from the Hive library. The Hive library is compatible with data from multiple protocols and multiple carriers and supports multi-threaded asynchronous queries. The rule management unit is used to load general rules from the pre-built rule base, and at the same time obtain user-inputted custom rules, and generate the rule base for this evaluation based on all general rules and all custom rules; The distributed computing unit is used to perform distributed aggregation computing on the data to be analyzed in this evaluation, and to conduct data quality assessment in combination with the rule base of this evaluation to obtain the evaluation results of this evaluation. The visualization and access control unit is used to visualize the evaluation results of this assessment and determine the access permissions of different users based on a preset three-level access control model, allowing users to access only what they are authorized to do. The log integration unit is used to generate evaluation logs that support rapid location based on the entire evaluation process.
9. An electronic device, characterized in that, include: A processor and a memory, wherein the memory stores instructions executable by the processor, and the processor is configured to, when executing the instructions, enable the electronic device to implement a Hive-based parsing data quality assessment method as described in any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it can implement a Hive-based data quality assessment method as described in any one of claims 1 to 7.