Method and system for probing original log data quality

Abstract

The invention provides a method and system for probing the data quality of original logs, and relates to the technical field of data probing, and the method comprises the steps: obtaining a plurality of original logs from a data source, and caching the original logs to a local server; the local server forms a probing process according to a plurality of probing rule nodes preset by probing personnel, and performs rule probing on each original log in sequence according to the probing process; and the local server performs processing according to rule probing results of all the original logs to obtain multiple data indexes, generates a probing analysis report and displays the probing analysis report. The method has the beneficial effects that an automatic probing process consisting of a plurality of probing rule nodes is constructed, so that low-efficiency and error-prone manual one-by-one verification is completely replaced, and the probing efficiency and accuracy are greatly improved; a plurality of data indexes and quality reports are generated based on a comprehensive rule exploration result, a set of objective and systematic data quality evaluation mechanism is established, logs with substandard quality are effectively prevented from entering a subsequent processing link, and high later repair cost is avoided.

Description

Technical Field

[0001] This invention relates to the field of data exploration technology, and in particular to a method and system for exploring the quality of raw log data. Background Technology

[0002] In the overall data governance process, data exploration, as a crucial step in the data access phase, bears the important responsibility of initially identifying data sources, testing connectivity, and obtaining raw log content. Traditional data exploration typically includes two core functions: first, exploration access, which verifies the connectivity of the data source and accesses the raw logs; second, manual verification, which involves manually checking the format, field content validity, and other aspects of the raw logs line by line. Only logs that have passed exploration and been confirmed to be of acceptable quality can proceed to subsequent data processing and storage stages.

[0003] However, existing data exploration methods have significant drawbacks, mainly in the following aspects: 1. Manual verification is inefficient and inaccurate: Manually verifying each original log is not only time-consuming and labor-intensive, but also makes it difficult to systematically obtain data quality indicators that require batch statistics, such as fill rate, resulting in a lack of comprehensiveness and objectivity in the investigation results.

[0004] 2. Inability to effectively evaluate the effect of log standardization: Due to the complexity and diversity of log formats, manual verification is difficult to simulate the actual effect of logs after standardization, and it is also impossible to predict their final form in the database, thus increasing the difficulty and uncertainty of data quality assessment.

[0005] 3. Impact on subsequent data processing performance and stability: The accuracy of manual probing is limited, which can easily lead to substandard logs flowing into the data processing module, resulting in a large number of parsing failures and log loss. This not only affects processing efficiency but may also cause a waste of system resources.

[0006] 4. Data quality is difficult to guarantee, and repair costs are high: If problematic logs are not effectively identified during the investigation phase, low-quality data will be imported into the database. Subsequent repairs often require data updates or reprocessing, which is complex and costly.

[0007] In summary, existing data exploration technologies heavily rely on manual intervention and lack automated, systematic quality assessment mechanisms, which has become one of the bottlenecks restricting the efficiency and quality improvement of data governance. Summary of the Invention

[0008] To address the problems existing in the prior art, this invention provides a method for probing the quality of raw log data, comprising: Step S1: Obtain multiple raw logs from the data source and cache them on the local server; Step S2: The local server forms an exploration process based on multiple exploration rule nodes pre-set by the explorers, and sequentially performs rule exploration on each of the original logs according to the exploration process. Step S3: The local server processes the results of the rule-based exploration of all the original logs to obtain multiple data indicators, generates an exploration analysis report, and displays it to the exploration personnel.

[0009] Preferably, the exploration rule nodes include: The data extraction node is used to standardize the data according to the structure type of the original log and map the standardized data to a preset target table. The format validation node is used to validate each field in the target table according to the field requirements to obtain multiple data indicators.

[0010] Preferably, the exploration rule node further includes: The data cleaning node is used to filter the raw logs according to the filtering requirements input by the investigators. And / or data association nodes, used to supplement the original log with data.

[0011] Preferably, the data structure type in the original log includes structured data, and / or semi-structured data, and / or unstructured data; The data extraction nodes include: JSON decoding nodes are used to standardize structured data; The CSV decoding node is used to standardize semi-structured data; The GROK decoding node is used to standardize unstructured data; Field mapping nodes are used to map the standardized data to the target table.

[0012] Preferably, the data cleaning node includes: A pre-filtering node is used to filter the raw logs according to the filtering requirements input by the investigators before the data extraction node is executed; And / or a post-filtering node, used to filter the raw logs according to the filtering requirements input by the investigators after the data extraction node is executed.

[0013] Preferably, the field requirements include field null value requirements, field format requirements, field type requirements, and field value range requirements; the data indicators include null value rate, format anomaly rate, type anomaly rate, and value range anomaly rate. The format validation node is used to calculate the null value rate based on all fields in the target table that do not meet the field null value requirement, the format anomaly rate based on all fields in the target table that do not meet the field format requirement, the type anomaly rate based on all fields in the target table that do not meet the field type requirement, and the value range anomaly rate based on all fields in the target table that do not meet the field value range requirement.

[0014] Preferably, step S3 includes: When data extraction fails during the execution of the data extraction node in any of the original logs, the local server records one parsing failure count; and When the local server encounters a field that does not meet the requirements during the execution of the format validation node in any of the original logs, it records the number of validation failures. When the local server executes the probe process without error in any of the original logs, it records the number of successful verifications; and The local server records the number of parsing failures, the number of verification failures, the number of verification successes, the reasons for the failures in the original logs of parsing failures, and the data indicators in the original logs of verification failures as the exploration and analysis report for display.

[0015] Preferably, a sample testing process is included before performing step S1, including: The local server initiates multiple connectivity requests with the data source based on the data source type using various communication protocols to test connectivity with the data source. After the connectivity test is passed, log samples are obtained from the data source for display testing, and corresponding probing restrictions are set according to the data source type.

[0016] Preferably, in step S2, the local server generates a corresponding node snapshot when executing each of the exploration rule nodes.

[0017] This invention provides a system for probing the quality of raw log data, applying the above-described method and running on the local server. The system includes: The log acquisition module is used to retrieve multiple raw logs from the data source and cache them on the local server. The log exploration module, connected to the log acquisition module, is used to form an exploration process based on multiple exploration rule nodes pre-set by the explorer, and to sequentially explore each of the original logs according to the exploration process. The analysis report module, connected to the log exploration module, is used to process the exploration results based on the rules of all the original logs to obtain multiple data indicators, generate an exploration analysis report, and display it.

[0018] The above technical solution has the following advantages or beneficial effects: 1. By constructing an automated exploration process consisting of multiple exploration rule nodes, the system completely replaces the inefficient and error-prone manual check-by-check. This system can perform batch and rapid rule scanning and quality inspection of logs, which not only significantly improves exploration efficiency and accuracy, but also makes it possible to systematically evaluate the standardization effect and expected storage format of logs, thereby identifying and intercepting quality problems in the early stages of data processing.

[0019] 2. Based on comprehensive rule exploration results, multiple data indicators and quality reports are generated, establishing an objective and systematic data quality assessment mechanism. This enables timely insight into data health status for data rectification or rule optimization. It ensures the quality of incoming data, effectively preventing substandard logs from entering subsequent processing stages, thus avoiding parsing failures, data loss, processing performance degradation, and high post-repair costs. Attached Figure Description

[0020] Figure 1 A flowchart illustrating a method for probing the quality of raw log data is provided in a preferred embodiment of the present invention. Figure 2 This is a schematic diagram of a system for probing the quality of raw log data, as a preferred embodiment of the present invention. Detailed Implementation

[0021] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. The present invention is not limited to this embodiment; other embodiments that conform to the spirit of the present invention may also fall within the scope of the present invention.

[0022] In a preferred embodiment of the present invention, based on the above-mentioned problems existing in the prior art, a method for probing the quality of raw log data is provided, such as... Figure 1 As shown, it includes: Step S1: Obtain multiple raw logs from the data source and cache them on the local server; Step S2: The local server forms an exploration process based on multiple exploration rule nodes pre-set by the explorers, and performs rule exploration on each original log in sequence according to the exploration process; Step S3: The local server processes the results of the investigation based on the rules of all the original logs to obtain multiple data indicators, generates an investigation analysis report, and displays it to the investigators.

[0023] Specifically, this embodiment addresses the problems of traditional data exploration by eliminating the manual exploration portion of traditional exploration schemes and instead using multiple exploration rule nodes to form an exploration process, thereby achieving automated rule exploration of raw logs.

[0024] Specifically, in step S1 of this embodiment, the local server retrieves multiple raw logs from the data source and caches them locally, achieving batch log retrieval and avoiding the inefficiency of manual collection of each log. Furthermore, existing technologies generally employ real-time probing of the data source, which continuously consumes the data source's computing, storage, and I / O resources. Especially in scenarios with large data volumes or high concurrency, this can significantly impact the data source's business performance. This solution, by retrieving and caching raw logs locally, decouples subsequent intensive rule probing and metric calculation processes from the data source, allowing probing operations to be completed independently locally. This avoids continuous resource consumption and performance interference with the data source, ensuring the stability and availability of the data source system.

[0025] This solution also avoids the problem of delays or interruptions in the exploration process caused by network fluctuations and data source load. After caching the data locally, all subsequent analysis operations are performed in a stable local environment, eliminating the uncertainty caused by network transmission. This makes the entire exploration process more stable, responds more quickly, and improves overall exploration efficiency.

[0026] In step S2 of this embodiment, the local server connects multiple probe rule nodes to form an automated probe process, sequentially probing the original logs according to the rules. This replaces manual verification, solving the problems of low efficiency and error-proneness of manual methods, and provides a foundation for batch analysis. It can identify and isolate logs that do not meet quality standards in advance, promptly notify the data source for data rectification, and prevent them from entering the data processing module, thereby reducing the risk of subsequent parsing failures, log loss, and performance impact.

[0027] In step S3, the local server generates multiple data metrics and analysis reports based on the rule-based probing results of all logs, achieving a systematic and objective quality assessment and improving accuracy and comprehensiveness. The local server can adjust data sources or rules in a timely manner based on data metrics, ensuring that only high-quality logs enter the database process, improving data quality from the source and avoiding costly repair operations such as updates or reprocessing due to low-quality data entering the database.

[0028] In a preferred embodiment of the present invention, a sample testing process is further included before performing step S1, comprising: The local server initiates multiple connection requests with the data source using various communication protocols based on the data source type to test the connectivity with the data source. After the connectivity test is passed, log samples are retrieved from the data source for display testing, and corresponding probing restrictions are set according to the data source type.

[0029] Specifically, existing technologies typically only allow data probing of structured data sources in databases, while this application supports data probing of data sources including structured, semi-structured, and unstructured data such as syslog, Kafka, JDBC, HTTP(S), and local files. Furthermore, before data probing, it initiates multiple connectivity requests to test the connectivity with different data sources by calling different communication protocols, ensuring that at least one connectivity method enables the network connectivity with the data source to meet the conditions for log probing.

[0030] Furthermore, after the connectivity test passes, log samples are retrieved and displayed. This allows users to intuitively see the actual content, format, and structure of the raw logs (e.g., whether it's a JSON string or CSV format) before configuring complex probing rules. This avoids the predicament of being unable to effectively configure rules when the raw log structure is unclear. By previewing the samples, users can design and select subsequent probing rules in a targeted manner (e.g., choosing GROK decoding or JSON decoding), greatly improving the accuracy and efficiency of rule configuration and reducing the difficulty of data quality assessment from the source.

[0031] Furthermore, to reduce the pressure on data sources caused by probing, such as excessive probing volume or continuous probing, this invention sets probing restrictions, probing quantity limits, and probing timeout limits for each type of data source, thereby reducing probing pressure. For example, for JDBC probing, filtering conditions can be set for database partition fields to reduce full table scans; for Kafka probing, logs can be retrieved starting from the current position or the starting position, and a timeout can be set to reduce the pressure caused by excessive Kafka probing volume or continuous probing due to the inability to find the target data.

[0032] In step S2 of a preferred embodiment of the present invention, a corresponding node snapshot is generated when each exploration rule node is executed.

[0033] Specifically, in this embodiment, each rule node (such as data extraction, cleaning, and association) generates a node snapshot during execution. This snapshot records the complete state of log data before it enters the node and after it has been processed by the node. Existing technologies are like a black box; investigators can only see the input and the final output, but cannot know how the data changed in between. This invention, through node snapshots, presents the entire standardization processing chain completely and clearly to investigators, enabling them to intuitively assess the specific impact of each rule on the log data. This completely solves the problem that manual verification cannot detect the effects of log standardization and cannot see the final results of data entry into the database.

[0034] Furthermore, investigators can verify the rationality of their designed Directed Acyclic Graph (DAG) and the optimal configuration of each rule plugin by observing the status changes of the logs at each stage. And when the final investigation results are unsatisfactory or a large number of parsing failures occur, investigators can backtrack these node snapshots to precisely pinpoint the rule node where the problem first appeared.

[0035] The detection rule nodes in a preferred embodiment of the present invention include: The data extraction node is used to standardize the data according to the structure type of the original log and map the standardized data to a preset target table. The format validation node is used to validate each field in the target table according to the field requirements to obtain multiple data indicators.

[0036] In a preferred embodiment of the present invention, the data structure types in the original log include structured data, and / or semi-structured data, and / or unstructured data; The data extraction nodes include: JSON decoding nodes are used to standardize structured data; The CSV decoding node is used to standardize semi-structured data; The GROK decoding node is used to standardize unstructured data; Field mapping nodes are used to map standardized data to the target table.

[0037] The preferred embodiments of the present invention include field null value requirements, field format requirements, field type requirements, and field value range requirements; data indicators include null value rate, format anomaly rate, type anomaly rate, and value range anomaly rate. The format validation node is used to calculate the null value rate based on all fields in the target table that do not meet the null value requirement, the format anomaly rate based on all fields in the target table that do not meet the format requirement, the type anomaly rate based on all fields in the target table that do not meet the type requirement, and the value range anomaly rate based on all fields in the target table that do not meet the value range requirement.

[0038] Specifically, in this embodiment, data extraction nodes standardize data of different structural types in the original logs. This process automatically identifies and processes structured, semi-structured, and unstructured data. For example, JSON decoding nodes process structured data, CSV decoding nodes process semi-structured data, and GROK decoding nodes parse unstructured data, ultimately mapping all data to a preset target table. This directly solves the inefficiency problem caused by the complex and diverse log formats in traditional manual verification, transforming the tedious process of manually parsing each log entry into automated batch processing, greatly improving data exploration efficiency.

[0039] More specifically, the format validation node systematically validates each field in the target table according to preset field requirements, including requirements for null values, format, type, and value range. Based on this, it generates quantitative indicators such as null value rate, format anomaly rate, type anomaly rate, and value range anomaly rate. This allows for a comprehensive quality assessment of standardized data during the exploration phase before formal data entry, addressing the pain point of traditional methods' inability to accurately obtain batch analysis indicators such as fill rate. By identifying data that does not meet entry standards in advance, it effectively prevents low-quality logs from entering subsequent processing stages, ensuring the quality of the final entered data from the source.

[0040] The collaborative work of the data extraction node and the format verification node forms a complete closed loop for data quality control. The data extraction node is responsible for converting the raw data into a standardized format, while the format verification node quantifies and evaluates the conversion results, generating specific quality indicators. This design allows the local server to clearly understand the actual effect of the raw logs after standardization, accurately assess the degree of data matching with the target system, and significantly reduce the difficulty of data quality assessment caused by the inability to predict the standardization effect in traditional methods.

[0041] In a preferred embodiment of the present invention, the detection rule node further includes: The data cleaning node is used to filter the raw logs according to the filtering requirements input by the investigators. And / or data association nodes, used to supplement the original log with data.

[0042] In a preferred embodiment of the present invention, the data cleaning node includes: The pre-filter node is used to filter the raw logs based on the filtering requirements input by the investigators before the data extraction node is executed; And / or post-filtering nodes, used to filter the raw logs based on the filtering requirements input by the investigators after the data extraction nodes are executed.

[0043] Specifically, in this embodiment, the data cleaning node enables precise filtering of raw logs at different stages of data processing based on the filtering requirements set by the investigators. For example, invalid data is removed by a pre-filter node before data standardization, and records that do not meet the requirements are removed by a post-filter node based on combined conditions after standardization. This hierarchical filtering mechanism ensures that only high-quality, relevant data enters subsequent processing stages, significantly reducing the amount of data processing and improving overall investigation efficiency, while also effectively preventing invalid data from interfering with subsequent analysis and data entry processes.

[0044] Through data association nodes, when other complete information tables are pre-configured in the data source, content can be extracted from these tables, and information can be intelligently supplemented into the standardized fields. Data association methods include static value mapping, dynamic data mapping, SQL mapping, geolocation mapping, and Redis mapping. Static value mapping plugins are used for association matching with static data; dynamic data mapping is used for association matching with data in modifiable value mapping tables; SQL mapping is used for association matching with data in database tables; geolocation mapping is used for association matching with location information; and Redis mapping is used for association matching with data in Redis. For example, if the security audit log only records the operator's ID number, by configuring an SQL mapping node, the system will automatically connect to the personnel information database, using the ID number as the association key, to query and supplement the person's name, gender, department, and other detailed information in real time. These supplemented fields will then be combined with the original log to form a complete standardized log.

[0045] This data augmentation capability effectively supplements missing or incomplete fields in the original logs, significantly improving data integrity and usability, and providing a richer and more accurate data foundation for subsequent data analysis and applications.

[0046] The exploration process in this invention is described in the form of a DAG process by arranging and combining multiple different rule exploration nodes, which is the final implementation of the exploration rules.

[0047] In a preferred embodiment of the present invention, step S3 includes: The local server records the number of parsing failures when data extraction fails during any raw log data extraction node process; and The local server records the number of validation failures when any field does not meet the format validation requirements during the execution of format validation nodes in any raw log; and The local server records the number of successful verifications when the probing process is successful in any raw log file; and The local server records the number of parsing failures, the number of verification failures, the number of verification successes, the reasons for the failures in the original logs of parsing failures, and the data metrics in the original logs of verification failures as an investigation and analysis report for display.

[0048] Specifically, in this embodiment, for each field in the target table, the data type, whether it is required, its length, and its value range are defined. By explicitly defining four core quality dimensions—field null value requirements, field format requirements, field type requirements, and field value range requirements—and accordingly calculating four quantitative indicators—null value rate, format anomaly rate, type anomaly rate, and value range anomaly rate—a comprehensive evaluation framework covering key aspects of data quality is constructed. This innovative design transforms data quality assessment from traditional subjective judgment to objective quantitative analysis, enabling precise identification of specific problems in data integrity, standardization, consistency, and validity, providing a clear direction for data quality improvement.

[0049] Furthermore, a complete quality status classification system was established by recording the number of parsing failures during the data extraction stage, the number of verification failures during the format verification stage, and the number of successful verifications for normally processed data. This design not only provides an overall overview of data quality, but more importantly, it enables precise differentiation of the sources of quality problems. It can clearly distinguish whether the problem is caused by parsing errors during the data extraction process or by data content that does not meet quality requirements, providing a clear path for subsequent problem investigation and rectification.

[0050] By including the original log content of parsing failures and their specific reasons for failure, as well as the logs of verification failures and their corresponding abnormal indicators in the investigation and analysis report, data governance personnel can not only understand which data is problematic, but also gain a deeper understanding of what the specific problems are and why they occur.

[0051] Therefore, by establishing multi-dimensional quality indicators, implementing full-process quality tracking, and providing in-depth problem analysis, this invention constructs a complete closed loop for data quality diagnosis and optimization, providing a solid technical guarantee for improving data source quality, ensuring data processing stability, and ensuring the reliability of the final data entering the database.

[0052] This invention provides a system for probing the quality of raw log data. The method described above runs on a local server, such as... Figure 2 As shown, the system includes: Log acquisition module 1 is used to retrieve multiple raw logs from the data source and cache them on the local server. Log exploration module 2 is connected to log acquisition module 1. It is used to form an exploration process based on multiple exploration rule nodes pre-set by the explorers, and to conduct rule exploration on each original log in sequence according to the exploration process. The analysis report module 3 connects to the log exploration module 2. It is used to process the exploration results based on the rules of all raw logs to obtain multiple data indicators, generate exploration analysis reports, and display them.

[0053] The above are merely preferred embodiments of the present invention and are not intended to limit the implementation methods and protection scope of the present invention. Those skilled in the art should recognize that any equivalent substitutions and obvious changes made using the content of this specification and illustrations should be included within the protection scope of the present invention.

Claims

1. A method of exploring raw log data quality, characterized by, The application relates to a method for generating a probe analysis report, and belongs to the technical field of probe analysis. The method comprises the following steps: S1, a plurality of original logs are remotely acquired from a data source and cached to a local server; S2, a probe flow is formed according to a plurality of probe rule nodes set by a probe personnel, and each original log is sequentially subjected to rule probing according to the probe flow; 2. The method of claim 1, wherein, S3, a plurality of data indexes are obtained by processing rule probing results of all the original logs, a probe analysis report is generated, and the probe analysis report is displayed to the probe personnel. The probe rule nodes comprise: a data extraction node, which is used for standardizing data in the original logs according to a structure type of the data and mapping the standardized data to a preset target table; 3. The method of claim 2, wherein, a format verification node, which is used for verifying each field in the target table according to field requirements to obtain a plurality of data indexes. The probe rule nodes further comprise: a data cleaning node, which is used for filtering the original logs according to filtering requirements input by the probe personnel; and / or a data association node, which is used for supplementing data of the original logs.

4. The method of claim 2, wherein, The structure type of the data in the original logs comprises structured data, semi-structured data and unstructured data; The data extraction node comprises: a JSON decoding node, which is used for standardizing the structured data; a CSV decoding node, which is used for standardizing the semi-structured data; a GROK decoding node, which is used for standardizing the unstructured data; and a field mapping node, which is used for mapping the standardized data to the target table.

5. The method of claim 2, wherein, The data cleaning node comprises: a pre-filtering node, which is used for filtering the original logs according to filtering requirements input by the probe personnel before the data extraction node is executed; and / or a post-filtering node, which is used for filtering the original logs according to filtering requirements input by the probe personnel after the data extraction node is executed. The field requirements comprise field null value requirements, field format requirements, field type requirements and field value domain requirements; and the data indexes comprise a null value rate, a format exception rate, a type exception rate and a value domain exception rate.

6. The method of claim 2, wherein, The format verification node is used for calculating the null value rate according to all fields in the target table that do not meet the field null value requirements, calculating the format exception rate according to all fields in the target table that do not meet the field format requirements, calculating the type exception rate according to all fields in the target table that do not meet the field type requirements, and calculating the value domain exception rate according to all fields in the target table that do not meet the field value domain requirements. The step S3 comprises:

7. The method of claim 6, wherein, the local server records a number of times of parsing failure when data extraction fails in the process of executing the data extraction node on any one of the original logs; and the local server records a number of times of verification failure when there is a field that does not meet the field requirements in the process of executing the format verification node on any one of the original logs; and the local server records a number of times of verification success when the probe flow is executed on any one of the original logs without error; and the local server records a number of times of verification success when the probe flow is executed on any one of the original logs without error. The local server displays the resolution failure number, the verification failure number, the verification success number, the failure reason of the original log of resolution failure, and the data index record of the original log of verification failure as the probe analysis report.

8. The method of claim 1, wherein, Before the step S1, a sample test process is further included, comprising: The local server initiates multiple connectivity test requests to the data source according to multiple different communication protocols corresponding to the data source type of the data source, and obtains log samples from the data source for display test after the connectivity test is passed, and sets corresponding probe limit conditions according to the data source type.

9. The method of claim 1, wherein, In the step S2, the local server generates a corresponding node snapshot when executing each probe rule node.

10. A system for exploring raw log data quality, characterized by, The system comprises: A log acquisition module for acquiring multiple original logs from a data source and caching the original logs to the local server; A log probe module connected to the log acquisition module, configured to form a probe process according to multiple probe rule nodes set by a probe personnel in advance, and sequentially perform rule probe on each original log according to the probe process; An analysis report module connected to the log probe module, configured to process multiple data indexes according to the rule probe results of all original logs to generate a probe analysis report and display the probe analysis report.

Images