Real-time data quality evaluation method and device, medium and program product

Abstract

The invention discloses a real-time data quality evaluation method and device, a medium and a program product, and relates to the technical field of data processing, and the method comprises the steps: obtaining multi-source monitoring data of a cluster, the multi-source monitoring data comprising monitoring index data of a server, production log data of a production end and consumption log data of a consumption end; respectively applying respective preset diagnosis rules to the monitoring index data, the production log data and the consumption log data to generate health scores of the terminals; and obtaining a data quality score of each end by fusing the server health score, the production end health score and the consumption end health score, and determining the health degree of the cluster according to the data quality scores. According to the method, the monitoring data of the server, the production end and the consumption end of the cluster are converted into quantifiable health state values, so that continuous and stable operation of services driven by real-time data can be guaranteed.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to real-time data quality assessment methods, electronic devices, storage media, and computer program products. Background Technology

[0002] Real-time data is increasingly valuable in modern work environments, especially in scenarios requiring rapid response, dynamic decision-making, and automated processing. Kafka (a distributed stream processing platform) clusters play a crucial role in modern real-time data technology architectures, serving as the "main artery" of data flow and the "nervous system" of business operations. Ensuring their high availability, high performance, and reliability is the cornerstone of business continuity. Therefore, assessing the quality of real-time data within a Kafka cluster is essential to guaranteeing continuous, stable, efficient, and reliable business operations. However, existing open-source monitoring systems (such as Kafka Manager) only provide single-metric monitoring visualization and threshold alerts on the service side, lacking a quantitative assessment of the overall health of the cluster. This makes it difficult for businesses relying on real-time data to operate stably. Summary of the Invention

[0003] The main purpose of this application is to provide a real-time data quality assessment method, electronic device, storage medium, and computer program product, which aims to solve the technical problem that existing open-source monitoring systems make it difficult for businesses that rely on real-time data to operate stably.

[0004] To achieve the above objectives, this application proposes a real-time data quality assessment method, which includes:

[0005] Obtain multi-source monitoring data of the cluster, wherein the multi-source monitoring data includes monitoring indicator data of the server, production log data of the production end, and consumption log data of the consumer end;

[0006] Each of the monitoring indicator data, production log data, and consumption log data is applied with its own preset diagnostic rules to generate a health score for each end. The health score for each end includes a server-side health score, a production-side health score, and a consumption-side health score.

[0007] The data quality score of each end is obtained by integrating the server-side health score, the production-side health score, and the consumer-side health score, and the health of the cluster is determined based on the data quality score.

[0008] In one embodiment, before the step of applying preset diagnostic rules to the monitoring indicator data, production log data, and consumption log data to generate health scores for each terminal, the following steps are included:

[0009] Based on a rolling time window, the system aggregates monitoring metrics data from the server, parses event types from production logs from the production side, and parses consumption events from consumption logs from the consumption side.

[0010] In one embodiment, the step of applying preset diagnostic rules to the monitoring indicator data to generate a server-side health score includes:

[0011] Based on the first diagnostic rule preset in the server, the aggregated monitoring indicator data is detected and a first detection result for each monitoring indicator data is generated. The first diagnostic rule includes a threshold condition for each monitoring indicator data and a preset score for each monitoring indicator data when the threshold condition is met.

[0012] Based on the first detection result and the preset score for each monitoring indicator, the server health score is determined.

[0013] In one embodiment, the step of applying preset diagnostic rules to the production log data to generate a production-side health score includes:

[0014] Based on the second diagnostic rule preset in the production end, the production risk feature data obtained by parsing the event type is detected, and a second detection result is generated for each production risk feature data. The second diagnostic rule includes the threshold condition for each production risk feature data and the preset score for each production risk feature data when the threshold condition is met.

[0015] Based on the second detection result and the preset score for each production risk characteristic data, the production health score is determined.

[0016] In one embodiment, the step of applying preset diagnostic rules to the consumption log data to generate a consumer health score includes:

[0017] Based on the third diagnostic rule preset in the consumer end, the abnormal consumption data obtained by parsing consumption events is detected, and a third detection result is generated for each abnormal consumption data. The third detection result includes the threshold condition for each abnormal consumption data and the preset score for each abnormal consumption data when the threshold condition is met.

[0018] Based on the third detection result and preset score for each abnormal consumption data, a consumer health score is determined.

[0019] In one embodiment, the step of obtaining the data quality score for each end by fusing the server-side health score, the production-side health score, and the consumer-side health score includes:

[0020] The server health score is multiplied by the server's first weight to obtain the server data quality score;

[0021] The production-side health score is multiplied by the first weight of the production side, and then combined with the server-side health score and the second weight of the server side to obtain the production-side data quality score.

[0022] The consumer health score is multiplied by the first weight of the consumer side, and then the production health score is multiplied by the second weight of the production side to obtain the consumer data quality score.

[0023] In one embodiment, the method includes:

[0024] Collect historical multi-source monitoring data and historical health status of the cluster to construct a training dataset. The historical multi-source monitoring data includes historical monitoring indicator data, historical production log data, and historical consumption log data.

[0025] Based on the training dataset, a prediction model is trained, wherein the input feature is historical multi-source monitoring data and the output feature is the health of the cluster.

[0026] The predicted health of the cluster is output by inputting real-time multi-source monitoring data into the prediction model.

[0027] Furthermore, to achieve the above objectives, this application also proposes a real-time data quality assessment device, which includes:

[0028] The acquisition module is used to acquire multi-source monitoring data of the cluster, wherein the multi-source monitoring data includes monitoring indicator data of the server, production log data of the production end, and consumption log data of the consumer end;

[0029] The diagnostic module is used to apply their respective preset diagnostic rules to the monitoring indicator data, production log data, and consumption log data to generate health scores for each end. The health scores for each end include server-side health scores, production-side health scores, and consumption-side health scores.

[0030] The fusion module is used to obtain the data quality score of each end by merging the server-side health score, the production-side health score, and the consumer-side health score, and to determine the health of the cluster based on the data quality score.

[0031] In addition, to achieve the above objectives, this application also proposes an electronic device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the real-time data quality assessment method as described above.

[0032] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the real-time data quality assessment method described above.

[0033] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the real-time data quality assessment method described above.

[0034] One or more technical solutions proposed in this application have at least the following technical effects:

[0035] Existing open-source monitoring systems can only issue single-metric threshold alerts for the Kafka cluster's server side, and cannot quantitatively assess the overall health status of the cluster. This creates blind spots in the detection of production-side failures and consumer-side potential problems, making it difficult for businesses that rely on real-time data to operate stably.

[0036] This application acquires multi-source monitoring data from three ends of the cluster (monitoring metrics data from the server, production log data from the production end, and consumption log data from the consumer end), and then calculates server health score, production health score, and consumer health score according to the preset diagnostic rules of each end. Finally, it performs a weighted fusion calculation of the server health score, production health score, and consumer health score based on preset weights to generate a data quality score for each end, and determines the health of the cluster based on the data quality score. The entire process can transform discrete multi-source monitoring data into quantifiable health status values, breaking through the one-sidedness of traditional single-end monitoring, and ultimately achieving accurate quantitative assessment of the health of the Kafka cluster, proactively covering potential risks on the production and consumption ends, and ensuring the continuous and stable operation of business driven by real-time data. Attached Figure Description

[0037] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0038] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 A flowchart illustrating an embodiment of the real-time data quality assessment method of this application;

[0040] Figure 2A flowchart illustrating yet another embodiment of the real-time data quality assessment method of this application;

[0041] Figure 3 A simplified flowchart illustrating the real-time data quality assessment method of this application;

[0042] Figure 4 This is a schematic diagram of the module structure of the real-time data quality assessment device of this application;

[0043] Figure 5 This is a schematic diagram of the equipment structure of the hardware operating environment involved in the real-time data quality assessment method of this application.

[0044] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0045] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0046] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0047] It should be noted that the executing entity in this embodiment can be an electronic device, or a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or processor capable of performing the above functions. The following description uses an electronic device as an example to illustrate this embodiment and the subsequent embodiments.

[0048] Based on this, embodiments of this application provide a real-time data quality assessment method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the real-time data quality assessment method of this application.

[0049] In this embodiment, the real-time data quality assessment method includes steps S10 to S30:

[0050] Step S10: Obtain multi-source monitoring data of the cluster, including monitoring indicator data of the server, production log data of the production side, and consumption log data of the consumer side.

[0051] It should be noted that in step S10, obtaining multi-source monitoring data of the cluster refers to synchronously collecting heterogeneous data sources that constitute the entire chain operation status of the Kafka cluster: monitoring indicator data of the server side, such as replica synchronization status, log flushing status, etc.; production log data of the producer side, including logs recorded by the producer during the message push process, such as data sending normality, data sending delay, and traffic fluctuation; and consumption log data of the consumer side, including logs generated by the consumer client when pulling messages, such as consumption delay time, message expiration, and rebalancing count.

[0052] The three types of data correspond to the writing (production end), storage / distribution (server end), and reading (consumer end) stages of the message lifecycle, respectively. By collecting data from the three ends in a coordinated manner, the limitations of traditional single-end monitoring can be overcome.

[0053] For example, the acquisition of server-side monitoring metric data can be achieved by: exposing monitoring metrics through server-side JMX (Java Management Extensions), developing a Java program, creating a JMXconnector to collect monitoring metric data in batches, and outputting it to the metric topic (table); or by directly acquiring it through a port; the specific method of acquisition is not limited. The acquisition of production log data on the production side and consumption log data on the consumption side can be achieved by: collecting logs through Flume (a data acquisition system) and outputting them to the corresponding log topic; or by integrating a monitoring SDK into the production / consumer code to proactively report log events to the corresponding log topic at critical operation nodes (such as message sending failures or rebalancing triggers); the specific method of acquisition is not limited.

[0054] Step S20: Apply the preset diagnostic rules to the monitoring indicator data, production log data, and consumption log data respectively to generate health scores for each end. The health scores for each end include server-side health scores, production-side health scores, and consumption-side health scores.

[0055] It should be noted that, for the three types of heterogeneous data in the Kafka cluster—server, producer, and consumer—each is diagnosed using its own preset diagnostic rules. These diagnostic rules include threshold conditions and corresponding scores for the detection items (i.e., monitoring indicator data, production risk feature data in production log data, and consumption anomaly data in consumption log data) of the server, producer, and consumer.

[0056] For monitoring metrics data, such as Broker (node) load, Kafka-GC (Garbage Collection) status, Kafka request normality, TCP connection status, etc., based on the server-side preset first diagnostic rules, the monitoring metrics data are checked (e.g., checking if the Broker load is less than 80%). The result of each check (1 or 0, where 1 indicates a pass and 0 indicates a fail) is multiplied and summed with the corresponding score to obtain the server-side health score. For production log data, production risk characteristic data (such as data sending normal, data sending delay, etc.) is parsed. Based on the production-side preset second diagnostic rules, production risk characteristic data is checked (e.g., whether the time written to Kafka minus the business time is greater than 2 seconds). The result of each check is multiplied and summed with the corresponding score to obtain the production-side health score. The consumption log data is parsed to obtain consumption anomaly data (such as consumption delay time, message expiration, rebalancing count, etc.). Based on the third diagnostic rules preset on the consumer side, the detection items, i.e. consumption anomaly data, are detected (e.g., whether the consumption delay time is greater than 2 seconds). Then, the detection results of each detection item are multiplied and accumulated with the corresponding item's score to obtain the consumer health score.

[0057] In one possible implementation, prior to step S20, the following is included:

[0058] Based on a rolling time window, the system aggregates monitoring metrics data from the server, parses event types from production logs from the production side, and parses consumption events from consumption logs from the consumption side.

[0059] It's important to note that the various rate metrics of the Kafka cluster are actually calculated using a 30-second rolling window. Inconsistent collection intervals may lead to inaccurate data. The rolling time window approach divides data processing time into fixed intervals (e.g., 10 minutes) to align the timeliness of data across the three ends of the Kafka cluster. Server-side monitoring metrics are aggregated (e.g., using mean / maximum values ​​within the window) to eliminate instantaneous fluctuations and generate a stable set of metrics reflecting the cyclical state. Production log data is parsed using event type analysis (e.g., regular expression matching or keyword extraction) to transform unstructured logs into standardized production events. Consumer log data is parsed using consumption events to extract key consumption behavior semantics.

[0060] In this implementation, rolling time windows are used to process multi-source monitoring data in a time-series manner, providing input data with time-aligned dimensions and a unified structure, thus improving the accuracy and completeness of real-time data quality assessment. Dynamic aggregation of server-side monitoring metrics within the window eliminates interference from instantaneous fluctuations; window parsing of production log event types overcomes the limitations of traditional single-point alarms, exposing related faults on the production end caused by itself or other ends; window parsing of consumption behavior events eliminates the risk of frequent rebalancing and message expiration that the consumer end cannot detect. These three elements together construct a time-series chain covering the entire data production, transmission, and consumption chain, enabling the integration of server-side resource status, production-side write reliability, and consumer-side processing robustness with a unified time benchmark. Ultimately, this transforms fragmented monitoring into a systematic and quantitative assessment, providing highly confident quantitative data for proactive defensive operations and maintenance.

[0061] In one embodiment, the step of applying preset diagnostic rules to monitoring indicator data to generate a server-side health score includes:

[0062] Based on the first diagnostic rule preset in the server, the aggregated monitoring indicator data is detected and a first detection result for each monitoring indicator data is generated. The first diagnostic rule includes the threshold condition for each monitoring indicator data and the preset score for each monitoring indicator data when the threshold condition is met.

[0063] Based on the first detection result and preset score of each monitoring indicator data, the server health score is determined.

[0064] It should be noted that, based on the first diagnostic rule (the preset threshold conditions and preset scores for each monitoring indicator data), the aggregated monitoring indicator data is tested—for example, comparing whether the CPU utilization in the Broker load exceeds the critical threshold of 80%, judging whether the Kafka-gc status is less than 2 seconds / minute, etc., to generate the first detection result. The first detection result can be 1 or 0, where 1 is a pass and 0 is a fail. Finally, each first detection result is calculated with the preset score in the first diagnostic rule (such as CPU score 20, GC score 10) to output a quantitative server health score.

[0065] In this implementation, the aggregated monitoring indicator data is tested using a preset first diagnostic rule, achieving a quantitative assessment of the server's health status. The aggregated monitoring indicator data in the scrolling window is compared with the threshold conditions in the first diagnostic rule to generate quantitative detection results representing the operational status of each dimension. Based on a preset score, a score is dynamically assigned to the first detection result. Finally, a server health score that integrates all monitoring indicators is calculated, which is a quantitative value reflecting the real-time health of the server. This approach overcomes the fragmentation defects of traditional single-indicator alarms and avoids the subjective bias of manual experience assessment, establishing a highly responsive and high-confidence decision-making benchmark for ensuring cluster stability.

[0066] For example, the preset threshold conditions and their corresponding scores for server-side monitoring items (i.e., monitoring indicator data) can be shown in Table 1.

[0067]

[0068] Table 1

[0069] In another implementation, the step of applying preset diagnostic rules to production log data to generate a production health score includes:

[0070] Based on the second diagnostic rule preset in the production end, the production risk feature data obtained by parsing the event type is detected, and a second detection result is generated for each production risk feature data. The second diagnostic rule includes the threshold condition for each production risk feature data and the preset score for each production risk feature data when the threshold condition is met.

[0071] Based on the second detection result and the preset score for each production risk characteristic data, the production health score is determined.

[0072] It should be noted that, based on the second diagnostic rule (the threshold conditions and preset scores for each production risk characteristic data), the production risk characteristic data is tested. The second test result can be 1 or 0, where 1 means passing the test and 0 means failing the test. Finally, each second test result is calculated with the preset score in the second diagnostic rule to output a quantitative production health score. This score can characterize the reliability, stability and compliance risks of the data writing process.

[0073] In this embodiment, a quantitative assessment of production-side data is achieved through detection using a preset second diagnostic rule. The parsed production risk characteristic data is compared with the threshold conditions in the second diagnostic rule to generate a second detection result. A score is dynamically assigned to the second detection result based on a preset score. Finally, a production-side health score that integrates all production risk characteristic data is calculated. This score reflects the real-time reliability of the production end, breaking through the limitations of traditional log keyword matching and avoiding the response delay of manual experience judgment. This establishes a highly sensitive and comprehensive proactive defense scheme for production-side data quality.

[0074] For example, the threshold conditions and their corresponding scores are preset for the production-side monitoring items (i.e., production risk characteristic data), as shown in Table 2.

[0075]

[0076] Table 2

[0077] In another implementation, the step of applying preset diagnostic rules to the consumption log data to generate a consumer health score includes:

[0078] Based on the pre-defined third diagnostic rules in the consumer end, the abnormal consumption data obtained from parsing consumption events is detected, and a third detection result is generated for each abnormal consumption data. The third detection result includes the threshold conditions for each abnormal consumption data and the preset score for each abnormal consumption data when the threshold conditions are met; the third detection result includes the detection result for each abnormal consumption data.

[0079] Based on the third detection result and preset score for each abnormal consumption data, a consumer health score is determined.

[0080] It should be noted that the abnormal consumption data is detected based on the third diagnostic rule (the threshold conditions and preset scores for each abnormal consumption data). The third detection result can be 1 or 0, where 1 means passing the detection and 0 means failing the detection. Finally, the third detection result is calculated with the preset score in the third diagnostic rule to output a quantitative consumer health score.

[0081] In this implementation, a pre-defined third diagnostic rule is used to detect abnormal consumption data, enabling a quantitative assessment of real-time consumer data. The parsed abnormal consumption data is compared with the threshold conditions in the third diagnostic rule to generate a third detection result. A score is dynamically assigned to the third detection result based on a pre-defined score. Finally, a consumer health score that integrates all abnormal consumption data is calculated. This score reflects the real-time robustness of the consumer end, overcoming the fragmentation of traditional single-point alarms and avoiding the lag of manual inspections. This establishes a highly sensitive and comprehensive proactive defense solution for the real-time data quality of the consumer end.

[0082] For example, the threshold conditions and their corresponding scores are preset for the consumer-side monitoring items (i.e., abnormal consumption data), as shown in Table 3.

[0083] Monitoring items Threshold condition Rating (out of 100) Consumption delay time Consumption delay time > 2 seconds 40 message expired Message expired and not consumed 50 Number of rebalancing Rebalance frequency > 20 times 10

[0084] Table 3

[0085] In addition, it should be noted that the threshold conditions and their scoring settings in the table above can be dynamically adjusted according to the actual business scenario, automatically triggering diagnostic rule optimization, so that the real-time data quality assessment system can continuously adapt to changes in business scenarios, break through the limitations of traditional static rules, and achieve iterative evolution of diagnostic rule optimization.

[0086] Step S30: By integrating the health scores of the server, production, and consumer ends, the data quality score of each end is obtained, and the health of the cluster is determined based on the data quality score.

[0087] It should be noted that in real-time data quality assessment, a weighted fusion mechanism is used to coordinate the three ends (server, consumer, and data processing) to jointly evaluate the health of the cluster. The server-side health score reflects the internal operating efficiency of the message cluster (such as resource load and request processing performance), the producer-side health score characterizes the stability of the data writing process (such as sending failure rate and traffic fluctuation), and the consumer-side health score reflects the reliability of data consumption (such as latency and rebalancing frequency).

[0088] Based on pre-defined weights for the business scenario, a weighted fusion calculation is performed on the three types of health scores to generate a data quality score for each endpoint. This score represents a quantitative value of the health status of the entire data transmission chain (production, service, and consumption). Finally, the overall health of the cluster is determined by comparing the data quality score with thresholds (e.g., >80 points indicates health). By acquiring data from the three endpoints and calculating their respective health scores, the higher the health score, the better the performance of each endpoint. The higher the final fusion score, the higher the data quality score of each endpoint. The health of the cluster is determined based on the data quality score; a higher data quality score indicates a better cluster health status. A better cluster health status ensures the continuous, stable, efficient, and reliable operation of the business.

[0089] In one feasible implementation, step S30 includes:

[0090] The server health score is multiplied by the server's first weight to obtain the server data quality score.

[0091] Multiply the production health score by the first weight of the production side, and then combine it with the server health score and the second weight of the server side to obtain the production data quality score.

[0092] The consumer health score is multiplied by the first weight of the consumer side, and then combined with the production health score and the second weight of the production side to obtain the consumer data quality score.

[0093] It should be noted that in the process of calculating the data quality score, the real-time data quality status of the entire cluster is quantified through a hierarchical weighted fusion mechanism: First, the server health score (reflecting the basic status of the cluster such as Broker resource load and replica synchronization latency) is multiplied by the first weight of the server (such as 1) to generate the server data quality score, which reflects the real-time data quality of the server.

[0094] For the production side, the production side health score (quantifying production stability such as normal data sending failures and data sending delays) is multiplied by the first weight of the production side (e.g., 0.75), and then the product of the server side health score and the second weight of the server side (characterizing the strength of the cluster's impact on production, for example, 0.25) is added. That is, production side weight × health score + server side weight × health score. The resulting production side data quality score indicates that production depends on its own stability and is also constrained by the basic state of the cluster (e.g., message writing depends on Broker replica synchronization).

[0095] For the consumer side, the consumer health score (which assesses consumption reliability such as message expiration and rebalancing frequency) is multiplied by the first weight of the consumer side (exemplarily, 0.8), and then combined with the product of the production side health score and the second weight of the production side (which reflects the impact of production quality on consumption, such as production delays inevitably leading to consumption backlog, exemplarily, 0.2). That is, consumer side weight × health score + production side weight × health score, to generate the consumer side data quality score.

[0096] The entire process enables the transmission of a complete real-time data stream across the Kafka cluster, from server to producer to consumer. This allows the data quality score to reflect both the independent health status of each endpoint and cross-end influencing factors, thus achieving a global assessment of the cluster. Furthermore, it should be noted that the calculated server-side, producer-side, and consumer-side data quality scores are collectively stored in a data table within a rolling time window.

[0097] In this implementation, the health scores of the server, producer, and consumer are weighted and fused: the server-side data quality score is directly generated by multiplying the server-side health score by the server-side first weight, reflecting the basic state of the cluster; the producer-side data quality score is fused with its own health score and the server-side health score, reflecting the dependence of production behavior on server-side resources; the consumer-side data quality score is further superimposed with the consumer-side's own health score and the producer-side health score, capturing the complete link state transmission of messages from production to consumption; breaking through the limitations of traditional single-layer weighted fusion, the dependence strength between each end is quantified through weight coefficients, so that the final generated data quality score accurately represents the health transmission effect of real-time data streams in the Kafka cluster, realizing the upgrade from local health status to full-link quality assessment, forming quantitative data, and ensuring the stability of real-time data services.

[0098] For example, see Figure 2 This document illustrates the real-time data quality assessment method of this application, including a schematic diagram of data quality score calculation. After calculating the server-side health score, producer-side health score, and consumer-side health score, the server-side data quality score is calculated based on the first weight of the server-side and its health score. Similarly, the producer-side data quality score is calculated based on the first weight of the producer-side, its health score, the second weight of the server-side, and its health score. Finally, the health of the cluster is quantitatively reflected based on the data quality scores of each end.

[0099] This embodiment provides a real-time data quality assessment method. It acquires multi-source monitoring data from three ends of the cluster (monitoring metrics data from the server, production log data from the production end, and consumption log data from the consumer end), and calculates server health scores, production health scores, and consumer health scores based on preset diagnostic rules for each end. Finally, it performs a weighted fusion calculation of the server health scores, production health scores, and consumer health scores based on preset weights to generate data quality scores for each end, and determines the cluster's health level based on these data quality scores. This entire process transforms discrete multi-source monitoring data into quantifiable health status values, overcoming the limitations of traditional single-end monitoring. Ultimately, it achieves accurate quantitative assessment of Kafka cluster health, proactively covering potential risks at both the production and consumer ends, and ensuring the continuous and stable operation of business driven by real-time data.

[0100] Based on the above embodiments of this application, in another embodiment of this application, the same or similar content as the above embodiments can be referred to the above description, and will not be repeated hereafter.

[0101] Collect historical multi-source monitoring data and historical health status of the cluster to build a training dataset. The historical multi-source monitoring data includes historical monitoring indicator data, historical production log data, and historical consumption log data.

[0102] Based on the training dataset, a prediction model is trained, where the input feature is historical multi-source monitoring data and the output feature is the health of the cluster.

[0103] The real-time multi-source monitoring data is input into the prediction model, and the predicted health of the cluster is output.

[0104] It should be noted that a training dataset is constructed based on the cluster's historical multi-source monitoring data and historical health status. The historical health status is obtained from the historical multi-source monitoring data using a real-time data quality assessment method. This training dataset is used to train a machine learning model, such as a long short-term memory network or a gradient boosting decision tree, for time-series prediction. This model learns the non-linear mapping relationship between the cluster's historical multi-source monitoring data and its historical health status. The input feature is the historical multi-source monitoring data, and the output feature is the cluster's historical health status. Training stops when the model's health status prediction error on the validation set consistently falls below a preset convergence threshold, resulting in a predictive model capable of predicting health status. In practical applications, real-time multi-source monitoring data can be input, and the model can output a predicted health status.

[0105] It should also be noted that the process of generating health scores for each endpoint by applying preset diagnostic rules to multi-source monitoring data, then merging the health scores of each endpoint to obtain the data quality score of each endpoint, and finally determining the health of the cluster based on the data quality score is complex. Therefore, this embodiment proposes a method using a predictive model to improve the speed of assessing the health of the entire cluster.

[0106] In this embodiment, a predictive model capable of predicting cluster health is trained by establishing a model of historical multi-source monitoring data and cluster health. When real-time multi-source monitoring data is input, the predictive model can overcome the limitations of weighted fusion and output a dynamic predicted value of cluster health based on complex nonlinear correlations learned from historical experience. This enables a shift from passive monitoring to proactive early warning, significantly improving the continuity and risk defense capabilities of real-time data services. It also ensures that the real-time data quality assessment system continuously adapts to changes in business scenarios and maintains a high degree of confidence in keeping the real-time data quality assessment in sync with the actual business health status.

[0107] For example, to help understand the implementation process of the real-time data quality assessment method obtained by combining this embodiment with the above embodiments, please refer to... Figure 3 , Figure 3 A simplified flowchart of a real-time data quality assessment method is provided, specifically:

[0108] Monitoring metrics and log data (including production logs from the producer and consumption logs from the consumer) are collected from the Kafka cluster and output to their respective topics; that is, monitoring metrics data is output to the metrics topic, and log data is output to the log topic. Then, based on the diagnostic rules for each end, a health score is calculated for each end. Finally, based on the health scores of the three ends, a data quality score is calculated to reflect the health of the cluster.

[0109] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the real-time data quality assessment method of this application. Any simple variations based on this technical concept, such as the interaction and combination of various embodiments, are all within the protection scope of this application.

[0110] This application also provides a real-time data quality assessment device; please refer to... Figure 4 The real-time data quality assessment device includes:

[0111] The acquisition module 10 is used to acquire multi-source monitoring data of the cluster, wherein the multi-source monitoring data includes monitoring indicator data of the server, production log data of the production end, and consumption log data of the consumer end.

[0112] The diagnostic module 20 is used to apply their respective preset diagnostic rules to the monitoring indicator data, production log data, and consumption log data to generate health scores for each terminal.

[0113] The fusion module 30 is used to obtain the data quality score of each end by merging the server-side health score, the production-side health score and the consumer-side health score, and to determine the health of the cluster based on the data quality score.

[0114] The real-time data quality assessment device includes a processing module, which is used to aggregate monitoring indicator data from the server, parse event types in production log data from the production side, and parse consumption events in consumption log data from the consumption side, based on a rolling time window.

[0115] The diagnostic module 20 is also used to detect the aggregated monitoring indicator data based on the first diagnostic rule preset in the server and generate a first detection result. The first diagnostic rule includes the threshold condition for each monitoring indicator data and the preset score for each monitoring indicator data. The first detection result includes the detection result for each monitoring indicator data.

[0116] Based on the first detection result and the preset score of each monitoring indicator data, the server health score is determined.

[0117] The diagnostic module 20 is also used to detect the production risk feature data obtained by parsing the event types based on the second diagnostic rules preset in the production end, and generate a second detection result. The second diagnostic rule includes the threshold conditions for each production risk feature data and the preset score for each production risk feature data. The second detection result includes the detection result for each production risk feature data.

[0118] Based on the second detection result and the preset score for each production risk characteristic data, the production health score is determined.

[0119] The diagnostic module 20 is also used to detect abnormal consumption data obtained by parsing consumption events based on the third diagnostic rules preset in the consumer terminal, and generate a third detection result, wherein the third detection result includes the threshold conditions for each abnormal consumption data and the preset score for each abnormal consumption data; the third detection result includes the detection result for each abnormal consumption data.

[0120] Based on the third detection result and the preset score for each abnormal consumption data, a consumer health score is determined.

[0121] The real-time data quality assessment device includes: an optimization module, used to collect server-side data quality scores, production-side data quality scores, consumer-side data quality scores, and cluster health scores over historical time periods, and to construct a training dataset;

[0122] Based on the training dataset, a prediction model is trained, wherein the input features are server-side data quality score, producer-side data quality score, and consumer-side data quality score, and the output feature is cluster health.

[0123] The real-time generated server-side data quality score, production-side data quality score, and consumer-side data quality score are input into the prediction model, and the predicted cluster health score is output.

[0124] When the predicted cluster health deviates from the actual health by more than a preset threshold, the preset diagnostic rules are optimized.

[0125] The fusion module 30 is also used to multiply the server health score by the first weight of the server to obtain the server data quality score;

[0126] The production-side health score is multiplied by the first weight of the production side, and then combined with the server-side health score and the second weight of the server side to obtain the production-side data quality score.

[0127] The consumer health score is multiplied by the first weight of the consumer side, and then the production health score is multiplied by the second weight of the production side to obtain the consumer data quality score.

[0128] The real-time data quality assessment device provided in this application, employing the real-time data quality assessment method described in the above embodiments, can solve the technical problem that existing open-source monitoring systems make it difficult for services relying on real-time data to operate stably. Compared with the prior art, the beneficial effects of the real-time data quality assessment device provided in this application are the same as those of the real-time data quality assessment method provided in the above embodiments, and other technical features in the real-time data quality assessment device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0129] This application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the real-time data quality assessment method in the first embodiment described above.

[0130] The following is for reference. Figure 5 The diagram illustrates a structural schematic of an electronic device suitable for implementing embodiments of this application. The electronic devices in these embodiments may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0131] like Figure 5As shown, the electronic device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the electronic device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. The communication device 1009 allows the electronic device to communicate wirelessly or wiredly with other devices to exchange data. Although the diagrams show electronic devices with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems may be implemented alternatively.

[0132] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0133] The electronic device provided in this application, employing the real-time data quality assessment method described in the above embodiments, can solve the technical problem that existing open-source monitoring systems make it difficult for services relying on real-time data to operate stably. Compared with the prior art, the beneficial effects of the electronic device provided in this application are the same as those of the real-time data quality assessment method provided in the above embodiments, and other technical features of this electronic device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0134] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0135] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0136] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the real-time data quality assessment method in the above embodiments.

[0137] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0138] The aforementioned computer-readable storage medium may be included in an electronic device or may exist independently without being assembled into an electronic device.

[0139] The aforementioned computer-readable storage medium carries one or more programs. When the aforementioned one or more programs are executed by an electronic device, the electronic device causes the electronic device to: acquire multi-source monitoring data of the cluster, wherein the multi-source monitoring data includes monitoring indicator data of the server, production log data of the production end, and consumption log data of the consumer end; apply their respective preset diagnostic rules to the monitoring indicator data, production log data, and consumption log data to generate health scores for each end; obtain data quality scores for each end by merging the server health scores, production health scores, and consumer health scores, and determine the health of the cluster based on the data quality scores.

[0140] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0141] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0142] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0143] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described real-time data quality assessment method. This solves the technical problem that existing open-source monitoring systems make it difficult for services relying on real-time data to operate stably. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the real-time data quality assessment method provided in the above embodiments, and will not be repeated here.

[0144] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the real-time data quality assessment method described above.

[0145] The computer program product provided in this application can solve the technical problem that existing open-source monitoring systems make it difficult for businesses relying on real-time data to operate stably. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the real-time data quality assessment method provided in the above embodiments, and will not be repeated here.

[0146] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A real-time data quality assessment method, characterized in that, The real-time data quality assessment method includes: Obtain multi-source monitoring data of the cluster, wherein the multi-source monitoring data includes monitoring indicator data of the server, production log data of the production end, and consumption log data of the consumer end; Each of the monitoring indicator data, production log data, and consumption log data is applied with its own preset diagnostic rules to generate a health score for each end. The health score for each end includes a server-side health score, a production-side health score, and a consumption-side health score. The data quality score of each end is obtained by integrating the server-side health score, the production-side health score, and the consumer-side health score, and the health of the cluster is determined based on the data quality score.

2. The method as described in claim 1, characterized in that, Before the step of applying their respective preset diagnostic rules to the monitoring indicator data, production log data, and consumption log data to generate health scores for each terminal, the following steps are included: Based on a rolling time window, the system aggregates monitoring metrics data from the server, parses event types from production logs from the production side, and parses consumption events from consumption logs from the consumption side.

3. The method as described in claim 2, characterized in that, The steps of applying preset diagnostic rules to the monitoring indicator data to generate a server-side health score include: Based on the first diagnostic rule preset in the server, the aggregated monitoring indicator data is detected and a first detection result for each monitoring indicator data is generated. The first diagnostic rule includes a threshold condition for each monitoring indicator data and a preset score for each monitoring indicator data when the threshold condition is met. Based on the first detection result and the preset score for each monitoring indicator, the server health score is determined.

4. The method as described in claim 2, characterized in that, The steps of applying preset diagnostic rules to the production log data to generate a production health score include: Based on the second diagnostic rule preset in the production end, the production risk feature data obtained by parsing the event type is detected, and a second detection result is generated for each production risk feature data. The second diagnostic rule includes the threshold condition for each production risk feature data and the preset score for each production risk feature data when the threshold condition is met. Based on the second detection result and the preset score for each production risk characteristic data, the production health score is determined.

5. The method as described in claim 2, characterized in that, The step of applying preset diagnostic rules to the consumption log data to generate a consumer health score includes: Based on the third diagnostic rule preset in the consumer end, the abnormal consumption data obtained by parsing consumption events is detected, and a third detection result is generated for each abnormal consumption data. The third detection result includes the threshold condition for each abnormal consumption data and the preset score for each abnormal consumption data when the threshold condition is met. Based on the third detection result and preset score for each abnormal consumption data, a consumer health score is determined.

6. The method as described in claim 1, characterized in that, The step of obtaining the data quality score for each end by fusing the server-side health score, the production-side health score, and the consumer-side health score includes: The server health score is multiplied by the server's first weight to obtain the server data quality score; The production-side health score is multiplied by the first weight of the production side, and then combined with the server-side health score and the second weight of the server side to obtain the production-side data quality score. The consumer health score is multiplied by the first weight of the consumer side, and then the production health score is multiplied by the second weight of the production side to obtain the consumer data quality score.

7. The method as described in claim 1, characterized in that, The method includes: Collect historical multi-source monitoring data and historical health status of the cluster to construct a training dataset. The historical multi-source monitoring data includes historical monitoring indicator data, historical production log data, and historical consumption log data. Based on the training dataset, a prediction model is trained, wherein the input feature is historical multi-source monitoring data and the output feature is the health of the cluster. The predicted health of the cluster is output by inputting real-time multi-source monitoring data into the prediction model.

8. An electronic device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the real-time data quality assessment method as described in any one of claims 1 to 7.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the real-time data quality assessment method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the real-time data quality assessment method as described in any one of claims 1 to 7.

Images