Real-time monitoring method, system and equipment for sample data and medium

Abstract

The invention discloses a real-time monitoring method, system and equipment for sample data and a medium, and relates to the field of data management.The method comprises the steps that multi-source sample data are input into Kafka, and real-time verification is conducted through Flink; data information and processing behaviors in the verification process are used as nodes and relation chains to construct a blood relationship map; when the data does not pass the verification, interrupting the training process of the current data in the target model, and marking the data which does not pass the verification as dirty data for isolation; obtaining a proportion value of the number of data which does not pass verification under each data verification rule in the isolation result in the total data number; according to an isolation result, querying whether a key node forbidding an error exists in an influence range in the blood relationship map; and stopping the training process of all data in the target model when the proportion value exceeds a preset data quality alarm threshold value or a key node forbidding errors exists. The method can automatically prevent dirty data from polluting the downstream according to different data quality problems.

Description

Technical Field

[0001] This invention relates to the field of data management, and in particular to a method, system, device and medium for real-time monitoring of sample data. Background Technology

[0002] With the rapid development of big data and artificial intelligence technologies, enterprises face the challenge of managing and processing multi-source data. In this process, data quality directly affects the operational efficiency and decision-making accuracy of enterprises.

[0003] Existing data governance methods include rule-based data quality inspection, statistical analysis, machine learning, and metadata management. However, when data problems are discovered, it's difficult to respond proactively based on the severity of the issue to quickly prevent downstream data from becoming contaminated. Therefore, how to rapidly prevent dirty data from polluting downstream data based on different data quality problems is a crucial issue that urgently needs to be addressed. Summary of the Invention

[0004] This invention provides a real-time monitoring method for sample data, which can solve the problem in the prior art that it is impossible to quickly prevent dirty data from contaminating downstream processes based on different data quality issues.

[0005] This invention provides a method for real-time monitoring of sample data, comprising the following steps: Obtain multi-source sample data for the target model; Multi-source sample data is input into the message queue Kafka to obtain Kafka data; the data processing engine Flink is used to perform real-time verification of the data type of Kafka data according to multiple preset data verification rules; the data information and processing behavior during the verification process are respectively used as nodes and relationship chains of the lineage graph to construct a lineage graph containing data lineage relationships. When Kafka data fails validation, the training process of the current data in the target model is interrupted; the Kafka data that fails validation is marked as dirty data and isolated using Kafka; the proportion of Kafka data that fails validation under each data validation rule in the isolation results is obtained through the monitoring metric tool Flink Metrics; based on the isolation results, the scope of influence is queried according to the data lineage relationship in the lineage graph, and the presence of critical nodes that prevent errors is checked within all nodes included in the scope of influence. When the ratio exceeds the preset data quality alarm threshold, or when there is a critical node that prevents errors, the training process of all data in the target model is stopped.

[0006] Furthermore, the method of using the Flink data processing engine to perform real-time validation of the data type of Kafka data according to multiple preset data validation rules includes: Initialize Flink and create the stream processing environment; Configure Flink's data stream source to be Kafka topic data; Use a JSON deserializer to convert Kafka messages into Java POJO objects; Define a custom processing function named DataValidationProcessFunction; this custom processing function inherits from the KeyedProcessFunction function in Flink.<String,DataRecord,ValidationResult> This allows custom processing functions to be applied to Flink's data stream. The data stream is partitioned according to String type keys. Each partition receives DataRecord type data as input, and after being processed by multiple preset data validation rules, it sends ValidationResult type data downstream as output. The preset multiple data validation rules include null values, numerical range, duplicate values, format errors, and data delay.

[0007] Furthermore, the step of using the data information and processing behavior during the verification process as nodes and relationship chains in the lineage graph, respectively, to construct a lineage graph containing data lineage relationships, includes: Design a lineage model, using data records and data processing steps in Flink as nodes in the lineage graph, and data flow and processing actions as the relationship chains between nodes in the lineage graph, and generate the lineage graph in the graph database Neo4j; After the Flink verification is completed, the information is written to the node and the relationship chain.

[0008] Furthermore, the step of marking the Kafka data that fails the verification as dirty data and isolating it using Kafka includes: Create nodes for dirty data that fail validation and nodes for clean data that pass validation in the lineage graph, and create a relationship chain between dirty data and clean data; Write the reason for the failure to pass the validation into the relationship chain.

[0009] Furthermore, the step of obtaining the proportion of Kafka data that failed validation under each data validation rule in the isolation results through the monitoring metric tool Flink Metrics, including: In the custom processing function DataValidationProcessFunction, a counter is defined for each data validation rule and the total data. Call the open method in the custom processing function and initialize the monitoring metric tool Flink Metrics; When a data stream passes through the custom processing function, the counter is updated according to the corresponding verification result; Configure the Flink Metrics Reporter to expose all counting results in Prometheus format; Configure the monitoring system Prometheus to capture target data in Prometheus format, and use the query language PromQL to set a proportion calculation function to obtain the proportion of Kafka data that failed the validation under each data validation rule to the total number of data based on the counting results.

[0010] Furthermore, the step of querying the scope of influence based on the bloodline relationship in the bloodline diagram, and checking whether there are any critical nodes that prevent errors within all nodes included in the scope of influence, includes: Manually mark critical nodes in the lineage diagram that are designed to prevent errors; Connect to the Neo4j graph database, and based on the data that failed the validation in the isolation results, use Cypher query commands to trace upstream of the lineage graph back to the data source, and trace downstream of the lineage graph all processing steps and products that consume the data; Filter out any marked critical nodes from the traced results to prevent errors.

[0011] Furthermore, the step of interrupting the training process of the current data in the target model when Kafka data fails validation includes: Set the circuit breaker parameters in the custom processing function DataValidationProcessFunction; When Kafka data fails validation, update the circuit breaker parameters to the enabled state. Set a function to trigger the circuit breaker. If the function checks that the circuit breaker parameter is in the "on" state, it will interrupt the training process of the current data in the target model and update the circuit breaker parameter to the "off" state.

[0012] This invention provides a real-time monitoring system for sample data, comprising: The data acquisition module is used to acquire multi-source sample data of the target model; The data verification module is used to input multi-source sample data into the message queue Kafka to obtain Kafka data; it uses the data processing engine Flink to perform real-time verification of the data type of Kafka data according to multiple preset data verification rules; and it uses the data information and processing behavior in the verification process as nodes and relationship chains in the lineage graph to construct a lineage graph containing data lineage relationships. The dirty data monitoring module is used to interrupt the training process of the current data in the target model when Kafka data fails validation; it marks the Kafka data that fails validation as dirty data and isolates it using Kafka; it obtains the proportion of Kafka data that fails validation under each data validation rule to the total number of data in the isolation results using the monitoring metric tool Flink Metrics; based on the isolation results, it queries the scope of influence according to the data lineage relationship in the lineage graph, and checks whether there are any critical nodes that prevent errors within all nodes included in the scope of influence. The data processing termination module is used to stop the training process of all data in the target model when the ratio exceeds the preset data quality alarm threshold or when there is a critical node that prevents errors.

[0013] This invention provides a computer device, including a memory and a processor; the memory stores a computer program, and the processor executes the computer program to implement the above-described method for real-time monitoring of sample data.

[0014] This invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the real-time monitoring method for sample data described above.

[0015] This invention provides a method, system, device, and medium for real-time monitoring of sample data, which has the following advantages compared with the prior art: Multi-source sample data is input into the Kafka message queue to obtain Kafka data. The data processing engine Flink is used to perform real-time validation of the Kafka data data according to multiple preset data validation rules. The data information and processing behavior during the validation process are respectively used as nodes and relationship chains in a lineage graph to construct a lineage graph containing data lineage relationships. When Kafka data fails validation, the training process of the current data in the target model is interrupted. The Kafka data that fails validation is marked as dirty data and isolated using Kafka. The proportion of Kafka data that fails validation under each data validation rule in the isolation results is obtained using the monitoring metric tool Flink Metrics. Based on the isolation results, the influence range is queried according to the data lineage relationships in the lineage graph, and the presence of critical nodes that prevent errors is checked within all nodes included in the influence range. When the proportion exceeds the preset data quality alarm threshold, or when a critical node that prevents errors exists, the training process of all data in the target model is stopped. In this process, firstly, the constructed lineage graph records all information about the data from upstream to downstream in the target model training process. Then, multiple preset data validation rules can detect different data quality issues. When data fails validation, the target model training process for the current data is interrupted, quickly preventing the dirty data from contaminating the downstream of the target model training process. After marking the failed validation results as dirty data, they are isolated, and the isolation results are used to query the lineage graph to see if there are any critical nodes within the affected area that prevent errors. Once a critical node preventing errors exists, or if the proportion of Kafka data that failed validation under each data validation rule in the isolation results exceeds the preset data quality alarm threshold, it indicates that an error has occurred in a critical node preventing errors in the target model training process, or that a large amount of dirty data that does not meet the target model training requirements has appeared. At this point, the training process of all data in the target model is immediately stopped, quickly preventing problems from occurring downstream in the target model training process. Attached Figure Description

[0016] Figure 1 A diagram illustrating the implementation steps of a real-time monitoring method for sample data provided in an embodiment of the present invention; Figure 2 A diagram illustrating the real-time data type verification steps of a real-time monitoring method for sample data provided in this embodiment of the invention; Figure 3 A diagram illustrating the steps for obtaining the proportion value in a real-time monitoring method for sample data provided in an embodiment of the present invention. Detailed Implementation

[0017] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0018] See Figure 1 This invention provides a method for real-time monitoring of sample data, comprising the following steps: Step 1: Obtain multi-source sample data for the target model.

[0019] Step 2: Input multi-source sample data into the message queue Kafka to obtain Kafka data; use the data processing engine Flink to perform real-time verification of the data type of Kafka data according to multiple preset data verification rules; use the data information and processing behavior during the verification process as nodes and relationship chains of the lineage graph to construct a lineage graph containing data lineage relationships.

[0020] Step 3: When Kafka data fails validation, interrupt the training process of the current data in the target model; mark the Kafka data that fails validation as dirty data and isolate it using Kafka; use the monitoring metric tool FlinkMetrics to obtain the proportion of Kafka data that fails validation under each data validation rule in the isolation results to the total number of data; based on the isolation results, query the scope of influence according to the data lineage relationship in the lineage graph, and check whether there are any critical nodes that prevent errors within all nodes included in the scope of influence.

[0021] Step 4: When the ratio exceeds the preset data quality alarm threshold, or when there is a critical node that prevents errors, stop the training process of all data in the target model.

[0022] The details of this invention are as follows: I. Multi-source sample data.

[0023] Enterprises inevitably need to use large amounts of data in their development of artificial intelligence technology. This data comes from diverse and complex sources, including multiple business databases, application log files, and third-party enterprise data. When training a target AI model, enterprises need to identify effective data from these multiple sources. This presents the challenge of validating and processing this multi-source data.

[0024] II. Processing of multi-source sample data and construction of lineage maps.

[0025] 1. Data formatting.

[0026] Transform multi-source sample data into a data stream in a unified format, including but not limited to JSON or Avro, and each data entry contains a globally unique ID (data_id) and a timestamp (event_time).

[0027] 2. The message queue Kafka receives data.

[0028] The transformed data stream is input into the data-input-topic of the Kafka message queue to obtain Kafka data.

[0029] 3. The data processing engine Flink consumes Kafka data and performs real-time verification.

[0030] Initialize Flink and create a stream processing environment (StreamExecutionEnvironment); configure the data stream source (Kafka Source) and subscribe to the data topic (data-input-topic); use a JSON deserializer to convert Kafka messages into Java POJO objects.

[0031] 4. Data processing.

[0032] Define a custom processing function named DataValidationProcessFunction; this custom processing function inherits from the KeyedProcessFunction function in Flink.<String,DataRecord,ValidationResult> This allows custom processing functions to be applied to Flink's data stream. The data stream is partitioned according to String keys. Each partition receives DataRecord type data as input, performs validation processing according to multiple preset data validation rules, and then sends ValidationResult type data downstream as output. The preset data validation rules include null values, numerical range, duplicate values, format errors, and data delay. Figure 2 As shown.

[0033] The code example is as follows (data validation is not limited to the validation rules listed in the code): Public class DataValidationProcessFunction extendskeyedProcessFunction<string,DataRecord,ValidationResult> { / / Validate for empty values.

[0034] private boolean checkForNulls(DataRecord record){ return record.getUserId()==null||record.getOrderAmount()==null;} / / Validating format errors.

[0035] private boolean checkFormat(DataRecord record){ Try{return record.getOrderAmount()>=0;} catch(Exception e){return true;} } / / Check for duplicate values.

[0036] Private ValueState <boolean>isDuplicateState; @override Public void open(configuration parameters){ valueStateDescriptor <boolean>descriptor=new ValueStateDescriptor<>("is-duplicate”,Types.BOOLEAN); StateTtlConfig ttlConfig=stateTtlConfig .newBuilder(Time.hours(24)) .setUpdateType(stateTtlConfig.UpdateType.OnCreateAndWrite) .setStateVisibility(StateTtlConfig.StateVisiblity.NeverReturnExpired) .build(); Descriptor.enableTimeToLive(ttlConfig); isDuplicateState=getRuntimeContext().getState(descriptor); } @Override Public void processElement(DataRecord record,Context ctx,Collector <validationresult>out)throws Exception{ ValidationResult result=new ValdationResult(); result.setOriginalRecord(record); result.setValidationPassed(true); List <string>failureReasons=new ArrayList<>(); / / Check for duplicate values.

[0037] If(isDuplicateState.value() !=null){ failureReasons.add("DUPLICATE_VALUE"); }else{isDuplicateState.update(true);} } / / Other validations.

[0038] if(checkForNulls(record))failureReasons.add("NULL_VALUE"); if(checkFormat(record))failureReasons.add("FORMAT_ERROR"); if(checkForDelay(record))failureReasons.add("DATA_DELAY"); If(!failureReasons.isEmpty()){ result.setValidationPassed(false); result.setFailureReasons(failureReasons);} out.collect(result);} } 5. Construction of the lineage chart.

[0039] The lineage model is designed, using data records (data_id) and data processing steps (such as FlinkValidation) in Flink as nodes in the lineage graph, and data flow and processing actions as the relationship chains between nodes in the lineage graph. The lineage graph is then generated in the Neo4j graph database. After Flink validation is completed (using the processElement method of DataValidationProcessFunction), the lineage information is written to the nodes and relationship chains.

[0040] A lineage diagram can also clearly show the destination of the original data, the source of a certain node, and the time, reason, and processing of the data, thereby accurately assessing the scope of impact and guiding subsequent work.

[0041] The lineage uses asynchronous write operations, marking data as dirty or clean during writing.

[0042] 6. Handling of data that fails validation.

[0043] Set the circuit breaker parameters in the custom processing function DataValidationProcessFunction; when Kafka data fails validation, update the circuit breaker parameters to the open state; set the function to trigger the circuit breaker. If the function checks that the circuit breaker parameters are in the open state, then the training process of the current data in the target model is interrupted, and the circuit breaker parameters are updated to the closed state.

[0044] (1) Use ValueState in DataValidationProcessFunction to store the circuit breaker state and related information.

[0045] / / Define an enumeration of fuse states.

[0046] public enum CircuitBreakerState{ CLOSED, OPEN, HALF_OPEN } public class DataValidationProcessFunction extendsKeyedProcessFunction<String,DataRecord,ValidationResult> { / / Fuse status.

[0047] private ValueState <circuitbreakerstate>CircuitBreakerState; / / Failure counter.

[0048] private ValueState <long>failureCount; / / Timestamp of the last failure.

[0049] private ValueState <long>lastFailureTimestamp; / / Configure circuit breaker related settings.

[0050] private static final long FAILURE_THRESHOLD=50; / / Failure threshold.

[0051] private static final long TIMEOUT_MS=120000; / / The duration the circuit breaker will last.

[0052] private static final long SUCCESS_THRESHOLD=2; / / In the half-open state, it needs to be restored after two consecutive times.

[0053] @Override public void open(Configuration parameters)throws Exception{ super.open(parameters); / / Fuse state initialization.

[0054] ValueStateDescriptor <circuitbreakerstate>stateDescriptor=newValueStateDescription<>("circuitBreakerState”,CircuitBreakerState.class); circuitBreakerState=getRuntimeContext().getState(stateDescriptor); ValueStateDescriptor <long>failureCountDescriptor=newValueStateDescriptor<>("failureCount”,long.class); failureCount=getRuntimeContext().getState(failureCountDescriptor); ValueStateDescriptor <long>lastFailureTimestampDescriptor=newValueStateDescriptor<>(lastFailureTimestamp=getRuntimeContext().getState(lastFailureTimestampDescriptor);) }} (2) Implement the circuit breaker logic in processElement and set the function that triggers the circuit breaker.

[0055] First, the current fuse status needs to be determined. If the initial state is open, the fuse is tripped and the circuit is directly isolated without further processing. If the initial state is closed, it is determined to be in a non-fuse-triggered state, and the subsequent verification steps are executed normally, with the result output.

[0056] If the circuit breaker is triggered and the system is isolated, the failure count is incremented, and the isolated data is processed.

[0057] 3. Isolate dirty data.

[0058] Create nodes for dirty data that failed validation and nodes for clean data that passed validation in the lineage graph, and create a relationship chain between dirty data and clean data; write the reason for failing validation into the relationship chain.

[0059] Specifically, within Flink's main workflow, the output `validationResult` of `DataValidationProcessFunction` can be split into clean and dirty data streams by `splite` or `filter` operations. The clean data stream, containing data where `result.isValidationPassed() == true`, is sent downstream (e.g., to a data warehouse). The dirty data stream, containing data where `result.isValidationPassed() == false`, is sent to the `dirty-data-topic` using `FlinkKafkaProducer`. During sending, the `validationResult` object is serialized into JSON, containing the original data and the failure reason. The failure reason is written into the relationship chain between the created dirty and clean data.

[0060] IV. Flink Metrics Monitoring.

[0061] In the custom processing function `DataValidationProcessFunction`, a counter is defined for each data validation rule and the total data. The `open` method is called within the custom processing function to initialize the monitoring metric tool `Flink Metrics`. When a data stream passes through the custom processing function, the counters are updated based on the corresponding validation results. The `Flink Metrics Reporter` is configured to expose all counting results in Prometheus format. The monitoring system `Prometheus` is configured to capture target data in Prometheus format, and a proportion calculation function is set using the query language `PromQL` to obtain the proportion of Kafka data that failed validation under each data validation rule to the total data volume based on the counting results. Figure 3 As shown.

[0062] (1) Define Metrics.

[0063] First, define the counter and initialize its state in the `open` method of `DataValidationProcessFunction`. Second, count the data that failed validation under each rule.

[0064] (2) In the processElement method, update the counter based on the verification result.

[0065] (3) Configure the Flink Metrics Reporter.

[0066] Configure the Prometheus reporter in flink-conf.yaml so that Prometheus can periodically obtain metrics.

[0067] Message queues in Flink only expose the counters; the rate needs to be calculated in detail in Prometheus. In this case, the `rate()` function can smoothly handle the counters and calculate the growth rate per second, making it suitable for monitoring flowing data types.

[0068] In summary, the above steps work closely together, integrating the message queue Kafka, the data processing engine Flink, the monitoring system Prometheus, and the graph database of lineage graphs to build an end-to-end and automated real-time data quality assurance system. It can detect and isolate problematic data in real time and utilize data lineage to achieve rapid and accurate impact scope analysis, thereby greatly improving the reliability of data processing and operational efficiency. The steps are closely linked and interact with each other, exhibiting high systemicity and synergy, as well as high scalability and practicality.

[0069] 5. Check the scope of impact and inspect the nodes.

[0070] In the lineage graph, critical nodes that are prohibited from making errors are manually marked; the graph database Neo4j is connected, and based on the data that fails the validation in the isolation results, the Cypher query command is used to trace upstream of the lineage graph back to the data source, and downstream of the lineage graph to trace all processing steps and products that consume the data; from the traced results, it is filtered to see if there are any marked critical nodes that are prohibited from making errors.

[0071] 1. Connect to the Neo4j graph database and use the Cypher graph query language to query the scope of impact, starting from the dirty data event, tracing upstream to the data source, and downstream to all processing steps and data products that consume this data.

[0072] The code first retrieves the parameter `$data_id`. Next, it searches for the data source associated with the dirty data event and traverses it using a variable-length path to find upstream and downstream nodes. It traces upstream and downstream to obtain all relevant information. Then, it extracts all unique nodes from the path. After extraction, it returns detailed information about the nodes, including whether they are critical nodes.

[0073] 2. Select key nodes within the scope of influence.

[0074] First, filter nodes where is_critical=true. Then, return all affected critical nodes. After returning, check if the affected critical nodes are critical nodes that cannot afford to fail.

[0075] This invention provides a real-time monitoring system for sample data, comprising: The data acquisition module is used to acquire multi-source sample data for the target model.

[0076] The data validation module is used to input multi-source sample data into the message queue Kafka to obtain Kafka data; it uses the data processing engine Flink to perform real-time validation of the data type of Kafka data according to multiple preset data validation rules; and it uses the data information and processing behavior in the validation process as nodes and relationship chains in the lineage graph to construct a lineage graph containing data lineage relationships.

[0077] The dirty data monitoring module is used to interrupt the training process of the current data in the target model when Kafka data fails validation; mark the Kafka data that fails validation as dirty data and isolate it using Kafka; obtain the proportion of Kafka data that fails validation under each data validation rule to the total number of data in the isolation results using the monitoring metric tool Flink Metrics; based on the isolation results, query the scope of influence according to the data lineage relationship in the lineage graph, and check whether there are any critical nodes that prevent errors within all nodes included in the scope of influence.

[0078] The data processing termination module is used to stop the training process of all data in the target model when the ratio exceeds the preset data quality alarm threshold or when there is a critical node that prevents errors.

[0079] This invention provides a computer device, including: a memory and a processor; the memory stores a computer program, and the processor executes the computer program to implement the steps of a method for real-time monitoring of sample data.

[0080] This invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of a method for real-time monitoring of sample data.

[0081] A specific example is as follows: This embodiment discloses a method for real-time monitoring of sample data, the specific steps of which are as follows: S1. Obtain multi-source sample data for the target model, including multiple business databases, application log files, and third-party enterprise data.

[0082] S2. Input multi-source sample data into the Kafka message queue to obtain Kafka data; use the Flink data processing engine to perform real-time validation of the Kafka data data according to multiple preset data validation rules. The validation process includes: initializing Flink and creating a stream processing environment; configuring Flink's data stream source to Kafka topic data; using a JSON deserializer to convert Kafka messages into Java POJO objects; defining a custom processing function named DataValidationProcessFunction; the custom processing function inherits the KeyedProcessFunction function in Flink.<String,DataRecord,ValidationResult> This allows custom processing functions to be applied to Flink's data stream. The data stream is partitioned according to String type keys. Each partition receives DataRecord type data as input, and after being processed by multiple preset data validation rules, it sends ValidationResult type data downstream as output. The preset data validation rules include null values, numerical range, duplicate values, format errors, and data delay.

[0083] S3. The data information and processing actions during the verification process are respectively used as nodes and relational chains in the lineage graph to construct a lineage graph containing data lineage relationships. The design steps include: using data records and data processing steps in Flink as nodes in the lineage graph, and using data flow and processing actions as relational chains between nodes in the lineage graph; generating the lineage graph in the graph database Neo4j; and writing the information into the nodes and relational chains after verification using Flink.

[0084] S4. When Kafka data fails validation, interrupt the training process of the current data in the target model; mark the Kafka data that fails validation as dirty data and isolate it using Kafka; obtain the proportion of Kafka data that fails validation under each data validation rule to the total number of data in the isolation results using the monitoring metric tool FlinkMetrics. The steps for obtaining the proportion include: defining a counter for each data validation rule and the total number of data in the custom processing function DataValidationProcessFunction; calling the open method in the custom processing function and initializing the monitoring metric tool Flink Metrics; updating the counters according to the corresponding validation results when a data stream passes through the custom processing function; configuring the reporter Flink MetricsReporter to expose all counting results in Prometheus format; configuring the monitoring system Prometheus to capture the target data in Prometheus format, and using the query language PromQL to set the proportion calculation function to obtain the proportion of Kafka data that fails validation under each data validation rule to the total number of data based on the counting results.

[0085] S5. Based on the isolation results, query the scope of influence according to the data lineage relationship in the lineage diagram, and check whether there are any critical nodes that prevent errors within all nodes included in the scope of influence.

[0086] S6. When the ratio exceeds the preset data quality alarm threshold, or when there is a critical node that prevents errors, stop the training process of all data in the target model.

[0087] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.< / long> < / long> < / circuitbreakerstate> < / long> < / long> < / circuitbreakerstate> < / string> < / validationresult> < / boolean> < / boolean>

Claims

1. A method for real-time monitoring of sample data, characterized in that, Includes the following steps: Obtain multi-source sample data for the target model; Multi-source sample data is input into the message queue Kafka to obtain Kafka data; the data processing engine Flink is used to perform real-time validation of the data type of Kafka data according to multiple preset data validation rules. The data information and processing behavior during the verification process are respectively used as nodes and relationship chains in the lineage graph to construct a lineage graph containing data lineage relationships; When Kafka data fails validation, the training process of the current data in the target model is interrupted; the Kafka data that fails validation is marked as dirty data and isolated using Kafka; the proportion of Kafka data that fails validation under each data validation rule in the isolation results is obtained by the monitoring metric tool Flink Metrics. Based on the isolation results, query the scope of influence according to the bloodline relationship in the bloodline diagram, and check whether there are any critical nodes that prevent errors within all nodes included in the scope of influence; When the ratio exceeds the preset data quality alarm threshold, or when there is a critical node that prevents errors, the training process of all data in the target model is stopped.

2. The real-time monitoring method for sample data as described in claim 1, characterized in that, The process utilizes the Flink data processing engine to perform real-time validation of Kafka data data according to multiple preset data validation rules, including: Initialize Flink and create the stream processing environment; Configure Flink's data stream source to be Kafka topic data; Use a JSON deserializer to convert Kafka messages into Java POJO objects; Define a custom processing function named DataValidationProcessFunction; this custom processing function inherits from the KeyedProcessFunction function in Flink.<String,DataRecord,ValidationResult> This allows custom processing functions to be applied to Flink's data stream. The data stream is partitioned according to String type keys. Each partition receives DataRecord type data as input, and after being processed by multiple preset data validation rules, it sends ValidationResult type data downstream as output. The preset multiple data validation rules include null values, numerical range, duplicate values, format errors, and data delay.

3. The real-time monitoring method for sample data as described in claim 1, characterized in that, The step of using data information and processing behavior during the verification process as nodes and relationship chains in the lineage graph to construct a lineage graph containing data lineage relationships includes: Design a lineage model, using data records and data processing steps in Flink as nodes in the lineage graph, and data flow and processing actions as the relationship chains between nodes in the lineage graph, and generate the lineage graph in the graph database Neo4j; After the Flink verification is completed, the information is written to the node and the relationship chain.

4. The real-time monitoring method for sample data as described in claim 1, characterized in that, The step of marking Kafka data that fails validation as dirty data and isolating it using Kafka includes: Create nodes for dirty data that fail validation and nodes for clean data that pass validation in the lineage graph, and create a relationship chain between dirty data and clean data; Write the reason for the failure to pass the validation into the relationship chain.

5. The real-time monitoring method for sample data as described in claim 2, characterized in that, The percentage of Kafka data that failed validation under each data validation rule in the isolation results, obtained through the monitoring metric tool Flink Metrics, is included in the following: In the custom processing function DataValidationProcessFunction, a counter is defined for each data validation rule and the total data. Call the open method in the custom processing function and initialize the monitoring metric tool Flink Metrics; When a data stream passes through the custom processing function, the counter is updated according to the corresponding verification result; Configure the Flink Metrics Reporter to expose all counting results in Prometheus format; Configure the monitoring system Prometheus to capture target data in Prometheus format, and use the query language PromQL to set a proportion calculation function to obtain the proportion of Kafka data that failed the validation under each data validation rule to the total number of data based on the counting results.

6. The real-time monitoring method for sample data as described in claim 1, characterized in that, The process involves querying the scope of influence based on the bloodline relationships in the bloodline diagram, and checking for critical nodes that prevent errors within all nodes included in the scope of influence, including: Manually mark critical nodes in the lineage diagram that are designed to prevent errors; Connect to the Neo4j graph database, and based on the data that failed the validation in the isolation results, use Cypher query commands to trace upstream of the lineage graph back to the data source, and trace downstream of the lineage graph all processing steps and products that consume the data; Filter out any marked critical nodes from the traced results to prevent errors.

7. The real-time monitoring method for sample data as described in claim 2, characterized in that, The step of interrupting the training process of the current data in the target model when Kafka data fails validation includes: Set the circuit breaker parameters in the custom processing function DataValidationProcessFunction; When Kafka data fails validation, update the circuit breaker parameters to the enabled state. Set a function to trigger the circuit breaker. If the function checks that the circuit breaker parameter is in the "on" state, it will interrupt the training process of the current data in the target model and update the circuit breaker parameter to the "off" state.

8. A real-time monitoring system for sample data, characterized in that, include: The data acquisition module is used to acquire multi-source sample data of the target model; The data validation module is used to input multi-source sample data into the message queue Kafka to obtain Kafka data; and uses the data processing engine Flink to perform real-time validation of the data type of Kafka data according to multiple preset data validation rules. The data information and processing behavior during the verification process are respectively used as nodes and relationship chains in the lineage graph to construct a lineage graph containing data lineage relationships; The dirty data monitoring module is used to interrupt the training process of the current data in the target model when Kafka data fails the validation; it marks the Kafka data that fails the validation as dirty data and isolates it using Kafka; and it obtains the proportion of the number of Kafka data that fails the validation under each data validation rule to the total number of data in the isolation results through the monitoring metric tool Flink Metrics. Based on the isolation results, query the scope of influence according to the bloodline relationship in the bloodline diagram, and check whether there are any critical nodes that prevent errors within all nodes included in the scope of influence; The data processing termination module is used to stop the training process of all data in the target model when the ratio exceeds the preset data quality alarm threshold or when there is a critical node that prevents errors.

9. A computer device, comprising: Memory and processor; The memory stores a computer program, characterized in that when the processor executes the computer program, it implements a real-time monitoring method for sample data according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements a real-time monitoring method for sample data as described in any one of claims 1 to 7.

Images