Intelligent data precision quality management system, method and device

Abstract

This invention discloses an intelligent data precision quality management system, method, and apparatus, relating to the field of data quality management technology. When rules change, a rule update event is generated and published via Kafka. During rule updates, differences between old and new rules are detected, and a rollback mechanism is automatically triggered when the difference exceeds expectations. Preprocessed data is validated using preset rules, and several abnormal data classes are obtained after classifying abnormal data. Classified abnormal data is automatically corrected. Abnormal data that still cannot be corrected is identified and classified, and the identified isolated data classes are migrated from the main data stream to an isolation queue. Abnormal data that cannot be processed immediately is retained in the isolation queue, and the data receiver is periodically reminded to process it. Abnormalities that cannot be processed immediately remain in the isolation queue, and the data receiver is notified at a constrained reminder frequency to ensure that potentially high-risk abnormalities do not accumulate indefinitely.

Description

Technical Field

[0001] This invention relates to the field of data quality management technology, specifically to an intelligent data precision quality management system, method, and apparatus. Background Technology

[0002] In the e-commerce industry, businesses need to process multi-source data in real time, including order transactions, product inventory, and user behavior. This data is not only massive in volume and diverse in type, but also experiences a significant increase in data traffic during peak promotional periods. The quality of this data directly impacts system stability, the accuracy of business decisions, and customer experience. For example, if data anomalies such as negative order amounts, incorrect inventory quantities, or abnormal timestamps are not detected and addressed promptly, they can lead to business process disruptions, inventory management errors, and increased customer complaints. To address this challenge, businesses need to build an intelligent, precise data quality management system capable of dynamically creating and updating rule networks, detecting and classifying abnormal data in real time, and ensuring efficient and stable operation under high concurrency and dynamic loads through automated error correction and resource optimization mechanisms.

[0003] Chinese invention patent application CN116756132A discloses a big data quality management system and method, including: performing data integrity management, constructing constraints, and setting integrity verification rules; performing data timeliness verification and constructing timeliness verification rules; performing data standardization verification and constructing standardization verification rules; performing data uniqueness verification according to a preset algorithm template; constructing accuracy judgment rules and performing accuracy verification through anomaly detection algorithms; performing consistency verification and visualizing the verified data. This invention provides a big data quality management method that presets multiple algorithm templates and verification rules to achieve automated monitoring of data quality, improve data quality monitoring efficiency, and effectively prevent inefficiencies and human errors caused by manual intervention.

[0004] Combining the above applications and existing technologies:

[0005] In the aforementioned complex scenarios, existing data quality management systems face two core technical challenges. First, achieving efficient and consistent synchronization of dynamic rule management and distribution in a distributed environment is difficult. Specifically, while the creation and storage of rule networks can support the visualization and efficient querying of multi-dimensional business logic, ensuring real-time synchronization and consistency of rule updates across all execution nodes through event-driven architecture and consistency protocols (such as Raft) remains a significant challenge when rules change. Failure to achieve real-time synchronization and consistency of rules, especially during peak periods or when rules change frequently, can lead to delays and inconsistencies in data verification, thereby affecting the accuracy of data processing and the overall stability of the system.

[0006] Secondly, in the processing of massive real-time data, another major challenge in ensuring data quality is how to efficiently detect, classify, and automatically correct abnormal data, while isolating and manually reviewing high-risk anomalies that cannot be automatically corrected. Traditional methods rely on static resource configuration and manual intervention, which are difficult to cope with dynamic load changes and complex business logic, resulting in low resource utilization and low processing efficiency.

[0007] To this end, the present invention provides an intelligent data precision quality management system, method, and apparatus. Summary of the Invention

[0008] (a) Technical problems to be solved

[0009] To address the shortcomings of existing technologies, this invention provides an intelligent data precision quality management system, method, and apparatus. It generates rule update events when rules change and publishes them via Kafka. During rule updates, it detects differences between old and new rules, automatically triggering a rollback mechanism when the difference exceeds expectations. It validates preprocessed data using preset rules, classifies abnormal data into several abnormal data categories, and automatically corrects the classified abnormal data. For abnormal data that still cannot be corrected, it identifies and classifies it, migrates the identified isolated data categories from the main data stream to an isolation queue, and retains abnormal data that cannot be processed immediately in the isolation queue, periodically reminding the data receiver to process it. Abnormalities that cannot be processed immediately remain in the isolation queue, with notifications sent to the data receiver, ensuring that potentially high-risk abnormalities do not accumulate indefinitely. This solves the technical problems described in the background section.

[0010] (II) Technical Solution

[0011] To achieve the above objectives, the present invention provides the following technical solution:

[0012] The intelligent data precision quality management method includes: after creating a rule network, generating rule update events and publishing them through Kafka when rules change; and after receiving new rules, dynamically adjusting the allocation of rule execution tasks based on real-time data traffic and rule complexity using a dynamic load balancing algorithm.

[0013] When updating rules, detect the differences between the old and new rules, generate a difference degree Xot from the difference data, and automatically trigger a rollback mechanism when the difference degree Xot exceeds the expectation. Collect performance indicators during the rule execution process to generate an abnormal indicator value Mav. If the abnormal indicator value Mav exceeds the expectation, issue a first-level alarm command to the outside.

[0014] The preprocessed data is validated using preset rules. After classifying the abnormal data, several abnormal data classes are obtained. The abnormal state data of each abnormal type are used to construct the data abnormality degree DAD(t). When the data abnormality degree DAD(t) exceeds the expectation, the classified abnormal data is automatically corrected.

[0015] For abnormal data that still cannot be corrected, identify and classify it, migrate the identified isolated data class from the main data stream to the isolation queue, retain abnormal data that cannot be processed immediately in the isolation queue, periodically remind the data receiving end to process it and constrain the reminder frequency;

[0016] When isolated abnormal data meets the alarm rules, an abnormal data report is sent to the data receiving end. After collecting the performance indicators of the data processing equipment, a load coefficient Λρ(t) is generated. When the load coefficient Λρ(t) exceeds the preset load threshold, the computing resources are expanded; otherwise, the computing resources are contracted.

[0017] Furthermore, a rule network is created and stored, with the rule structure including triggering conditions, execution actions, and priorities. A rule definition language based on a domain-specific language is used, and a graph database is utilized to model the dependencies and priorities between rules. An event-driven architecture is adopted, and an event stream is built through Apache Kafka for real-time push of rule updates. When a rule changes, a rule update event is generated and published to a Kafka topic. After subscribing to the relevant topic, the latest rules are obtained and applied in real time, and the Raft protocol is integrated.

[0018] Furthermore, upon receiving a new rule, it is parsed into an Abstract Syntax Tree (AST) based on the domain-specific language RDL; the AST is converted into an Intermediate Representation (IR), and the IR is dynamically compiled into efficient machine code using just-in-time (JIT) compilation, and the rule execution path is optimized through a JIT compiler.

[0019] Furthermore, a difference comparison algorithm is used to detect the differences between the old and new rules. After continuously acquiring the difference data of several rules, a difference score Xot is generated. The difference ratio Xs between two old and new rules is linearly normalized, and the corresponding data values ​​are mapped to the interval [0,1], as follows:

[0020]

[0021] Where n is the number of use cases, Xs i Xs is the difference ratio between the old and new rules in the i-th group. a The average difference ratio; weighting coefficients: 0≤F1≤1, 0≤F2≤1, and F1+F2=1.

[0022] Furthermore, real-time monitoring and logging are implemented to collect performance metrics during rule execution. Metrics exceeding preset ranges are identified as anomalous. The proportion and weight of anomalous metrics exceeding the range, along with the time of the anomaly, are recorded. These are then combined to generate anomaly values ​​(Mav), as follows:

[0023] Mav=∫∫∫ D (w L ·P L (t,x,y)+w T ·P T (t,x,y)+w R ·P R (t,x,y))dydxdt

[0024] In the formula: D is the integration region, defined as a specific time range [t0,t1], a spatial range [x0,x1], and other relevant dimensions [y0,y1]; w L To handle the weighting coefficients for delay, P L (t,x,y) represents the proportion of exceptions that handle delays; w T The weighting factor P for throughput T (t,x,y), P T (t,x,y) represents the proportion of abnormal throughput; P R (t,x,y) represents the anomaly proportion of the rule hit rate, w R This is the weighting coefficient for the rule hit rate.

[0025] Furthermore, upon receiving an alarm command, the received data is preprocessed, and each piece of preprocessed data is verified using preset verification rules. Data that violates the rules is marked as abnormal. The detected abnormal data is classified according to predefined classification standards, and classification standards are formulated based on the type of abnormality. The abnormal data is classified in multiple dimensions to obtain several abnormal data classes.

[0026] Furthermore, a multi-dimensional index is established for the categorized abnormal data, and the dynamic changes of the abnormal data are monitored to obtain the abnormal state data for each abnormality type; the data anomaly degree DAD(t) is generated from the abnormal state data.

[0027] When the data anomaly degree DAD(t) exceeds the preset threshold, each type of anomaly corresponds to a specific error correction rule. After summarizing, a set of data error correction rules is generated. The preset rule-based error correction logic is applied to the anomaly data in the anomaly data class, and the classified anomaly data is automatically corrected to generate corrected data.

[0028] Furthermore, taking abnormal data that cannot be automatically corrected by preset rules as input, the trained auxiliary correction model is used to predict and correct complex anomalies, and outputs reasonable predicted values ​​to replace or correct the original abnormal data.

[0029] Establish an isolation queue or database table to store abnormal data that cannot be automatically corrected, remove the isolation data class that cannot be automatically corrected from the main data stream, and migrate it to the isolation queue.

[0030] Furthermore, when transferring abnormal data to the isolation queue, detailed tags and metadata are automatically added to each piece of data, including but not limited to: abnormal type, rule ID, data source, occurrence time and processing status. The tags and metadata of abnormal data are updated during the migration process.

[0031] Furthermore, the isolated data enters a pending review state, waiting for the data review end to process it. After correcting the abnormal data, the data review end will re-inject the corrected data into the main data stream, or mark it as processed based on the processing result.

[0032] For abnormal data that cannot be processed immediately, it is kept in an isolation queue and the data receiver is periodically reminded to process it. The reminder frequency Rop is constrained. After a reminder interval that meets the constraints, a reminder instruction is sent to the data receiver.

[0033] Furthermore, alarm rules are set based on the type and quantity of anomalies. When isolated abnormal data meets the alarm rules, an abnormal data report containing detailed information about the abnormal data and processing suggestions is sent to the data receiving end.

[0034] Record the processing results of abnormal data and generate processing logs, including processing time, processing method and processing result. Feed the processing logs back to the rule engine and machine learning model to optimize the abnormal data processing strategy.

[0035] The intelligent data precision quality management system includes a rule building unit. After creating a rule network, it generates rule update events when rules change and publishes them through Kafka. Upon receiving new rules, the dynamic load balancing algorithm dynamically adjusts the allocation of rule execution tasks based on real-time data traffic and rule complexity.

[0036] The data difference analysis unit detects the differences between the old and new rules when the rules are updated, generates the difference degree Xot from the difference data, and automatically triggers the rollback mechanism when the difference degree Xot exceeds the expectation. It collects the performance indicators during the rule execution process and generates the indicator outlier value Mav. If the indicator outlier value Mav exceeds the expectation, it sends a first-level alarm command to the outside.

[0037] The data correction unit uses preset rules to verify the preprocessed data, classifies the abnormal data to obtain several abnormal data classes, constructs the data abnormality degree DAD(t) from the abnormal state data of each abnormality type, and automatically corrects the classified abnormal data when the data abnormality degree DAD(t) exceeds the expectation.

[0038] The abnormal data isolation unit identifies and classifies abnormal data that cannot be corrected, migrates the identified isolated data classes from the main data stream to the isolation queue, retains abnormal data that cannot be processed immediately in the isolation queue, periodically reminds the data receiving end to process it, and constrains the reminder frequency.

[0039] The response processing unit sends an abnormal data report to the data receiving end when the isolated abnormal data meets the alarm rules. After collecting the performance indicators of the data processing equipment, it generates a load coefficient Λρ(t). When the load coefficient Λρ(t) exceeds the preset load threshold, it expands the computing resources; otherwise, it shrinks the computing resources.

[0040] An intelligent data precision quality management device includes at least one processor;

[0041] A memory for storing executable instructions that are executed by at least one of the processors to cause the at least one of the processors to perform the steps of the intelligent data precision quality management method.

[0042] (III) Beneficial Effects

[0043] This invention provides an intelligent data precision quality management system, method, and apparatus, which have the following beneficial effects:

[0044] 1. The rollback method not only reduces the possibility of errors in manual investigation, but also avoids the impact on online business, ensuring the stability of continuous operation and data verification. By quantitatively recording the change points and differences and quickly formulating response measures, it can prevent unintentional major rule modifications from causing serious damage to the data processing flow, and ultimately protect the reliability and maintainability of the system under changing business needs.

[0045] 2. By collecting key performance indicators during rule execution and combining them with outlier detection and alarm strategies, we provide precise health monitoring and risk warning methods, and remind relevant teams to investigate and handle issues in a timely manner. This multi-dimensional and visualized performance monitoring not only prevents system bottlenecks caused by decreased rule execution efficiency or improper resource usage, but also reliably identifies potential risks and responds quickly in the face of complex and ever-changing e-commerce transaction environments.

[0046] 3. Pre-defined error correction rules are applied to abnormal data to automate the correction of most common anomalies. By combining multi-dimensional classification results with automatic correction actions, manual intervention is significantly reduced and the spread of errors to critical business processes is prevented. The corrected data obtained after applying the error correction rules can be quickly sent back to the normal data stream or subjected to secondary testing, further ensuring a complete closed loop for data quality.

[0047] 4. The tagging information includes the anomaly type, rule ID, data source, occurrence time, and processing status, which facilitates quick location and assignment of processing responsibilities. It effectively prevents difficult-to-correct abnormal data from entering the core business processing flow, avoiding a chain of errors or performance burdens in subsequent business logic.

[0048] 5. Manual error correction or marking is performed through the data audit end, so that complex or uncorrectable anomalies can be addressed with targeted solutions. Anomalies that cannot be handled immediately are kept in the isolation queue and the data receiver is notified with a constrained reminder frequency to ensure that potentially high-risk anomalies do not accumulate or are ignored in a long time.

[0049] 6. Automated intervention based on alarm rules effectively reduces the burden of manual screening, allowing operation and maintenance and business teams to make more efficient use of limited manpower to deal with truly urgent and high-risk issues, ensuring the overall operational quality of the system and business continuity.

[0050] 7. By adopting elastic scaling and recovery strategies based on load factors, the problems of resource waste or performance deficiencies in static configuration mode can be solved, improving processing capacity while maintaining cost efficiency. When high load anomalies occur frequently, resources can be prioritized for critical processes and high-risk tasks, achieving dual protection of performance and quality. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of the intelligent data precision quality management method of the present invention;

[0052] Figure 2 This is a schematic diagram of the intelligent data precision quality management system of the present invention;

[0053] Figure 3 This is a schematic diagram of the intelligent data precision quality management device of the present invention. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] Please see Figure 1 This invention provides an intelligent data precision quality management method, including:

[0056] Step 1: After creating the rule network, a rule update event is generated and published through Kafka when a rule changes. Upon receiving a new rule, the dynamic load balancing algorithm dynamically adjusts the allocation of rule execution tasks based on real-time data traffic and rule complexity.

[0057] Step one includes the following:

[0058] Step 101: Create and store a rule network, such as data processing rules, filtering rules, and detection rules. The rule structure includes elements such as triggering conditions, execution actions, and priorities. A rule definition language based on a domain-specific language is used, and the dependencies and priorities between rules are modeled using a graph database to achieve efficient querying and execution optimization of complex rule logic.

[0059] When used, it enables visualization and efficient querying of multi-dimensional business logic in complex data scenarios. By clearly decomposing elements such as triggering conditions, execution actions, and priorities, it ensures precise scheduling in a sequential or parallel manner during rule execution, thereby enhancing the controllability of the data processing flow. In a large-scale distributed environment, it can significantly improve the overall efficiency and reliability of data quality management.

[0060] Step 102: Adopt an event-driven architecture, using Apache Kafka to build a high-throughput, low-latency event stream for real-time push of rule updates; generate rule update events when rules change and publish them to Kafka topics; after subscribing to relevant topics, obtain and apply the latest rules in real time, and integrate the Raft protocol to ensure the consistency and reliability of rule updates across all execution nodes; through Kafka publishing, subscribing to, and receiving events, achieve instant synchronization and application of rules, ensuring the real-time performance and consistency of data verification; it should be noted that the key steps of the Raft protocol include: nodes electing a leader through voting; the leader receiving rule update requests and replicating them as log entries to all followers; after a majority of nodes confirm the log entries, the leader commits and notifies followers to apply the updates; all nodes apply the log entries to the state machine in the order of commit, ensuring rule consistency;

[0061] Kafka topics are a core concept in Apache Kafka, used to organize and categorize message streams. A topic can be viewed as a logical category of messages; producers publish messages to specified topics, while consumers subscribe to and read messages from these topics. To achieve high throughput and parallel processing, topics are typically divided into multiple partitions, each of which can be processed independently on different servers, thereby improving system scalability and fault tolerance. The design of topics enables Kafka to efficiently handle large-scale real-time data streams and is widely used in scenarios such as log collection, real-time analysis, and event-driven architectures, ensuring reliable data transmission and low latency.

[0062] Through an event-driven architecture, rule changes can be rapidly distributed to all execution nodes in a large-scale, multi-node distributed environment. By generating update events and publishing them to Kafka topics when rules change, subscribing nodes can obtain and apply the latest rules in real time, thereby ensuring the dynamic consistency of data verification and processing logic. This ensures that the accuracy and real-time performance of data processing can be maintained even during peak periods or when rules change frequently, and significantly reduces data anomalies caused by rule asynchrony.

[0063] Step 103: Upon receiving a new rule, the domain-specific language RDL is parsed into an abstract syntax tree (AST); the AST is converted into an intermediate representation (IR), and just-in-time (JIT) compilation is used to dynamically compile the intermediate representation IR into efficient machine code, thereby improving rule execution performance and response speed. The rule execution path is also optimized through the JIT compiler.

[0064] It integrates a dynamic load balancing algorithm to dynamically adjust the allocation of rule execution tasks based on real-time data traffic and rule complexity, thereby optimizing resource utilization and performance.

[0065] When using this method, refer to steps 101 to 103:

[0066] The JIT compiler can dynamically adjust the compilation strategy according to the real-time runtime environment and data characteristics, improving execution efficiency and reducing resource waste. In addition, the integrated dynamic load balancing algorithm allocates execution nodes according to rule complexity and traffic levels, which can further enhance the adaptability to high concurrency and uneven load scenarios, and effectively ensure the real-time performance and scalability of the data quality management process.

[0067] Step 2: Detect the differences between the old and new rules when the rules are updated, generate the difference index Xot from the difference data, and automatically trigger the rollback mechanism when the difference index Xot exceeds the expectation. Collect the performance indicators during the rule execution process to generate the indicator outlier value Mav. If the indicator outlier value Mav exceeds the expectation, issue a level 1 alarm command to the outside.

[0068] Step two includes the following:

[0069] Step 201: Record detailed change history and metadata when updating rules. Use a difference comparison algorithm to detect differences between old and new rules. After continuously acquiring the difference data of several rules, generate the difference score Xot.

[0070] The difference ratio Xs between the two rules, old and new, is linearly normalized, mapping the corresponding data values ​​to the interval [0,1] as follows:

[0071]

[0072] Where n is the number of use cases, Xs i Xs is the difference ratio between the old and new rules in the i-th group. a The average difference ratio; weighting coefficients: 0≤F1≤1, 0≤F2≤1, and F1+F2=1;

[0073] If the obtained difference Xot exceeds the preset difference threshold, it indicates that the difference between the new rule and the previous rule is large. At this time, the rollback mechanism is automatically triggered to restore the stable previous version of the rule, ensuring the security and traceability of the rule update and maintaining the continuity of data verification. The difference comparison algorithm (diff algorithm) is used to detect the difference between the new and old rules. If the obtained difference exceeds the set threshold, the rollback mechanism is automatically triggered, which can avoid the spread of potential risks caused by large changes.

[0074] This rollback method not only reduces the possibility of errors in manual investigation, but also avoids impacting online business and ensures the stability of continuous operation and data verification. By quantitatively recording change points and differences and quickly formulating response measures, it can prevent unintentional major rule modifications from causing serious damage to the data processing flow, ultimately protecting the reliability and maintainability of the system under changing business needs.

[0075] Step 202: Real-time monitoring and logging. Collect performance metrics during rule execution, such as processing latency, throughput, and rule hit rate. Metrics exceeding preset ranges are identified as anomalous. Record the proportion and weight of anomalous metrics, as well as the time of the anomaly. Under dimensionless conditions, combine these metrics to generate anomaly value Mav, as follows:

[0076] Mav=∫∫∫ D (w L ·P L (t,x,y)+w T ·P T (t,x,y)+w R ·P R (t,x,y))dy dx dt

[0077] In the formula: D is the integration region, defined as a specific time range [t0,t1], spatial range [x0,x1], and other relevant dimensions [y0,y1]; L(t,x,y) is the processing delay at time t and spatial location (x,y); w L To handle the weighting coefficients for delay, P L (t,x,y) represents the proportion of anomalies in the processing latency; T(t,x,y) represents the throughput at time t and spatial location (x,y); w T The weighting factor P for throughput T (t,x,y), P T (t,x,y) represents the anomaly ratio of throughput; R(t,x,y) represents the rule hit rate at time t and spatial location (x,y); P R (t,x,y) represents the anomaly proportion of the rule hit rate, w R These are the weighting coefficients for the rule hit rate; all weighting coefficients range from 0 to 1.

[0078] Based on historical data and management expectations for performance indicators, anomaly thresholds are preset. If the obtained abnormal value Mav exceeds the preset anomaly threshold, it indicates that there is an anomaly in the current rule execution process, and timely response and processing are required, issuing a level one alarm command to the outside.

[0079] When using it, refer to steps 201 and 202:

[0080] By collecting key performance indicators (such as processing latency, throughput, and rule hit rate) during rule execution, and combining these with outlier detection and alerting strategies, precise health monitoring and risk warning methods are provided, reminding relevant teams to investigate and address issues promptly. This multi-dimensional and visualized performance monitoring not only prevents system bottlenecks caused by decreased rule execution efficiency or improper resource usage, but also provides specific and quantifiable decision-making basis for continuous optimization of data processing workflows. In the face of complex and ever-changing e-commerce transaction environments, it can reliably identify potential risks and respond quickly, ensuring reliable business continuity.

[0081] Step 3: Verify the preprocessed data using preset rules, classify the abnormal data to obtain several abnormal data classes, construct the data anomaly degree DAD(t) from the abnormal state data of each anomaly type, and automatically correct the classified abnormal data when the data anomaly degree DAD(t) exceeds the expectation.

[0082] Step three includes the following:

[0083] Step 301: After receiving the alarm command, preprocess the received data, and use the preset verification rules to verify each piece of preprocessed data, marking data that violates the rules as abnormal;

[0084] The detected abnormal data is classified according to predefined classification criteria, and classification criteria are formulated based on the type of abnormality (such as numerical abnormality, format abnormality, logical abnormality, etc.). Combining different dimensions of the data (such as order amount, inventory quantity, date format, etc.), the abnormal data is classified in multiple dimensions to obtain several abnormal data classes.

[0085] By performing necessary preprocessing on the data and applying preset verification rules to each item, data that violates the rules can be quickly filtered out in high-concurrency environments, forming an accurate set of abnormal data. By further subdividing the abnormal data according to classification standards such as numerical abnormalities, format abnormalities, and logical abnormalities, the targeting of abnormality handling can be improved. Cross-checking information on key dimensions such as order amount, inventory quantity, and date format can more effectively discover deep-seated or hidden anomalies, providing protection against risk control vulnerabilities in e-commerce operations.

[0086] Step 302: Establish a multi-dimensional index for the classified abnormal data and monitor the dynamic changes of the abnormal data. After setting corresponding thresholds and rules for each type of abnormal data, obtain the abnormal data volume of each abnormal data type, the time node of receiving the abnormal data, and the similarity between different types of abnormal data, and summarize to generate abnormal status data; generate the data anomaly degree DAD(t) from the abnormal status data, as follows:

[0087]

[0088] In the formula: A is the weight vector, defined as Where w i The weight coefficients of the i-th type of anomaly satisfy the following condition: S is the similarity matrix, an N×N symmetric matrix, with elements S ij Let S represent the similarity between the i-th and j-th anomaly data, with a value range of [0.1], where S... ii =1; V(t′) is the outlier data vector, representing the data volume of each type of outlier at time t′, defined as... Where V i (t′) represents the amount of abnormal data received at time t′ for the i-th type of anomaly; W(t,t′) is the time weighting function matrix, an N×N matrix, defined as W ij (t,t′)=e -λ(t-t′) δ ij Where λ is the time decay factor, with a value greater than 1, controlling the rate at which the influence of outlier data decreases over time, and δ ijIt is the Kronecker delta function, where δ = j. ij =1, otherwise δij=0; o is the Hadamard product, representing element-wise multiplication of corresponding elements. Tensor product, representing the outer product of two vectors or matrices, generates a high-dimensional tensor. For convolution integral, it represents the integral over time t′ from t0 to the current time t, N: the total number of anomaly types;

[0089] When the data anomaly degree DAD(t) exceeds the preset threshold, it indicates that the current abnormal data class has a high degree of data anomaly and needs to be handled in a targeted manner. At this time, a level 2 alarm command is issued to the outside.

[0090] By building a multi-dimensional index for the classified abnormal data and monitoring the dynamic changes of the abnormal data, different types of abnormal data can still be quickly retrieved and processed in large-scale, high-concurrency scenarios; by generating the data anomaly degree DAD(t), it helps to grasp the evolution and trend of abnormal behavior in real-time operation and reduce the loss caused by information lag.

[0091] Specifically designed for peak e-commerce transaction periods, it can help identify and prioritize high-risk abnormal data types, preventing serious errors from escalating in downstream processes or affecting other business processes.

[0092] Step 303: After receiving the second-level alarm command, specific error correction rules are assigned to each type of abnormality. After summarizing, a set of data error correction rules is generated. The preset rule-based error correction logic is applied to the abnormal data in the abnormal data class to automatically correct the classified abnormal data and generate corrected data.

[0093] It should be noted that these error correction rules can be defined by business experts based on historical data and business needs, covering various common anomaly types, such as negative order amounts, abnormal inventory quantities, or incorrect date formats.

[0094] When using this method, refer to steps 301 to 303:

[0095] Pre-defined error correction rules are applied to abnormal data to automate the correction of most common anomalies. By combining multi-dimensional classification results with automatic correction actions, manual intervention is significantly reduced and the spread of errors to critical business processes is prevented. Corrected data obtained after applying the error correction rules can be quickly returned to the normal data stream or subjected to secondary testing, further ensuring a complete closed loop for data quality. When the error correction rules are inapplicable or automatic correction fails, the system can also intelligently trigger isolation processes or manual review channels, improving efficiency while ensuring security and stability.

[0096] Step 4: Identify and classify the abnormal data that still cannot be corrected, migrate the identified isolated data class from the main data stream to the isolation queue, retain the abnormal data that cannot be processed immediately in the isolation queue, periodically remind the data receiving end to process it and constrain the reminder frequency;

[0097] Step four includes the following:

[0098] Step 401: Train a machine learning algorithm using labeled sample data to obtain a trained auxiliary correction model; use abnormal data that cannot be automatically corrected by preset rules as input, use the trained auxiliary correction model to predict and correct complex anomalies, and output the predicted reasonable value to replace or correct the original abnormal data.

[0099] For anomalous data that still cannot be corrected, identify and classify it to identify and isolate data categories, including:

[0100] Complex anomalies: Anomalies involving multiple field dependencies or complex business logic, such as cross-table data inconsistency; Uncommon anomalies: Anomaly patterns that are rare or unseen in historical data; High-risk anomalies: Anomaly data that may have a significant impact on business processes or customer experience.

[0101] For abnormal data that cannot be automatically corrected by preset rules, deeper prediction and correction are performed. By analyzing historical data and complex business logic, more accurate correction suggestions or values ​​can be given for anomalies that deviate from the normal range but do not meet simple error correction rules. This significantly reduces the reliance on manual review and effectively makes up for the possible coverage deficiencies of rule-based error correction strategies. After identifying complex, uncommon, and high-risk anomalies that are difficult to correct, further isolation or high-level alarms are performed based on their potential harm or dependency relationships. In the highly dynamic e-commerce environment, the system achieves a full integration of automated and intelligent processing, enabling it to maintain a high detection rate and low false alarm rate when facing a large and ever-changing anomaly scenario, and continuously improve data quality and business reliability.

[0102] Step 402: Establish an isolation queue or database table to store abnormal data that cannot be automatically corrected, remove the isolation data class that cannot be automatically corrected from the main data stream, and migrate it to the isolation queue.

[0103] When transferring abnormal data to the isolation queue, detailed tags and metadata are automatically added to each piece of data, including but not limited to: Abnormal type: such as numerical abnormality, format abnormality, and logical abnormality; Rule ID: the specific rule identifier that triggered the abnormality; Data source: the source system or module of the abnormal data; Occurrence time: the specific timestamp of the abnormal data generation; Processing status: the current processing stage, such as pending review, in process, processed, etc.

[0104] During the migration process, update the tags and metadata of abnormal data, and record the processing history of all isolated data, including data migration time, personnel involved, and processing results.

[0105] When used, the tagging information includes anomaly type, rule ID, data source, occurrence time, and processing status, which facilitates quick location and assignment of processing responsibilities and reduces the cost of repeated investigations; the isolated data is also persistently managed, and by recording migration time, processing personnel, and processing results, the traceability of the entire processing process is ensured.

[0106] It effectively prevents uncorrectable abnormal data from entering the core business processing flow, avoiding a chain of errors or performance burdens in subsequent business logic; when downstream processing or manual review is required, the system can immediately retrieve isolated data for in-depth investigation or manual error correction, achieving a balance between security and efficiency.

[0107] Step 403: The isolated data enters the pending review state, waiting for the data review end to process it. After the data review end corrects the abnormal data, it will re-inject the corrected data into the main data stream, or mark it as processed according to the processing result.

[0108] For abnormal data that cannot be processed immediately, it is retained in an isolation queue, and the data receiver is periodically reminded to process it. The reminder frequency (Rop) is constrained. After a reminder interval that meets the constraints, a reminder instruction is sent to the data receiver. The constraint method is as follows:

[0109]

[0110] In the formula: N is the total number of exception types, i is the index of the exception type, i = 1, 2, ..., N, Δ i σ is the weighting coefficient for anomaly type i. i (t) is the priority function for exception type i, based on the current system load U(t) and the urgency level of the exception E. i (t) and historical processing efficiency H i (t) is calculated;

[0111]

[0112] Among them, E i H(t) represents the urgency score of anomaly type i at time t; U(t) is the load index at time t, ranging from [0, 1], H i (t) represents the historical processing efficiency of anomaly type i at time t, and ∈ is a small constant to prevent the denominator from being zero;

[0113] H(x) is the Heaviside step function, defined as: tlast,i The time point when the alert command was last sent for exception type i;

[0114] Δt i (t) represents the minimum dynamic alert interval for exception type i. Δt 0,i This is the minimum alert interval for exception type i. This is a load regulation parameter, with a value between 0 and 1, γ. i This is a reminder frequency adjustment parameter that controls the degree to which the current reminder frequency affects the reminder interval; its value is between 0 and 1.

[0115] P i (t) represents the current alert frequency for exception type i, φ i (N i (t)): The abnormal data volume function of abnormal type i, defined as: N i (t) represents the amount of abnormal data in the isolation queue for abnormal type i at time t; N i (t) represents the threshold for the amount of abnormal data of anomaly type i.

[0116] When using this method, refer to steps 401 and 403:

[0117] Once isolated data enters the pending review state, manual error correction or marking is performed through the data review platform. This allows for targeted solutions to complex or uncorrectable anomalies, ultimately ensuring data quality management. Anomalies that cannot be processed immediately remain in the isolation queue, with data receivers notified at a constrained frequency to ensure that potentially high-risk anomalies do not accumulate or go unnoticed. This approach ensures both flexibility and security in highly dynamic e-commerce transaction scenarios. It can handle numerous common anomalies through automated logic while also allowing for orderly manual intervention for complex or high-risk anomalies, forming a multi-layered, closed-loop control mechanism.

[0118] Step 5: When the isolated abnormal data meets the alarm rules, send an abnormal data report to the data receiving end, collect the performance indicators of the data processing equipment and generate the load coefficient Λρ(t), and take corresponding response strategies based on the relationship between the load coefficient Λρ(t) and the load threshold.

[0119] Step five includes the following:

[0120] Step 501: Set alarm rules based on anomaly type and quantity. When isolated abnormal data meets the alarm rules, send an abnormal data report containing detailed information about the abnormal data and processing suggestions to the data receiving end.

[0121] Record the processing results of abnormal data and generate processing logs, including information such as processing time, processing method and processing results. Feed the processing logs back to the rule engine and machine learning model to optimize the abnormal data processing strategy.

[0122] When in use, it outputs alarm commands to the data receiving end along with detailed abnormal data reports and handling suggestions, which greatly improves the response speed and handling efficiency of high-risk anomalies. This automated intervention based on alarm rules effectively reduces the burden of manual screening, allowing operation and maintenance and business teams to make more efficient use of limited manpower to deal with truly urgent and high-risk problems, ensuring the overall operational quality of the system and business continuity.

[0123] Step 502: Collect key performance indicators of each component during data processing, including CPU utilization, memory usage, network bandwidth and disk I / O rate, etc. Under dimensionless conditions, generate the load coefficient Λρ(t) from the performance indicators as follows;

[0124]

[0125] P(t): Performance index vector, defined as... P C (t) represents CPU utilization, P M (t) represents memory usage, P N (t) represents network bandwidth usage, P D (t) represents the disk I / O rate, and W is the weighting matrix, which is a K×K symmetric matrix. w kk Let w be the weight of the performance index k. km λ is the interaction weight between performance indicators k and m, and λ is the time dynamic coefficient, which takes a value greater than 0, to control the sensitivity of the load coefficient to the rate of load change. δ is the rate of change of the load factor at time (t-1). k For nonlinear interaction coefficients, the value is greater than 1, K = 4, which is the total number of performance indicators; e can be 2.718.

[0126] Define strategies for resource expansion and contraction, including upper and lower limits for resource allocation, adjustment thresholds and step sizes; expand computing resources, such as adding more servers, when the load coefficient Λρ(t) exceeds the preset load threshold, and contract computing resources, such as reducing the configuration of existing computer resources.

[0127] When using this method, refer to steps 501 and 502:

[0128] By collecting key performance indicators of each component and generating a load factor Λρ(t), and adopting elastic scaling and recycling strategies based on the load factor Λρ(t), the problems of resource waste or performance deficiencies in static configuration mode can be solved. It can continuously adapt to market fluctuations in the ever-changing e-commerce environment, improve processing capacity while maintaining cost efficiency. When high load anomalies occur frequently, resources can be prioritized for critical processes and high-risk tasks, achieving dual protection of performance and quality.

[0129] Please see Figure 2 and 3 This invention provides an intelligent data precision quality management system, including:

[0130] The rule building unit creates a rule network and generates rule update events when rules change, which are then published via Kafka. Upon receiving new rules, the dynamic load balancing algorithm dynamically adjusts the allocation of rule execution tasks based on real-time data traffic and rule complexity.

[0131] The data difference analysis unit detects the differences between the old and new rules when the rules are updated, generates the difference degree Xot from the difference data, and automatically triggers the rollback mechanism when the difference degree Xot exceeds the expectation. It collects the performance indicators during the rule execution process and generates the indicator outlier value Mav. If the indicator outlier value Mav exceeds the expectation, it sends a first-level alarm command to the outside.

[0132] The data correction unit uses preset rules to verify the preprocessed data, classifies the abnormal data to obtain several abnormal data classes, constructs the data abnormality degree DAD(t) from the abnormal state data of each abnormality type, and automatically corrects the classified abnormal data when the data abnormality degree DAD(t) exceeds the expectation.

[0133] The abnormal data isolation unit identifies and classifies abnormal data that cannot be corrected, migrates the identified isolated data classes from the main data stream to the isolation queue, retains abnormal data that cannot be processed immediately in the isolation queue, periodically reminds the data receiving end to process it, and constrains the reminder frequency.

[0134] The response processing unit sends an abnormal data report to the data receiving end when the isolated abnormal data meets the alarm rules. After collecting the performance indicators of the data processing equipment, it generates a load coefficient Λρ(t). When the load coefficient Λρ(t) exceeds the preset load threshold, it expands the computing resources; otherwise, it shrinks the computing resources.

[0135] This invention provides an intelligent data precision quality management device, comprising at least one processor;

[0136] A memory for storing executable instructions that are executed by at least one of the processors to cause the at least one of the processors to perform the steps of the intelligent data precision quality management method.

[0137] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0138] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0139] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0140] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0141] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An intelligent data-driven precision quality management method, characterized by: include, After creating the rule network, a rule update event is generated and published through Kafka when a rule changes. Upon receiving a new rule, the dynamic load balancing algorithm dynamically adjusts the allocation of rule execution tasks based on real-time data traffic and rule complexity. When updating rules, detect the differences between the old and new rules. When the difference degree Xot generated by the difference data exceeds the expectation, automatically trigger the rollback mechanism. Collect performance indicators during the rule execution process to generate an abnormal indicator value Mav. If the abnormal indicator value Mav exceeds the expectation, issue a first-level alarm command to the outside. The preprocessed data is validated using preset rules. After classifying the abnormal data, several abnormal data classes are obtained. The abnormal state data of each abnormal type are used to construct the data abnormality degree DAD(t). When the data abnormality degree DAD(t) exceeds the expectation, the classified abnormal data is automatically corrected. For abnormal data that still cannot be corrected, identify and classify it, migrate the identified isolated data class from the main data stream to the isolation queue, retain abnormal data that cannot be processed immediately in the isolation queue, periodically remind the data receiving end to process it and constrain the reminder frequency; When isolated abnormal data meets the alarm rules, an abnormal data report is sent to the data receiving end. After collecting the performance indicators of the data processing equipment, a load coefficient Λρ(t) is generated. When the load coefficient Λρ(t) exceeds the preset load threshold, the computing resources are expanded; otherwise, the computing resources are contracted.

2. The intelligent data precision quality management method according to claim 1, characterized in that: Create and store a rule network. The rule structure includes triggering conditions, execution actions, and priorities. A rule definition language based on a domain-specific language is used, and a graph database is utilized to model the dependencies and priorities between rules. An event-driven architecture is adopted, and an event stream is built through Apache Kafka for real-time push of rule updates. When a rule changes, a rule update event is generated and published to a Kafka topic. After subscribing to the relevant topic, the latest rules are obtained and applied in real time, and the Raft protocol is integrated. Upon receiving a new rule, it is parsed into an Abstract Syntax Tree (AST) based on the domain-specific language RDL; the AST is then converted into an Intermediate Representation (IR), which is dynamically compiled into efficient machine code using just-in-time (JIT) compilation, and the rule execution path is optimized using a JIT compiler.

3. The intelligent data precision quality management method according to claim 2, characterized in that: A difference comparison algorithm is used to detect the differences between new and old rules. After continuously acquiring the difference data of several rules, a difference score Xot is generated. The difference ratio Xs between two new and old rules is linearly normalized, and the corresponding data values ​​are mapped to the interval [0, 1], as follows: Where n is the number of use cases, Xs i Xs is the difference ratio between the old and new rules in the i-th group. a The average difference ratio; weighting coefficients: 0≤F1≤1, 0≤F2≤1, and F1+F2=1.

4. The intelligent data precision quality management method according to claim 3, characterized in that: Real-time monitoring and logging are implemented to collect performance metrics during rule execution. Metrics exceeding preset ranges are identified as anomalous. The proportion and weight of anomalous metrics exceeding the range, along with the time of the anomaly, are recorded. These are then combined to generate the anomalous metric value Mav, as follows: Mav=∫∫∫ D (w L ·P L (t,x,y)+w T ·P T (t,x,y)+w R ·P R (t,x,y))dy dx dt In the formula: D is the integration region, defined as a specific time range [t0, t1], a spatial range [x0, x1], and other relevant dimensions [y0, y1]; w L To handle the weighting coefficients for delay, P L (t, x, y) represents the proportion of exceptions that handle delays; w T P is the weighting factor for throughput. T (t, x, y) represents the proportion of abnormal throughput; P R (t, x, y) represents the anomaly proportion of the rule hit rate, w R This is the weighting coefficient for the rule hit rate.

5. The intelligent data precision quality management method according to claim 4, characterized in that: Upon receiving an alarm command, the received data is preprocessed, and each piece of preprocessed data is verified using preset verification rules. Data that violates the rules is marked as abnormal. The detected abnormal data is classified according to predefined classification criteria, and classification criteria are formulated based on the anomaly type; the abnormal data is classified in multiple dimensions to obtain several abnormal data classes; A multi-dimensional index is built for the classified abnormal data and the dynamic changes of the abnormal data are monitored to obtain the abnormal status data of each abnormal type; the abnormality degree DAD(t) is generated from the abnormal status data. When the data anomaly degree DAD(t) exceeds the preset threshold, each type of anomaly corresponds to a specific error correction rule. After summarizing, a set of data error correction rules is generated. The preset rule-based error correction logic is applied to the anomaly data in the anomaly data class, and the classified anomaly data is automatically corrected to generate corrected data.

6. The intelligent data precision quality management method according to claim 5, characterized in that: Using abnormal data that cannot be automatically corrected by preset rules as input, the trained auxiliary correction model is used to predict and correct complex anomalies, and outputs reasonable predicted values ​​to replace or correct the original abnormal data. Establish an isolation queue or database table to store abnormal data that cannot be automatically corrected, remove the isolation data class that cannot be automatically corrected from the main data stream, and migrate it to the isolation queue; When transferring abnormal data to the isolation queue, detailed tags and metadata are automatically added to each piece of data, including but not limited to: abnormal type, rule ID, data source, occurrence time and processing status. The tags and metadata of abnormal data are updated during the migration process.

7. The intelligent data precision quality management method according to claim 6, characterized in that: After isolation, the data enters a pending review state, waiting for the data review end to process it. After correcting the abnormal data, the data review end will re-inject the corrected data into the main data stream, or mark it as processed based on the processing result. For abnormal data that cannot be processed immediately, it is kept in an isolation queue and the data receiver is periodically reminded to process it. The reminder frequency Rop is constrained. After a reminder interval that meets the constraints, a reminder instruction is sent to the data receiver.

8. The intelligent data precision quality management method according to claim 7, characterized in that: Set alarm rules based on anomaly type and quantity. When isolated abnormal data meets the alarm rules, send an abnormal data report containing detailed information about the abnormal data and processing suggestions to the data receiving end. Record the processing results of abnormal data and generate processing logs, including processing time, processing method and processing result. Feed the processing logs back to the rule engine and machine learning model to optimize the abnormal data processing strategy.

9. An intelligent data-driven precision quality management system, characterized by: include, The rule building unit creates a rule network and generates rule update events when rules change, which are then published via Kafka. Upon receiving new rules, the dynamic load balancing algorithm dynamically adjusts the allocation of rule execution tasks based on real-time data traffic and rule complexity. The data difference analysis unit detects the differences between the old and new rules when the rules are updated, generates the difference degree Xot from the difference data, and automatically triggers the rollback mechanism when the difference degree Xot exceeds the expectation. It collects the performance indicators during the rule execution process and generates the indicator outlier value Mav. If the indicator outlier value Mav exceeds the expectation, it sends a first-level alarm command to the outside. The data correction unit uses preset rules to verify the preprocessed data, classifies the abnormal data to obtain several abnormal data classes, constructs the data abnormality degree DAD(t) from the abnormal state data of each abnormality type, and automatically corrects the classified abnormal data when the data abnormality degree DAD(t) exceeds the expectation. The abnormal data isolation unit identifies and classifies abnormal data that cannot be corrected, migrates the identified isolated data classes from the main data stream to the isolation queue, retains abnormal data that cannot be processed immediately in the isolation queue, periodically reminds the data receiving end to process it, and constrains the reminder frequency. The response processing unit sends an abnormal data report to the data receiving end when the isolated abnormal data meets the alarm rules. After collecting the performance indicators of the data processing equipment, it generates a load coefficient Λρ(t). When the load coefficient Λρ(t) exceeds the preset load threshold, it expands the computing resources; otherwise, it shrinks the computing resources.

10. An intelligent data-driven precision quality management device, characterized in that: include, At least one processor; A memory for storing executable instructions that are executed by at least one of the processors to cause at least one of the processors to perform the steps of the method as claimed in any one of claims 1 to 8.

Images