Intelligent accurate data quality management system, method and device

Abstract

The invention discloses an intelligent accurate data quality management system, method and device, and relates to the technical field of data quality management. When a rule is changed, a rule updating event is generated and published through Kafka, when the rule is updated, the difference between a new rule and an old rule is detected, and when the difference exceeds the expectation, a rollback mechanism is automatically triggered, so that the accuracy of the data quality management is improved. Verifying the preprocessed data by using a preset rule, classifying the abnormal data to obtain a plurality of abnormal data classes, and automatically correcting the classified abnormal data; identifying and classifying abnormal data which still cannot be corrected, migrating an isolated data class obtained by identification to an isolated queue from a main data stream, retaining the abnormal data which cannot be processed immediately in the isolated queue, and regularly reminding a data receiving end to process the abnormal data; and the exceptions which cannot be processed in real time are continuously left in the isolation queue, and a data receiving end is notified at a constrained reminding frequency so as to ensure that the potential high-risk exceptions are not accumulated for a long time.

Description

Technical Field

[0001] The present invention relates to the technical field of data quality management, and specifically to an intelligent data precision quality management system, method and device. Background Art

[0002] In the e-commerce industry, enterprises need to process multi-source data such as order transactions, product inventory, and user behavior in real time. These data are not only large in volume and diverse in types, but also have a significant increase in data traffic during peak promotion periods. The quality of data directly affects the stable operation of the system, the accuracy of business decisions, and customer experience. For example, if data anomalies such as negative order amounts, incorrect inventory quantities, or abnormal timestamps are not detected and processed in a timely manner, it will lead to business process interruptions, inventory management errors, and increased customer complaints. To meet this challenge, enterprises need to build an intelligent data precision quality management system that can dynamically create and update rule networks, detect and classify abnormal data in real time, and ensure that the system can still operate efficiently and stably under high concurrency and dynamic loads through automated error correction and resource optimization mechanisms.

[0003] In the Chinese invention patent with application publication number CN116756132A, a big data quality management system and method are disclosed, including: integrity management of data, constraint construction, and setting integrity verification rules; timeliness verification of data, construction of timeliness verification rules; normative verification of data, construction of normative verification rules; uniqueness verification of data according to preset algorithm templates; construction of accuracy determination rules, and accuracy verification through anomaly detection algorithms; consistency verification, and visualization of verified data. A big data quality management method provided by an embodiment of the present invention presets multiple algorithm templates and verification rules to realize automatic monitoring of data quality, improve data quality monitoring efficiency, and effectively prevent the possibility of inefficiency and human errors caused by manual intervention.

[0004] Combining the above applications and the prior art:

[0005] In the above complex scenarios, existing data quality management systems face two core technical problems. First, dynamic rule management and distribution are difficult to achieve efficient and consistent synchronization in a distributed environment. Specifically, although the creation and storage of rule networks can support the visualization and efficient query of multi-dimensional business logic, when the rules change, how to ensure the real-time synchronization and consistency of rule updates among all execution nodes through event-driven architecture and consistency protocols (such as Raft) is still a major challenge. If the instant synchronization and consistency of rules cannot be achieved, especially during peak periods or when the rules change frequently, it may cause delays and inconsistencies in data verification, which in turn affects the accuracy of data processing and the overall stability of the system.

[0006] Secondly, in the processing of massive real-time data, how to efficiently detect, classify and automatically correct abnormal data, while isolating and manually reviewing high-risk anomalies that cannot be automatically corrected, is another major challenge in ensuring data quality. Traditional methods rely on static resource configuration and manual intervention, which are difficult to cope with dynamic load changes and complex business logic, resulting in low resource utilization and low processing efficiency.

[0007] To this end, the present invention provides an intelligent data precision quality management system, method and device. Summary of the invention

[0008] 1. Technical issues to be resolved

[0009] In view of the deficiencies in the prior art, the present invention provides an intelligent data precision quality management system, method and device, which generates a rule update event when a rule is changed and publishes it through Kafka, detects the difference between the new and old rules when the rule is updated, and automatically triggers a rollback mechanism when the difference exceeds expectations, verifies the preprocessed data using preset rules, obtains several abnormal data classes after classifying the abnormal data, and automatically corrects the classified abnormal data; identifies and classifies the abnormal data that still cannot be corrected, migrates the identified and obtained isolated data classes from the main data stream to the isolation queue, retains the abnormal data that cannot be processed immediately in the isolation queue, and regularly reminds the data receiving end to process it; the exceptions that cannot be processed immediately are continued to remain in the isolation queue, and the data receiving end is reminded to ensure that potential high-risk exceptions will not accumulate for a long time; thereby solving the technical problems recorded in the background technology.

[0010] (II) Technical solution

[0011] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0012] Intelligent data precision quality management method, including, after creating the rule network, generating rule update events when the rules are changed and publishing them through Kafka, and after receiving the new rules, the dynamic load balancing algorithm dynamically adjusts the allocation of rule execution tasks according to the real-time data flow and rule complexity;

[0013] When updating rules, the difference between the new and old rules is detected, and the difference degree Xot is generated from the difference data. When the difference degree Xot exceeds expectations, the rollback mechanism is automatically triggered. The performance indicators during the rule execution process are collected to generate the indicator abnormal value Mav. If the indicator abnormal value Mav exceeds expectations, a first-level alarm instruction is issued to the outside.

[0014] Use preset rules to verify the preprocessed data, classify the abnormal data and obtain several abnormal data classes, construct the data abnormality degree DAD(t) from the abnormal state data of each abnormal type, and automatically correct the classified abnormal data when the data abnormality degree DAD(t) exceeds expectations;

[0015] Identify and classify abnormal data that still cannot be corrected, migrate the identified isolated data class from the main data stream to the isolation queue, retain the abnormal data that cannot be processed immediately in the isolation queue, regularly remind the data receiving end to process it and restrict the reminder frequency;

[0016] When the isolated abnormal data meets the alarm rules, an abnormal data report is sent to the data receiving end. After collecting the performance indicators of the data processing equipment, the load coefficient Λρ(t) is generated. When the load coefficient Λρ(t) exceeds the preset load threshold, the computing resources are expanded, otherwise the computing resources are contracted.

[0017] Furthermore, a rule network is created and stored, and the rule structure includes trigger conditions, execution actions and priorities. A rule definition language based on a domain-specific language is used, and the dependencies and priorities between rules are modeled using a graph database. An event-driven architecture is used to build event streams through Apache Kafka for real-time push of rule updates. When rules change, rule update events are generated and published to Kafka topics. After subscribing to relevant topics, the latest rules are obtained and applied in real time and the Raft protocol is integrated.

[0018] Furthermore, after receiving the new rule, the domain-specific language RDL is parsed into an abstract syntax tree AST; the AST is converted into an intermediate representation IR, and the intermediate representation IR is dynamically compiled into efficient machine code using just-in-time compilation, and the rule execution path is optimized through the JIT compiler.

[0019] Furthermore, a difference comparison algorithm is used to detect the difference between the new and old rules. After continuously obtaining the difference data of several rules, the difference degree Xot is generated, wherein the difference ratio Xs between the two new and old rules is linearly normalized, and the corresponding data value is mapped to the interval [0,1] in the following manner:

[0020]

[0021] Where n is the number of usage scenarios, Xs i is the difference ratio between the new and old rules of the i-th group, Xs a It is the difference ratio average value; weight coefficient: 0≤F1≤1, 0≤F2≤1, and F1+F2=1.

[0022] Furthermore, real-time monitoring and log recording are performed to collect performance indicators during rule execution, and performance indicators that exceed the preset range are regarded as abnormal indicators; the proportion and weight of abnormal indicators exceeded, and the abnormal time node are recorded, and then the indicator abnormal value Mav is generated after combining them, as follows:

[0023] Mav=∫∫∫ D (w L ·P L (t,x,y)+w T ·P T (t,x,y)+w R ·P R (t,x,y))dydxdt

[0024] Where: D is the integration area, which is defined as a specific time range [t0, t1], spatial range [x0, x1] and other related dimensions [y0, y1]; w L is the weight coefficient of processing delay, P L (t,x,y) is the abnormal proportion of processing delay; w T is the throughput weight coefficient P T (t,x,y),P T (t,x,y) is the abnormal proportion of throughput; P R (t,x,y) is the abnormal proportion of the rule hit rate, w R is the weight coefficient of the rule hit rate.

[0025] Furthermore, after receiving the alarm instruction, the received data is preprocessed, and the preprocessed data is checked one by one using preset verification rules, and the data that violates the rules is marked as abnormal; the detected abnormal data is classified according to predefined classification standards, and classification standards are formulated according to the abnormal type; the abnormal data is classified in multiple dimensions to obtain several abnormal data classes.

[0026] Furthermore, a multi-dimensional index is established for the classified abnormal data and the dynamic changes of the abnormal data are monitored to obtain the abnormal state data of each abnormal type; the data abnormality degree DAD(t) is generated from the abnormal state data.

[0027] When the data anomaly degree DAD(t) exceeds the preset threshold, each type of anomaly corresponds to a specific error correction rule, which is summarized to generate a data error correction rule set. The preset regularized error correction logic is applied to the abnormal data in the abnormal data class, and the classified abnormal data is automatically corrected to generate corrected data.

[0028] Furthermore, taking abnormal data that cannot be automatically corrected by preset rules as input, the trained auxiliary correction model is used to predict and correct complex anomalies, and the predicted reasonable value is output to replace or correct the original abnormal data;

[0029] Establish an isolation queue or database table to store abnormal data that cannot be automatically corrected, remove the isolated data class that cannot be automatically corrected from the main data flow, and migrate it to the isolation queue.

[0030] Furthermore, when transferring abnormal data to the isolation queue, detailed tags and metadata are automatically added to each piece of data, including but not limited to: abnormality type, rule ID, data source, occurrence time and processing status, and the tags and metadata of abnormal data are updated during the migration process.

[0031] Furthermore, the isolated data enters a pending review state, waiting for processing by the data review end. After the data review end corrects the abnormal data, it re-injects the corrected data into the main data stream, or marks it as processed according to the processing result;

[0032] For abnormal data that cannot be processed immediately, it is retained in the isolation queue, and the data receiving end is regularly reminded to process it, and the reminder frequency Rop is constrained. After the reminder interval that meets the constraint conditions, a reminder instruction is issued to the data receiving end.

[0033] Furthermore, alarm rules based on the type and quantity of abnormalities are set, and when the isolated abnormal data meets the alarm rules, an abnormal data report containing detailed information of the abnormal data and processing suggestions is sent to the data receiving end;

[0034] Record the processing results of abnormal data and generate a processing log, including the processing time, processing method and processing results. Feed the processing log back to the rule engine and machine learning model to optimize the abnormal data processing strategy.

[0035] Intelligent data precision quality management system, including: rule building unit, after creating the rule network, generates rule update events when the rules are changed and publishes them through Kafka. After receiving the new rules, the dynamic load balancing algorithm dynamically adjusts the allocation of rule execution tasks according to the real-time data flow and rule complexity;

[0036] The data difference analysis unit detects the difference between the new and old rules when the rules are updated, generates the difference degree Xot from the difference data, and automatically triggers the rollback mechanism when the difference degree Xot exceeds expectations. It collects the performance indicators in the rule execution process to generate the indicator abnormal value Mav. If the indicator abnormal value Mav exceeds expectations, a first-level alarm instruction is issued to the outside.

[0037] The data correction unit verifies the preprocessed data using preset rules, obtains several abnormal data classes after classifying the abnormal data, constructs the data abnormality degree DAD(t) from the abnormal state data of each abnormal type, and automatically corrects the classified abnormal data when the data abnormality degree DAD(t) exceeds expectations;

[0038] The abnormal data isolation unit identifies and classifies abnormal data that cannot be corrected, migrates the identified isolated data class from the main data stream to the isolation queue, retains the abnormal data that cannot be processed immediately in the isolation queue, regularly reminds the data receiving end to process it and restricts the reminder frequency;

[0039] The response processing unit sends an abnormal data report to the data receiving end when the isolated abnormal data meets the alarm rules, generates a load coefficient Λρ(t) after collecting the performance indicators of the data processing equipment, and expands the computing resources when the load coefficient Λρ(t) exceeds the preset load threshold, otherwise it shrinks the computing resources.

[0040] An intelligent data precision quality management device, comprising at least one processor;

[0041] The memory is used to store executable instructions, and the instructions are executed by at least one of the processors to enable at least one of the processors to perform the steps of the intelligent data precision quality management method.

[0042] (III) Beneficial effects

[0043] The present invention provides an intelligent data precision quality management system, method and device, which have the following beneficial effects:

[0044] 1. The rollback method not only reduces the possibility of manual troubleshooting errors, but also avoids the impact on online business, ensuring the stability of continuous operation and data verification; by quantitatively recording the change points and differences and quickly formulating response measures, it can prevent unintentional major rule modifications from causing serious damage to the data processing process, and ultimately protect the reliability and maintainability of the system under changing business requirements.

[0045] 2. By collecting key performance indicators during the rule execution process, combined with the abnormal value judgment and alarm strategy of these indicators, accurate health monitoring and risk warning methods are provided, and relevant teams are reminded to investigate and deal with them in a timely manner; this multi-dimensional, visual performance monitoring not only prevents system bottlenecks caused by reduced rule execution efficiency or improper resource use, but also can reliably identify potential risks and respond quickly when facing complex and changeable e-commerce transaction environments.

[0046] 3. Apply preset error correction rule logic to abnormal data to achieve automatic correction of most common abnormal data; by combining multi-dimensional classification results with automatic correction actions, it significantly reduces manual intervention and avoids the spread of errors to key business links. The corrected data obtained after applying the error correction rules can be quickly sent back to the normal data flow or for secondary detection, further ensuring the complete closed loop of data quality.

[0047] 4. Marking information includes exception type, rule ID, data source, occurrence time and processing status, which facilitates rapid location and allocation of processing responsibilities, effectively preventing difficult-to-correct abnormal data from entering the core business processing flow, and avoiding chain errors or performance burdens in subsequent business logic.

[0048] 5. Manual error correction or marking is performed on the data audit end to provide targeted solutions for complex or automatically uncorrectable anomalies. Exceptions that cannot be handled immediately will remain in the isolation queue and the data receiving end will be notified at a constrained reminder frequency to ensure that potential high-risk anomalies will not accumulate or be ignored for a long time.

[0049] 6. Automated intervention based on alarm rules effectively reduces the burden of manual screening, allowing operation and maintenance and business teams to more efficiently use limited manpower to deal with truly urgent and high-risk issues, ensuring the overall operation quality of the system and business continuity.

[0050] 7. Adopting elastic expansion and recovery strategies based on the load factor can solve the problem of resource waste or performance deficiency in the static configuration mode, improve processing capacity while maintaining cost efficiency. When high-load exceptions occur frequently, resources can be allocated to key processes and high-risk tasks first, achieving dual guarantees of performance and quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is a flow chart of the intelligent data precision quality management method of the present invention;

[0052] Figure 2 This is a schematic diagram of the structure of the intelligent data precision quality management system of the present invention;

[0053] Figure 3 It is a schematic diagram of the structure of the intelligent data precision quality management device of the present invention. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] See also Figure 1 The present invention provides an intelligent data precision quality management method, including:

[0056] Step 1: After creating the rule network, when the rules are changed, rule update events are generated and published through Kafka. After receiving the new rules, the dynamic load balancing algorithm dynamically adjusts the allocation of rule execution tasks according to the real-time data traffic and rule complexity.

[0057] The step 1 includes the following contents:

[0058] Step 101: Create and store a rule network, such as data processing rules, screening rules, and detection rules. The rule structure includes trigger conditions, execution actions, and priorities. A rule definition language based on a domain-specific language is used, and the dependency and priority between modeling rules are modeled using a graph database to achieve efficient query and execution optimization of complex rule logic.

[0059] When in use, it realizes visualization and efficient query of multi-dimensional business logic in complex data scenarios. By clearly splitting factors such as trigger conditions, execution actions and priorities, it ensures accurate scheduling in sequential or parallel manner when executing rules, thereby enhancing the controllability of the data processing process. In a large-scale distributed environment, it can significantly improve the overall efficiency and reliability of data quality management.

[0060] Step 102: Adopt an event-driven architecture and build a high-throughput, low-latency event stream through Apache Kafka for real-time push of rule updates; generate rule update events when rules change and publish them to Kafka topics, obtain and apply the latest rules in real time after subscribing to relevant topics, and integrate the Raft protocol to ensure the consistency and reliability of rule updates among all execution nodes; publish, subscribe and receive events through Kafka to achieve instant synchronization and application of rules, and ensure the real-time and consistency of data verification; Among them, it should be noted that the key steps of the Raft protocol include nodes electing leaders through voting, leaders receiving rule update requests and copying them as log entries to all followers, after the majority of nodes confirm the log entries, the leader submits and notifies followers to apply the updates, and all nodes apply the log entries to the state machine in the order of submission to ensure rule consistency;

[0061] Kafka topic is a core concept in Apache Kafka, which is used to organize and classify message streams. A topic can be regarded as a logical message category. Producers publish messages to designated topics, and consumers subscribe to and read messages from these topics. In order to achieve high throughput and parallel processing, topics are usually divided into multiple partitions. Each partition can be processed independently on different servers, thereby improving the scalability and fault tolerance of the system. The design of topics enables Kafka to efficiently process large-scale real-time data streams. It is widely used in scenarios such as log collection, real-time analysis, and event-driven architecture to ensure the reliability and low latency of data transmission.

[0062] Through the event-driven architecture, rule changes can be quickly distributed to each execution node in a large-scale, multi-node distributed environment; by generating update events when the rules change and publishing them to the Kafka topic, subscribing nodes can obtain and apply the latest rules in real time, thereby ensuring the dynamic consistency of data verification and processing logic. This can ensure that data processing accuracy and real-time performance can be maintained during peak periods or when rules change frequently, greatly reducing data anomalies caused by rule asynchrony.

[0063] Step 103: After receiving the new rule, parse the domain-specific language RDL into an abstract syntax tree AST; convert the AST into an intermediate representation IR, and use just-in-time compilation to dynamically compile the intermediate representation IR into efficient machine code to improve the rule execution performance and response speed, and optimize the rule execution path through the JIT compiler;

[0064] Integrated dynamic load balancing algorithm to dynamically adjust the allocation of rule execution tasks according to real-time data traffic and rule complexity to optimize resource utilization and performance;

[0065] When using, combine the contents in steps 101 to 103:

[0066] The JIT compiler can dynamically adjust the compilation strategy according to the real-time operating environment and data characteristics, improve execution efficiency and reduce resource waste; in addition, the integrated dynamic load balancing algorithm allocates execution nodes according to rule complexity and traffic volume, which can further improve the adaptability to high concurrency and uneven load scenarios, and can effectively ensure the real-time and scalability of the data quality management process.

[0067] Step 2: Detect the difference between the new and old rules when the rules are updated, generate the difference degree Xot from the difference data, and automatically trigger the rollback mechanism when the difference degree Xot exceeds expectations. Collect the performance indicators in the rule execution process to generate the indicator abnormal value Mav. If the indicator abnormal value Mav exceeds expectations, issue a first-level alarm instruction to the outside.

[0068] The step 2 includes the following contents:

[0069] Step 201: When updating a rule, a detailed change history and metadata are recorded, and a difference comparison algorithm is used to detect the difference between the new and old rules. After continuously acquiring the difference data of several rules, a difference degree Xot is generated, where:

[0070] The difference between the two new and old rules is linearly normalized Xs, and the corresponding data values ​​are mapped to the interval [0,1] as follows:

[0071]

[0072] Where n is the number of usage scenarios, Xs i is the difference ratio between the new and old rules of the i-th group, Xs a is the difference ratio average; weight coefficient: 0≤F1≤1, 0≤F2≤1, and F1+F2=1;

[0073] If the obtained difference Xot exceeds the preset difference threshold, it means that the new rule has a large difference. At this time, the rollback mechanism is automatically triggered to restore to the stable previous version of the rule to ensure the security and traceability of the rule update and maintain the continuity of data verification; the difference comparison algorithm (diff algorithm) is used to detect the difference between the new and old rules. If the obtained difference exceeds the set threshold, the rollback mechanism is automatically triggered to avoid the spread of potential risks caused by major changes;

[0074] This rollback method not only reduces the possibility of manual troubleshooting errors, but also avoids impact on online business, ensuring the stability of continuous operation and data verification; by quantitatively recording change points and differences and quickly formulating response measures, it can prevent unintentional major rule modifications from causing serious damage to the data processing process, ultimately protecting the reliability and maintainability of the system under changing business requirements.

[0075] Step 202: Real-time monitoring and log recording, collect performance indicators during the rule execution process, such as processing delay, throughput, and rule hit rate, and take performance indicators that exceed the preset range as abnormal indicators; record the proportion and weight of the abnormal indicator, and the abnormal time node, and generate the indicator abnormal value Mav after combining them under dimensionless conditions, as follows:

[0076] Mav=∫∫∫ D (w L ·P L (t,x,y)+w T ·P T (t,x,y)+w R ·P R (t,x,y))dy dx dt

[0077] Where: D is the integration area, which is defined as a specific time range [t0, t1], spatial range [x0, x1] and other related dimensions [y0, y1]; L(t, x, y) is the processing delay at time t and spatial location (x, y); w L is the weight coefficient of processing delay, P L (t,x,y) is the abnormal proportion of processing delay; T(t,x,y) is the throughput at time t and spatial location (x,y), w T is the throughput weight coefficient P T (t,x,y),P T (t,x,y) is the abnormal proportion of throughput; R(t,x,y) is the rule hit rate at time t and spatial position (x,y), P R (t,x,y) is the abnormal proportion of the rule hit rate, w R is the weight coefficient of the rule hit rate; the weight coefficient is between 0 and 1;

[0078] Based on historical data and management expectations for performance indicators, anomaly thresholds are pre-set; if the acquired indicator anomaly value Mav exceeds the preset anomaly threshold, it indicates that anomalies exist in the current rule execution process and require timely response and processing, and a first-level alarm instruction is issued to the outside world;

[0079] When using, combine the contents in steps 201 and 202:

[0080] By collecting key performance indicators during the rule execution process (such as processing delay, throughput, rule hit rate, etc.), and combining the abnormal value judgment and alarm strategy of these indicators, it provides accurate health monitoring and risk warning means, and reminds relevant teams to investigate and deal with them in time; this multi-dimensional, visual performance monitoring not only prevents system bottlenecks caused by reduced rule execution efficiency or improper resource use, but also provides specific and quantitative decision-making basis for the continuous optimization of data processing processes; in the face of complex and changeable e-commerce transaction environments, it can reliably identify potential risks and respond quickly to achieve reliable protection of business continuity.

[0081] Step 3: Use preset rules to verify the preprocessed data, classify the abnormal data and obtain several abnormal data classes, construct the data anomaly degree DAD(t) from the abnormal state data of each abnormal type, and automatically correct the classified abnormal data when the data anomaly degree DAD(t) exceeds expectations;

[0082] The step three includes the following contents:

[0083] Step 301: After receiving the alarm instruction, pre-process the received data, check the pre-processed data one by one using the preset verification rules, and mark the data that violates the rules as abnormal;

[0084] The detected abnormal data is classified according to the predefined classification standards, and the classification standards are formulated according to the abnormality type (such as numerical abnormality, format abnormality, logical abnormality, etc.); the abnormal data is classified in multiple dimensions according to different dimensions of the data (such as order amount, inventory quantity, date format, etc.) to obtain several abnormal data classes;

[0085] By preprocessing the data as necessary and applying preset verification rules one by one, data violating the rules can be quickly screened out in a high-concurrency environment to form an accurate set of abnormal data; by segmenting abnormal data in multiple dimensions according to classification standards such as numerical anomalies, format anomalies, and logical anomalies, the pertinence of abnormal processing can be improved; cross-checking information on key dimensions such as order amount, inventory quantity, and date format can more effectively discover deep-level or hidden anomalies, providing protection for e-commerce operations to prevent risk control loopholes;

[0086] Step 302: Establish a multi-dimensional index for the classified abnormal data and monitor the dynamic changes of the abnormal data. After setting corresponding thresholds and rules for each type of abnormal data, obtain the amount of abnormal data of each abnormal type, the time node when the abnormal data is received, and the similarity between different types of abnormal data, and summarize and generate abnormal state data; generate data abnormality degree DAD(t) from the abnormal state data in the following manner:

[0087]

[0088] Where: A is the weight vector, defined as where w i Represents the weight coefficient of the i-th type of anomaly, satisfying S is the similarity matrix, an N×N symmetric matrix, with element S ij Represents the similarity between the abnormal data of the i-th category and the j-th category, and its value range is [0.1], where S ii =1; V(t′) is the abnormal data volume vector, and the amount of data of each type of abnormality at time t′ is defined as Where V i (t′) represents the amount of abnormal data received by the i-th type of anomaly at time t′; W(t,t′) is the time weight function matrix, an N×N matrix, defined as W ij (t,t′)=e -λ(t-t′) δ ij , where λ is the time decay factor, which is greater than 1 and controls the rate at which the influence of abnormal data decreases over time, δ ijis the Kronecker delta function, when i = j, delta ij =1, otherwise δij=0; o is the Hadamard product, which means the element-by-element multiplication of the corresponding elements. is the tensor product, which represents the outer product of two vectors or matrices, generating a high-dimensional tensor. is the convolution integral, which represents the integral of time t′ from t0 to the current time t, N: the total number of abnormal types;

[0089] When the data anomaly degree DAD(t) exceeds the preset threshold, it means that the data anomaly degree of the current abnormal data class is high and needs to be processed in a targeted manner. At this time, a secondary alarm instruction is issued to the outside;

[0090] By establishing multi-dimensional indexes for classified abnormal data and monitoring the dynamic changes of abnormal data, different types of abnormal data can still be quickly retrieved and processed in large-scale, high-concurrency scenarios; by generating data anomaly degree DAD(t), it helps to grasp the evolution and trend of abnormal behavior during real-time operation and reduce the loss caused by information lag;

[0091] Especially during peak e-commerce transaction periods, it can help identify and prioritize high-risk abnormal data types to prevent serious errors from fermenting in downstream links or affecting other business processes.

[0092] Step 303: after receiving the secondary alarm instruction, a specific error correction rule is corresponding to each type of anomaly, and a data error correction rule set is generated after being summarized. The preset regularized error correction logic is applied to the abnormal data in the abnormal data class, and the classified abnormal data is automatically corrected to generate corrected data;

[0093] It should be noted that these error correction rules can be defined by business experts based on historical data and business needs, covering various common exception types, such as negative order amounts, abnormal inventory quantities, or incorrect date formats.

[0094] When using, combine the contents in steps 301 to 303:

[0095] Apply preset error correction rule logic to abnormal data to achieve automatic correction of most common abnormal data; by combining multi-dimensional classification results with automatic correction actions, it significantly reduces manual intervention and prevents errors from spreading to key business links. The corrected data obtained after applying the error correction rules can be quickly sent back to the normal data flow or undergo secondary testing, further ensuring a complete closed loop of data quality. When the error correction rules cannot be applied or the automatic correction fails, the system can also intelligently trigger the isolation process or manual review channel to improve efficiency while ensuring security and stability.

[0096] Step 4: Identify and classify the abnormal data that still cannot be corrected, migrate the identified isolated data class from the main data stream to the isolation queue, retain the abnormal data that cannot be processed immediately in the isolation queue, regularly remind the data receiving end to process it and restrict the reminder frequency;

[0097] The step 4 includes the following contents:

[0098] Step 401: train a machine learning algorithm with the labeled sample data to obtain a trained auxiliary correction model; use the abnormal data that cannot be automatically corrected by the preset rules as input, use the trained auxiliary correction model to predict and correct complex abnormalities, and output a predicted reasonable value to replace or correct the original abnormal data;

[0099] Identify and classify abnormal data that still cannot be corrected, and identify and obtain isolated data categories, including:

[0100] Complex anomalies: anomalies involving multi-field dependencies or complex business logic, such as inconsistent data across tables; Uncommon anomalies: abnormal patterns that are rare or unseen in historical data; High-risk anomalies: abnormal data that may have a significant impact on business processes or customer experience.

[0101] It conducts deeper prediction and correction of abnormal data that cannot be automatically corrected by preset rules. By analyzing past historical data and complex business logic, it can give more accurate correction suggestions or values ​​for anomalies that deviate from the normal range but do not conform to simple error correction rules, thereby significantly reducing reliance on manual review and effectively making up for the possible insufficient coverage of regularized error correction strategies. After identifying complex anomalies, uncommon anomalies, and high-risk anomalies that are difficult to correct, they are further isolated or high-level alarmed according to their potential hazards or dependencies. In a highly dynamic e-commerce environment, it achieves a full integration of automation and intelligent processing, allowing the system to maintain a high detection rate and low false alarm rate in the face of large and changeable abnormal scenarios, and continuously improve data quality and business reliability.

[0102] Step 402: Establish an isolation queue or database table for storing abnormal data that cannot be automatically corrected, remove the isolated data class that cannot be automatically corrected from the main data stream, and migrate it to the isolation queue;

[0103] When transferring abnormal data to the isolation queue, detailed tags and metadata are automatically added to each piece of data, including but not limited to: abnormal type: such as numerical abnormality, format abnormality and logical abnormality; rule ID: the specific rule identifier that triggers the abnormality; data source: the source system or module of the abnormal data; occurrence time: the specific timestamp when the abnormal data is generated; processing status: the current processing stage, such as pending review, processing, processed, etc.

[0104] During the migration process, the tags and metadata of abnormal data are updated, and the processing history of all isolated data is recorded, including data migration time, processing personnel, and processing results.

[0105] When in use, the tag information includes exception type, rule ID, data source, occurrence time and processing status, which facilitates quick location and allocation of processing responsibilities, reducing the cost of repeated investigations; the isolated data is also managed persistently, and the traceability of the entire processing process is ensured by recording the migration time, processing personnel and processing results;

[0106] It effectively prevents difficult-to-correct abnormal data from entering the core business processing flow, avoiding chain errors or performance burdens in subsequent business logic; when downstream processing or manual review is required, the system can immediately retrieve isolated data for in-depth investigation or manual error correction to achieve a balance between security and efficiency.

[0107] Step 403: The isolated data enters a pending review state, waiting for processing by the data review end. After the data review end corrects the abnormal data, it re-injects the corrected data into the main data stream, or marks it as processed according to the processing result;

[0108] For abnormal data that cannot be processed immediately, it is retained in the isolation queue, and the data receiving end is regularly reminded to process it, and the reminder frequency Rop is constrained. After the reminder interval that meets the constraint conditions, a reminder instruction is issued to the data receiving end; the constraint method is as follows:

[0109]

[0110] Where: N is the total number of anomaly types, i is the index of the anomaly type, i = 1, 2, ..., N, Δ i is the weight coefficient of anomaly type i, σ i (t) is the priority function of exception type i, based on the current system load U(t), the emergency degree of the exception E i (t) and historical processing efficiency H i (t) calculated;

[0111]

[0112] Among them, E i (t) is the urgency score of anomaly type i at time t; U(t) is the load index at time t, ranging from [0, 1], H i (t) is the historical processing efficiency of anomaly type i at time t, ∈ is a small constant to prevent the denominator from being zero;

[0113] H(x) is the Heaviside step function, defined as: tlast,i The time point when the reminder instruction was last sent for exception type i;

[0114] Δt i (t) is the dynamic minimum reminder interval of abnormal type i, Δt 0,i is the basic minimum reminder interval of exception type i, is the load regulation parameter, with a value between 0 and 1, γ i It is the reminder frequency adjustment parameter, which controls the influence of the current reminder frequency on the reminder interval, and its value is between 0 and 1;

[0115] P i (t) is the current reminder frequency of anomaly type i, φ i (N i (t)): The abnormal data volume function of abnormal type i, defined as: N i (t) is the amount of abnormal data in the isolation queue of abnormal type i at time t; N i (t) is the abnormal data volume threshold of abnormal type i.

[0116] When using, combine the contents in steps 401 and 403:

[0117] After the isolated data enters the pending review state, manual error correction or marking is performed through the data review end, so that complex or uncorrectable anomalies can obtain targeted solutions, thereby achieving the ultimate guarantee of data quality management. Exceptions that cannot be handled immediately will continue to remain in the isolation queue, and the data receiving end will be notified with a constrained reminder frequency to ensure that potential high-risk anomalies will not accumulate or be ignored for a long time. In highly dynamic e-commerce transaction scenarios, flexibility and security are guaranteed. It can not only rely on automated logic to handle a large number of common anomalies, but also carry out orderly manual intervention on difficult or high-risk anomalies, forming a closed-loop control mechanism with multi-layer protection.

[0118] Step 5: When the isolated abnormal data meets the alarm rules, an abnormal data report is sent to the data receiving end, and the load coefficient Λρ(t) is generated after collecting the performance indicators of the data processing equipment. The corresponding response strategy is adopted according to the relationship between the load coefficient Λρ(t) and the load threshold;

[0119] The step five includes the following contents:

[0120] Step 501: setting alarm rules based on abnormal types and quantities, and when the isolated abnormal data meets the alarm rules, sending an abnormal data report containing detailed information of the abnormal data and processing suggestions to the data receiving end;

[0121] Record the processing results of abnormal data and generate processing logs, including processing time, processing methods, and processing results. Feedback the processing logs to the rule engine and machine learning model to optimize the abnormal data processing strategy.

[0122] When in use, it outputs alarm instructions to the data receiving end and attaches detailed abnormal data reports and processing suggestions, which greatly improves the response speed and handling efficiency of high-risk anomalies; this automated intervention based on alarm rules effectively reduces the burden of manual screening, allowing operation and maintenance and business teams to more efficiently use limited manpower to deal with truly urgent and high-risk issues, and ensure the overall operation quality of the system and business continuity.

[0123] Step 502: collect key performance indicators of each component during data processing, including CPU usage, memory usage, network bandwidth, and disk I / O rate, etc. Under dimensionless conditions, generate a load factor Λρ(t) from the performance indicators in the following manner;

[0124]

[0125] P(t): performance indicator vector, defined as P C (t) is the CPU usage, P M (t) is the memory usage, P N (t) is the network bandwidth usage, P D (t) is the disk I / O rate, W is the weighting matrix, which is a K×K symmetric matrix, w kk is the deadweight of performance index k, w km is the interaction weight between performance indicators k and m, λ is the time dynamic coefficient, which is greater than 0 and controls the sensitivity of the load factor to the load change rate; is the rate of change of the load factor at time (t-1), δ k is the nonlinear interaction coefficient, the value is greater than 1, K = 4, is the total number of performance indicators; e can be 2.718;

[0126] Define resource expansion and contraction strategies, including upper and lower limits of resource allocation, adjustment thresholds and step sizes, etc.; expand computing resources when the load factor Λρ(t) exceeds the preset load threshold, such as adding more servers; otherwise, contract computing resources, such as reducing the existing computer resource configuration;

[0127] When using, combine the contents in steps 501 and 502:

[0128] By collecting the key performance indicators of each component and generating the load coefficient Λρ(t), elastic expansion and recovery strategies are adopted according to the load coefficient Λρ(t). This can solve the problem of resource waste or performance deficiency in the static configuration mode. It can continuously adapt to market fluctuations in the ever-changing e-commerce environment, improve processing capabilities while maintaining cost efficiency. When high-load anomalies occur frequently, resources can be allocated to key processes and high-risk tasks in priority, achieving dual guarantees of performance and quality.

[0129] See also Figure 2 and 3 The present invention provides an intelligent data precision quality management system, including:

[0130] The rule building unit, after creating the rule network, generates rule update events when the rules are changed and publishes them through Kafka. After receiving the new rules, the dynamic load balancing algorithm dynamically adjusts the allocation of rule execution tasks according to the real-time data flow and rule complexity;

[0131] The data difference analysis unit detects the difference between the new and old rules when the rules are updated, generates the difference degree Xot from the difference data, and automatically triggers the rollback mechanism when the difference degree Xot exceeds expectations. It collects the performance indicators in the rule execution process to generate the indicator abnormal value Mav. If the indicator abnormal value Mav exceeds expectations, a first-level alarm instruction is issued to the outside.

[0132] The data correction unit verifies the preprocessed data using preset rules, obtains several abnormal data classes after classifying the abnormal data, constructs the data abnormality degree DAD(t) from the abnormal state data of each abnormal type, and automatically corrects the classified abnormal data when the data abnormality degree DAD(t) exceeds expectations;

[0133] The abnormal data isolation unit identifies and classifies abnormal data that cannot be corrected, migrates the identified isolated data class from the main data stream to the isolation queue, retains the abnormal data that cannot be processed immediately in the isolation queue, regularly reminds the data receiving end to process it and restricts the reminder frequency;

[0134] The response processing unit sends an abnormal data report to the data receiving end when the isolated abnormal data meets the alarm rules, generates a load coefficient Λρ(t) after collecting the performance indicators of the data processing equipment, and expands the computing resources when the load coefficient Λρ(t) exceeds the preset load threshold, otherwise it shrinks the computing resources.

[0135] The present invention provides an intelligent data precision quality management device, comprising at least one processor;

[0136] The memory is used to store executable instructions, and the instructions are executed by at least one of the processors to enable at least one of the processors to perform the steps of the intelligent data precision quality management method.

[0137] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0138] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0139] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only some logical function divisions. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0140] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0141] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. Intelligent data precision quality management method, characterized by: include, After the rule network is created, rule update events are generated and published through Kafka when the rules are changed. After receiving the new rules, the dynamic load balancing algorithm dynamically adjusts the distribution of rule execution tasks based on real-time data traffic and rule complexity; When updating rules, the difference between the new and old rules is detected. When the difference Xot generated by the difference data exceeds expectations, the rollback mechanism is automatically triggered. The performance indicators during the rule execution process are collected to generate the indicator abnormal value Mav. If the indicator abnormal value Mav exceeds expectations, a first-level alarm instruction is issued to the outside. Use preset rules to verify the preprocessed data, classify the abnormal data and obtain several abnormal data classes, construct the data abnormality degree DAD(t) from the abnormal state data of each abnormal type, and automatically correct the classified abnormal data when the data abnormality degree DAD(t) exceeds expectations; Identify and classify abnormal data that still cannot be corrected, migrate the identified isolated data class from the main data stream to the isolation queue, retain the abnormal data that cannot be processed immediately in the isolation queue, regularly remind the data receiving end to process it and restrict the reminder frequency; When the isolated abnormal data meets the alarm rules, an abnormal data report is sent to the data receiving end. After collecting the performance indicators of the data processing equipment, the load coefficient Λρ(t) is generated. When the load coefficient Λρ(t) exceeds the preset load threshold, the computing resources are expanded, otherwise the computing resources are contracted.

2. The intelligent data precision quality management method according to claim 1, characterized in that: Create and store rule networks. The rule structure includes trigger conditions, execution actions, and priorities. A rule definition language based on a domain-specific language is used, and the dependency and priority between rules are modeled using a graph database. Adopting an event-driven architecture, we build event streams through Apache Kafka for real-time push of rule updates. When rules are changed, we generate rule update events and publish them to Kafka topics. After subscribing to relevant topics, we obtain and apply the latest rules in real time and integrate the Raft protocol. After receiving the new rules, the domain-specific language RDL is parsed into an abstract syntax tree AST; the AST is converted into an intermediate representation IR, and the intermediate representation IR is dynamically compiled into efficient machine code using just-in-time compilation, and the rule execution path is optimized through the JIT compiler.

3. The intelligent data precision quality management method according to claim 2, characterized in that: The difference comparison algorithm is used to detect the difference between the new and old rules. After continuously obtaining the difference data of several rules, the difference degree Xot is generated. Among them, the difference ratio Xs between the two new and old rules is linearly normalized, and the corresponding data value is mapped to the interval [0, 1] in the following way: Where n is the number of usage scenarios, Xs i is the difference ratio between the new and old rules of the i-th group, Xs a It is the difference ratio average value; weight coefficient: 0≤F1≤1, 0≤F2≤1, and F1+F2=1.

4. The intelligent data precision quality management method according to claim 3, characterized in that: Real-time monitoring and log recording collect performance indicators during rule execution, and treat performance indicators that exceed the preset range as abnormal indicators; record the proportion and weight of abnormal indicators exceeded, and the abnormal time node, and combine them to generate the indicator abnormal value Mav, as follows: Mav=∫∫∫ D (w L ·P L (t,x,y)+w T ·P T (t,x,y)+w R ·P R (t,x,y))dy dx dt Where: D is the integration area, which is defined as a specific time range [t0, t1], spatial range [x0, x1] and other related dimensions [y0, y1]; w L is the weight coefficient of processing delay, P L (t, x, y) is the abnormal proportion of processing delay; w T is the throughput weight coefficient P T (t, x, y), P T (t, x, y) is the abnormal ratio of throughput; P R (t, x, y) is the abnormal proportion of the rule hit rate, w R is the weight coefficient of the rule hit rate.

5. The intelligent data precision quality management method according to claim 4, characterized in that: After receiving the alarm instruction, the received data is preprocessed, and the preprocessed data is checked one by one using the preset verification rules, and the data that violates the rules is marked as abnormal; Classify the detected abnormal data according to the predefined classification standards, and formulate classification standards according to the abnormal type; classify the abnormal data in multiple dimensions to obtain several abnormal data classes; Establish multi-dimensional indexes for the classified abnormal data and monitor the dynamic changes of abnormal data to obtain abnormal status data of each abnormal type; generate data abnormality degree DAD(t) from the abnormal status data; When the data anomaly degree DAD(t) exceeds the preset threshold, each type of anomaly corresponds to a specific error correction rule, which is summarized to generate a data error correction rule set. The preset regularized error correction logic is applied to the abnormal data in the abnormal data class, and the classified abnormal data is automatically corrected to generate corrected data.

6. The intelligent data precision quality management method according to claim 5, characterized in that: Taking abnormal data that cannot be automatically corrected by preset rules as input, the trained auxiliary correction model is used to predict and correct complex anomalies, and the predicted reasonable value is output to replace or correct the original abnormal data; Establish an isolation queue or database table for storing abnormal data that cannot be automatically corrected, remove the isolated data class that cannot be automatically corrected from the main data flow, and migrate it to the isolation queue; When transferring abnormal data to the isolation queue, detailed tags and metadata are automatically added to each piece of data, including but not limited to: abnormality type, rule ID, data source, occurrence time and processing status. The tags and metadata of abnormal data are updated during the migration process.

7. The intelligent data precision quality management method according to claim 6, characterized in that: The isolated data enters the pending review state, waiting for processing by the data review end. After the data review end corrects the abnormal data, it will re-inject the corrected data into the main data stream, or mark it as processed according to the processing results; For abnormal data that cannot be processed immediately, it is retained in the isolation queue, and the data receiving end is regularly reminded to process it, and the reminder frequency Rop is constrained. After the reminder interval that meets the constraint conditions, a reminder instruction is issued to the data receiving end.

8. The intelligent data precision quality management method according to claim 7, characterized in that: Set alarm rules based on abnormal types and quantities. When isolated abnormal data meets the alarm rules, send an abnormal data report containing detailed information of the abnormal data and processing suggestions to the data receiving end; Record the processing results of abnormal data and generate a processing log, including the processing time, processing method and processing results. Feed the processing log back to the rule engine and machine learning model to optimize the abnormal data processing strategy.

9. Intelligent data precision quality management system, characterized by: include, The rule building unit, after creating the rule network, generates rule update events when the rules are changed and publishes them through Kafka. After receiving the new rules, the dynamic load balancing algorithm dynamically adjusts the allocation of rule execution tasks according to the real-time data flow and rule complexity; The data difference analysis unit detects the difference between the new and old rules when the rules are updated, generates the difference degree Xot from the difference data, and automatically triggers the rollback mechanism when the difference degree Xot exceeds expectations. It collects the performance indicators in the rule execution process to generate the indicator abnormal value Mav. If the indicator abnormal value Mav exceeds expectations, a first-level alarm instruction is issued to the outside. The data correction unit verifies the preprocessed data using preset rules, obtains several abnormal data classes after classifying the abnormal data, constructs the data abnormality degree DAD(t) from the abnormal state data of each abnormal type, and automatically corrects the classified abnormal data when the data abnormality degree DAD(t) exceeds expectations; The abnormal data isolation unit identifies and classifies abnormal data that cannot be corrected, migrates the identified isolated data class from the main data stream to the isolation queue, retains the abnormal data that cannot be processed immediately in the isolation queue, regularly reminds the data receiving end to process it and restricts the reminder frequency; The response processing unit sends an abnormal data report to the data receiving end when the isolated abnormal data meets the alarm rules, generates a load coefficient Λρ(t) after collecting the performance indicators of the data processing equipment, and expands the computing resources when the load coefficient Λρ(t) exceeds the preset load threshold, otherwise it shrinks the computing resources.

10. Intelligent data precision quality management device, characterized by: include, at least one processor; A memory for storing executable instructions, wherein the instructions are executed by at least one of the processors to enable at least one of the processors to perform the steps of the method according to any one of claims 1 to 8.

Images