A rule-based data-driven data cleaning method and data cleaning engine

Abstract

The present application relates to a data cleaning method and a data cleaning engine based on rule data driving, belonging to the technical field of data processing. The data cleaning method includes: receiving the original data streams of multiple heterogeneous business systems; extracting multi-source data features and cross-system field association features to construct a data feature fingerprint; performing similarity matching on the data feature fingerprint, screening rules with a matching degree higher than a preset threshold to obtain a candidate rule set; converting the candidate rule set into a directed acyclic graph; marking conflict rule pairs to obtain a conflict warning list; inserting a virtual isolation layer in the directed acyclic graph to generate a path identification table; performing topological sorting according to the directed acyclic graph to generate a rule execution sequence and performing cleaning processing on the original data streams; allocating data execution channels according to the path identification table; and outputting the cleaned clean data stream and a cleaning log including the rule execution path. The present application can efficiently and accurately process data from different business systems.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data cleaning method and a data cleaning engine based on rule-based data driving. Background Art

[0002] With the rapid development of information technology, the amount of data has increased exponentially, especially in various enterprises. Various heterogeneous business systems have generated a large amount of structured, semi-structured and unstructured data. These data come from various sources, including customer transaction records, product information, user behavior data, sensor data, social media texts, etc. At the same time, more and more enterprises are gradually adopting emerging technologies such as big data technology, cloud computing technology and the Internet of Things (IoT), which further promote the generation, transmission and storage of enterprise data. Enterprise business systems are also becoming increasingly complex, and often use different technology stacks (such as relational databases, NoSQL databases, file systems, log systems, etc.) to store and manage different types of data.

[0003] However, the heterogeneity and dispersion of data and the ever-expanding diversified needs make data management and cleaning extremely complex. For various business systems, how to accurately extract useful information from multiple different data sources, integrate them reasonably, and ensure the accuracy, consistency, integrity and efficiency of data is an important challenge in the current enterprise informatization process.

[0004] At present, common data cleaning methods often focus on the cleaning and processing of a single data source, and the processing capabilities and intelligence level of multi-source data are still insufficient. Especially when processing data from different business systems, there are often large differences in data formats, transmission protocols, and storage structures between these systems, resulting in high complexity, high costs, and high risks in the data processing process.

[0005] Therefore, faced with complex data sources, diverse business needs and large-scale distributed data processing, how to efficiently and accurately process data from different business systems has become a key issue that needs to be solved urgently. Summary of the invention

[0006] In order to efficiently and accurately process data from different business systems, the present application provides a data cleaning method and a data cleaning engine based on rule-based data drive.

[0007] In the first aspect, the present application provides a data cleaning method based on rule-based data drive, which adopts the following technical solution:

[0008] A data cleaning method based on rule data drive, the data cleaning method comprising:

[0009] Receive the original data streams of multiple heterogeneous business systems;

[0010] Extract the multi-source data features and cross-system field association features from the original data streams, and construct a data feature fingerprint;

[0011] Based on the conditional features in the preset rule library, perform similarity matching on the data feature fingerprint, and filter out the rules with a matching degree higher than the preset threshold to obtain a candidate rule set;

[0012] Convert the candidate rule set into a directed acyclic graph; wherein, the nodes of the directed acyclic graph represent rules, and the edges represent data dependency relationships;

[0013] Analyze the input-output dependency relationships of each candidate rule in the candidate rule set, mark the conflicting rule pairs, and obtain a conflict warning list;

[0014] Insert a virtual isolation layer into the directed acyclic graph according to the conflict warning list, bind the isolation layer identifier to the conflicting rule pairs, and generate a path identifier table;

[0015] Perform topological sorting on the directed acyclic graph after inserting the virtual isolation layer to generate a rule execution sequence;

[0016] Clean the original data streams according to the rule execution sequence, and attach a trace label to each piece of data; the trace label includes a rule path identifier, a field modification history, and an isolation layer identifier;

[0017] Allocate data execution channels according to the path identifier table; wherein, when it is detected that the isolation layer identifier in the trace label matches the conflict warning list, data rerouting is triggered;

[0018] Output the cleaned clean data stream and a cleaning log including the rule execution path.

[0019] By adopting the above technical solution, through multi-source data fusion, rule dependency modeling, conflict isolation, and topological execution, an efficient and reliable data cleaning system is formed. When facing multi-source data, this technical solution can flexibly and accurately handle the relationships between different systems, and can effectively detect and handle rule conflicts, reduce the interference of conflicting rule pairs, ensure data quality, improve data cleaning throughput and efficiency, and thus provide accurate and stable data support for various business systems.

[0020] Optionally, the original data streams include structured numerical data, text data, or binary files; the steps of extracting the multi-source data features from the original data streams include:

[0021] Perform dynamic distribution statistics on the structured numerical data to generate data distribution features;

[0022] Extract semantic vector features from the text data;

[0023] Parse the binary file and extract metadata features;

[0024] Obtain multi-source data features according to the data distribution features, semantic vector features and metadata features.

[0025] By adopting the above technical solution, a multi-source data feature set is formed, which provides rich information for subsequent data cleaning, anomaly detection and pattern recognition. It not only improves the accuracy and efficiency of data processing, but also can intelligently process diverse heterogeneous data to ensure the comprehensiveness and high quality of data cleaning.

[0026] Optionally, the steps of analyzing the input-output dependency relationships of each candidate rule in the candidate rule set, marking the conflicting rule pairs, and obtaining the conflict warning list include:

[0027] Receive the candidate rule set, parse the input field set and output field set of each candidate rule, and generate a rule input-output field table;

[0028] Based on the input-output field table, construct a rule dependency graph;

[0029] Traverse the rule dependency graph, detect the rule pairs whose output fields have intersections and have no dependency paths, and mark them as conflicting rule pairs;

[0030] Determine the confidence level of the conflicting rule pairs according to the cross-system field association features in the data feature fingerprint;

[0031] Match the corresponding preset routing strategy according to the confidence level of each conflicting rule pair, and generate a conflict warning list.

[0032] By adopting the above technical solution, comprehensively using the rule dependency graph, cross-system field association features and confidence adjustment mechanism, effectively identify and handle the rule conflict problems in the data cleaning process. By clarifying the dependency relationships between rules, timely discovering potential conflicts, adjusting the confidence level of conflicts, and optimizing the processing flow according to the preset routing strategy, finally generate an efficient and intelligent conflict warning list and processing path, thereby improving the accuracy, processing efficiency and flexibility of data cleaning.

[0033] Optionally, the steps of inserting a virtual isolation layer into the directed acyclic graph according to the conflict warning list, binding the conflicting rule pairs with isolation layer identifiers, and generating a path identifier table include:

[0034] According to the directed acyclic graph and the conflict warning list, locate the nearest common ancestor node of the conflicting rule pairs;

[0035] Insert a virtual isolation layer node after the out-edge of the nearest common ancestor node pointing to the conflict rule pair to generate an updated directed acyclic graph;

[0036] Inject an isolation layer identifier, a conflict rule pair, and a bypass copy generation flag into the virtual isolation layer node;

[0037] Generate a path identifier table according to the preset routing policy in the conflict warning list and the cross-system association characteristics of the data feature fingerprint.

[0038] By adopting the above technical solution, based on the LCA positioning, the root cause of the conflict rule is accurately found and a virtual isolation layer is inserted at the appropriate position, successfully isolating the conflict path. Through the analysis of the intelligent routing policy and cross-system characteristics, it is ensured that the data can flow according to the appropriate path, effectively solving the problems such as incomplete isolation of data conflicts and poor cross-system consistency.

[0039] Optionally, the steps of generating a rule execution sequence by performing a topological sort on the directed acyclic graph after inserting the virtual isolation layer include:

[0040] Inject node labels into the directed acyclic graph after inserting the virtual isolation layer according to the path identifier table;

[0041] Initialize a high-priority queue and a low-priority queue according to the node label type, and sort the nodes in the queue based on the service priority policy; the node label types include main chain nodes, branch nodes, and virtual isolation layer nodes;

[0042] Traverse the high-priority queue and the low-priority queue layer by layer to generate a main execution sequence and a branch execution sequence;

[0043] Obtain a rule execution sequence according to the main execution sequence and the branch execution sequence.

[0044] By adopting the above technical solution, based on the injection of node labels, the system can clearly identify main chain nodes and branch nodes, and reasonably schedule resources according to the priority policy; at the same time, the layer-by-layer traversal ensures that the rules are executed according to the priority, making the execution process of complex rules more stable and efficient.

[0045] Optionally, the steps of traversing the high-priority queue and the low-priority queue layer by layer to generate a main execution sequence and a branch execution sequence include:

[0046] Circularly extract the main chain nodes in the high-priority queue and add them to the main execution sequence;

[0047] When the high-priority queue is empty, extract the current branch node in the low-priority queue and add it to the branch execution sequence;

[0048] Traverse all outgoing edges of the current branch node and filter successor nodes whose node label type is a branch;

[0049] Update the in-degree of the successor nodes, set the in-degree to zero, and add successor nodes whose node label type is a branch to the low-priority queue.

[0050] By adopting the above technical solution, the system can efficiently manage the execution order and priority of nodes, ensure the priority execution of main-chain nodes and the timely processing of branch nodes after the main-chain nodes are executed, flexibly schedule tasks according to the dependency relationships and resource status of nodes, optimize resource utilization, avoid unordered execution and conflicts, and at the same time improve the parallelism and efficiency of task execution.

[0051] Optionally, after the step of outputting the cleaned clean data stream and associated cleaning logs, the following steps are further included:

[0052] Based on the rule execution success rate and business feedback data in the cleaning log, eliminate rules with an execution success rate lower than the preset success rate threshold and correct the feature extraction parameters;

[0053] Encode the historical rule execution paths into a rule gene pool that can be genetically optimized, and iteratively update the rule library version.

[0054] By adopting the above technical solution, during the rule execution process, the system can monitor the success rate of rules through the cleaning log, eliminate inefficient rules, and continuously correct the feature extraction parameters, thereby ensuring the efficiency and accuracy of the rule library. At the same time, by converting the historical rule execution paths into a genetically optimized gene pool and using genetic algorithms for iterative updating of rules, the rule library is enabled to have the ability of self-optimization. This technical solution improves the adaptive ability of the system in complex data processing and business execution, and ensures that the rule library always maintains an efficient, accurate and continuously optimizable state in the face of changing business requirements.

[0055] In a second aspect, the present application provides a rule data-driven data cleaning engine, adopting the following technical solution:

[0056] A rule data-driven data cleaning engine, the data cleaning engine includes:

[0057] A receiving module, configured to receive the original data streams of multiple heterogeneous business systems;

[0058] A feature extraction module, configured to extract multi-source data features and cross-system field association features in the original data stream, and construct a data feature fingerprint;

[0059] A rule screening module, configured to perform similarity matching on the data feature fingerprint based on the conditional features in a preset rule library, screen rules with a matching degree higher than a preset threshold, and obtain a candidate rule set;

[0060] A conversion module, configured to convert the candidate rule set into a directed acyclic graph; wherein, nodes of the directed acyclic graph represent rules, and edges represent data dependency relationships;

[0061] A conflict marking module, configured to analyze the input-output dependency relationships of each candidate rule in the candidate rule set, mark conflict rule pairs, and obtain a conflict warning list;

[0062] A path identification table generation module, configured to insert virtual isolation layers in the directed acyclic graph according to the conflict warning list, bind the conflict rule pairs with isolation layer identifiers, and generate a path identification table;

[0063] A topological sorting module, configured to perform topological sorting on the directed acyclic graph after inserting virtual isolation layers, and generate a rule execution sequence;

[0064] A cleaning processing module, configured to perform cleaning processing on the original data stream according to the rule execution sequence, and attach a track label to each piece of data; the track label includes a rule path identifier, a field modification history, and an isolation layer identifier;

[0065] An execution channel allocation module, configured to allocate data execution channels according to the path identification table; wherein, when it is detected that the isolation layer identifier in the track label matches the conflict warning list, data rerouting is triggered;

[0066] An output module, configured to output a clean data stream after cleaning and a cleaning log including a rule execution path.

[0067] In a third aspect, the present application provides a computer device, adopting the following technical solution:

[0068] A computer device includes a memory, a processor, and a computer program stored on the memory, and the processor executes the computer program to implement the steps of the method as described in the first aspect.

[0069] In a fourth aspect, the present application provides a computer-readable storage medium, adopting the following technical solution:

[0070] A computer-readable storage medium stores a computer program that can be loaded and executed by a processor to implement any one of the methods in the first aspect.

[0071] In summary, the present application includes at least one of the following beneficial technical effects: The present application can effectively extract and match data features from multiple heterogeneous business systems, optimize the order of rule execution, and handle cross-system data dependencies. By constructing a directed acyclic graph of rules, marking conflict rule pairs, and introducing a virtual isolation layer, the reliability of rule execution and the accuracy of data are ensured. At the same time, the generation of trace tags and the data rerouting mechanism provide detailed execution records and dynamic control for the data cleaning process, enabling flexible adjustment of the data processing path according to business requirements. This technical solution not only improves the efficiency and accuracy of data cleaning but also provides comprehensive traceability and transparency for data quality control, greatly enhancing the intelligence and automation level of data processing, being able to handle complex heterogeneous data environments, improve data quality, and support more accurate data decision-making and business operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 FIG. is a first process schematic diagram of a rule data-driven data cleaning method according to one embodiment of the present application.

[0073] Figure 2 FIG. is a second process schematic diagram of a rule data-driven data cleaning method according to one embodiment of the present application.

[0074] Figure 3 FIG. is a third process schematic diagram of a rule data-driven data cleaning method according to one embodiment of the present application.

[0075] Figure 4 FIG. is a fourth process schematic diagram of a rule data-driven data cleaning method according to one embodiment of the present application.

[0076] Figure 5 FIG. is a fifth process schematic diagram of a rule data-driven data cleaning method according to one embodiment of the present application.

[0077] Figure 6 FIG. is a sixth process schematic diagram of a rule data-driven data cleaning method according to one embodiment of the present application.

[0078] Figure 7 FIG. is a seventh process schematic diagram of a rule data-driven data cleaning method according to one embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0079] In order to make the objectives, technical solutions, and advantages of the present application clearer, the following further describes the present application in detail with reference to the accompanying Figure 1-7 drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0080] An embodiment of this application discloses a data cleaning method based on rule data driving.

[0081] Refer to Figure 1 , a data cleaning method based on rule data driving, the data cleaning method includes:

[0082] Step S101, receiving the original data streams of multiple heterogeneous business systems;

[0083] Among them, the original data streams include structured numerical data tables, text data, and binary files;

[0084] Specifically, data is uniformly accessed from multiple heterogeneous business systems. Since business systems often use different technology stacks (such as relational databases, NoSQL databases, logging systems, or binary files, etc.), this step needs to use the adapter pattern to encapsulate different data source access interfaces to achieve the unified access of different data formats.

[0085] Exemplarily, the e-commerce system pushes order data (structured data) through the Kafka message queue, the logistics system uploads signed photos (binary files) through SFTP, and the payment system returns transaction flows (Protobuf format) through the gRPC interface. After these data are accessed through the adapter, they are uniformly converted into the Parquet format for storage.

[0086] Step S102, extracting multi-source data features and cross-system field association features in the original data streams, and constructing a data feature fingerprint;

[0087] Among them, the data feature fingerprint (DFP) is a global feature set, including data distribution features, semantic vector features, and cross-system field association features of multiple business systems.

[0088] Specifically, numerical distribution features: for example, calculating statistics such as mean, variance, skewness, and kurtosis for data fields such as transaction amount and user age; semantic vector features: using a pre-trained language model (such as BERT) to convert unstructured text into semantic vectors to capture the deep semantics of the text; cross-system field association features: constructing a field association network between different business systems, for example, recording the association relationship between the order system and the payment system through a graph database.

[0089] It can be understood that through multi-dimensional feature fusion, the DFP can provide rich context information for subsequent rule matching and enhance the accuracy of data cleaning. Numerical features can identify abnormal patterns, text features can accurately match specific content, and cross-system association features are used to ensure the consistency of rules in different business systems.

[0090] Step S103: Based on the conditional features in the preset rule library, perform similarity matching on the data feature fingerprints, and filter out the rules with a matching degree higher than the preset threshold to obtain a candidate rule set;

[0091] Among them, based on the conditional features in the preset rule library, perform similarity matching on the data feature fingerprints (DFP), and filter out the rules with a matching degree higher than the preset threshold. Each rule can consist of a conditional part (such as the amount being greater than 1000) and an action part (such as marking it as a suspicious transaction).

[0092] Specifically, the matching process includes structured condition matching, semantic condition matching, and cross-system field matching. Structured condition matching, such as the numerical condition in the amount field matching rule; semantic condition matching, such as the semantic matching of text fields, calculating the similarity between the semantic vectors of the data feature fingerprints and the rule conditions; cross-system field matching, such as when a rule depends on fields in two systems, checking whether the field association strength in the data feature fingerprints meets the requirements.

[0093] Step S104: Convert the candidate rule set into a directed acyclic graph; among them, the nodes of the directed acyclic graph represent rules, and the edges represent data dependencies;

[0094] Specifically, convert the dependency relationships between the various rules in the candidate rule set into a directed acyclic graph (DAG) to clarify the rule execution order. The nodes represent rules, and the edges represent the dependency relationships between rules. Dependency relationships include explicit dependencies (such as the output of rule R1 being the input of rule R2) and implicit dependencies (such as the business logic requiring rule R1 to be executed first).

[0095] Exemplarily, in the credit risk control scenario, rule R201 (calculate credit score) and rule R202 (approve the credit limit based on the credit score) constitute an explicit dependency, while rule R203 (verify the work certificate) and R204 (verify the income statement) have no data dependency but need to be executed in the order of business logic.

[0096] It can be understood that DAG modeling clearly shows the dependency relationships between rules, ensures that the rules are executed in the correct order, and avoids execution errors. By detecting circular dependencies, the risk of rule design is reduced, and the failure rate in the production environment is decreased.

[0097] Step S105: Analyze the input-output dependency relationships of each candidate rule in the candidate rule set, mark the conflicting rule pairs, and obtain a conflict warning list;

[0098] Among them, conflicting rules refer to multiple rules modifying the same field without a dependency relationship, which may lead to contradictions in the data results. This step traverses the dependency relationships between rules, analyzes the input and output of the rules, identifies all possible conflicting rule pairs, and obtains a conflict warning list.

[0099] Exemplarily, in a logistics system, rule R301 (timeout marked as "delayed") and rule R302 (manually marked as "normal" by a customer service representative) modify the same "logistics status" field and have no dependency relationship, so they are marked as a conflicting rule pair.

[0100] Step S106: Insert a virtual isolation layer in the directed acyclic graph according to the conflict warning list, bind the conflicting rule pair to the isolation layer identifier, and generate a path identifier table.

[0101] Among them, the virtual isolation layer is a logical marker node inserted in the directed acyclic graph, used to split the execution paths of the conflicting rule pairs. The insertion position is after the lowest common ancestor (LCA) node of the conflicting rule pair to ensure that the conflicting rules are executed in different branches. The path identifier table records the isolation layer ID, the conflicting rule pair, and the rerouting target (such as a manual review interface or a cross-system verification service).

[0102] For example, if rule R7 conflicts with rule R8, the LCA node is rule R6. Insert a virtual isolation layer L3 after R6 and generate a path identifier table, indicating that the conflicting rules R7 and R8 need to be processed through manual review.

[0103] It can be understood that the virtual isolation layer ensures that the conflicting rules do not affect the main process and prevents incorrect data from entering the downstream. The insertion management of the path identifier table makes the system more maintainable and reduces the response time of manual intervention in production.

[0104] Step S107: Perform a topological sort on the directed acyclic graph after inserting the virtual isolation layer to generate a rule execution sequence.

[0105] In one embodiment of the present application, the topological sort is implemented based on the Kahn algorithm: Initialize the nodes with an in-degree of 0 (rules without pre-dependencies), remove the nodes one by one and update the in-degree of the successor nodes until a complete sequence is generated. For nodes with multiple in-degrees (such as multiple rules depending on the same pre-rule), they are sorted according to the business priority. The non-conflicting paths (main execution sequence) and the conflicting paths (branch execution sequence) are sorted separately. The main chain is executed according to the dependency order, and the branches are processed asynchronously.

[0106] For example, the final sorting result is R1 → R2 → R4 → R5 (main chain), and the conflicting branch R3 → L3 → R6 is executed asynchronously. The main chain rules are assigned to high-priority computing nodes, and the branch rules are assigned to low-priority queues.

[0107] It can be understood that the topological sort can ensure the logical correctness of the main execution chain, and the asynchronous processing of the conflicting branches avoids blocking the main process.

[0108] Step S108, clean the original data stream according to the rule execution sequence, and attach a trace tag to each piece of data; the trace tag includes a rule path identifier, a field modification history, and an isolation layer identifier;

[0109] Specifically, during the data cleaning process, the trace tag is the key metadata for tracking the data flow. Among them, the rule path identifier is the unique identifier of the rule sequence through which the data flows, generated by the hash algorithm. This path identifier is dynamically generated based on the topologically sorted rule execution sequence to ensure the uniqueness and anti-tampering of the execution chain. The field modification history stores the modification records of the fields in the form of key-value pairs, including the original value, the modified value, the execution rule ID, and the timestamp. These modifications are incrementally updated through the version control mechanism to support data rollback and comparison of historical versions.

[0110] In addition, when the data enters the virtual isolation layer, the isolation layer identifier and the current processing status (such as L3→Pending) are injected, and bound to the conflicting rule pair in the path identifier table to trigger subsequent rerouting decisions.

[0111] Step S109, allocate a data execution channel according to the path identifier table; among them, when it is detected that the isolation layer identifier in the trace tag matches the conflict warning list, data rerouting is triggered;

[0112] Specifically, the path identifier table is used to record the mapping relationship between the virtual isolation layer and the conflicting rule pair. When the matching condition is met, the data rerouting is triggered according to the preset routing strategy in the path identifier table: create a data copy and send it to the specified channel (such as manual review), and the main thread caches the intermediate state and waits for the result.

[0113] Exemplarily, after a certain order data executes R1 (format verification) → R2 (risk control scoring), the trace tag is [R1, R2]; if it enters the isolation layer L3, the copy is sent to manual review, and the tag is updated to [R1, R2, L3→pending], and after manual confirmation, R6 is continued to be executed.

[0114] It can be understood that the trace tag supports data lineage tracking to help locate the root cause of cleaning errors. The rerouting mechanism ensures that conflicting data does not contaminate the clean stream, and the quality of the finally output data is guaranteed.

[0115] Step S110, output the clean data stream after cleaning and the cleaning log including the rule execution path.

[0116] Specifically, for the data stream that does not trigger any conflicting rule pairs during the cleaning process, it is cleaned according to each rule in the topological sequence to ensure that the output data meets the preset quality standards. The cleaning log will record the final rule execution path and the field modification history for downstream system parsing.

[0117] Specifically, the cleaning log includes execution metadata (data ID, start and end times of cleaning, resource consumption), details of the rule execution path (list of successful / failed rules, conflict handling records), and performance metrics (time taken for a single rule, cross-system call latency).

[0118] In the above embodiments, through multi-source data fusion, rule dependency modeling, conflict isolation, and topological execution, an efficient and reliable data cleaning system is formed. When facing multi-source data, this technical solution can flexibly and accurately handle the relationships between different systems, effectively detect and handle rule conflicts, reduce the interference of conflicting rule pairs, ensure data quality, improve data cleaning throughput and efficiency, and thus provide accurate and stable data support for various business systems.

[0119] Referring to Figure 2 , as an embodiment of step S102, the steps of extracting multi-source data features from the original data stream include:

[0120] Step S201, performing dynamic distribution statistics on structured numerical data to generate data distribution features;

[0121] Among them, structured numerical data is usually data in tabular form, containing multiple numerical fields. In this process, the system performs dynamic distribution statistical analysis on each numerical field. This means that the system will calculate the statistical features of these data fields, such as mean, variance, skewness, kurtosis, maximum value, minimum value, etc.

[0122] It can be understood that the purpose of dynamic distribution statistics is to analyze the distribution characteristics of data, understand the central tendency, dispersion degree of data, and whether there are outliers. Through these statistical features, the system can identify potential problems in the data (such as skewed distribution or extreme outliers), providing a basis for subsequent data cleaning and processing.

[0123] Step S202, extracting semantic vector features from text data;

[0124] Among them, text data is usually unstructured, such as log files, comments, articles, etc. By using natural language processing (NLP) techniques and leveraging pre-trained language models (such as BERT, Word2Vec, etc.), the text data can be converted into semantic vectors. Semantic vector features can capture the deep semantic information in the text, rather than just simple features based on word frequency. Through this step, text data can be transformed from abstract and unstructured content into numerical features that can be processed by a computer.

[0125] Step S203, parsing binary files and extracting metadata features;

[0126] Among them, binary files (such as images, audio, video files, compressed files, etc.) contain structured binary data and cannot be directly used for traditional data analysis. Metadata features in these files can be extracted through specialized parsing tools (such as parsing EXIF data of images, parsing metadata of audio, etc.). Metadata usually includes information such as the creation date, size, format, resolution, file author, encoding method, etc. of the file. These metadata help to understand the basic characteristics of the binary data and provide key information for subsequent processing and analysis.

[0127] Step S204: Obtain multi-source data features based on the data distribution features, semantic vector features, and metadata features.

[0128] Among them, by fusing the distribution features of structured data, the semantic features of text data, and the metadata features of binary data, a comprehensive multi-source data feature set is generated. Fusing these multi-source data features can strengthen the connection between data and improve the intelligence and accuracy in the data cleaning process.

[0129] In the above implementation, a multi-source data feature set is formed, providing rich information for subsequent data cleaning, anomaly detection, and pattern recognition. It not only improves the accuracy and efficiency of data processing but also can intelligently process diverse heterogeneous data to ensure the comprehensiveness and high quality of data cleaning.

[0130] Refer to Figure 3 , as an implementation of step S105, the steps of analyzing the input-output dependency relationships of each candidate rule in the candidate rule set and marking conflict rule pairs to obtain a conflict warning list include:

[0131] Step S301: Receive the candidate rule set, parse the input field set and output field set of each candidate rule, and generate a rule input-output field table;

[0132] Among them, each rule usually consists of a condition (input field) and an action (output field). Therefore, these fields need to be extracted and classified. The input field set refers to the condition fields required for the rule to be triggered, while the output field set is the data fields that are changed or generated after the rule is applied. During the parsing process, a hybrid parsing engine is adopted, combining a syntax tree parsing module and a semantic entity recognition module to ensure the structural and semantic understanding of the rule conditions.

[0133] It can be understood that by generating a rule input-output field table, the system can clearly identify the dependent fields and target fields of each rule, laying a foundation for subsequent construction of a rule dependency graph and conflict detection. The technical effect of this step is to provide an accurate field mapping for rule management in the data cleaning process, enabling efficient support for subsequent rule dependency and conflict analysis.

[0134] Step S302: Based on the input-output field table, construct a rule dependency graph;

[0135] Among them, the rule dependency graph is a directed graph, where nodes represent candidate rules and edges represent explicit or implicit dependency relationships between rules. An explicit dependency means that when the output field of rule A is used as the input field of rule B, rule A explicitly depends on rule B; an implicit dependency is based on business logic constraints, and the execution of some rules must be sequential. The establishment of this dependency relationship helps the system understand the relationships between rules and provides support for subsequent conflict detection and priority scheduling.

[0136] Exemplarily, assume that the output field of rule A is "suspicious transaction flag", and the input field of rule B also includes "suspicious transaction flag". If rule A is executed first and rule B depends on the output field of rule A, then in the rule dependency graph, we will add an explicit dependency edge from rule A to rule B.

[0137] Step S303: Traverse the rule dependency graph, detect rule pairs with intersecting output fields and no dependency paths, and mark them as conflicting rule pairs;

[0138] Among them, the system traverses the rule dependency graph, analyzes the dependency relationships between rules, and checks whether there are rule pairs with intersecting output fields. If the output fields of two rules intersect and there is no dependency path between these two rules (i.e., they are independent of each other), it means that these two rules may cause conflicts and need to be marked as conflicting rule pairs. For example, when two rules both try to modify the same field without a dependency relationship in the execution order, it will lead to data inconsistency or uncertainty.

[0139] Step S304: Determine the confidence level of the conflicting rule pairs according to the cross-system field association features in the data feature fingerprint;

[0140] Among them, the system determines the confidence level of the conflicting rule pairs according to the cross-system field association features in the data feature fingerprint (DFP). The data feature fingerprint models the association strength between cross-system fields to evaluate the actual impact between conflicting rule pairs. For example, two rules may modify related fields in different systems. If their cross-system association strength is high, the risk of conflict is greater; on the contrary, if the cross-system association between them is weak, the possibility of conflict is lower.

[0141] For example, assume that rule A and rule B modify fields in two different business systems respectively, but there is a strong cross-system association between these two fields (such as the order status and payment status of the same user). The system will increase the confidence level of the conflicting rule pair according to this association.

[0142] Step S305: Match the corresponding preset routing policies according to the confidence level of each conflict rule pair, and generate a conflict warning list.

[0143] Among them, the conflict warning list lists all conflict rule pairs and their confidence levels. After that, the system can select an appropriate processing method according to the preset routing policy template, and assign the conflict rule pairs to specific adjudication interfaces. The routing policy template determines whether to process the conflict rules through manual review, rerouting, etc. according to the confidence level of the conflict rule pairs and business requirements.

[0144] For example, assume that the confidence levels of conflict rule pairs A and B are relatively high, and the system will route them to the manual review interface, while the rules with relatively low conflict confidence levels will directly enter the automated processing flow.

[0145] In the above implementation, by comprehensively using the rule dependency graph, cross-system field association features, and confidence level adjustment mechanism, the rule conflict problems in the data cleaning process are effectively identified and processed. By clarifying the dependency relationships between rules, potential conflicts are discovered in a timely manner, the confidence levels of conflicts are adjusted, and the processing flow is optimized according to the preset routing policy, finally generating an efficient and intelligent conflict warning list and processing path, thereby improving the accuracy, processing efficiency, and flexibility of data cleaning.

[0146] Referring to Figure 4 , as an implementation of step S106, the steps of inserting a virtual isolation layer in the directed acyclic graph according to the conflict warning list and binding the conflict rule pairs with isolation layer identifiers to generate a path identifier table include:

[0147] Step S401: Locate the nearest common ancestor node of the conflict rule pair according to the directed acyclic graph and the conflict warning list;

[0148] Among them, by analyzing the rule dependency relationships in the directed acyclic graph (DAG), the nearest common ancestor (LCA) node of the conflict rule pair is located. The LCA node is the last common node between two rules in the DAG, which means that these two rules have a common execution path starting from this node.

[0149] In some embodiments, the breadth-first search (BFS) algorithm can be used for traversal to identify the common path between the conflict rule pairs and find their LCA nodes. LCA location is a key step in resolving rule conflicts because all conflict rules depend on the path starting from the LCA node, and the virtual isolation layer will be inserted after the LCA node to ensure that rule conflicts are isolated.

[0150] For example, assume there is a conflict between rule A and rule B, and they share a common node C as the LCA node in the DAG. Then after LCA location, the dependency paths of the conflicting rule pair A and B will be marked as paths starting from node C.

[0151] Step S402, insert a virtual isolation layer node after the out-edge of the nearest common ancestor node pointing to the conflicting rule pair to generate an updated directed acyclic graph;

[0152] Among them, once the LCA node is determined, the system will modify the dependency path between the LCA node and the conflicting rule pair. Specifically, the system will cut off the original edge from the LCA node to the conflicting rule pair and insert a virtual isolation layer node between the LCA node and the conflicting rule pair. The virtual isolation layer is a logical node used to separate conflicting rules to ensure that they do not share the same execution path, thus avoiding conflicts.

[0153] For example, assume the LCA node is C, and the conflicting rules A and B respectively depend on the output paths of node C. In this case, the system cuts off the original dependency edges from C to A and B, and inserts a virtual isolation layer node L between node C and A, B to generate new branch paths C → L → A, C → L → B.

[0154] Step S403, inject an isolation layer identifier, a conflicting rule pair, and a bypass copy generation mark for the virtual isolation layer node;

[0155] Among them, after inserting the virtual isolation layer node, the system needs to inject an isolation layer ID, a conflicting rule pair, and a bypass copy generation mark for this node. These identifiers ensure the uniqueness and traceability of the isolation layer. The isolation layer ID is the unique identifier of this isolation layer, used to trace the execution path of this layer; the conflicting rule pair records the rules isolated in the isolation layer; the bypass copy generation mark indicates whether a copy needs to be generated and processed through the bypass. These marks are crucial for subsequent routing decisions, data processing, and isolation management.

[0156] Exemplarily, assume a virtual isolation layer L is inserted in the DAG, the system assigns an isolation layer ID (such as "L1") to node L, and records the execution situations of conflicting rules A and B in this isolation layer. If node L triggers copy generation, the bypass copy mark will indicate that this data needs to take another path for processing.

[0157] Step S404, generate a path identification table according to the preset routing policy in the conflict warning list and the cross-system association characteristics of the data feature fingerprints.

[0158] Among them, the system generates a path identification table based on the preset routing policy in the conflict warning list and the cross-system association features in the data feature fingerprint (DFP). The path identification table includes the isolation layer ID of the conflict rule pair, the rerouting target, and the cross-system verification rule.

[0159] Specifically, the routing policy determines whether the data needs to be manually reviewed or verified through cross-system services based on the confidence level of the conflict rule. The cross-system association feature matrix provides an assessment of the field association strength between systems. If the strength exceeds the preset threshold, the cross-system verification service is triggered.

[0160] For example, assume that the confidence levels of conflict rules A and B are relatively high, and the cross-system association strength between them is also strong. The system will route them to the manual review interface and mark "manual review" as the target in the path identification table. If the association strength is low, the system may directly pass the data to the automatic processing path.

[0161] In the above implementation, the root cause of the conflict rule is accurately found based on LCA positioning and a virtual isolation layer is inserted at the appropriate position, successfully isolating the conflict path. Through the analysis of intelligent routing policies and cross-system features, it is ensured that the data can flow along the appropriate path, effectively solving problems such as incomplete isolation of data conflicts and poor cross-system consistency.

[0162] Refer to Figure 5 , as an implementation of step S107, the steps of generating a rule execution sequence by performing a topological sort on the directed acyclic graph after inserting the virtual isolation layer include:

[0163] Step S501, injecting node labels into the directed acyclic graph after inserting the virtual isolation layer according to the path identification table; the node label types include main chain nodes, branch nodes, and virtual isolation layer nodes;

[0164] Among them, each node in the directed acyclic graph (DAG) after inserting the virtual isolation layer is injected with a label to distinguish different types of nodes. The labels include main chain nodes, branch nodes, and virtual isolation layer nodes. The main chain node represents a rule node that does not depend on the virtual isolation layer, the branch node represents a rule node branched by the isolation layer, and the virtual isolation layer node itself represents the isolation logic layer. The path identification table helps the system understand the execution status of each node and the conflict path of the rules, thus providing input for the subsequent generation of the rule execution sequence.

[0165] Step S502, initializing a high-priority queue and a low-priority queue according to the node label types, and sorting the nodes in the queue based on the business priority policy;

[0166] Among them, the system initializes two priority queues according to the types of node tags (main chain nodes, branch nodes, virtual isolation layer nodes): a high-priority queue and a low-priority queue. Main chain nodes usually have a higher business priority because they do not depend on other nodes, while branch nodes and virtual isolation layer nodes are given a lower priority. The nodes in the queue are sorted according to the business priority policy (such as time requirements, computational complexity, etc.), so as to ensure that main chain nodes are executed first, and branch nodes are processed subsequently according to business requirements.

[0167] Specifically, the initialization steps include: screening main chain nodes with an in-degree of 0 and adding them to the high-priority queue, and screening branch nodes with an in-degree of 0 and adding them to the low-priority queue.

[0168] Step S503, traverse the high-priority queue and the low-priority queue layer by layer to generate a main execution sequence and a branch execution sequence;

[0169] Among them, the system extracts main chain nodes from the high-priority queue by means of layer-by-layer traversal and adds them to the main execution sequence. When the high-priority queue is empty, the system extracts branch nodes from the low-priority queue and adds them to the branch execution sequence. During the traversal process, the system updates the in-degree of successor nodes and determines whether to add these nodes to the low-priority queue according to the dependency relationship of the nodes (especially for branch nodes, if the nodes they depend on have been executed, their in-degree will be zero, and the system adds them to the low-priority queue waiting for execution).

[0170] Step S504, obtain a regular execution sequence according to the main execution sequence and the branch execution sequence.

[0171] In the above embodiments, based on the injection of node tags, the system can clearly identify main chain nodes and branch nodes and reasonably schedule resources according to the priority policy; at the same time, layer-by-layer traversal ensures that the rules are executed according to the priority, making the execution process of complex rules more stable and efficient.

[0172] Refer to Figure 6 , as an implementation manner of step S503, the steps of traversing the high-priority queue and the low-priority queue layer by layer to generate a main execution sequence and a branch execution sequence include:

[0173] Step S601, repeatedly extract main chain nodes in the high-priority queue and add them to the main execution sequence;

[0174] Among them, the system repeatedly extracts nodes in the high-priority queue and adds them to the main execution sequence. Main chain nodes are executed first because they usually do not depend on the execution of other nodes and represent the backbone of the process. Therefore, the main chain nodes in the high-priority queue should be processed first, which can ensure that the most core part of the business process is completed first.

[0175] Exemplarily, assume that the nodes in the high-priority queue are A, B, and C, and these nodes are all main-chain nodes. The system sequentially extracts these main-chain nodes from the queue and adds them to the main execution sequence in order. For example, rule A is extracted and added to the main execution sequence, and then rules B and C are added in sequence.

[0176] Step S602, when the high-priority queue is empty, extract the current branch node in the low-priority queue and add it to the branch execution sequence;

[0177] Among them, when all the main-chain nodes in the high-priority queue have been executed, the system turns to extract the branch nodes in the low-priority queue and adds them to the branch execution sequence. The low-priority queue usually contains branch nodes with more complex dependencies or less resource requirements, so they are executed after the main-chain nodes are processed.

[0178] Step S603, traverse all the out-edges of the current branch node, and filter out the successor nodes with the node label type of branch;

[0179] Among them, after the current branch node is extracted and added to the branch execution sequence, the system will traverse all the out-edges of this node and filter out the successor nodes. These successor nodes need to be checked for their types, especially to determine whether the node label type is a branch node. If the successor node is of the branch type, then they will become the next branch nodes to be processed and continue to be added to the low-priority queue to wait for execution.

[0180] Step S604, update the in-degree of the successor nodes, set the in-degree to zero and add the successor nodes with the node label type of branch to the low-priority queue.

[0181] Among them, the system will update the in-degree of the filtered successor nodes. Whenever a node is executed, the in-degree of all its successor nodes will be reduced by 1. When the in-degree of a certain successor node becomes zero and the node is a branch node, the system will add this node to the low-priority queue to wait for subsequent execution. The in-degree becoming zero means that all the predecessor nodes of this node have been completed and the node can be executed.

[0182] For example, assume that rule C is executed, and the out-edge of rule C points to rule D and rule E. Assume that the in-degree of rule E is 1, while the in-degree of rule D is 2. When rule C is executed, the in-degrees of rule D and E are each reduced by 1. Assume that the in-degree of rule D is reduced to 0, and rule D is a branch node, then rule D will be added to the low-priority queue to wait for processing.

[0183] It can be understood that through in-degree update, the system can ensure that nodes are executed in an orderly manner according to the dependency relationship. The judgment of in-degree being zero can avoid deadlocks or incorrect execution order, ensure that all dependency relationships are correctly processed, and thus effectively organize and optimize the execution process.

[0184] In the above embodiments, the system can efficiently manage the execution order and priority of nodes, ensure the priority execution of main-chain nodes and the timely processing of branch nodes after the execution of main-chain nodes, flexibly schedule tasks according to the dependency relationship and resource status of nodes, optimize resource utilization, avoid disorderly execution and conflicts, and at the same time improve the parallelism and efficiency of task execution.

[0185] Refer to Figure 7 , as a further embodiment of the data cleaning method, after the step of outputting the cleaned clean data stream and the associated cleaning log, it further includes:

[0186] Step S701, based on the rule execution success rate and business feedback data in the cleaning log, eliminate the rules with an execution success rate lower than the preset success rate threshold and correct the feature extraction parameters;

[0187] Among them, the effectiveness of each rule is evaluated by analyzing the rule execution success rate recorded in the cleaning log and the business feedback data. If the execution success rate of a certain rule is lower than the preset success rate threshold, this rule will be regarded as less efficient and may need to be eliminated or adjusted. Eliminating inefficient rules can avoid resource waste and ensure that the rules in the rule library can effectively promote the business process. At the same time, by combining business feedback data and cleaning logs, the system can also correct the feature extraction parameters, thereby improving the accuracy and efficiency of rule execution.

[0188] Specifically, the execution success rate of a rule is calculated based on the number of successes and failures in the execution log, and the business feedback data usually comes from the results of actual business, reflecting the actual effect after rule execution. If a rule often fails or fails to achieve the expected effect, the system will optimize the rule based on this data. Correcting the feature extraction parameters means adjusting the conditions or algorithms used for data extraction in the rule to improve the performance of the rule.

[0189] Step S702, encode the historical rule execution path into a rule gene pool that can be genetically optimized, and iteratively update the rule library version.

[0190] Among them, the historical rule execution paths are transformed into a rule gene pool that can be genetically optimized, and the rule library version is iteratively updated. Through the analysis of historical rule execution paths, the system can identify the optimal execution paths and rule combinations. These historical paths are encoded in the form of "genes", and this encoding method enables the rule library to have the ability of "genetic optimization", that is, to continuously optimize the rule execution process by simulating natural selection and genetic algorithms.

[0191] Specifically, during the process of encoding the execution paths of rules into genes, the system extracts information such as the key parameters, execution order, and dependencies of each rule execution, and then stores them in the form of gene encoding. These rule genes can evolve continuously through the optimization process of genetic algorithms during subsequent executions to find the best rule combinations and execution orders.

[0192] Exemplarily, assume that the rule execution path is Rule A -> Rule B -> Rule C. In genetic algorithms, each rule and its execution order are encoded as a gene. As the execution path is optimized, the system may find a better path. For example, the order of Rule B and Rule C can be exchanged to improve the overall execution efficiency. Through genetic algorithms, the system gradually eliminates inappropriate rule combinations and retains the optimal path.

[0193] In the above embodiments, during the rule execution process, the system can monitor the success rate of rules by cleaning logs, eliminate inefficient rules, and continuously correct the feature extraction parameters, thereby ensuring the efficiency and accuracy of the rule library. At the same time, by transforming the historical rule execution paths into a genetically optimized gene pool and using genetic algorithms for iterative update of rules, the rule library is enabled to have the ability of self-optimization. This technical solution improves the adaptive ability of the system in complex data processing and business execution, ensuring that the rule library always maintains an efficient, accurate, and continuously optimizable state in the face of changing business requirements.

[0194] The embodiments of this application also disclose a data cleaning engine based on rule data driving.

[0195] A data cleaning engine based on rule data driving, the data cleaning engine includes:

[0196] A receiving module, configured to receive the original data streams of multiple heterogeneous business systems;

[0197] A feature extraction module, configured to extract multi-source data features and cross-system field association features in the original data streams, and construct a data feature fingerprint;

[0198] A rule screening module, configured to perform similarity matching on the data feature fingerprint based on the conditional features in a preset rule library, and screen the rules with a matching degree higher than a preset threshold to obtain a candidate rule set;

[0199] A conversion module for converting a candidate rule set into a directed acyclic graph, where the nodes of the directed acyclic graph represent rules and the edges represent data dependency relationships;

[0200] A conflict marking module for analyzing the input-output dependency relationships of each candidate rule in the candidate rule set, marking conflict rule pairs, and obtaining a conflict warning list;

[0201] A path identification table generation module for inserting virtual isolation layers in the directed acyclic graph according to the conflict warning list, binding the conflict rule pairs to the isolation layer identifiers, and generating a path identification table;

[0202] A topological sorting module for performing topological sorting on the directed acyclic graph after inserting virtual isolation layers to generate a rule execution sequence;

[0203] A cleaning processing module for cleaning the original data stream according to the rule execution sequence and attaching a trace label to each piece of data; the trace label includes a rule path identifier, a field modification history, and an isolation layer identifier;

[0204] An execution channel allocation module for allocating data execution channels according to the path identification table; where when it is detected that the isolation layer identifier in the trace label matches the conflict warning list, data rerouting is triggered;

[0205] An output module for outputting the cleaned clean data stream and a cleaning log including the rule execution path.

[0206] A data cleaning engine based on rule data driving according to an embodiment of the present application can implement any of the above data cleaning methods, and the specific working processes of each module in the data cleaning engine can refer to the corresponding processes in the above method embodiments.

[0207] In several embodiments provided by the present application, it should be understood that the provided methods and systems can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for example, the division of a certain module is only a logical function division, and there may be other division methods in actual implementation. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0208] An embodiment of the present application also discloses a computer device.

[0209] The computer device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements a data cleaning method based on rule data driving as described above.

[0210] An embodiment of the present application also discloses a computer-readable storage medium.

[0211] A computer-readable storage medium stores a computer program that can be loaded and executed by a processor to perform any one of the above-described rule-based data-driven data cleaning methods.

[0212] Among them, the computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in conjunction with an instruction execution system, apparatus, or device; the program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to wireless, wire, optical fiber cable, RF, etc., or any suitable combination of the above.

[0213] It should be noted that in the above embodiments, the descriptions of each embodiment have their own focuses. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0214] The above are all preferred embodiments of the present application. The protection scope of the present application is not limited thereby. Any feature disclosed in this specification (including the abstract and drawings), unless specifically described, can be replaced by other equivalent or similar-purpose alternative features. That is, unless specifically described, each feature is only an example of a series of equivalent or similar features.

Claims

1. A data cleaning method based on rule data driving, characterized in that The data cleaning method includes: Receiving the original data streams of multiple heterogeneous business systems; Extracting multi-source data features and cross-system field association features from the original data streams, and constructing a data feature fingerprint; Performing similarity matching on the data feature fingerprint based on the conditional features in a preset rule library, screening the rules with a matching degree higher than a preset threshold, and obtaining a candidate rule set; Converting the candidate rule set into a directed acyclic graph; wherein, the nodes of the directed acyclic graph represent rules, and the edges represent data dependency relationships; Analyzing the input-output dependency relationships of the candidate rules in the candidate rule set, marking the conflicting rule pairs, and obtaining a conflict warning list; Inserting a virtual isolation layer into the directed acyclic graph according to the conflict warning list, binding the isolation layer identifier to the conflicting rule pairs, and generating a path identifier table; Performing topological sorting on the directed acyclic graph after inserting the virtual isolation layer to generate a rule execution sequence; Performing cleaning processing on the original data streams according to the rule execution sequence, and attaching a track label to each piece of data; the track label includes a rule path identifier, a field modification history, and an isolation layer identifier; Allocating a data execution channel according to the path identifier table; wherein, when it is detected that the isolation layer identifier in the track label matches the conflict warning list, data rerouting is triggered; Outputting the cleaned clean data stream and a cleaning log including the rule execution path; The step of inserting a virtual isolation layer into the directed acyclic graph according to the conflict warning list, binding the isolation layer identifier to the conflicting rule pairs, and generating a path identifier table includes: Locating the nearest common ancestor node of the conflicting rule pairs according to the directed acyclic graph and the conflict warning list; Inserting a virtual isolation layer node after the out-edge of the nearest common ancestor node pointing to the conflicting rule pairs to generate an updated directed acyclic graph; Injecting an isolation layer identifier, a conflicting rule pair, and a bypass copy generation mark into the virtual isolation layer node; Generating a path identifier table according to the preset routing strategy in the conflict warning list and the cross-system association features of the data feature fingerprint; The step of performing topological sorting on the directed acyclic graph after inserting the virtual isolation layer to generate a rule execution sequence includes: Injecting node labels into the directed acyclic graph after inserting the virtual isolation layer according to the path identifier table; Initializing a high-priority queue and a low-priority queue according to the node label type, and sorting the nodes in the queues based on the service priority strategy; the node label type includes main chain nodes, branch nodes, and virtual isolation layer nodes; Traversing the high-priority queue and the low-priority queue layer by layer to generate a main execution sequence and a branch execution sequence; Obtaining a rule execution sequence according to the main execution sequence and the branch execution sequence.

2. The data cleaning method based on rule data driving according to claim 1, wherein: The original data streams include structured numerical data, text data, or binary files; The step of extracting multi-source data features from the original data streams includes: Performing dynamic distribution statistics on the structured numerical data to generate data distribution features; Extracting semantic vector features from the text data; Parsing the binary file and extracting metadata features; Based on the data distribution characteristics, semantic vector characteristics, and metadata characteristics, multi-source data characteristics are obtained.

3. A data cleaning method based on rule data driving according to claim 1, characterized in that The steps of analyzing the input-output dependency relationships of each candidate rule in the candidate rule set and marking conflict rule pairs to obtain a conflict warning list include: Receiving a candidate rule set, parsing the input field set and output field set of each candidate rule, and generating a rule input-output field table; Based on the input-output field table, constructing a rule dependency graph; Traversing the rule dependency graph, detecting rule pairs where the output fields have an intersection and there is no dependency path, and marking them as conflict rule pairs; Determining the confidence level of the conflict rule pairs according to the cross-system field association characteristics in the data feature fingerprint; Matching the corresponding preset routing strategy according to the confidence level of each conflict rule pair, and generating a conflict warning list.

4. A data cleaning method based on rule data driving according to claim 1, characterized in that The steps of hierarchically traversing the high-priority queue and the low-priority queue to generate a main execution sequence and a branch execution sequence include: Circularly extracting the main chain nodes in the high-priority queue and adding them to the main execution sequence; When the high-priority queue is empty, extracting the current branch node in the low-priority queue and adding it to the branch execution sequence; Traversing all the outgoing edges of the current branch node, and screening the successor nodes whose node label type is branch; Updating the in-degree of the successor nodes, and adding the successor nodes whose in-degree is zero and node label type is branch to the low-priority queue.

5. A data cleaning method based on rule data driving according to any one of claims 1 to 4, characterized in that After the step of outputting the cleaned clean data stream and associated cleaning logs, it further includes: Based on the rule execution success rate and business feedback data in the cleaning log, eliminating the rules with an execution success rate lower than the preset success rate threshold and correcting the feature extraction parameters; Encoding the historical rule execution paths into a rule gene pool that can be genetically optimized, and iteratively updating the rule library version.

6. A rule-based data-driven data cleaning engine, characterized in that, The data cleaning engine includes: A receiving module for receiving the original data streams of multiple heterogeneous business systems; A feature extraction module for extracting multi-source data characteristics and cross-system field association characteristics in the original data stream, and constructing a data feature fingerprint; A rule screening module for performing similarity matching on the data feature fingerprint based on the conditional characteristics in the preset rule library, screening the rules with a matching degree higher than the preset threshold, and obtaining a candidate rule set; A conversion module for converting the candidate rule set into a directed acyclic graph; wherein, the nodes of the directed acyclic graph represent rules, and the edges represent data dependency relationships; A conflict marking module for analyzing the input-output dependency relationships of each candidate rule in the candidate rule set, marking conflict rule pairs, and obtaining a conflict warning list; A path identification table generation module for inserting a virtual isolation layer into the directed acyclic graph according to the conflict warning list, binding the conflict rule pairs with isolation layer identifiers, and generating a path identification table; A topological sorting module for performing topological sorting on the directed acyclic graph after inserting the virtual isolation layer to generate a rule execution sequence; A cleaning processing module for cleaning the original data stream according to the rule execution sequence and attaching a trajectory label to each piece of data; the trajectory label includes a rule path identifier, a field modification history, and an isolation layer identifier; An execution channel allocation module for allocating data execution channels according to the path identification table; wherein, when it is detected that the isolation layer identifier in the trajectory label matches the conflict warning list, data rerouting is triggered; An output module for outputting a cleaned clean data stream and a cleaning log including a rule execution path; The path identification table generation module is further configured to: Insert a virtual isolation layer in the directed acyclic graph according to the conflict warning list, and bind the conflict rule pair to the isolation layer identifier. The steps of generating the path identification table include: Locate the nearest common ancestor node of the conflict rule pair according to the directed acyclic graph and the conflict warning list; Insert a virtual isolation layer node after the out-edge of the nearest common ancestor node pointing to the conflict rule pair to generate an updated directed acyclic graph; Inject an isolation layer identifier, a conflict rule pair, and a bypass copy generation flag into the virtual isolation layer node; Generate a path identification table according to the preset routing policy in the conflict warning list and the cross-system association characteristics of the data feature fingerprint; The topological sorting module is further configured to: Inject node labels into the directed acyclic graph after inserting the virtual isolation layer according to the path identification table; Initialize a high-priority queue and a low-priority queue according to the node label type, and sort the nodes in the queue based on the service priority policy; the node label types include main chain nodes, branch nodes, and virtual isolation layer nodes; Traverse the high-priority queue and the low-priority queue layer by layer to generate a main execution sequence and a branch execution sequence; Obtain a rule execution sequence according to the main execution sequence and the branch execution sequence.

7. A computer device, characterized in that: It includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: Stores a computer program that can be loaded and executed by a processor to implement the method according to any one of claims 1 to 5.

Images