A heterogeneous data automatic cleaning method and system based on a configurable rule engine

Abstract

The present application relates to the technical field of data cleaning, and particularly relates to a heterogeneous data automatic cleaning method and system based on a configurable rule engine. The method comprises the following steps: acquiring a heterogeneous data source, uniformly converting a data mode and an instance into a multi-dimensional heterogeneous fact graph, calculating an execution weight for each conflict rule according to an authority degree of the data source and a priority of the rule when detecting a write operation conflict, splitting data partitions by using a connected component and a node betweenness centrality, forming a rule subset corresponding to each data partition, creating an isolated Drools session instance to perform parallel cleaning, generating a traceability log in a parallel cleaning process, aggregating the traceability log after all data partitions are cleaned, re-evaluating and updating the authority degree of each data source, and solving a rule conflict in a next cleaning cycle. The scheme of the present application can accurately process a write conflict, improve the safety of parallel cleaning, shorten a processing period, and form a continuously optimized processing closed loop.

Description

Technical Field

[0001] This invention relates to the field of data cleaning technology. More specifically, this invention relates to an automated method and system for cleaning heterogeneous data based on a configurable rule engine. Background Technology

[0002] Heterogeneous data typically originates from different business systems, databases, sensor networks, or third-party platforms, and these data vary in structure, format, semantics, and quality standards. This diversity of data sources and the differences in standards easily lead to increasingly serious data silos and data quality problems. Therefore, effectively cleaning heterogeneous data has become one of the core challenges in current data governance and application fields.

[0003] To address these challenges, among numerous data cleaning technologies, rule engine-based methods have been widely adopted due to their flexibility, interpretability, and ease of maintenance. Drools, a mature, high-performance open-source business rule management system, is particularly popular. Drools effectively separates business logic from application code and defines rules through DRLs, thereby enabling data verification, transformation, and correction.

[0004] However, the aforementioned Drools-based data cleaning methods still have many shortcomings in practical applications. First, regarding rule conflict handling, when multiple data sources trigger contradictory cleaning rules on the same target entity, there is a lack of effective discrimination mechanisms, easily leading to inferior data sources overwriting superior ones. Second, in terms of execution efficiency, when faced with massive amounts of interconnected data, a single Drools session instance often becomes a performance bottleneck, resulting in high processing latency. Moreover, it typically uses simple data sharding for parallel cleaning, ignoring the relationships between data, which may compromise data integrity or trigger data competition, making truly secure parallel cleaning difficult. Furthermore, key parameters such as the authority of the data source are usually set based on human experience, unable to automatically learn and adjust according to the actual effects of the cleaning process, and the traceability of the cleaning process is insufficient, making it difficult to conduct compliance audits and attribution analysis for every change in data values. Summary of the Invention

[0005] The purpose of this invention is to propose an automated heterogeneous data cleaning method and system based on a configurable rule engine, in order to solve the problems in the prior art where the high-quality data source is covered, resulting in low accuracy of data cleaning results, low security and long processing cycle of parallel cleaning, and low traceability of the cleaning process; to this end, this invention provides solutions in the following two aspects.

[0006] In a first aspect, the present invention provides an automated heterogeneous data cleaning method based on a configurable rule engine, comprising:

[0007] Obtain heterogeneous data sources and uniformly convert the data patterns and data instances of these data sources into a multidimensional heterogeneous fact graph. Nodes represent data objects, node attributes represent data items, and edges are constructed based on inherent logical or foreign key relationships between data. Parse the DRL rule file and detect write operation conflicts. When multiple rules are detected to constitute a write operation conflict for the same data item of the same data object in the multidimensional heterogeneous fact graph, calculate the execution weight for each conflicting rule based on the authority of each data source and the priority of the rules within the current period. Select the rule with the highest execution weight as the sole execution rule. Based on the topological structure of the multidimensional heterogeneous fact graph, utilize connected components and the betweenness centrality of nodes to segment the multidimensional heterogeneous fact graph into multiple... The system divides the data into partitions; it categorizes the parsed rules into rule subsets corresponding to each data partition; it creates isolated Drools session instances for each data partition and rule subset for parallel cleaning; during parallel cleaning, it generates a source log for each rule execution, which records the original value of the data item, the new value after cleaning, the applied rule identifier, and the data source identifier corresponding to the data that triggered the rule execution; after all data partitions are cleaned, it aggregates the source logs of all Drools session instances, analyzes the data value changes triggered by each data source based on the source logs, re-evaluates and updates the authority of each data source to resolve rule conflicts in the next cleaning cycle.

[0008] Preferably, the step of uniformly converting the data schema and data instances of the data source into a multidimensional heterogeneous fact graph includes: for each data source, traversing the data tables contained in the data source; for each row of data in the data table, creating a node representing a data object, wherein the type of the node is determined by the name of the data table where the data object is located, and using the field names and field values ​​of the data as node attributes; if there is a foreign key relationship between two rows of data in two data tables, then creating a directed edge between the corresponding two nodes.

[0009] Preferably, the step of calculating the execution weight for each conflicting rule includes: extracting the preset integer priority of each rule from the DRL rule file; obtaining the current authority of the data source to which the data item triggering the current rule belongs; and calculating the product of the current authority of the data source to which the data item triggering the current rule belongs and the preset integer priority of the triggered rule as the execution weight of the conflicting rule.

[0010] Preferably, the step of dividing the multidimensional heterogeneous fact graph into multiple data partitions using the betweenness centrality of connected components and nodes includes: identifying all connected components in the multidimensional heterogeneous fact graph; treating each connected component with fewer than a preset threshold as an independent data partition; for connected components with more than or equal to the preset threshold, calculating the betweenness centrality of each node, removing the nodes and their associated edges that rank in the top 5% by betweenness centrality, treating the multiple new connected subgraphs formed after removal as independent data partitions, and assigning the removed nodes to one or more independent data partitions.

[0011] Preferably, the step of classifying the parsed rules to form a rule subset corresponding to each data partition includes: for each rule, statically analyzing the data object type matched by the condition part; identifying all nodes belonging to the type in the multidimensional heterogeneous fact graph, and determining the data partition set of node distribution; adding the rule identifier of the rule to the rule subset corresponding to each data partition in the set.

[0012] Preferably, generating a source log for each rule execution includes: when a rule is successfully executed and modifies a data item, generating a source log in JSON format; the source log record includes: timestamp, i.e., the UTC timestamp at the time of execution; objectID, i.e., the unique identifier of the modified data object; attribute, i.e., the name of the modified data item; originalValue, i.e., the value before modification; newValue, i.e., the value after modification; ruleID, i.e., the unique identifier of the executed rule; triggeringSourceID, i.e., the unique identifier of the data source that triggered the rule condition; modifiedSourceID, i.e., the unique identifier of the original data source that modified the data item.

[0013] Preferably, the reassessment and updating of the authority of each data source includes: for each data source The number of times the data source appears as triggeringSourceID in all source tracing logs and where originalValue and newValue are not equal. ; Count the number of times a modifiedSourceID appears and its originalValue is not equal to its newValue. ;Will and The sum of values ​​is used as the total number of evaluations. ;when At that time, the formula for updating authority is: ;when hour, In the formula, For the updated data source Authority Data source before update Authority This is the preset learning rate.

[0014] Preferably, the step of parsing the DRL rule file and detecting write operation conflicts includes: loading and parsing the DRL text to construct an abstract syntax tree of the rules; traversing the abstract syntax tree to identify the execution parts of all rules; establishing a mapping table to record the write operations performed by each rule on each data item of each data object; and based on the mapping table, when multiple rules are determined to modify the same data item of the same data object, it is determined to be a write operation conflict.

[0015] Preferably, the identification of all connected components in the multidimensional heterogeneous fact graph includes: identification using a breadth-first search or depth-first search algorithm.

[0016] In the second aspect, a heterogeneous data automated cleaning system based on a configurable rule engine includes:

[0017] The processor and memory, wherein the memory stores computer program instructions for automated cleaning of heterogeneous data based on a configurable rule engine, which, when executed by the processor, implement the aforementioned method for automated cleaning of heterogeneous data based on a configurable rule engine.

[0018] The beneficial effects of this invention are as follows: By unifying heterogeneous data sources into a multidimensional heterogeneous fact graph, this invention can fully identify the complex relationships between data. In terms of rule conflict handling, it can calculate the execution weight of each conflicting rule by utilizing the authority of the data source and the priority of the rule, thereby handling data write conflicts more accurately, avoiding the overwriting of high-quality data by low-quality data sources, and improving the accuracy and reliability of data cleaning results. This invention also uses graph partitioning technology to divide the data into loosely coupled partitions, achieving secure parallel cleaning and shortening the processing cycle of large-scale data. Moreover, this invention utilizes detailed traceability logs to ensure that the entire cleaning process is fully traceable and auditable, and provides objective evidence for the subsequent evaluation and updating of the authority of the data source, which helps to make better decisions in subsequent cleaning tasks, thus forming a continuously optimized processing loop. Attached Figure Description

[0019] Figure 1 This illustration schematically shows a flowchart of the steps of an automated heterogeneous data cleaning method based on a configurable rule engine in this embodiment;

[0020] Figure 2 The diagram illustrates the structure of an automated heterogeneous data cleaning system based on a configurable rule engine in this embodiment. Detailed Implementation

[0021] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0022] like Figure 1 As shown, this embodiment of the automated heterogeneous data cleaning method based on a configurable rule engine includes steps S1 to S4:

[0023] Step S1: Obtain heterogeneous data sources and convert the data patterns and data instances of the data sources into a multidimensional heterogeneous fact graph. Nodes represent data objects, node attributes represent data items, and edges are constructed based on the inherent logic or foreign key relationships between data.

[0024] Specifically, it connects to various data sources such as relational databases, NoSQL databases, and file systems. For relational databases, each data table is mapped to a type of node, each row of data in the table is converted into a specific node instance, and the data columns become node attributes. For semi-structured data such as JSON or XML, each top-level object or document is mapped to a node, and internal fields or elements become node attributes. Directed edges are established between corresponding nodes based on foreign key constraints in the database or implicit ID relationships in the data content. For example, a row of data in the customer table is converted into a customer node, the corresponding row of data in the order table pointed to by the order ID foreign key is converted into an order node, and a pointing edge is established between the customer node and the order node.

[0025] In an optional embodiment, the step of uniformly converting the data schema and data instances of the data source into a multidimensional heterogeneous fact graph includes:

[0026] For each data source, iterate through the data tables contained in the data source; for each row of data in the data table, create a node representing the data object. The type of the node is determined by the name of the data table where the data object is located, and use the field names and field values ​​of the data as node attributes; if there is a foreign key relationship between two rows of data in two data tables, then create a directed edge between the corresponding two nodes.

[0027] Specifically, when there are two data sources, namely a Customer Relationship Management System (CRM) and an Order Management System, the CRM contains a Customer table with one row containing ID equal to 123, Name equal to Zhang San, and City equal to Shanghai. The Order Management System contains an Order table with one row containing Order Number equal to A001, Customer ID equal to 123, and Amount equal to 500. During the conversion process, when processing the Customer table, the row containing Zhang San is read, and a new node is created. This node is set to Customer type and has three node attributes: ID equal to 123, Name equal to Zhang San, and City equal to Shanghai. During the conversion process, when processing the Order table, for the row containing Order Number A001, a new node of another type, Order, is created and three node attributes are added: Order Number equal to A001, Customer ID equal to 123, and Amount equal to 500. The system detects that the Customer ID field in the Order table is a foreign key, linked to the primary key ID in the Customer table. Since the Customer ID of order A001 matches the ID of customer Zhang San (both are 123), a directed edge is created between the node representing order A001 and the node representing customer Zhang San, pointing from the order node to the customer node, indicating that the order belongs to that customer. By repeating this process for all rows in all tables, a complete multidimensional heterogeneous fact graph is constructed.

[0028] Step S2: Parse the DRL rule file and detect write operation conflicts; when multiple rules are detected to constitute a write operation conflict for the same data item of the same data object in the multidimensional heterogeneous fact graph, calculate the execution weight for each conflicting rule according to the authority of each data source and the priority of the rule in the current period; select the rule with the highest execution weight as the only execution rule.

[0029] In one embodiment, parsing the DRL rule file and detecting write operation conflicts includes:

[0030] Load and parse the DRL text to construct an abstract syntax tree of rules; traverse the abstract syntax tree to identify the execution part of all rules; establish a mapping table to record the write operations performed by each rule on each data item of each data object; based on the mapping table, when multiple rules are determined to modify the same data item of the same data object, it is determined to be a write operation conflict.

[0031] Specifically, the DRL text is loaded and parsed, and an abstract syntax tree of rules is constructed. By traversing the syntax tree, the execution part of all rules, i.e. the then statement block, is identified. A mapping table is established to record which data item of which data object each rule performed a write operation on. When multiple rules are found to modify the same data item, it is determined to be a conflict.

[0032] In one embodiment, calculating the execution weight for each conflicting rule includes:

[0033] Extract the preset integer priority of each rule from the DRL rule file; obtain the current authority of the data source to which the data item that triggered the current rule belongs; calculate the product of the current authority of the data source to which the data item that triggered the current rule belongs and the preset integer priority of the triggered rule, and use it as the execution weight of the conflicting rule.

[0034] Specifically, the priority of the rule is the preset integer priority of the rule, which is a pre-set integer priority for each rule, and can be set according to the salience attribute. The authority is a floating-point number between 0 and 1, which is the value calculated after the end of the previous cleaning cycle, with an initial value of 0.5. Further, there is a DRL rule used to verify the postal code format, with the rule ID Rule-ZipCheck. In the rule file, this rule is assigned a preset integer priority of 20, indicating that it has high importance. There are two data sources: the government department database Source-Gov and the user online reporting database Source-User. After the previous round of data cleaning, the authority of the two data sources was updated after evaluation. Source-Gov's authority was updated to 0.9 due to its higher data quality, while Source-User's authority was 0.3 due to more data errors. In this round of cleaning, when data from both Source-Gov and Source-User triggered the Rule-ZipCheck rule, the execution weights were calculated to be 18 and 6, respectively.

[0035] In the event of multiple conflicting rules, the decision engine will prioritize the cleaning result produced by the rule with the higher execution weight. Therefore, it will select the rule execution instance triggered by Source-Gov as the sole execution rule for this modification, and disable the rule execution instance triggered by Source-User as the modification in the execution engine.

[0036] The execution weight of each conflict rule satisfies the expression: In the formula, To apply weights, For the sake of authority, The default integer priority for the rule.

[0037] Step S3: Based on the topology of the multidimensional heterogeneous fact graph, the multidimensional heterogeneous fact graph is divided into multiple data partitions using the betweenness centrality of connected components and nodes; the parsed rules are classified to form rule subsets corresponding to each data partition; isolated Drools session instances are created for each data partition and rule subset for parallel cleaning.

[0038] In one embodiment, the method of segmenting a multidimensional heterogeneous fact graph into multiple data partitions using connected components and the betweenness centrality of nodes includes:

[0039] Identify all connected components in the multidimensional heterogeneous fact graph; treat each connected component with fewer than a preset threshold as an independent data partition; for connected components with more than or equal to the preset threshold, calculate the betweenness centrality of each node, remove the top 5% of nodes and their associated edges, treat the resulting new connected subgraphs as independent data partitions, and assign the removed nodes to one or more independent data partitions.

[0040] The identification of all connected components in the multidimensional heterogeneous fact graph includes: using a breadth-first search or depth-first search algorithm for identification.

[0041] Specifically, the preset threshold is set empirically, and can be set to 5000 nodes. Breadth-first search or depth-first search algorithms are used to identify all connected components in the multidimensional heterogeneous fact graph. For example, if the entire multidimensional heterogeneous fact graph contains 100,000 nodes, and the set threshold is 5000 nodes, three connected components are identified: component A with 4000 nodes, component B with 3000 nodes, and component C with 93000 nodes. Components A and B both have fewer nodes than the preset threshold of 5000, so they are directly divided into two independent data partitions, namely data partition 1 and data partition 2. Component C, with 93000 nodes, exceeds the preset threshold and requires further segmentation. For the excessively large connected component component C, the betweenness centrality of all nodes within component C is further calculated. This index represents the importance of nodes as bridges in the multidimensional heterogeneous fact graph. After the calculation, all nodes are sorted from high to low according to their betweenness centrality scores, and the top 5% of nodes, i.e., 4650 nodes, are selected as split points. The 4650 bridge nodes and all their connecting edges are temporarily removed from component C. The removal operation splits the originally tightly connected component C into several smaller, disconnected subgraphs, which become data partitions. For example, if it splits into 20 new connected subgraphs, these 20 subgraphs become data partitions 3 to 22. The 4650 removed nodes can be uniformly placed into a special data partition 23, or assigned to one or more adjacent data partitions according to their connection with the new partitions.

[0042] In one embodiment, classifying the parsed rules to form a rule subset corresponding to each data partition includes:

[0043] For each rule, the data object type matched by the static analysis condition is determined; all nodes belonging to the type in the multidimensional heterogeneous fact graph are identified, and the data partition set of node distribution is determined; the rule identifier of the rule is added to the rule subset corresponding to each data partition in the set.

[0044] Specifically, it iterates through all parsed rules, analyzes the data object types involved in the matching conditions (i.e., the "when" part), and when a rule needs to match customer nodes and order nodes, and a certain data partition happens to contain both types of nodes, then the rule is classified into the rule subset of this data partition. In this way, the parsed rules can be classified according to the data objects they affect, forming rule subsets corresponding to each data partition. For example, in the previous step, the multidimensional heterogeneous fact graph was divided into 10 data partitions, namely data partition 1 to data partition 10. Now, we start processing a DRL rule with the ID R101, which checks all data objects of the customer type and verifies whether the email address attribute is formatted correctly. Static analysis is performed on rule ID R101, which parses out that the data object type pointed to by the condition part is customer. The entire multidimensional heterogeneous fact graph is queried to find all nodes of the customer type. The query results show that all customer nodes are distributed in data partition 1, data partition 3, and data partition 7, while the other data partitions (data partition 2 and data partition 4) do not contain any customer type nodes at all. Based on the information obtained, R101 is added to the rule list associated with data partition 1, data partition 3, and data partition 7, thus generating a rule subset containing R101 for the above three data partitions. When subsequent distributed cleaning tasks are started, the worker nodes processing data partitions 1, 3, and 7 will load and execute rule R101, while the worker nodes processing other data partitions will not load this rule. This avoids executing useless rules on irrelevant data and improves overall processing efficiency.

[0045] During data cleaning, an independent thread is created for each data partition. Within this thread, a Drools KieSession is instantiated, and all data nodes of that data partition are inserted into the session as facts. At the same time, only the corresponding subset of rules is loaded, thereby enabling multi-threaded parallel execution of data cleaning.

[0046] Step S4: During the parallel cleaning process, a source log is generated for each rule execution. The source log records the original value of the data item, the new value after cleaning, the rule identifier applied, and the data source identifier corresponding to the data that triggered the rule execution. After all data partitions are cleaned, the source logs of all Drools session instances are aggregated. Based on the source logs, the data value changes triggered by each data source are analyzed, and the authority of each data source is re-evaluated and updated to resolve rule conflicts in the next cleaning cycle.

[0047] Specifically, a custom agenda event listener, `AgendaEventListener`, is registered in the Drools engine. When the listener detects the `afterMatchFired` event, it indicates that a rule has been successfully executed. In the event handling method, the unique name of the activated rule is obtained from the event object as the rule identifier; the modified data object is extracted from the match object; before executing the rule's `then` statement block, the original value of the target data item is saved by deep copying or reserving a field in the data object; after the rule performs the modification operation, the cleaned new value of the data item is obtained; and the identifier of the source system is read from a preset property of the data object as the data source identifier. This information—original value, new value, rule identifier, and data source identifier—is formatted into a JSON or text log and written to a centralized log storage system, such as Elasticsearch or a database table.

[0048] After all parallel threads have finished executing, a master program is responsible for collecting all source logs generated during the current cycle from the log storage system. The master program iterates through each log entry; for each log entry recording a data value change, it identifies the data source providing the new value—the data source that triggered the rule—and also identifies the data source corresponding to the modified old value. Whenever a data source appears as a contributing source, its score increases by a fixed value; whenever it appears as a corrective source, its score decreases by a fixed value. After iterating through all logs, the scores from all data sources are normalized to generate a new round of authority, which is then stored for use in calculating the execution weights for rule conflicts at the start of the next data cleaning cycle.

[0049] In an optional embodiment, generating a source log for each rule execution includes:

[0050] When a rule is successfully executed and modifies a data item, a source log is generated in JSON format. The source log record includes: timestamp, which is the UTC timestamp at the time of execution; objectID, which is the unique identifier of the modified data object; attribute, which is the name of the modified data item; originalValue, which is the value before modification; newValue, which is the value after modification; ruleID, which is the unique identifier of the executed rule; triggeringSourceID, which is the unique identifier of the data source that triggered the rule condition; and modifiedSourceID, which is the unique identifier of the original data source that modified the data item.

[0051] Specifically, a customer data object with ID C001 is being processed. Its phone number attribute value initially comes from data source S2 and is 86-010-12345678. During the cleaning process, data from another data source S1 triggers a rule with ID R-PhoneFormat, which is used to standardize phone numbers. The rule R-PhoneFormat is successfully matched and executed, changing the phone number attribute value of customer C001 from 86-010-12345678 to 01012345678. At the moment the modification operation is completed, a traceability log is generated, which records multiple key-value pairs. Furthermore, this JSON-formatted source log record contains: the timestamp key's value is the current UTC timestamp, such as 2023-10-27T08:00:00Z; the objectID key's value is C001; the attribute key's value is the phone number; the originalValue key's value is 86-010-12345678; the newValue key's value is 01012345678; the ruleID key's value is R-PhoneFormat; the triggeringSourceID key's value is S1, because the rule was triggered by data from S1; and the modifiedSourceID key's value is S2, because the modified original value came from S2.

[0052] In an optional embodiment, the reassessment and updating of the authority of each data source includes:

[0053] For each data source The number of times the data source appears as triggeringSourceID in all source tracing logs and where originalValue and newValue are not equal. ; Count the number of times a modifiedSourceID appears and its originalValue is not equal to its newValue. ;Will and The sum of values ​​is used as the total number of evaluations. ;when At that time, the formula for updating authority is: ;when hour, In the formula, For the updated data source Authority Data source before update Authority This is the preset learning rate.

[0054] Specifically, the preset learning rate The range of values ​​is When a data cleaning cycle has just ended, and there are data source C and data source D, the implementers can determine the preset learning rate based on experience. The specific value of the learning rate, as preset in this embodiment. The authority level was 0.2; prior to this update, the current authority level of data source C was... The current authority of data source D is 0.7. The value is 0.4. All source logs generated during this period were summarized and analyzed. Statistics show that data source C, acting as the triggeringSourceID, triggered 80 data corrections, i.e. Meanwhile, other data sources, acting as modifiedSourceIDs, triggered 20 modifications, i.e. For data source D, it triggered 30 corrections as the triggeringSourceID, i.e. Other data sources, acting as modifiedSourceIDs, triggered 70 modifications, i.e. Calculate the authority of each data source after the update. For data source C, the total number of evaluations is... The updated authority level is For data source D, the total number of evaluations is: The updated authority level is After the update, the authority of data source C increased from 0.7 to 0.72, while the authority of data source D decreased from 0.4 to 0.38. The updated authority values ​​will be used in the next cleaning cycle.

[0055] This invention also provides an automated heterogeneous data cleaning system based on a configurable rule engine. For example... Figure 2As shown, the system includes a processor and a memory. The memory stores computer program instructions for automated cleaning of heterogeneous data based on a configurable rule engine. When the computer program instructions are executed by the processor, they implement the above-described method for automated cleaning of heterogeneous data based on a configurable rule engine according to the present invention.

[0056] The system also includes other components well known to those skilled in the art, such as communication buses and communication interfaces, the settings and functions of which are known in the art and therefore will not be described in detail here.

[0057] In this invention, the aforementioned memory can be any tangible medium containing or storing a program that can be used or combined with an instruction execution system, apparatus, or device. For example, a computer-readable storage medium can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc., or any other medium that can be used to store desired information and can be accessed by an application, module, or both. Any such computer storage medium can be part of a device or accessible to or connected to a device. Any application or module described in this invention can be implemented by computer-readable / executable instructions stored or otherwise maintained on such a computer-readable medium.

[0058] In the description of this specification, "multiple" means at least two, such as two, three or more, etc., unless otherwise expressly and specifically defined.

[0059] While various embodiments of the invention have been shown and described in this specification, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Many modifications, alterations, and alternatives will occur to those skilled in the art without departing from the spirit and essence of the invention.

Claims

1. A method for automatic cleansing of heterogeneous data based on a configurable rules engine, characterized in that, The method comprises the following steps: acquiring heterogeneous data sources, and uniformly converting data modes and data instances of the data sources into a multi-dimensional heterogeneous fact graph, wherein nodes represent data objects, node attributes represent data items, and edges are constructed based on inherent logic or foreign key relationships between data; parsing a DRL rule file and detecting write operation conflicts; when it is detected that multiple rules constitute write operation conflicts on the same data item of the same data object in the multi-dimensional heterogeneous fact graph, calculating an execution weight for each conflicting rule according to the authority degrees of the data sources in the current period and the priorities of the rules; selecting a rule with the highest execution weight as the only execution rule; based on the topological structure of the multi-dimensional heterogeneous fact graph, segmenting the multi-dimensional heterogeneous fact graph into multiple data partitions by using connected components and betweenness centrality of nodes, including: identifying all connected components in the multi-dimensional heterogeneous fact graph; regarding each connected component with a node number less than a preset threshold as an independent data partition; for a connected component with a node number greater than or equal to the preset threshold, calculating the betweenness centrality of each node in the connected component, removing the top 5% nodes in terms of betweenness centrality and associated edges, and regarding multiple new connected subgraphs formed after the removal as independent data partitions, and meanwhile, regarding the removed nodes as one or more independent data partitions; classifying the parsed rules to form rule subsets corresponding to the data partitions, including: for each rule, statically analyzing a data object type matched by a condition part; identifying all nodes belonging to the type in the multi-dimensional heterogeneous fact graph, and determining a data partition set in which the nodes are distributed; and adding a rule identifier of the rule to a rule subset corresponding to each data partition in the set; creating isolated Drools session instances for the data partitions and the rule subsets to perform parallel cleaning; generating a traceability log for each rule execution in the parallel cleaning process, wherein the traceability log records an original value of a data item, a new value after cleaning, a rule identifier applied, and a data source identifier corresponding to data triggering the rule execution; after all the data partitions are cleaned, aggregating traceability logs of all the Drools session instances, analyzing data value change conditions triggered by the data sources based on the traceability logs, reevaluating and updating the authority degrees of the data sources, so as to solve rule conflicts in the next cleaning period; re-evaluating and updating the authority of each data source, including: counting the number of times that each data source appears as a triggeringSourceID in all provenance logs and originalValue is not equal to newValue ; counting the number of times that each data source appears as a modifiedSourceID and originalValue is not equal to newValue ; counting the sum of and as the total number of times participating in evaluation ; when , the formula for updating the authority is: ; when , ; in the formula, is the authority of the data source after updating, is the authority of the data source before updating, is a preset learning rate.

2. The heterogeneous data automated cleansing method based on configurable rule engine according to claim 1, wherein, the conversion of the data modes and the data instances of the data sources into the multi-dimensional heterogeneous fact graph comprises the following steps: for each data source, traversing data tables contained in the data source; for each row of data in the data tables, creating a node representing a data object, wherein the type of the node is determined by the name of the data table in which the data object is located, and field names and field values of the data are taken as node attributes; if there is a foreign key association between two rows of data of two data tables, a directed edge is created between the corresponding two nodes.

3. The heterogeneous data automated cleansing method based on configurable rule engine according to claim 1, wherein, the calculation of the execution weight for each conflicting rule comprises the following steps: The preset integer priority of each rule is extracted from the DRL rule file; the current authority degree of the data source to which the data item triggering the current rule belongs is obtained; the product of the current authority degree of the data source to which the data item triggering the current rule belongs and the preset integer priority of the triggered rule is calculated as the execution weight of the conflict rule.

4. The heterogeneous data automated cleansing method based on configurable rule engine according to claim 1, wherein, The traceability log is generated for each rule execution, including: When a rule is successfully executed and a data item is modified, a traceability log is generated in JSON format; the traceability log record includes: timestamp, the UTC timestamp at the time of execution; objectID, the unique identifier of the modified data object; attribute, the name of the modified data item; originalValue, the value before modification; newValue, the value after modification; ruleID, the unique identifier of the executed rule; triggeringSourceID, the unique identifier of the data source triggering the rule condition; modifiedSourceID, the unique identifier of the original data source of the modified data item.

5. The heterogeneous data automated cleansing method based on configurable rule engine according to claim 1, wherein, The DRL rule file is parsed and the write operation conflict is detected, including: The DRL text is loaded and parsed to construct the abstract syntax tree of the rule; the execution part of all rules is identified by traversing the abstract syntax tree; a mapping table is established to record the write operation of each rule on each data item of each data object; based on the mapping table, when multiple rules are determined to modify the same data item of the same data object, the write operation conflict is determined.

6. The heterogeneous data automated cleansing method based on configurable rule engine according to claim 1, wherein, The all connected components in the multi-dimensional heterogeneous fact graph are identified, including: using the breadth-first search or depth-first search algorithm for identification.

7. A heterogeneous data automated cleansing system based on a configurable rules engine, characterized in that, Including: A processor and a memory, the memory stores computer program instructions for automatic cleaning of heterogeneous data based on a configurable rule engine, when the computer program instructions are executed by the processor, a method for automatic cleaning of heterogeneous data based on a configurable rule engine according to any one of claims 1-6 is realized.

Images