A Method and System for Archival Data Quality Inspection and Repair Based on Rule Conflict Resolution

Abstract

This invention provides a method and system for archival data quality detection and repair based on rule conflict resolution, belonging to the field of archival data technology. The method includes constructing a semantic graph of archival data, generating a dependent directed acyclic graph through rule conflict resolution, compiling a chain of detection operators to verify the features of catalog data and original text, and determining the optimal repair scheme through error root cause localization and impact domain analysis. This invention achieves accurate detection and minimal-impact repair of archival data quality, improves data consistency and reliability, and reduces data repair costs.

Description

Technical Field

[0001] This invention relates to the field of archival data technology, and in particular to a method and system for archival data quality detection and repair based on rule conflict resolution. Background Technology

[0002] Archival data is a vital historical record and information resource, crucial for ensuring historical continuity, supporting government decision-making, and facilitating scientific research. With the deepening of e-government and digital archives development, the scale of archival data has expanded dramatically, while data quality issues have become increasingly prominent. Archival data typically comprises two core components: a catalog data structure and original text data objects, which must maintain strict mapping consistency. Current archival data quality management primarily employs rule-based inspection, verifying the integrity, consistency, and accuracy of data through predefined rule sets, and then implementing corrective actions based on the inspection results.

[0003] Traditional detection rules suffer from semantic conflicts. When multiple rules are applied to the same data object simultaneously, the priority and dependencies between rules are difficult to manage effectively, leading to inconsistent or even contradictory detection results, affecting the accuracy and reliability of the detection. There is a lack of effective methods for verifying the consistency between catalog data and original text data. Existing technologies mainly focus on rule verification within a single data source, making it difficult to achieve feature extraction and association verification across data sources, and failing to accurately identify mapping errors between catalog data and original text entities. Errors in archival data often have propagation characteristics. Existing remediation methods lack systematic analysis of the scope of error impact, making it difficult to trace the root cause of errors and assess the impact of remediation. This leads to the introduction of new inconsistencies during the remediation process, and may even cause wider-ranging data quality problems. Summary of the Invention

[0004] This invention provides a method and system for archival data quality detection and repair based on rule conflict resolution, which can solve the problems in the prior art.

[0005] A first aspect of the present invention provides a method for archival data quality detection and repair based on rule conflict resolution, comprising:

[0006] Obtain the catalog data structure and the original data object, parse the field attribute dependencies in the catalog data structure and the format feature identifiers in the original data object, and construct a semantic graph of the archive data.

[0007] Based on the semantic graph of the archive data, the detection rules are semantically parsed and conflict resolved to generate a rule-dependent directed acyclic graph; conflict propagation path analysis and priority inheritance are performed on the rule conflict nodes in the rule-dependent directed acyclic graph to obtain the set of resolved detection rules.

[0008] The resolved detection rule set is compiled into a detection operator chain including an attribute verification operator, a feature extraction operator, and an association verification operator. The association verification operator receives the directory attribute feature vector output by the attribute verification operator and the original text physical feature vector output by the feature extraction operator. A similarity matrix is ​​constructed to perform a mapping consistency measurement and generate a structured detection result set.

[0009] Based on the error propagation path in the structured detection result set, reverse tracing is performed on the semantic graph of the archive data to locate the root cause data node that caused the error, and all downstream dependent nodes of the root cause data node are identified based on the graph traversal algorithm to generate the error impact domain topology.

[0010] Based on the error impact domain topology, assess the impact of repairing the root cause data node on the downstream dependent nodes, and generate a repair scheme that minimizes the impact.

[0011] The process of parsing the field attribute dependencies in the catalog data structure and the format feature identifiers in the original data object to construct a semantic graph of the archival data includes:

[0012] Syntax analysis is performed on the field definitions in the directory data structure to identify the reference dependencies between fields. A field dependency propagation chain is constructed by recursively traversing the reference dependencies. The field dependency propagation chain records the propagation type identifier from the source field to the target field.

[0013] The binary stream of the original data object is matched with a format signature to determine the format type identifier. Based on the format type identifier, page segmentation markers and metadata region identifiers are extracted. The timestamp and author identifier are extracted from the metadata region identifier as time dimension feature identifiers and source dimension feature identifiers, respectively.

[0014] The source and target fields in the field dependency propagation chain are converted into directory field nodes, the propagation type identifier is converted into dependency edge type attributes connecting the directory field nodes, the page corresponding to the page segmentation mark is converted into a raw text object node, the time dimension feature identifier and the source dimension feature identifier are appended as attributes of the raw text object node, and field nodes with file reference semantics are identified in the directory field nodes and cross-domain association edges are established with the raw text object nodes.

[0015] The directory field nodes, the dependency edge type attributes, the original text object nodes, and the cross-domain association edges are merged to form the archive data semantic graph.

[0016] Based on the semantic graph of the archive data, semantic parsing and conflict resolution are performed on the detection rules to generate a directed acyclic graph of rule dependencies, including:

[0017] The dependent edge type attribute and the attribute of the original object node are respectively converted into a field scope constraint set and an original scope constraint set;

[0018] Extract the object identifier and constraint expression of the rules and clauses in different levels of testing standards;

[0019] Semantic matching is performed between the detection object identifier and the field scope constraint set and the original text scope constraint set to determine the target node set pointed to by the detection object identifier; the constraint expression is converted into a formal logical expression based on the target node set.

[0020] Each rule clause is converted into a rule node. The derivation relationship between the precondition subexpression and the postcondition subexpression in the formal logical expression is extracted and converted into a directed dependency edge connecting the rule node.

[0021] The cross-domain association edges in the semantic graph of the archive data are traversed to identify rule node pairs that act on the same set of target nodes. When the constraint expressions in the rule node pairs have a logical mutual exclusion relationship, a priority identifier is added according to the priority of the hierarchical detection standard to which the rule node pairs belong.

[0022] The rule nodes and the directed dependency edges are topologically sorted. When a loop structure is detected, the dependency edge of the rule node with the lowest priority is disconnected according to the priority identifier, thus forming the rule-dependent directed acyclic graph.

[0023] For the conflicting nodes in the directed acyclic graph that the rules depend on, perform conflict propagation path analysis and priority inheritance to obtain the resolved set of detection rules, including:

[0024] Traverse the rule nodes in the directed acyclic graph that the rules depend on, and identify conflicting rule nodes that act on the same set of target nodes;

[0025] For each of the rule conflict nodes, traverse backwards along the directed dependency edges to trace the set of predecessor nodes of the rule conflict node to form a conflict propagation path;

[0026] Extract the priority identifier of each rule node in the conflict propagation path, and propagate the priority from the root node to the rule conflict node along the conflict propagation path. When the priority identifier of the predecessor node is higher than the priority identifier of the successor node, the priority identifier of the predecessor node is inherited to the successor node, the priority identifier of the successor node is updated, and a priority inheritance chain is formed.

[0027] Based on the updated priority identifier in the priority inheritance chain, the conflicting rule nodes are compared pairwise. The rule nodes with higher priority identifiers are retained, and the rule nodes with lower priority identifiers are removed from the directed acyclic graph of rule dependencies. The retained rule nodes and the rule nodes that have not conflicted are merged to form the resolved detection rule set.

[0028] The resolved detection rule set is compiled into a detection operator chain including an attribute verification operator, a feature extraction operator, and an association verification operator. The association verification operator receives the directory attribute feature vector output by the attribute verification operator and the original text physical feature vector output by the feature extraction operator. A similarity matrix is ​​constructed to perform a mapping consistency measure, generating a structured detection result set including:

[0029] Traverse the formal logical expressions in the resolved detection rule set and extract the constraint condition sub-expression types from the formal logical expressions;

[0030] According to the type of the constraint condition sub-expression, the rule nodes are classified into attribute verification rule nodes, feature extraction rule nodes and association verification rule nodes, and compiled into the attribute verification operator, the feature extraction operator and the association verification operator respectively. The call dependency relationship is constructed according to the directed dependency edge in the directed acyclic graph of the rule dependency to form the detection operator chain.

[0031] The attribute verification operator performs constraint verification on the attribute values ​​of the directory field nodes and encodes the verified attribute values ​​as directory attribute feature vectors; the feature extraction operator extracts physical structure features from the original text object nodes and encodes them as original text physical feature vectors.

[0032] The association verification operator calculates the similarity value between each feature component in the directory attribute feature vector and each feature component in the original text physical feature vector, and fills the similarity value into the corresponding position of the similarity matrix;

[0033] Based on the similarity matrix, the mapping relationship between the directory attribute feature vector and the original text physical feature vector is identified, and the structured detection result set is generated.

[0034] Based on the error propagation path in the structured detection result set, reverse tracing is performed on the semantic graph of the archive data to locate the root cause data node leading to the error. Then, based on a graph traversal algorithm, all downstream dependent nodes of the root cause data node are identified, generating an error impact domain topology including:

[0035] Mark the directory field nodes and original object nodes that fail to validate the mapping relationship as error nodes;

[0036] For each error node, a reverse traversal is performed along the semantic relationship edge in the semantic graph of the archive data. The data integrity identifier of each predecessor node in the predecessor node chain of the error node is traced. Based on the data integrity identifier, it is determined whether the predecessor node has missing data or abnormal data format. If so, the predecessor node is marked as the root cause data node.

[0037] Starting with the root cause data node, a forward traversal is performed along the semantic relationship edges in the semantic graph of the archive data to construct a directed subgraph with the root cause data node as the root. The directed subgraph contains all downstream data nodes that can be reached by the root cause data node. All downstream data nodes in the directed subgraph are marked as downstream dependent nodes.

[0038] Based on the directionality and connectivity of the semantic relationship edges between the root data node and the downstream dependent node, a topological connection matrix is ​​constructed between the nodes. The topological connection matrix records the propagation relationship of the influence of the root data node on each downstream dependent node.

[0039] Based on the topological connection matrix, the root cause data nodes, the downstream dependent nodes, the semantic relationship edges, and the influence propagation relationships are organized to obtain the error influence domain topology structure.

[0040] Based on the error impact domain topology, assess the impact of repairing the root cause data node on the downstream dependent nodes, and generate a repair scheme that minimizes the impact, including:

[0041] The in-degree value of each downstream dependent node is counted; based on the in-degree value and the hierarchical depth of the downstream dependent node in the semantic graph of the archive data, the influence weight coefficient of each downstream dependent node is calculated, and the influence weight coefficient is proportional to the in-degree value and the hierarchical depth.

[0042] A set of candidate repair operations is generated for the root cause data node; for each candidate repair operation in the set of candidate repair operations, the execution process of the candidate repair operation on the error influence domain topology is simulated based on the repair operation dependency graph, the amount of data state change of the candidate repair operation on each downstream dependent node is tracked, and the amount of data state change is weighted and summed with the adjusted influence weight coefficient to obtain the total influence value.

[0043] The candidate repair operation that minimizes the total impact value is selected as the optimal repair operation.

[0044] The optimal repair operation, the node identifier of the root cause data node, the list of node identifiers of the downstream dependent nodes, and the total impact value are encapsulated into the repair scheme.

[0045] A second aspect of the present invention provides an archival data quality detection and repair system based on rule conflict resolution, comprising:

[0046] The first unit is used to obtain the catalog data structure and the original data object, parse the field attribute dependencies in the catalog data structure and the format feature identifiers in the original data object, and construct a semantic graph of the archive data.

[0047] The second unit is used to perform semantic parsing and conflict resolution on the detection rules based on the semantic graph of the archive data, and generate a rule-dependent directed acyclic graph; and to perform conflict propagation path analysis and priority inheritance on the rule conflict nodes existing in the rule-dependent directed acyclic graph to obtain the set of resolved detection rules.

[0048] The third unit is used to compile the resolved detection rule set into a detection operator chain including an attribute verification operator, a feature extraction operator, and an association verification operator. The association verification operator receives the directory attribute feature vector output by the attribute verification operator and the original text physical feature vector output by the feature extraction operator, and generates a structured detection result set by constructing a similarity matrix to perform a mapping consistency measurement.

[0049] The fourth unit is used to perform reverse tracing on the semantic graph of the archive data according to the error propagation path in the structured detection result set, locate the root cause data node that caused the error, and identify all downstream dependent nodes of the root cause data node based on the graph traversal algorithm to generate the error impact domain topology.

[0050] The fifth unit is used to assess the impact of repairing the root cause data node on the downstream dependent nodes based on the topology of the error impact domain, and to generate a repair scheme that minimizes the impact.

[0051] A third aspect of the present invention,

[0052] An electronic device is provided, comprising:

[0053] processor;

[0054] Memory used to store processor-executable instructions;

[0055] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0056] Fourth aspect of the present invention,

[0057] A computer-readable storage medium is provided, having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0058] The beneficial effects of this application are as follows:

[0059] By constructing a semantic graph of archival data and resolving rule conflicts based on it, a directed acyclic graph of rule dependencies is generated. This enables propagation path analysis and priority inheritance of conflicting nodes, effectively avoiding the inconsistency in detection results caused by multiple rule conflicts in traditional methods. The resolved rule set is compiled into a chain of detection operators, including attribute verification operators, feature extraction operators, and association verification operators. In particular, by constructing a similarity matrix for mapping consistency measurement, the accuracy of archival data quality detection is significantly improved. Reverse tracing is performed on the archival data semantic graph based on the structured detection result set, which can accurately locate the root cause data node causing the error, rather than just stopping at the level of identifying surface error phenomena. By identifying all downstream dependent nodes of the root cause data node and generating the error impact domain topology, the impact of repair on downstream nodes is assessed, and a repair scheme that minimizes the impact is generated, avoiding the data consistency problems caused by traditional "in-situ repair". Attached Figure Description

[0060] Figure 1 This is a flowchart illustrating the method for archival data quality detection and repair based on rule conflict resolution, as described in an embodiment of the present invention.

[0061] Figure 2 A schematic diagram illustrating the process of constructing a semantic graph of archival data. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0063] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0064] Figure 1 This is a flowchart illustrating the archival data quality detection and repair method based on rule conflict resolution, as described in an embodiment of the present invention. Figure 1 As shown, the method includes:

[0065] Obtain the catalog data structure and the original data object, parse the field attribute dependencies in the catalog data structure and the format feature identifiers in the original data object, and construct a semantic graph of the archive data.

[0066] Based on the semantic graph of the archive data, the detection rules are semantically parsed and conflict resolved to generate a rule-dependent directed acyclic graph; conflict propagation path analysis and priority inheritance are performed on the rule conflict nodes in the rule-dependent directed acyclic graph to obtain the set of resolved detection rules.

[0067] The resolved detection rule set is compiled into a detection operator chain including an attribute verification operator, a feature extraction operator, and an association verification operator. The association verification operator receives the directory attribute feature vector output by the attribute verification operator and the original text physical feature vector output by the feature extraction operator. A similarity matrix is ​​constructed to perform a mapping consistency measurement and generate a structured detection result set.

[0068] Based on the error propagation path in the structured detection result set, reverse tracing is performed on the semantic graph of the archive data to locate the root cause data node that caused the error, and all downstream dependent nodes of the root cause data node are identified based on the graph traversal algorithm to generate the error impact domain topology.

[0069] Based on the error impact domain topology, assess the impact of repairing the root cause data node on the downstream dependent nodes, and generate a repair scheme that minimizes the impact.

[0070] In one optional implementation, parsing the field attribute dependencies in the catalog data structure and the format feature identifiers in the original data object to construct a semantic graph of the archive data includes:

[0071] Syntax analysis is performed on the field definitions in the directory data structure to identify the reference dependencies between fields. A field dependency propagation chain is constructed by recursively traversing the reference dependencies. The field dependency propagation chain records the propagation type identifier from the source field to the target field.

[0072] The binary stream of the original data object is matched with a format signature to determine the format type identifier. Based on the format type identifier, page segmentation markers and metadata region identifiers are extracted. The timestamp and author identifier are extracted from the metadata region identifier as time dimension feature identifiers and source dimension feature identifiers, respectively.

[0073] The source and target fields in the field dependency propagation chain are converted into directory field nodes, the propagation type identifier is converted into dependency edge type attributes connecting the directory field nodes, the page corresponding to the page segmentation mark is converted into a raw text object node, the time dimension feature identifier and the source dimension feature identifier are appended as attributes of the raw text object node, and field nodes with file reference semantics are identified in the directory field nodes and cross-domain association edges are established with the raw text object nodes.

[0074] The directory field nodes, the dependency edge type attributes, the original text object nodes, and the cross-domain association edges are merged to form the archive data semantic graph.

[0075] like Figure 2 As shown, the method includes:

[0076] A lexical analyzer is used to parse the field definition statements, extracting information such as field names, data types, and constraints. Reference relationships are determined by detecting other field reference markers appearing in the field definition, such as "reference," "dependency," or specific symbol markers (e.g., $, @). For example, in the field definition "File type: $archive type.attribute value," it can be identified that this field references the "attribute value" sub-attribute of the "archive type" field.

[0077] A depth-first search algorithm is used to recursively traverse reference dependencies to construct a field dependency propagation chain. For each field, the set of fields it directly depends on is recorded. For each directly dependent field, the recursive traversal continues until a source field without dependencies is found or a circular dependency is detected. During the traversal, dependency propagation type identifiers are also recorded, such as "numerical reference," "text inheritance," and "conditional trigger." For example, the propagation type "location reference" exists from the "file number" field to the "file location" field, indicating that the file number determines the physical storage location of the file.

[0078] The binary stream of the original data object is subjected to format signature matching. Specific bytes in the file header are read and compared with a preset format signature library. The format signature library contains magic numbers and characteristic byte sequences for common document types, such as the %PDF mark for PDF files and the PK header identifier for DOCX files. After determining the format type identifier through comparison, the corresponding parsing rules are applied according to different format types.

[0079] For PDF files, it can identify page number marks and page separators; for DOCX files, it can parse paragraph tags and page breaks; and for TXT files, it can identify page breaks or page number annotation lines. The extraction of metadata area identifiers employs different strategies for different formats: PDF files are obtained from the document information dictionary; DOCX files are extracted from core.xml; and TXT files are parsed from specific formatted lines in the file header.

[0080] The timestamp is extracted from the metadata area, and common date and time formats such as "YYYY-MM-DD HH:MM:SS" or Unix timestamps are identified and converted to a standard format. The author identifier is extracted from fields such as "Author", "Creator", or "Modifier" in the metadata, and is cleaned and standardized as necessary.

[0081] The source and target fields in the field dependency propagation chain are created as directory field nodes in the graph. Each node contains attributes such as field name, data type, and value range. The propagation type identifier is converted into dependency edges connecting nodes, with edge attributes including dependency type, strength coefficient, and propagation direction. Dependency edge types can be divided into different levels such as "mandatory dependency," "conditional dependency," and "reference dependency."

[0082] The pages corresponding to the page segmentation markers are converted into original text object nodes. Each page creates a node, which contains attributes such as page number, content summary, and keywords. Time dimension feature identifiers and source dimension feature identifiers are added as additional attributes of the original text object nodes, representing the document's time attributes (creation time, modification time, review time, etc.) and source attributes (author, department, channel, etc.), respectively.

[0083] The process of identifying field nodes with file citation semantics involves detecting fields whose names contain keywords such as "attachment," "document number," or "citation," or fields with special types such as file path or URL. Cross-domain association edges are established between these field nodes and the original text object nodes, with association edge attributes including citation type, citation strength, and citation conditions.

[0084] By merging all nodes and edges using a graph database manipulation language, a complete semantic graph of the archival data is formed. During the merging process, operations such as duplicate node elimination, conflict reconciliation, and structural optimization are performed to ensure the consistency and integrity of the graph. Once completed, the graph can be used for applications such as intelligent querying, association analysis, and knowledge reasoning of archival data.

[0085] In practical applications, for the implementation of a historical archive, the first step is to analyze the fields such as "Archive Number," "Subject Classification," and "Confidentiality Level" in its directory structure. It is identified that the "Archive Number" consists of "Year" and "Serial Number," and the "Confidentiality Level" references a "Security Level Dictionary." Through recursive analysis, a dependency propagation chain is constructed, starting with "Year," passing through "Archive Number," and ending at "Storage Location." Simultaneously, the original PDF file is parsed, extracting metadata information created on "2022-03-15" and authored by "Archives Department," identifying a document containing 53 pages. Finally, the directory field nodes and original page nodes are linked through the "Archive Number" field, forming a complete semantic graph of the archive, achieving comprehensive semantic understanding and relational analysis of the archive.

[0086] In one optional implementation, semantic parsing and conflict resolution of the detection rules based on the semantic graph of the archive data, generating a rule-dependent directed acyclic graph, includes:

[0087] The dependent edge type attribute and the attribute of the original object node are respectively converted into a field scope constraint set and an original scope constraint set;

[0088] Extract the object identifier and constraint expression of the rules and clauses in different levels of testing standards;

[0089] Semantic matching is performed between the detection object identifier and the field scope constraint set and the original text scope constraint set to determine the target node set pointed to by the detection object identifier; the constraint expression is converted into a formal logical expression based on the target node set.

[0090] Each rule clause is converted into a rule node. The derivation relationship between the precondition subexpression and the postcondition subexpression in the formal logical expression is extracted and converted into a directed dependency edge connecting the rule node.

[0091] The cross-domain association edges in the semantic graph of the archive data are traversed to identify rule node pairs that act on the same set of target nodes. When the constraint expressions in the rule node pairs have a logical mutual exclusion relationship, a priority identifier is added according to the priority of the hierarchical detection standard to which the rule node pairs belong.

[0092] The rule nodes and the directed dependency edges are topologically sorted. When a loop structure is detected, the dependency edge of the rule node with the lowest priority is disconnected according to the priority identifier, thus forming the rule-dependent directed acyclic graph.

[0093] In practical applications, the semantic graph of archival data provides the foundation for analyzing and detecting rules. It transforms the dependency edge type attributes and original object node attributes in the semantic graph to obtain the set of field scope constraints and the set of original scope constraints. Dependency edge type attributes, such as "belong to" and "component," describe the subordinate relationships between nodes. These attributes are converted into field scope constraints; for example, the "archive number" field scope constraint is "applies to all archival entities." Original object node attributes, such as "document type" and "creation date," describe the characteristics of the original archives and are converted into original scope constraints, such as "applies to financial documents created after 2020."

[0094] The system extracts the detection object identifiers and constraint expressions from rule clauses based on different levels of detection standards. Detection object identifiers are entities such as "financial statements" and "personnel files," while constraint expressions include phrases such as "creation date is no earlier than filing date" and "must include approval signature." The extraction process uses natural language processing technology to identify the subject, predicate, and object structures in the standard text and map them to detection objects and constraints.

[0095] To determine the set of target nodes pointed to by the detection object identifier, semantic matching is performed between the detection object identifier and the previously obtained set of field scope constraints and original text scope constraints. The matching process utilizes word vector similarity calculation and semantic inference rules. For example, "financial vouchers" in the detection criteria is matched with nodes such as "invoices" and "receipts" in the archival semantic graph. After matching, the constraint expression is converted into a formal logical expression based on the target node set. The conversion uses predicate logic representation, such as... For financial vouchers, there exists y ∈ approval record such that x.approvalID = y.ID".

[0096] Each rule clause is converted into a rule node. Formal logical expressions are parsed using a syntax tree, breaking them down into atomic propositions and logical connectors. For expressions containing implication relations, such as " For financial vouchers, if x.creation date > filing date, it is marked as "abnormal". The part to the left of the implied symbol "→" is identified as the precondition subexpression "x.creation date > filing date", and the part to the right is identified as the postcondition subexpression "marked as abnormal". For expressions containing conjunctive relations, such as " The approval record, y.approval status='approved' ∧ y.approval date≤archive date", identifies the atomic propositions connected by the conjunctive symbol "∧" as parallel preconditions.

[0097] Analyze the derivation relationship between precondition and postcondition subexpressions, and extract the data nodes and their attributes involved in the expressions. For the precondition "x.creation date > filing date", identify the nodes that need to be accessed: "creation date" and "filing date". For the postcondition "marked as abnormal", identify the node that needs to be operated on: "abnormal marker". Establish directed dependency edges from nodes involved in the preconditions to nodes involved in the postconditions, indicating that executing this rule requires first obtaining the data from the precondition nodes and then performing operations on the postcondition nodes.

[0098] For example, the rule "If the file is an electronic version, it must contain an electronic signature" can be broken down into the precondition "the file type is electronic version" and the postcondition "contains an electronic signature", forming a dependency edge from the "determine the file type" node to the "verify the electronic signature" node.

[0099] The system traverses cross-domain association edges in the semantic graph of archival data, identifies rule node pairs that act on the same target node set, and adds priority markers to these rule nodes based on the priority of the hierarchical detection standard to which they belong when they find that the constraints of these rule nodes are logically mutually exclusive. For example, when national standards and industry standards have different requirements for the same document attribute, the national standard is marked as having higher priority.

[0100] A topological sort is performed on the regular nodes and directed dependent edges to construct a directed graph of regular dependencies. The topological sort uses the Kahn algorithm, which adds nodes with an in-degree of 0 to a queue, continuously removes nodes from the queue and removes their related edges, and updates the in-degree of other nodes. If a cycle structure is detected during the sorting process, the dependent edges of the regular node with the lowest priority are disconnected according to the priority identifier to ensure that a directed acyclic graph is finally formed.

[0101] By generating a directed acyclic graph (DAG) of rule dependencies, the document management system can clearly identify the dependencies between rules, avoid executing contradictory detection rules, and improve the accuracy and efficiency of document management. This method is particularly suitable for complex document management scenarios involving multiple departments and standards, effectively solving the consistency problem caused by rule conflicts in traditional document detection.

[0102] In one optional implementation, conflict propagation path analysis and priority inheritance are performed on the conflicting nodes in the directed acyclic graph that the rules depend on, resulting in a set of resolved detection rules including:

[0103] Traverse the rule nodes in the directed acyclic graph that the rules depend on, and identify conflicting rule nodes that act on the same set of target nodes;

[0104] For each of the rule conflict nodes, traverse backwards along the directed dependency edges to trace the set of predecessor nodes of the rule conflict node to form a conflict propagation path;

[0105] Extract the priority identifier of each rule node in the conflict propagation path, and propagate the priority from the root node to the rule conflict node along the conflict propagation path. When the priority identifier of the predecessor node is higher than the priority identifier of the successor node, the priority identifier of the predecessor node is inherited to the successor node, the priority identifier of the successor node is updated, and a priority inheritance chain is formed.

[0106] Based on the updated priority identifier in the priority inheritance chain, the conflicting rule nodes are compared pairwise. The rule nodes with higher priority identifiers are retained, and the rule nodes with lower priority identifiers are removed from the directed acyclic graph of rule dependencies. The retained rule nodes and the rule nodes that have not conflicted are merged to form the resolved detection rule set.

[0107] A rule-dependent directed acyclic graph (DAG) is a data structure representing the dependencies between rules, where nodes represent rules and directed edges represent the dependencies between rules. Rule conflict occurs when multiple rules act on the same set of target nodes but produce different or even contradictory detection results. A rule conflict is identified by traversing all rule nodes in the graph using either depth-first search or breadth-first search, examining the target node set of each rule node. When the target node sets of two or more rule nodes overlap, these rule nodes are marked as rule conflict nodes. For example, if rule R1 and rule R2 both act on network traffic packet P1, but R1 classifies the packet as normal traffic while R2 classifies it as abnormal traffic, then R1 and R2 constitute rule conflict nodes.

[0108] Starting from the node with the conflict, a backward traversal is performed along the directed dependency edges to trace all the predecessor nodes of the conflict node, forming a conflict propagation path. This process is implemented recursively, starting from the conflict node and tracing back the predecessor of each predecessor node until the root node with no incoming edges is reached. For example, for conflict nodes R1 and R2, the backward traversal reveals that R1 depends on the root node R0, R2 depends on the intermediate node R3, and R3 depends on the root node R0, thus forming conflict propagation paths R0→R1 and R0→R3→R2.

[0109] The priority identifier of each rule node in the path is extracted. This priority identifier is a predefined value or marker used during rule design to indicate the importance or execution priority of the rule. Priority propagation occurs along the conflict propagation path from the root node to the conflicting nodes, implementing a priority inheritance mechanism. If the priority identifier of the predecessor node is higher than that of the successor node, the predecessor node's priority identifier is inherited by the successor node, updating the successor node's priority identifier. This process propagates level by level, forming a priority inheritance chain.

[0110] For example, suppose the original priorities of rule nodes R0, R1, R2, and R3 are 4, 2, 1, and 3, respectively. During priority propagation, since R0(4) > R1(2), the priority of R1 is updated to 4; similarly, since R0(4) > R3(3), the priority of R3 is updated to 4; and further, since R3(4) > R2(1), the priority of R2 is updated to 4. Finally, a priority inheritance chain is formed: R0(4) → R1(4) and R0(4) → R3(4) → R2(4).

[0111] Based on the updated priority identifier, conflicting nodes are compared pairwise, retaining the node with higher priority and removing the node with lower priority. When priorities are the same, additional strategies such as rule creation time, rule complexity, or domain-specific knowledge can be used for further judgment. In the previous example, since R1 and R2 both have a priority of 4 after priority inheritance, their creation times can be further compared. Assuming R1 was created on January 1, 2023, and R2 was created on January 5, 2023, the newer R2 is retained, and R1 is removed.

[0112] By removing low-priority rule nodes and merging the remaining rule nodes with non-conflicting rule nodes, a resolved set of detection rules is formed. This set no longer contains conflicting rules and can perform consistency checks on the target system.

[0113] The key advantage of this method is that it not only considers the direct conflicts between rules, but also the indirect dependencies between rules through conflict propagation path analysis. It also ensures that the judgment of high-priority rules can be effectively transmitted on the dependency chain through the priority inheritance mechanism, thus providing a more comprehensive and systematic solution for resolving rule conflicts.

[0114] In one optional implementation, the resolved detection rule set is compiled into a detection operator chain including an attribute verification operator, a feature extraction operator, and an association verification operator. The association verification operator receives the directory attribute feature vector output by the attribute verification operator and the original text physical feature vector output by the feature extraction operator. A similarity matrix is ​​constructed to perform a mapping consistency measure, generating a structured detection result set including:

[0115] Traverse the formal logical expressions in the resolved detection rule set and extract the constraint condition sub-expression types from the formal logical expressions;

[0116] According to the type of the constraint condition sub-expression, the rule nodes are classified into attribute verification rule nodes, feature extraction rule nodes and association verification rule nodes, and compiled into the attribute verification operator, the feature extraction operator and the association verification operator respectively. The call dependency relationship is constructed according to the directed dependency edge in the directed acyclic graph of the rule dependency to form the detection operator chain.

[0117] The attribute verification operator performs constraint verification on the attribute values ​​of the directory field nodes and encodes the verified attribute values ​​as directory attribute feature vectors; the feature extraction operator extracts physical structure features from the original text object nodes and encodes them as original text physical feature vectors.

[0118] The association verification operator calculates the similarity value between each feature component in the directory attribute feature vector and each feature component in the original text physical feature vector, and fills the similarity value into the corresponding position of the similarity matrix;

[0119] Based on the similarity matrix, the mapping relationship between the directory attribute feature vector and the original text physical feature vector is identified, and the structured detection result set is generated.

[0120] The resolved set of detection rules is compiled into a chain of detection operators, including attribute verification operators, feature extraction operators, and association verification operators. The formal logical expressions in the resolved set of detection rules are traversed to extract the constraint sub-expression types. Based on the different types of constraint sub-expressions, rule nodes can be divided into three different categories and compiled into corresponding operators, ultimately forming the chain of detection operators.

[0121] When traversing the resolved set of detection rules, the syntax structure of each formal logical expression is parsed to identify the constraint sub-expressions. For example, for the expression "IF directory.file size > 0 THEN original.file size = directory.file size", it can be identified that "directory.file size > 0" is an attribute constraint, while "original.file size = directory.file size" is an association constraint. The parser can extract the types of these constraints, preparing for subsequent rule node classification.

[0122] Based on the extracted constraint sub-expression type, rule nodes are categorized into attribute validation, feature extraction, and association validation rule nodes. Attribute validation rule nodes are responsible for validating whether the attribute values ​​of the directory field nodes meet specific conditions; feature extraction rule nodes are responsible for extracting physical structure features from the original text object; and association validation rule nodes are responsible for validating the mapping relationship between directory attributes and original text physical features. For example, the constraint "directory.filenameMATCHES '[a-zA-Z0-9]+''" is classified as an attribute validation rule node, the constraint "original text.extracted page number()" is classified as a feature extraction rule node, and the constraint "directory.page number = original text.page number" is classified as an association validation rule node.

[0123] After classification, different types of rule nodes are compiled into corresponding operators. The compilation process is as follows: An abstract syntax tree is constructed for the formal logical expressions in the rule nodes, decomposing the expressions into operators, operands, and logical relationships. For the attribute validation rule node "directory.file size > 0", the parser identifies the comparison operator ">", the left operand "directory.file size", and the right operand "0". By traversing the left operand nodes of the abstract syntax tree, the data source identifier "directory" and the field identifier "file size" are extracted and mapped to the input parameter type and parameter name of the operator. Specifically, the "file size" field attribute under the "directory" node is queried from the archival data semantic graph, and the data type of this field is determined to be integer, thus confirming that the input parameter of the operator is an "integer file size value".

[0124] After determining the input parameters, the operator types of the root node of the abstract syntax tree are analyzed. For the comparison operator ">", a corresponding verification logic code template is generated, which compares the left operand parameter with the right operand constant. The verification logic template is instantiated into an executable function by the code generator. The function body includes three parts: parameter reception, condition judgment, and result return. To determine the output result, the outgoing edges of the rule node in the directed acyclic graph of rule dependencies are analyzed to identify the input data type required by the subsequent nodes that depend on the rule node. Since the subsequent nodes of the attribute verification operator are usually association verification operators, which need to receive feature vectors as input, the output result of the attribute verification operator is determined to be "the feature vector encoded by the verified attribute value". The encoding process uses one-hot encoding or numerical normalization to convert the verified "file size" value into a component of the feature vector and append a verification pass flag.

[0125] For the feature extraction rule node "Original Text.ExtractPageNumber()", the parser identifies the function call operator "()", the data source identifier "Original Text", and the function name "ExtractPageNumber". It queries the physical structure attributes of the "Original Text" node from the archival data semantic graph to determine the file parsing interface associated with this node, mapping the function name to a specific file parsing method. When determining the input parameters, it identifies that this method requires the original text file object as input and outputs the extracted page number value. The code generator maps the function call to code that calls the file parsing interface and encodes the extraction result into components of the original text's physical feature vector.

[0126] For the association verification rule node "Table of Contents.Page Number = Original Text.Page Number", the parser identifies the equality comparison operator "=", the left operand "Table of Contents.Page Number", and the right operand "Original Text.Page Number". When determining the input parameters, it recognizes that this operator requires both the table of contents attribute feature vector and the original text physical feature vector as input. By querying the incoming edges in the directed acyclic graph of rule dependencies, it confirms that these two inputs come from the outputs of the attribute verification operator and the feature extraction operator, respectively. When determining the output, the semantics of the equality comparison operation are analyzed to generate the similarity calculation logic, and the output is the corresponding element value in the similarity matrix.

[0127] Call dependencies are constructed based on directed dependency edges in the rule-dependent directed acyclic graph, forming a chain of detection operators. This chain represents the call order and data flow between operators. During construction, it is ensured that the output of preceding operators can serve as the input of subsequent operators, guaranteeing the continuity of the data flow. For example, the output of the attribute verification operator (directory attribute feature vector) will serve as part of the input to the association verification operator, and the output of the feature extraction operator (original text physical feature vector) will serve as another part of the input to the association verification operator.

[0128] The attribute validation operator performs constraint validation on the attribute values ​​of directory field nodes and encodes the validated attribute values ​​into a directory attribute feature vector. The validation process includes type checking, range checking, and format checking. For validated attribute values, they are converted into a component of the feature vector for subsequent similarity calculation. For example, for the "file size" field in the directory, if its value is 1024 and satisfies the constraint "file size > 0", then 1024 is encoded as a component of the directory attribute feature vector.

[0129] Feature extraction operators extract physical structure features from the original text object nodes and encode them into original text physical feature vectors. The extraction process includes parsing the file structure, identifying the file format, and calculating the file size. For example, for a PDF file, physical features such as the actual number of pages, the actual file size, and the chapter structure can be extracted and encoded into components of the original text physical feature vector.

[0130] The association verification operator calculates the similarity value between each feature component in the directory attribute feature vector and each feature component in the original text physical feature vector, and fills the corresponding positions in the similarity matrix with the similarity value. Different metrics can be used for similarity calculation, such as Euclidean distance and cosine similarity. For example, for the directory attribute "file size" and the original text physical feature "actual file size," the relative error between the two can be calculated as the similarity value.

[0131] Based on the similarity matrix, the mapping relationship between directory attribute feature vectors and original text physical feature vectors is identified. The similarity matrix is ​​traversed row by row and column by column, extracting the maximum similarity value in each row and its corresponding column index to establish a preliminary mapping relationship from directory attribute components to original text physical feature components. A bidirectional optimal matching strategy is applied to refine the mapping relationship. When the optimal row match and the optimal column match point to the same pair of feature components, it is marked as a strong mapping relationship, and the confidence score is set to the similarity value itself. When they are inconsistent, the mapping relationship with higher similarity is retained as a weak mapping relationship.

[0132] A threshold filtering strategy is applied to remove mapping items with confidence scores below a threshold. For the remaining high-confidence mappings, directory attribute field identifiers and original text physical feature identifiers are extracted and encapsulated into a triple structure: "(directory attribute field identifier, original text physical feature identifier, confidence score)". For directory attribute components that fail to establish a mapping relationship, inconsistency detection records are generated, with the record content including "(directory attribute field identifier, candidate original text physical feature identifier list, maximum similarity value, inconsistency type identifier)". All high-confidence mapping triples and inconsistency detection records are grouped and aggregated according to the hierarchical structure of directory nodes. Each detection result entry includes a directory node identifier, a consistency mapping relationship list, an inconsistency detection record list, and a comprehensive consistency score, along with an additional detection rule version identifier, detection execution timestamp, and detection operator chain identifier, forming a structured detection result set.

[0133] A structured detection result set is generated. The identification process includes finding the maximum similarity value, applying threshold filtering, and cluster analysis. The final detection result set contains the mapping relationship between directory attributes and physical features of the original text, as well as the confidence score of each mapping. For example, the detection result shows that "the 'page number' field in the directory has a 92% similarity to the 'actual page number' feature extracted from the original text." Such structured results can be used for subsequent consistency assessment and anomaly detection.

[0134] Through the above process, the abstract set of detection rules can be transformed into an executable chain of detection operators, enabling automated detection of consistency between directory attributes and original text physical characteristics, thereby improving the efficiency and accuracy of data quality management.

[0135] In one optional implementation, based on the error propagation path in the structured detection result set, reverse tracing is performed on the semantic graph of the archive data to locate the root cause data node leading to the error, and all downstream dependent nodes of the root cause data node are identified based on a graph traversal algorithm to generate an error impact domain topology, including:

[0136] Mark the directory field nodes and original object nodes that fail to validate the mapping relationship as error nodes;

[0137] For each error node, a reverse traversal is performed along the semantic relationship edge in the semantic graph of the archive data. The data integrity identifier of each predecessor node in the predecessor node chain of the error node is traced. Based on the data integrity identifier, it is determined whether the predecessor node has missing data or abnormal data format. If so, the predecessor node is marked as the root cause data node.

[0138] Starting with the root cause data node, a forward traversal is performed along the semantic relationship edges in the semantic graph of the archive data to construct a directed subgraph with the root cause data node as the root. The directed subgraph contains all downstream data nodes that can be reached by the root cause data node. All downstream data nodes in the directed subgraph are marked as downstream dependent nodes.

[0139] Based on the directionality and connectivity of the semantic relationship edges between the root data node and the downstream dependent node, a topological connection matrix is ​​constructed between the nodes. The topological connection matrix records the propagation relationship of the influence of the root data node on each downstream dependent node.

[0140] Based on the topological connection matrix, the root cause data nodes, the downstream dependent nodes, the semantic relationship edges, and the influence propagation relationships are organized to obtain the error influence domain topology structure.

[0141] Directory field nodes and original text object nodes that fail to validate the mapping relationship are marked as error nodes. In the semantic graph of archival data, each node has a unique identifier and type attribute. Directory field nodes represent archival metadata information, while original text object nodes represent the actual document content. When these nodes are detected to be inconsistent with the expected mapping relationship, an error_flag=true flag is added to the node attribute, and the error type (error_type) (such as "format mismatch", "missing key information", etc.) and the error discovery time (error_timestamp) are recorded.

[0142] A depth-first search algorithm is used to perform a reverse traversal of the semantic graph of the archive data for each marked erroneous node, tracing back along the semantic relationship edges from the erroneous node. Each predecessor node has a data integrity flag, `data_integrity_flag`, whose values ​​include "complete", "partial_missing", and "format_error". During the traversal, a set of visited records, `visited_nodes`, is maintained to prevent infinite loops caused by circular dependencies. A priority queue, `queue`, is used to store nodes to be visited, sorted by node timestamp to ensure that the most recently updated node is checked first.

[0143] When a predecessor node `node_i` is accessed, its `data_integrity_flag` value is checked. If the value is "partial_missing" or "format_error", the node is marked as a root cause data node `root_cause_node`, and the discovery path `path_to_root_cause` is recorded. In some complex scenarios, there may be multiple root cause nodes; in this case, all of them are stored in the root cause node set `root_cause_set`. For each root cause node, its specific abnormal attributes also need to be recorded, such as the specific missing fields `missing_fields` or the specific format anomalies `format_anomalies`.

[0144] A breadth-first search algorithm is employed to perform a forward traversal in the semantic graph of the archival data to construct the scope of influence. Starting from the root node, the traversal proceeds along the forward direction of the semantic relationship edges. A queue is maintained to store nodes to be visited, and a set `affected_nodes` records all affected downstream nodes. The traversal process is performed hierarchically, first visiting directly connected nodes, then indirectly connected nodes, ensuring that all influence paths are captured.

[0145] For each downstream node visited, its distance from the root cause node is recorded as its impact level (impact_level), with closer nodes having a greater impact. Simultaneously, a comprehensive impact score (impact_score) is calculated based on the node type (e.g., "document node," "directory node," "relationship node") and importance weight (preset based on the node's importance within the entire archive system). When the impact_score exceeds a preset threshold, the node is added to the critically affected node set (critical_affected_nodes).

[0146] Based on the traversal results, a topology connection matrix is ​​constructed, creating an N×N matrix `connection_matrix` (where N is the total number of the root node and all its downstream dependent nodes). The matrix element `value[i][j]` represents the influence relationship between node i and node j. The influence relationship can be direct influence (value 1), indirect influence (value between 0 and 1, decreasing according to path length), or no influence (value 0). For directly connected nodes, the edge type `edge_type` (such as "containment relationship", "reference relationship", etc.) and weight `edge_weight` are also recorded for subsequent analysis of important paths in error propagation.

[0147] Organize the topology of the error impact domain by creating a directed graph data structure error_impact_topology, which includes a set of nodes (root data nodes and downstream dependent nodes), a set of edges (semantic relationship edges between nodes), and an impact matrix.

[0148] In practical applications, taking a company's document management system as an example, when a mismatch is detected between expense reimbursement documents and invoice amounts in the financial records, the expense reimbursement document node is marked as an error node. By tracing back, an abnormal data format (incorrect decimal point placement) is found in the invoice entry node, which is then marked as the root cause node. Forward traversal identifies downstream dependent nodes such as accounting vouchers and financial statements, constructing an error impact topology graph. This helps managers quickly locate the source of the problem and assess its scope, effectively supporting data governance decisions.

[0149] The error impact domain topology constructed in this way not only clearly presents the error propagation path, but also provides an intuitive view of the root cause of the error and its scope of impact, providing strong support for archival data quality management.

[0150] In one optional implementation, assessing the impact of repairing the root cause data node on the downstream dependent nodes based on the error impact domain topology, and generating a repair scheme that minimizes the impact includes:

[0151] The in-degree value of each downstream dependent node is counted; based on the in-degree value and the hierarchical depth of the downstream dependent node in the semantic graph of the archive data, the influence weight coefficient of each downstream dependent node is calculated, and the influence weight coefficient is proportional to the in-degree value and the hierarchical depth.

[0152] A set of candidate repair operations is generated for the root cause data node; for each candidate repair operation in the set of candidate repair operations, the execution process of the candidate repair operation on the error influence domain topology is simulated based on the repair operation dependency graph, the amount of data state change of the candidate repair operation on each downstream dependent node is tracked, and the amount of data state change is weighted and summed with the adjusted influence weight coefficient to obtain the total influence value.

[0153] The candidate repair operation that minimizes the total impact value is selected as the optimal repair operation.

[0154] The optimal repair operation, the node identifier of the root cause data node, the list of node identifiers of the downstream dependent nodes, and the total impact value are encapsulated into the repair scheme.

[0155] The in-degree value of each downstream dependent node is calculated. The in-degree value refers to the number of edges pointing to that downstream dependent node in the semantic graph of the archive data, representing the number of upstream data nodes that the node depends on. By traversing each downstream dependent node in the error-affected domain topology, the number of edges pointing to that node is calculated and recorded. For example, for downstream dependent node A, if three nodes B, C, and D each point to node A, then the in-degree value of node A is 3.

[0156] Based on the in-degree value and the hierarchical depth of downstream dependent nodes in the semantic graph of the archive data, the influence weight coefficient of each downstream dependent node is calculated. Hierarchical depth refers to the shortest path length from the root data node to the downstream dependent node. The formula for calculating the influence weight coefficient can be set as: Weight Coefficient = In-degree value × Hierarchical depth × Adjustment factor. The adjustment factor can be set according to specific business scenarios, with a default value of 1. For example, if the in-degree value of downstream dependent node A is 3 and the hierarchical depth is 2, then its influence weight coefficient is 6. The higher the weight coefficient, the greater the impact of the repair operation on the downstream dependent node, and the more attention it should receive when assessing the overall impact.

[0157] A set of candidate repair operations is generated for each root cause data node. These candidate repair operations may include, but are not limited to, data value correction, data structure adjustment, and modification of data relationships. For example, for erroneous numerical data, candidate repair operations may include setting the value to the average, setting the value to the median, or setting the value to a linear regression prediction of historical data, among other repair methods.

[0158] For each candidate repair operation in the generated set of candidate repair operations, the execution process of that candidate repair operation on the error-affected domain topology is simulated based on the repair operation dependency graph. The repair operation dependency graph describes the dependencies between different repair operations, ensuring that the repair operations are executed in the correct order. The simulation execution process includes the following steps: starting from the root cause data node, applying the candidate repair operation; propagating the repair impact to downstream nodes along the edges of the topology; and recording the changes in the data state of each downstream dependent node.

[0159] During the simulation, the amount of data state change for each downstream dependent node is tracked based on the candidate repair operation. The amount of data state change can be calculated in various ways, such as absolute difference or relative percentage change for numerical data, and structural similarity change for structured data. For example, if the value of a downstream node is 100 before repair and becomes 90 after repair, its data state change is 10 or 10%.

[0160] The total influence is calculated by weighting and summing the changes in data state with the adjusted influence weight coefficients. The formula is: Total Influence = ∑(Changes in data state of each downstream dependent node × Corresponding influence weight coefficient). For example, if there are three downstream dependent nodes with changes in data state of 10, 5, and 15, and influence weight coefficients of 6, 3, and 9, the total influence is 10×6 + 5×3 + 15×9 = 60 + 15 + 135 = 210.

[0161] The candidate repair operation that minimizes the total impact is selected as the optimal repair operation. By comparing the total impact values ​​of all candidate repair operations, the one with the smallest impact is chosen as the final repair solution. This ensures that the root cause data node is repaired while minimizing the impact on the entire data system.

[0162] The optimal repair operation, the node identifier of the root cause data node, the list of node identifiers of downstream dependent nodes, and the total impact value are encapsulated into a repair scheme. The data structure of the repair scheme can be designed as follows: {"rootNodeId": "Node A", "repairOperation": "Correct the value to 95", "affectedNodes": ["Node B", "Node C", "Node D"], "totalImpact": 210, "repairTime": "2023-05-15 14:30:00"}.

[0163] Such a repair plan not only includes specific repair operation information, but also records the affected nodes and the overall impact, which makes it easier for data managers to assess the rationality of the repair decision, and at the same time provides a reference for the repair of similar problems in the future.

[0164] This invention provides an archival data quality detection and repair system based on rule conflict resolution, the system comprising:

[0165] The first unit is used to obtain the catalog data structure and the original data object, parse the field attribute dependencies in the catalog data structure and the format feature identifiers in the original data object, and construct a semantic graph of the archive data.

[0166] The second unit is used to perform semantic parsing and conflict resolution on the detection rules based on the semantic graph of the archive data, and generate a rule-dependent directed acyclic graph; and to perform conflict propagation path analysis and priority inheritance on the rule conflict nodes existing in the rule-dependent directed acyclic graph to obtain the set of resolved detection rules.

[0167] The third unit is used to compile the resolved detection rule set into a detection operator chain including an attribute verification operator, a feature extraction operator, and an association verification operator. The association verification operator receives the directory attribute feature vector output by the attribute verification operator and the original text physical feature vector output by the feature extraction operator, and generates a structured detection result set by constructing a similarity matrix to perform a mapping consistency measurement.

[0168] The fourth unit is used to perform reverse tracing on the semantic graph of the archive data according to the error propagation path in the structured detection result set, locate the root cause data node that caused the error, and identify all downstream dependent nodes of the root cause data node based on the graph traversal algorithm to generate the error impact domain topology.

[0169] The fifth unit is used to assess the impact of repairing the root cause data node on the downstream dependent nodes based on the topology of the error impact domain, and to generate a repair scheme that minimizes the impact.

[0170] A third aspect of the present invention provides an electronic device, comprising:

[0171] processor;

[0172] Memory used to store processor-executable instructions;

[0173] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0174] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0175] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0176] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for archival data quality detection and repair based on rule conflict resolution, characterized in that, include: Obtain the catalog data structure and the original text data object, parse the field attribute dependencies in the catalog data structure and the format feature identifiers in the original text data object, and construct a semantic graph of the archival data, including: Syntax analysis is performed on the field definitions in the directory data structure to identify the reference dependencies between fields. A field dependency propagation chain is constructed by recursively traversing the reference dependencies. The field dependency propagation chain records the propagation type identifier from the source field to the target field. The binary stream of the original data object is matched with a format signature to determine the format type identifier. Based on the format type identifier, page segmentation markers and metadata region identifiers are extracted. The timestamp and author identifier are extracted from the metadata region identifier as time dimension feature identifiers and source dimension feature identifiers, respectively. The source and target fields in the field dependency propagation chain are converted into directory field nodes, the propagation type identifier is converted into dependency edge type attributes connecting the directory field nodes, the page corresponding to the page segmentation mark is converted into a raw text object node, the time dimension feature identifier and the source dimension feature identifier are appended as attributes of the raw text object node, and field nodes with file reference semantics are identified in the directory field nodes and cross-domain association edges are established with the raw text object nodes. The directory field nodes, the dependency edge type attributes, the original text object nodes, and the cross-domain association edges are merged to form the archive data semantic graph; Based on the semantic graph of the archive data, the detection rules are semantically parsed and conflict resolved to generate a directed acyclic graph of rule dependencies; Analyze the conflict propagation path and perform priority inheritance on the conflicting nodes in the directed acyclic graph that the rules depend on, and obtain the set of detection rules after resolution. The resolved detection rule set is compiled into a detection operator chain including an attribute verification operator, a feature extraction operator, and an association verification operator. The association verification operator receives the directory attribute feature vector output by the attribute verification operator and the original text physical feature vector output by the feature extraction operator. A similarity matrix is ​​constructed to perform a mapping consistency measure, generating a structured detection result set, including: Traverse the formal logical expressions in the resolved detection rule set and extract the constraint condition sub-expression types from the formal logical expressions; According to the type of the constraint condition sub-expression, the rule nodes are classified into attribute verification rule nodes, feature extraction rule nodes and association verification rule nodes, and compiled into the attribute verification operator, the feature extraction operator and the association verification operator respectively. The call dependency relationship is constructed according to the directed dependency edge in the directed acyclic graph of the rule dependency to form the detection operator chain. The attribute verification operator performs constraint verification on the attribute values ​​of the directory field nodes and encodes the verified attribute values ​​into a directory attribute feature vector; The feature extraction operator extracts physical structure features from the original text object nodes and encodes them into original text physical feature vectors; The association verification operator calculates the similarity value between each feature component in the directory attribute feature vector and each feature component in the original text physical feature vector, and fills the similarity value into the corresponding position of the similarity matrix; Based on the similarity matrix, the mapping relationship between the directory attribute feature vector and the original text physical feature vector is identified, and the structured detection result set is generated; Based on the error propagation path in the structured detection result set, reverse tracing is performed on the semantic graph of the archive data to locate the root cause data node that caused the error, and all downstream dependent nodes of the root cause data node are identified based on the graph traversal algorithm to generate the error impact domain topology. Based on the error impact domain topology, assess the impact of repairing the root cause data node on the downstream dependent nodes, and generate a repair scheme that minimizes the impact.

2. The method according to claim 1, characterized in that, Based on the semantic graph of the archive data, semantic parsing and conflict resolution are performed on the detection rules to generate a directed acyclic graph of rule dependencies, including: The dependent edge type attribute and the attribute of the original object node are respectively converted into a field scope constraint set and an original scope constraint set; Extract the object identifier and constraint expression of the rules and clauses in different levels of testing standards; The detection object identifier is semantically matched with the field scope constraint set and the original text scope constraint set to determine the target node set pointed to by the detection object identifier; The constraint expression is converted into a formal logical expression based on the target node set; Each rule clause is converted into a rule node. The derivation relationship between the precondition subexpression and the postcondition subexpression in the formal logical expression is extracted and converted into a directed dependency edge connecting the rule node. The cross-domain association edges in the semantic graph of the archive data are traversed to identify rule node pairs that act on the same set of target nodes. When the constraint expressions in the rule node pairs have a logical mutual exclusion relationship, a priority identifier is added according to the priority of the hierarchical detection standard to which the rule node pairs belong. The rule nodes and the directed dependency edges are topologically sorted. When a loop structure is detected, the dependency edge of the rule node with the lowest priority is disconnected according to the priority identifier, thus forming the rule-dependent directed acyclic graph.

3. The method according to claim 2, characterized in that, Analyze the conflict propagation path and perform priority inheritance on the conflicting nodes in the directed acyclic graph that the rules depend on, and obtain the set of resolved detection rules, including: Traverse the rule nodes in the directed acyclic graph that depends on the rules, and identify conflicting rule nodes that act on the same set of target nodes; For each of the rule conflict nodes, traverse backwards along the directed dependency edges to trace the set of predecessor nodes of the rule conflict node to form a conflict propagation path; Extract the priority identifier of each rule node in the conflict propagation path, and propagate the priority from the root node to the rule conflict node along the conflict propagation path. When the priority identifier of the predecessor node is higher than the priority identifier of the successor node, the priority identifier of the predecessor node is inherited to the successor node, the priority identifier of the successor node is updated, and a priority inheritance chain is formed. Based on the updated priority identifier in the priority inheritance chain, the conflicting rule nodes are compared pairwise. The rule nodes with higher priority identifiers are retained, and the rule nodes with lower priority identifiers are removed from the directed acyclic graph of rule dependencies. The retained rule nodes and the rule nodes that have not conflicted are merged to form the resolved detection rule set.

4. The method according to claim 1, characterized in that, Based on the error propagation path in the structured detection result set, reverse tracing is performed on the semantic graph of the archive data to locate the root cause data node leading to the error. Then, based on a graph traversal algorithm, all downstream dependent nodes of the root cause data node are identified, generating an error impact domain topology including: Mark the directory field nodes and original object nodes that fail to validate the mapping relationship as error nodes; For each error node, a reverse traversal is performed along the semantic relationship edge in the semantic graph of the archive data. The data integrity identifier of each predecessor node in the predecessor node chain of the error node is traced. Based on the data integrity identifier, it is determined whether the predecessor node has missing data or abnormal data format. If so, the predecessor node is marked as the root cause data node. Starting with the root cause data node, a forward traversal is performed along the semantic relationship edges in the semantic graph of the archive data to construct a directed subgraph with the root cause data node as the root. The directed subgraph contains all downstream data nodes that can be reached by the root cause data node. All downstream data nodes in the directed subgraph are marked as downstream dependent nodes. Based on the directionality and connectivity of the semantic relationship edges between the root data node and the downstream dependent node, a topological connection matrix is ​​constructed between the nodes. The topological connection matrix records the propagation relationship of the influence of the root data node on each downstream dependent node. Based on the topological connection matrix, the root cause data nodes, the downstream dependent nodes, the semantic relationship edges, and the influence propagation relationships are organized to obtain the error influence domain topology structure.

5. The method according to claim 4, characterized in that, Based on the error impact domain topology, assess the impact of repairing the root cause data node on the downstream dependent nodes, and generate a repair scheme that minimizes the impact, including: Calculate the in-degree value of each of the downstream dependent nodes; Based on the in-degree value and the hierarchical depth of the downstream dependent node in the semantic graph of the archive data, the influence weight coefficient of each downstream dependent node is calculated, and the influence weight coefficient is proportional to the in-degree value and the hierarchical depth. A set of candidate repair operations is generated for the root cause data nodes; For each candidate repair operation in the candidate repair operation set, the execution process of the candidate repair operation on the error impact domain topology is simulated based on the repair operation dependency graph. The amount of data state change of the candidate repair operation on each downstream dependent node is tracked. The amount of data state change is weighted and summed with the adjusted impact weight coefficient to obtain the total impact value. The candidate repair operation that minimizes the total impact value is selected as the optimal repair operation. The optimal repair operation, the node identifier of the root cause data node, the list of node identifiers of the downstream dependent nodes, and the total impact value are encapsulated into the repair scheme.

6. A system for detecting and repairing archival data quality based on rule conflict resolution, used to implement the method as described in any one of claims 1-5, characterized in that, include: The first unit is used to obtain the catalog data structure and the original data object, parse the field attribute dependencies in the catalog data structure and the format feature identifiers in the original data object, and construct a semantic graph of the archive data. The second unit is used to perform semantic parsing and conflict resolution on the detection rules based on the semantic graph of the archive data, and generate a rule-dependent directed acyclic graph. Analyze the conflict propagation path and perform priority inheritance on the conflicting nodes in the directed acyclic graph that the rules depend on, and obtain the set of detection rules after resolution. The third unit is used to compile the resolved detection rule set into a detection operator chain including an attribute verification operator, a feature extraction operator, and an association verification operator. The association verification operator receives the directory attribute feature vector output by the attribute verification operator and the original text physical feature vector output by the feature extraction operator, and generates a structured detection result set by constructing a similarity matrix to perform a mapping consistency measurement. The fourth unit is used to perform reverse tracing on the semantic graph of the archive data according to the error propagation path in the structured detection result set, locate the root cause data node that caused the error, and identify all downstream dependent nodes of the root cause data node based on the graph traversal algorithm to generate the error impact domain topology. The fifth unit is used to assess the impact of repairing the root cause data node on the downstream dependent nodes based on the topology of the error impact domain, and to generate a repair scheme that minimizes the impact.

7. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 5.

Images