Rule engine-based electronic archive data quality automatic detection method and system

Abstract

The application provides a rule engine-based electronic archive data quality automatic detection method and system, and relates to the technical field of data processing.The method comprises the following steps: step 1, obtaining a metadata record set of an electronic archive to be detected and a pre-set structured check constraint library; step 2, according to the metadata record set, analyzing and generating an initial state feature vector of each data node; step 3, according to the structured check constraint library, performing discrete logic matching on the initial state feature vector to generate a local compliance judgment result; step 4, extracting time sequence correlation parameters in the local compliance judgment result to generate a cross-node continuity check intermediate quantity; and step 5, based on the cross-node continuity check intermediate quantity, performing an analytical differential operation in a continuous time dimension to generate a metadata time sequence drift rate analysis result.The application improves the automation level of bulk cargo port cargo damage assessment.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to an automatic detection method and system for electronic archive data quality based on a rule engine. Background Technology

[0002] In a multi-source heterogeneous environment, the metadata of electronic records often carries timestamps from different business systems and acquisition terminals. Due to the limited synchronization accuracy of clocks at each source (e.g., deviations on the order of seconds or even minutes), network transmission latency jitter, and sequential misalignment during batch acquisition, multiple metadata records associated with the same document entity will gradually accumulate small but not negligible nonlinear drifts on the timeline. Taking a group company's contract archive as an example, its process involves an OA system, a financial system, and an image scanning system. The local clocks of these three systems drift independently with a deviation of approximately 0.05 seconds per day. After several months of operation, the time sequence of the three metadata records—contract approval, financial payment confirmation, and scanning and archiving registration—may be reversed, creating a logical problem of payment before approval.

[0003] However, existing rule-based quality inspection methods can only perform static checks at the discrete field level, such as checking whether the time field of a single metadata record is empty, conforms to the YYYY-MM-DD format, or is within the valid value range. These methods cannot extract time-series correlation parameters such as the time interval and sequence of adjacent operation records across nodes, and cannot form cross-record continuity checks. When the metadata status undergoes hidden distortions along the time dimension in a non-linear manner, traditional discrete logic matching mechanisms are completely unaware of it. This causes the logical inconsistencies in the archival event chain caused by gradual time-series changes to become a systemic blind spot, affecting the evidentiary value of electronic archives and the accuracy of business traceability. Summary of the Invention

[0004] This invention provides an automatic detection method and system for electronic archive data quality based on a rule engine, which improves the automation level of cargo damage assessment at bulk cargo ports.

[0005] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows: Firstly, a method for automatically detecting the quality of electronic archival data based on a rule engine, the method comprising: Step 1: Obtain the metadata record set of the electronic document to be inspected and the pre-set structured verification constraint library; Step 2: Based on the metadata record set, parse and generate the initial state feature vector of each data node; Step 3: Based on the structured verification constraint library, perform discrete logic matching on the initial state feature vector to generate a local compliance judgment result; Step 4: Extract the time-series correlation parameters from the local compliance determination results and generate intermediate quantities for cross-node continuity verification; Step 5: Based on the intermediate quantity of cross-node continuity verification, perform analytical differentiation operation in the continuous time dimension to generate metadata time-series drift rate analysis results; Step 6: For the metadata time-series drift rate parsing results, perform cumulative integral calculation of multi-source heterogeneous archive data to generate a global correlation consistency total evaluation result; Step 7: Based on the total global correlation consistency assessment results, solve the evolution differential equation under the nonlinear distortion state to generate the data state smooth transition trajectory solution; Step 8: Compare the solution of the smooth transition trajectory of the data state with the preset compliance boundary conditions to generate a set of abnormal data node identifiers; Step 9: Based on the set of abnormal data node identifiers, perform an abnormal dimension space separation operation to generate the final data quality defect location result.

[0006] Secondly, an automatic quality detection system for electronic archival data based on a rule engine includes: The data acquisition module is used to acquire the metadata record set of the electronic archives to be inspected and the pre-set structured verification constraint library; The feature parsing module is used to parse and generate the initial state feature vector of each data node based on the metadata record set; The discrete matching module is used to perform discrete logic matching on the initial state feature vector based on the structured verification constraint library to generate a local compliance judgment result. The time-series extraction module is used to extract time-series correlation parameters from the local compliance judgment results and generate intermediate quantities for cross-node continuity verification. The differential operation module is used to perform analytical differential operations in the continuous time dimension based on the intermediate quantity of cross-node continuity verification, and generate metadata time series drift rate analysis results; The integral calculation module is used to perform cumulative integral calculation on the metadata time-series drift rate parsing results, and generate a global correlation consistency total evaluation result. The equation solving module is used to solve the evolution differential equation under nonlinear distortion state based on the global correlation consistency total evaluation result, and generate the data state smooth transition trajectory solution; The boundary comparison module is used to compare the solution of the smooth transition trajectory of the data state with the preset compliance boundary conditions to generate a set of abnormal data node identifiers. The spatial separation module is used to perform anomaly dimension spatial separation operation based on the set of abnormal data node identifiers, and generate the final data quality defect location result.

[0007] The above-described solution of the present invention has at least the following beneficial effects: Temporal correlation parameters are extracted from the local compliance judgment results to form intermediate quantities for cross-node continuity verification. Metadata temporal drift rate is generated by analytical differential operation to accurately quantify the small and non-linear offset on the time axis.

[0008] This invention utilizes cumulative integral calculations to assess the overall consistency of multi-source heterogeneous archives. By solving the evolutionary differential equations under nonlinear distortion states, it derives the smooth transition trajectory of data states and accurately compares it with compliance boundary conditions to generate a set of abnormal data node identifiers. Following anomaly dimensional space separation, it precisely outputs the data quality defect location results. Therefore, this invention can automatically detect and pinpoint temporally gradual anomalies and cross-record logical inconsistencies that traditional static rules cannot detect, improving the comprehensiveness, time sensitivity, and accuracy of hidden defect location in electronic archive data quality inspection. Attached Figure Description

[0009] Figure 1 This is a flowchart illustrating an automatic detection method for electronic archive data quality based on a rule engine, provided by an embodiment of the present invention.

[0010] Figure 2 This is a schematic diagram of an automatic electronic archive data quality detection system based on a rule engine, provided by an embodiment of the present invention. Detailed Implementation

[0011] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0012] like Figure 1 As shown, an embodiment of the present invention proposes an automatic detection method for electronic archive data quality based on a rule engine, the method comprising the following steps: Step 1: Obtain the metadata record set of the electronic document to be inspected and the pre-set structured verification constraint library; Step 2: Based on the metadata record set, parse and generate the initial state feature vector of each data node; Step 3: Based on the structured verification constraint library, perform discrete logic matching on the initial state feature vector to generate a local compliance judgment result; Step 4: Extract the time-series correlation parameters from the local compliance determination results and generate intermediate quantities for cross-node continuity verification; Step 5: Based on the intermediate quantity of cross-node continuity verification, perform analytical differentiation operation in the continuous time dimension to generate metadata time-series drift rate analysis results; Step 6: For the metadata time-series drift rate parsing results, perform cumulative integral calculation of multi-source heterogeneous archive data to generate a global correlation consistency total evaluation result; Step 7: Based on the total global correlation consistency assessment results, solve the evolution differential equation under the nonlinear distortion state to generate the data state smooth transition trajectory solution; Step 8: Compare the solution of the smooth transition trajectory of the data state with the preset compliance boundary conditions to generate a set of abnormal data node identifiers; Step 9: Based on the set of abnormal data node identifiers, perform an abnormal dimension space separation operation to generate the final data quality defect location result.

[0013] In this embodiment of the invention, by acquiring a set of metadata records and performing discrete logic matching, and by extracting time-series correlation parameters and generating intermediate quantities for cross-node continuity verification, the limitation of traditional methods that can only perform static field verification on a single node is overcome, and continuity verification across business processes is achieved. Furthermore, through analytical differential operations in the continuous time dimension and cumulative integral operations on multi-source heterogeneous data, the minute and hidden nonlinear time-series drift rate on the time axis is accurately quantified, and a global correlation consistency assessment is completed. Further, this embodiment, by solving the evolutionary differential equation under nonlinear distortion states, derives the smooth transition trajectory of the data state and performs precise comparison and anomaly dimension space separation with compliance boundary conditions, automatically discovering and locking time-series gradual anomalies and cross-record logical inconsistencies that traditional static rules cannot detect.

[0014] It effectively eliminates the systemic blind spots of inconsistencies in the logical chain of archival events, and improves the comprehensiveness, time sensitivity, and accuracy of locating hidden defects in electronic archival data quality inspection.

[0015] In a preferred embodiment of the present invention, step 1 above may include: Step 1.1 involves acquiring the metadata record set and pre-set structured validation constraint library of the electronic archives to be inspected. Specifically, based on the business scope of the electronic archives to be inspected, the system first connects sequentially to each business source system through the archive management middleware. For the group enterprise contract archive scenario, these source systems include office automation systems, financial accounting systems, and image scanning and archiving systems. The connection method uses pre-set direct database connection interfaces between systems or a lightweight data exchange protocol based on the representational state transition specification to ensure unified access capabilities for source systems implemented with different technology stacks. After the connection is established, the system sends metadata query instructions to each source system. The query instructions carry the start and end time range of the archives to be inspected, the archive category identifier, and the business entity identifier, thereby limiting the search scope and preventing irrelevant data from being mixed in.

[0016] Upon receiving the query command, each source system returns its stored metadata records. The metadata returned by the office automation system includes approval process node identifiers, operator identifiers, operation timestamps, approval opinion summaries, and process status markers; the metadata returned by the financial accounting system includes payment confirmation timestamps, amount values, payment method codes, and accounting voucher numbers; and the metadata returned by the image scanning system includes scan registration timestamps, image file storage paths, scan page counts, and image clarity scores. Because the three systems each use independent data storage formats—the office automation system uses Extensible Markup Language (XML) fragments, the financial accounting system uses relational database table structures, and the image scanning system uses semi-structured documents in key-value pair format—the system needs to parse the format of the raw data returned by each source system during the extraction phase. This involves converting the node attribute values ​​in the XML fragments, the column values ​​in the relational database tables, and the key-value pairs in the semi-structured documents into a unified, internally standardized key-value pair structure. Each metadata record is represented as a five-tuple: document entity identifier, business process identifier, attribute field name, attribute field value, and record generation timestamp.

[0017] Because the local clocks of each source system drift independently—for example, the local clocks of the office automation system, financial accounting system, and image scanning system each drift by approximately 0.05 seconds per day—the accumulated time discrepancies between these systems can reach several seconds to tens of seconds after several months. Therefore, after extracting the metadata, the system needs to align all recorded timestamps to a unified global time base. Specifically, the system initiates a Network Time Protocol (NTP) request to a trusted time service node to obtain the precise value of the current Coordinated Universal Time (UTC), which is then recorded as the global base time. Simultaneously, the system records the local server time returned by each source system when connecting to each source system. , , Calculate the offset of each source relative to the global reference time: The time offset of the office automation system is ; The time offset of the financial accounting system is ; The time offset of the image scanning system is ; The original record timestamp in each metadata record The offset is corrected according to the source system to which it belongs, resulting in a corrected unified timestamp. After this step, the timestamps of all metadata records are aligned to the same global time base.

[0018] After completing timestamp alignment, the system uses the archive entity identifier as the grouping key to group all metadata records belonging to the same archive entity into a record group. Multiple metadata records within each record group document the status information of that archive entity during its flow through different business processes. Then, within each record group, the system aligns the metadata according to the corrected unified timestamp. The records are sorted in ascending order from smallest to largest, forming a sequence of metadata records unfolding along the timeline. After sorting, each record is assigned a monotonically increasing sequence number, which will serve as the index for determining the relationship between adjacent nodes in subsequent steps. Finally, the set of all record groups constitutes the metadata record set of the electronic document to be inspected.

[0019] While acquiring the metadata record set, the system loads a pre-built structured validation constraint library from the rule configuration storage area. The construction process of this constraint library is as follows: Archival business experts, based on archival management specifications, industry compliance standards, and internal audit requirements, write validation rules one by one. Each validation rule contains the following five components: validation target field, validation type, judgment condition, tolerance range, and violation severity level. All validation rules are grouped and indexed according to two dimensions: archival category and business process, establishing a rule directory tree structure. The judgment conditions in each rule are structured and encoded, transforming them into a system-parsable conditional expression syntax tree. The leaf nodes of the syntax tree are atomic judgment units, and the intermediate nodes are logical connect words. The encoded rule set and the rule directory tree are serialized and stored together as a structured validation constraint library file for use in the detection process. When the system loads, the constraint library file is deserialized into a rule object graph in memory, and a hash mapping index from the validation target field to the rule entry is established.

[0020] In a preferred embodiment of the present invention, step 2 above may include: Step 2.1: Based on the metadata record set, perform storage path parsing and original field stripping operations to separate unstructured text tags from core business values, obtaining a sequence of independent field data pairs. Specifically, this includes: reading the metadata record set of the electronic archive to be inspected. This set is typically stored in a nested file directory structure or a hierarchical data table structure. Each metadata record contains storage path information, such as a path string composed of hierarchical identifiers like the archive's archival collection, year, retention period, and organization code. Iterate through each metadata record in this set and perform path segmentation on the storage path field. Specifically, the path string is segmented using a path separator to extract text fragments from each level of paragraph. For each extracted level of paragraph, it is matched against a pre-set archive classification dictionary to confirm the corresponding business meaning category. For example, approval process identifiers, financial subject codes, image batch numbers, etc., appearing in the path paragraph are all labeled according to their classification tags registered in the dictionary.

[0021] For a single metadata record after path resolution, a raw field stripping operation is performed. The task of raw field stripping is to separate the field name and field value from the field body containing mixed information. For highly structured metadata records, such as records stored in key-value pairs or relational table rows, the metadata information of the field definitions is directly read, the field name is extracted as a text label, and the field value is extracted as the core business value, forming an independent field data pair. For semi-structured or unstructured metadata records, such as log text or free-formatted memo fields, a regular expression pattern matching method is used: naming patterns for various field names are pre-registered in the structured validation constraint library, including the correspondence between Chinese field identifiers and English abbreviations. The raw text content of the metadata record is compared one by one with these patterns to locate the position where the field name appears, and the numerical content immediately following the field name is extracted as the corresponding field value.

[0022] After the above processing, each metadata record is split into several sets of independent field data pairs. Each set of independent field data pairs can be represented as a tuple of field name label and field value, for example, contract approval time: 2024-03-15 14:30:22, payment amount: 185000.00, number of scanned pages: 1. All independent field data pairs generated from all metadata records are arranged in the order of record source.

[0023] Step 2.2: Based on the sequence of independent field data pairs, perform data type classification and redundant character filtering operations to remove invalid placeholders and correct garbled fields, obtaining a list of clean numeric attributes. Specifically, this includes: for each field value in the sequence of independent field data pairs, classify it according to the field type definition registered in the structured validation constraint library. The type classification rules include: if the field value matches a standard date and time format, such as a four-digit year followed by a hyphen, then a two-digit month, then a hyphen, then a two-digit day, or followed by spaces and hours, minutes, and seconds, it is classified as a time type attribute; if the field value consists of pure numeric characters and does not contain letters or other symbols, it is classified as a numeric type attribute; if the field value is one of the items in a finite enumeration set, such as the approval status having only three values: approved, rejected, and pending, it is classified as an enumeration type attribute; all other cases are classified as text type attributes. After classification, a type tag is added to each field value.

[0024] For numeric and time type attributes, the filter iterates through each character position of the field value, detecting and removing invalid placeholders. Common invalid placeholders include: thousands separator commas, leading and trailing spaces, invisible zero-width characters, currency symbol prefixes or suffixes, and redundant quotation marks. The filtering operation employs a character-level blacklist / whitelist mechanism: the whitelist registers the set of legal characters allowed for numeric and time types, including numbers zero to nine, decimal points, hyphens, colons, and spaces; any characters not in the whitelist are deleted. For enumeration type attributes, the filtering operation removes leading and trailing whitespace characters from the field value and converts all English letters to lowercase to eliminate interference from case differences.

[0025] For garbled text fragments caused by encoding conversion errors in text type attributes, a character encoding feature detection method is used for identification. Specifically, the encoding value of each character in the field value is scanned. If three or more consecutive non-printable characters appear, the fragment is determined to be a garbled area. When repairing garbled areas, if a copy of the original encoding format is preserved in the metadata record, the content at the corresponding position is extracted from the copy and replaced; otherwise, the garbled area is marked as a null placeholder to ensure that subsequent processing is not interrupted.

[0026] The set of field values ​​after completing the above three processes is the list of pure numeric attributes.

[0027] Step 2.3: Based on the list of pure numerical attributes, perform unit of measurement unification and dimension standardization operations by referring to the preset archive classification dictionary. This converts discrete business indicators into calculable continuous physical parameters, resulting in a sequence of physical quantities with unified dimensions. Specifically, this includes: loading the preset archive classification dictionary, which registers the standard unit of measurement and dimension conversion factor for each type of business indicator. For example, the standard unit of measurement for monetary indicators is RMB yuan, and its lower-level units are registered as ten thousand yuan, thousand yuan, and fractional units: the conversion factor corresponding to ten thousand yuan is... That is, 10,000 yuan equals 10,000 yuan, and the conversion factor for 1,000 yuan is... The corresponding conversion factor is The standard unit of measurement for time-related indicators is the second, and its sub-units include conversion factors for hours, minutes, milliseconds, etc.: the conversion factor for hours is... The conversion factor corresponding to minutes is The conversion factor for milliseconds is The standard unit of measurement for length indicators is millimeters, for area indicators it is square meters, and so on.

[0028] Iterate through each numeric type attribute in the clean numeric attribute list, extract its field name tag, and search the archive classification dictionary for the standard unit of measurement corresponding to that field name and the currently used original unit of measurement. If the original unit of measurement is consistent with the standard unit of measurement, no conversion is needed, and the original value is directly retained. If they are inconsistent, perform a conversion operation. The conversion formula is: the standard value equals the original value multiplied by the conversion factor corresponding to the original unit. For example, the original value of a payment amount field is... The original unit is ten thousand yuan, and the conversion factor is [missing value]. Then the standard value is equal to The standard unit is yuan. For example, the original value of a certain approval time field is... The original unit is minutes, and the conversion factor is [missing information]. Then the standard value is equal to The standard unit is seconds.

[0029] For time-type attributes, the same standardization process is performed, unifying the parsing of time strings of various formats into continuous timestamp values ​​in seconds. The parsing is based on Coordinated Universal Time (UTC) 00:00:00 on January 1, 1970. Any given date and time string is converted into the cumulative number of seconds after this base zero point. The conversion process is as follows: first, the year, month, day, hour, minute, and second components in the date and time string are parsed; then, leap years are determined based on the year to ascertain the total number of days in that year; the total number of seconds in all complete years prior to that date is accumulated; each year is calculated as 365 or 366 days multiplied by 86400 seconds; the total number of seconds in all complete months prior to that date, the number of seconds in all complete days prior to that date in that month, and the number of seconds corresponding to the hour, minute, and second within that day are calculated; finally, a continuous timestamp value in seconds is obtained.

[0030] After completing the above conversion and standardization processes, all numerical and time-type attributes are expressed using a unified unit of measurement and dimension, resulting in a unified dimensional physical quantity sequence.

[0031] Step 2.4: Based on the unified dimension physical quantity sequence, perform spatial coordinate alignment and vector dimension concatenation operations along the generation timestamp order, integrate the numerical distribution states of each dimension, and construct a multi-dimensional matrix to obtain the initial state feature vector of each data node. Specifically, this includes: counting all unique field name label types appearing in the unified dimension physical quantity sequence, assuming there are a total of If there are different field name labels, then the dimension of the feature vector is . Assign a fixed vector dimension index to each field name label, with the indexes arranged sequentially from the first dimension to the second. Dimensions are defined in various ways. For example, the dimension index for contract approval time is 1, the dimension index for payment amount is 2, the dimension index for number of scanned pages is 3, the dimension index for approval agency code is 4, and so on. This dimension index mapping relationship is globally fixed and consistent across all data nodes to ensure that feature vectors from different nodes can be compared and processed in the same space.

[0032] Each physical quantity record in the unified-dimensional physical quantity sequence is associated with the generation timestamp of its corresponding metadata record. All physical quantity records are sorted in ascending order of their generation timestamps. For multiple physical quantity records with identical generation timestamps, they are aggregated into a single data node. The aggregation method is as follows: using the generation timestamp as the time identifier of the data node, mapping the field name labels of all physical quantities under that time identifier to the corresponding vector dimension index, and filling the standard values ​​into the positions of that dimension.

[0033] For each data node, initialize a length of... The zero vector, i.e., all The initial values ​​of each dimension are all set to zero. Based on the physical quantities actually included in the node, the values ​​of each dimension are filled into the corresponding positions. For dimensions not involved in the node, there are three possibilities: 1. The field is indeed not applicable to this business process, in which case the dimension value remains zero; 2. The field should have a value but is missing due to data collection omissions, in which case linear interpolation from adjacent nodes is required to complete the data. The linear interpolation calculation method is as follows: Let the missing dimension be the _th ... Dimension, searching backwards on the timeline for the most recent element containing the first element. Node A with a valid value is timestamped as , No. Dimension value Search backwards for the most recent one containing the first element. Node B with a valid value is timestamped as , No. Dimension value The timestamp of the current node is recorded as Then the current node's th Interpolation results of dimension for: ; If a node containing a valid value for that dimension is found only on one side, then that node's value is used directly. If no node is found on either side, the value remains zero, and the filled value is then used. The values ​​of each dimension are concatenated in order of their dimension indices to form a complete value. A 3D row vector. This vector is the initial state feature vector of the corresponding data node.

[0034] In this embodiment of the invention, by performing storage path parsing and original field stripping operations, unstructured text tags and core business values ​​are accurately separated, eliminating the interference of unstructured redundant data on subsequent numerical calculations, and obtaining a sequence of clearly structured independent field data pairs. Through data type classification and redundant character filtering operations, invalid placeholders are effectively removed and garbled fields are corrected, improving the purity and usability of the underlying data and forming a clean list of numerical attributes.

[0035] By comparing the data with a pre-defined archive classification dictionary and performing unit of measurement and dimension standardization operations, the originally discrete and incomparable business indicators were transformed into continuous physical parameters with unified standards. This eliminated the dimensional barriers between multi-source systems and yielded a sequence of physical quantities with unified dimensions. By performing spatial coordinate alignment and vector dimension concatenation operations along the generated timestamp sequence, the multi-dimensional numerical distribution was successfully integrated into a structured multi-dimensional matrix.

[0036] In a preferred embodiment of the present invention, step 3 above may include: Step 3.1: Based on the initial state feature vectors of each data node, extract discrete dimension state identifiers and retrieve the structured validation constraint library to locate the associated judgment conditions and tolerance intervals, establish the correspondence between feature dimensions and validation entries, and obtain a discrete state constraint mapping table. Specifically, this includes: loading the initial state feature vectors of each data node; for each initial state feature vector, sequentially reading the values ​​of each dimension from the 1st to the Nth dimension according to the dimension index; and simultaneously extracting the corresponding field name label and data type tag for each dimension value read. Since field name labels have been fixedly assigned to each dimension index in step 2.4, the corresponding field name labels can be obtained directly by querying the dimension index. The dimension index, field name label, and the actual value of that dimension in the current node are combined to form a discrete dimension state identifier for that node. This process is performed on all dimensions of all data nodes to obtain a complete set of discrete dimension state identifiers. Load the pre-built structured validation constraint library. This library is organized in tabular form, with each row corresponding to a validation constraint entry. Each entry contains the following core fields: applicable field name label or field name label regular expression pattern, decision condition type, lower tolerance limit, upper tolerance limit, constraint effective time range, and applicable file classification range. Decision condition types are categorized as follows: range constraint (requires the value to fall between the lower and upper tolerance limits); lower limit constraint (requires only the value to be no less than the lower tolerance limit); upper limit constraint (requires only the value to be no greater than the upper tolerance limit); enumeration constraint (requires the value to belong to a specified finite enumeration set); non-null constraint (requires the value to be neither null nor zero); and time-series monotonic constraint (requires the value to monotonically increase or decrease along the time axis). The constraint retrieval and matching process iterates through each field name label in the discrete dimension state identifier set, searching the structured validation constraint library for all applicable validation constraint entries. The retrieval employs multi-condition joint matching: precise or fuzzy matching between the field name label and the field name label pattern in the constraint entry; intersection determination between the file category range of the data node and the applicable scope of the constraint entry; and inclusion relationship determination between the generation timestamp of the data node and the effective time range of the constraint entry. For successfully matched constraint entries, all of them are retrieved and associated with the corresponding field name label and dimension index. The matching results are organized into a discrete state constraint mapping table. The table's structure is as follows: each row records the correspondence between a feature dimension and a set of validation constraint entries. The row content includes the dimension index, field name label, and a list of applicable validation constraint entries. Each entry contains the judgment condition type, lower tolerance limit, upper tolerance limit or enumerated set of allowed values, and constraint weight coefficient. The constraint weight coefficient is a fixed parameter pre-set in the structured validation constraint library, taking the value of a real number between 0 and 1, reflecting the importance of the constraint entry in the overall compliance judgment. For example, the range constraint weight coefficient for the core amount field is set to... The weight coefficient for the non-null constraint of the auxiliary remarks field is set to .

[0037] Step 3.2: Based on the discrete state constraint mapping table, perform a logical boundary comparison operation, verify the inclusion relationship between the actual value and the tolerance range component by component, identify out-of-bounds data blocks and mark the positive and negative deviation directions to obtain local deviation state identifiers. Specifically, this includes: traversing the initial state feature vector of each data node; for each dimension of the current node, searching for all verification constraint entries corresponding to that dimension in the discrete state constraint mapping table; if a dimension has no constraint entries in the mapping table, it means that the dimension is not within the current quality inspection range and is skipped without processing; for each verification constraint entry found, perform the corresponding logical boundary comparison operation according to its judgment condition type, as detailed below: When the judgment condition type is a range constraint, the current actual value of the dimension is compared with the lower tolerance limit and the upper tolerance limit. If the actual value is greater than or equal to the lower tolerance limit and less than or equal to the upper tolerance limit, it is considered compliant, and the deviation is set to zero. If the actual value is less than the lower tolerance limit, it is considered a negative out-of-bounds error, and the deviation is equal to the lower tolerance limit minus the actual value. If the actual value is greater than the upper tolerance limit, it is considered a positive out-of-bounds error, and the deviation is equal to the actual value minus the upper tolerance limit. When the judgment condition type is a lower limit constraint, only the actual value is compared with the lower tolerance limit. If the actual value is greater than or equal to the lower tolerance limit, it is considered compliant, and the deviation is set to zero. If the actual value is less than the lower tolerance limit, it is considered a negative out-of-bounds error, and the deviation is equal to the lower tolerance limit minus the actual value. When the judgment condition type is an upper limit constraint, only the actual value is compared with the upper tolerance limit. If the actual value is less than or equal to the tolerance limit, it is considered compliant, and the deviation is set to zero. If the actual value is greater than the tolerance limit, it is considered a positive out-of-bounds error, and the deviation is equal to the actual value minus the tolerance limit. When the judgment condition type is an enumeration constraint, the actual value is compared with each value in the enumerated set of allowed values ​​one by one. If the actual value is equal to any allowed value in the set, it is considered compliant, and the deviation is set to zero. If the actual value is not equal to any allowed value in the set, it is considered a deviation-type out-of-bounds error, and the deviation is marked with a special identifier value of -1, indicating that it does not belong to the allowed enumeration range. When the judgment condition type is a non-null constraint: check whether the actual value is null or zero. If it is not null and not zero, it is considered compliant, and the deviation is set to zero. If it is null or zero, it is considered a missing-type out-of-bounds error, and the deviation is marked with a special identifier value of -2. When the judgment condition type is a time-series monotonic constraint: it is not processed separately in this sub-step, but the comparison task is suspended and marked, to be checked in step 4 in conjunction with the time series data of adjacent nodes for continuity verification.

[0038] After comparing a constraint entry, the comparison result is recorded as a local deviation status identifier. Each local deviation status identifier contains the following information: the timestamp of the data node to which it belongs, the index of the dimension to which it belongs, the field name label, the constraint entry number, the judgment condition type, the actual value, the lower tolerance limit, the upper tolerance limit, the deviation amount, and the out-of-bounds direction marker. The out-of-bounds direction marker has the following value rules: zero indicates compliance and no out-of-bounds, positive one indicates a positive out-of-bounds actual value is too large, negative one indicates a negative out-of-bounds actual value is too small, negative two indicates a missing out-of-bounds, and negative three indicates an enumerated deviation-of-bounds. After comparing all constraint entries of the current node and the current dimension, all local deviation status identifiers for that dimension are summarized. The same operation is performed on all nodes and all dimensions to finally obtain a set of local deviation status identifiers covering the entire scope to be detected.

[0039] Step 3.3: Based on the local deviation status identifiers, perform compliance weight allocation and binary classification judgment, summarize the matching status of each dimension and remove invalid alarms, generate a Boolean verification conclusion, and obtain the local compliance judgment result. Specifically, this includes: traversing each data node, collecting all local deviation status identifiers generated by all dimensions under that node, and for each local deviation status identifier, extracting its corresponding constraint weight coefficient. This constraint weight coefficient comes from the preset weight value of the constraint entry in the discrete state constraint mapping table. Multiply the absolute value of the deviation by the constraint weight coefficient to obtain the weighted deviation score of that deviation status. The calculation formula is: For example, the deviation in a certain payment amount dimension is Yuan, positive out-of-bounds, its constraint weight coefficient is The weighted deviation score is then... For example, if the deviation of a certain remarks field is a missing type out-of-bounds error, it is marked as -2, and the deviation is recorded as... Its constraint weight coefficient is The weighted deviation score is then... .

[0040] For deviations with special identifiers of -1, -2, and -3, the absolute value of the deviation is uniformly represented by the numerical value. They are included in the weighted calculation because these types of out-of-bounds errors do not involve continuous numerical ranges, but only focus on the fact that an out-of-bounds error has occurred.

[0041] After the weighted deviation score is calculated, check whether each local deviation status identifier has an alarm suppression condition. Alarm suppression conditions include: the dimension is marked as a condition-optional dimension in the structured validation constraint library, and the file category to which the data node belongs is not within the mandatory inspection range; or although the deviation of the dimension is not zero, its absolute value is less than a preset ignore threshold. Ignore thresholds are set separately for each dimension, typically one-thousandth of the tolerance interval width of that dimension. If the absolute value of a deviation is less than the corresponding ignore threshold, the deviation status identifier is considered an invalid alarm, its weighted deviation score is forcibly set to zero, and it is not included in the subsequent compliance judgment summary.

[0042] The weighted deviation scores, after invalid alarm removal, are summed by data node. Assume the current node has a total of... There are 10 valid local deviation status indicators, and the weighted deviation score for each indicator is 1. ( from arrive Then the total deviation score of that node is... It equals the sum of the weighted deviation scores: ; The total deviation score for each data node The node's compliance score is compared to a preset threshold. This threshold is a globally preset parameter set based on the stringency of the document quality inspection. If the total deviation score is less than or equal to the node's compliance threshold, the node is deemed compliant, and the Boolean verification result is set to true. If the total deviation score is greater than the node's compliance threshold, the node is deemed non-compliant, and the Boolean verification result is set to false.

[0043] The judgment results for each data node are summarized into a single record, which includes: the timestamp identifier of the data node, a Boolean verification conclusion, the total deviation score, and a detailed list of deviations for each dimension under that node. The deviation details list lists the field name label, constraint entry number, out-of-bounds direction marker, and weighted deviation score for each item.

[0044] In this embodiment of the invention, by extracting discrete dimension state identifiers and retrieving a structured verification constraint library, a correspondence between feature dimensions and verification entries is established, resulting in a discrete state constraint mapping table. This achieves accurate mapping between verification conditions and multi-source heterogeneous data features, avoiding the waste of computational resources caused by blind verification. By performing logical boundary comparison operations, the actual values ​​and tolerance ranges are checked component by component, out-of-bounds data blocks are identified and positive and negative deviation directions are marked, resulting in local deviation state identifiers. This effectively overcomes the shortcomings of traditional static verification, which can only determine compliance but cannot perceive the degree and direction of deviation, thus improving the granularity of deviation identification.

[0045] By performing compliance weight allocation and binary classification judgment, the matching status of each dimension is summarized and invalid alarms are removed, a Boolean verification conclusion is generated, and a local compliance judgment result is obtained, which reduces the false alarm rate and redundant alarm interference of data quality detection in complex business scenarios.

[0046] In a preferred embodiment of the present invention, step 4 above may include: Step 4.1: Based on the local compliance judgment results, extract the deviation data nodes containing timestamp attributes, filter static isolated data without time-series dependencies, and obtain a subset of time-series correlated data. Specifically, this includes: loading the full set of local compliance judgment results, where each record contains the timestamp identifier of the data node, a Boolean verification conclusion, the total deviation score, and a detailed list of deviations for each dimension; traversing the set and performing a first round of screening based on the value of the Boolean verification conclusion: extracting all nodes with false verification conclusions (i.e., nodes judged as non-compliant) to form the initial screening result set for this round. These nodes carry candidate deviation information that needs to be analyzed for time-series correlation; for each non-compliant data node in the initial screening result set, checking the completeness of its timestamp attribute. The timestamp attribute includes the generation timestamp of the node's metadata record and the values ​​of each time type dimension processed. If the timestamp attribute of a node is null or invalid, such as an abnormal timestamp of zero or a negative number, then the node is determined not to meet the basic conditions for time-series analysis, is classified into the static isolated data category, and is directly removed from the candidate set.

[0047] The core logic of performing time-series dependency judgment is to check whether the current non-compliant data node has a predecessor or successor node on the same archive flow chain. Specifically, it searches forward from the current node's generation timestamp, looking for other data nodes whose generation timestamps are earlier than the current node and whose time difference is within a preset maximum business interval (e.g., 30 calendar days), and backward from the current node, looking for other data nodes whose generation timestamps are later than the current node and whose time difference is within the preset maximum business interval. This maximum business interval is a global parameter set based on the actual flow cycle of the archive business, and can be configured separately for different archive categories. If no other data node is found within either the forward or backward search range, the current node is considered a statically isolated data node without time-series dependency. If at least one predecessor or successor node is found, the current node is considered a dynamically flowing data node with time-series dependency. Typical scenarios for statically isolated data nodes without time-series dependency include: a single isolated imported metadata record whose business process has no preceding or following records in other systems; or incomplete archive fragments caused by data migration. Deviations in these nodes cannot be confirmed or corrected through cross-node continuity checks, and therefore are filtered out. After the above filtering operation, all non-compliant data nodes remain, forming a subset of time-series related data. Each node in this subset simultaneously meets three conditions: the verification conclusion is non-compliant, the timestamp attribute is complete and valid, and there is at least one adjacent node located on the same archive transfer chain.

[0048] Step 4.2: Based on the aforementioned time-series associated data subset, perform adjacent node difference calculation and state transition tracking along the archive circulation timeline to extract continuously changing business indicator values ​​and obtain a time-series associated parameter sequence. Specifically, this includes: arranging all data nodes in the time-series associated data subset in ascending order of their generation timestamps; after sorting, establishing directed adjacency relationships for nodes on the same archive circulation chain. The criterion for determining whether two nodes belong to the same archive circulation chain is: the two nodes share at least one business indicator field of the same dimension, and the value of this shared field in the two nodes has a reasonable business continuity relationship. For example, if the contract approval node and the financial payment node share the contract number dimension and the contract number values ​​are consistent, then they belong to the same circulation chain. For each node, determine its direct predecessor node and direct successor node.

[0049] For each pair of nodes with direct preceding-following relationships in the time-series associated data subset, predecessor node A and successor node B, calculate their numerical differences across each shared business dimension. Let predecessor node A be at the... The value of dimension is The successor node B is in the th The value of dimension is Then the first node between the two nodes absolute difference of dimensions The calculation formula is: ; Simultaneously calculate the time interval between the two nodes. The calculation formula is: ; in This is the generation timestamp of the predecessor node A (a continuous value in seconds). This is the generation timestamp of the successor node B.

[0050] For each pair of adjacent nodes, not only is the numerical difference calculated, but the direction and pattern of state transitions also need to be tracked. State transition directions are categorized as positive, negative, and zero transitions. State transition patterns are labeled based on the business semantics of that dimension: the expected monotonic direction of that dimension is queried in a structured validation constraint library; the actual transition direction is compared with the expected monotonic direction, and labeled as conforming to expectation, deviating from expectation, or reversing direction. For example, the approval level sequence dimension in a contract approval process is expected to be monotonically increasing; if the actual transition occurs... If the result is negative, it is marked as a direction reversal.

[0051] The results of the above calculations and tracking are summarized to generate a time-series correlation parameter record for each pair of adjacent nodes. This record contains the following fields: predecessor node timestamp. Successor node timestamp Time interval Numerical differences between shared dimensions The system includes state transition direction markers and state transition mode markers for each dimension. Additionally, a comprehensive change magnitude parameter is calculated to characterize the overall degree of change between adjacent nodes. This parameter is calculated by weighted summing of the absolute values ​​of the numerical differences across shared dimensions, with weight coefficients taken from the constraint weight coefficients corresponding to each dimension in the discrete state constraint mapping table from step 3. The calculation formula is as follows: ; The time-series correlation parameter records of all adjacent node pairs are arranged in ascending order of the predecessor node timestamp, forming the output of this sub-step: the time-series correlation parameter sequence.

[0052] Step 4.3: Based on the time-series correlation parameter sequence, perform time window sliding and data alignment operations to eliminate phase offset caused by acquisition time differences, obtaining a standardized time-series correlation parameter set. Specifically, this includes: analyzing the generation timestamp source of all data nodes in the time-series correlation parameter sequence, identifying the source system identifier of each timestamp. Since the OA system, financial system, and image scanning system described in the background technology each use independent local clocks, there is a fixed or slowly changing offset between the timestamps recorded in different systems for the same business event. One system is selected, typically the one with the highest clock accuracy or synchronized with the standard time service, as the standard time reference. The clocks of the remaining systems are considered as subordinate clocks with offsets. For each source system that is not a standard time reference, the difference between the timestamp recorded by that source system and the timestamp of the same business event recorded by the standard time reference system is extracted from the time-series correlation parameter sequence. Since the same business event may be recorded in different systems, for example, for the event of contract approval completion, the OA system records the approval time, while the financial system also records the receipt time when receiving the approval result. The clock offset can be estimated by the timestamp difference between the corresponding events. Assuming the same business event occurs in the standard system The timestamp recorded in the middle is In subordinate systems The timestamp recorded in the middle is Then the subordinate system Compared to the standard system clock offset The calculation formula is: ; For subordinate systems To avoid the influence of individual outliers, the median of all offsets for which corresponding events can be found is taken to obtain the estimated clock offset of the slave system. .

[0053] Define a time window with a fixed width. Based on the average pace of archival workflow, the time window is typically set to 3600 seconds, or 1 hour. The starting position of the time window is aligned with the earliest timestamp in the time-series associated parameter sequence, and the window covers the period from the starting time to the next specified time. Record all time-series correlation parameters within the time interval. Then, slide the window to the right, with each slide step being half the window width, i.e. The window size is set to allow for 50% overlap between adjacent windows to prevent data at boundary locations from losing continuity due to window splitting.

[0054] Within each time window, a clock skew correction operation is performed on the timestamps of data nodes from non-standard time reference systems. The correction formula is: ; in, This represents the median clock offset for the slave system. After correction, the timestamps of all data nodes are aligned to the standard time reference, eliminating the impact of source clock asynchrony.

[0055] After clock offset correction, there may be multiple data nodes within the time window whose timestamps are very close but not exactly equal. This is due to slight differences in network transmission latency and system processing latency. For such timestamps, the median time of the window is used as the alignment anchor. All data nodes whose absolute difference between the corrected timestamps within the window and the alignment anchor is less than a preset alignment tolerance, typically set to 60 seconds, are uniformly adjusted to the alignment anchor. The alignment anchor is calculated as follows: take the median of all corrected timestamps within the window; after clock offset correction and window alignment, recalculate the time interval between adjacent node pairs in the timing correlation parameter sequence. Differences between each dimension By using the corrected and aligned timestamps and the re-associated adjacent node pairs, a standardized timing correlation parameter set is obtained. This parameter set eliminates phase noise caused by multi-source clock differences.

[0056] Step 4.4: Based on the standardized temporal correlation parameter set, construct a state transition matrix between adjacent data nodes to quantify the logical dependency strength of continuous business links and obtain intermediate quantities for cross-node continuity verification. Specifically, this includes: for each pair of adjacent nodes in the standardized temporal correlation parameter set, predecessor node A and successor node B, construct a... OK The state transition matrix of the column, where This represents the number of business dimensions shared by this pair of nodes. The matrix's [number]th [dimension]. Line number Column elements Indicates that the predecessor node A is at the 1st epoch. The numerical change in dimension affects the successor node B in the th dimension. Factors influencing the numerical change of dimensions.

[0057] The impact factor is calculated as follows: for the diagonal elements of the matrix, i.e. Self-transfer of the same dimension between preceding and following nodes, influencing factor The calculation formula is: ; in, For the predecessor node A in the th The numerical value of the dimension. For the successor node B in the th The numerical value of the dimension. Let be a very small positive number, taking the value of This is used to prevent the denominator from being zero. A positive number indicates that the dimension is increasing along the direction of flow; a negative number indicates that it is decreasing; if... The absolute value of the value is close to zero, indicating that the change in this dimension is very small.

[0058] For the off-diagonal elements of the matrix, i.e. This indicates the cross-influence between different dimensions; the influencing factor. The calculation needs to be combined with business logic dependencies. If the structured validation constraint library contains constraints from the first... Dimension to the If the business logic association rules of a dimension (e.g., the approval amount dimension affects the payment amount dimension) are used, the calculation method is as follows: ; in, The cross-effect weight coefficient is a pre-defined value in this association rule, ranging from zero to one, representing the first... Victory The strength of business logic constraints in the dimension. If the constraint library does not contain constraints from the first dimension... Dimension to the Association rules for dimensions, then Set it to zero.

[0059] Based on the constructed state transition matrix The Frobenius norm of this matrix is ​​calculated as a quantitative indicator of the strength of the logical dependency between these adjacent nodes. The Frobenius norm is calculated as the square root of the sum of the squares of all elements in the matrix: ; The larger the norm value, the stronger the impact of the predecessor node's state change on the successor node's state change, indicating a tighter logical coupling between the two business processes. Conversely, the smaller the norm value, the closer the state transitions between the predecessor and successor nodes are to independent changes, indicating a weaker logical dependency.

[0060] The state transition matrix of each pair of adjacent nodes The strength of their logical dependencies is summarized into a single intermediate record for cross-node continuity verification. This record includes: the timestamp of the predecessor node. The timestamp of the successor node State transition matrix Logical dependency strength value, time interval The parameters include a comprehensive change range parameter. In addition, a continuity score is added to comprehensively evaluate whether the state transition of the pair of adjacent nodes meets the continuity expectations of the archival workflow. The formula for calculating the continuity score is: ; The meaning of this formula is: under the same logical dependency strength, the longer the time interval, the lower the continuity score, because the uncertainty of state transition under a long time interval is greater; the shorter the time interval, the higher the continuity score, because the state transition under a short time interval is more in line with the characteristics of tightly coupled business links. The intermediate quantity records of cross-node continuity verification of all adjacent node pairs are arranged in ascending order of the predecessor node timestamp, which constitute the intermediate quantity of cross-node continuity verification.

[0061] In this embodiment of the invention, by extracting deviation data nodes containing timestamp attributes and filtering static isolated data without time-series dependencies, a subset of time-series related data is obtained. This accurately focuses on deviation nodes with time-series evolution characteristics, eliminates the interference of irrelevant static data on time-series analysis, and reduces the computational complexity of subsequent time-series analysis.

[0062] Building upon this foundation, by performing adjacent node difference calculations and state transition tracking along the archive flow timeline, continuously changing business indicator values ​​are extracted, resulting in a time-series correlation parameter sequence. This effectively captures the dynamic evolution of business indicators during continuous flow, achieving a leap from static single-point verification to dynamic continuous tracking. By performing time window sliding and data alignment operations, phase offsets caused by acquisition time differences are eliminated, resulting in a standardized time-series correlation parameter set. This overcomes the time-series misalignment problem caused by clock asynchrony or network latency in multi-source heterogeneous systems, ensuring the comparability and consistency of data from different sources under the same time reference.

[0063] By constructing a state transition matrix between adjacent data nodes, the logical dependency strength of continuous business processes is quantified, and intermediate quantities for cross-node continuity verification are obtained, successfully transforming abstract business logic dependencies into a computable mathematical model.

[0064] In a preferred embodiment of the present invention, step 5 above may include: Step 5.1: Based on the cross-node continuity verification intermediate quantities, extract the business indicator state transition trajectory under the continuous time dimension, construct a continuously differentiable temporal evolution function model, and obtain the temporal state analytical expression. Specifically, this includes: loading the generated complete set of cross-node continuity verification intermediate quantities, where each record contains the predecessor node timestamp, successor node timestamp, state transition matrix, logical dependency strength value, time interval, and continuity score. Extract the timestamps of all data nodes that have appeared from these records, arrange these timestamps in ascending order to form a discrete time point sequence, where each time point in the time point sequence corresponds to the position of a data node on the file flow chain; for each business dimension, extract the state value of that dimension at all discrete time points, for the ... For each business dimension, iterate through each time point in the time sequence, find the initial state feature vector of the data node corresponding to that time point, and extract the first... The numerical value of the dimension. If there is no directly corresponding value for a dimension at a certain point in time, for example, if the metadata record corresponding to that point in time does not contain that dimension, and interpolation failed to complete it in step 2.4, then linear interpolation is performed using the nearest known value before and after that point in time. The interpolation formula is the same as in step 2.4, thus obtaining the numerical value of the first dimension. Each business dimension represents a discrete state sequence over the entire time period, i.e., a series of discrete point pairs in the form of timestamps and dimension values. Connecting these discrete points in timestamp order forms the state transition trajectory of the business indicators for that dimension.

[0065] To construct a continuously differentiable time-series evolution function model, for the state transition trajectory of business indicators in each business dimension, a continuous function is needed to fit these discrete points. This function must be continuous and first-order differentiable everywhere in its domain, meaning the function curve is smooth, without discontinuities or sharp angles. This is achieved using piecewise polynomial interpolation. Specifically, the entire time axis is divided into several sub-intervals, each with two adjacent discrete time points as endpoints. Within each sub-interval, a cubic spline interpolation function is used to approximate the state change curve. The construction process of the cubic spline interpolation function is as follows: Assume the entire time axis has... The discrete time points are denoted in ascending order as follows: The corresponding number The state values ​​of each dimension are denoted as follows: For the first A range, from arrive ,in Take from 1 The cubic polynomial function in this interval The form is: ; in These are four coefficients to be determined. These are continuous-time variables. These four coefficients need to satisfy three sets of constraints: the first set is the function value continuity condition, i.e. , The second set is the condition for continuity of the first derivative, which states that the first derivatives of two adjacent intervals are equal at their common endpoints. The third group is the condition for continuity of the second derivative, which states that the second derivatives of two adjacent intervals are equal at their common endpoints. In addition, two boundary conditions are added. Usually, the second derivatives at the two endpoints are set to zero, which are natural boundary conditions. This can uniquely determine the polynomial coefficients of all subintervals.

[0066] When solving for the coefficients, first establish a tridiagonal linear system of equations based on the continuity condition of the second derivative. Let each internal time point... The second derivative value at is For From 2 to For each interior point, the equation is satisfied: ; Combined with boundary conditions and The tridiagonal system of equations was solved using the chasing method, yielding all the results. The second derivative values ​​at each time point arrive Find the four coefficients in reverse order for each interval: , , , .

[0067] After the above fitting process, the first... The time-series state analytical expression for each business dimension is a piecewise defined cubic spline function, given any continuous time variable. The state value of this dimension and its first and second derivatives can be calculated for all... Perform the above operations on each business dimension to obtain the results. A sequence state parsing expression.

[0068] Step 5.2: Based on the analytical expression of the time-series state, perform analytical differentiation operations on the continuous-time variables to calculate the instantaneous rate of change of the business indicators over time, and obtain the instantaneous rate of change distribution curve. Specifically, this includes: for the first... Temporal state parsing expressions for each business dimension (Dimension subscripts are omitted here for brevity) Perform first-order analytic differentiation. Because It is a piecewise cubic polynomial function, and its derivative is... It can be obtained directly by differentiating each term of the polynomial in each subinterval, without the need for numerical difference approximation, thus ensuring computational accuracy. For the Sub-intervals polynomials on : ; The polynomial with respect to time Find the first derivative to obtain the instantaneous rate of change function on this subinterval. : ; This function represents the time interval at any point within the subinterval. Above, the first The instantaneous rate of change of a business indicator. The unit is the standard physical quantity unit of that dimension divided by seconds. The instantaneous rate of change distribution curve is generated by sampling continuous time variables across the entire time axis. The sampling method is: starting from the earliest time point... From the beginning to the latest time point End, with a fixed sampling step size. The time interval is typically set to 60 seconds, meaning a sample is taken every minute, sequentially capturing the sampling time points. For each sampling time point... Determine which subinterval it falls into, and then use the corresponding subinterval's... The formula calculates the instantaneous rate of change at that moment. If the sampling time point happens to fall at the endpoint of a sub-interval, the average value is calculated from both the left and right sub-intervals to eliminate any possible small numerical jumps at the endpoints.

[0069] All sampling time points and their corresponding instantaneous rate of change values ​​are combined into a series. Plotting points on a two-dimensional coordinate plane and connecting them yields the instantaneous rate of change distribution curve for that business dimension. The horizontal axis of the curve represents continuous time, and the vertical axis represents the instantaneous rate of change. For all... Perform the above analytical differentiation and sampling operations on each business dimension respectively to obtain... A curve showing the instantaneous rate of change.

[0070] Step 5.3: Based on the instantaneous rate of change distribution curve, identify the extreme points of sudden increases or decreases in the rate of change, locate the abnormal abrupt change intervals of metadata in the continuous flow process, and obtain the set of time-series abrupt change characteristic intervals. Specifically, this includes: calculating the second derivative of the instantaneous rate of change, i.e., the acceleration of the rate of change, and applying the second derivative of the instantaneous rate of change to the first derivative of the second derivative of the third derivative of the second derivative of the third derivative of the first derivative of the second derivative of the third derivative of the first derivative of the second derivative of the third derivative of the first derivative of the second derivative of the first ... instantaneous rate of change function of dimension Again about time Differentiate. Because It is a quadratic polynomial over the subinterval, and its second derivative is... That is, the original state function The third derivative is a constant over the subinterval: ; However, a more rigorous approach is: ,so This is the acceleration function of the rate of change. acceleration of the rate of change It indicates how quickly the instantaneous rate of change changes with time. When When the absolute value is very large, it indicates that the rate of change is changing rapidly, which may correspond to an abnormal change in the business status.

[0071] instantaneous rate of change The extreme point occurs on its first derivative. The position equal to zero, or it appears at the endpoints of a subinterval (because a piecewise polynomial may not be differentiable to higher orders at the endpoints). For each subinterval, solve the equation. To obtain the possible extreme point locations : ; If the extreme point falls within a sub-interval Within the range, it is a valid extreme point; its position and corresponding value are recorded. value.

[0072] Iterate through all sampling time points and calculate the absolute difference in the instantaneous rate of change between adjacent sampling points. Let the th sampling point be... The instantaneous change rate of each sampling point is , No. The instantaneous change rate of each sampling point is Then the rate jump variable between adjacent sampling points is: ; The rate jump variable is compared with a preset mutation detection threshold. The mutation detection threshold is set based on the historical normal fluctuation range of this business dimension, typically three times the standard deviation of the instantaneous change rate across all sampling points of that dimension. If the rate jump variable is greater than the mutation detection threshold, it is marked as starting from the [previous] [period]. From the sampling point to the... The interval between each sampling point is a mutation candidate interval. Adjacent mutation candidate intervals are merged into a continuous mutation interval. The merging rule is: if the interval between two mutation candidate intervals is less than or equal to 2 sampling steps, i.e. 120 seconds, then they and the interval region in between are merged into the same mutation interval.

[0073] The mutation interval includes the start and end timestamps, the maximum and minimum instantaneous rate of change within the interval, the cumulative value of the rate jump variables within the interval (i.e., the sum of the rate jump variables of all adjacent sampling points within the interval), and a mutation type marker: Sudden increase: the rate of change within the interval generally increases; Sudden decrease: the rate of change within the interval generally decreases; Oscillating: the rate of change within the interval fluctuates violently. The mutation type marker is determined by calculating the difference between the first and last instantaneous rate of change values ​​within the interval. .like If the value is greater than 0.3 times the cumulative value of the rate jump variable, it is marked as a sudden increase; if If the value is less than -0.3 times the cumulative value of the rate jump variable, it is marked as a sudden drop; otherwise, it is marked as an oscillation. After summing all mutation intervals across all business dimensions, a set of time-series mutation feature intervals is obtained. Each element in this set contains: the index of the business dimension to which it belongs, the start timestamp of the mutation interval, the end timestamp of the mutation interval, the mutation type flag, the maximum rate jump variable, and the cumulative value of the rate jump variable.

[0074] Step 5.4: Based on the set of time-series abrupt change characteristic intervals, calculate the average rate of change within each interval and compare it with the preset dynamic drift boundary to quantify the magnitude of deviation from the normal flow pattern and obtain the metadata time-series drift rate analysis result. Specifically, this includes: calculating the average rate of change within each abrupt change interval; for each abrupt change interval in the set of time-series abrupt change characteristic intervals, extracting the instantaneous rate of change values ​​of all sampling points within that interval from the instantaneous rate of change distribution curve; assuming there are a total of [missing information] within the interval. There are 10 sampling points, and the instantaneous rate of change values ​​of each sampling point are as follows: Then the average rate of change in that interval The calculation formula is: ; Simultaneously calculate the standard deviation of the instantaneous rate of change within this interval. This is used to measure the dispersion of the rate of change within an interval. ; Load the pre-defined dynamic drift boundaries. These dynamic drift boundaries are the normal drift tolerance range pre-defined in the structured validation constraint library for each business dimension, and include two thresholds: the upper drift boundary and the lower drift boundary. and drift lower boundary These two boundaries are set based on the statistical distribution of the instantaneous rate of change observed in the historical normal file flow process for this business dimension. Specifically, a 95% confidence interval is taken for the instantaneous rate of change of the historical normal sample, and the upper limit of the confidence interval is the upper boundary of the drift. The lower limit of the confidence interval is the lower boundary of the drift. For dimensions lacking historical samples, business experts directly specify boundary values ​​according to document management standards. For example, the contract amount dimension should not undergo abrupt changes during normal circulation, and its lower boundary of drift is set to... Yuan per second, with the upper drift boundary set to Yuan per second; while the normal drift boundary of the approval process sequence number dimension is set to arrive Every second.

[0075] Perform a quantitative comparison of drift amplitude; for each abrupt change interval, calculate the average rate of change for that interval. Compare with the dynamic drift boundary. The comparison is divided into three cases: Scenario 1: At this point, the average rate of change within the interval exceeds the upper boundary of normal drift, constituting positive over-limit drift. Positive drift exceeding the limit... The calculation formula is: ; Scenario 2: At this point, the average rate of change within the interval is lower than the lower boundary of normal drift, meaning the rate of change is too slow or even reversed, which constitutes negative over-limit drift. Negative drift exceeds the limit. The calculation formula is: ; Scenario 3: If the average rate of change within the interval is within the normal drift tolerance range, it does not constitute a valid drift anomaly. The mutation interval is marked as a non-drift mutation and is removed.

[0076] The fourth step is to calculate the drift rate.

[0077] For abrupt change intervals identified as either positive or negative out-of-limit drift, calculate the metadata time-series drift rate for that interval. The drift rate represents the relative degree to which the time-series changes of business metrics deviate from normal flow patterns within that interval. Drift Rate The calculation formula is: When it is a positive over-limit drift: ; When it is a negative over-limit drift: ; in, The smallest positive number defined in step 4.4 To prevent the denominator from being zero, the drift rate is expressed as a percentage; a higher value indicates a more severe deviation.

[0078] Summarize the metadata time-series drift rate parsing results and generate a parsing result record for each valid drift abrupt change interval. The record includes: the relevant business dimension index and field name label, the start and end timestamps of the abrupt change interval, and the average change rate of the interval. Standard deviation of the rate of change in the interval Dynamic drift upper boundary Dynamic drift lower boundary Drift direction marker, absolute value of drift excess, drift rate The interval contains a list of data nodes and timestamps, and the parsing results records for all dimensions are sorted by drift rate. The results of metadata time-series drift rate parsing are arranged in descending order from largest to smallest.

[0079] In this embodiment of the invention, by extracting the state transition trajectory of business indicators in the continuous time dimension, a continuously differentiable temporal evolution function model is constructed, resulting in an analytical expression for the temporal state. This successfully transforms discrete business state data into an evolutionary model with mathematical continuity, providing a rigorous theoretical foundation and computational premise for subsequent calculus operations. Based on the analytical expression for the temporal state, analytical differentiation operations are performed on the continuous time variables to calculate the instantaneous rate of change of business indicators over time, obtaining the instantaneous rate of change distribution curve. This effectively overcomes the shortcomings of traditional static or discrete verification methods that cannot perceive dynamic trends within small time scales, achieving precise micro-dynamic perception of data flow status.

[0080] Based on the instantaneous rate of change distribution curve, extreme points of sudden increases or decreases in the rate of change are identified, and abnormal mutation intervals in the continuous flow of metadata are located, resulting in a set of temporal mutation feature intervals. This improves the sensitivity to hidden temporal distortions and can accurately pinpoint abnormal mutation time windows that are easily missed by traditional conventional threshold detection. Thus, by calculating the average rate of change in each interval and comparing it with the preset dynamic drift boundary, the magnitude of deviation from the normal flow pattern is quantified, and the metadata temporal drift rate analysis result is obtained, successfully transforming abstract temporal anomalies into measurable and specific numerical indicators.

[0081] In a preferred embodiment of the present invention, step 6 above may include: Step 6.1: Based on the metadata time-series drift rate parsing results, extract the association dimension identifiers of the multi-source heterogeneous archive data and map them to a unified global verification space to obtain a global association dimension mapping set. Specifically, this includes: loading the metadata time-series drift rate parsing results. Each record in this result carries a business dimension index, field name label, drift rate, and the identifier of the data source system to which it belongs. Traverse all parsing result records, extracting the field name label and the identifier of the data source system to which each record belongs, and after deduplication, obtain a cross-data source field name label list. The field name labels in this list come from different systems and have different naming rules. For example, in the OA system, it is called "approval amount," in the financial system, it is called "total payment," and in the imaging system, it is called "invoice face value." Although the names are different in their respective systems, they actually point to the same business concept: transaction amount. Load the pre-set global association dimension dictionary. This dictionary is a pre-built standardized mapping table. Its construction process involves: collecting metadata dictionary documents from all data source systems, inviting archival business experts to review each field, and grouping semantically similar or identical fields into the same global dimension. Each global dimension has a globally unique dimension identifier and a standardized Chinese name. For example, the approval amount from the OA system, the total payment amount from the financial system, and the face value of invoices from the imaging system are uniformly mapped to global dimension number 001, with the standard name "Transaction Amount." Each record in the global relational dimension dictionary includes: global dimension number, standard dimension name, a list of corresponding original field name labels from each source system, unit conversion factors for the field values ​​from each source system, and the weight of this global dimension in multi-source cross-validation.

[0082] Perform dimension mapping operations, iterating through each field name tag in the cross-data source field name tag list and searching for the corresponding global dimension number in the global associated dimension dictionary. If an exact match exists in the dictionary, a one-to-one mapping relationship is established between the field name tag and the global dimension number. If no exact match is found, the text similarity between the field name tag and the original field name tags of each entry in the dictionary is calculated, and the entry with the highest similarity exceeding a preset threshold (usually set to 0.85) is taken as the matching result. If the highest similarity is still below the threshold, a new temporary global dimension number is assigned to the field name tag and marked as awaiting manual confirmation. For each successfully mapped field name tag, the corresponding unit conversion factor is extracted synchronously. The unit conversion factor is used to unify the values ​​of different source systems to a standard unit. For example, if the approval amount in the OA system is in yuan and the total payment amount in the financial system is in fen, then the unit conversion factor for this field in the financial system is... That is, multiplying the numerical value in cents by It is equal to the value in yuan. The original value is multiplied by the unit conversion factor to obtain the standardized value, which ensures the consistency of dimensions when comparing across sources.

[0083] Summarize all mapping relationships to generate a global relational dimension mapping set. This mapping set is organized with the global dimension number as the primary key. Each global dimension contains: the standard dimension name, a list of all source systems participating in this global dimension, the original field name label corresponding to each source system, the unit conversion factor for each source system, and the participation weight of this global dimension in multi-source cross-validation.

[0084] Step 6.2: Based on the global correlation dimension mapping set, perform cross-data source cumulative integration calculations on the continuous variation intervals in the time series drift rate analysis results, and accumulate the total deviation under the continuous time dimension to obtain the total cumulative deviation of multi-source data. Specifically, this includes: reorganizing the drift data according to the global dimension; using the global correlation dimension mapping set, mapping each record in the metadata time series drift rate analysis results to the corresponding global dimension number; for multiple records belonging to the same global dimension, merging them according to the start timestamp of the drift interval; since the time boundaries of the drift intervals of different source systems may not be completely aligned, it is necessary to first perform standardized segmentation processing on the time intervals. The standardized segmentation rule is: take the minimum value of the start timestamps of all drift intervals under this global dimension as the unified start time, take the maximum value of the end timestamps of all drift intervals as the unified end time, and then divide this entire continuous time segment into several time micro-elements of equal length according to a fixed step size.

[0085] For each time element, perform the instantaneous value assignment of the deviation; for the th The first global dimension Each time element has a time range of 1 / 2. ,in The initial moment of the infinitesimal element. Sampling step size Within each infinitesimal second, check whether each source system associated with the global dimension has a valid drift mutation interval covered within that infinitesimal second. If a source system reported a drift mutation interval covering the infinitesimal second in step 5.4, then extract the drift rate of the source system within that interval. and the average rate of change of the interval The instantaneous deviation contribution value of the source system within this infinitesimal time element. ( The source system number is defined as follows: ; in, This represents the average rate of change of the source system within this drift interval. The drift rate of the source system in this interval (expressed as a decimal, i.e., a percentage divided by). ), The participation weights assigned to the source system in the global correlation dimension mapping set. If the source system does not cover any drift mutation intervals within this time infinitesimal, then... The value is zero.

[0086] Summing across data sources within the same global dimension, for the ... The first global dimension Each time element, associated with all of the global dimensions. The instantaneous deviation contributions of each source system are added together to obtain the multi-source instantaneous deviation of the time element. : ; This summation operation aggregates the deviations from different data sources at the same time microelement, reflecting the degree of deviation that multiple data sources point to at that moment.

[0087] Perform cumulative integration along the continuous time dimension, and then... All under a global dimension The sum of multi-source instantaneous biases in each time element is accumulated and integrated along the time axis. Since the time elements are discrete samples with equal intervals, the accumulated integral can be approximated by summing the sum of the multi-source instantaneous biases multiplied by the time element step size. The total cumulative deviation of multi-source data across all global dimensions throughout the entire analysis period. The calculation formula is: ; When unfolded, it is: ; in, For source system In the The average rate of change of each time element corresponding to the drift interval This represents the corresponding drift rate. Sampling step size The physical meaning of this cumulative integration operation is: to integrate the area of ​​the deviation across data sources over a continuous time dimension, similar to calculating the total area under the deviation curve. Integration result. The larger the value, the more severe the accumulated time-series bias of that global dimension in the multi-source data. Performing the above four steps on all global dimensions yields a set of the total accumulated bias from the multi-source data. Each record in the set contains: global dimension number, standard dimension name, start and end timestamps of the analysis period, total number of time micro-elements, and total accumulated bias from the multi-source data. .

[0088] Step 6.3: Based on the total accumulated deviation of the multi-source data, and combined with the pre-set logical association constraints of archival business, perform cross-validation and weight aggregation between heterogeneous data to obtain the global association consistency deviation index. Specifically, this includes: loading the pre-set logical association constraint rule library for archival business. This rule library defines the logical association relationships between different global dimensions in the archival business process. Each rule contains the following elements: association constraint number, a list of antecedent global dimension numbers indicating the dimensions involved in the preconditions, a consequent global dimension number indicating the target dimension that should be satisfied under the preconditions, logical association type, association constraint strength coefficient, and the mathematical expression or inequality of the constraint. The logical association types are divided into the following categories: equality association, proportional association, and temporal causal association. The association constraint strength coefficient... It is a value in arrive The real value between the contract amount and the payment amount is assessed by business experts based on the importance of the constraint within the archival quality assurance system. For example, the equivalence constraint strength coefficient between the contract amount and the payment amount is set to... The most stringent constraint is set to a coefficient of strength for the ratio between the number of approval days and the number of scanned pages. Relatively lenient.

[0089] Construct a cross-dimensional deviation correlation matrix, assuming the total number of global dimensions is . , build a OK Cross-dimensional deviation correlation matrix of columns Elements of the matrix Indicates the first The total cumulative deviation of the first global dimension and the first To determine whether there are logical constraints among the cumulative deviations of each global dimension, and the strength of these constraints, for each constraint rule in the rule base, extract its set of antecedent global dimension numbers and consequent global dimension numbers. For each global dimension number in the antecedent set... Number it and the global dimension of the consequent. Forming associative pairs in matrix elements Enter the associated constraint strength coefficient of this rule in the middle. .like arrive If multiple rules exist, then take... The maximum value. If no rules exist, then Set it to zero. This matrix is ​​an asymmetric matrix, i.e. and They can be different.

[0090] Cross-validation is performed for each non-zero element in the matrix. (i.e., the first) Victory (Due to the existence of logical association constraints in dimension), calculate the first... peacekeeping The degree of deviation in terms of the cumulative total of actual deviations. Degree of deviation The calculation formula is: When the logical association type is an equality association: ; When the logical association type is proportional association, let the preset proportional range be... The actual ratio is (when (When), if the actual proportion falls within the proportion range, then Take zero, otherwise: ; When the logical association type is a sequential causal association, check and The sign of the positive and negative signs: If both signs are the same (both positive or both negative), then Take zero; if the signs are opposite: ; in, To prevent the denominator from being zero.

[0091] Weighted aggregation calculates the global correlation consistency deviation index, which represents the degree of deviation corresponding to all non-zero elements in the matrix. By performing a weighted summation and then normalizing by dividing by the sum of all correlation constraint strength coefficients, the global correlation consistency deviation index is obtained. : ; The global correlation consistency deviation index is equal to the sum of all cross-dimensional deviations divided by the sum of all correlation constraint strength coefficients. The index typically ranges from [value missing in original text]. Up to positive infinity, The closer it is to zero, the more consistent the distribution pattern of the deviation in the multi-source heterogeneous data is with the constraints of the archival business logic, that is, there are no contradictions between the deviations. The larger the value, the more it indicates that the cross-dimensional deviation distribution deviates significantly from the expected business logic.

[0092] Step 6.4: Based on the global correlation consistency deviation index, perform total threshold determination and consistency level classification to quantify the overall matching degree of multi-source archival data and obtain the total global correlation consistency assessment result. Specifically, this includes: determining the global correlation consistency deviation index... Execute threshold determination, with three preset thresholds, the first threshold... Usually set to Second threshold Usually set to Third threshold Usually set to These three thresholds will The range of values ​​is divided into four intervals: when When the global correlation consistency level is set to Level 1, it indicates that the multi-source data is highly consistent and the deviation distribution fully meets the business logic correlation expectations; when When the global consistency level is set to level two, it means that the multi-source data is basically consistent, the deviation distribution basically meets the business logic association expectations, but there are slight deviations in individual dimensions; when When the global consistency level is set to level three, it indicates that there is significant inconsistency between multi-source data, and the deviation distribution contradicts the expected business logic, requiring close attention; when At this time, the global consistency level is set to level four, indicating that the multi-source data is seriously inconsistent, the deviation distribution is seriously deviating from the expected business logic, and triggers a warning of major defects in the archive quality.

[0093] Total cumulative bias of multi-source data across all global dimensions Perform a summary assessment, and all The total global bias is obtained by summing the absolute values ​​of the cumulative total biases from multiple sources across all global dimensions. : ; Simultaneously, the number of global dimensions whose cumulative absolute value of deviation exceeds a preset dimension-level threshold is counted. Dimension-level thresholds are set based on the business importance and historical normal fluctuation range of each global dimension. For example, the dimension-level threshold for the contract amount dimension is set to 10000, meaning that a cumulative deviation not exceeding 10000 is considered normal. This threshold is determined based on the 95th percentile of the historical normal sample cumulative deviation for this dimension. Let the dimension-level thresholds be respectively... arrive Then the number of dimensions exceeding the limit The calculation method is: traversal from arrive ,like If the count is increased by one, then the count is incremented by one.

[0094] Calculate the overall matching score of multi-source archival data. It is a 100-point score that comprehensively considers the global correlation consistency deviation index. and total global deviation The calculation formula is: ; in This is the normalized global association consistency deviation index, and the normalization method is as follows: ; Guaranteed The range of values ​​is within arrive between, smaller The closer to zero, The bigger The closer it gets to one.

[0095] The total global bias is the normalized value. The normalization method is as follows: ; in, The reference normalization cardinality is the sum of the thresholds for all global dimensions, i.e. , and These are the weighting coefficients. .generally Pick , Pick This indicates that the global correlation consistency deviation index contributes slightly more to the overall matching degree than the total global deviation. The range is theoretically arrive A higher score indicates a better overall match between the multi-source archival data. In practical applications, it is usually based on... The passing score is set at 100%.

[0096] Summarize the overall global consistency assessment results and generate the final assessment report, which includes the following: The overall assessment consists of a global association consistency level of one to four and an overall matching score. Global Association Consistency Deviation Index Total global deviation Count of dimensions exceeding the standard With total dimensions The ratio; the detailed section by dimension is the cumulative total of deviations from multi-source data. Sort in descending order from largest to smallest, and list the standard dimension name and total accumulated bias for each global dimension. Dimensional threshold Whether it exceeds the standard, the number of source systems involved in this dimension, and the maximum degree of deviation involved in this dimension. The evaluation conclusion and recommendations section automatically generates corresponding archive quality evaluation conclusions based on the global correlation consistency level and overall matching degree score. For example, Level 1 corresponds to excellent data quality and it is recommended to archive normally; Level 4 corresponds to serious data quality defects and it is recommended to postpone archiving and start a full-chain data verification. The overall global correlation consistency evaluation result is the final output of the entire metadata quality management method.

[0097] In this embodiment of the invention, by extracting the association dimension identifiers of multi-source heterogeneous archive data and mapping them to a unified global verification space, a global association dimension mapping set is obtained. This successfully breaks down data silos between different business systems, achieving dimensional alignment and unified mapping of heterogeneous data from a global perspective, and laying a solid structural foundation for cross-source evaluation. For the continuous variation intervals in the time-series drift rate analysis results, cumulative integration across data sources is performed, accumulating the total deviation in the continuous time dimension to obtain the cumulative total deviation of multi-source data. This effectively captures and quantifies the cumulative effect of small nonlinear drifts in the continuous time dimension, overcoming the defect of traditional single-point discrete verification that easily overlooks the logical inversion caused by the accumulation of long-term small deviations.

[0098] By combining pre-defined constraints related to archival business logic, cross-validation and weight aggregation are performed between heterogeneous data to obtain a global correlation consistency deviation index. This transforms the purely mathematical deviation into an evaluation indicator that fits the actual business scenario, improving the business interpretability and logical rationality of the evaluation results and avoiding blind calculations divorced from business reality. By performing total threshold determination and consistency level classification, the overall matching degree of multi-source archival data is quantified, resulting in a global correlation consistency total evaluation result, which intuitively and scientifically defines the overall health status of multi-source archival data.

[0099] In a preferred embodiment of the present invention, step 7 above may include: Step 7.1: Based on the overall global correlation consistency assessment result, extract abnormal data nodes with excessive consistency deviation index, identify nonlinear distortion features in the data flow process, and obtain a nonlinear distortion status identifier set. Specifically, this includes: loading the output overall global correlation consistency assessment result, reading the specific values ​​of the global correlation consistency level and the global correlation consistency deviation index, and simultaneously loading the constructed cross-dimensional deviation correlation matrix and the degree of deviation of each dimension pair. Details. Based on the degree of deviation. The numerical value will correlate all dimensions according to the pair. Sort in descending order from largest to smallest, and extract... All dimension-related pairs exceeding a preset divergence threshold. The divergence threshold is typically set to all non-zero pairs. The value is 1.5 times the median. These extracted dimensional association pairs are the main contributors to the excessive global association consistency deviation index, and are called the high deviation association pair set. For each dimensional association pair in the high deviation association pair set, we trace back to the calculation process of the cumulative total of multi-source data deviation for each of the two global dimensions in step 6.2. In the time micro-level data of step 6.2, we locate the cumulative total of multi-source data deviation. and The time interval with the greatest contribution. The specific location method is as follows: for the global dimension... All of them The time micro-elements are sorted in descending order of the absolute value of the sum of instantaneous deviations from multiple sources. The top 10% of time micro-elements with the largest absolute values ​​are selected, and their timestamp ranges are recorded. For the global dimension... Perform the same operation. The union of the timestamp ranges of the two high-contribution time micro-elements is the anomalous mutation time interval.

[0100] From the metadata time-series drift rate parsing results, extract all drift records falling within the abnormal mutation time interval. Each drift record includes: the business dimension index to which it belongs, the source system identifier, the start and end timestamps of the mutation interval, the drift rate, and the drift direction marker. These drift records are merged according to their original data nodes. Since the input of step 5 comes from the intermediate quantity of cross-node continuity verification in step 4, and step 4 comes from the local compliance judgment result of step 3, we can trace back along the processing chain to finally locate those original data nodes that were judged to be non-compliant in step 3 and participated in the time-series correlation analysis. These nodes are the abnormal data nodes with excessive consistency deviation index.

[0101] For each anomalous data node, nonlinear distortion feature identification is performed. Nonlinear distortion means that during the archive transfer process, the state changes of the data node no longer follow the smooth, continuous pattern described by the cubic spline function established in step 5.1, but instead exhibit drastic jumps or inflections deviating from the spline curve. Nonlinear distortion feature identification is quantified using the following three indicators: The first metric is the degree of local curvature anomaly. In the constructed cubic spline function, the second derivative over each subinterval represents the degree of curvature of the curve. For the time points where the outlier data nodes occur... Calculate the second derivative of the spline function at that point. Simultaneously calculate the median of the second derivative at each of the two adjacent time points before and after this node. Local curvature anomaly The calculation formula is: ; in, for .like If the value exceeds the preset curvature anomaly threshold (usually set to 2.0), then the node is marked as having curvature distortion.

[0102] The second metric is the state transition amplitude ratio. This is calculated by dividing the absolute value of the state difference between the anomalous data node and its predecessor by the standard deviation of the state difference over the entire time series. Let the anomalous data node be at the nth... The value of dimension is Its predecessor node is at the The value of dimension is , No. The standard deviation of the state differences between all adjacent nodes is The amplitude of the state transition is greater than The calculation formula is: ; like If the jump amplitude exceeds the preset threshold (usually set to 3.0, i.e., the three-standard-deviation principle), then the node is marked as having jump distortion.

[0103] The third indicator is the drift direction reversal. Check if the drift direction marker of the drift interval containing the abnormal data node is consistent with the drift direction marker of the interval containing its predecessor node. If the predecessor interval is drifting positively while the current interval is drifting negatively, or vice versa, then the node is marked as having a direction reversal distortion.

[0104] If any one of the above three indicators exceeds the standard, the abnormal data node is determined to have nonlinear distortion characteristics. A nonlinear distortion status identifier is generated for each abnormal data node exhibiting nonlinear distortion characteristics. This identifier includes: a globally unique identifier for the abnormal data node, its source system identifier, its global dimension number and standard dimension name, the node's generation timestamp, and the local curvature anomaly degree. The numerical value and the amplitude of state transitions are compared to The numerical value, drift direction reversal degree marker, and distortion type label.

[0105] Step 7.2: Based on the nonlinear distortion state identifier set, construct a continuous differential equation model describing the evolution of data state over time during business flow. Set initial compliance states and boundary constraints to obtain the nonlinear evolution equation expression. Specifically, this includes: determining the order and basic form of the differential equation. For the data state evolution in archival business flow, a first-order ordinary differential equation is used as the basic model framework because the rate of change of most business indicators depends on the difference between the current state and the target state, which conforms to the characteristics of first-order dynamics. The basic form of the first-order ordinary differential equation is: The first derivative of the state function with respect to time is equal to the state evolution rate function. Right now: ; in, Let be the state function to be solved. Let be the state evolution rate function, which describes the state evolution rate at the current state. and the current moment Below, the speed and direction of state changes.

[0106] Determine the state evolution rate function The specific form of the state evolution rate function is as follows: it consists of two superimposed parts: a normal flow driving term and a nonlinear distortion disturbance term. This describes the natural progression of business metrics as records circulate under ideal compliance conditions. For most dimensions of record-related business, the normal circulation drivers are represented by the differential form of the logistic growth model: ; in, The intrinsic growth rate represents the rate of growth of a state when resources are not limited. This represents the upper limit of environmental capacity, indicating the natural saturation value of this business dimension during the document circulation process. and The value of is obtained by fitting the logistic curve to the state change trajectory of this dimension on normal historical samples. Specifically, the fitting method involves collecting (timestamp, state value) point pairs for this dimension across all normal data nodes and solving for the logistic function using the least squares method. Parameters in , and .

[0107] Nonlinear distortion perturbation term This describes the deviation effect of factors such as data acquisition errors, system clock skew, and human error on the normal flow pattern. This disturbance term is directly related to the nonlinear distortion characteristic index of abnormal data nodes, and its form is a decaying Gaussian impulse function. ; in, This is the central moment when the distortion occurs. For the disturbance amplitude, This represents the time span of the disturbance's effect. Disturbance amplitude. The calculation method is as follows: the local curvature anomaly degree of the abnormal data node in step 7.1 is calculated. State transition amplitude ratio The weighted sum is calculated using the drift direction reversal flag (1 for yes, 0 for no): ; Among them, the weighting coefficient , , Based on experience, values ​​of 0.4, 0.35, and 0.25 are used respectively, and the sum of these three values ​​is 1, representing the time width of the disturbance's effect. It is proportional to the length of the drift mutation interval where the abnormal data node is located: ; in The duration of this drift mutation interval (end timestamp minus start timestamp).

[0108] Superimposing the normal flow driving term and the nonlinear distortion disturbance term yields the complete state evolution rate function: ; Set the initial compliance state and boundary constraints. Initial compliance state The value is the status value of the predecessor node of the abnormal data node in the corresponding business dimension. The predecessor node is assumed to be in a compliant state unaffected by nonlinear distortion.

[0109] The boundary constraints are divided into left and right ends. The left boundary condition is the initial value condition: ; in, This is the generation timestamp for the preceding compliant node.

[0110] The right-hand boundary condition is a final value constraint, which takes the state value of the nearest successor node to the abnormal data node that has been determined to be compliant. If all nodes following the abnormal data node are in an abnormal state (i.e., no compliant successor node has been found), then the final value constraint is set to the theoretical target value of this global dimension in the current flow stage. This theoretical target value is obtained by querying the pre-set archive business flow status baseline table.

[0111] Assemble the nonlinear evolution equation expression by combining the above state evolution rate function, initial conditions, and boundary constraints to obtain the complete nonlinear evolution equation expression for this anomalous data node: ; in, For each anomalous data node in the nonlinear distortion state identifier set, the above four steps are used to generate its own nonlinear evolution equation expression, which is the timestamp corresponding to the right-hand boundary condition.

[0112] Step 7.3: Based on the expression of the nonlinear evolution equation, perform a one-time analytical solution operation for the abnormal data nodes to calculate the continuous change path of the data state from the distorted point to the compliant state, and obtain the initial smooth transition trajectory set. Specifically, this includes: determining the type of the nonlinear evolution equation and selecting the corresponding solution strategy. The differential equation constructed in step 7.2 is: ; This is a first-order nonhomogeneous nonlinear ordinary differential equation. The homogeneous part (i.e., the normal flow driving term) The equation is in Bernoulli form, and the non-homogeneous part is a time-dependent equation. The known explicit function. Since the perturbation term is time... explicit functions rather than states The equation can be solved using the method of variation of constants combined with the Bernoulli equation solution formula. The overall solution approach is as follows: first, find the general solution of the homogeneous equation; then, construct a particular solution of the non-homogeneous equation using the method of variation of constants or the Green's function method; finally, determine the integration constant using the initial conditions.

[0113] Solving the homogeneous equation is as follows: ; This is a logistic differential equation. Rewriting it in standard Bernoulli form and solving it yields the solution. The general solution of the homogeneous equation is obtained: ; in, , where is an undetermined constant.

[0114] Using the Green's function method, a particular solution to the nonhomogeneous equation can be constructed. The nonhomogeneous equation can be written as: ; Due to nonhomogeneous terms Only at the distortion center moment Since there are non-zero values ​​nearby, the disturbance can be considered as a local impulse. Under this approximation, the impulse response method is used to solve the problem. First, consider the time at time... Apply a strength of The effect of a transient pulse on the system state. For homogeneous systems. At any moment Apply small perturbation Then, the system will evolve along the trajectory of the homogeneous solution, and the non-homogeneous particular solution. It can be expressed using convolution integrals. However, since the original equation is nonlinear, the principle of linear superposition cannot be directly applied. Therefore, an approximate linearization approach is adopted: before and after the distortion occurs, the state values... Distance from environmental capacity limit There is usually still a lot of space at this time. The homogeneous equation can be approximated as 1. This degenerates into a linear equation. Under this approximation, the equation becomes: ; This is a first-order linear nonhomogeneous ordinary differential equation, which can be solved analytically using the method of integrating factors. Integrating factors Multiply both sides of the equation by the integrating factor: Right now: ; From both sides arrive integral: ; Expand the Gaussian impulse convolution integral on the right. Since the integrand is the product of a Gaussian function and an exponential function, this integral does not have a closed-form solution in elementary function form, but it can be expressed using an error function. Let: ; Combine exponential terms through the formula: ; make Then the exponential part is reduced to: ; The second term is related to the integral variable. Irrelevant constant factors are denoted as : ; then .

[0115] This integral is the cumulative distribution form of the Gaussian function, which can be expressed using the error function. Express: ; Therefore, the complete expression for the particular solution is: ; Substitution By refining and simplifying the expression, we obtain the particular solution under the linear approximation: ; The linear approximation solution is modified by embedding it into a nonlinear framework.

[0116] The above linear approximation is in much smaller It was established at that time, but when near At this point, the logistic saturation effect cannot be ignored. Therefore, an iterative correction method is adopted: the linear approximation solution... Substituting into the right-hand side of the original nonlinear equation Among the factors, calculate the adjusted effective growth rate: ; use Replace the constants in the original linear equation The integral factor method is then re-executed. After one correction, the state function is obtained. This is the initial smooth transition trajectory.

[0117] Perform the above four-step solution operation on all abnormal data nodes to obtain the initial set of smooth transition trajectories. Each trajectory is defined by a continuous function. The formal expression, the domain is from arrive The complete time interval, and the value range is the natural range of values ​​for this business dimension under compliant conditions.

[0118] Step 7.4: Based on the initial smooth transition trajectory set, perform trajectory continuity verification and discrete noise point filtering operations to restore the true business data evolution path obscured by nonlinear distortion, and obtain the data state smooth transition trajectory solution. Specifically, this includes: trajectory continuity verification, for the initial smooth transition trajectory... In its domain With uniform step size Dense sampling was performed to obtain a series of sampling points. ,in This is the sampling point number. A continuity check is performed on adjacent sampling points: Calculate the state change between adjacent sampling points .

[0119] Calculate the rate of state change between adjacent sampling points .

[0120] examine Whether it is within a physically reasonable range, the upper limit of which is the maximum instantaneous rate of change that business dimension may occur in real business. For example, in the dimension of contract amount, under any legitimate business operation, it is impossible for the amount to suddenly change by more than 50% of the total contract amount within one second. The upper limit of the physically reasonable range. The setting is based on twice the maximum rate of change of adjacent data nodes in the historical normal samples of this dimension.

[0121] like The absolute value is greater than If the sampling interval is marked as a continuity breakpoint, then the sampling interval is marked as a continuity breakpoint.

[0122] The sampling intervals near the continuity breakpoints are collected to analyze the break mode. The break modes are divided into two categories: isolated spike type and persistent drift type. Isolated spike type breakage is usually caused by the numerical residue at the tail of the Gaussian pulse in step 7.3, belonging to the solution artifact; persistent drift type breakage usually indicates that the parameters of the nonlinear evolution equation in step 7.2 ( , or There is a deviation.

[0123] Discrete noise point filtering. Different filtering strategies are adopted based on the fracture mode classification results. For isolated spike-type fracture points, a three-point moving median filter is used. Let the sampling time point corresponding to the fracture point be... There is one normal sampling point before and after it. and .Will Replace with and Arithmetic mean: ; Then, for the filtered point and the two sampling points before and after it (a total of five points), a least-squares fitting smoothing is performed using a local cubic polynomial. The local fitting polynomial is: ; With five sampling points ( arrive The data fits four coefficients. arrive The fitting objective is to minimize the sum of squared residuals at the five points. Then, using... exist Replace the original value at the location For persistent drift-type breakpoints, it is necessary to adjust the parameters of the nonlinear evolution equation. Specifically, the adjustment method is to use the actual sampling point data within the persistent drift interval as a new fitting constraint to reduce the parameters. The search space is determined, and the logistic curve fitting in step 7.2 is re-executed. The solution in step 7.3 is then re-executed using the adjusted parameters to obtain the corrected trajectory.

[0124] Endpoint smoothing is implemented, with the initial smooth transition trajectory at the left endpoint. Requires the actual state value of the predecessor compliance node Precise match, at the right end point Requires comparison with the actual state values ​​of subsequent compliance nodes. Precise matching. Due to the use of linear approximation and iterative correction in step 7.3, there may be slight numerical deviations at the trajectory endpoints. The specific operation for endpoint smoothing is as follows: Near the left endpoint ( arrive The interval is used to apply a decay correction term to the trajectory, ensuring that the trajectory passes strictly through the point. The correction item is in the form of: ; in, Pick One-fifth.

[0125] Near the right endpoint, a similar attenuation correction is applied: ; Verify the physical consistency of the corrected trajectory. For the corrected trajectory... Perform continuous re-examination across the entire range: again using step size Sampling to confirm the rate of change of all adjacent sampling points. The absolute values ​​of all of them do not exceed Simultaneously, the monotonicity of the trajectory is verified to ensure it conforms to the expected monotonic direction for this business dimension. For local intervals that do not meet the monotonicity constraint, an order-preserving smoothing algorithm is used for forced adjustment. After the above continuity verification and noise filtering, the obtained data state is the smooth transition trajectory solution for the abnormal data node. This trajectory solution fully describes the continuous change path of the data state from the compliant state before distortion, through distortion disturbance, and then smoothly transitioning to the compliant state after distortion. For each abnormal data node in the nonlinear distortion state identifier set, a smooth transition trajectory solution is obtained, which, when summarized, constitutes the data state smooth transition trajectory solution set.

[0126] In this embodiment of the invention, by extracting abnormal data nodes with excessive consistency deviation index and identifying nonlinear distortion characteristics in the data flow process, a set of nonlinear distortion state identifiers is obtained. This accurately identifies core abnormal nodes, removes interference from normal data, and provides precise targets for subsequent targeted modeling. Based on the set of nonlinear distortion state identifiers, a continuous differential equation model describing the evolution of data state over time is constructed. Initial compliance states and boundary constraints are set to obtain the nonlinear evolution equation expression. This successfully transforms the complex business data distortion process into a rigorous mathematical continuous evolution model, ensuring that state deduction has a solid theoretical foundation and clear boundary constraints.

[0127] Based on the nonlinear evolution equation expression, a one-time analytical solution operation is performed on the abnormal data nodes to calculate the continuous change path of the data state from the distortion point to the compliant state, obtaining an initial smooth transition trajectory set. This scientifically deduces a reasonable evolution path for the data to regress from abnormal to compliant, avoiding secondary logical contradictions caused by abrupt data truncation or crude correction. By performing trajectory continuity verification and discrete noise point filtering operations, the true business data evolution path obscured by nonlinear distortion is restored, yielding a smooth transition trajectory solution for the data state. This effectively filters out random noise interference and accurately restores the true evolution logic of the business data.

[0128] In a preferred embodiment of the present invention, step 8 above may include: Step 8.1: Based on the data state smooth transition trajectory solution, extract the values ​​of continuous state nodes in the trajectory and compare them point-by-point with the preset compliance boundary conditions to obtain the boundary deviation state record. Specifically, this includes loading the output data state smooth transition trajectory solution set. This set contains the smooth transition trajectory function generated for each abnormal data node. The domain is defined as the timestamp of the predecessor compliant node. Timestamp of subsequent compliance nodes The complete continuous time interval is loaded. Simultaneously, the nonlinear distortion status identifier corresponding to the abnormal data node is loaded, and its global dimension number and standard dimension name are obtained; the pre-defined compliance boundary conditions corresponding to this global dimension are loaded. The compliance boundary conditions are pre-defined in the structured validation constraint library, defining the legal range in which the value of this business dimension should fall at each stage of the document flow. The compliance boundary conditions include three elements: the compliance lower limit value... Compliance upper limit The compliance lower and upper limits are defined by the same source as the dynamic drift boundary in step 5.4, but the focus here is on the absolute value range rather than the rate of change range. The specific form of the compliance boundary differs for different types of business dimensions: the compliance lower limit for the contract amount dimension is zero, and the compliance upper limit is 120 / 5 of the total contract amount; the compliance lower limit for the approval process sequence number dimension is one, and the compliance upper limit is the total number of steps defined in the approval process.

[0129] Perform dense sampling of the trajectory for smooth transition trajectory functions In its domain The above uses fine sampling step size Perform uniform sampling. Fine sampling step size. Typically, a 1-second sampling time is used, much smaller than the 60-second sampling step in step 5.2, to ensure that any brief out-of-bounds states on the trajectory can be captured. The specific sampling operation is as follows: starting from the initial time... Beginning, with Sampling time points are generated incrementally until the end time is reached or slightly exceeded. Assume co-generation The sampling time points are denoted as follows: , , until For each sampling time point ,Will Substitute into the smooth transition trajectory function Calculate the trajectory state value at that moment: ; This yields a continuous sequence of state node values ​​for the trajectory, where each element is a pair of points.

[0130] Perform point-by-point numerical comparison; for each sampling point in the continuous sequence of state node values, its state value is... The compliance minimum value of this dimension at this stage of the process. and compliance cap The comparisons are performed sequentially. The comparison and judgment rules are as follows: like and Then the sampling point is marked as compliant; if If the sampling point exceeds the lower limit, then the deviation from the lower limit is calculated: ; like If the sampling point exceeds the upper limit, then the deviation from the upper limit is calculated: ; For each sampling point marked as exceeding the lower or upper limit, a boundary deviation state record is generated. Each record contains the following fields: a globally unique identifier for the anomalous data node, its global dimension number and standard dimension name, sampling timestamp, and trajectory state value. Compliance minimum value Compliance upper limit Cross-boundary direction marker, absolute value of deviation, and relative proportion of deviation.

[0131] The relative deviation ratio is used to measure the relative severity of the boundary breach, and its calculation formula is as follows: When it exceeds the lower limit: ; When the limit is exceeded: ; in .

[0132] The above sampling and comparison operations are performed on the smooth transition trajectory solutions of all abnormal data nodes, and all boundary deviation state records are summarized.

[0133] Step 8.2: Based on the boundary deviation state records, filter out continuous state nodes whose values ​​exceed the compliance upper or lower limit, mark the corresponding archival business process and timestamp, and obtain a set of out-of-bounds state nodes. Specifically, this includes: extracting all out-of-bounds sampling points from the boundary deviation state records, traversing the generated complete set of boundary deviation state records, and filtering out all records whose out-of-bounds direction is marked as exceeding the lower or upper limit. The sampling points corresponding to these records are the out-of-bounds state nodes. Sort the filtered records in ascending order according to their respective abnormal data node numbers and sampling timestamps to form a sequence of out-of-bounds state nodes grouped by node and arranged by time.

[0134] For each out-of-bounds status node, its sampling timestamp determines which stage of the archive circulation lifecycle it falls within. The archive circulation lifecycle has been standardized and defined in step 1 using a business stage template library; each stage has a clear start and end time boundary. The pre-defined business stage time boundary table is traversed to find the stage to which the sampling timestamp belongs. The unique ID, stage name, and sequence number of that stage in the circulation chain are then entered into the out-of-bounds status node's record. If the sampling timestamp crosses the boundaries of two adjacent business stages, the stage with the longer time span is taken as the belonging stage.

[0135] The cumulative out-of-bounds degree of the calculated nodes is determined by recording the absolute value of the deviation for each out-of-bounds node, as well as calculating the cumulative impact of that node on the entire abnormal interval. For a group of out-of-bounds nodes belonging to the same abnormal data node and occurring in consecutive time, the deviation is accumulated sequentially from the first out-of-bounds node to the last. Let there be a total of [number missing] out-of-bounds nodes in this group. There are 1, and the absolute values ​​of the deviations at each node are as follows: , until Then the first in that group The cumulative forward deviation of each node is: ; Simultaneously calculate the cumulative deviation percentage of this node in the entire group: ; Mark the core out-of-bounds features. For each out-of-bounds state node, add three auxiliary markers based on its relative deviation ratio and location.

[0136] The peak value is marked as the absolute value of the deviation of a node being greater than that of its preceding and following nodes. A node is marked as an out-of-bounds peak node if the absolute value of its deviation is greater than twice the median of the absolute values ​​of the deviations of all out-of-bounds nodes belonging to its abnormal data nodes. Boundary crossing is marked as follows: if the preceding sampling point of a node is compliant, the node is marked as the boundary crossing start node; if the following sampling point of a node is compliant, the node is marked as the boundary crossing end node. Duration is marked by counting the number of consecutive out-of-bounds sampling points in an increasing time direction starting from the boundary crossing start node, multiplied by the sampling step size. This gives the duration of the continuous boundary crossing segment.

[0137] Assemble the out-of-bounds state node set. All out-of-bounds state nodes processed above are aggregated and organized into a set using the abnormal data node number as the primary key and the sampling timestamp as the secondary sort key. Each element in the set includes: a globally unique identifier for the abnormal data node, its global dimension number and standard dimension name, its business process number and name, sampling timestamp, trajectory status value, out-of-bounds direction marker, absolute value of deviation, relative deviation ratio, forward cumulative deviation, cumulative deviation percentage, peak value marker, boundary crossing marker, and duration.

[0138] Step 8.3: Based on the set of out-of-bounds state nodes, perform a continuous block merging operation on out-of-bounds state nodes with adjacent timestamps to obtain aggregated out-of-bounds blocks. Specifically, this includes: initializing the basic parameters of the aggregation operation and defining the merging criteria between two out-of-bounds state nodes. The merging criteria include three parallel conditions: The temporal continuity condition is that two out-of-bounds state nodes are adjacent in their respective out-of-bounds state node sequences after being sorted by sampling timestamps, and the difference between the sampling timestamp of the latter node and the sampling timestamp of the former node is less than or equal to the merging time interval threshold. Merging time interval threshold The setting needs to consider the actual workflow of document processing, and is typically set to 60 seconds. If the time interval between two boundary-crossing nodes exceeds... This indicates that a compliant state interval separates the two and they should not be merged; the consistency condition for the outbound direction is that the outbound direction markers of the two outbound state nodes are the same, that is, both exceeding the upper limit or both exceeding the lower limit. Outbound events with different directions reflect different types of distortions and need to be aggregated separately; the business process continuity condition is that the difference in the sequence number of the business process to which the two outbound state nodes belong in the file transfer chain does not exceed 1, that is, they belong to the same business process or adjacent business processes. Outbound events that span multiple business processes need to remain independent in order to accurately locate the problematic process.

[0139] Perform continuous block merging based on time windows. For each out-of-bounds state node sequence under an abnormal data node, scan sequentially according to the sampling timestamp from smallest to largest, and group adjacent nodes that meet the above three merging judgment conditions into the same aggregated out-of-bounds block.

[0140] Suppose that there are a total of [number] abnormal data nodes. The nodes in the out-of-bounds state are sorted by time as follows: ,node Until the node Creates an empty list of aggregated out-of-bounds blocks, and a block object currently being built.

[0141] From node Begin by setting up nodes Place it in the current block. Then process the nodes in sequence. To the node For the currently processed node : Check nodes With nodes Does the three merging condition meet? If all three conditions are met, then merge the nodes. Also add it to the current block, and update the end timestamp of the current block to the node. The sampling timestamp.

[0142] If at least one condition is not met, close the current block and create a new current block, then set the node... Place it at the beginning of the new current block.

[0143] After scanning all After each node, the last current block is also saved to the list.

[0144] Calculate the summary statistics for each aggregated out-of-bounds block. For each aggregated out-of-bounds block, the following metrics are calculated: Block start timestamp is the sampling timestamp of the first out-of-bounds state node within the block; Block end timestamp is the sampling timestamp of the last out-of-bounds state node within the block; Block duration is equal to the block end timestamp minus the block start timestamp; Total number of out-of-bounds state nodes within the block. The sum of the absolute values ​​of the deviations of all out-of-bounds state nodes within the block. : ; Block average deviation : ; Maximum deviation of block To retrieve all blocks The maximum value is recorded, along with the corresponding sampling timestamp and out-of-bounds direction; the block deviation from the cumulative integral is recorded. To approximate the calculation as the integral of the deviation over time, since the sampling step size is equal... The integral formula is: ; The list of business segments covered by the block is obtained by extracting unique business segment numbers from the out-of-bounds state nodes within the block; the out-of-bounds direction marker is the out-of-bounds direction that accounts for the majority of the out-of-bounds state nodes within the block.

[0145] The post-merging processing of aggregated out-of-bounds blocks involves two aggregated out-of-bounds blocks belonging to the same anomalous data node and overlapping or closely adjacent in time. If their out-of-bounds directions are consistent and the covered business processes overlap, a second-level merge is performed, combining them into a larger aggregated out-of-bounds block. All summary statistics are updated. After the above merging operation, a set of aggregated out-of-bounds blocks is obtained. Each block in the set is an aggregate record, containing: the globally unique identifier of the anomalous data node to which it belongs, the global dimension number and standard dimension name to which it belongs, the block start timestamp, the block end timestamp, the block duration, the total number of out-of-bounds nodes within the block, the total deviation, the average deviation, the maximum deviation and its corresponding timestamp, the cumulative deviation score, the list of covered business processes, and the out-of-bounds direction marker.

[0146] Step 8.4: Based on the aggregated out-of-bounds blocks, extract the core data node identifier of each block and generate a structured label containing the source of the anomaly and the degree of deviation, resulting in a set of anomalous data node identifiers. Specifically, this includes: extracting core data node identifiers. For each aggregated out-of-bounds block, trace upwards layer by layer along steps 7, 6, and 5 to determine the core data nodes involved in the block. The tracing path is as follows: the anomalous data nodes directly associated with the aggregated out-of-bounds blocks have globally unique identifiers already in the block record. Based on this identifier, trace back to the nonlinear distortion state identifier set in step 7.1 to obtain the nonlinear distortion state identifier of the anomalous data node, and extract from it: source system identifier, node generation timestamp, and distortion type label.

[0147] Then, based on this abnormal data node, trace back to the calculation process of the cumulative total of multi-source data deviation in step 6.2, and locate the multi-source instantaneous deviation of the global dimension to which this node belongs in the corresponding time interval. And its percentile ranking in all time infinitesimals. Simultaneously, backtracking to the cross-dimensional deviation correlation matrix in step 6.3, identify all high-divergence correlation pairs with this global dimension as the antecedent or consequent, and record the global dimension number and degree of divergence of the correlated counterpart. Then, going back to the metadata time-series drift rate parsing results in step 5.4, we extract the drift rate of the drift interval where the abnormal data node is located. Drift direction markers and interval average rate of change Then, backtracking to the intermediate quantity of cross-node continuity verification in step 4, we extract the logical dependency strength value and time interval between the abnormal data node and its predecessor node; finally, backtracking to the local compliance judgment result in step 3, we extract the rule number, non-compliance type and non-compliance degree score of the abnormal data node that was initially judged to be non-compliant; all the information obtained above constitutes the core data node identification context of the aggregated out-of-bounds block.

[0148] Calculate the overall anomaly severity score of the block. It is a percentage score used to quantify the threat posed by the anomaly represented by the out-of-bounds aggregate block to overall data quality. Its calculation formula is: ; The calculation methods for the four sub-items are as follows: Drift severity sub-item : Retrieve the drift rate of the drift interval corresponding to the abnormal data node. ,Right now: ; Deviation range sub-items : Maximum deviation of the block The ratio of this ratio to the tolerance of compliance boundaries in this dimension determines: ; Scope of impact : Determined by the duration of the block (in seconds) and the preset severity duration threshold The ratio (usually taken as 1800 seconds, or thirty minutes) determines the outcome: ; Consistency violation items The maximum degree of cross-dimensional divergence involved by this anomalous data node. The ratio to 1 determines: ; Weighting coefficient , , , The sum of these values ​​is 1, and they are typically set to 0.30, 0.25, 0.20, and 0.25 respectively. The weights for drift severity and consistency violation are slightly higher, reflecting the greater impact of these two factors on data quality.

[0149] Structured anomaly labels are generated. For each aggregated out-of-bounds block, a structured anomaly data node label is generated based on its core data node identifier context and comprehensive anomaly severity score. This label contains three levels of tag information: The first level is the anomaly source label, which includes the source system identifier, source system name, globally unique identifier of the abnormal data node, node generation timestamp, name of the business process to which it belongs, name of the global dimension standard to which it belongs, and a summary of the violation rule number. The second level is the anomaly manifestation label, which includes the distortion type (curvature distortion, jump distortion, direction reversal distortion, or a combination thereof), the direction of the boundary crossing (exceeding the upper limit, exceeding the lower limit, or bidirectional boundary crossing), and the cumulative score of block deviation. Maximum deviation of the block and its proportion and drift rate in the compliance boundary tolerance of this dimension. The third level is the abnormal impact label, which includes a comprehensive score for the severity of the abnormality. The associated global dimension list and the corresponding degree of deviation. The contribution percentage of this anomaly to the total cumulative deviation of multi-source data, and its impact on consistency levels.

[0150] The abnormal data node identifiers are sorted and summarized, and the structured anomaly labels corresponding to all aggregated out-of-bounds blocks are scored according to the overall anomaly severity. Sort in descending order from largest to smallest. For anomaly labels with the same score, sort by block deviation cumulative score. The sorting is based on a secondary ranking from largest to smallest. After sorting, each anomaly tag is assigned a priority level: the top 20% are marked as high-priority red alerts, the middle 30% as medium-priority orange alerts, and the remaining 50% as low-priority yellow alerts. If the total number of anomaly tags is less than five, an absolute threshold is used for determination. As a high priority, Medium priority It has low priority.

[0151] The final result is a set of abnormal data node identifiers. Each identifier record in this set includes: priority level, full fields of the abnormal source label, full fields of the abnormal behavior label, full fields of the abnormal impact label, and the type of suggested repair action.

[0152] In this embodiment of the invention, by extracting the values ​​of continuous state nodes in the trajectory and comparing them point by point with preset compliance boundary conditions, boundary deviation state records are obtained. This accurately quantifies the deviation of the data state from the compliance standard, providing an objective and rigorous numerical comparison benchmark for subsequent anomaly identification. By filtering continuous state nodes whose values ​​exceed the upper or lower limit of compliance and marking the corresponding archival business process and timestamp, a set of out-of-bounds state nodes is obtained. This effectively filters out normal data fluctuations within the compliance range, accurately locks down substantial out-of-bounds nodes, and achieves spatiotemporal dual positioning of anomaly information.

[0153] By performing a continuous block merging operation on out-of-bounds state nodes with adjacent timestamps, aggregated out-of-bounds blocks are obtained. This overcomes the shortcomings of traditional methods that fragment continuous anomalies into isolated alarms, integrating fragmented out-of-bounds nodes into anomaly blocks with business continuity, reducing the redundancy and processing complexity of alarm information. By extracting the core data node identifier of each block, a structured label containing the source of the anomaly and the degree of deviation is generated, resulting in an anomaly data node identifier set. This successfully transforms complex anomaly states into a structured expression that is machine-readable and human-understandable.

[0154] In a preferred embodiment of the present invention, step 9 above may include: Step 9.1: Based on the abnormal data node identifier set, parse the business attribute dimension and flow link dimension corresponding to each abnormal node to obtain multi-dimensional abnormal feature tags. Specifically, this includes: loading the output abnormal data node identifier set, where each record already contains three levels of information: abnormal source tag, abnormal manifestation tag, and abnormal impact tag. Traverse each abnormal identifier record in the set, extracting the globally unique identifier of the abnormal data node, the source system identifier, and the standard name of the global dimension from the abnormal source tag. Simultaneously, extract the distortion type, out-of-bounds direction, and maximum block deviation from the abnormal manifestation tag; parse the business attribute dimension features. The business attribute dimension refers to the category system to which the abnormal data node belongs in the document business semantic space, including but not limited to the following sub-dimensions: data domain type sub-dimension, such as amount, date, text, enumeration, identifier, etc.; business entity sub-dimension, such as contract, payment slip, approval slip, invoice, voucher, etc.; data generation method sub-dimension, such as manual entry, automatic system generation, external system push, scanning and recognition extraction, etc.; data sensitivity level sub-dimension, such as core business data, auxiliary business data, log data, etc.

[0155] The parsing process employs an automatic classification method based on rule matching. A pre-defined classification rule table for business attribute dimensions is used, where each rule defines the mapping logic from existing fields of the anomaly identifier to the values ​​of each sub-dimension. For example, for the data domain type sub-dimension, keyword matching is performed using the standard name of the global dimension to which the anomaly node belongs: nodes with keywords such as amount, total amount, and price in the standard name are classified into the amount category; nodes with keywords such as date, time, and moment are classified into the date category; and nodes with keywords such as name, description, and remarks are classified into the text category. For the business entity sub-dimension, the source system identifier of the anomaly node is matched against a pre-defined system and entity mapping table: anomaly nodes from the OA system are classified into the approval form entity, nodes from the finance system are classified into the payment form entity, and nodes from the image scanning system are classified into the invoice entity.

[0156] The analysis focuses on the dimensional characteristics of the data flow stages. The dimensional characteristics refer to the stage position of the abnormal data node in the lifecycle of the archive business flow. Step 8.2 already associated each out-of-bounds node with its corresponding business stage number and name. Here, we further enrich the dimensional characteristics of the data flow stages based on the list of business stages covered by the aggregated out-of-bounds block. These characteristics include: the name of the first business stage involved by the abnormal node, the name of the last business stage, the total number of business stages crossed, whether it crosses multiple non-adjacent business stages (i.e., the difference in stage numbers is greater than 1), and the macro-level flow stage it belongs to, such as the data acquisition stage, data processing stage, data review stage, or data archiving stage.

[0157] The macro-level workflow stage is divided as follows: the first third of the business process sequence numbers are assigned to the data acquisition stage, the middle third to the data processing stage, and the last third to the data review stage. The archiving stage is independently assigned to the data archiving stage. The cross-contribution of multi-dimensional anomaly features is calculated. Business attribute dimensions and workflow stage dimensions are cross-combined to form the Cartesian product of the business attribute sub-dimension values ​​and the workflow stage sub-dimension values. For each cross-combination, the contribution percentage of that combination to the anomaly severity is calculated. The specific calculation method is as follows: Count the number of out-of-bounds states belonging to this cross combination under this abnormal data node, and denote it as . Count the total number of out-of-bounds nodes under this abnormal data node, and denote it as . The percentage of nodes in this cross combination is: ; Next, calculate the sum of the absolute values ​​of the deviations of the out-of-bounds state nodes belonging to this crossover combination under the abnormal data node, and denot it as... The sum of the absolute values ​​of the deviations of all out-of-bounds state nodes under this abnormal data node is denoted as . The deviation percentage of this cross combination is: ; The cross-contribution is calculated as the weighted average of the percentage of nodes and the percentage of deviation. ; The percentage of deviation has a greater weight because the magnitude of the deviation is a better indicator of the true severity of the anomaly than the number of nodes.

[0158] Assemble multidimensional anomaly feature labels. For each anomaly data node identifier, generate its own unique multidimensional anomaly feature label. This label contains the following components: The first part is the feature vector of business attribute dimensions, recording the values ​​of each sub-dimension in key-value pairs, such as data domain type equals amount, business entity equals payment slip, data generation method equals manual entry, and data sensitivity level equals core business data. The second part is the feature vector of circulation stage dimensions, recording fields such as the name of the first business stage, the name of the last business stage, the total number of stages crossed, and the macro circulation stage. The third part is the cross-contribution matrix, recording the cross-contribution values ​​of each cross combination, arranged in descending order of cross-contribution, retaining combinations with a contribution of more than 5%. The fourth part is the comprehensive anomaly severity score. The abnormal data node identifiers are directly inherited from step 8.4. After the above parsing, all abnormal data node identifiers form a multidimensional abnormal feature label set.

[0159] Step 9.2: Based on the multidimensional anomaly feature labels, perform anomaly dimension space separation operation to divide anomaly nodes of the same business attribute dimension or the same flow link dimension into independent verification groups to obtain dimension-isolated anomaly groups. Specifically, this includes: determining the main axis and separation granularity of dimension separation, where the business attribute dimension and the flow link dimension are two orthogonal separation main axes. On the main axis of business attributes, three sub-dimensions—data domain type, business entity, and data generation method—are selected as the separation criteria. On the main axis of circulation process, the macro-circulation stage is selected as the separation criteria. These four separation criteria together constitute a four-dimensional separation space. The combination of values ​​for each dimension in this separation space defines a basic unit for independent verification grouping. The preparatory work before separation includes traversing the multi-dimensional abnormal feature label set, collecting all possible values ​​for the four separation criteria, and forming the value domain for each separation criterion. For example, the value domain for data domain type may be {amount, date, text, enumeration, identifier}; the value domain for business entity may be {contract, payment slip, approval slip, invoice, voucher}; the value domain for data generation method may be {manual entry, automatic system generation, external system push, scanning and recognition extraction}; and the value domain for macro-circulation stage may be {data acquisition stage, data processing stage, data review stage, data archiving stage}. Perform a dimensional space separation operation. For each anomaly node in the multidimensional anomaly feature label set, determine its precise coordinates in the four-dimensional separation space based on the business attribute dimension feature vector and the flow link dimension feature vector in its multidimensional anomaly feature label. Traverse all anomaly nodes, grouping anomaly nodes with identical quadruple coordinates into the same initial group. Assume that after deduplication, a total of [number] nodes are formed. Each of the four different quadruples of coordinates corresponds to For each initial group, count the number of abnormal nodes within that group. and comprehensive abnormality severity score The sum .Will Initial groups smaller than the preset minimum group size threshold, typically set to 3, are marked as sparse groups.

[0160] Perform a sparse grouping merging operation. For an initial group labeled as a sparse group, search for neighboring groups along each dimension of the four-dimensional separation space that differ from the target group in only one dimension and have the same values ​​in the other three dimensions. If multiple such neighboring groups exist, calculate the merged benefit between the target group and each candidate neighboring group. The formula for calculating the merged benefit is: ; in, The number of outlier nodes in the target sparse group. This represents the number of abnormal nodes in the candidate adjacent groups. The average severity score for the target group, The average severity score for candidate groups.

[0161] A positive post-merge benefit indicates that the severity within the groups is more balanced after merging, making the merge statistically significant. The candidate adjacent groups with the highest post-merge benefit are selected for merging. The merging method is as follows: all outlier nodes in the sparse group are moved to the selected adjacent group, the sparse group is deleted, and the values ​​of the adjacent group on the difference dimension are expanded to wildcards. After merging, the scores of the merged groups are recalculated. and Repeat the merge operation until no more merges are found. Groups smaller than the minimum group size threshold are grouped together.

[0162] Calculate the group feature vector for each dimension-isolated outlier group. For each dimension-isolated outlier group that is finally determined, calculate the following group-level statistical indicators: Total number of abnormal nodes in the group Mean score of overall abnormality severity in the group .

[0163] The standard deviation of the group's overall severity of abnormality score is calculated using the standard formula for standard deviation: ; The distribution of distortion types within the group, including the proportions of curvature distortion, jump distortion, direction reversal distortion, and composite distortion.

[0164] Distribution of outbound directions within the group, statistics on the proportion of exceeding the upper limit, exceeding the lower limit, and bidirectional outbounds, and the total cumulative deviation within the group. ,in Iterate through all out-of-bounds blocks corresponding to abnormal nodes within the group.

[0165] The dominant anomaly pattern is determined by combining the most frequent distortion type and the out-of-bounds direction within the group. The final output is a set of dimensionally isolated anomaly groups. Each group record includes: group number, four-dimensional separation coordinates, total number of anomaly nodes in the group, mean and standard deviation of the group's overall anomaly severity score, distortion type distribution, out-of-bounds direction distribution, total cumulative deviation, dominant anomaly pattern, and a list of anomaly node identifiers within the group.

[0166] Step 9.3: Isolate abnormal groups according to the aforementioned dimensions, trace the upstream data dependencies of abnormal nodes within each group, identify the source data fields and verification constraint entries that caused the anomalies, and obtain the defect tracing association chain. Specifically, this includes: loading a data flow dependency graph, which is gradually accumulated and constructed during the processing of steps 1 to 4, recording the entire chain of data flow from initial collection to final archiving. Nodes in the graph represent data field instances, and directed edges represent dependencies between data fields. Dependencies are divided into three types: Type 1 is direct transitive dependency: the value of upstream node A is directly copied or simply converted and used as the source of the value for downstream node B. For example, the payment amount field in a payment order directly references the approval amount field in an approval order; Type 2 is aggregate computation dependency: the value of downstream node B is derived from multiple upstream nodes. , , The values ​​are obtained through aggregation operations. For example, the monthly summary amount is obtained by summing the payment amounts of all payment orders for the current month; Type 3 is a verification judgment dependency: the compliance status judgment of downstream node B depends on whether upstream node A meets a certain verification constraint item. For example, the local compliance judgment result in step 3 depends on the output of the field-level compliance judgment in step 2; For each abnormal node in the dimension-isolated abnormal group, starting from the globally unique identifier of the abnormal node, a breadth-first traversal is performed in the data flow dependency graph along the directed edges in reverse direction, i.e., from downstream to upstream, tracing its upstream dependent nodes layer by layer. The upper limit of the tracing depth is set to five levels to avoid infinite backtracking.

[0167] Construct an upstream dependency tree for abnormal nodes. For an abnormal node X, the root node of its upstream dependency tree is X itself. The first-level parent nodes are all upstream nodes that X directly depends on in the dependency graph. The second-level parent nodes are the upstream nodes that each parent node in the first level directly depends on, and so on until the fifth level or until there are no more upstream nodes in the graph. For each node in the upstream dependency tree, record the following information: the name of the data field represented by the node, the identifier of the source system, the name of the business process, the node generation timestamp, the verification constraint entry number corresponding to the node, whether the node was judged as non-compliant in step 3 and its degree of non-compliance score.

[0168] Identify source data fields. In the upstream dependency tree, a source data field is defined as a node that meets one of the following conditions: Condition 1, the node has no upstream dependent nodes in the dependency graph; Condition 2, although the node has upstream dependent nodes in the graph, all of its upstream dependent nodes were determined to be compliant in step 3. Traverse the upstream dependency tree, scanning from the root node to the leaf nodes, to find all nodes that meet the above conditions. For each such node, identify it as a candidate source data field for the anomalous node. If the same anomalous node has multiple candidate source data fields, the priority is determined according to the following rules: The priority determination rules are as follows: candidate sources judged as non-compliant have higher priority than candidate sources judged as compliant; those with higher non-compliance scores have higher priority than those with lower scores; those closer to the abnormal node in the dependency chain have higher priority; those with higher business sensitivity have higher priority; and the data field of the candidate source with the highest priority is taken as the final source data field of the abnormal node.

[0169] Identify the validation constraint entries that caused the anomaly. Following the forward dependency path from the source data field to the anomalous node, examine the validation constraint entries associated with each node along the path. For each validation constraint entry, calculate its contribution to the anomalous node; sort all validation constraint entries on all path nodes in descending order of contribution, and select the entry with a contribution exceeding 10% as the primary validation constraint entry that caused the anomaly.

[0170] Assemble the defect tracing chain. For each anomalous node, connect its complete dependency path from the source data field to the node itself, along with the validation constraints and contribution responsibilities of each node along the path, into a defect tracing chain. The structured description of the defect tracing chain includes six parts: The first ring is the complete location of the source data field; the second ring is the location and amount of the source field's first deviation from the compliance boundary; the third ring is the sequence of intermediate nodes along the deviation propagation along the dependency link, with each intermediate node recording the field name, the link it belongs to, and the propagation amplification coefficient; the fourth ring is the cumulative calculation of the propagation amplification coefficient; the fifth ring is the ranking of the contribution responsibility of each verification constraint item on the path; and the sixth ring is the out-of-bounds behavior finally presented by the abnormal node.

[0171] The propagation amplification factor is calculated as follows: for two adjacent nodes in a dependent link... and Upstream, For downstream nodes, if the dependency type is a direct transitive dependency (downstream equals upstream multiplied by a conversion factor), then the propagation amplification factor equals the conversion factor. If the dependency type is an aggregate computation dependency (downstream equals the sum of multiple upstream nodes), then the propagation amplification factor equals the weight percentage of that upstream node in the aggregate computation. The cumulative propagation amplification factor across the entire link from the source to the abnormal node is the product of the propagation amplification factors of each segment: ; in This represents the number of adjacent node pairs in the link. The propagation amplification coefficient for each segment is used. The above tracing operation is performed on all abnormal nodes within the dimension-isolated abnormal group, and the result is a set of defect source tracing association chains.

[0172] Step 9.4: Based on the defect tracing association chain, summarize the defect distribution frequency and association constraint entries for each dimension, generate a structured report containing specific field locations and error types, and obtain the final data quality defect location result. Specifically, this includes: summarizing the defect distribution by business attribute dimension, using data domain type, business entity, and data generation method as primary keys, grouping and statistically analyzing all abnormal nodes in the defect tracing association chain set, and for each combination value, calculating the following indicators: The total number of abnormal nodes under this combination The proportion of abnormal nodes in this combination to the total number of abnormal nodes. The average comprehensive anomaly severity score of the abnormal nodes under this combination. The number of duplicate source data fields involved in the abnormal nodes under this combination. The defect density for this combination is defined as the total number of anomalous nodes in this combination divided by the total frequency of this combination appearing in all data nodes. If the total frequency of a combination is unknown or zero, the defect density is replaced by the absolute value of the total number of anomalous nodes, calculated as follows: Arrange them in descending order from largest to smallest to form a defect distribution table based on business attribute dimensions.

[0173] Defect distribution is summarized by workflow stage. Using the macro-level workflow stage and specific business process name as the primary key, similar statistical operations to the first step are performed. In addition, two additional indicators are calculated at the workflow stage level: The process defect injection rate is defined as the number of abnormal nodes in the process where the data field originates from that business process, divided by the total number of all data nodes within that process. This metric reflects the probability that each process is the starting point of a defect. The process defect detection rate is defined as the number of out-of-bounds instances of abnormal nodes that are first detected in that business process, divided by the total number of all data nodes within that process. This metric reflects the probability that each process is the exposure point of a defect.

[0174] When the defect injection rate of a certain process is higher than the defect detection rate, it means that the defect was not detected in time after it was generated in that process and spread downstream, indicating that the process needs to strengthen the verification constraints. When the defect detection rate of a certain process is higher than the defect injection rate, it means that the verification constraints of that process are relatively strict and intercept defects from upstream.

[0175] Defect distribution is summarized by source data field. Using each source data field as a statistical unit, the following information is summarized: The complete location of the source data field; the total number of abnormal nodes caused by this source data field. The percentage of anomalous nodes caused by this source data field among all anomalous nodes; the sum of the comprehensive anomalous severity scores of all anomalous nodes in the defect tracing chain associated with this source data field. The impact range score of the defect in this source data field. ,according to Sort the data in descending order of size to identify the key source fields that have the greatest impact on overall data quality. Typically, [these fields are selected]. The top five fields are prioritized for root cause remediation.

[0176] Defect distribution is summarized by validation constraint item. Using each validation constraint item as a statistical unit, the following information is summarized: The validation constraint entry number and rule description summary, along with the total number of times this entry is associated as the primary validation constraint entry in all defect tracing chains. The average contribution responsibility of this item The deduplication list and count of deduplicated source data fields involved in the abnormal node associated with this entry; the defect detection effectiveness score for this entry: ; A higher validity score indicates that the constraint item plays a greater practical role in defect detection and that its detection weight is set reasonably. Items with validity scores lower than the average level of similar constraint items suggest that their detection weights may need to be adjusted. The final structured data quality defect location result report is then generated.

[0177] In this embodiment of the invention, by parsing the business attribute dimension and flow link dimension corresponding to each abnormal node, multi-dimensional abnormal feature labels are obtained, accurately depicting the business context of the abnormal data. This overcomes the information loss caused by traditional single-dimensional detection and provides a multi-dimensional feature foundation for subsequent refined analysis. By performing an abnormal dimension space separation operation, abnormal nodes with the same business attribute dimension or the same flow link dimension are divided into independent verification groups, resulting in dimension-isolated abnormal groups. This effectively decouples and isolates complex abnormal data, avoids the ambiguity in location caused by mutual interference between abnormalities of different dimensions, and reduces the complexity of defect analysis.

[0178] By tracing the upstream data dependencies of abnormal nodes within each group, the source data fields and validation constraints that triggered the anomalies were identified, resulting in a defect tracing chain. This approach overcomes the limitations of traditional methods that can only output surface-level anomaly results, achieving precise backtracking to the underlying data source and clarifying the root cause of data quality issues. By summarizing the frequency of defect distribution and related constraint entries across various dimensions, a structured report containing specific field locations and error types was generated, yielding the final data quality defect location results. This successfully transformed complex tracing information into concrete conclusions that can directly guide data cleaning and system optimization, eliminating systemic blind spots caused by inconsistencies in the archive event chain logic.

[0179] like Figure 2 As shown, embodiments of the present invention also provide an automatic electronic archive data quality detection system based on a rule engine, comprising: The data acquisition module is used to acquire the metadata record set of the electronic archives to be inspected and the pre-set structured verification constraint library; The feature parsing module is used to parse and generate the initial state feature vector of each data node based on the metadata record set; The discrete matching module is used to perform discrete logic matching on the initial state feature vector based on the structured verification constraint library to generate a local compliance judgment result. The time-series extraction module is used to extract time-series correlation parameters from the local compliance judgment results and generate intermediate quantities for cross-node continuity verification. The differential operation module is used to perform analytical differential operations in the continuous time dimension based on the intermediate quantity of cross-node continuity verification, and generate metadata time series drift rate analysis results; The integral calculation module is used to perform cumulative integral calculation on the metadata time-series drift rate parsing results, and generate a global correlation consistency total evaluation result. The equation solving module is used to solve the evolution differential equation under nonlinear distortion state based on the global correlation consistency total evaluation result, and generate the data state smooth transition trajectory solution; The boundary comparison module is used to compare the solution of the smooth transition trajectory of the data state with the preset compliance boundary conditions to generate a set of abnormal data node identifiers. The spatial separation module is used to perform anomaly dimension spatial separation operation based on the set of abnormal data node identifiers, and generate the final data quality defect location result.

[0180] It should be noted that this system is a system corresponding to the above method. All implementation methods in the above method embodiments are applicable to this embodiment and can achieve the same technical effect.

[0181] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for automatic detection of electronic archive data quality based on a rule engine, characterized in that, The method includes: Step 1: Obtain the metadata record set of the electronic document to be inspected and the pre-set structured verification constraint library; Step 2: Based on the metadata record set, parse and generate the initial state feature vector of each data node; Step 3: Based on the structured verification constraint library, perform discrete logic matching on the initial state feature vector to generate a local compliance judgment result; Step 4: Extract the time-series correlation parameters from the local compliance determination results and generate intermediate quantities for cross-node continuity verification; Step 5: Based on the intermediate quantity of cross-node continuity verification, perform analytical differentiation operation in the continuous time dimension to generate metadata time-series drift rate analysis results; Step 6: For the metadata time-series drift rate parsing results, perform cumulative integral calculation of multi-source heterogeneous archive data to generate a global correlation consistency total evaluation result; Step 7: Based on the total global correlation consistency assessment results, solve the evolution differential equation under the nonlinear distortion state to generate the data state smooth transition trajectory solution; Step 8: Compare the solution of the smooth transition trajectory of the data state with the preset compliance boundary conditions to generate a set of abnormal data node identifiers; Step 9: Based on the set of abnormal data node identifiers, perform an abnormal dimension space separation operation to generate the final data quality defect location result.

2. The method for automatic detection of electronic archive data quality based on a rule engine according to claim 1, characterized in that, Step 2: Based on the metadata record set, parse and generate the initial state feature vector for each data node, including: Based on the metadata record set, perform storage path parsing and original field stripping operations to separate unstructured text tags and core business values, resulting in a sequence of independent field data pairs; Based on the sequence of independent field data pairs, perform data type classification and redundant character filtering operations, remove invalid placeholders and correct garbled fields to obtain a list of pure numerical attributes. Based on the list of pure numerical attributes, the unit of measurement and dimension standardization operations are performed by referring to the preset archive classification dictionary to convert discrete business indicators into calculable continuous physical parameters, thereby obtaining a sequence of physical quantities with unified dimensions. Based on the unified dimensional physical quantity sequence, spatial coordinate alignment and vector dimension concatenation operations are performed along the generation timestamp order to integrate the numerical distribution states of each dimension and construct a multidimensional matrix to obtain the initial state feature vector of each data node.

3. The method for automatic detection of electronic archive data quality based on a rule engine according to claim 2, characterized in that, Step 3: Based on the structured verification constraint library, perform discrete logic matching on the initial state feature vector to generate a local compliance determination result, including: Based on the initial state feature vector of each data node, the discrete dimension state identifier is extracted and the structured verification constraint library is retrieved. The associated judgment conditions and tolerance intervals are located, and the correspondence between feature dimensions and verification items is established to obtain the discrete state constraint mapping table. According to the discrete state constraint mapping table, a logical boundary comparison operation is performed to check the inclusion relationship between the actual value and the tolerance range for each component, identify out-of-bounds data blocks and mark the positive and negative deviation directions to obtain the local deviation state identifier. Based on the local deviation status identifier, compliance weight allocation and binary classification judgment are performed, the matching status of each dimension is summarized and invalid alarms are removed, a Boolean verification conclusion is generated, and the local compliance judgment result is obtained.

4. The automatic detection method for electronic archive data quality based on a rule engine according to claim 3, characterized in that, Step 4: Extract the time-series correlation parameters from the local compliance determination results to generate intermediate quantities for cross-node continuity verification, including: Based on the local compliance determination results, the deviation data nodes containing timestamp attributes are extracted, and static isolated data without time-series dependencies are filtered out to obtain a subset of time-series related data. Based on the aforementioned time-series associated data subset, the difference between adjacent nodes and state transition tracking are performed along the archive flow time axis to extract continuously changing business indicator values ​​and obtain a time-series associated parameter sequence; Based on the time-series correlation parameter sequence, perform time window sliding and data alignment operations to eliminate phase shift caused by acquisition time difference and obtain a standardized time-series correlation parameter set; Based on the standardized time-series correlation parameter set, a state transition matrix between adjacent data nodes is constructed to quantify the logical dependency strength of continuous business links and obtain intermediate quantities for cross-node continuity verification.

5. The method for automatic detection of electronic archive data quality based on a rule engine according to claim 4, characterized in that, Step 5: Based on the intermediate quantity of the cross-node continuity verification, perform analytical differentiation operation in the continuous time dimension to generate metadata time-series drift rate analysis results, including: Based on the intermediate quantities of the cross-node continuity verification, the business indicator state transition trajectory in the continuous time dimension is extracted, a continuously differentiable time-series evolution function model is constructed, and the time-series state analytical expression is obtained. Based on the time-series state analytical expression, analytical differentiation is performed on the continuous time variable to calculate the instantaneous rate of change of business indicators over time, and the instantaneous rate of change distribution curve is obtained. Based on the instantaneous rate of change distribution curve, identify the extreme points of sudden increase or decrease in rate of change, locate the abnormal change intervals of metadata in the continuous flow process, and obtain the set of time-series change feature intervals. Based on the set of time-series mutation feature intervals, the average rate of change within each interval is calculated and compared with the preset dynamic drift boundary to quantify the magnitude of deviation from the normal flow pattern, thereby obtaining the metadata time-series drift rate analysis result.

6. The method for automatic detection of electronic archive data quality based on a rule engine according to claim 5, characterized in that, Step 6: Based on the metadata time-series drift rate parsing result, perform cumulative integration calculation on the multi-source heterogeneous archive data to generate a global correlation consistency total evaluation result, including: Based on the metadata time-series drift rate parsing results, the association dimension identifiers of multi-source heterogeneous archive data are extracted and mapped to a unified global verification space to obtain a global association dimension mapping set; Based on the global correlation dimension mapping set, for the continuous change interval in the time series drift rate analysis result, perform cross-data source cumulative integration operation, accumulate the total deviation under the continuous time dimension, and obtain the total cumulative deviation of multi-source data. Based on the total cumulative deviation of the multi-source data, combined with the pre-set archive business logic association constraints, cross-validation and weight aggregation between heterogeneous data are performed to obtain the global association consistency deviation index. Based on the global correlation consistency deviation index, the total threshold determination and consistency level classification are performed to quantify the overall matching degree of multi-source archive data and obtain the global correlation consistency total assessment result.

7. The method for automatic detection of electronic archive data quality based on a rule engine according to claim 6, characterized in that, Step 7: Based on the overall global correlation consistency assessment results, solve the evolutionary differential equation under the nonlinear distortion state to generate a smooth transition trajectory solution for the data state, including: Based on the overall global correlation consistency assessment results, abnormal data nodes with excessive consistency deviation index are extracted, nonlinear distortion features in the data flow process are identified, and a nonlinear distortion status identifier set is obtained. Based on the nonlinear distortion state identifier set, a continuous differential equation model describing the evolution of data state over time is constructed. An initial compliance state and boundary constraints are set to obtain the nonlinear evolution equation expression. Based on the nonlinear evolution equation expression, a one-time analytical solution operation is performed on the abnormal data nodes to calculate the continuous change path of the data state from the distortion point to the compliant state, and to obtain the initial smooth transition trajectory set; Based on the initial set of smooth transition trajectories, trajectory continuity verification and discrete noise point filtering operations are performed to restore the evolution path of the real business data that was obscured by nonlinear distortion, and the solution of the smooth transition trajectory of the data state is obtained.

8. The method for automatic detection of electronic archive data quality based on a rule engine according to claim 7, characterized in that, Step 8: Compare the solution of the smooth transition trajectory of the data state with the preset compliance boundary conditions to generate a set of abnormal data node identifiers, including: Based on the smooth transition trajectory solution of the data state, the values ​​of continuous state nodes in the trajectory are extracted and compared with the preset compliance boundary conditions point by point to obtain the boundary deviation state record. Based on the boundary deviation status record, filter out continuous status nodes whose values ​​exceed the compliance upper or lower limit, mark the corresponding archive business process and timestamp, and obtain the set of out-of-bounds status nodes. Based on the set of out-of-bounds state nodes, a continuous block merging operation is performed on out-of-bounds state nodes with adjacent timestamps to obtain aggregated out-of-bounds blocks; Based on the aggregated out-of-bounds blocks, the core data node identifier of each block is extracted, and a structured label containing the source of the anomaly and the degree of deviation is generated to obtain a set of abnormal data node identifiers.

9. The method for automatic detection of electronic archive data quality based on a rule engine according to claim 8, characterized in that, Step 9: Based on the set of abnormal data node identifiers, perform an abnormal dimension space separation operation to generate the final data quality defect location result, including: Based on the set of abnormal data node identifiers, the business attribute dimension and flow link dimension corresponding to each abnormal node are parsed to obtain multi-dimensional abnormal feature labels; Based on the multidimensional anomaly feature labels, perform anomaly dimension space separation operation to divide anomaly nodes of the same business attribute dimension or the same flow link dimension into independent verification groups to obtain dimension-isolated anomaly groups. Based on the aforementioned dimensions, isolate abnormal groups, trace the upstream data dependencies of abnormal nodes within each group, identify the source data fields and verification constraint entries that caused the abnormalities, and obtain the defect tracing association chain; Based on the defect tracing chain, the defect distribution frequency and associated constraint entries of each dimension are summarized to generate a structured report containing specific field locations and error types, thus obtaining the final data quality defect location result.

10. An automatic electronic archive data quality detection system based on a rule engine, wherein the system implements the method as described in any one of claims 1 to 9, characterized in that, include: The data acquisition module is used to acquire the metadata record set of the electronic archives to be inspected and the pre-set structured verification constraint library; The feature parsing module is used to parse and generate the initial state feature vector of each data node based on the metadata record set; The discrete matching module is used to perform discrete logic matching on the initial state feature vector based on the structured verification constraint library to generate a local compliance judgment result. The time-series extraction module is used to extract time-series correlation parameters from the local compliance judgment results and generate intermediate quantities for cross-node continuity verification. The differential operation module is used to perform analytical differential operations in the continuous time dimension based on the intermediate quantity of cross-node continuity verification, and generate metadata time series drift rate analysis results; The integral calculation module is used to perform cumulative integral calculation on the metadata time-series drift rate parsing results, and generate a global correlation consistency total evaluation result. The equation solving module is used to solve the evolution differential equation under nonlinear distortion state based on the global correlation consistency total evaluation result, and generate the data state smooth transition trajectory solution; The boundary comparison module is used to compare the solution of the smooth transition trajectory of the data state with the preset compliance boundary conditions to generate a set of abnormal data node identifiers. The spatial separation module is used to perform anomaly dimension spatial separation operation based on the set of abnormal data node identifiers, and generate the final data quality defect location result.

Images