Cross-library data consistency verification method and device, equipment and storage medium

Abstract

The invention relates to the field of data processing, and discloses a cross-library data consistency verification method, device and equipment and a storage medium, and the method is used for carrying out risk assessment on cross-library data derived values. The method comprises the following steps: acquiring a verification task and extracting a task identifier, double data source connection information of the same business index, a derived value calculation rule, a static consistency threshold and a consistency judgment strategy; obtaining dual-data-source original data based on the connection information, and converting the dual-data-source original data into data vectors; obtaining double derived values of the original data of the double data sources according to a calculation rule, and calculating an absolute difference value; if the absolute difference value exceeds a static consistency threshold value, calculating rule attributes, data vectors and double derived values in combination with task historical statistical information, and obtaining a business logic inconsistency risk probability value and root cause classification through a pre-training probabilistic evaluation model; and generating a final result according to the probability value and the judgment strategy, and integrating related information to generate a verification report.

Description

Technical Field

[0001] This invention relates to the field of data processing, and in particular to a method, apparatus, device, and storage medium for cross-database data consistency verification. Background Technology

[0002] As enterprises deepen their digital transformation, it has become commonplace for data assets to be distributed across various heterogeneous data storage systems (such as relational databases, NoSQL databases, data lakes, and data warehouses). Maintaining consistency of key business metrics across these dispersed data sources is crucial to ensuring the accuracy of decision support, business analysis, and report generation. These metrics are typically not direct mappings of the original data, but rather are aggregated, correlated, and calculated from multiple sources based on complex business logic (i.e., derived value calculation rules). Examples include daily active users, monthly gross profit, and inventory turnover rate.

[0003] Currently, the industry mainly relies on the following traditional methods to implement cross-database data consistency verification: Simple comparison based on a fixed threshold: This is the most common method. It involves performing the same computational logic in both the source and target databases to obtain derived values, then calculating their absolute difference or relative ratio, and comparing it to a preset, fixed static threshold. If the difference exceeds the threshold, it is considered inconsistent. While this method is simple to implement and computationally fast, it has significant drawbacks: threshold setting heavily relies on human experience and lacks flexibility; it cannot distinguish between acceptable normal business fluctuations and genuine data errors or logical defects; for complex rules, minor differences may be amplified into false positives, while genuine anomalies may be missed because the fixed threshold is not met.

[0004] Regular batch processing and manual review: This method involves writing scripts to run verification tasks on a schedule, generating a list of results that exceed thresholds, which are then handed over to data engineers or analysts for manual review. This approach shifts the judgment burden to human labor, resulting in low efficiency, slow response times, and a sharp increase in manual costs as the number of verification tasks and the complexity of the rules rise, making it difficult to scale.

[0005] Rule-based enhanced validation: In addition to simple comparison, this method introduces business rules for filtering, such as distinguishing between weekdays and holidays, and ignoring data from specific sources. This method improves flexibility to some extent, but the rules still require manual maintenance and are essentially binary yes / no judgments, which cannot quantify the risk level of inconsistencies and are difficult to cope with changes in data distribution and emerging anomaly patterns.

[0006] Therefore, existing technologies still need improvement and development. Summary of the Invention

[0007] This invention provides a method, apparatus, device, and storage medium for cross-database data consistency verification, used for risk assessment of cross-database data derived values.

[0008] The first aspect of this invention provides a cross-database data consistency verification method, comprising: acquiring a verification task; obtaining a task identifier, connection information of a first data source and a second data source corresponding to the same business indicator, derived value calculation rules, a static consistency threshold, and a consistency determination strategy based on the verification task; acquiring, based on the connection information, first original data and second original data required for calculating derived values ​​from the first data source and the second data source respectively, and converting the first original data and the second original data into a first data vector and a second data vector; calculating, based on the derived value calculation rules, calculating a first derived value using the first original data and calculating a second derived value using the second original data, and calculating the first derived value and... The absolute difference value of the second derived value; compare the absolute difference value with the static consistency threshold. If the absolute difference value exceeds the static consistency threshold, obtain the historical statistical information of the verification task based on the task identifier, extract the attributes of the derived value calculation rule, and use a pre-trained probabilistic evaluation model based on the first derived value, the second derived value, the first data vector, the second data vector, the historical statistical information, and the attribute historical statistical information of the derived value calculation rule to evaluate the probability value and root cause classification representing the risk of inconsistency at the business logic level. Generate a final judgment result based on the probability value and the consistency judgment strategy, and generate a verification report based on the probability value, the root cause classification, and the final judgment result.

[0009] Optionally, in a first implementation of the first aspect of the present invention, the step of obtaining the verification task, and obtaining a task identifier, connection information of a first data source and a second data source corresponding to the same business indicator, derived value calculation rules, static consistency threshold, and consistency determination strategy based on the verification task, includes: obtaining and parsing the verification task to obtain a task identifier; retrieving and reading the corresponding structured task configuration file from a preset configuration management center based on the task identifier; parsing the structured task configuration file to obtain multiple fields, and extracting the connection information of the first data source and the second data source corresponding to the same business indicator, derived value calculation rules, static consistency threshold, and consistency determination strategy from the multiple fields.

[0010] Optionally, in a second implementation of the first aspect of the present invention, the step of obtaining the first original data and the second original data required for calculating the derived value from the first data source and the second data source respectively based on the connection information, and converting the first original data and the second original data into a first data vector and a second data vector, includes: generating query requests for the first data source and the second data source based on the connection information to obtain the first original data and the second original data required for calculating the derived value; performing data cleaning on the first original data and the second original data to obtain first cleaned data and second cleaned data; and using feature engineering methods to convert the first cleaned data and the second cleaned data into a first data vector and a second data vector with the same dimension, respectively.

[0011] Optionally, in a third implementation of the first aspect of the present invention, the step of calculating a first derived value using the first original data and a second derived value using the second original data based on the derived value calculation rule, and calculating the absolute difference between the first derived value and the second derived value, includes: parsing the derived value calculation rule into a computational logic tree and compiling the computational logic tree into task code that can be run in a computational engine; using the computational engine to calculate the first original data and the second original data to obtain the first derived value and the second derived value; and calculating the absolute difference between the first derived value and the second derived value.

[0012] Optionally, in a fourth implementation of the first aspect of the present invention, the step of comparing the absolute difference value with a static consistency threshold, and if the absolute difference value exceeds the static consistency threshold, then obtaining historical statistical information of the verification task based on the task identifier, extracting the attributes of the derived value calculation rule, and using a pre-trained probabilistic evaluation model based on the first derived value, the second derived value, the first data vector, the second data vector, the historical statistical information, and the attribute historical statistical information of the derived value calculation rule, evaluating the probability value and root cause classification representing the risk of inconsistency at the business logic level, includes: comparing the absolute difference value with the static consistency threshold, and if the absolute difference value exceeds the static consistency threshold, obtaining historical statistical information of the verification task based on the task identifier, extracting the attributes of the derived value calculation rule, and using a pre-trained probabilistic evaluation model to evaluate the probability value and root cause classification representing the risk of inconsistency at the business logic level. If a static consistency threshold is specified, then based on the task identifier, historical statistical information of the verification task within a preset historical period is queried from a preset historical knowledge base, and the attributes of the derived value calculation rule are extracted. The historical statistical information includes average difference, difference variance, and historical inconsistency cases. The attributes of the derived value calculation rule include rule complexity score, type and number of operators involved. A comprehensive vector is constructed based on the first derived value, the second derived value, the first data vector, the second data vector, the historical statistical information, and the attributes of the derived value calculation rule. The comprehensive vector is input into a pre-trained probabilistic evaluation model to obtain the probability value and root cause classification of the risk of inconsistency at the business logic level represented by the output of the pre-trained probabilistic evaluation model.

[0013] Optionally, in a fifth implementation of the first aspect of the present invention, the step of inputting the comprehensive vector into a pre-trained probabilistic evaluation model to obtain the probability value and root cause classification of the representation of business logic inconsistency risk output by the pre-trained probabilistic evaluation model includes: constructing and training a probabilistic evaluation model based on a lightweight gradient boosting tree algorithm; standardizing and normalizing the comprehensive vector to obtain a pre-processed vector; and inputting the pre-processed vector into the pre-trained probabilistic evaluation model to obtain the probability value and root cause classification of the representation of business logic inconsistency risk output by the probabilistic evaluation model.

[0014] Optionally, in a sixth implementation of the first aspect of the present invention, the step of generating a final judgment result based on the probability value and the consistency judgment strategy, and generating a verification report based on the probability value, the root cause classification, and the final judgment result, includes: matching the probability value with a risk level threshold in the consistency judgment strategy to determine that the final judgment result is consistent, inconsistent, or pending; combining the probability value, the root cause classification, and the final judgment result to generate a verification report, the verification report including a difference summary, risk level, root cause inference, and processing suggestions; storing the verification report in a structured format, and triggering an alarm according to a preset alarm rule.

[0015] A second aspect of the present invention provides a cross-database data consistency verification device, comprising: a parsing module, configured to acquire a verification task, and based on the verification task, obtain a task identifier, connection information of a first data source and a second data source corresponding to the same business indicator, a derived value calculation rule, a static consistency threshold, and a consistency determination strategy; a conversion module, configured to, based on the connection information, acquire first original data and second original data required for calculating derived values ​​from the first data source and the second data source respectively, and convert the first original data and the second original data into a first data vector and a second data vector; and a calculation module, configured to, based on the derived value calculation rule, calculate a first derived value using the first original data, calculate a second derived value using the second original data, and calculate the first derived value and the second derived value using the second original data, and calculate the first derived value and the second derived value. The absolute difference value of the two derived values; the evaluation module is used to compare the absolute difference value with the static consistency threshold. If the absolute difference value exceeds the static consistency threshold, the historical statistical information of the verification task is obtained based on the task identifier, and the attributes of the derived value calculation rule are extracted. Based on the first derived value, the second derived value, the first data vector, the second data vector, the historical statistical information, and the historical statistical information of the attributes of the derived value calculation rule, a pre-trained probabilistic evaluation model is used to evaluate the probability value and root cause classification that represent the risk of inconsistency at the business logic level; the generation module is used to generate a final judgment result based on the probability value and the consistency judgment strategy, and generate a verification report based on the probability value, the root cause classification, and the final judgment result.

[0016] A third aspect of the present invention provides a cross-database data consistency verification device, comprising: a memory and at least one processor, wherein the memory stores computer-readable instructions, and the memory and the at least one processor are interconnected via a circuit; the at least one processor invokes the computer-readable instructions in the memory to cause the cross-database data consistency verification device to perform the various steps of the cross-database data consistency verification method described above.

[0017] A fourth aspect of the present invention provides a computer-readable storage medium storing computer-readable instructions that, when executed on a computer, cause the computer to perform the steps of the cross-database data consistency verification method as described above.

[0018] The technical solution provided by this invention introduces a two-stage progressive architecture of fast filtering with static thresholds and in-depth evaluation with probabilistic models. First, it efficiently filters out a massive number of tasks with no significant differences using preset thresholds, greatly saving computing resources and ensuring the efficiency of basic verification. For potential anomalies exceeding the threshold, it integrates the current difference value, the vectorized representation of the original data, the historical statistical baseline, and the attribute features of the derived rules themselves to form a multi-dimensional, context-rich comprehensive judgment basis. The pre-trained probabilistic evaluation model performs in-depth analysis based on this, and its output of the inconsistency risk probability value and root cause classification probability distribution fundamentally changes the limitations of traditional binary judgment, providing quantitative and interpretable decision support for operation and maintenance personnel, and significantly improving the accuracy and operability of alarms. Attached Figure Description

[0019] Figure 1 This is a first flowchart of a cross-database data consistency verification method provided in an embodiment of the present invention; Figure 2 This is a second flowchart of the cross-database data consistency verification method provided in an embodiment of the present invention; Figure 3 This is a third flowchart of the cross-database data consistency verification method provided in the embodiments of the present invention; Figure 4 This is a fourth flowchart of the cross-database data consistency verification method provided in this embodiment of the invention; Figure 5 This is a fifth flowchart of the cross-database data consistency verification method provided in this embodiment of the invention; Figure 6 This is a sixth flowchart of the cross-database data consistency verification method provided in this embodiment of the invention; Figure 7 This is a schematic diagram of the cross-database data consistency verification device provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of the cross-database data consistency verification device provided in an embodiment of the present invention. Detailed Implementation

[0020] The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" or "having" and any variations thereof are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0021] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 The first embodiment of a cross-database data consistency verification method in this invention includes: S101. Obtain the verification task, and based on the verification task, obtain the task identifier, the connection information of the first data source and the second data source used to calculate the same business indicator, the derived value calculation rules, the static consistency threshold, and the consistency judgment strategy.

[0022] In this embodiment, it can be understood that the execution subject of the present invention can be a cross-database data consistency verification device, or it can be a terminal or a server; the specific implementation is not limited here. This embodiment of the present invention uses a server as an example for illustration.

[0023] In this embodiment, a verification task submitted from a business preset or user-defined source is received, and the task parsing module is activated to quickly break down the verification task. Key core information is directly extracted from the task instructions, including a task identifier to uniquely identify the task, connection information between the first and second data sources corresponding to the same business metric (such as database address, port, account password), calculation rules for derived values ​​of the business metric (such as formulas and statistical logic), a static consistency threshold for preliminary consistency assessment, and a consistency determination strategy including risk level classification. After extraction, the configuration information is checked for completeness. If key information is found to be missing, a completion prompt is immediately triggered to ensure the verification task can proceed normally, providing complete configuration support for subsequent cross-database data processing and verification.

[0024] S102. Based on the connection information, obtain the first original data and the second original data required for calculating the derived value from the first data source and the second data source respectively, and convert the first original data and the second original data into a first data vector and a second data vector.

[0025] In this embodiment, based on the connection information, secure connections are established with the first and second data sources through standardized data access interfaces. This eliminates the need to generate complex query scripts; the required raw data fields are directly extracted based on derived value calculations. The acquired first and second raw data undergo simple standardization processing, including removing obvious duplicate data and unifying data formats (such as dates and numerical units). Subsequently, a lightweight feature processing method is used to map the two types of standardized raw data into data vectors of the same dimension and uniform numerical range, namely the first data vector and the second data vector. This ensures the comparability of the two data sources in subsequent calculations and evaluations, while also adapting to the input requirements of the pre-trained model.

[0026] S103. Based on the derived value calculation rules, calculate the first derived value using the first original data, and calculate the second derived value using the second original data, and calculate the absolute difference between the first derived value and the second derived value. In this embodiment, the extracted derived value calculation rules are directly converted into executable calculation logic to calculate business indicators for the first and second original data, respectively, obtaining a first derived value and a second derived value representing the same business indicator. During the calculation process, data types are automatically adapted to ensure the accuracy of numerical calculations. Subsequently, an absolute value difference calculation method is used to subtract the first derived value from the second derived value and take the absolute value to obtain the absolute difference value between the two. This difference value intuitively reflects the degree of data deviation between the two data sources on the same business indicator.

[0027] S104. Compare the absolute difference value with the static consistency threshold. If the absolute difference value exceeds the static consistency threshold, obtain the historical statistical information of the verification task based on the task identifier, extract the attributes of the derived value calculation rule, and use a pre-trained probabilistic evaluation model based on the first derived value, the second derived value, the first data vector, the second data vector, the historical statistical information, and the attribute historical statistical information of the derived value calculation rule to evaluate the probability value and root cause classification that represent the risk of inconsistency at the business logic level.

[0028] In this embodiment, the calculated absolute difference value is directly compared with the static consistency threshold. If the absolute difference value does not exceed the static consistency threshold, it is determined to be initially consistent. If it exceeds the static consistency threshold, the historical verification data of the verification task is quickly retrieved through the task identifier to collect historical statistical information such as average difference and difference fluctuation. At the same time, the core attributes of the derived value calculation rule (such as rule complexity and the type of operator involved) are extracted. Combining the first derived value, the second derived value, the dual data vector, historical statistical information, and rule attributes, a multi-dimensional feature set is constructed and input into a pre-trained probabilistic evaluation model. The model quickly outputs the risk probability value and root cause classification of inconsistency at the business logic level (such as data source synchronization delay and deviation in understanding the calculation rule), realizing in-depth mining from data differences to business logic risks.

[0029] S105. Generate the final judgment result based on the probability value and consistency judgment strategy, and generate a verification report based on the probability value, root cause classification and the final judgment result.

[0030] In this embodiment, the risk probability value output by the pre-trained probabilistic evaluation model is matched with the risk level threshold in the consistency determination strategy. If the probability value is lower than the low-risk threshold, it is determined to be consistent; if it is higher than the high-risk threshold, it is determined to be inconsistent; and if it is in between, it is determined to be pending, forming the final determination result. Subsequently, the probability value, root cause classification, and final determination result are integrated to generate a structured verification report. The report includes a summary of data differences, risk level labeling, the basis for root cause inference, and targeted processing suggestions, such as checking the data source synchronization mechanism and verification calculation rules.

[0031] This embodiment provides a cross-database data consistency verification method. It adopts a two-level progressive architecture by introducing a static threshold for rapid filtering and a probabilistic model for in-depth evaluation. First, it uses a preset threshold to efficiently filter out a large number of tasks with no obvious differences, which greatly saves computing resources and ensures the efficiency of basic verification. For potential anomalies that exceed the threshold, it integrates the current difference value, the vectorized representation of the original data, the historical statistical baseline, and the attribute features of the derived rules themselves to form a multi-dimensional and context-rich comprehensive judgment basis. The pre-trained probabilistic evaluation model performs in-depth analysis based on this. Its output inconsistency risk probability value and root cause classification probability distribution fundamentally change the limitations of traditional binary judgment, providing quantitative and interpretable decision support for operation and maintenance personnel, and significantly improving the accuracy and operability of alarms.

[0032] Please see Figure 2 The second embodiment of the cross-database data consistency verification method in this invention includes: S201. Obtain and parse the verification task to get the task identifier.

[0033] In this embodiment, a verification task submitted by a business system or user is obtained through a preset task receiving interface. The task can be in the form of a JSON-formatted instruction or a form submission request. The task parsing engine is activated to perform structured parsing of the verification task, extracting a task identifier to uniquely identify the task. This identifier can be a string containing the business type, timestamp, and random sequence, ensuring uniqueness throughout the system. During parsing, the task format is validated. If the format does not conform to the preset standard (e.g., a missing task identifier field), a format error message is immediately returned, requiring the submitter to correct and resubmit. After extracting the task identifier, it and the verification task are temporarily stored in a temporary cache to provide an index for subsequent configuration file retrieval.

[0034] S202. Based on the task identifier, retrieve and read the corresponding structured task configuration file from the preset configuration management center.

[0035] In this embodiment, a retrieval request is initiated to the configuration management center based on the extracted task identifier. The configuration management center adopts a distributed storage architecture to centrally manage the structured configuration files of all verification tasks. Each configuration file uses the task identifier as its filename for easy and quick location. During the retrieval process, search keywords are constructed using the task identifier, and a hash index algorithm is used to quickly query the corresponding configuration file, avoiding retrieval delays caused by full scans. If the corresponding structured configuration file is found, its integrity is verified to check for any damage or missing critical modules. If the file is not found, the configuration file creation process is triggered, reminding the administrator to supplement the configuration.

[0036] S203. Parse the structured task configuration file to obtain multiple fields, and extract the connection information of the first data source and the second data source corresponding to the same business indicator, the derived value calculation rules, the static consistency threshold, and the consistency judgment strategy from the multiple fields.

[0037] In this embodiment, a structured configuration file parser is activated to parse the downloaded configuration file field by field according to preset file format specifications (such as XML tag definitions and JSON field name conventions). The configuration file contains multiple standardized fields. By matching field names, the connection information (including database type, address, port, login credentials, table name, and field name) of the first and second data sources corresponding to the same business metric is accurately extracted, as well as the derived value calculation rules (such as arithmetic formulas, aggregate functions, and conditional judgment logic), static consistency thresholds (such as numerical thresholds and percentage thresholds), and consistency judgment strategies (such as risk level classification and threshold range definitions).

[0038] In this embodiment, after parsing, the extracted field information is cross-validated to ensure that the dual data source connection information is complete and the calculation rule logic is valid, so as to avoid the subsequent verification process being affected by missing or incorrect fields, and finally form a standardized set of configuration information.

[0039] In this embodiment, a unique association between the verification task and the configuration file is established through a task identifier, ensuring accurate matching of configuration information and avoiding confusion between different task configurations. Furthermore, relying on the configuration management center to centrally store structured task configuration files, standardized management of configuration information is achieved, facilitating subsequent maintenance and modification, and overcoming the drawbacks of traditional distributed configurations that are prone to loss and difficult to trace. Finally, by parsing the structured configuration file to extract core fields, key configurations such as dual data source connection information and calculation rules are clarified, ensuring the accuracy and completeness of the extraction process and avoiding verification task failure due to configuration extraction omissions or errors.

[0040] Please see Figure 3 A third embodiment of a cross-database data consistency verification method according to the present invention includes: S301. Based on the connection information, generate query requests for the first data source and the second data source to obtain the first original data and the second original data required for calculating the derived value.

[0041] In this embodiment, based on the extracted connection information between the first and second data sources, and combined with the derived value calculation rules, the required raw data fields (such as time fields, numerical fields, and identifier fields) are identified. A targeted query request is generated using Structured Query Language (SQL). The query request includes key information such as the data table name, field list, and filtering conditions (such as time range constraints) to ensure that only necessary data is extracted, reducing data transmission volume. Secure connections are established with the first and second data sources respectively through standardized database connection protocols such as JDBC and ODBC. Encrypted transmission mechanisms are used during the connection process to ensure data security. The generated query request is sent to the corresponding data source, and the returned first and second raw data are received.

[0042] S302. Perform data cleaning on the first original data and the second original data to obtain the first cleaned data and the second cleaned data.

[0043] In this embodiment, a multi-step cleaning operation is performed on the acquired first and second raw data. First, duplicate data is removed by identifying and deleting completely duplicate or highly similar redundant data based on primary key fields or core feature fields. Second, missing value processing is performed by filling missing values ​​in key fields with the mean, median, or business logic-based derivation, while missing values ​​in non-key fields are directly marked and retained. Then, outlier detection and processing are performed by identifying abnormal data that exceeds the normal range using box plots and Z-scores, and determining whether to remove or correct it based on business rules. Finally, data format is unified by converting date, time, and numerical data into a preset standard format (e.g., dates are unified to YYYY-MM-DD, and numerical values ​​are retained to two decimal places) to ensure the comparability of the two data sources, ultimately yielding the first and second cleaned data.

[0044] S303. Using feature engineering methods, the first cleaned data and the second cleaned data are converted into a first data vector and a second data vector with the same dimension, respectively.

[0045] In this embodiment, feature engineering methods are used to perform vector transformation on the first and second cleaned data. First, categorical data (such as region identifiers and business types) are numerically processed using one-hot encoding or label encoding. Continuous data (such as amounts and quantities) are normalized using Min-Max normalization or Z-score normalization to map the values ​​to the [0,1] or [-1,1] interval, eliminating the influence of units. Then, based on derived value calculation rules and business indicator characteristics, core feature fields are selected to determine the dimensions of the data vectors, ensuring complete consistency in the dimensions of the data vectors from both data sources. Finally, the processed feature data are arranged in a preset order to construct the first and second data vectors, where each element corresponds to the numerical result of a feature field.

[0046] In this embodiment, a precise query request is generated based on the connection information to ensure efficient extraction of the original data required for calculating derived values ​​from the dual data sources, avoiding data redundancy or missing values. Then, an outlier, duplicate, and missing value is removed from the original data through a data cleaning process, which greatly improves the data quality and provides a clean data source for subsequent calculations and evaluations. Finally, feature engineering methods are used to convert the cleaned data into data vectors of the same dimension, which not only unifies the data format of the dual data sources but also effectively preserves the core features of the data, adapting to the input requirements of the subsequent probabilistic evaluation model.

[0047] Please see Figure 4 A fourth embodiment of a cross-database data consistency verification method according to the present invention includes: S401. Parse the derived value calculation rules into a calculation logic tree, and compile the calculation logic tree into task code that can run in the calculation engine.

[0048] In this embodiment, the computation rule parsing module is activated to perform syntactic analysis and logical decomposition on the derived value computation rules, converting the linear computation rules into a hierarchical computation logic tree. The leaf nodes of the logic tree are the original data fields, the intermediate nodes are operators or aggregate functions, and the root node is the final derived value. During the parsing process, the syntactic correctness of the computation rules is verified. If a syntax error exists (such as operator mismatch or field non-existence), a rule error message is returned. Subsequently, the code generation engine is invoked to compile the computation logic tree into directly executable task code according to the syntax specifications of the computation engine (such as Spark or Flink). The code includes data input interfaces, computation logic implementation, and result output interfaces, ensuring that the computation engine can load and execute it quickly.

[0049] S402. Use the calculation engine to calculate the first original data and the second original data to obtain the first derived value and the second derived value.

[0050] In this embodiment, the first and second cleaned data are encapsulated into data structures recognizable by the computing engine and passed to the compiled task code through the data input interface. The computing engine executes the calculation operations according to the logical order in the task code, first performing basic operations on the original fields of the leaf nodes, and then progressively performing aggregation and combination operations on the intermediate nodes upwards, finally obtaining the derived values ​​corresponding to the root node, namely the first and second derived values. During the calculation process, the calculation progress is monitored in real time. If abnormalities such as data type mismatch or numerical overflow occur, the calculation is immediately paused and an exception log is returned for troubleshooting. After the calculation is completed, the derived values ​​are stored as high-precision numerical types, retaining sufficient decimal places.

[0051] S403. Calculate the absolute difference between the first derived value and the second derived value.

[0052] In this embodiment, the calculated first and second derived values ​​are obtained. First, the numerical type and precision of the two derived values ​​are checked for consistency to ensure they are values ​​of the same type and precision (e.g., both are Double type, retaining four decimal places). Then, the absolute difference calculation formula is used: Absolute difference value = |First derived value - Second derived value|. The calculation is performed using a high-precision numerical calculation library to avoid precision errors caused by traditional calculation methods. After calculation, the absolute difference value and the derived values ​​are stored together and associated with the corresponding task identifier and data source information to form a difference record. If the derived value is a relative indicator such as a percentage or ratio, the relative difference value (absolute difference value / baseline derived value) is calculated simultaneously to provide a multi-dimensional reference for subsequent consistency determination.

[0053] In this embodiment, the derived value calculation rules are parsed into a calculation logic tree, clearly decomposing complex calculation logic and avoiding logical ambiguity or calculation errors caused by directly executing the rules. Then, the calculation logic tree is compiled into task code that can be run by the calculation engine, realizing the automated and standardized execution of the calculation rules, which greatly improves the efficiency and accuracy of derived value calculation and solves the problems of error-proneness and poor adaptability of traditional manual calculation or script calculation. Finally, the derived values ​​of the two data sources are obtained through a unified calculation process, and the absolute difference value is accurately calculated, providing a quantitative basis for subsequent consistency judgment.

[0054] Please see Figure 5 The fifth embodiment of a cross-database data consistency verification method in this invention includes: S501. Compare the absolute difference value with the static consistency threshold. If the absolute difference value exceeds the static consistency threshold, based on the task identifier, query the historical statistical information of the verification task within the preset historical period from the preset historical knowledge base, and extract the attributes of the derived value calculation rule. The historical statistical information includes the average difference, difference variance, and historical inconsistency cases. The attributes of the derived value calculation rule include the rule complexity score, the type and number of operators involved.

[0055] In this embodiment, the difference value is compared with a static consistency threshold. If the difference value exceeds the threshold, a query request is immediately initiated to a pre-built historical knowledge base based on the task identifier. The pre-built historical knowledge base stores the verification records of all verification tasks within a preset historical period (e.g., 3 months). Historical data of the current verification task is filtered out, and historical statistics such as average difference value, difference variance, number of historical inconsistencies, and root cause distribution are obtained. Simultaneously, feature analysis is performed on the derived value calculation rules to extract core attributes such as rule complexity score (based on the number of fields involved and operator hierarchy calculation), the types and number of operators involved (e.g., arithmetic operators, logical operators, aggregation operators), and whether conditional judgment logic is included. The historical statistical information is associated with the calculation rule attributes and stored to provide data support for subsequent comprehensive vector construction.

[0056] S502. Construct a comprehensive vector based on the attributes of the first derived value, the second derived value, the first data vector, the second data vector, historical statistical information, and the derived value calculation rules.

[0057] In this embodiment, a comprehensive feature vector is constructed by integrating the attributes of the first derived value, the second derived value, the absolute difference value, the first data vector, the second data vector, historical statistical information, and the derived value calculation rules. First, various data types are standardized: numerical features such as derived values, absolute difference values, and historical average differences are standardized using Z-scores; categorical features (such as operator types) in the calculation rule attributes are one-hot encoded; and count features such as the number of historical inconsistencies are normalized. Then, according to a preset feature order, all processed feature data are concatenated into a one-dimensional comprehensive vector. The vector dimension is dynamically adjusted according to the number of features to ensure that all key information is included. After construction, the comprehensive vector is validated to check for missing or outlier values. If any are found, interpolation or outlier removal is used to correct them, ensuring that the vector accurately reflects the consistency-related features of cross-database data.

[0058] S503. Input the comprehensive vector into the pre-trained probabilistic evaluation model to obtain the probability value and root cause classification of the risk of inconsistency at the business logic level in the representation output by the pre-trained probabilistic evaluation model.

[0059] In this embodiment, the comprehensive vector is input into a pre-trained probabilistic evaluation model to obtain the probability value and root cause classification of the inconsistency risk at the business logic level in the representation output by the pre-trained probabilistic evaluation model. Specifically, this includes: constructing and training a probabilistic evaluation model based on a lightweight gradient boosting tree algorithm; standardizing and normalizing the comprehensive vector to obtain a pre-processed vector; and inputting the comprehensive vector into the pre-trained probabilistic evaluation model to obtain the probability value and root cause classification of the inconsistency risk at the business logic level in the representation output by the probabilistic evaluation model.

[0060] In this embodiment, Lightweight Gradient Boosting Tree (LightGBM) is selected as the basic model architecture. This model employs a histogram-based decision tree algorithm, which has the advantages of fast training speed and low memory consumption. First, a massive amount of cross-database data consistency verification samples are collected. These samples include comprehensive vectors (features), inconsistent business logic labeling results (labels, 0 = no risk, 1 = risky), and corresponding root cause classifications (labels). The samples are divided into training and validation sets in an 8:2 ratio. Data augmentation is performed on the training set to improve the model's generalization ability. Then, model hyperparameters (such as learning rate, tree depth, and number of leaf nodes) are configured, and the model is trained using a five-fold cross-validation method. During training, AUC (area under the curve) is used as the evaluation metric for risk prediction, and accuracy is used as the evaluation metric for root cause classification. Hyperparameters are continuously optimized until the model performance on the validation set reaches the preset standards (e.g., AUC ≥ 0.95, accuracy ≥ 0.9). The model training is then completed, and the parameters are fixed, resulting in a probabilistic evaluation model.

[0061] In this embodiment, the constructed composite vector undergoes a two-step processing: The first step is standardization, using the Z-score standardization method to transform numerical features (such as derived values, variance values, and historical statistical values) in the vector, ensuring that the mean of each feature is 0 and the variance is 1, eliminating the influence of dimensional differences. The second step is normalization, using the Min-Max normalization method to map the standardized features and other types of features (such as encoded classification features) to the [0,1] interval, ensuring that the numerical range of all features is consistent and avoiding excessive sensitivity of the probabilistic evaluation model to some features due to large numerical differences. During preprocessing, the standardization parameters (mean, variance) and normalization parameters (minimum, maximum) of each feature are recorded for unified processing of subsequent new samples. After processing, a preprocessed vector is obtained, which has the characteristics of uniform feature distribution and uniform numerical range, effectively improving the prediction accuracy of the model.

[0062] In this embodiment, the preprocessed vector is input into a pre-trained probabilistic evaluation model. The probabilistic evaluation model performs feature calculations and logical judgments on the vector using pre-defined parameters and decision rules. In the risk prediction branch, the probabilistic evaluation model outputs a probability value representing the risk of inconsistency at the business logic level. This value is mapped to the [0,1] interval using the Sigmoid function, intuitively reflecting the risk level. In the root cause classification branch, the probabilistic evaluation model uses multi-classification logic to output the probability distribution of each root cause category, selecting the category with the highest probability as the root cause classification. After outputting the results, the results are validated for reasonableness. If there is a logical contradiction between the risk probability value and the root cause classification (e.g., low probability risk corresponds to a serious root cause), the result is marked as suspicious and prompts manual review to ensure the reliability of the output results.

[0063] In this embodiment, when the difference exceeds the threshold, the historical statistical information of the verification task is actively associated, and the historical data patterns are integrated to improve the reliability of the assessment, avoiding the one-sidedness of relying solely on the current data. Furthermore, the core attributes of the derived value calculation rules are extracted, and the calculation logic features are incorporated into the assessment dimension, making the assessment more aligned with business scenarios. By integrating multi-dimensional data features to construct a comprehensive vector, comprehensive input information is provided to the pre-trained model, ensuring that the model can make comprehensive judgments from multiple aspects such as data differences, historical patterns, and calculation logic. Finally, the model outputs the inconsistency risk probability value and root cause classification at the business logic level, achieving an upgrade from data difference identification to business risk diagnosis. This effectively solves the pain points of traditional verification, which struggles to distinguish between data errors and business logic errors, and is difficult to locate root causes. It provides operation and maintenance personnel with precise troubleshooting directions and significantly improves the efficiency of resolving cross-database data consistency issues.

[0064] Please see Figure 6 The sixth embodiment of a cross-database data consistency verification method in this invention includes: S601. Match the probability value with the risk level threshold in the consistency judgment strategy to determine the final judgment result as consistent, inconsistent, or pending.

[0065] The consistency assessment strategy retrieves preset risk level thresholds, typically divided into three levels: low-risk threshold (e.g., 0.3) and high-risk threshold (e.g., 0.7). The risk probability value output by the probabilistic assessment model is compared with the threshold: if the probability value ≤ low-risk threshold, it is considered consistent, indicating that the data difference is within normal error and there is no business logic inconsistency risk; if the probability value ≥ high-risk threshold, it is considered inconsistent, indicating a clear business logic inconsistency problem; if low-risk threshold < probability value < high-risk threshold, it is considered pending, indicating that the data difference has potential risk and requires further manual review. The final assessment result is stored in association with the risk probability value and root cause classification.

[0066] S602. Combining probability values, root cause classification, and final judgment results, generate a verification report. The verification report includes a summary of differences, risk level, root cause prediction, and treatment recommendations.

[0067] Following a pre-defined report template, the final judgment results, risk probability values, root cause classifications, and key data from the verification process are integrated to generate a structured verification report. The core content of the report includes a difference summary (names of both data sources, business metrics, derived values, absolute difference values, and thresholds), risk levels (labeled as low risk, high risk, or potential risk based on the judgment results), root cause inferences (detailed explanation of possible root cause categories and inference basis, such as data source synchronization delays, with 30% of historical inconsistency cases being this root cause, and the current difference value highly matching the characteristics of historical synchronization delay cases), and handling recommendations (providing specific operational guidance for root cause classifications, such as recommending checking the synchronization scheduling tasks of data source A and data source B to confirm whether there are execution failures or delays).

[0068] S603. Store the verification report in a structured format and trigger alarms according to preset alarm rules.

[0069] The generated structured verification reports are stored in a distributed database, using a naming convention of task identifier + timestamp to ensure report uniqueness and traceability. The database supports CRUD operations on reports and provides access control, allowing only authorized personnel to view or modify reports. Simultaneously, corresponding alarms are triggered according to preset alarm rules: if the determination result is inconsistent, an email and pop-up alarm are immediately sent to the operations manager and business manager, along with a report link, requesting urgent handling; if the determination result is pending, a reminder notification is sent to operations personnel, requiring them to complete the review within a specified time (e.g., 24 hours); if the determination result is consistent, only a log is recorded, without triggering an alarm. Alarm information includes the task identifier, risk level, and core discrepancies, facilitating quick understanding of the situation and prompt action by relevant personnel, ensuring timely handling of cross-database data consistency issues.

[0070] In this embodiment, by precisely matching the risk probability value with the risk level threshold in the judgment strategy, a three-level judgment of consistency, inconsistency, and pending is achieved, avoiding the rigidity of a single threshold judgment and making the judgment results more in line with actual business scenarios. Moreover, by generating a structured verification report containing a difference summary, risk level, root cause inference, and handling suggestions, not only are the verification results presented, but also the problem analysis and solution direction are provided, solving the problem of traditional reports having limited information and weak guidance. Finally, the report is stored in a structured manner and alarms are triggered according to preset alarm rules to ensure that the verification results are traceable and queryable, while enabling relevant personnel to respond to high-risk issues in a timely manner.

[0071] The cross-database data consistency verification method in the embodiments of the present invention has been described above. The apparatus in the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 7 The implementation methods of the cross-database data consistency verification device in this invention include: The parsing module 701 is used to obtain the verification task and, based on the verification task, obtain the task identifier, the connection information of the first data source and the second data source corresponding to the same business indicator, the derived value calculation rule, the static consistency threshold, and the consistency judgment strategy. The conversion module 702 is used to obtain, based on the connection information, the first original data and the second original data required for calculating the derived value from the first data source and the second data source respectively, and convert the first original data and the second original data into a first data vector and a second data vector. The calculation module 703 is used to calculate a first derived value using the first original data based on the derived value calculation rules, and to calculate a second derived value using the second original data, and to calculate the absolute difference between the first derived value and the second derived value. Evaluation module 704 is used to compare the absolute difference value with the static consistency threshold. If the absolute difference value exceeds the static consistency threshold, the historical statistical information of the verification task is obtained based on the task identifier, and the attributes of the derived value calculation rule are extracted. Based on the first derived value, the second derived value, the first data vector, the second data vector, the historical statistical information, and the attribute historical statistical information of the derived value calculation rule, a pre-trained probabilistic evaluation model is used to evaluate the probability value and root cause classification that represent the risk of inconsistency at the business logic level. The generation module 705 is used to generate a final judgment result based on the probability value and the consistency judgment strategy, and to generate a verification report based on the probability value, the root cause classification and the final judgment result.

[0072] In this embodiment, the parsing module 701 includes: an acquisition unit 7011, used to acquire and parse the verification task to obtain a task identifier; a reading unit 7012, used to retrieve and read the corresponding structured task configuration file from a preset configuration management center based on the task identifier; and a first parsing unit 7013, used to parse the structured task configuration file to obtain multiple fields, and extract from the multiple fields the connection information of the first data source and the second data source corresponding to the same business indicator, the derived value calculation rules, the static consistency threshold, and the consistency judgment strategy.

[0073] In this embodiment, the conversion module 702 includes: a query unit 7021, used to generate query requests for the first data source and the second data source based on the connection information to obtain the first original data and the second original data required for calculating the derived value; a cleaning unit 7022, used to clean the first original data and the second original data to obtain the first cleaned data and the second cleaned data; and a conversion unit 7023, used to use feature engineering methods to convert the first cleaned data and the second cleaned data into a first data vector and a second data vector with the same dimension, respectively.

[0074] In this embodiment, the calculation module 703 includes: a second parsing unit 7031, used to parse the derived value calculation rules into a calculation logic tree and compile the calculation logic tree into task code that can be run in the calculation engine; a first calculation unit 7032, used to use the calculation engine to calculate the first original data and the second original data to obtain the first derived value and the second derived value; and a second calculation unit 7033, used to calculate the absolute difference between the first derived value and the second derived value.

[0075] In this embodiment, the evaluation module 704 includes: a comparison unit 7041, used to compare the absolute difference value with the static consistency threshold; if the absolute difference value exceeds the static consistency threshold, then based on the task identifier, query the historical statistical information of the verification task within a preset historical period from a preset historical knowledge base, and extract the attributes of the derived value calculation rule. The historical statistical information includes the average difference, difference variance, and historical inconsistency cases. The attributes of the derived value calculation rule include the rule complexity score, the type and number of operators involved; a construction unit 7042, used to construct a comprehensive vector based on the first derived value, the second derived value, the first data vector, the second data vector, the historical statistical information, and the attributes of the derived value calculation rule; and an evaluation unit 7043, used to input the comprehensive vector into a pre-trained probabilistic evaluation model, and obtain the probability value and root cause classification of the risk of inconsistency at the business logic level as represented by the output of the pre-trained probabilistic evaluation model.

[0076] In this embodiment, the generation module 705 includes: a matching unit 7051, used to match the probability value with the risk level threshold in the consistency determination strategy to determine the final determination result as consistent, inconsistent, or pending; a generation unit 7052, used to combine the probability value, the root cause classification, and the final determination result to generate a verification report, the verification report including a difference summary, risk level, root cause inference, and processing suggestions; and an alarm unit 7053, used to store the verification report in a structured format and trigger an alarm according to a preset alarm rule.

[0077] In this embodiment, a two-tiered progressive architecture of fast static threshold filtering and in-depth evaluation using a probabilistic model is introduced. First, a large number of tasks with no significant differences are efficiently screened out using a preset threshold, which greatly saves computing resources and ensures the efficiency of basic verification. For potential anomalies that exceed the threshold, a multi-dimensional and context-rich comprehensive judgment basis is formed by integrating the current difference value, the vectorized representation of the original data, the historical statistical baseline, and the attribute features of the derived rules themselves. The pre-trained probabilistic evaluation model performs in-depth analysis based on this, and its output of the inconsistency risk probability value and root cause classification probability distribution fundamentally changes the limitations of traditional binary judgment, providing quantitative and interpretable decision support for operation and maintenance personnel, and significantly improving the accuracy and operability of alarms.

[0078] Figure 7 The structure of the cross-database data consistency verification device shown does not constitute a limitation on the cross-database data consistency verification device, and can implement the steps of the cross-database data consistency verification methods provided in the above method embodiments.

[0079] above Figure 7The cross-database data consistency verification device in this embodiment of the invention is described in detail from the perspective of modular functional entities. The cross-database data consistency verification device in this embodiment of the invention is described in detail below from the perspective of hardware processing.

[0080] Figure 8 This is a schematic diagram of a cross-database data consistency verification device provided in an embodiment of the present invention. The device 800 can vary significantly due to different configurations or performance characteristics. It may include one or more central processing units (CPUs) 810 (e.g., one or more processors) and a memory 820, and one or more storage media 830 (e.g., one or more mass storage devices) for storing application programs 833 or data 832. The memory 820 and storage media 830 can be temporary or persistent storage. The program stored in the storage media 830 may include one or more modules (not shown), each module including a series of instruction operations on the device 800. Furthermore, the processor 810 may be configured to communicate with the storage media 830 and execute the series of instruction operations on the storage media on the device 800.

[0081] Device 800 may also include one or more power supplies 840, one or more wired or wireless network interfaces 850, one or more input / output interfaces 860, and / or one or more operating systems 831, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc.

[0082] This invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. The computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the steps of a cross-database data consistency verification method.

[0083] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system, device, or unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0084] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0085] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for cross-database data consistency verification, characterized in that, The cross-database data consistency verification method includes: Obtain the verification task, and based on the verification task, obtain the task identifier, the connection information of the first data source and the second data source corresponding to the same business indicator, the derived value calculation rule, the static consistency threshold, and the consistency judgment strategy. Based on the connection information, the first original data and the second original data required for calculating the derived value are obtained from the first data source and the second data source, respectively, and the first original data and the second original data are converted into a first data vector and a second data vector. Based on the derived value calculation rules, a first derived value is calculated using the first original data, and a second derived value is calculated using the second original data. The absolute difference between the first derived value and the second derived value is then calculated. The absolute difference value is compared with the static consistency threshold. If the absolute difference value exceeds the static consistency threshold, the historical statistical information of the verification task is obtained based on the task identifier, and the attributes of the derived value calculation rule are extracted. Based on the first derived value, the second derived value, the first data vector, the second data vector, the historical statistical information, and the attribute historical statistical information of the derived value calculation rule, a pre-trained probabilistic evaluation model is used to evaluate the probability value and root cause classification that represent the risk of inconsistency at the business logic level. A final determination result is generated based on the probability value and the consistency determination strategy, and a verification report is generated based on the probability value, the root cause classification, and the final determination result.

2. The cross-database data consistency verification method according to claim 1, characterized in that, The process of obtaining the verification task includes, based on the verification task, obtaining the task identifier, connection information of the first data source and the second data source corresponding to the same business indicator, derived value calculation rules, static consistency threshold, and consistency determination strategy, including: Obtain and parse the verification task to get the task identifier; Based on the task identifier, the corresponding structured task configuration file is retrieved and read from the preset configuration management center; The structured task configuration file is parsed to obtain multiple fields, and the connection information of the first data source and the second data source corresponding to the same business indicator, the derived value calculation rules, the static consistency threshold, and the consistency judgment strategy are extracted from the multiple fields.

3. The cross-database data consistency verification method according to claim 1, characterized in that, The step of obtaining the first and second original data required for calculating the derived value from the first and second data sources respectively based on the connection information, and converting the first and second original data into a first data vector and a second data vector, includes: Based on the connection information, a query request is generated for the first data source and the second data source to obtain the first raw data and the second raw data required to calculate the derived value; The first raw data and the second raw data are cleaned to obtain first cleaned data and second cleaned data. Using feature engineering methods, the first cleaned data and the second cleaned data are respectively converted into a first data vector and a second data vector with the same dimension.

4. The cross-database data consistency verification method according to claim 1, characterized in that, The step of calculating a first derived value using the first original data and a second derived value using the second original data, based on the derived value calculation rule, and calculating the absolute difference between the first derived value and the second derived value, includes: The derived value calculation rules are parsed into a calculation logic tree, and the calculation logic tree is compiled into task code that can run in the calculation engine; The calculation engine is used to calculate the first and second original data to obtain the first derived value and the second derived value. Calculate the absolute difference between the first derived value and the second derived value.

5. The cross-database data consistency verification method according to claim 1, characterized in that, The absolute difference value is compared with a static consistency threshold. If the absolute difference value exceeds the static consistency threshold, historical statistical information of the verification task is obtained based on the task identifier, and the attributes of the derived value calculation rule are extracted. Based on the first derived value, the second derived value, the first data vector, the second data vector, the historical statistical information, and the attribute historical statistical information of the derived value calculation rule, a pre-trained probabilistic evaluation model is used to evaluate the probability value and root cause classification representing the risk of inconsistency at the business logic level, including: The absolute difference value is compared with the static consistency threshold. If the absolute difference value exceeds the static consistency threshold, then based on the task identifier, the historical statistical information of the verification task within the preset historical period is queried from the preset historical knowledge base, and the attributes of the derived value calculation rule are extracted. The historical statistical information includes the average difference, the difference variance, and historical inconsistency cases. The attributes of the derived value calculation rule include the rule complexity score, the type and number of operators involved. A comprehensive vector is constructed based on the attributes of the first derived value, the second derived value, the first data vector, the second data vector, the historical statistical information, and the derived value calculation rules; The comprehensive vector is input into a pre-trained probabilistic evaluation model to obtain the probability value and root cause classification of the risk of inconsistency at the business logic level in the output of the pre-trained probabilistic evaluation model.

6. The cross-database data consistency verification method according to claim 5, characterized in that, The step of inputting the comprehensive vector into a pre-trained probabilistic evaluation model to obtain the probability value and root cause classification of the inconsistency risk at the business logic level represented by the output of the pre-trained probabilistic evaluation model includes: A probabilistic evaluation model was constructed and trained based on a lightweight gradient boosting tree algorithm. The composite vector is standardized and normalized to obtain a preprocessed vector; The preprocessed vector is input into a pre-trained probabilistic evaluation model to obtain the probability value and root cause classification of the risk of inconsistency at the business logic level represented by the output of the probabilistic evaluation model.

7. The cross-database data consistency verification method according to claim 1, characterized in that, The process of generating a final determination result based on the probability value and the consistency determination strategy, and generating a verification report based on the probability value, the root cause classification, and the final determination result, includes: The probability value is matched with the risk level threshold in the consistency determination strategy to determine the final determination result as consistent, inconsistent, or pending. Based on the probability value, the root cause classification, and the final determination result, a verification report is generated, which includes a difference summary, risk level, root cause inference, and treatment recommendations. The verification report is stored in a structured format, and alarms are triggered according to preset alarm rules.

8. A cross-database data consistency verification device, characterized in that, include: The parsing module is used to obtain the verification task and, based on the verification task, obtain the task identifier, the connection information of the first data source and the second data source corresponding to the same business indicator, the derived value calculation rule, the static consistency threshold, and the consistency judgment strategy. The conversion module is used to obtain, based on the connection information, the first original data and the second original data required for calculating the derived value from the first data source and the second data source respectively, and convert the first original data and the second original data into a first data vector and a second data vector. The calculation module is used to calculate a first derived value using the first original data based on the derived value calculation rules, and to calculate a second derived value using the second original data, and to calculate the absolute difference between the first derived value and the second derived value. The evaluation module is used to compare the absolute difference value with the static consistency threshold. If the absolute difference value exceeds the static consistency threshold, the historical statistical information of the verification task is obtained based on the task identifier, and the attributes of the derived value calculation rule are extracted. Based on the first derived value, the second derived value, the first data vector, the second data vector, the historical statistical information, and the attribute historical statistical information of the derived value calculation rule, a pre-trained probabilistic evaluation model is used to evaluate the probability value and root cause classification that represent the risk of inconsistency at the business logic level. The generation module is used to generate a final judgment result based on the probability value and the consistency judgment strategy, and to generate a verification report based on the probability value, the root cause classification and the final judgment result.

9. A cross-database data consistency verification device, characterized in that, It includes a memory and at least one processor, wherein the memory stores computer-readable instructions; The at least one processor invokes the computer-readable instructions in the memory to perform the steps of the cross-database data consistency verification method as described in any one of claims 1-7.

10. A computer-readable storage medium storing computer-readable instructions thereon, characterized in that, When the computer-readable instructions are executed by a processor, they implement the steps of the cross-database data consistency verification method as described in any one of claims 1-7.

Images