Multi-data-source automatic acquisition and comparison system

Abstract

The application discloses a multi-data-source automatic acquisition and comparison system, relates to the technical field of data processing and automatic test, and comprises the following steps: a scheduling center generates a task context containing a task identifier and a time range; an intelligent sampler determines a sample set based on a strategy-driven random algorithm and a hash modulo operation; a data extraction agent acquires multi-source heterogeneous data in parallel through a unified interface and encapsulates the multi-source heterogeneous data into a virtual data context; a rule execution engine loads a declarative rule, converts data into standard semantic values, constructs a rule execution topology graph, traverses nodes according to a dependency relationship to execute logical assertions or blockages; and a report generation module outputs a diagnostic report by summarizing checking results and original data snapshots. Through full-link data tracking and topological rule checking, the application realizes accurate automatic comparison of multi-source heterogeneous data consistency in a distributed environment.

Description

Technical Field

[0001] This invention relates to the field of data processing and automated testing technology, specifically to a multi-data source automatic acquisition and comparison system. Background Technology

[0002] With the rapid development of Internet technology, modern software system architecture has become increasingly complex, gradually evolving from monolithic architecture to distributed and microservice architecture. When dealing with core business such as order flow, fund settlement or inventory management, a complete business action often requires cross-component data interaction and flow between multiple independent application modules, relational databases, memory caches and message middleware. Data is constantly generated, transmitted, transformed and persisted in these heterogeneous components.

[0003] To ensure the accuracy of this business data, testers typically use assertions on the status codes or key fields returned by the API to determine whether the business is successful, or they write fixed database query scripts to verify some core tables after the test. At the same time, the operations and maintenance team also deploys monitoring systems to track database performance metrics, such as slow queries or deadlocks, to indirectly assess the health of the system.

[0004] However, with the surge in business concurrency, the flow of data between heterogeneous data sources inevitably introduces time delays. This inherent eventual consistency characteristic of distributed systems makes existing static verification methods prone to misjudgment: comparing data at the moment before it is fully synchronized will cause the verification logic to incorrectly report data inconsistency, generating a large number of false alarms. Conversely, if the verification conditions are relaxed or only manual sampling is performed in order to avoid false alarms, it is easy to miss deep data pollution caused by logical defects or network jitter in the face of massive amounts of data.

[0005] This makes it difficult for existing technologies to adapt to frequent changes in business logic while establishing an efficient data quality assurance mechanism that can perform end-to-end full-link tracing across multiple heterogeneous storage systems and intelligently distinguish between normal system latency and real logical errors. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides an automatic data acquisition and comparison system for multiple data sources.

[0007] To achieve the above objectives, the technical solution of the present invention is as follows: In a first aspect, the present invention discloses an automatic data source acquisition and comparison system, comprising: The scheduling center is used to obtain preset scheduling configuration information and generate a task context containing a unique task identifier and a time range to be verified based on the scheduling configuration information. The intelligent sampler is used to select candidate datasets from the business data stream within the time range to be verified based on the task context and a preset strategy-driven random algorithm. It then performs hash and modulo operations on the business primary keys in the candidate datasets to determine the sample set that meets the preset coverage factor. The data extraction agent is used to obtain multi-source heterogeneous data distributed in databases, caching systems and message queues in parallel through a unified data access interface based on each business primary key in the sample set, and encapsulate the multi-source heterogeneous data into a virtual data context. The rule execution engine is used to load a pre-defined declarative rule configuration model, which includes a virtual field mapping table and logical validation rules. Based on the virtual field mapping table, the multi-source heterogeneous data in the virtual data context is mapped and converted into standardized business semantic values; Based on the pre-dependencies between logical verification rules, a rule execution topology graph in the form of a directed acyclic graph with a hierarchical structure is constructed. Standardized business semantic values ​​are injected into the rule execution topology graph, and each rule node is traversed in topological order. During the traversal, it is determined whether the predecessor dependency node of the current rule node has been executed successfully. If so, the logical assertion of the current rule node is executed based on the injected business semantic value to obtain the verification result; otherwise, the execution of the current rule node and its child nodes is blocked. The report generation module is used to summarize the verification results of all executed nodes in the rule execution topology graph and the corresponding original data snapshots, generate a structured diagnostic report and output it.

[0008] Secondly, this invention discloses a method for automatic collection and comparison of multiple data sources, comprising the following steps: Obtain the preset scheduling configuration information, and generate a task context containing a unique task identifier and a time range to be verified based on the scheduling configuration information; Based on the task context, a random algorithm driven by a preset strategy is used to filter candidate datasets from the business data stream within the time range to be verified, and hash and modulo operations are performed on the business primary keys in the candidate datasets to determine the sample set that meets the preset coverage factor. Based on each business primary key in the sample set, multi-source heterogeneous data distributed in databases, caches and message queues are obtained in parallel through a unified data access interface, and the multi-source heterogeneous data is encapsulated into a virtual data context. Load the pre-defined declarative rule configuration model, which includes a virtual field mapping table and logical validation rules; Based on the virtual field mapping table, the multi-source heterogeneous data in the virtual data context is mapped and converted into standardized business semantic values; Based on the pre-dependencies between logical verification rules, a rule execution topology graph in the form of a directed acyclic graph with a hierarchical structure is constructed. Standardized business semantic values ​​are injected into the rule execution topology graph, and each rule node is traversed in topological order. During the traversal, it is determined whether the predecessor dependency node of the current rule node has been executed successfully. If so, the logical assertion of the current rule node is executed based on the injected business semantic value to obtain the verification result; otherwise, the execution of the current rule node and its child nodes is blocked. The system summarizes the verification results of all executed nodes in the rule execution topology graph, along with the corresponding original data snapshots, and generates and outputs a structured diagnostic report.

[0009] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. By using a strategy-driven intelligent sampling algorithm and hashing mechanism, combined with the avalanche effect of algorithms such as MurmurHash3, the coverage blind spot problem of traditional random sampling when the data distribution is skewed is solved, ensuring uniform coverage of the underlying physical shards; at the same time, an adaptive sampling probability based on the failure rate is introduced to realize intelligent focusing on high-risk business modules, which reduces system load and improves the efficiency of problem discovery. 2. A dynamic tolerance processing and logical clock alignment mechanism based on the business lifecycle was introduced. By calculating the entropy change pattern of business data, the comparison threshold is dynamically adjusted, and combined with an exponential backoff retry strategy with jitter, false alarms caused by eventual consistency delays in distributed systems are effectively eliminated, while avoiding the thundering herd effect caused by concurrent retries. 3. By using virtual data context and declarative rule model, physical storage and logical verification are decoupled, supporting unified collection and standardized normalization of various heterogeneous sources such as databases, caches, and message queues; at the same time, by constructing rule execution topology graph (DAG) and snapshot embedding technology, the complete restoration of the fault scene and accurate location of the root cause are achieved. 4. By using the circuit breaker mechanism of rule topology and the quantitative assessment based on weighted risk scores, redundant errors caused by failure of preceding dependencies are automatically filtered out, and hierarchical notifications are implemented according to the risk classification matrix, which greatly reduces the troubleshooting cost for operation and maintenance personnel and improves the automation level of data quality assurance. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a block diagram of the system architecture of Embodiment 1 of the present invention; Figure 2 This is a timing diagram of the intelligent sampling process in the system of Embodiment 1 of the present invention; Figure 3 This is a flowchart of the rule execution engine in the system of Embodiment 1 of the present invention; Figure 4 This is a flowchart illustrating the overall execution process of the method in Embodiment 2 of the present invention. Detailed Implementation

[0012] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] Application Overview: In the field of modern distributed software engineering, especially in complex business systems involving high-concurrency transactions and fund transfers, the eventual consistency and integrity of end-to-end data are regarded as core indicators for measuring system reliability. This high-quality data state is essentially an entropy reduction process of a closed business logic loop at the information entropy level. That is, through atomic operations triggered by front-end users (such as placing payment orders), data is driven to flow and persist in an orderly manner in heterogeneous storage media such as relational databases, memory caches, and message middleware, thereby leaving a business snapshot with a strict mathematical correspondence at the physical storage level.

[0014] However, existing technologies lack a deep verification mechanism for the synchronization of the state between the surface interface response and the underlying data persistence. This makes it impossible to accurately identify the temporal delay illusions and logical data loss problems that exist in the asynchronous processing of distributed systems. The temporal delay illusion manifests as the interface returning a successful response, but the backend data has not yet been synchronized due to network jitter or queue backlog, presenting a temporary inconsistency. Logical data loss manifests as the business process seemingly running, but due to code defects, specific branch data fails to be written to the database or messages are lost. As a result, a strict spatiotemporal correspondence cannot be established between the interface return status and the final data persistence, causing the monitoring system to misjudge or miss data quality, thereby affecting the accurate assessment of system stability and the timeliness of fault diagnosis.

[0015] For example, in real-world scenarios during large-scale e-commerce promotions, when a user initiates an order payment, existing monitoring systems can only capture a successful HTTP 200 response through conventional interface assertions, but cannot distinguish whether there are hidden data gaps in the background. Furthermore, when the system experiences master-slave database synchronization delays due to high load, the monitoring script only records the mismatch between the database and cached data values ​​at the current moment, failing to monitor data convergence trends and logical clock differences throughout the business lifecycle. Specifically, the system misjudges normal synchronization delays as serious data errors, triggering an alarm storm, or categorizes asynchronous write failures (such as inventory not being deducted) as normal processing time, leading to a sense of complacency among operations and maintenance personnel and an inability to identify truly fatal defects that undermine business integrity from massive amounts of data.

[0016] If the above issues are not addressed, the data quality assurance system will continuously lose its ability to objectively assess the health of the business. Specifically, unidentified time-series delays will lead to a proliferation of false alarms, masking real faults and forcing operations teams into a protracted and ineffective troubleshooting war. Simultaneously, failure to promptly correct logical data loss will cause silent data pollution, preventing financial reconciliation and inventory management from achieving a closed-loop system, ultimately leading to severe financial losses or customer complaints. Therefore, inaccurate data verification will systematically hinder enterprises from building a highly reliable distributed architecture, impacting the continued stable operation of core businesses.

[0017] Example 1: like Figures 1-3 As shown, the multi-data source automatic acquisition and comparison system includes: The scheduling center is used to obtain preset scheduling configuration information and generate a task context containing a unique task identifier and a time range to be verified based on the scheduling configuration information. As the startup hub of the entire multi-data source automatic collection and comparison system, the scheduling center first performs in-depth analysis and loading of the preset scheduling configuration information. To ensure that the system can adapt to the different demands of various business lines (such as payment, orders, and inventory) on real-time performance and system resource consumption, the scheduling center first reads the scheduling configuration information stored in the configuration center or database. This information includes key parameters such as business category identifier, task trigger cycle, peak business restriction period, and data accumulation delay threshold. After obtaining these parameters, the scheduling center does not immediately trigger the task. Instead, it uses a time-driven mechanism based on a time wheel or a distributed scheduled task framework (such as a Quartz cluster) to perform periodic scanning. Once the current system time triggers the preset cycle of a certain business category, the system immediately enters the calculation phase of the time range to be verified.

[0018] To address the "false inconsistency" problem arising from eventual consistency in distributed systems—that is, to avoid verification when data has just been generated but has not yet been synchronized across all heterogeneous components—the scheduling center does not simply select the current time when generating the time range to be verified. Specifically, the system calculates a retrospective time interval based on the current system time, the data settling delay threshold in the configuration information, and the verification window length. This ensures that data falling within this interval has theoretically completed its flow and resting in the database, cache, and message queue. Assume the current system time is... The preset data settling delay threshold is (Used to skip periods of high-frequency data change due to instability), the length of the verification window covered by a single task is [length missing]. Then the start time of the time range to be verified With end time The calculation formula is as follows: The closed interval obtained by the above formula This effectively mitigates the risk of false alarms caused by network jitter or asynchronous processing delays. After determining the time range, the dispatch center then executes the peak traffic avoidance logic, by judging the current time... Whether the task falls within the peak business restriction period defined in the scheduling configuration information, if it is within the restriction period, the generation of the task will be automatically suspended or postponed to prevent the computing resources occupied by the verification task from affecting the stability of the online core business. This boundary condition judgment ensures the robustness of the system and low intrusion into the production environment.

[0019] Once the time window calculation is complete and it's confirmed to be a non-peak business period, the scheduling center immediately enters the task context instantiation phase. To achieve full lifecycle traceability for each verification task, the system needs to generate a globally unique task identifier. This identifier generation doesn't solely rely on a simple UUID; for easier log retrieval and problem attribution, a combined coding logic is typically used, concatenating and hashing the business category identifier, the current timestamp, and a random salt value. Let the business category identifier be... The current timestamp is Random salt value Then the task unique identifier The generation logic can be expressed as: in This represents a string concatenation operation. This is a high-collision-resistance hash function such as SHA-256. Ultimately, the scheduling center will calculate the... Time interval and generation It is encapsulated as an immutable data object, namely the task context, and pushed to the downstream message channel or directly called the interface of the intelligent sampler, thereby completing the data flow from static configuration to dynamic execution instructions, providing accurate spatiotemporal coordinates and tracking credentials for subsequent intelligent sampling and multi-source data comparison.

[0020] The intelligent sampler is used to select candidate datasets from the business data stream within the time range to be verified based on the task context and a preset strategy-driven random algorithm. It then performs hash and modulo operations on the business primary keys in the candidate datasets to determine the sample set that meets the preset coverage factor. As the core decision-maker connecting scheduling instructions and actual data extraction, the intelligent sampler's execution begins with receiving and parsing the task context issued by the scheduling center. This context not only defines the time boundary to be verified but also implicitly contains the target business category identifier for this task. To overcome the resource waste or risk omission caused by the "one-size-fits-all" approach of traditional random sampling when facing business modules with different stability, the intelligent sampler first initiates an adaptive probability calculation process based on historical failure rates. This process queries diagnostic reports stored in the historical database to calculate the failure rate of the current business category identifier within the most recent preset period (e.g., the last 7 days). Based on this statistical data, the system dynamically weights the basic random probability using a preset sensitivity factor, thereby deriving the dynamic sampling probability for the current task to achieve an intelligent focusing effect of "the higher the failure rate, the denser the sampling." Assuming the target business category identifier is... The preset base random probability is Sensitivity factor is The statistical failure rate is Then the dynamic sampling probability The calculation formula is as follows: By introducing The function serves as a boundary condition, ensuring that the final probability value will not exceed 100%, thereby guaranteeing the numerical robustness of the computational logic under extreme failure conditions.

[0021] After determining the sampling intensity for this task, the intelligent sampler then enters the initial screening stage of the candidate dataset, based on the time range to be verified in the task context. By accessing the time index or Binlog index of the business database, data that meets the time requirements and matches the aforementioned dynamic sampling probability can be quickly identified from the massive business data stream. The initial data forms a candidate dataset. However, relying solely on time and probability-based filtering cannot guarantee the uniformity of sample distribution across the underlying physical storage. If the sampling results are concentrated on certain specific database shards, potential problems in other shards (such as data loss due to incorrect shard routing rule configuration) will be masked. Therefore, the intelligent sampler then performs a secondary filtering based on hashing and modulo operations on each business primary key in the candidate dataset, aiming to ensure uniform coverage of the underlying physical architecture through mathematical determinism. Specifically, the system first filters the business primary keys... Perform a high-collision-resistance hash operation, then compare the hash value with the total number of underlying data storage shards. Perform modulo operation to obtain the shard index value to which the data belongs, and determine whether the index value is within the preset sampling coverage factor. Within the defined interval. Its core discrimination logic is as follows: The business primary key will only be retained in the sample set if the above inequality holds true; otherwise, it will be removed.

[0022] It is worth noting that this system uses either MurmurHash3 or SHA-256 algorithms when performing hash operations. This is because the business primary key ( Hash values ​​often exhibit monotonically increasing characteristics (such as auto-incrementing IDs in databases) or specific prefixes (such as order number rules). Directly taking the modulo operation can lead to data skew. The aforementioned hash algorithms exhibit a significant avalanche effect, meaning that small changes in the input data (such as adding 1 to the last digit of the ID) can result in huge differences in the output hash value, thus dispersing consecutive business IDs evenly across the dataset. to In the numerical space, it is ensured that even with an extremely low sampling rate, samples can fall on each physical partition with a probability of near uniform distribution, avoiding the problem of "oversampling of hot partitions and omission of cold partitions".

[0023] To prevent the generated sample set from becoming too large in extremely high-concurrency scenarios, which could cause an OutOfMemory (OOM) error in the subsequent rule execution engine, the intelligent sampler performs a capacity circuit breaker check before outputting the final result. This check determines whether the size of the sample set exceeds the system's preset maximum sample threshold. If the number exceeds the limit, random truncation is performed again within the current set until the number meets the safety boundary. After the above-mentioned layer-by-layer processing from adaptive probability calculation, time-stream initial screening, physical sharding coverage verification to capacity circuit breaking, the intelligent sampler finally outputs a sample set that is both statistically representative and covers the risks of the underlying architecture. This set is then passed to the data extraction agent, thereby maximizing the utilization of computing resources while ensuring the depth of data quality monitoring.

[0024] The data extraction agent is used to obtain multi-source heterogeneous data distributed in databases, caching systems and message queues in parallel through a unified data access interface based on each business primary key in the sample set, and encapsulate the multi-source heterogeneous data into a virtual data context. The data extraction agent acts as a bridge connecting logical sampling and physical data execution. It eliminates the heterogeneity differences of the underlying storage media and solves the data acquisition latency problem caused by serial I / O in high-concurrency scenarios. When the data extraction agent receives the sample set from the intelligent sampler, to avoid the "lane congestion" effect caused by processing each sample one by one, the system first initializes an asynchronous I / O thread pool or coroutine scheduler based on the Reactor pattern. For each business primary key in the sample set, it concurrently creates an independent data acquisition task group. Within each task group, the data extraction agent calls a preset unified data access interface. This interface encapsulates driver adapters for different data sources, enabling it to simultaneously convert the same business primary key into specific query commands for relational databases (such as MySQL), key-value caches (such as Redis), and message queue indexes (such as Elasticsearch or RocketMQ Dashboard), thereby achieving a parallel fetching mechanism of "one trigger, multiple responses."

[0025] Specifically, when performing database-level crawling, the unified data access interface injects the business primary key into the query conditions according to a predefined SQL template. To prevent performance fluctuations caused by full table scans, the agent will forcibly check whether the query statement hits the primary key index or unique index of the physical table. Simultaneously, when performing cache-level crawling, the system uses preset key generation rules (such as "prefix:business ID") to directly locate key-value pairs in memory. For Redis in cluster mode, the agent will automatically calculate the slot to ensure the accuracy of routing. In the most complex message queue crawling stage, since the native message queue usually does not support random reading based on business ID, the data extraction agent will connect to a pre-built message trace store and quickly retrieve the message body (payload) and its metadata (such as generation time and topic offset) associated with the business primary key through the index.

[0026] To address the limitation of message queues not supporting random reads, the message trace store (Message TraceStore) is constructed using a dual-write asynchronous consumption architecture. This means that simultaneously with message sending by the business system, or through a separate consumer group, the mapping relationship between the message's key (business primary key) and MessageID / Offset is written in real-time to a columnar database (such as HBase) or search engine (such as Elasticsearch) that supports high-concurrency random read / write. When the data extraction agent crawls, it first queries this trace store to obtain the precise physical location, and then uses the seek (offset) mechanism to directly locate the message body, thereby reducing the time complexity to [missing information]. .

[0027] To ensure the stability of this parallel I / O operation, the system sets a strict timeout and circuit breaker threshold for each data source request. If the response time of a source exceeds the threshold, the agent will immediately interrupt the request and return a "null" placeholder to prevent a single point of failure from dragging down the entire comparison task.

[0028] Once all data sources have returned responses (or timeout signals), the data extraction agent enters the data encapsulation phase, which involves building a virtual data context. At this point, the data collected from various sources varies widely in format—databases return structured ResultRows, caches return serialized binary streams or JSON strings, and message queues may return Base64-encoded text.

[0029] At this stage, the data extraction agent also needs to perform encoding normalization. For text fields returned by the database, binary streams in Redis, and payloads in MQ, the system uniformly converts them to UTF-8 format strings, or while retaining the original binary, explicitly marks their original encoding format in the context metadata to prevent subsequent rule engines from failing verification due to encoding inconsistencies (such as GBK and UTF-8) when performing string comparison.

[0030] To provide a standardized input view for the subsequent rule execution engine, the data extraction agent encapsulates this raw data into a standardized container object without modification. Assume this is for a specific business primary key. , involving Heterogeneous data sources The obtained raw data are as follows: (If the fetch fails or times out, then...) Then the virtual data context The construction logic can be defined by the following set mapping formula: in, Identify the data source type (e.g., DB, Cache). Record the physical source path of the data (e.g., database name.table name). The original data payload after encapsulation. The status of this fetch is marked (success / timeout / not found). This encapsulation method ensures the complete preservation of the "original context", enabling subsequent comparison not only of data values ​​but also tracing the data's source and acquisition status. Finally, the encapsulated virtual data context is pushed in batches to the memory queue, awaiting consumption by the rule execution engine, thus completing the transformation from physical fragments to a logical whole.

[0031] The rule execution engine is used to load a pre-defined declarative rule configuration model, which includes a virtual field mapping table and logical validation rules. Based on the virtual field mapping table, the multi-source heterogeneous data in the virtual data context is mapped and converted into standardized business semantic values; Based on the pre-dependencies between logical verification rules, a rule execution topology graph in the form of a directed acyclic graph with a hierarchical structure is constructed. Standardized business semantic values ​​are injected into the rule execution topology graph, and each rule node is traversed in topological order. During the traversal, it is determined whether the predecessor dependency node of the current rule node has been executed successfully. If so, the logical assertion of the current rule node is executed based on the injected business semantic value to obtain the verification result; otherwise, the execution of the current rule node and its child nodes is blocked. Once the rule execution engine receives the encapsulated virtual data context from the data extraction agent, it first triggers the initialization loading process based on the declarative rule configuration model. To address the pain point of strong coupling between physical data and business logic, the engine does not directly manipulate the raw data; instead, it first parses the virtual field mapping table in the model. This table defines the mapping relationship from physical data paths (such as "db_order.t_payment.amount") to virtual business fields (such as "VO.Payment.Amount"). The engine then traverses the virtual data context, utilizing pre-defined atomic normalization operators. For raw physical data The cleaning process involves performing standardized operations, including unit unification (e.g., fraction to integer conversion), type conversion (e.g., String to Decimal), and null value handling, to obtain standardized business semantic values. This process can be expressed by the following formula: in For the target data type, This is a precision or unit conversion factor. This process ensures that all subsequent verification logic is built on a unified semantic plane, eliminating interference caused by differences in underlying storage.

[0032] After data standardization, to avoid a storm of invalid alerts caused by missing basic data, the rule execution engine needs to determine the correct order of rule execution. The engine reads the logical verification rules from the configuration, parses the input parameters of each rule to identify its dependencies, and then constructs a rule execution topology in the form of a directed acyclic graph (DAG). The construction logic follows the principle of "existence over consistency," setting the integrity verification rule that checks whether key fields are empty as the root node of the topology. Subsequently, it identifies consistency verification rules (such as database and cache comparisons) and business logic correctness verification rules (such as amount calculation formulas) that depend on the root node, establishing directed edges from the dependent rules to the dependent rules. This topology ensures that the verification process is progressive, with deeper logical deductions only performed when the underlying facts are true.

[0033] In terms of specific construction algorithms, the engine employs either the Kahn algorithm or a topological sorting algorithm based on Depth-First Search (DFS). The system first counts the in-degree of all rule nodes and adds the root node with an in-degree of 0 to the execution queue. As a node completes execution, logically, the in-degree of that node and its emanating edges is decremented by 1, and the in-degree of all subsequent nodes is decremented. Once the in-degree reaches zero, the node is added to the executable queue. This algorithm mechanism ensures that, in complex mesh dependencies, the execution order always strictly follows logical causality and can automatically detect and report circular dependency configurations.

[0034] After constructing the execution path, the engine injects standardized business semantic values ​​into the rule execution topology graph and begins traversing each rule node in topological order. Before actually executing the logical assertion of a particular node, to address the data synchronization latency issue unique to distributed systems, the engine first performs logical clock alignment processing. The system extracts the business version numbers carried by different data sources from the virtual data context. And calculate its maximum difference. If the maximum difference If the preset version synchronization threshold is exceeded, it indicates that the data between components has not been synchronized (e.g., the database has been updated but the cache has not expired). Forcing a comparison at this time will inevitably lead to false alarms. Therefore, the engine will pause the execution of the current node, suspend the current thread and enter a sleep state. After a preset time, a retry mechanism will be triggered to re-acquire data until the synchronization conditions are met or the maximum number of retries is reached. This ensures that the comparison operation is performed at the "same logical moment".

[0035] It is worth noting that when triggering the retry mechanism, to avoid the Thundering Herd Problem (where a large number of concurrent tasks wake up simultaneously and overwhelm the data source), the engine employs an exponential backoff strategy with jitter. That is, the... The calculation of the waiting time for the second retry introduces a random variable: This nondeterministic waiting strategy effectively breaks up retry requests over time, smoothing out the system's I / O load.

[0036] Once the version alignment check passes, the engine enters the assertion execution phase. Considering that data consistency requirements change dynamically during business state transitions, the engine introduces dynamic tolerance handling based on the business lifecycle. First, the engine extracts the master state field. (e.g., order status), using a preset drift function Calculate the dynamic tolerance threshold for the current scenario. For the two business semantic values ​​to be compared... and The system performs the following inequality check: when When in a high-frequency change state such as "processing", It is assigned a relatively large tolerance threshold, allowing for numerical fluctuations or time-related differences within a certain range; while when When the system is in the "completed" final state, the threshold is reset to zero, and strict matching is performed. This mechanism gives the system the intelligent ability to distinguish between "normal delays" and "real errors".

[0037] Specifically, the drift function It can be designed as a piecewise step function: when In the initial creation state, Set to the maximum value, and only perform existence checks; when in an intermediate steady state... Set to an allowable small error (e.g., 0.01); when in the final state, Set to 0. This segmented design accurately maps the entropy changes of business data at different lifecycle stages.

[0038] During the traversal, the engine consistently executes strict circuit breaker and blocking logic. For each rule node, the system checks whether its preceding dependent nodes have been executed successfully. If a preceding node fails, the engine identifies the error type: if it is due to missing physical data or dependency blocking, the current node and all its child nodes are directly marked as "Skipped" and not included in error statistics, thus achieving precise pruning; only when it is confirmed that a logical assertion has failed (i.e., the numerical comparison still exceeds the acceptable range even with complete data and version synchronization) is it marked as "Failed," and a snapshot of the current input data is captured.

[0039] For example, suppose the task is to verify an e-commerce order with a "payment successful" status, involving a MySQL database (source A) and a Redis cache (source B).

[0040] Loading and Mapping: The engine loads the configuration and converts t_order.pay_amt (unit: cents) from source A and cache:order:amt (unit: yuan) from source B into standard values ​​using atomic normalization operators. .

[0041] Topology construction: Construct a DAG, with the root node being "Order ID existence verification" and the child nodes being "Amount consistency verification".

[0042] Clock alignment: The engine checks the version number and finds source version A. Source B version The difference is 1. The preset version synchronization threshold is 3. No need to retry, continue execution.

[0043] Dynamic tolerance: Extract the master state field Order_Status=PAID (final state). Based on the drift function, the final state tolerance threshold is determined. .

[0044] Logical assertion: Perform computation The assertion passed.

[0045] Output: If the amount in source B is 99.00, the assertion fails, and the engine will record a "amount mismatch" error, along with... And the Order_Status snapshot at this time, generate the final report.

[0046] The report generation module is used to summarize the verification results of all executed nodes in the rule execution topology graph and the corresponding original data snapshots, generate a structured diagnostic report and output it.

[0047] The report generation module, as the final output of the entire automated data collection and comparison system, transforms the discrete and complex topology traversal states generated by the rule execution engine into structured, readable, and traceable diagnostic conclusions. After the rule execution engine completes its traversal of all sample sets, the report generation module first initiates the result aggregation and noise reduction process. Because rule execution employs a directed acyclic graph (DAG) structure, a single fundamental failure (such as a database connection failure) can cause dozens of subsequent dependent rule nodes to be "blocked" or "skipped." Directly outputting the status of all nodes would result in a report filled with invalid and redundant information. Therefore, the report generation module traverses the metadata of all executed nodes in the rule execution topology graph and performs logical filtering based on error type: for nodes marked as "skipped," the system defaults to folding or hiding them, retaining only the root cause node that triggered the blocking mechanism (i.e., nodes with missing physical data or blocked prerequisites) and all nodes with failed logical assertions. This aggregation logic ensures that the report directly addresses the core issue, avoiding the interference of an "alarm storm" on maintenance personnel.

[0048] After performing logical noise reduction, the report generation module needs to quantitatively assess the health of each business sample to facilitate subsequent alarm classification. The system introduces a calculation logic based on weighted risk scores for each sample. The system iterates through all rule nodes that are judged as "failed". According to the preset weights in the rule configuration (For example, rules related to funds have higher weights, while rules related to presentation have lower weights) Calculate the risk score for this sample. The calculation formula is as follows: in, For rules The corresponding severity coefficient (e.g., general error = 1, severe error = 10). If If the alarm threshold is exceeded, the sample will be marked as "Abnormal"; otherwise, it will only be marked as "Warning" or "Pass".

[0049] To facilitate cross-business line comparisons, the report generation module calculates the original risk scores. Then, it will be mapped against a preset risk classification matrix. This matrix uses the error severity coefficient and the frequency of error occurrence as two dimensions to divide the diagnostic results into four levels: P0 (interruption fault), P1 (critical fault), P2 (general warning), and P3 (indication message). The alarm system will decide whether to trigger a telephone voice alarm or only send an instant message based on the level.

[0050] Next, the report generation module performs a crucial snapshot embedding operation. The system extracts the original data snapshot of the sample at the time of verification (including the original physical data and standardized business semantic values) from the virtual data context, serializes this data into JSON or Protocol Buffers format, and embeds it into the details field of the diagnostic report. To prevent the leakage of sensitive data (such as user mobile phone numbers and ID cards), a pre-built masking filter is also invoked during this process to mask specific fields, thereby providing a complete "fault scene" reconstruction while ensuring data security.

[0051] Ultimately, the generated structured data is rendered into an easy-to-read diagnostic report. This report includes not only task-level statistics (such as total time, coverage, and pass rate) but also detailed attribution analysis at the sample level. The report generation module pushes this structured object to persistent storage (such as Elasticsearch or MongoDB) via an asynchronous message queue for historical review. Simultaneously, it triggers different levels of notification strategies (such as email, DingTalk, and SMS) based on the calculated risk score. This entire process achieves value transformation from underlying data verification to upper-level business decision-making, ensuring that technical metrics are translated into quality insights that business personnel can understand.

[0052] Example 2: like Figure 4 As shown, the method for automatic collection and comparison of multiple data sources includes the following steps: Obtain the preset scheduling configuration information, and generate a task context containing a unique task identifier and a time range to be verified based on the scheduling configuration information; Based on the task context, a random algorithm driven by a preset strategy is used to filter candidate datasets from the business data stream within the time range to be verified, and hash and modulo operations are performed on the business primary keys in the candidate datasets to determine the sample set that meets the preset coverage factor. Based on each business primary key in the sample set, multi-source heterogeneous data distributed in databases, caches and message queues are obtained in parallel through a unified data access interface, and the multi-source heterogeneous data is encapsulated into a virtual data context. Load the pre-defined declarative rule configuration model, which includes a virtual field mapping table and logical validation rules; Based on the virtual field mapping table, the multi-source heterogeneous data in the virtual data context is mapped and converted into standardized business semantic values; Based on the pre-dependencies between logical verification rules, a rule execution topology graph in the form of a directed acyclic graph with a hierarchical structure is constructed. Standardized business semantic values ​​are injected into the rule execution topology graph, and each rule node is traversed in topological order. During the traversal, it is determined whether the predecessor dependency node of the current rule node has been executed successfully. If so, the logical assertion of the current rule node is executed based on the injected business semantic value to obtain the verification result; otherwise, the execution of the current rule node and its child nodes is blocked. The system summarizes the verification results of all executed nodes in the rule execution topology graph, along with the corresponding original data snapshots, and generates and outputs a structured diagnostic report.

[0053] The above description is merely an example and illustration of the structure of the present invention. Those skilled in the art can make various modifications or additions to the specific embodiments described, or use similar methods to replace them, as long as they do not deviate from the structure of the invention or exceed the scope defined in the claims, all of which should fall within the protection scope of the present invention.

[0054] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0055] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to any specific implementation. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A multi-data source automatic acquisition and comparison system, characterized in that, include: The scheduling center is used to obtain preset scheduling configuration information and generate a task context containing a unique task identifier and a time range to be verified based on the scheduling configuration information. The intelligent sampler is used to select candidate datasets from the business data stream within the time range to be verified by driving a random algorithm according to the task context and a preset strategy, and to perform hash and modulo operations on the business primary keys in the candidate datasets to determine the sample set that meets the preset coverage factor. A data extraction agent is used to obtain multi-source heterogeneous data distributed in databases, cache systems and message queues in parallel through a unified data access interface based on each business primary key in the sample set, and to encapsulate the multi-source heterogeneous data into a virtual data context. A rule execution engine is used to load a pre-defined declarative rule configuration model, which includes a virtual field mapping table and logical validation rules. Based on the virtual field mapping table, the multi-source heterogeneous data in the virtual data context is mapped and converted into standardized business semantic values; based on the pre-dependencies between the logical verification rules, a rule execution topology graph in the form of a directed acyclic graph with a hierarchical structure is constructed. The standardized business semantic values ​​are injected into the rule execution topology graph, and each rule node is traversed in topological order. During the traversal, it is determined whether the predecessor dependency node of the current rule node has been executed. If so, the logical assertion of the current rule node is executed based on the injected business semantic value to obtain the verification result. Otherwise, block the execution of the current rule node and its child nodes; The report generation module is used to summarize the verification results of all executed nodes in the rule execution topology graph and the corresponding original data snapshots, generate a structured diagnostic report and output it.

2. The multi-data source automatic acquisition and comparison system according to claim 1, characterized in that: The process of determining the sample set includes: A hash value is obtained by performing a hash operation on any business primary key in the candidate dataset. The hash value is moduloed by the preset total number of data storage shards to obtain the shard index value; Determine whether the fragment index value falls within the numerical range defined by the preset sampling coverage factor; If yes, the business primary key is retained in the sample set; otherwise, the business primary key is removed from the candidate dataset.

3. The multi-data source automatic acquisition and comparison system according to claim 1, characterized in that: The screening process for the candidate dataset includes: Parse the scheduling configuration information to determine the business category identifier to which the current task belongs; Obtain historical diagnostic reports generated within a preset number of historical task periods; Based on the historical diagnostic reports, the failure rate corresponding to the business category identifier is calculated; Based on the failure rate and a preset sensitivity factor, calculate the dynamic sampling probability for the business category identifier; The candidate dataset is obtained by randomly sampling the business data stream according to the dynamic sampling probability. The formula for calculating the dynamic sampling probability is as follows: in, Indicates business category identifier The current dynamic sampling probability, Represents the preset base random probability This represents the preset sensitivity factor. Indicates business category identifier The failure rate within a preset number of historical task cycles.

4. The multi-data source automatic acquisition and comparison system according to claim 1, characterized in that: The process of obtaining the business semantic values ​​includes: Parse the virtual field mapping table to obtain multiple mapping records, where each mapping record defines the correspondence between the physical data path used to locate the original data in the virtual data context and the virtual business field used to identify standard business attributes; Based on the correspondence, the original physical data is extracted from the virtual data context; By applying a pre-defined atomic normalization operator, the original physical data is subjected to unit conversion and data type conversion to obtain the standardized business semantic values ​​in a unified format. Establish a key-value pair index between the virtual business field and the standardized business semantic value, for use in the rule execution topology graph.

5. The multi-data source automatic acquisition and comparison system according to claim 1, characterized in that: The logical verification rules include integrity verification rules, consistency verification rules, and business logic correctness verification rules. The process of constructing the rule execution topology graph includes: Set the integrity verification rule as the root node of the rule execution topology graph; Parse the input parameters of the consistency verification rule and the business logic correctness verification rule to identify parameter dependencies; When any one of the logical verification rules is used as the first rule, and the execution of the first rule depends on the result of another rule in the logical verification rules being used as the second rule, a directed edge is established from the second rule to the first rule. The rule execution topology graph is generated based on the root node and the directed edges.

6. The multi-data source automatic acquisition and comparison system according to claim 1, characterized in that: The logical assertion of the current rule node is executed based on the injected business semantic value to obtain the verification result, and dynamic tolerance processing based on the business lifecycle is also included: Extract the main state field representing the current data service lifecycle from the virtual data context; The current business stage of the data is determined based on the main status field; Based on the business stage, the dynamic tolerance threshold of the current rule node is calculated using a preset drift function; Calculate the absolute value of the difference between the two business semantic values ​​to be compared, and determine whether the absolute value of the difference is less than or equal to the dynamic tolerance threshold; If so, the logical assertion of the current rule node is deemed to have passed; Otherwise, the logical assertion is deemed to have failed.

7. The multi-data source automatic acquisition and comparison system according to claim 6, characterized in that: Whether the absolute value of the difference is less than or equal to the dynamic tolerance threshold is determined by the following inequality: in, and These represent standardized business semantic values ​​corresponding to the same business object in different data sources. This indicates the value of the main status field. Represents state-based The calculated dynamic tolerance threshold; when When it belongs to the preset set of high-frequency change states, The value is set to the preset maximum tolerance threshold; when When it belongs to the preset set of final states, The value is zero.

8. The multi-data source automatic acquisition and comparison system according to claim 1, characterized in that: Before executing the logical assertion of the current rule node based on the injected business semantic value, logical clock alignment processing is also included: Extract the business version number corresponding to each data source from the virtual data context; Calculate the difference in business version numbers between different data sources; Determine whether the difference exceeds a preset version synchronization threshold; If so, the execution of the current rule node is paused, and after a preset sleep time, the process of obtaining multi-source heterogeneous data distributed in the database, cache system and message queue is retried until the difference is less than or equal to the version synchronization threshold or the maximum number of retries is reached. Otherwise, continue executing the logical assertion.

9. The multi-data source automatic acquisition and comparison system according to claim 1, characterized in that: The blocking of the execution of the current rule node and its child nodes specifically includes: Identify the error type of the rule node that failed to execute; If the error type is missing physical data or blocked prerequisites, the current rule node and all its successor child nodes are marked as skipped and not recorded as a business error. If the error type is a logical assertion failure, then the current rule node is marked as a failure, and a snapshot of the input data when the assertion fails is recorded. At the same time, the execution of all child nodes that depend on the current rule node is terminated.

10. A method for automatic acquisition and comparison of multiple data sources, characterized in that, Includes the following steps: Obtain preset scheduling configuration information, and generate a task context containing a unique task identifier and a time range to be verified based on the scheduling configuration information; Based on the task context, a preset strategy drives a random algorithm to filter candidate datasets from the business data streams within the time range to be verified, and performs hashing and modulo operations on the business primary keys in the candidate datasets to determine the sample set that satisfies the preset coverage factor. Based on each business primary key in the sample set, multi-source heterogeneous data distributed in database, cache system and message queue are obtained in parallel through a unified data access interface, and the multi-source heterogeneous data is encapsulated into a virtual data context. Load a pre-defined declarative rule configuration model, which includes a virtual field mapping table and logical validation rules; Based on the virtual field mapping table, the multi-source heterogeneous data in the virtual data context is mapped and converted into standardized business semantic values; Based on the pre-dependencies between the logical verification rules, a rule execution topology graph in the form of a directed acyclic graph with a hierarchical structure is constructed. The standardized business semantic values ​​are injected into the rule execution topology graph, and each rule node is traversed in topological order. During the traversal, it is determined whether the predecessor dependency node of the current rule node has been executed. If so, the logical assertion of the current rule node is executed based on the injected business semantic value to obtain the verification result. Otherwise, block the execution of the current rule node and its child nodes; The system summarizes the verification results of all executed nodes in the rule execution topology graph and the corresponding original data snapshots, generates a structured diagnostic report, and outputs it.

Images