Incremental data detection method and device, equipment and storage medium

Abstract

The invention discloses an incremental data detection method and device, equipment and a storage medium, and relates to the technical field of incremental data detection.The incremental data detection method comprises the steps that a service primary key and a data fingerprint are generated for each data record in a heterogeneous data source; constructing a data snapshot according to the business primary key of each data record and the corresponding data fingerprint; comparing the data snapshot with a stored historical data snapshot to identify a data change record; and outputting an incremental data result set containing the data change record. According to the method, the universality of incremental data detection can be improved.

Description

Technical Field

[0001] This application relates to the field of incremental data detection technology, and in particular to incremental data detection methods, apparatus, equipment and storage media. Background Technology

[0002] In application scenarios such as enterprise data platforms, big data analytics, and real-time data warehouses, it is often necessary to integrate multiple heterogeneous data sources from different systems. Incremental data synchronization has become the preferred solution for synchronizing data from heterogeneous data sources due to its high efficiency.

[0003] Currently, incremental data detection employs various methods, including filtering based on timestamps or auto-incrementing IDs (Identity), CDC (Change Data Capture) based on database logs, trigger-based methods, and full table comparisons. However, each method has its own limitations. Specifically, filtering based on timestamps or auto-incrementing IDs requires the data source to contain physical fields such as `last_updated_time` or `auto_increment_id`. This makes it unsuitable for legacy systems or file-based data sources lacking such fields. More importantly, this method typically fails to detect physical deletion operations of data records, resulting in incomplete incremental data. CDC based on database logs... Solution 1: This solution heavily relies on the proprietary log formats and parsing tools (such as Debezium, Canal) of specific database management systems (e.g., Oracle, MySQL). It is difficult to scale to heterogeneous data sources other than database types, such as file systems, message queues (Kafka), or NoSQL databases (MongoDB), because these data sources typically do not provide standard transaction log interfaces. Solution 2: Full table comparison. While theoretically universal, this requires retrieving the entire dataset using SELECT statements (command statements for retrieving data from database tables) and then comparing each row. Its computational complexity and I / O (input / output) overhead increase linearly with the data volume, resulting in poor performance and failing to meet the real-time or near real-time synchronization requirements for large datasets. In summary, existing technical solutions, due to their tight coupling with the physical structure or specific mechanisms of the data source, lack universality and are difficult to apply to diverse heterogeneous data sources within an enterprise.

[0004] In summary, improving the universality of incremental data detection has become a pressing technical problem that needs to be solved in this field. Summary of the Invention

[0005] The main objective of this application is to provide an incremental data detection method, apparatus, device, and storage medium, aiming to improve the universality of incremental data detection.

[0006] To achieve the above objectives, this application proposes an incremental data detection method, which includes: Generate a business primary key and data fingerprint for each data record in a heterogeneous data source; Construct a data snapshot based on the business primary key and corresponding data fingerprint of each data record; The data snapshot is compared with the stored historical data snapshot to identify data change records; Output an incremental data result set containing the data change records.

[0007] In one embodiment, prior to the step of generating a business primary key and data fingerprint for each data record in the heterogeneous data source, the method further includes: A connection is established with heterogeneous data sources through an adapter layer, wherein the adapter layer configures a corresponding adapter for each type of heterogeneous data source, and the adapter is configured with data reading rules for the heterogeneous data source. The heterogeneous data source includes at least one of relational databases, NoSQL (notnolysql), file systems, and message queues. Each adapter reads the data records of the corresponding heterogeneous data source.

[0008] In one embodiment, the step of generating a business primary key and data fingerprint for each data record in a heterogeneous data source includes: The fingerprint generation rule for identifying the heterogeneous data source includes a business identifier field type as the business primary key and a content monitoring field type as the data fingerprint. For each data record in the heterogeneous data source, extract the field whose field type is the business identifier field type from the data record to generate the business primary key of the data record; and extract the field whose field type is the content monitoring field type from the data record to generate the value string of the data record. The hash value of the value string is calculated to obtain the data fingerprint of the data record.

[0009] In one embodiment, the step of constructing a data snapshot based on the business primary key and corresponding data fingerprint of each data record includes: For each data record in the heterogeneous data source, the business primary key and the corresponding data fingerprint are stored in the form of key-value pairs to obtain a set of key-value pairs; The set of key-value pairs for each of the data records is taken as a data snapshot.

[0010] In one embodiment, the step of comparing the data snapshot with stored historical data snapshots to identify data change records includes: The data snapshot is compared with the set of key-value pairs in the stored historical data snapshot. Business primary keys that exist in the data snapshot but not in the historical data snapshot are identified as new records. Business primary keys that exist in the historical data snapshot but not in the data snapshot are identified as deleted records. Business primary keys that exist in both the data snapshot and the historical data snapshot but have different data fingerprints are identified as update records. A data change record is generated based on the newly added record, the deleted record, and the updated record.

[0011] In one embodiment, after the step of outputting an incremental data result set containing the data change records, the method further includes: In response to an incremental data read command, the incremental data result set is transmitted to a downstream system so that the downstream system can pull incremental data from the heterogeneous data source based on the incremental data result set.

[0012] In one embodiment, the method further includes: Determine whether the heterogeneous data source supports the native incremental capture interface, wherein the native incremental capture interface includes at least one of the following: a change data capture interface based on database transaction logs, an incremental query interface provided by the data source, and a query interface based on data update timestamps or auto-incrementing sequence fields; If so, incremental data detection is performed through the native incremental capture interface. If not, then perform the steps of generating a business primary key and data fingerprint for each data record in the heterogeneous data source, as well as subsequent steps.

[0013] Furthermore, to achieve the above objectives, this application also proposes an incremental data detection device, which includes: The fingerprint generation module is used to generate a business primary key and a data fingerprint for each data record in a heterogeneous data source. The snapshot construction module is used to construct data snapshots based on the business primary key and corresponding data fingerprint of each data record; The snapshot comparison module is used to compare the data snapshot with the stored historical data snapshot in order to identify data change records; The result output module is used to output an incremental data result set containing the data change records.

[0014] In addition, to achieve the above objectives, this application also proposes an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the incremental data detection method as described above.

[0015] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the incremental data detection method described above.

[0016] This application proposes an incremental data detection method. In this method, a business primary key and data fingerprint are generated for each data record in a heterogeneous data source. A data snapshot is constructed based on the business primary key and corresponding data fingerprint of each data record. The data snapshot is compared with stored historical data snapshots to identify data change records. Finally, an incremental data result set containing these data change records is output. In summary, this application generates a business primary key and data fingerprint for each data record in a heterogeneous data source, constructs a data snapshot, and performs a comparison. This method is independent of specific fields, database types, or versions, and does not require complex modifications to the data source or add extra burden. Regardless of the structure and characteristics of the data source, it can accurately generate business primary keys and data fingerprints, construct data snapshots, and perform effective comparisons to identify data change records. Therefore, it has high universality and can effectively solve the problem of low universality in current incremental data detection methods, meeting the needs of incremental data synchronization from heterogeneous data sources in application scenarios such as enterprise data platforms, big data analytics, and real-time data warehouses. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating an embodiment of the incremental data detection method of this application. Figure 2 This is a schematic diagram of the incremental data detection system architecture provided in Embodiment 2 of the incremental data detection method of this application; Figure 3 This is a simplified flowchart illustrating the incremental data detection method provided in Embodiment 2 of the incremental data detection method of this application. Figure 4 This is a schematic diagram of the module structure of the incremental data detection device according to an embodiment of this application; Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the incremental data detection method in this application embodiment.

[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0022] In application scenarios such as enterprise data platforms, big data analytics, and real-time data warehouses, it is often necessary to integrate multiple heterogeneous data sources from different systems. Incremental data synchronization has become the preferred solution for synchronizing data from heterogeneous data sources due to its high efficiency.

[0023] Currently, incremental data detection can be achieved through various methods, such as filtering based on timestamps or auto-incrementing IDs, CDC based on database logs, trigger-based methods, and full table comparisons. However, these methods have their own shortcomings, lack universality, and are difficult to apply to various data sources.

[0024] Specifically, filtering schemes based on timestamps or auto-incrementing IDs require the data source to contain physical fields such as `last_updated_time` or `auto_increment_id`. This is unsuitable for legacy systems or file-based data sources lacking such fields. More importantly, this scheme typically fails to detect physical deletion operations of data records, leading to incomplete incremental data. Database log-based CDC schemes heavily rely on proprietary log formats and parsing tools (such as Debezium and Canal) of specific database management systems (e.g., Oracle, MySQL). They are difficult to extend to heterogeneous data sources other than database types, such as file systems, message queues (Kafka), or NoSQL databases (MongoDB), because these data sources typically do not provide standard transaction log interfaces. Full table comparison schemes, while theoretically universal, require retrieving the entire dataset using methods like SELECT and then comparing row by row. Their computational complexity and I / O overhead increase linearly with the data volume, resulting in poor performance and failing to meet the real-time or near-real-time synchronization requirements for large datasets. In summary, existing technical solutions are all tightly coupled with the physical structure or specific mechanisms of the data source, resulting in low universality and difficulty in being applied to diverse heterogeneous data sources within an enterprise.

[0025] In summary, improving the universality of incremental data detection has become a pressing technical problem that needs to be solved in this field.

[0026] This application provides a solution that generates a business primary key and a data fingerprint for each data record in a heterogeneous data source; constructs a data snapshot based on the business primary key and the corresponding data fingerprint of each data record; compares the data snapshot with the stored historical data snapshot to identify data change records; and outputs an incremental data result set containing data change records.

[0027] In summary, the embodiments of this application generate business primary keys and data fingerprints for each data record in heterogeneous data sources, construct data snapshots, and perform comparisons. This approach is independent of specific fields, database types, or versions, and does not require complex modifications to the data source or incur additional overhead. Regardless of the structure and characteristics of the data source, it can accurately generate business primary keys and data fingerprints, construct data snapshots, and perform effective comparisons, thereby identifying data change records. Therefore, it has high universality and effectively solves the problem of low universality in current incremental data detection methods, meeting the needs of incremental data synchronization from heterogeneous data sources in application scenarios such as enterprise data platforms, big data analytics, and real-time data warehouses.

[0028] It should be noted that the executing entity in this embodiment can be an electronic device with data processing, network communication, and program execution functions, such as a computer, host computer, controller, etc., or an electronic device capable of performing the above functions. The following description uses an incremental data detection system as an example to illustrate this embodiment and the subsequent embodiments.

[0029] Based on this, embodiments of this application provide an incremental data detection method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the incremental data detection method of this application.

[0030] In this embodiment, the incremental data detection method includes steps S10 to S40: Step S10: Generate a business primary key and data fingerprint for each data record in the heterogeneous data source; It should be noted that heterogeneous data sources refer to multiple data sources that differ in logical structure, physical storage method, data format, access protocol, etc. These types include, but are not limited to, relational databases (such as MySQL and Oracle), NoSQL (such as MongoDB and Redis), data warehouses, file systems (such as systems storing CSV, JSON, and log files), and message queues (such as Kafka and RabbitMQ). A data record refers to the smallest unit of data representing an independent entity or event within a specific data source. For example, it corresponds to a row in a database table, an object in a JSON (JavaScript Object Notation) file, a line of text in a log file, and a message in a message queue. A business primary key refers to a combination of one or more fields that can uniquely identify a data record based on business logic. The composition of this business primary key does not rely on mechanisms such as database auto-incrementing primary keys, but is determined according to business semantics. A data fingerprint refers to a fixed-length digest string obtained by calculating the content of all or specified monitoring fields of a data record using a specific hash algorithm. This string serves as a unique representation of the record's content; any minor change in the content will cause a significant change in the data fingerprint.

[0031] Raw data records from various heterogeneous data sources are processed in a standardized manner and transformed into standardized information elements that are easy to calculate and compare, namely business primary keys for unique identification and data fingerprints for content consistency judgment.

[0032] Specifically, for each data record in the heterogeneous data source, the values ​​of one or more specific fields are extracted from the data record. If the business primary key consists of multiple fields, the values ​​of these fields are concatenated in a fixed order to generate a globally unique string as the business primary key of that data record. Simultaneously, all or some fields in the data record whose changes need to be monitored are extracted, and the values ​​of these fields are concatenated in a fixed order to form a value string. A preset hash algorithm is then used to convert the value string into a data fingerprint to represent the content status of the data record at the current moment.

[0033] In one feasible embodiment, steps A10 to A20 are included before step S10: Step A10: Establish a connection with the heterogeneous data source through the adapter layer. The adapter layer configures a corresponding adapter for each type of heterogeneous data source. The adapter is set with data reading rules for the heterogeneous data source. The heterogeneous data source includes at least one of relational databases, NoSQL, file systems and message queues. It should be noted that the adapter layer refers to the abstract architecture layer that interacts with external data sources. This layer consists of a series of adapter components for specific types of data sources. Each adapter component encapsulates the specific protocols, drivers, API (Application Programming Interface) call details, and connection management logic required to communicate with its corresponding data source type. Data reading rules refer to a series of parameters and instructions pre-configured in each adapter to precisely control how data is extracted from the corresponding data source. These data reading rules specifically define the dimensions and methods of data access. For example, for relational databases, a rule can be an SQL query statement or a stored procedure call; for file systems, a rule can be a specific file path wildcard, file encoding format, and parsing mode; for message queues, a rule can be a consumer group identifier, topic subscription expression, and offset reset strategy. These data reading rules also specify the triggering conditions for reading, such as polling at a fixed time period or triggering based on the accumulated data volume.

[0034] By encapsulating the complexities of access protocols and interaction methods of various heterogeneous data sources at the underlying level through the adapter layer, a unified data access interface is provided to the upstream incremental detection algorithm, thereby enabling it to handle all types of data sources with completely consistent processing logic.

[0035] Step A20: Read the data records of the corresponding heterogeneous data source through each adapter.

[0036] When a data read command is triggered by any means such as a timer, external event, or manual operation, each adapter is called concurrently or sequentially. After receiving the call command, each adapter performs a specific operation corresponding to the data source to obtain the data records of the data source.

[0037] Therefore, by introducing an adapter layer, when a new data source type needs to be supported, only one adapter conforming to a unified interface specification needs to be developed for this new type of data source for seamless integration. This improves the versatility of the entire technical solution and reduces the future maintenance costs and expansion difficulty of the system.

[0038] Furthermore, in one feasible implementation, the adapter can implement fault tolerance and performance optimization mechanisms. For example, when performing data reading, if network fluctuations or temporary unavailability of the data source are encountered, the adapter can automatically retry multiple times and adopt an exponential backoff strategy. For data sources that support incremental queries, the adapter can intelligently maintain query watermarks, such as the offset of the last read message, the file read position, or the database log sequence number, to achieve breakpoint resume and avoid full reads every time.

[0039] Step S20: Construct a data snapshot based on the business primary key and corresponding data fingerprint of each data record; It should be noted that a data snapshot is a lightweight image of the complete set of business primary keys and data fingerprint correspondences of all valid data records in a heterogeneous data source. It includes the overall content state of the heterogeneous data source without storing the complete original data.

[0040] The business primary keys and data fingerprints generated from the scattered records of massive data are aggregated to form a data snapshot that can be retrieved and compared in subsequent periods.

[0041] Specifically, it iterates through all processed data records. For each data record, it uses its business primary key as the key and its data fingerprint as the value to form a key-value pair. The set of key-value pairs of all data records is used as a data snapshot of the current incremental data detection period, and this data snapshot is persistently stored in the preset storage area.

[0042] Step S30: Compare the data snapshot with the stored historical data snapshot to identify data change records; It should be noted that historical data snapshots refer to data snapshots generated and stored in a successful detection cycle immediately preceding the current cycle; data change records refer to the set of business primary keys that represent data records that were added, deleted, or updated within the time interval between the two consecutive data snapshots, identified by comparing data snapshots from two consecutive cycles.

[0043] Retrieve and load the historical data snapshot (SnapshotT-1) stored in the last detection period from persistent storage. By comparing the current data snapshot (SnapshotT) with the business primary key and data fingerprint in the historical data snapshot (SnapshotT-1), the data change record is identified based on the synchronization of the two in the business primary key and / or data fingerprint.

[0044] Step S40: Output an incremental data result set containing data change records.

[0045] The output contains an incremental data result set containing data change records. The incremental data result set refers to a structured data output result that clearly lists all change records identified during the current detection period. It is usually classified by change type and includes at least the business primary key of these records. It may also include information such as change timestamps. This incremental data result set can be provided to downstream systems for incremental data querying.

[0046] Thus, in this embodiment, a business primary key and data fingerprint are generated for each data record in a heterogeneous data source, a data snapshot is constructed and compared, without relying on specific fields, database types or versions, and without requiring complex modifications to the data source or adding extra burden. Regardless of the structure and characteristics of the data source, it can accurately generate business primary keys and data fingerprints, construct data snapshots and perform effective comparisons, thereby identifying data change records. Therefore, it has high universality and can effectively solve the problem of low universality of current incremental data detection methods, meeting the needs of incremental data synchronization from heterogeneous data sources in application scenarios such as enterprise data platforms, big data analysis and real-time data warehouses.

[0047] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description and will not be repeated hereafter. Based on this, step S10 may include steps S101 to S103: Step S101: Identify the fingerprint generation rules of heterogeneous data sources. The fingerprint generation rules include the business identifier field type as the business primary key and the content monitoring field type as the data fingerprint. Before processing data records from a specific data source, a pre-configured fingerprint generation rule for that data source is loaded. This rule can be customized based on the actual application scenario and is recorded in the adapter. The fingerprint generation rule explicitly defines how to extract the business primary key and data fingerprint from data records of heterogeneous data sources. Specifically, it lists the identifiers of one or more fields in the corresponding data source used to constitute the business primary key. The selection of these fields is entirely based on the uniqueness requirements of the business logic. For example, in the user information table, the "user_id" field is specified as the business identifier field type for the business primary key; in the order details table, the combination of the "order_id" and "product_id" fields is specified as the business identifier field type for the business primary key. The fingerprint generation rule also lists the identifiers of one or more fields whose content needs to be monitored for changes. The combination of these fields collectively determines the content fingerprint of the data record. For example, if the monitoring fields "customer_name, delivery_address, phone_number" are specified as the content monitoring field types for the data fingerprint, then any change in the value of any of these fields will cause a change in the final calculated data fingerprint.

[0048] Step S102: For each data record in the heterogeneous data source, extract the field of type business identifier field from the data record to generate the business primary key of the data record; and extract the field of type content monitoring field from the data record to generate the value string of the data record. For each data record read through the adapter, the value of the corresponding field is extracted from the data record as the business primary key according to the business identifier field type list defined in the fingerprint generation rules, and the value of the corresponding field is extracted from the data record as the value string according to the content monitoring field type list defined in the fingerprint generation rules.

[0049] If the business primary key consists of only one field, then the string representation of that field is used directly as the business primary key; if it consists of multiple fields, then the field values ​​are concatenated according to the field order specified in the fingerprint generation rules and using a predefined separator (such as underscore _, vertical bar | or other characters) to form a globally unique business primary key string.

[0050] Step S103: Calculate the hash value of the value string to obtain the data fingerprint of the data record.

[0051] Hash values ​​are calculated for each string value from heterogeneous data sources. The hash function used can be selected based on the actual application scenario. In scenarios with extremely high data integrity requirements but no need for encryption security, fast non-encryption hash functions such as xxHash, MurmurHash3, or CityHash are preferred. In scenarios with security requirements, encryption hash functions such as MD5, SHA-1, or SHA-256 can be used. After the hash function performs the calculation, it outputs a fixed-length string, which is the data fingerprint of the data record at the current moment.

[0052] Therefore, by introducing configurable fingerprint generation rules, it is possible to freely define what constitutes a unique identifier (i.e., business primary key) of a record and what constitutes the core content to be monitored (i.e., data fingerprint) based on actual business semantics rather than the physical structure of the database. This makes the incremental data system in this embodiment not only suitable for well-designed modern application databases, but also easily cope with scenarios such as legacy systems, third-party data interfaces, and semi-structured log files, thereby enabling it to handle various non-standardized or heterogeneous data with complex business logic.

[0053] In one feasible embodiment, step S20 may include steps S201-S202: Step S201: For each data record in the heterogeneous data source, store the business primary key and the corresponding data fingerprint in key-value pair format to obtain a key-value pair set. Iterate through each data record in the heterogeneous data source, store its business primary key and corresponding data fingerprint in key-value pairs, forming a set containing all the key-value pairs corresponding to all data records.

[0054] Step S202: Take the set of key-value pairs of each data record as a data snapshot.

[0055] The set of key-value pairs is used as a snapshot of the heterogeneous data source at the current point in time. This snapshot is used to record the current data state of the heterogeneous data source.

[0056] Therefore, by using a set of key-value pairs to construct a data snapshot, the existence determination and fingerprint value acquisition of any business primary key can be extremely fast when comparing the current data snapshot with historical data snapshots. This reduces the complexity of the entire snapshot comparison process to near zero, and compared with the traditional row-by-row full comparison scheme, it can bring a great performance improvement when processing large-scale data.

[0057] In addition, in one feasible implementation, data compression technology can be used to further reduce storage space usage when saving data snapshots.

[0058] In one feasible embodiment, step S30 may include steps S301 to S302: Step S301: Compare the key-value pair set in the data snapshot with the stored historical data snapshot. Business primary keys that exist in the data snapshot but not in the historical data snapshot are identified as new records. Business primary keys that exist in the historical data snapshot but not in the data snapshot are identified as deleted records. Business primary keys that exist in both the data snapshot and the historical data snapshot but have different data fingerprints are identified as update records. The system first loads a historical data snapshot (SnapshotT-1) from the persistent storage of the data from the previous detection period, and compares the set of key-value pairs stored in it with the set of key-value pairs in the current data snapshot (SnapshotT). The comparison logic is as follows: For each business primary key in SnapshotT, check if the business primary key exists in the key set of SnapshotT-1. If the check result is that the business primary key does not exist in SnapshotT-1, then the data record corresponding to the business primary key is determined to be a newly added record.

[0059] For each business primary key in SnapshotT-1, check if the business primary key exists in the key set of SnapshotT. If the check result is that the business primary key does not exist in SnapshotT, then the data record corresponding to the business primary key is determined to be a deleted record.

[0060] For a business primary key that exists in both SnapshotT and SnapshotT-1, check whether their data fingerprints are the same. If the two fingerprint values ​​are different, determine that the data record corresponding to the business primary key is an updated record.

[0061] Step S302: Generate data change records based on newly added records, deleted records, and updated records.

[0062] By comparing data snapshots, newly added, deleted, and updated records from heterogeneous data sources are identified. These records are then integrated to obtain data change records that can be consumed by downstream systems. These data change records include the data change type and the corresponding business primary key, and may also include the time range of this detection, the snapshot generation timestamp, and the data source identifier.

[0063] Therefore, by comparing data snapshots from two consecutive time points, all types of data changes that occurred during that time can be identified. In other words, as long as the data changes, it can be detected and correctly classified, thus achieving a complete record of all "add, delete, and modify" change operations.

[0064] Furthermore, in a feasible implementation, in order to improve comparison efficiency, especially when comparing ultra-large-scale snapshots, the snapshot comparison process can be executed in parallel using multi-threading technology. For example, a thread pool can be used to shard the key set in the data snapshot, and multiple threads can concurrently perform the identification of newly added records. Similarly, the key set in the historical data snapshot can be sharded and deleted records can be identified concurrently. For the identification of updated records, the common key set in the data snapshot and the historical data snapshot can be sharded and processed concurrently.

[0065] In one feasible embodiment, step S40 may be followed by step S50: In step S50, in response to the incremental data read instruction, the incremental data result set is transmitted to the downstream system so that the downstream system can pull incremental data from the heterogeneous data source based on the incremental data result set.

[0066] When an instruction to trigger incremental data reading is received, the previously acquired incremental data result set is encapsulated according to a preset transmission protocol and data format. Then, the encapsulated incremental data result set is sent to the downstream system through a pre-established communication link. This allows the downstream system to actively locate and retrieve the complete incremental data that matches the result set from the corresponding heterogeneous data source based on the incremental data identification information contained in the result set, thereby completing further processing of the incremental data by the downstream system.

[0067] In one feasible embodiment, the incremental data detection method may further include steps B10 to B30: Step B10: Determine whether the heterogeneous data source supports the native incremental capture interface. The native incremental capture interface includes at least one of the following: a change data capture interface based on the database transaction log, an incremental query interface provided by the data source, and a query interface based on the data update timestamp or auto-incrementing sequence field. It's important to clarify that native incremental capture interfaces refer to mechanisms or interfaces built into the data source itself for capturing its own data changes; change data capture interfaces based on database transaction logs refer to technologies that capture all data manipulation language changes by parsing the database's transaction logs (such as MySQL's binlog, Oracle's RedoLog, and PostgreSQL's WAL), with representative tools including Debezium and Canal; incremental query interfaces provided by the data source refer to dedicated interfaces exposed directly by some data sources or SaaS services through APIs for querying recent data changes, such as the incremental data query operation in Salesforce's SOAP API; and query interfaces based on data update timestamps or auto-incrementing sequence fields refer to methods that utilize fields in the data table that can identify the time of record addition or modification (such as last_modified_time) or monotonically increasing sequence fields (such as auto-incrementing primary key id) to obtain the increment by recording the previous maximum value and then querying records greater than that value.

[0068] Determine whether the heterogeneous data source to be monitored has a usable native incremental capture interface. The interface includes at least one of the following: a change data capture interface based on database transaction logs, an incremental query interface provided by the data source, and a query interface based on data update timestamps or auto-incrementing sequence fields.

[0069] Step B20: If yes, then perform incremental data detection through the native incremental capture interface; If the determination result is yes, the native incremental capture interface corresponding to the data source will be enabled and called first to perform efficient and real-time incremental data detection. In this path, the system performs incremental data detection through the native incremental capture interface and directly generates incremental results.

[0070] Step B30: If not, proceed with the steps of generating a business primary key and data fingerprint for each data record in the heterogeneous data source, and subsequent steps.

[0071] If the result is negative, meaning the data source does not support any native incremental mechanism, the aforementioned snapshot comparison process is executed to ensure reliable incremental detection can be completed on any type of heterogeneous data source.

[0072] Therefore, for data sources with efficient native mechanisms, their capabilities can be fully utilized to achieve incremental detection with low latency and low overhead; while for data sources without native mechanisms, incremental detection can also be achieved by using snapshot comparison, enabling the system to efficiently and reliably handle all types of data sources.

[0073] For example, to help understand the implementation flow of the incremental data detection method obtained by combining the above embodiments, please refer to... Figure 2 , Figure 2 A schematic diagram of an incremental data detection system architecture employing an incremental data detection method is provided, specifically: In this example, the heterogeneous data sources handled by the incremental data inspection system include, but are not limited to, relational databases such as MySQL and PostgreSQL, NoSQL databases such as MongoDB and ElasticSearch, and CSV files. These data sources differ significantly in structure and access methods. To uniformly handle these different data sources, the system sets up an adapter layer. This layer provides a dedicated adapter for each type of data source. When an inspection task starts, the system establishes connections with the corresponding data sources by calling these adapters and reads the data records, thereby standardizing the heterogeneous data access methods. The read data records are sent to the hash fingerprint calculation module. This module extracts specific fields from each data record according to the fingerprint rules pre-configured for each data source, concatenates them into a business primary key, extracts other specified fields, concatenates them into a value string, and then uses a hash algorithm to calculate the value string to generate a data fingerprint that uniquely represents the content of the record. Then, the snapshot management module collects the "business primary key-data fingerprint" key-value pairs of all current data records, constructs a current data snapshot, and stores this snapshot. This module is also responsible for retrieving the last stored historical data snapshot when needed. Then, the difference analysis module acquires the current data snapshot and historical data snapshots. By comparing the key-value pair sets of the two snapshots, it identifies primary keys that exist in the current data snapshot but not in the historical data snapshot as new records, primary keys that exist in the historical data snapshot but not in the current data snapshot as deleted records, and primary keys that exist in both snapshots but have different data fingerprint values ​​as update records. Finally, the system aggregates the primary key information of these identified change records into an incremental data result set for downstream systems, thus completing a full incremental detection cycle.

[0074] The implementation flow of the incremental data detection method obtained in this embodiment, combined with the above embodiments, is as follows: Figure 3As shown, firstly, heterogeneous data sources and adapters are configured, that is, the type of each heterogeneous data source to be monitored is specified and the corresponding connection and reading rules are configured. After configuration, all current data records are read from the corresponding data source through the configured adapter. Then, a hash fingerprint is generated for each read data record, namely the business primary key and data fingerprint. After generating fingerprints for all records, these "business primary key-data fingerprint" key-value pairs are collected to construct the current data snapshot SnapshotT for this detection cycle. Then, the historical data snapshot SnapshotT-1 saved from the last detection is read and its existence is determined. If the historical data snapshot does not exist, it indicates that the system is running for the first time or there is no baseline data. In this case, the system treats all records in the current snapshot SnapshotT as new records and directly generates an incremental data result set. If the historical data snapshot exists, the current data snapshot SnapshotT is compared with the historical data snapshot SnapshotT-1 to find the added, deleted, and updated records, and an incremental data result set is generated based on this. After outputting the results, the current snapshot SnapshotT generated this time is persisted and saved, making it a historical data snapshot for the next detection. Finally, based on the preset scheduling strategy, it is determined whether to start the next detection cycle immediately; if the next cycle is entered, the entire process from reading data to saving a snapshot is repeated; if not, the process ends.

[0075] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the incremental data detection method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0076] This application also provides an incremental data detection device, please refer to... Figure 4 The incremental data detection device includes: Fingerprint generation module 10 is used to generate business primary keys and data fingerprints for each data record in heterogeneous data sources; The snapshot building module 20 is used to build data snapshots based on the business primary key and corresponding data fingerprint of each data record; The snapshot comparison module 30 is used to compare the data snapshot with the stored historical data snapshot in order to identify data change records; The result output module 40 is used to output an incremental data result set containing data change records. Optionally, the incremental data detection device also includes a data reading module (not shown), which is used for: A connection is established with heterogeneous data sources through an adapter layer. The adapter layer configures a corresponding adapter for each type of heterogeneous data source. The adapter is set with data reading rules for heterogeneous data sources. Heterogeneous data sources include at least one of relational databases, NoSQL, file systems, and message queues. Each adapter reads the data records from the corresponding heterogeneous data source.

[0077] Optionally, the fingerprint generation module 10 is also used for: Fingerprint generation rules for identifying heterogeneous data sources include business identifier field types as business primary keys and content monitoring field types as data fingerprints; For each data record in the heterogeneous data source, extract the field of type business identifier from the data record to generate the business primary key of the data record, and extract the field of type content monitoring from the data record to generate the value string of the data record; Calculate the hash value of the value string to obtain the data fingerprint of the data record.

[0078] Optionally, the fingerprint generation module 10 is also used for: For each data record in the heterogeneous data source, the business primary key and the corresponding data fingerprint are stored in the form of key-value pairs to obtain a set of key-value pairs; The set of key-value pairs for each data record is used as a data snapshot.

[0079] Optionally, the snapshot comparison module 30 is also used for: The key-value pair set in the data snapshot is compared with the set of key-value pairs in the stored historical data snapshot. Business primary keys that exist in the data snapshot but not in the historical data snapshot are identified as new records, business primary keys that exist in the historical data snapshot but not in the data snapshot are identified as deleted records, and business primary keys that exist in both the data snapshot and the historical data snapshot but have different data fingerprints are identified as update records. Data change records are generated based on newly added, deleted, and updated records.

[0080] Optionally, the result output module 40 is also used for: In response to an incremental data read command, the incremental data result set is transmitted to the downstream system so that the downstream system can pull incremental data from the heterogeneous data source based on the incremental data result set.

[0081] Optionally, the incremental data detection device further includes a hybrid detection module (not shown), which is used for: Determine whether the heterogeneous data source supports the native incremental capture interface. The native incremental capture interface includes at least one of the following: a change data capture interface based on the database transaction log, an incremental query interface provided by the data source, and a query interface based on the data update timestamp or auto-increment sequence field. If so, incremental data detection is performed through the native incremental capture interface; If not, then execute the steps of generating a business primary key and data fingerprint for each data record in the heterogeneous data source, as well as subsequent steps.

[0082] The incremental data detection device provided in this application, employing the incremental data detection method described in the above embodiments, can improve the universality of incremental data detection. Compared with the prior art, the beneficial effects of the incremental data detection device provided in this application are the same as those of the incremental data detection method provided in the above embodiments, and other technical features in the incremental data detection device are the same as those disclosed in the incremental data detection method of the above embodiments, and will not be repeated here.

[0083] This application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the incremental data detection method in the first embodiment described above.

[0084] The following is for reference. Figure 5 The diagram illustrates a structural schematic of an electronic device suitable for implementing embodiments of this application. The electronic devices in these embodiments may include, but are not limited to, mobile terminals such as laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0085] like Figure 5As shown, the electronic device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the electronic device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. The communication device 1009 allows the electronic device to communicate wirelessly or wiredly with other devices to exchange data. Although the diagrams show electronic devices with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems may be implemented alternatively.

[0086] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0087] The electronic device provided in this application, employing the incremental data detection method described in the above embodiments, can improve the universality of incremental data detection. Compared with the prior art, the beneficial effects of the electronic device provided in this application are the same as those of the incremental data detection method provided in the above embodiments, and other technical features in this electronic device are the same as those disclosed in the incremental data detection method of the previous embodiment, and will not be repeated here.

[0088] It should be understood that the various parts disclosed in the embodiments of this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0089] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0090] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the incremental data detection method in the above embodiments.

[0091] The computer-readable storage medium provided in this application embodiment may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0092] The aforementioned computer-readable storage medium may be included in an electronic device or may exist independently without being assembled into an electronic device.

[0093] The aforementioned computer-readable storage medium carries one or more programs that, when executed by an electronic device, cause the electronic device to: generate a business primary key and a data fingerprint for each data record in the heterogeneous data source; construct a data snapshot based on the business primary key and the corresponding data fingerprint of each data record; compare the data snapshot with stored historical data snapshots to identify data change records; and output an incremental data result set containing data change records.

[0094] Computer program code for performing the operations of the embodiments of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0095] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0096] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0097] The readable storage medium provided in this application embodiment is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described incremental data detection method, thereby improving the universality of incremental data detection. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application embodiment are the same as the beneficial effects of the incremental data detection method provided in the above embodiments, and will not be repeated here.

[0098] The above are only some embodiments of this application and do not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. An incremental data detection method, characterized in that, The incremental data detection method includes: Generate a business primary key and data fingerprint for each data record in a heterogeneous data source; Construct a data snapshot based on the business primary key and corresponding data fingerprint of each data record; The data snapshot is compared with the stored historical data snapshot to identify data change records; Output an incremental data result set containing the data change records.

2. The incremental data detection method as described in claim 1, characterized in that, Before the step of generating a business primary key and data fingerprint for each data record in the heterogeneous data source, the method further includes: A connection is established with heterogeneous data sources through an adapter layer, wherein the adapter layer configures a corresponding adapter for each type of heterogeneous data source, and the adapter is configured with data reading rules for the heterogeneous data source. The heterogeneous data source includes at least one of relational databases, NoSQL, file systems, and message queues. Each adapter reads the data records of the corresponding heterogeneous data source.

3. The incremental data detection method as described in claim 1, characterized in that, The steps for generating a business primary key and data fingerprint for each data record in a heterogeneous data source include: The fingerprint generation rule for identifying the heterogeneous data source includes a business identifier field type as the business primary key and a content monitoring field type as the data fingerprint. For each data record in the heterogeneous data source, extract the field whose field type is the business identifier field type from the data record to generate the business primary key of the data record; and extract the field whose field type is the content monitoring field type from the data record to generate the value string of the data record. The hash value of the value string is calculated to obtain the data fingerprint of the data record.

4. The incremental data detection method as described in claim 3, characterized in that, The step of constructing a data snapshot based on the business primary key and corresponding data fingerprint of each data record includes: For each data record in the heterogeneous data source, the business primary key and the corresponding data fingerprint are stored in the form of key-value pairs to obtain a set of key-value pairs; The set of key-value pairs for each of the data records is taken as a data snapshot.

5. The incremental data detection method as described in claim 1, characterized in that, The step of comparing the data snapshot with stored historical data snapshots to identify data change records includes: The data snapshot is compared with the set of key-value pairs in the stored historical data snapshot. Business primary keys that exist in the data snapshot but not in the historical data snapshot are identified as new records. Business primary keys that exist in the historical data snapshot but not in the data snapshot are identified as deleted records. Business primary keys that exist in both the data snapshot and the historical data snapshot but have different data fingerprints are identified as update records. A data change record is generated based on the newly added record, the deleted record, and the updated record.

6. The incremental data detection method as described in claim 1, characterized in that, After the step of outputting an incremental data result set containing the data change records, the method further includes: In response to an incremental data read command, the incremental data result set is transmitted to a downstream system so that the downstream system can retrieve incremental data from the heterogeneous data source based on the incremental data result set.

7. The incremental data detection method according to any one of claims 1 to 6, characterized in that, The method further includes: Determine whether the heterogeneous data source supports the native incremental capture interface, wherein the native incremental capture interface includes at least one of the following: a change data capture interface based on database transaction logs, an incremental query interface provided by the data source, and a query interface based on data update timestamps or auto-incrementing sequence fields; If so, incremental data detection is performed through the native incremental capture interface. If not, then perform the steps of generating a business primary key and data fingerprint for each data record in the heterogeneous data source, as well as subsequent steps.

8. An incremental data detection device, characterized in that, The incremental data detection device includes: The fingerprint generation module is used to generate a business primary key and a data fingerprint for each data record in a heterogeneous data source. The snapshot construction module is used to construct data snapshots based on the business primary key and corresponding data fingerprint of each data record; The snapshot comparison module is used to compare the data snapshot with the stored historical data snapshot in order to identify data change records; The result output module is used to output an incremental data result set containing the data change records.

9. An electronic device, characterized in that, The electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the incremental data detection method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the incremental data detection method as described in any one of claims 1 to 7.

Images