Data change capture method, system, device, medium, and program product

Abstract

The application discloses a data change capture method, system, device, medium and program product, relates to the technical field of data processing, and comprises the following steps: constructing a snapshot metadata table, recording a creation operation and a deletion operation of a data file and a submission timestamp corresponding to the operation through the snapshot metadata table, wherein the data file comprises a table data file and a tombstone data file, actual business data of a table is stored in the table data file, and deletion data of the table is stored in the tombstone data file; determining an effective file list in a target time period of capturing data change by querying the submission timestamp in the snapshot metadata table, wherein the effective file list comprises an effective table data file and / or an effective tombstone data file; and filtering and merging the effective file list based on the submission timestamp in the target time period to obtain final changed data. The application improves the processing efficiency of data change capture.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data change capture method, a data change capture system, an electronic device, a storage medium, and a computer program product. Background Art

[0002] Existing data change capture systems capture database changes primarily by directly reading the database's transaction log and parsing it to extract incremental change data. However, this log-based method of capturing change data involves writing massive amounts of data changes to the log as files. For example, file-level changes, such as the addition or deletion of new files, are recorded in the log. However, directly reading the log only yields records of file changes. To obtain the true changes to the massive amounts of data stored in the files, it is necessary to infer the file changes recorded in the log. For example, this involves locating the changed file using the file path recorded in the log, then parsing the file content of the changed file to obtain the changed data. Because both log and file parsing are required during the inference process, the complexity and latency of data parsing increase dramatically. In short, capturing change data by parsing logs is an inefficient approach. Summary of the Invention

[0003] The main purpose of this application is to provide a data change capture method, a data change capture system, an electronic device, a storage medium and a computer program product, aiming to solve the technical problem of low efficiency in processing existing changed data.

[0004] To achieve the above objectives, the present application proposes a data change capture method, which includes:

[0005] Constructing a snapshot metadata table to record creation and deletion operations of data files and corresponding commit timestamps of the operations, wherein the data files include table data files and tombstone data files. The table data files store the actual business data of the table, and the tombstone data files store the deletion data of the table.

[0006] Determine a valid file list within a target time period for capturing data changes by querying the commit timestamp in the snapshot metadata table, wherein the valid file list includes valid table data files and / or valid tombstone data files;

[0007] The valid file list within the target time period is filtered and merged based on the submission timestamp to obtain the final change data.

[0008] In one embodiment, the step of filtering and merging the valid file list based on the submission timestamps within the target time period to obtain the final change data includes:

[0009] Merging the data in the valid table data file based on the submission timestamp to obtain initial change data;

[0010] Based on the target field in the valid tombstone data file, the initial change data is filtered to obtain final change data, wherein the target field in the valid tombstone data file is associated with the physical address corresponding to the data in the valid table data file.

[0011] In one embodiment, the step of determining a valid file list within a target time period for capturing data changes includes:

[0012] If, when querying the snapshot metadata table, the commit timestamp of the creation operation of the data file is less than or equal to the target time period, and the deletion operation of the same data file is before the target time period, then each data file is added to the valid file list of the target time period.

[0013] In one embodiment, the data change capture method further includes:

[0014] responding to an operation instruction on a data file and determining whether the operation instruction is a delete operation;

[0015] If it is a delete operation, a tombstone data file is created; if it is a non-delete operation, a table data file is created;

[0016] The operation record of the tombstone data file or the table data file is written into the snapshot metadata table, wherein the operation record includes a data file type identifier, an operation type, and a corresponding commit timestamp.

[0017] In one embodiment, before the step of writing the operation record of the tombstone data file or the table data file into the snapshot metadata table, the step includes:

[0018] Check whether the operation record contains privacy fields;

[0019] If a privacy field is included, the privacy field is encrypted using a preset encryption key to generate an encrypted metadata record;

[0020] The encrypted metadata record is stored in the snapshot metadata table, and when reading the valid table data file or the valid tombstone data file, the privacy field is automatically decrypted according to the encrypted metadata record.

[0021] In one embodiment, the snapshot metadata table records a storage path identifier of a data file; and the data change capture method includes:

[0022] When reading a valid table data file or a valid tombstone data file, determining whether the valid table data file or the valid tombstone data file is located in an external storage location according to the storage path identifier;

[0023] If it is located in an external storage location, the file content of the valid table data file or the valid tombstone data file is loaded from the external storage location to the local storage space and then read.

[0024] In addition, to achieve the above objectives, the present application also proposes a data change capture system, which executes the steps of the data change capture method described above when responding to a data change capture request.

[0025] In addition, to achieve the above-mentioned purpose, the present application also proposes an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data change capture method as described above.

[0026] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium. A computer program is stored on the storage medium, and when the computer program is executed by the processor, the steps of the data change capture method described above are implemented.

[0027] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the steps of the data change capture method as described above.

[0028] One or more technical solutions proposed in this application have at least the following technical effects:

[0029] The technical solution of this application directly records the creation and deletion operations of table data files and tombstone data files, as well as their commit timestamps, by constructing a snapshot metadata table. Specifically, the snapshot metadata table is strictly sorted based on commit timestamps to ensure operation timing, and tombstone data files are specifically used to store deleted data. This avoids the traditional solution of scanning the entire table primary key to determine the specific location when capturing change data. In this application, the file operation records in the snapshot metadata table can be used to directly locate the change data, replacing the traditional transaction log parsing mechanism.

[0030] After building the snapshot metadata table, the list of valid files within the target time period for capturing data changes, i.e., the file snapshot mechanism, is determined by querying the commit timestamp in the snapshot metadata table. This allows accurate positioning of valid table data files and valid tombstone data files within the target time period, thus avoiding computational redundancy caused by the need for multiple layers of indirect log parsing in traditional solutions. Finally, the present application filters and merges the valid file list based on the commit timestamp within the target time period to obtain the final change data. This method of capturing change data based on metadata management and the file snapshot mechanism not only greatly reduces the complexity of data processing, but also significantly improves the processing efficiency of capturing massive data changes, resolving the drawback of inefficient capture of change data by existing log parsing. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0032] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0033] Figure 1 A flowchart illustrating an embodiment of a data change capture method of this application;

[0034] Figure 2 This is a schematic diagram of the device structure of the hardware operating environment involved in the data change capture method of this application.

[0035] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0036] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0037] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0038] CDC (Change Data Capture) captures changed data in a data change capture system and converts it into a format that can be consumed by other systems. Its core process can be divided into four stages: capture, parsing, transmission, and consumption.

[0039] Traditional CDC solutions for data change capture systems are mainly based on transaction logs, directly reading the transaction logs of the data change capture system and then parsing the transaction logs to extract incremental change events.

[0040] It should be noted that the execution entity of this embodiment can be a data change capture system (referred to as the system), or a computing service device with data processing, network communication, and program execution capabilities, such as a tablet computer, personal computer, or mobile phone, or a processor capable of performing the aforementioned functions. This embodiment and the following embodiments will be described below using the data change capture system as an example.

[0041] Based on this, the embodiment of the present application provides a data change capture method, referring to Figure 1 , Figure 1 This is a flow chart of an embodiment of the data change capture method of this application.

[0042] In this embodiment, the data change capture method includes steps S10 to S30:

[0043] Step S10: Construct a snapshot metadata table to record the creation and deletion operations of data files and the corresponding commit timestamps of the operations. The data files include table data files and tombstone data files. The table data files store the actual business data of the table, and the tombstone data files store the deletion data of the table.

[0044] It's important to note that the snapshot metadata table is a system-level data structure and a system table for data change capture. It can be viewed as a collection of multiple table data, each of which can be viewed as a collection of multiple column data. It's used to track the complete lifecycle of data files. Each record in the snapshot metadata table records a change event for a table data file (a physical file that stores actual user business data) or a tombstone data file (a physical file specifically used to record deletion operations). Specifically, both create and delete operations can be considered change events. The commit timestamps corresponding to these operations are time series identifiers, serving as kernel-level timestamps (typically 64-bit integers) within the data change capture system. The snapshot metadata table transforms the specific content, typically scattered across logs in traditional solutions, into structured records that can be directly queried using SQL. Furthermore, the data files in the snapshot metadata table can include only table data files, only tombstone data files, or both.

[0045] For example, a statement for creating a snapshot metadata table may be: CREATE TABLE meta_table(TableID INT, FileName VARCHAR(32), IsTombstone BOOLEAN, OP_Type INT, CommitTS INT).

[0046] Table 1 shows the organizational structure of a snapshot metadata table. The table includes TableID (a unique identifier for the snapshot metadata table), FileName (the table name of the data file), IsTombstone (indicates whether the file is a tombstone data file; if IsTombstone=true, it indicates a tombstone data file that records delete operations), OP_Type (operation type, such as create or delete), and CommitTS (the commit timestamp corresponding to the operation).

[0047] Table 1

[0048]

[0049] Step S20, determining a valid file list within a target time period for capturing data changes by querying a commit timestamp in the snapshot metadata table, wherein the valid file list includes valid table data files and / or valid tombstone data files;

[0050] It's important to note that SQL queries the snapshot metadata table's submission timestamps to filter out all valid files with submission times earlier than or equal to the target time period. This also ensures that the files haven't been deleted before the target time period (i.e., no earlier deletion records exist). The valid file list is a collection of file snapshots dynamically generated by this process. Valid table data files contain business data rows that haven't expired, while valid tombstone data files store deletion location markers that haven't been overwritten.

[0051] For example, the SQL query statement may be:

[0052] SELECT t1.FileName, t1.IsTombstone;

[0053] FROM meta_table t1;

[0054] LEFT JOIN meta_table t2 ON;

[0055] t1.FileName = t2.FileName;

[0056] AND t1.CommitTS <t2.CommitTS;

[0057] AND t2.CommitTS>1720728301;

[0058] AND t2.CommitTS<= 1720728400;

[0059] WHERE

[0060] t1.TableID = 1;

[0061] AND t1.CommitTS>1720728301;

[0062] AND t1.CommitTS<= 1720728400;

[0063] AND t1.OP_Type = 'CREATE' ;

[0064] AND NOT EXISTS (

[0065] SELECT 1

[0066] FROM meta_table t3 ;

[0067] WHERE

[0068] t3.FileName = t1.FileName

[0069] AND t3.CommitTS>t1.CommitTS;

[0070] AND t3.CommitTS<= 1720728400;

[0071] AND t3.OP_Type = 'DELETE';

[0072] ).

[0073] In addition, it should be noted that the valid file list may contain only valid table data files, or only valid tombstone data files, or both valid table data files and valid tombstone data files.

[0074] It can be understood that the inefficient mode of log playback is replaced by snapshot metadata table query and timestamp comparison in step S20. Specifically, the traditional solution needs to scan the entire transaction log history to reconstruct the status, while the present application directly determines the validity of the file by the status of the creation or deletion record.

[0075] In one embodiment, step S20 includes: if, when querying the snapshot metadata table, the commit timestamp of the creation operation of the data file is less than or equal to the target time period, and the deletion operation of the same data file is before the target time period, then each data file is added to the valid file list of the target time period.

[0076] It's important to note that during snapshot metadata table queries, the generation of the valid file list follows strict validation rules. Specifically, the commit timestamp of the create operation must be less than or equal to the target time period, meaning the file creation moment has been recorded in the data change capture system's transaction history. This commit timestamp, serving as a globally ordered transaction ID, is less than the target time period, indicating that the file was created before the target time period. Deletion operations on the same data file before the target time period indicate that the file is valid at the target time period only if a deletion record for that file with a commit time before the target time period is detected.

[0077] This method of determining valid files based on a target time period is a snapshot mechanism. A file is added to the valid list only when both the creation time limit exists and the deletion time limit has not been triggered (there is no earlier deletion record).

[0078] In this embodiment, this mechanism of filtering valid files based on snapshots eliminates the ambiguity of inferring historical states in traditional log solutions, ensuring the accuracy and efficiency of massive file screening.

[0079] Step S30: filtering and merging the valid file list within the target time period based on the submission timestamp to obtain the final change data.

[0080] It's important to note that efficient processing of file changes within a target time period by querying the snapshot metadata table relies on the utilization of file operation records and timestamps. Based on the commit timestamp, valid files modified within the target time period (i.e., files whose commit timestamps fall within this time window and haven't been overwritten by later operations) are directly filtered and merged. The final change data is then obtained by filtering and merging the file contents.

[0081] In one embodiment, step S30 includes: merging data in the valid table data file based on the commit timestamp to obtain initial change data;

[0082] Based on the target field in the valid tombstone data file, the initial change data is filtered to obtain the final change data, wherein the target field in the valid tombstone data file is associated with the physical address corresponding to the data in the valid table data file.

[0083] It's important to note that the database change capture process first parses valid table data files, which store actual business data identified by primary keys. These files contain changes following create or delete operations. When the target time period for capturing changes begins, the system performs a time window merge on the data in the valid table data files based on the commit timestamps. Only valid records with commit timestamps falling within the target time period are retained to form the initial change data.

[0084] At the same time, the target field, tombstone_rowid field, and the corresponding value are read from the valid tombstone data file. This target field stores the physical address of the data and can be directly mapped to the specific location of the table data file to establish a mapping relationship between the tombstone mark and the business data. This can avoid the performance loss caused by the traditional solution of secondary query through primary key identifier.

[0085] During the filtering phase, the system uses the physical addresses parsed from the target field's value to precisely locate the corresponding physical addresses in the initial change data. These marked data are logically marked as deleted and removed. The resulting final change data is essentially a filtered and merged snapshot of the business data state, including valid creation records within the time window while excluding any spatially invalid data marked by tombstone files.

[0086] In this implementation, this mechanism, which is directly based on physical address positioning, can achieve low-latency response even on a large data scale, avoiding the computing resource consumption caused by full table scanning followed by positioning in traditional log parsing, and solving the throughput bottleneck of change capture in massive data scenarios.

[0087] This embodiment provides a data change capture method that directly records the creation and deletion operations of table data files and tombstone data files, along with their commit timestamps, by constructing a snapshot metadata table. Specifically, the snapshot metadata table is strictly sorted based on commit timestamps to ensure operation timing, and tombstone data files are specifically stored to store deleted data, eliminating the need for traditional solutions to scan the entire table's primary key to determine the specific location when capturing change data. In this application, based on the file operation records in the snapshot metadata table, change data can be directly located, replacing the traditional transaction log parsing mechanism. After constructing the snapshot metadata table, the commit timestamp in the snapshot metadata table is queried to determine the list of valid files within the target time period for capturing data changes, i.e., the file snapshot mechanism. This allows for precise location of valid table data files and valid tombstone data files within the target time period, avoiding the computational redundancy caused by the multiple layers of indirect log parsing required in traditional solutions. Finally, this application filters and merges the valid file list within the target time period based on commit timestamps to obtain the final change data. This method of capturing change data based on metadata management and a file snapshot mechanism not only significantly reduces data processing complexity but also significantly improves the processing efficiency of capturing massive amounts of data changes, resolving the drawback of existing log parsing, which inefficiently captures change data.

[0088] For example, an embodiment:

[0089] As shown in Table 2, Table 2 is a snapshot metadata table named demo;

[0090] Table 2

[0091]

[0092] At this point, you need to obtain the changes in snapshot metadata within the target time period [1720728301, 1720728400):

[0093] The valid file list obtained according to the time period is: file3, file4, tb_file2, tb_file3.

[0094] As shown in Table 3, the file content of file3 can be:

[0095] Table 3

[0096]

[0097] As shown in Table 4, the content of file4 can be:

[0098] Table 4

[0099]

[0100] After merging file3 and file4 based on file operations, the result is shown in Table 5. Among them, when the PK (primary key) identifier is the same, the data with the latest timestamp is used:

[0101] Table 5

[0102]

[0103] However, since file3 is deleted, all the contents in file3 must be deleted. PK=2 and PK=3 are the deleted contents in file3. Therefore, the results are as shown in Table 6:

[0104] Table 6

[0105]

[0106] However, since file4 is created, the content in file4 needs to be added. PK3, PK=4, PK=5, and PK=6 are all newly added content in file4. Therefore, the results are as shown in Table 7:

[0107] Table 7

[0108]

[0109] Because PK=4 is added after deletion, which is equivalent to no change. The snapshot metadata table records the changed data, so PK=4 is not recorded in the initial change data. PK=2, on the other hand, is not added after deletion, which is equivalent to a deletion operation. Therefore, it needs to be recorded in the initial change data. The initial data table is shown in Table 8:

[0110] Table 8

[0111]

[0112] At this point, it is determined that PK=2 is deleted and PK=3, 5, 6 are created. However, whether PK=3, 5, 6 are actually created needs to be determined based on the tombstone data, that is, the data stored in the tombstone data file;

[0113] Assume that the file content of tb_file3 is as shown in Table 9. From Table 9, we can see that PK=6 has been deleted. Therefore, we need to filter out the change record with PK=6:

[0114] Table 9

[0115]

[0116] According to the contents of Table 8 and Table 9, the final change data is obtained, as shown in Table 10:

[0117] Table 10

[0118]

[0119] Therefore, the changes during this time period are: the actual business data corresponding to PK=2 is deleted, and the business data corresponding to PK=3 and PK=5 are created. Based on the above analysis, we can conclude that snapshot-based change capture can merge some historical operations, reducing the amount of data synchronized downstream. Specifically, the data sent downstream is DELETE PK=2, INSERT PK=3, and INSERT PK=5.

[0120] Based on the above embodiment of the present application, in another embodiment of the present application, the same or similar contents as the above embodiment can be referred to the above introduction, and no further details will be given later. The data change capture method further includes steps D10 to D30:

[0121] Step D10, responding to an operation instruction on the data file and determining whether the operation instruction is a delete operation;

[0122] It should be noted that when the data change capture system receives an operation instruction (such as write, update, or delete) for a data file, it first analyzes the operation type of the operation instruction, specifically, identifying whether the operation instruction is a delete operation.

[0123] Step D20: If it is a delete operation, a tombstone data file is created; if it is a non-delete operation, a table data file is created;

[0124] It should be noted that if the operation is a delete, a tombstone data file is generated to store the deleted data in the table. The tombstone data file includes the PK (primary key identifier), tombstone_rowid (target field, whose value is the physical address of the target deleted row), and CommitTS (a hidden column that stores the commit timestamp corresponding to the data change operation). If the operation is non-delete (such as add or update), a table data file is created to store the actual business data. The table data file includes the PK (primary key identifier), Column (column value), and CommitTS (a hidden column that stores the commit timestamp corresponding to the data change operation).

[0125] For example, except for the different organizational structures, the storage structure of the table data file and the tombstone data file is exactly the same. The table data file and the tombstone data file of the same table are stored separately and associated through tombstone_rowid. They can be stored in multiple files in a column-based organization form, forming two file lists.

[0126] Through step D10, valid data and deletion markers can be physically isolated, so that the change capture process can directly locate the deletion target, avoiding the overhead of reversely inferring the deletion relationship from the log in traditional solutions.

[0127] Step D30 , writing the operation record of the tombstone data file or the table data file into the snapshot metadata table, wherein the operation record includes the data file type identifier, the operation type and the corresponding commit timestamp.

[0128] It's important to note that whether you create a tombstone file or a table data file, a change record must be written to the snapshot metadata table. Each record can include a data file type flag (IsTombstone) to clearly distinguish whether the current file is a tombstone file (true) or a table data file (false); an operation type (OP_Type) to record the file-level operation (e.g., CREATE for adding a file, DELETE for deleting a file); and a commit timestamp (CommitTS) to mark the precise time when the operation took effect.

[0129] For example, a delete operation generates a record with IsTombstone=true and OP_Type=CREATE (indicating a newly added tombstone file), while a non-delete operation generates a record with IsTombstone=false and OP_Type=CREATE. This step constructs a complete timeline of file change events, providing a foundation for subsequent time-window-based snapshot analysis.

[0130] In this embodiment, when the system receives a data file operation instruction, it first identifies the operation type. If it is a deletion operation, a tombstone data file is generated specifically to store the deleted data. If it is not a deletion operation, a table data file is generated to store the actual business data. The file creation process is simultaneously recorded in the snapshot metadata table. This processing method generates metadata records when a file changes, fundamentally avoiding the metadata ambiguity caused by mixed writing of transaction logs in traditional solutions. Recording file change status and corresponding timestamps in the snapshot metadata table not only eliminates the file path inference process required for traditional log parsing, but more importantly, through pre-classification of file type identification, subsequent change capture can directly skip the redundant primary key matching calculations in massive data filtering, thus minimizing the change data extraction path.

[0131] Traditional data change capture systems face technical issues such as privacy leakage risks and compliance deficiencies. Specifically, change data is stored in plain text in physical files, and any direct access to table data files or tombstone files can leak privacy. Furthermore, traditional encryption schemes prevent change data from being parsed properly.

[0132] In one embodiment, before step D30, the following steps are included:

[0133] Check whether the operation record contains privacy fields;

[0134] If a privacy field is included, the preset encryption key is used to encrypt the privacy field and generate an encrypted metadata record;

[0135] The encrypted metadata record is stored in the snapshot metadata table. When reading the valid table data file or valid tombstone data file, the privacy field is automatically decrypted according to the encrypted metadata record.

[0136] It should be noted that when processing operation records, the system first performs privacy field scanning and detection through a predefined sensitive field rule library (such as field identifiers containing keywords such as "ID card" and "mobile phone number"); when a field containing privacy attributes is identified, the preset encryption key is called to perform encryption algorithm conversion on the privacy field to generate a structured encrypted record containing ciphertext data and encryption metadata (such as algorithm identifier, initial vector, key version number, etc.); the encrypted record is then stored in the snapshot metadata table to form a mapping binding from the plaintext data field to the encrypted record.

[0137] When reading physical files subsequently (whether table data files or tombstone files), the system automatically reconstructs the decryption environment parameters by parsing the associated encrypted metadata records to complete the plaintext restoration of the private fields.

[0138] In this embodiment, through the security process of privacy field detection, real-time encryption to seamless decryption, sensitive data is always encrypted in storage and is only decrypted on demand during business use, achieving privacy protection while ensuring efficient change capture.

[0139] In distributed databases, table data files and tombstone files are usually stored in external heterogeneous storage systems. Traditional data change capture solutions require pulling the full data from remote storage across the network each time a file is read, resulting in capture delays.

[0140] In another embodiment, the data change capture method further includes:

[0141] When reading a valid table data file or a valid tombstone data file, determining whether the valid table data file or the valid tombstone data file is located in an external storage location according to a storage path identifier;

[0142] If it is located in an external storage location, the file contents of the valid table data file or the valid tombstone data file are loaded from the external storage location to the local storage space and then read.

[0143] It's important to note that during the file read phase of the data change capture system, the physical location of the file is determined by parsing the storage path identifier. If the storage path identifier contains a characteristic protocol identifier pointing to an external storage system (such as cloud object storage, distributed file systems, or other non-local disk storage facilities), the system triggers the secure load protocol: first, the file content is fully retrieved from the external storage location through an encrypted channel and temporarily stored in local storage space (an isolated temporary storage area at the operating system level). The file content is then parsed in the local storage space.

[0144] In this embodiment, this conversion of cross-network access into local reading not only eliminates the risk of latency fluctuations in remote reading, but also establishes a security boundary through data localization, blocking the attack surface that may be generated by the external storage interface, and realizing an efficient and secure access closed loop for external files.

[0145] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the data change capture method of the present application. More forms of simple transformations based on this technical concept, such as the interaction and combination of various embodiments, are all within the scope of protection of the present application.

[0146] The present application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the data change capture method in the above-mentioned first embodiment.

[0147] Reference below Figure 2 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application. The electronic devices in the embodiments of the present application may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (such as in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 2 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0148] like Figure 2As shown, the electronic device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in a read-only memory 1002 or programs loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the electronic device. The processing device 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems may be connected to the input / output interface 1006: an input device 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; a storage device 1003 including, for example, a magnetic tape or hard disk; and a communication device 1009. The communication device 1009 may allow the electronic device to communicate with other devices wirelessly or wired to exchange data. Although the figures show electronic devices with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have instead.

[0149] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a read-only memory 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are performed.

[0150] The electronic device provided in this application utilizes the data change capture method in the above-described embodiment to address the low efficiency of existing data change processing. Compared to the prior art, the electronic device provided in this application achieves the same beneficial effects as the data change capture method in the above-described embodiment. Other technical features of the electronic device are the same as those disclosed in the above-described embodiment and are not further detailed here.

[0151] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0152] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0153] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer program) stored thereon, wherein the computer-readable program instructions are used to execute the data change capture method in the above-mentioned embodiment.

[0154] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0155] The computer-readable storage medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0156] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by an electronic device, the electronic device: constructs a snapshot metadata table, and records the creation operation, deletion operation and the submission timestamp corresponding to the operation of the data file through the snapshot metadata table, wherein the data file includes a table data file and a tombstone data file, the actual business data of the table is stored in the table data file, and the deletion data of the table is stored in the tombstone data file; by querying the submission timestamp in the snapshot metadata table, the valid file list within the target time period for capturing the data change is determined, wherein the valid file list includes valid table data files and / or valid tombstone data files; within the target time period, the valid file list is filtered and merged based on the submission timestamp to obtain the final change data.

[0157] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0158] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0159] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0160] The computer-readable storage medium provided in this application stores computer-readable program instructions (i.e., a computer program) for executing the aforementioned data change capture method. This computer-readable storage medium can address the low efficiency of existing data change processing. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are similar to those of the data change capture method provided in the aforementioned embodiments and are not further elaborated here.

[0161] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned data change capture method when executed by a processor.

[0162] The computer program product provided by this application can solve the technical problem of low efficiency in processing changed data. Compared with the existing technology, the beneficial effects of the computer program product provided by this application are the same as those of the data change capture method provided by the above embodiment, and will not be repeated here.

[0163] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A data change capture method, characterized in that: The data change capture method includes: Constructing a snapshot metadata table to record creation and deletion operations of data files and corresponding commit timestamps of the operations, wherein the data files include table data files and tombstone data files. The table data files store the actual business data of the table, and the tombstone data files store the deletion data of the table. Determine a valid file list within a target time period for capturing data changes by querying the commit timestamp in the snapshot metadata table, wherein the valid file list includes valid table data files and valid tombstone data files; The valid file list is filtered and merged based on the submission timestamp within the target time period to obtain final change data. The step of filtering and merging the valid file list based on the submission timestamp within the target time period to obtain the final change data includes: merging the data in the valid table data file based on the submission timestamp to obtain initial change data; establishing a mapping relationship between the physical address and the initial change data based on the physical address indicated by the target field in the valid tombstone data file, and removing the invalid data in the valid table data file based on the mapping relationship to obtain the final change data, wherein the target field in the valid tombstone data file is associated with the physical address corresponding to the data in the valid table data file.

2. The data change capture method according to claim 1, wherein: The step of determining a valid file list within a target time period for capturing data changes comprises: If, when querying the snapshot metadata table, the commit timestamp of the creation operation of the data file is less than or equal to the target time period, and the deletion operation of the same data file is before the target time period, then each data file is added to the valid file list of the target time period.

3. The data change capture method according to claim 1, wherein: The data change capture method further includes: responding to an operation instruction on a data file and determining whether the operation instruction is a delete operation; If it is a delete operation, a tombstone data file is created; if it is a non-delete operation, a table data file is created; The operation record of the tombstone data file or the table data file is written into the snapshot metadata table, wherein the operation record includes a data file type identifier, an operation type, and a corresponding commit timestamp.

4. The data change capture method according to claim 3, wherein: Before the step of writing the operation record of the tombstone data file or the table data file into the snapshot metadata table, the method includes: Check whether the operation record contains privacy fields; If a privacy field is included, the privacy field is encrypted using a preset encryption key to generate an encrypted metadata record; The encrypted metadata record is stored in the snapshot metadata table, and when reading a valid table data file or a valid tombstone data file, the privacy field is automatically decrypted according to the encrypted metadata record.

5. The data change capture method according to claim 1, wherein: The snapshot metadata table records the storage path identifier of the data file; The data change capture method includes: When reading a valid table data file or a valid tombstone data file, determining whether the valid table data file or the valid tombstone data file is located in an external storage location according to the storage path identifier; If it is located in an external storage location, the file content of the valid table data file or the valid tombstone data file is loaded from the external storage location to the local storage space and then read.

6. A data change capture system, characterized in that: The data change capture system executes the steps of the data change capture method according to any one of claims 1 to 5 when responding to the data change capture request.

7. An electronic device, characterized in that: The electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data change capture method according to any one of claims 1 to 5.

8. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data change capture method according to any one of claims 1 to 5 are implemented.

9. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the data change capture method according to any one of claims 1 to 5 are implemented.

Images