Data synchronization method and apparatus, electronic device, and computer program product

Abstract

This application discloses a data synchronization method, a data synchronization device, an electronic device, and a computer program product. The method includes: constructing source event data corresponding to the source data to be synchronized; determining a corresponding custom data synchronization processing identifier based on the source event data; dynamically loading and executing corresponding custom data synchronization processing logic based on the custom data synchronization processing identifier to perform custom data synchronization processing on the source event data to obtain target event data; and synchronizing the target event data to a target endpoint. This application's solution can improve the flexibility of data synchronization and meet the synchronization needs of various types of data.

Description

Technical Field

[0001] This application belongs to the field of database technology, and in particular relates to a data synchronization method, a data synchronization device, an electronic device, and a computer program product. Background Technology

[0002] In data synchronization technology, data is typically synchronized from a source database to a target database according to a pre-defined mapping relationship using a data synchronization service to achieve data transfer between different databases. Currently, the data synchronization process is generally based on fixed synchronization rules, such as directly copying data to the target end according to a pre-configured mapping relationship. While this approach can meet the needs of routine data migration or data replication, because the synchronization rules are fixed during task configuration, it is difficult to perform differentiated processing based on the characteristics of different source data during the data synchronization process. This results in poor flexibility in data synchronization when facing scenarios with complex business structures or diverse data formats, making it difficult to meet different data synchronization needs. Summary of the Invention

[0003] This application provides a data synchronization method, a data synchronization device, an electronic device, and a computer program product, which can improve the flexibility of data synchronization and meet the synchronization needs of various types of data.

[0004] Firstly, this application provides a data synchronization method, including: Construct the source event data corresponding to the source data to be synchronized; Determine the corresponding custom data synchronization processing identifier based on the source event data; Based on the custom data synchronization processing identifier, the corresponding custom data synchronization processing logic is dynamically loaded and executed to perform custom data synchronization processing on the source event data to obtain the target event data; Synchronize the target event data to the target device.

[0005] Secondly, this application provides a data synchronization device, comprising: The first construction module is used to construct the source event data corresponding to the source data to be synchronized; The determination module is used to determine the corresponding custom data synchronization processing identifier based on the source event data; The first processing module is used to dynamically load and execute the corresponding custom data synchronization processing logic based on the custom data synchronization processing identifier, so as to perform custom data synchronization processing on the source event data to obtain the target event data. The synchronization module is used to synchronize target event data to the target end.

[0006] Thirdly, this application provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the method described in the first aspect.

[0007] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described in the first aspect above.

[0008] Fifthly, this application provides a computer program product comprising a computer program that, when executed by one or more processors, implements the steps of the method described in the first aspect.

[0009] The beneficial effects of this application compared to the prior art are as follows: This application constructs source event data corresponding to source data and determines a corresponding custom data synchronization processing identifier based on the source event data, thereby enabling the selection of corresponding data synchronization processing methods for different source data. Furthermore, based on the custom data synchronization processing identifier, the corresponding custom data synchronization processing logic is dynamically loaded and executed to process the source event data and obtain target event data, which is then synchronized to the target end. Thus, by dynamically loading and executing different data synchronization processing logics during the data synchronization process, different source data can be processed using different methods before synchronization, thereby improving the flexibility of the data synchronization process. It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect above, and will not be repeated here. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram illustrating the implementation process of the data synchronization method provided in the embodiments of this application; Figure 2 This is a structural block diagram of the data synchronization device provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0012] The embodiments of the technical solution of this application will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of this application and are therefore merely examples, and should not be used to limit the scope of protection of this application.

[0013] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0014] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly indicating the number, specific order, or primary and secondary relationship of the indicated technical features.

[0015] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0016] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0017] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), unless otherwise expressly and specifically defined.

[0018] This application provides a data synchronization method. This data synchronization method can be applied to an electronic device that is equipped with a Data Transmission Service (DTS). Please refer to... Figure 1 , Figure 1 The implementation flow of the data synchronization method applied to this electronic device is presented, and detailed below: Step 101: Construct the source event data corresponding to the source data to be synchronized.

[0019] The data synchronization service first acquires the source data to be synchronized and constructs it into source event data. Source data refers to the original data records to be synchronized, retrieved from the source database. Source event data is a data object obtained after unified processing and encapsulation of the source data, used as a unified data processing carrier during the data synchronization process. Specifically, source event data may include information such as the library identifier, table identifier, field data, and operation type corresponding to the source data, used to describe a data record to be synchronized.

[0020] In practice, the data synchronization service can read the data to be synchronized from the source database, parse each data record into its corresponding field structure, and then encapsulate the parsed data into source event data according to a preset data structure. For example, after reading a record from the source database, the data synchronization service can encapsulate the database name, table name, and field values ​​in that record into a source event data object, thereby obtaining source event data in a unified format.

[0021] This step converts source data from different sources and / or with different structures into standardized data objects, facilitating unified data processing in subsequent steps and providing a basis for determining the appropriate processing method based on the source event data.

[0022] Step 102: Determine the corresponding custom data synchronization processing identifier based on the source event data.

[0023] After obtaining the source event data, the data synchronization service can determine the corresponding custom data synchronization processing identifier based on the source event data. Specifically, this custom data synchronization processing identifier is an identifier that indicates which custom data synchronization processing logic is used. In some examples, this custom data synchronization processing identifier can be a unique identifier, such as a processing logic number, processing class identifier, or processing logic name. It can be understood that different custom data synchronization processing identifiers can correspond to different data processing methods.

[0024] This step allows you to select different data processing methods for different source data during the data synchronization process, thus providing a basis for dynamically loading and executing the corresponding data synchronization processing logic later.

[0025] Step 103: Dynamically load and execute the corresponding custom data synchronization processing logic based on the custom data synchronization processing identifier to perform custom data synchronization processing on the source event data to obtain the target event data.

[0026] Custom data synchronization processing logic refers to the logical unit that processes source event data during data synchronization; different custom data synchronization processing logics can implement different data processing methods. Target event data refers to the data object processed by the custom data synchronization processing logic; its data content may differ from the source event data, and it is used to ultimately synchronize to the target end.

[0027] In some examples, custom data synchronization processing logic can be executed by instances of custom data synchronization processing classes that implement a predefined EventProcessor interface. The EventProcessor interface takes source event data as input and outputs target event data. Here, the EventProcessor interface is a unified interface specification for defining custom data synchronization processing logic; a custom data synchronization processing class instance refers to a concrete processing class object that implements the EventProcessor interface, used to execute the corresponding data processing logic during data synchronization. By defining a unified EventProcessor interface, different custom data synchronization processing logics can exist in the form of different implementation classes, and each implementation class processes data according to the same input and output method. This facilitates subsequent expansion of possible processing operations based on actual needs.

[0028] Specifically, the EventProcessor interface is used to receive source event data as input and output processed target event data. Since different custom data synchronization processing logics may produce different processing results for the source event data, the output of the EventProcessor interface can be target event data, and the number of target event data is not limited, thus supporting the generation of zero, one, or multiple target event data corresponding to one source event data.

[0029] In practice, after determining the corresponding processing logic based on the custom data synchronization processing identifier, the data synchronization service can locate the custom data synchronization processing class that implements the EventProcessor interface and instantiate this class to obtain a custom data synchronization processing class instance. Subsequently, the data synchronization service calls the processing method provided by this custom data synchronization processing class instance, passing the source event data as input to the instance for processing to obtain the target event data. For example, when the source event data corresponds to a certain custom data synchronization processing identifier, the data synchronization service loads the corresponding processing class that implements the EventProcessor interface and creates an instance of that class; then, the source event data is input into the processing class instance, which executes the corresponding data processing logic and outputs the processed target event data.

[0030] Based on the implementation process of the custom data synchronization processing logic described above, this step can be implemented in the following way: First, the data synchronization service can determine the target processing class information based on a custom data synchronization processing identifier. This custom data synchronization processing identifier can correspond to unique processing class configuration information, which indicates the location and class identifier of the custom data synchronization processing class. For example, the processing class configuration information may include an extended directory, the processing class file name, and the processing class path information, etc., which are not limited here.

[0031] Next, the data synchronization service can locate the target processing class based on the processing class configuration information. Specifically, the data synchronization service locates the corresponding extension component directory based on the extension directory, searches for the corresponding processing class file in that directory, and then determines the custom data synchronization processing class that implements the corresponding EventProcessor interface based on the processing class path information.

[0032] Then, the data synchronization service can dynamically load custom data synchronization processing classes. Through a class loading mechanism, the data synchronization service can dynamically load custom data synchronization processing classes during runtime and create corresponding instances of these classes. Dynamic loading means loading the corresponding data synchronization processing logic during runtime based on a defined custom data synchronization processing identifier, rather than loading it at system initialization. Because the custom data synchronization processing classes are dynamically loaded during runtime, different custom data synchronization processing classes can be loaded as needed based on different custom data synchronization processing identifiers, without having to load all custom data synchronization processing classes during system initialization.

[0033] Finally, the data synchronization service can execute custom data synchronization processing class instances. That is, the data synchronization service can call the EventProcessor interface of the custom data synchronization processing class instance, pass in the source event data, and execute the corresponding data processing logic.

[0034] This step involves using a custom data synchronization processing class instance that implements the preset EventProcessor interface to execute custom data synchronization processing logic. This allows for modular encapsulation of different data processing methods and calls through a unified interface. This enables flexible expansion of different data processing logic during data synchronization while ensuring consistent input and output structures across different processing logics. Furthermore, the corresponding processing class can be dynamically loaded during runtime based on different processing identifiers, thereby enabling on-demand expansion of custom data synchronization processing logic and improving system scalability.

[0035] Step 104: Synchronize the target event data to the target device.

[0036] After obtaining the target event data, the data synchronization service can synchronize the target event data to the target endpoint. The target endpoint refers to the target database or target data storage system that receives the synchronized data. Furthermore, synchronizing to the target endpoint means generating corresponding data records on the target endpoint based on the data content within the target event data, thereby completing the data synchronization.

[0037] In practice, the data synchronization service can generate corresponding data operation statements based on the table information and field data in the target event data, and send these statements to the target end for execution, thereby writing the target event data to the target end. Alternatively, the data synchronization service can directly call the data writing interface provided by the target end to write the target event data to the target end.

[0038] In some examples, the data synchronization service can generate corresponding data insertion or update operations based on the target event data and execute them in the target database, thereby synchronizing the processed data to the target database.

[0039] This step allows you to write the data, which has undergone custom data synchronization processing, into the target system, thereby completing the data synchronization.

[0040] In some embodiments, the data synchronization method proposed above is applicable not only to full synchronization scenarios but also to incremental synchronization scenarios. The processing flow for these two types of synchronization scenarios is basically the same; both can be processed according to the custom data synchronization processing logic described above, except for the difference in the method of acquiring the source event data in the early stages. The process of acquiring source event data in these two scenarios is described below: In a full synchronization scenario, step 101 may specifically include: A1 retrieves the snapshot data obtained from the source database after the snapshot backup.

[0041] The data synchronization service can initiate a snapshot backup request to the source database, which will then generate corresponding snapshot data; alternatively, it can export the entire source database using a data export tool; or it can obtain snapshot data by streaming the full data. Snapshot data refers to the data set obtained after taking a snapshot backup of the source database at a specific moment, reflecting the full data state of the source database at that time. This snapshot data can be stored as a file or transmitted as a data stream. This application does not limit the specific method of obtaining the snapshot data, as long as the full data of the source database can be obtained.

[0042] This step allows you to obtain the full data content of the source database, thus providing the original data source for subsequent construction of source event data.

[0043] A2 parses the snapshot data to obtain the source event data.

[0044] The data synchronization service can parse each data record in the snapshot data, convert each parsed record into a data object in a unified format, and construct source event data. Specifically, the data synchronization service can read each record from the snapshot data, parse out the corresponding table information and field data, and then encapsulate the parsed information into a source event data object.

[0045] For example, the data synchronization service reads a data record belonging to a source table from the snapshot data, parses out the table identifier and field values ​​of the record, and encapsulates the parsed data into a source event data record, thus obtaining the corresponding source event data. For multiple records in the snapshot data, the data synchronization service can repeat the above process to construct multiple source event data records respectively.

[0046] This step converts all data in the source database into a unified source event data format, enabling consistent data processing in subsequent processes and providing a unified data input for performing custom data synchronization processing based on the source event data.

[0047] In the scenario of incremental synchronization, step 101 may specifically include: B1 retrieves incremental data from the source database using a pre-defined change data capture tool.

[0048] Data synchronization services can use change data capture tools to read data change information from the source database's log system, such as the database's change log or transaction log, and parse out the changed data records, i.e., incremental data. The change data capture tool is a component used to capture data change information in the source database, specifically to obtain incremental data. This incremental data refers to the data changes that occurred in the source database within a certain time period due to add, modify, or delete operations.

[0049] This step allows for timely acquisition of data changes in the source database, providing an incremental data source for subsequent construction of source event data. Synchronization is performed only on data that has changed, thereby improving data synchronization efficiency.

[0050] B2 filters and extracts incremental data to obtain source event data.

[0051] After acquiring incremental data, it needs to be filtered and extracted because this data may contain information that does not require synchronization. Specifically, the data synchronization service can filter data records that meet preset synchronization conditions from the incremental data and remove data that does not need to be synchronized. This data that does not need to be synchronized includes, but is not limited to, heartbeat data, DDL statement data, query statement data, and table data that does not need to be synchronized (without configured table mapping relationships). Subsequently, the data synchronization service extracts the corresponding data records from the filtered incremental data and converts the extracted data records into data objects of a unified format, thereby constructing the source event data.

[0052] For example, the data synchronization service extracts a data change record from incremental data, parses the corresponding table information and field data, and encapsulates the parsed data into a source event record, thus obtaining the corresponding source event data. For multiple incremental data records, the above process can be repeated to construct the corresponding source event data for each record.

[0053] This step converts incremental changes in the source database into a unified source event data format, enabling consistent data processing in subsequent processes and providing a unified data input for custom data synchronization based on the source event data.

[0054] In some embodiments, step 102 specifically includes: C1 retrieves the synchronization task identifier information and source table information from the source event data.

[0055] The source table information refers to the table identifier information in the source database corresponding to the source event data, used to identify which table in the source database the source event data originates from. The source table information may include at least one of the database name and table name, used to uniquely identify the table to which the source data belongs.

[0056] The synchronization task identifier is used to identify data synchronization tasks, and different data synchronization tasks correspond to different synchronization task identifiers. Specifically, the synchronization task identifier can be a pipelineId, which is used to uniquely identify a data synchronization task.

[0057] Specifically, when constructing source event data based on the current synchronization requirements, the data synchronization service has already encapsulated the table identifier information and the corresponding data synchronization task identifier information of the source data within the source event data. Therefore, the data synchronization service can directly read the corresponding source table information and the corresponding data synchronization task identifier information from the source event data to determine the synchronization task and source table to which the source event data belongs.

[0058] It is understandable that in practical applications, multiple data synchronization tasks may exist simultaneously in a data synchronization system, meaning there may be multiple different pipelineIds; furthermore, a single data synchronization task may correspond to multiple table mapping relationships. Therefore, the corresponding table mapping relationship cannot be uniquely determined solely based on the source table information; it is necessary to combine the synchronization task identifier information and the source table information to determine the corresponding table mapping relationship. Hence, it is necessary to obtain the synchronization task identifier information and the source table information here.

[0059] For example, two different data synchronization tasks may have source tables with the same name, but their corresponding table mapping relationships may differ due to different data synchronization goals or processing logic. Therefore, it is necessary to combine the pipelineId and source table information for matching.

[0060] C2 queries the pre-configured table mapping relationship based on the synchronization task identifier information and the source table information.

[0061] Table mapping relationships describe the correspondence between source tables and target data structures. These mapping relationships can be pre-configured during the data synchronization task configuration phase and used to find the corresponding processing rules for different source tables during data synchronization. Specifically, a table mapping relationship can simultaneously associate synchronization task identifier information and source table information, allowing source tables with the same name in different data synchronization tasks to correspond to different table mapping relationships. Furthermore, the table mapping relationship can also associate corresponding custom data synchronization processing identifiers based on the actual synchronization processing requirements.

[0062] After obtaining the synchronization task identifier and source table information, the data synchronization service can perform a matching query in the pre-configured table mapping relationship based on the synchronization task identifier and the source table information to obtain the table mapping relationship corresponding to the synchronization task identifier and the source table. For example, when the synchronization task identifier corresponds to the first data synchronization task and the source table information corresponds to a certain source table, the data synchronization service can search for table mapping relationships that match both the synchronization task identifier and the source table information in the table mapping relationship configuration and obtain the corresponding mapping configuration.

[0063] C3 determines the corresponding custom data synchronization processing identifier from the table mapping relationship.

[0064] After obtaining the table mapping relationship, the data synchronization service can read the corresponding custom data synchronization processing identifier from the table mapping relationship and use this custom data synchronization processing identifier as the basis for dynamically loading custom data synchronization processing logic. For example, if a custom data synchronization processing identifier is configured for a source table of a data synchronization task in the table mapping relationship, the data synchronization service can determine the corresponding custom data synchronization processing identifier when it matches that source table of the data synchronization task.

[0065] It is important to note that if, for a given source table in a data synchronization task, the corresponding custom data synchronization processing identifier cannot be determined from the table mapping relationship—that is, if no associated custom data synchronization processing identifier exists in the table mapping relationship—then custom synchronization processing is unnecessary for the data to be synchronized. This is the most common data synchronization scenario, where it is sufficient to directly perform database-table-column mapping and synchronization based on the table mapping relationship, ensuring complete data consistency before and after synchronization. In other words, the custom data synchronization processing proposed in this application can be used in scenarios where additional computational processing is required before and after synchronization, and the processed data is inconsistent with the original data, such as data splitting, data merging, data processing, or data filtering scenarios discussed later.

[0066] In some embodiments, the process of a custom data synchronization processing instance performing custom data synchronization processing includes: D1 receives source event data through the EventProcessor interface.

[0067] After loading and creating an instance of the custom data synchronization processing class, the data synchronization service calls the corresponding EventProcessor interface to pass the source event data to be processed into that custom data synchronization processing class instance. As described above, the EventProcessor interface can serve as a unified data processing entry point, enabling different custom data synchronization processing class instances to receive source event data in a consistent manner.

[0068] D2 performs at least one of the following processing methods based on the source event data: data splitting, data merging, data processing, or data filtering.

[0069] The data synchronization service supports any of the following custom processing operations before data synchronization: data splitting, data merging, data processing, and data filtering. Data splitting refers to the process of splitting a source event data set into multiple target event data sets, such as splitting a set field in the source event data into multiple records. Data merging refers to the process of generating a target event data set based on multiple fields or related data in the source event data. Data processing refers to the process of generating target event data by performing calculations, transformations, or supplementary processing on the field data in the source event data. Data filtering refers to the process of determining whether to discard source event data based on preset conditions.

[0070] After receiving source event data, a custom data synchronization processing class instance can perform at least one of the above processing methods on the source event data according to the needs of the actual scenario. For example, in some scenarios, the custom data synchronization processing class instance can perform data splitting processing on the source event data to generate multiple target event data; in other scenarios, it can also perform data processing on the source event data, transforming the field data to generate target event data. This application embodiment does not limit the specific processing method used.

[0071] D3 outputs the processed target event data.

[0072] After completing the above processing, the custom data synchronization processing class instance can output the processing result as target event data. This target event data is the data object obtained after the custom data synchronization processing, and its quantity can be one or more. It should be noted that when data filtering is performed and the filtering conditions are met, the number of output target event data can also be 0, thus achieving filtering of the source event data.

[0073] Through the above steps, the data synchronization service can perform at least one of the following processing methods on the source event data based on the EventProcessor interface: data splitting, data merging, data processing, or data filtering. This enables the data synchronization process to support multiple data processing methods, thereby improving the flexibility of data synchronization.

[0074] In some examples, the various possible processing methods mentioned above are explained in detail below: Regarding data splitting: E1 receives source event data through the EventProcessor interface.

[0075] E2 performs data splitting based on whether external data sources need to be accessed, resulting in individual sub-data items.

[0076] E3 generates corresponding event data based on each sub-data item.

[0077] E4 outputs multiple generated event data as target event data.

[0078] A custom data synchronization processing class instance obtains source event data through the EventProcessor interface and parses the fields within it to determine the fields requiring splitting. Without accessing an external data source, the instance can extract the fields to be split from the source event data and parse them to obtain multiple sub-data items. For example, when the field to be split is a collection type field, the instance treats each item in the collection as a sub-data item. When accessing an external data source is required, the instance accesses the external data source based on key fields in the source event data, queries multiple data records associated with the source event data, and then uses these records as the split sub-data items. The instance can then copy common fields from the source event data for each sub-data item and write the sub-data item into the corresponding fields, thereby generating multiple event data records and obtaining the final output target event data.

[0079] By using the above methods, a source event data can be split into multiple target event data, thereby meeting the requirement of expanding a single data into multiple data before synchronization.

[0080] Regarding data merging: F1 receives source event data through the EventProcessor interface.

[0081] F2 performs data merging processing based on whether external data sources need to be accessed.

[0082] F3 constructs target event data based on target field data.

[0083] F4 outputs the constructed target event data.

[0084] A custom data synchronization processing class instance receives source event data through the EventProcessor interface and retrieves multiple fields from the source event data that need to be merged. When no external data source is needed, the custom data synchronization processing class instance extracts multiple fields from the source event data and performs aggregation calculations on these fields according to preset merging rules. For example, it performs concatenation or calculations on these fields to generate the merged target field data. When an external data source is needed, the custom data synchronization processing class instance accesses the external data source based on key fields in the source event data, retrieves data records associated with the source event data from the external data source, and then performs aggregation calculations on the fields to be merged in the source event data with the data retrieved from the external data source to generate the merged target field data. It's important to note that the records before and after merging can come from different tables; after merging, simply update the `tableName` of `EventData`. The custom data synchronization processing class instance can then write the merged field data into the target event data and update the corresponding field content.

[0085] The above methods can be used to merge multiple fields or multiple related data into a single target event data, thereby meeting the need for data aggregation and then synchronization.

[0086] For data processing: G1 receives source event data through the EventProcessor interface.

[0087] G2 performs data processing based on whether or not external data sources need to be accessed.

[0088] G3 writes the processed field data into the event data.

[0089] G4 outputs the processed target event data.

[0090] A custom data synchronization processing class instance receives source event data through the EventProcessor interface and parses the field content within it. When no external data source is needed, the instance performs calculations on the field data according to preset processing rules. For example, it performs type conversions, numerical calculations, time zone processing, or field mapping on the field data to generate processed field data. When external data source access is required, the instance accesses the external data source based on key fields in the source event data, retrieves relevant data, and then performs calculations and transformations on the retrieved data and the field data in the source event data. It can even delete certain fields to generate processed field data. The instance then uses the processed field data to update the field content in the target event data, or it can add corresponding field content.

[0091] The above methods allow for flexible data processing of source event data, thereby meeting the needs of field transformation or field supplementation.

[0092] For data filtering: H1 receives source event data through the EventProcessor interface.

[0093] H2 performs data filtering based on whether external data sources need to be accessed.

[0094] H3 determines whether to output the target event data based on the judgment result.

[0095] H4 outputs the filtered target event data.

[0096] A custom data synchronization processing class instance obtains source event data through the EventProcessor interface and parses the field data within it. When no external data source is needed, the instance compares the field values ​​in the source event data with preset filtering conditions. For example, it determines whether to filter the source event data based on the field value range or matching criteria. When external data source access is required, the instance accesses the external data source based on key fields in the source event data, retrieves data related to the source event data, and then combines the query results with preset filtering rules to determine whether filtering is necessary. When the filtering conditions are met, the instance does not output the target event data (i.e., the output target event data is empty); when the filtering conditions are not met, the source event data is output as the target event data.

[0097] By using the above methods, data that meets the criteria can be filtered during the data synchronization process, thereby preventing unwanted data from being synchronized to the target end and improving the accuracy of data synchronization.

[0098] In some embodiments, since the above data synchronization method solves the limitation that the source data and target data must be consistent in traditional data synchronization, the source data and target data can be inconsistent. Therefore, its data verification process also differs from the data verification process of traditional data synchronization. Based on this, the data verification process in the embodiments of this application is described below: I1 constructs the source verification event data corresponding to the source data to be verified in the source database.

[0099] The source data to be verified refers to the data records in the source database used to perform data consistency verification. The source verification event data is a data object obtained by encapsulating the source data to be verified. Its data structure is consistent with the aforementioned source event data, and it is used as a unified data processing input during the data verification process.

[0100] The data synchronization service can read the source data to be verified from the source database and encapsulate it into source verification event data in the same way as source event data is constructed. For example, the data synchronization service can read a record from the source database and encapsulate the table information and field data of that record into a source verification event data, thereby obtaining the corresponding source verification event data.

[0101] I2 executes the corresponding custom data synchronization processing logic on the source verification event data to obtain the expected event data.

[0102] The data synchronization service can determine the corresponding custom data synchronization processing identifier based on the source verification event data and load the corresponding custom data synchronization processing logic. This allows it to execute the same custom data synchronization processing logic on the source verification event data as in the data synchronization phase, resulting in the processed expected event data. It can be understood that after processing the expected event data according to this custom data synchronization processing logic, it should theoretically be synchronized to the target end, providing a basis for subsequent consistency comparisons with the target end data.

[0103] I3 retrieves the corresponding target event data from the target end based on the expected event data.

[0104] The data synchronization service can query the corresponding data records from the target end based on the identification information (such as table information and primary key information) in the expected event data, and encapsulate the retrieved data into target event data. In other words, the target event data is actually the actual synchronization result data obtained from the target end.

[0105] I4 compares the expected event data with the target event data to complete the data verification.

[0106] The data synchronization service compares the field values ​​of the expected event data and the target event data field by field to determine whether the data validation passes based on the comparison results. When the data content is consistent, the data synchronization result is correct; when differences exist, inconsistencies exist during the data synchronization process. Therefore, the data synchronization service can verify whether data processed by custom data synchronization is correctly synchronized to the target end, thereby validating the data synchronization results and improving the reliability of data synchronization.

[0107] Through the above steps, the data synchronization service can execute custom data synchronization processing logic consistent with the data synchronization phase on the source verification event data, and perform consistency comparison with the target data, thereby verifying whether the data synchronization result after custom data synchronization processing is correct.

[0108] As can be seen from the above, this embodiment constructs source event data corresponding to the source data and determines a corresponding custom data synchronization processing identifier based on the source event data, thereby enabling the selection of appropriate data synchronization processing methods for different source data. Furthermore, based on the custom data synchronization processing identifier, the corresponding custom data synchronization processing logic is dynamically loaded and executed to process the source event data and obtain target event data, which is then synchronized to the target end. Thus, by dynamically loading and executing different data synchronization processing logics during the data synchronization process, different source data can be processed using different methods before synchronization, thereby improving the flexibility of the data synchronization process.

[0109] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0110] Corresponding to the data synchronization method provided above, this application also provides a data synchronization device. Please refer to... Figure 2 The data synchronization device 2 in this embodiment includes: The first construction module 201 is used to construct the source event data corresponding to the source data to be synchronized; The determination module 202 is used to determine the corresponding custom data synchronization processing identifier based on the source event data; The first processing module 203 is used to dynamically load and execute the corresponding custom data synchronization processing logic based on the custom data synchronization processing identifier, so as to perform custom data synchronization processing on the source event data to obtain the target event data. Synchronization module 204 is used to synchronize target event data to the target end.

[0111] In some embodiments, the first building module 201 includes: The first acquisition unit is used to acquire snapshot data of the source database after snapshot backup; The parsing unit is used to parse the snapshot data to obtain the source event data.

[0112] In some embodiments, the first building module 201 includes: The second acquisition unit is used to acquire incremental data from the source database using a preset change data capture tool; The filtering unit is used to filter and extract incremental data to obtain source event data.

[0113] In some embodiments, the determining module 202 includes: The third acquisition unit is used to obtain synchronization task identification information and source table information from the source event data; The query unit is used to query the pre-configured table mapping relationship based on the synchronization task identifier information and the source table information; The determination unit is used to determine the corresponding custom data synchronization processing identifier from the table mapping relationship.

[0114] In some embodiments, the custom data synchronization processing logic is executed by an instance of a custom data synchronization processing class that implements the preset EventProcessor interface, and the input of the EventProcessor interface is source event data, and the output is target event data.

[0115] In some embodiments, the first processing module 203 includes: The receiving unit is used to receive source event data through the EventProcessor interface; The processing unit is used to perform at least one of the following processing methods based on the source event data: data splitting, data merging, data processing, or data filtering. The output unit is used to output the target event data obtained after processing.

[0116] In some embodiments, the data synchronization device 2 further includes: The second construction module is used to construct the source verification event data corresponding to the source data to be verified in the source database; The second processing module is used to execute the corresponding custom data synchronization processing logic on the source verification event data to obtain the expected event data. The acquisition module is used to acquire the corresponding target event data from the target end based on the expected event data; The comparison module is used to compare the expected event data with the target event data to complete the data verification.

[0117] As can be seen from the above, this embodiment constructs source event data corresponding to the source data and determines a corresponding custom data synchronization processing identifier based on the source event data, thereby enabling the selection of appropriate data synchronization processing methods for different source data. Furthermore, based on the custom data synchronization processing identifier, the corresponding custom data synchronization processing logic is dynamically loaded and executed to process the source event data and obtain target event data, which is then synchronized to the target end. Thus, by dynamically loading and executing different data synchronization processing logics during the data synchronization process, different source data can be processed using different methods before synchronization, thereby improving the flexibility of the data synchronization process.

[0118] Corresponding to the data synchronization method provided above, this application also provides an electronic device. Please refer to... Figure 3 The electronic device 3 in this application embodiment includes: a memory 301, and one or more processors 302. Figure 3 (Only one is shown in the image) and a computer program stored in memory 301 and executable on the processor. Specifically, the processor 302 performs the following steps by running the aforementioned computer program stored in memory 301: Construct the source event data corresponding to the source data to be synchronized; Determine the corresponding custom data synchronization processing identifier based on the source event data; Based on the custom data synchronization processing identifier, the corresponding custom data synchronization processing logic is dynamically loaded and executed to perform custom data synchronization processing on the source event data to obtain the target event data; Synchronize the target event data to the target device.

[0119] Assuming the above is the first possible implementation, then in the second possible implementation, which is based on the first possible implementation, the source event data corresponding to the source data to be synchronized is constructed, including: Retrieve snapshot data of the source database after snapshot backup; The snapshot data is parsed to obtain the source event data.

[0120] In the third possible implementation provided based on the first possible implementation described above, the source event data corresponding to the source data to be synchronized is constructed, including: Incremental data is obtained from the source database using a pre-defined change data capture tool; The incremental data is filtered and extracted to obtain the source event data.

[0121] In the fourth possible implementation provided based on the first possible implementation described above, the corresponding custom data synchronization processing identifier is determined according to the source event data, including: Obtain the synchronization task identifier information and source table information from the source event data; Based on the synchronization task identifier information and source table information, query the pre-configured table mapping relationship; Determine the corresponding custom data synchronization processing identifier from the table mapping relationship.

[0122] In the fifth possible implementation based on the first possible implementation described above, the custom data synchronization processing logic is executed by an instance of a custom data synchronization processing class that implements the preset EventProcessor interface, and the input of the EventProcessor interface is source event data, and the output is target event data.

[0123] In the sixth possible implementation provided based on the fifth possible implementation described above, the process of a custom data synchronization processing class instance performing custom data synchronization processing includes: Receive source event data through the EventProcessor interface; Perform at least one of the following processes based on the source event data: data splitting, data merging, data processing, or data filtering; Output the target event data obtained after processing.

[0124] In a seventh possible implementation provided based on the first, second, third, fourth, fifth, or sixth possible implementations described above, the processor 302 further performs the following steps when running the computer program stored in the memory 301: Construct source verification event data corresponding to the source data to be verified in the source database; Execute the corresponding custom data synchronization processing logic on the source verification event data to obtain the expected event data; Based on the expected event data, obtain the corresponding target event data from the target end; Compare the expected event data with the target event data to ensure consistency, thus completing the data verification.

[0125] It should be understood that, in the embodiments of this application, the processor 302 may be a central processing unit (CPU), but it may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0126] Memory 301 may include read-only memory and random access memory, and provides instructions and data to processor 302. Some or all of memory 301 may also include non-volatile random access memory. For example, memory 301 may also store device type information.

[0127] As can be seen from the above, this embodiment constructs source event data corresponding to the source data and determines a corresponding custom data synchronization processing identifier based on the source event data, thereby enabling the selection of appropriate data synchronization processing methods for different source data. Furthermore, based on the custom data synchronization processing identifier, the corresponding custom data synchronization processing logic is dynamically loaded and executed to process the source event data and obtain target event data, which is then synchronized to the target end. Thus, by dynamically loading and executing different data synchronization processing logics during the data synchronization process, different source data can be processed using different methods before synchronization, thereby improving the flexibility of the data synchronization process.

[0128] This application also provides a computer program product that, when run on an electronic device, enables the electronic device to perform the steps described in the various method embodiments above.

[0129] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the above device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0130] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0131] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of external device software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0132] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules or units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units through some interfaces, and may be electrical, mechanical, or other forms.

[0133] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0134] If the integrated units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing associated hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable storage medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer-readable storage device, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the contents of the aforementioned computer-readable storage media may be appropriately added to or subtracted from the contents according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media may not include electrical carrier signals and telecommunication signals.

[0135] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method of data synchronization, the method comprising: include: Construct the source event data corresponding to the source data to be synchronized; Determine the corresponding custom data synchronization processing identifier based on the source event data; Based on the custom data synchronization processing identifier, the corresponding custom data synchronization processing logic is dynamically loaded and executed to perform custom data synchronization processing on the source event data to obtain the target event data; Synchronize the target event data to the target device.

2. The data synchronization method of claim 1, wherein, The source event data corresponding to the source data to be synchronized includes: Retrieve snapshot data of the source database after snapshot backup; The snapshot data is parsed to obtain the source event data.

3. The data synchronization method of claim 1, wherein, The source event data corresponding to the source data to be synchronized includes: Incremental data is obtained from the source database using a pre-defined change data capture tool; The incremental data is filtered and extracted to obtain the source event data.

4. The data synchronization method of claim 1, wherein, The step of determining the corresponding custom data synchronization processing identifier based on the source event data includes: Obtain the synchronization task identifier information and source table information from the source event data; Based on the synchronization task identifier information and the source table information, query the pre-configured table mapping relationship; The corresponding custom data synchronization processing identifier is determined from the table mapping relationship.

5. The data synchronization method of claim 1, wherein, The custom data synchronization processing logic is executed by an instance of a custom data synchronization processing class that implements the preset EventProcessor interface, and the input of the EventProcessor interface is the source event data, and the output is the target event data.

6. The data synchronization method of claim 5, wherein, The process of executing custom data synchronization processing by the instance of the custom data synchronization processing class includes: The source event data is received through the EventProcessor interface; Perform at least one of the following processes based on the source event data: data splitting, data merging, data processing, or data filtering; Output the target event data obtained after processing.

7. The data synchronization method of any one of claims 1 to 6, wherein, The data synchronization method further includes: Construct source verification event data corresponding to the source data to be verified in the source database; The corresponding custom data synchronization processing logic is executed on the source verification event data to obtain the desired event data; Based on the expected event data, obtain the corresponding target event data from the target end; The consistency between the expected event data and the target event data is compared to complete the data verification.

8. A data synchronization apparatus, characterized by comprising: include: The first construction module is used to construct the source event data corresponding to the source data to be synchronized; The determination module is used to determine the corresponding custom data synchronization processing identifier based on the source event data; The first processing module is used to dynamically load and execute the corresponding custom data synchronization processing logic based on the custom data synchronization processing identifier, so as to perform custom data synchronization processing on the source event data to obtain the target event data. The synchronization module is used to synchronize the target event data to the target terminal.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 7.

10. A computer program product, characterised in that, The computer program product includes a computer program that, when executed by one or more processors, implements the method as described in any one of claims 1 to 7.

Images