Data synchronization method, electronic device, storage medium, and program product

Abstract

This disclosure provides a data synchronization method, electronic device, storage medium, and program product. In response to a data change operation on a source database, a change record corresponding to the data change operation is added to a data change record table. The change record includes source information of the source data targeted by the data change operation. Based on the change records included in the data change record table, incremental data markers are added to the source data indicated by each change record in the source data table included in the source database. These incremental data markers indicate that the corresponding source data belongs to incremental data. Based on the incremental data markers included in the source data table, incremental data synchronization is performed on the target data table in the target database corresponding to the source data table. Therefore, incremental data can be efficiently and accurately filtered from the source data table for incremental data synchronization, helping to improve the synchronization efficiency and accuracy of incremental data synchronization.

Description

Technical Field

[0001] This disclosure relates to data management technology, and in particular to a data synchronization method, electronic device, storage medium, and program product. Background Technology

[0002] In today's information age, enterprises and organizations typically rely on multiple database systems to store and manage massive amounts of data. As businesses continue to grow, the need for data synchronization between different databases is increasing, especially for synchronizing incremental data.

[0003] In related technologies, a timestamp-based synchronization method can be used to synchronize incremental data. By comparing the timestamps of corresponding records in the source database and the target database, the incremental data can be determined and synchronized. However, this method has limitations. If the timestamp is incorrect, it will lead to misjudgment or omission of incremental data. Summary of the Invention

[0004] This disclosure provides a data synchronization method, electronic device, storage medium, and program product to address the aforementioned technical problems to a certain extent.

[0005] One aspect of this disclosure provides a data synchronization method, the method comprising: In response to a data change operation on the source database, a change record corresponding to the data change operation is added to the data change record table. The change record includes the source information of the source data targeted by the data change operation. Based on the change records included in the data change record table, an incremental data marker is added to the source data indicated by each change record in the source data table included in the source database. The incremental data marker is used to indicate that the corresponding source data belongs to incremental data. Based on the incremental data tags included in the source data table, incremental data synchronization is performed on the target data table in the target database corresponding to the source data table.

[0006] In an exemplary embodiment, the step of adding a change record corresponding to the data change operation to the data change record table in response to a data change operation on the source database includes: In response to a data change operation on the source database, the change type corresponding to the data change operation, the source table identifier of the source data table where the target source data is located, and the source record identifier of the source data in the source data table are determined. In the data change record table, add the change record that includes the change type, the source table identifier, and the source record identifier.

[0007] In an exemplary embodiment, the step of adding incremental data markers to the source data indicated by each change record in the source data table included in the source database, based on the change records included in the data change record table, includes: For the target change records included in the data change record table, obtain the target change type, target source table identifier, and target source record identifier included in the target change record; Based on the target source table identifier and the target source record identifier, a target incremental data marker is added to the target source data in the target source data table, and the target incremental data marker includes the target change type.

[0008] In an exemplary embodiment, the step of synchronizing incremental data of a target table in the target database corresponding to the source data table based on the incremental data tags included in the source data table includes: Based on the source data in the source data table that has the incremental data tag and the change type included in the incremental data tag, incremental data synchronization is performed on the target data table.

[0009] In an exemplary embodiment, performing incremental data synchronization on the target data table based on source data in the source data table having the incremental data tag and the change type included in the incremental data tag includes: In response to the incremental data tag including a change type of insertion, the source data corresponding to the incremental data tag is inserted into the target data table; In response to the incremental data tag including a change type of update, the target data record in the target data table is updated based on the source data; In response to the incremental data tag including a change type of deletion, the target data record corresponding to the source data in the target data table is deleted.

[0010] In one exemplary embodiment, the change record includes the change time; The step of adding incremental data markers to the source data indicated by each change record in the source data table included in the source database, based on the change records included in the data change record table, includes: The system acquires periodic change records corresponding to a synchronization period at a preset time interval. The periodic change records include change records added within the synchronization time interval corresponding to the synchronization period. Based on the periodic change records, incremental data markers are added to the source data indicated by each periodic change record in the source data table included in the source database.

[0011] In an exemplary embodiment, the step of synchronizing incremental data of a target table in the target database corresponding to the source data table based on the incremental data tags included in the source data table includes: Based on the incremental data markers added to the source data table during the synchronization period, data synchronization is performed on the target data table in the target database corresponding to the source data table for the corresponding period.

[0012] Another aspect of this disclosure provides a data synchronization apparatus, the apparatus comprising: The change record module is used to add a change record corresponding to the data change operation to the data change record table in response to a data change operation to the source database. The change record includes the source information of the source data targeted by the data change operation. The data tagging module is used to add incremental data tags to the source data indicated by each change record in the source data table included in the source database, based on the change records included in the data change record table. The incremental data tags are used to indicate that the corresponding source data belongs to incremental data. The data synchronization module is used to perform incremental data synchronization on the target data table in the target database corresponding to the source data table, based on the incremental data tags included in the source data table.

[0013] In another aspect of this embodiment, an electronic device is provided, comprising: Memory, used to store computer programs; A processor is configured to execute a computer program stored in the memory, wherein when the computer program is executed, it implements the data synchronization method described in any of the above embodiments.

[0014] In another aspect of this embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the data synchronization method described in any of the above embodiments.

[0015] In another aspect of this embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the data synchronization method described in any of the above embodiments.

[0016] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0017] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0018] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein: Figure 1 A flowchart illustrating a data synchronization method provided as an exemplary embodiment of this disclosure; Figure 2 A flowchart for adding a change record to a data change record table is provided as an exemplary embodiment of this disclosure; Figure 3 A flowchart for adding incremental data markers to a source data table, provided as an exemplary embodiment of this disclosure; Figure 4 A flowchart for adding incremental data markers to a source data table, provided as another exemplary embodiment of this disclosure; Figure 5 A flowchart of a data change recording process provided as an exemplary embodiment of this disclosure; Figure 6 A flowchart of an incremental data tagging process provided as an exemplary embodiment of this disclosure; Figure 7 A flowchart of an incremental data synchronization process provided as an exemplary embodiment of this disclosure; Figure 8 A schematic diagram of the structure of a data synchronization device provided in an exemplary embodiment of this disclosure; Figure 9 A schematic diagram of the structure of an electronic device provided for an exemplary embodiment of this disclosure. Detailed Implementation

[0019] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0020] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0021] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0022] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0023] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0024] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0025] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0026] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0027] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0028] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0029] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0030] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0031] In related technologies, incremental data synchronization can be achieved using timestamp-based synchronization methods. However, this method is limited by timestamp records. When data update operations are not standardized and the timestamp field is not updated correctly, it may lead to misjudgment or omission of incremental data. Alternatively, a log mining-based synchronization method can be used. This method can parse the database's transaction logs and extract information related to data changes to determine incremental data. However, this approach is highly dependent on the database's log format. Different databases have significantly different log formats, requiring dedicated log parsing development for each database. This may result in parsing errors that affect the accuracy of incremental data determination. Furthermore, the log mining process is relatively complex, inefficient, and consumes significant system resources.

[0032] Alternatively, a full comparison can be used to determine the incremental data and synchronize it. This involves periodically comparing the source and target databases to identify the incremental data. However, this method requires a full comparison of a large amount of data, making it extremely inefficient. Especially with massive datasets, it consumes significant system time and resources, severely impacting the normal operation of the system.

[0033] This disclosure provides a data synchronization method that records each data change operation based on a data change record table and marks incremental data in the source data table based on the change records included in the data change record table. This allows for accurate and comprehensive identification of incremental data in the source data table. Data synchronization is then performed based on the incremental data marks in the source data table, thus enabling accurate and efficient synchronization of incremental data from the source data table to the target data table. This disclosure can be applied to any data synchronization scenario between databases, specifically for incremental data synchronization between databases.

[0034] In one possible implementation, such as Figure 1 The diagram illustrates a flowchart of a data synchronization method provided by an exemplary embodiment of this disclosure. The method provided in this disclosure can be used in the electronic devices described in the above embodiments, such as... Figure 1 As shown, it includes the following steps 110-130: Step 110: In response to the data change operation on the source database, add the corresponding change record to the data change record table.

[0035] A data modification operation refers to an operation that modifies data in the source tables of the source database. Optionally, a data modification operation can be a data insertion operation, a data update operation, or a data deletion operation.

[0036] The data insertion operation is used to add new data records to the source data table. The data update operation is used to update the data in existing records in the source data table. The data deletion operation is used to delete data records from the source data table. Source data refers to the data recorded in the source data table.

[0037] In this embodiment, a data change record table is set up to record various data change operations targeting source data tables in the source database. Each change record can record the source information of the source data targeted by the data change operation. Upon receiving any data change operation targeting a source data table in the source database, the data change operation can be executed on the source database, and a change record corresponding to the data change operation can be added to the data change record table. The change record includes the source information of the source data targeted by the data change operation. Optionally, the source information may include the source data table to which the source data belongs and the specific data record to which it belongs in the source data table. In one possible implementation, upon receiving a data change operation on the source database and successfully executing the data change operation, a change record corresponding to the data change operation is added to the data change record table.

[0038] In one example, if the data change operation is a data insertion operation, the corresponding change record may include the source information of the newly added data. If the data change operation is a data update operation, the corresponding change record may include the source information of the updated data. If the data change operation is a data deletion operation, the corresponding change record may include the source information of the deleted data.

[0039] Optionally, a data change log table can be set up in the source database to record data change operations performed on the source database.

[0040] Step 120: Based on the change records included in the data change record table, add incremental data tags to the source data indicated by each change record in the source data table included in the source database.

[0041] In one possible implementation, for any change record included in the data change record table, the source data in the source data table targeted by the data change operation can be located based on the source information included in the change record, and an incremental data marker can be added to the source data targeted by the data change operation. The incremental data marker is used to indicate that the corresponding source data belongs to incremental data.

[0042] Incremental data refers to data that has been added to or changed in the source data table since the last synchronization or at a specific time. Incremental data can include at least one of the following: newly added data, updated data, and deleted data. Optionally, an incremental data marker can be added to the data record corresponding to the source data. Then, the incremental data corresponding to the source data table can be quickly filtered based on the incremental data marker field.

[0043] In one possible implementation, change records in the data change log table can be retrieved according to the incremental data synchronization requirements, and incremental data tags can be added based on the retrieved change records. For example, if data after a specific time needs to be synchronized, change records after that specific time can be retrieved from the data change log table, and incremental data tags can be added to the source data in the source data table based on these change records. As another example, change records newly added to the data change log table since the last synchronization can also be retrieved, and incremental data tags can be added to the source data in the source data table based on these newly added change records.

[0044] In another possible implementation, a preset time interval can be used to periodically acquire new change records in the data change record table. Based on the change records acquired in each period, incremental data can be periodically marked in the source data table, thereby periodically synchronizing the incremental data.

[0045] Step 130: Based on the incremental data tags included in the source data table, perform incremental data synchronization on the target data table in the target database corresponding to the source data table.

[0046] During the process of synchronizing incremental data from the source data table to the target data table corresponding to the source data table in the target database, the incremental data tags included in the source data table can be obtained. The data in the source data table with incremental data tags are the incremental data. The incremental data in the source data table can be filtered based on the incremental data tags, and then the incremental data can be synchronized to the target data table.

[0047] In this embodiment, a data change record table is set up. Upon receiving a data change operation on the source database, a change record corresponding to the data change operation can be added to the data change record table. This ensures accurate recording of each data change operation and guarantees the comprehensiveness of the data change records. Furthermore, the change record includes the data source information of the source data targeted by the data change operation. Based on this information, the source data that has been changed in the source data table can be accurately located. Then, based on the change records included in the data change record table, incremental data marking can be applied to the changed source data in the source database to indicate that the corresponding source data is incremental data. This allows for accurate marking of incremental data. Subsequently, based on the incremental data markings included in the source data table, data synchronization can be performed on the target data table in the target database corresponding to the source data table. This achieves incremental data synchronization from the source data table to the target data table. The method of data synchronization based on incremental data marking can efficiently and accurately filter incremental data from the source data table, contributing to improved synchronization efficiency and accuracy of incremental data synchronization.

[0048] Furthermore, the synchronization method provided in this disclosure does not depend on the database log format, eliminating the need for development for different log parsing methods. This overcomes the development complexity and limited universality issues caused by log format differences, reducing development and maintenance costs. This disclosure can achieve incremental data marking through common database table operations and field addition operations, demonstrating strong universality and wide applicability to various database systems.

[0049] In one possible implementation, such as Figure 2 As shown, in Figure 1 Based on the embodiments, the execution process of step 110 above may include the following steps 1101-1102: Step 1101: In response to a data change operation on the source database, determine the change type corresponding to the data change operation, the source table identifier of the source data table where the source data is located, and the source record identifier of the source data in the source data table.

[0050] Upon receiving a data change operation on the source database, the change type can be determined. Change types include insert, update, and delete. If the data change operation is an insert operation, the change type is insert. If the data change operation is an update operation, the change type is update. If the data change operation is a delete operation, the change type is delete.

[0051] Furthermore, the source table identifier of the source data table containing the source data targeted by the data change operation can be determined. The source table identifier uniquely identifies the source data table in the source database. For example, the source table identifier can be the name of the source data table to clearly identify the source data table to which the source data belongs. Additionally, the source data identifier of the corresponding data record in the source data table can be determined. The source record identifier uniquely identifies that record in the source data table. Optionally, the source data identifier can be the primary key value of that source data record. Based on the primary key value, it can be accurately linked to the specific data record in the source data table. The primary key value can include specific fields or combinations of specific fields. The source table identifier and the source data identifier together constitute the source information of the source data.

[0052] In one example, for an update operation on the source data included in the first data record in the source data table A, the change type can be determined to be an update type, and the source table identifier of the source data table A and the source record identifier of the first data record can be obtained.

[0053] Step 1102: Add a change record to the data change record table, including the change type, source table identifier, and source record identifier.

[0054] After determining the change type, the source table identifier, and the source record identifier of the data change operation, a change record can be added that includes the change type, source table identifier, and source record identifier. Different change types can be represented by different numbers, fields, or symbols. For example, an insert type can be represented by the letter "I", an update type by the letter "U", and a delete type by the letter "D".

[0055] In this embodiment, upon receiving a data change operation, the source table identifier of the source data table to which the target source data belongs and the source record identifier of the source data in the source data table can be determined to quickly locate the source data in the source data table. Furthermore, the change type corresponding to the data change operation can be recorded, which can quickly determine the synchronization operation type corresponding to the incremental data during the subsequent data synchronization process, thus helping to improve synchronization efficiency.

[0056] In one possible implementation, Figure 1 , Figure 2 Based on the embodiments, such as Figure 3 As shown, the execution process of step 120 above may include the following steps 1201-1202: Step 1201: For the target change records included in the data change record table, obtain the target change type, target source table identifier, and target source record identifier included in the target change record.

[0057] The target change record can be any change record in the data change record table. For each target change record, the target change type, target source table identifier, and target source record identifier can be retrieved. The target change type refers to the operation type of the target data change operation corresponding to the target change record, while the target source table identifier and target source record identifier represent the source information of the target source data targeted by the target data change operation.

[0058] Step 1202: Based on the target source table identifier and the target source record identifier, add a target incremental data marker to the target source data in the target source data table.

[0059] When adding target incremental data markers to target source data based on target change records, the target source data table to which the target source data belongs can be determined based on the target source table identifier. Then, the data record corresponding to the target source data in the target source data table can be located based on the target source record identifier. After that, target incremental data markers can be added to the data record corresponding to the target source data (such as the record row where the target source data is located). The target incremental data markers include the target change type.

[0060] In one possible implementation, the target incremental data tag can be marked in the form of a field. A target incremental data field can be added to the data record corresponding to the target source data. The target incremental data field can be a field used to indicate the type of target change.

[0061] Optionally, the content of the target incremental data field can be the same as the field in the target change record that indicates the target change type. That is, for each change type, the same field content can be used in both the incremental data marker and the change record. For example, if the field in the target change record that indicates the target change type is "I", then the target incremental data field "I" can be added to the data record corresponding to the target source data. Alternatively, the content of the target incremental data field can also be different from the field in the target change record that indicates the target change type. That is, for each change type, different field content can be used in the incremental data marker and the change record. For example, if the field in the target change record that indicates the target change type is "I", representing an insert type, and the incremental data marker can use the "Add" field to represent the insert type, then the target incremental data field "Add" can be added to the data record corresponding to the target source data.

[0062] In this embodiment, for any change record included in the data change record table, the corresponding source data record can be located based on the source table identifier and source record identifier included in the change record. Then, an incremental data marker for indicating the change type is added to the data record corresponding to the source data. Thus, the incremental data included in the source data table can be marked accurately and comprehensively, so as to accurately determine each incremental data during the incremental data synchronization process. This can effectively reduce the probability of missed or misjudged incremental data and improve the accuracy of incremental data synchronization.

[0063] In one possible implementation, during the process of synchronizing incremental data in the target data table in the target database corresponding to the source data table based on the incremental data tags included in the source data table, the incremental data synchronization in the target data table can be performed based on the source data in the source data table that has incremental data tags and the change types included in the incremental data tags.

[0064] In one possible implementation, data records with incremental data tags can be filtered from the source data table to obtain the source data marked by the incremental data tags, where the source data marked by the incremental data tags is the incremental data. After obtaining the incremental data from the source data table based on the incremental data tags, for any incremental data, a corresponding type of change operation can be performed on the target data table based on the change type indicated by the incremental data tag corresponding to that incremental data to achieve data synchronization of the incremental data.

[0065] Optionally, if the incremental data tag includes a deletion type, the target data record corresponding to the source data in the target data table can be deleted. During this process, the target data record corresponding to the source data can be determined in the target data table based on the source record identifier (such as a primary key value), and then that target data record can be deleted. Specifically, identical data records in the source and target data tables are uniquely identified using the same identifier (such as the same primary key value).

[0066] Optionally, if the change type included in the incremental data marker is an insert or update type, the source data needs to be extracted for corresponding data synchronization. Specifically, in response to the change type included in the incremental data marker being an insert type, the source data corresponding to the incremental data marker is inserted into the target data table. In the case where the change type included in the incremental data marker is an insert type, the source data corresponding to the incremental data marker can be extracted and inserted into the target data table. Optionally, the insertion position in the target data table can be the same as the record position of the source data in the source data table.

[0067] In response to the incremental data tag indicating an update type of change, the target data record in the target data table is updated based on the source data. Specifically, when the incremental data tag indicates an update type of change, the source data corresponding to the incremental data tag and its source record identifier can be extracted. Based on the source record identifier, the target data record corresponding to the source data in the target data table is determined. Then, the target data record is updated based on the source data.

[0068] In this embodiment, incremental data can be synchronized based on source data with incremental data markers in the source data table, ensuring the comprehensiveness of incremental data synchronization. Furthermore, during this process, corresponding change operations can be performed on the target data table corresponding to the source data table based on the change types included in the incremental data markers, enabling efficient completion of incremental data synchronization.

[0069] In one possible implementation, the change record includes a change time, which indicates the time when the data change operation recorded in the change record occurred, and can subsequently be used to determine the time range to which the corresponding incremental data belongs. Optionally, the change time can be recorded using a preset time format, for example, the preset time format can be YYYY-MM-DD HH:MM:SS, where the YYYY field represents the year, the MM field represents the month, the DD field represents the specific date, and HH (hour):MM (minute):SS (second) represents the specific time.

[0070] In one possible implementation, such as Figure 4 As shown, based on the above embodiments, the process of adding incremental data markers to the source data indicated by each change record in the source data table included in the source database, based on the change records included in the data change record table, may include the following steps: Step 120a: Obtain the periodic change record corresponding to the synchronization period with a preset time interval as the period. The periodic change record includes the change record added within the synchronization time interval corresponding to the synchronization period.

[0071] The preset time interval can be set according to data synchronization requirements, system performance, etc. For example, the preset time interval can be 1 hour, 1 day, etc.

[0072] A synchronization period refers to a period corresponding to a preset time interval. For a given synchronization period, the added change records in the data change record table within the corresponding synchronization time interval can be retrieved to obtain the periodic change record for that synchronization period.

[0073] Step 120b: Based on the periodic change records, add incremental data markers to the source data indicated by each periodic change record in the source data table included in the source database.

[0074] After determining the periodic change record, an incremental data marker can be added to the source data in the source data table indicated by any of the change records included in the periodic change record, thereby completing the incremental data marking process corresponding to this synchronization cycle.

[0075] In one possible implementation, an incremental data marking process can be initiated each time a preset time interval is reached. During this process, based on the change times included in each change record in the data change record table, the change records added to the data change record table within the time range corresponding to the current synchronization cycle are determined, resulting in the periodic change records corresponding to the current synchronization cycle. Then, based on these periodic change records, the incremental data marking for the current synchronization cycle is completed. The time range of the synchronization time interval is determined according to the start and end times of the current synchronization cycle.

[0076] For example, if the cycle is 1 hour, the incremental data marking process can be started every 1 hour. The historical 1 hour is the synchronization time interval corresponding to the current synchronization cycle. The newly added change records in the data change record table within the historical 1 hour can be obtained to obtain the cycle change record. Then, incremental data marking is performed based on the cycle change record.

[0077] The method for incremental data marking based on each change record included in the periodic change record can be referred to the above. Figure 3 The implementation method of incremental data marking based on target change records in the corresponding embodiment will not be described in detail here.

[0078] In one possible implementation, after incremental data marking based on periodic change records, data synchronization can be performed on the target data table in the target database corresponding to the source data table based on the incremental data markings added to the source data table within the synchronization period. That is, after incremental data marking is completed in each synchronization period, corresponding data synchronization can be performed on the target data table in the target database based on the incremental data markings added in the current synchronization period, achieving periodic incremental data synchronization. The method of performing corresponding data synchronization on the target data table in the target database based on the incremental data markings added in the current synchronization period can refer to the specific implementation method described above, which involves performing incremental data synchronization on the target data table based on source data with incremental data markings in the source data table and the change types included in the incremental data markings; it will not be elaborated further here.

[0079] In this embodiment, change records can be acquired periodically, incremental data can be processed periodically, and data synchronization can be performed periodically. This reduces the frequency of data processing, helps improve data synchronization efficiency, and saves processing resources.

[0080] In one possible implementation, the data synchronization process may include a data change recording process, an incremental data marking process, and an incremental data synchronization process. For example, Figure 5 As shown, the data change record process may include the following steps: Step 210: Detect data change operations targeting the source database.

[0081] Step 220: Perform the corresponding data change operation on the source database.

[0082] Step 230: Insert the change record corresponding to the data change operation into the data change record table.

[0083] The inserted change record includes the change type, change time, source table identifier, and source record identifier corresponding to the source data.

[0084] The specific implementation methods for steps 210-230 can be referred to the above. Figure 2 Examples are not described in detail here.

[0085] In one possible implementation, such as Figure 6 As shown, the incremental data tagging process may include the following steps: Step 310: Determine the synchronization time interval.

[0086] Step 320: Query the change records within the current synchronization interval from the data change record table.

[0087] Step 330: Determine if the corresponding change record is found. If yes, proceed to step 340; otherwise, proceed to step 360.

[0088] Step 340: Determine the data record of the source data based on the source table identifier and source record identifier included in the change record.

[0089] Step 350: Add an incremental data tag, including the change type, to the data record of the source data.

[0090] Step 360: Wait for the next synchronization interval to be reached.

[0091] The specific implementation methods for steps 310-360 can be referred to the above. Figure 3 , Figure 4 The corresponding implementation examples will not be described in detail here.

[0092] In one possible implementation, such as Figure 7 As shown, the process of synchronizing incremental data may include the following steps: Step 410: Extract the corresponding incremental data based on the incremental data tags included in the source data table.

[0093] Step 420: Obtain the change types included in the incremental data tags.

[0094] Step 430: If the change type is insert type, insert the incremental data into the target data table.

[0095] Step 440: If the change type is update type, update the corresponding data record in the target data table based on the incremental data.

[0096] Step 450: If the change type is deletion, delete the data record corresponding to the incremental data in the target data table.

[0097] The data change recording method provided in this embodiment can record data change operation information in detail. Then, based on the detailed change record, the incremental data in the source data table can be marked accurately and comprehensively. Compared with the synchronization method based on timestamps or database logs in related technologies, this method can reduce the probability of missed or misjudged incremental data and ensure the accuracy of incremental data determination.

[0098] Furthermore, processing and synchronizing incremental data based on synchronization time intervals allows for centralized processing of incremental data. Concise data extraction and synchronization can be performed based on the incremental data tags included in the source data table, which helps improve synchronization efficiency.

[0099] In one possible implementation, such as Figure 8 The diagram illustrates a structural schematic of a data synchronization device provided in an exemplary embodiment of this disclosure. The data synchronization device can be deployed in the aforementioned electronic device to implement the data synchronization method described in any of the above embodiments. The device includes: The change record module 810 is used to add a change record corresponding to the data change operation to the data change record table in response to a data change operation on the source database. The change record includes the source information of the source data targeted by the data change operation. The data tagging module 820 is used to add incremental data tags to the source data indicated by each change record in the source data table included in the source database based on the change records included in the data change record table. The incremental data tags are used to indicate that the corresponding source data belongs to incremental data. The data synchronization module 830 is used to perform incremental data synchronization on the target data table in the target database corresponding to the source data table based on the incremental data tags included in the source data table.

[0100] In an exemplary embodiment, the change record module 810 is further configured to: In response to a data change operation on the source database, the change type corresponding to the data change operation, the source table identifier of the source data table where the target source data is located, and the source record identifier of the source data in the source data table are determined. In the data change record table, add the change record that includes the change type, the source table identifier, and the source record identifier.

[0101] In an exemplary embodiment, the data tagging module 820 described above is further configured to: For the target change records included in the data change record table, obtain the target change type, target source table identifier, and target source record identifier included in the target change record; Based on the target source table identifier and the target source record identifier, a target incremental data marker is added to the target source data in the target source data table, and the target incremental data marker includes the target change type.

[0102] In an exemplary embodiment, the data synchronization module 830 described above is further configured to: Based on the source data in the source data table that has the incremental data tag and the change type included in the incremental data tag, incremental data synchronization is performed on the target data table.

[0103] In an exemplary embodiment, the data synchronization module 830 described above is further configured to: In response to the incremental data tag including a change type of insertion, the source data corresponding to the incremental data tag is inserted into the target data table; In response to the incremental data tag including a change type of update, the target data record in the target data table is updated based on the source data; In response to the incremental data tag including a change type of deletion, the target data record corresponding to the source data in the target data table is deleted.

[0104] In one exemplary embodiment, the change record includes the change time; The aforementioned data tagging module 820 is also used for: The system acquires periodic change records corresponding to a synchronization period at a preset time interval. The periodic change records include change records added within the synchronization time interval corresponding to the synchronization period. Based on the periodic change records, incremental data markers are added to the source data indicated by each periodic change record in the source data table included in the source database.

[0105] In an exemplary embodiment, the data synchronization module 830 described above is further configured to: Based on the incremental data markers added to the source data table during the synchronization period, data synchronization is performed on the target data table in the target database corresponding to the source data table for the corresponding period.

[0106] The data synchronization device in this disclosure corresponds to the data synchronization method described above, and the relevant content can be referred to each other, so it will not be repeated here. The beneficial technical effects of the data synchronization device in this disclosure can be found in the corresponding beneficial technical effects in the exemplary method section above, so it will not be repeated here.

[0107] In addition, this disclosure also provides an electronic device, including: Memory, used to store computer programs; A processor is configured to execute a computer program stored in the memory, wherein when the computer program is executed, it implements the data synchronization method described in any of the above embodiments of the present disclosure.

[0108] Figure 9 This is a schematic diagram illustrating the structure of an application embodiment of the electronic device disclosed herein. Below, reference is made to… Figure 9 To describe an electronic device according to embodiments of this disclosure. For example... Figure 9 As shown, the electronic device includes one or more processors and memory.

[0109] A processor can be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and can control other components in an electronic device to perform desired functions.

[0110] The memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and a processor may execute the program instructions to implement the data synchronization methods of the various embodiments of this disclosure described above and / or other desired functions.

[0111] In one example, the electronic device may also include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0112] In addition, the input device may include, for example, a keyboard, a mouse, etc.

[0113] This output device can output various information to the outside, including determined distance information, direction information, etc. The output device may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0114] Of course, for the sake of simplicity, Figure 9 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0115] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the data synchronization methods according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0116] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0117] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the data synchronization methods according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0118] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0119] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as ROM, RAM, magnetic disk, or optical disk.

[0120] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0121] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0122] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0123] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0124] It should also be noted that in the apparatus, devices, and methods disclosed herein, the components or steps can be disassembled and / or recombined. Such disassembly and / or recombination should be considered as equivalent solutions to this disclosure.

[0125] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0126] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A data synchronization method, characterized in that, The method includes: In response to a data change operation on the source database, a change record corresponding to the data change operation is added to the data change record table. The change record includes the source information of the source data targeted by the data change operation. Based on the change records included in the data change record table, an incremental data marker is added to the source data indicated by each change record in the source data table included in the source database. The incremental data marker is used to indicate that the corresponding source data belongs to incremental data. Based on the incremental data tags included in the source data table, incremental data synchronization is performed on the target data table in the target database corresponding to the source data table.

2. The method according to claim 1, characterized in that, The step of adding a change record corresponding to the data change operation to the data change record table in response to a data change operation in the source database includes: In response to a data change operation on the source database, the change type corresponding to the data change operation, the source table identifier of the source data table where the target source data is located, and the source record identifier of the source data in the source data table are determined. In the data change record table, add the change record that includes the change type, the source table identifier, and the source record identifier.

3. The method according to claim 2, characterized in that, The step of adding incremental data markers to the source data indicated by each change record in the source data table included in the source database, based on the change records included in the data change record table, includes: For the target change records included in the data change record table, obtain the target change type, target source table identifier, and target source record identifier included in the target change record; Based on the target source table identifier and the target source record identifier, a target incremental data marker is added to the target source data in the target source data table, and the target incremental data marker includes the target change type.

4. The method according to claim 3, characterized in that, The step of synchronizing incremental data in the target data table in the target database corresponding to the source data table, based on the incremental data tags included in the source data table, includes: Based on the source data in the source data table that has the incremental data tag and the change type included in the incremental data tag, incremental data synchronization is performed on the target data table.

5. The method according to claim 4, characterized in that, The step of performing incremental data synchronization on the target data table based on the source data in the source data table that has the incremental data tag and the change type included in the incremental data tag includes: In response to the incremental data tag including a change type of insertion, the source data corresponding to the incremental data tag is inserted into the target data table; In response to the incremental data tag including a change type of update, the target data record in the target data table is updated based on the source data; In response to the incremental data tag including a change type of deletion, the target data record corresponding to the source data in the target data table is deleted.

6. The method according to any one of claims 1 to 5, characterized in that, The change record includes the change time; The step of adding incremental data markers to the source data indicated by each change record in the source data table included in the source database, based on the change records included in the data change record table, includes: The system acquires periodic change records corresponding to a synchronization period at a preset time interval. The periodic change records include change records added within the synchronization time interval corresponding to the synchronization period. Based on the periodic change records, incremental data markers are added to the source data indicated by each periodic change record in the source data table included in the source database.

7. The method according to claim 6, characterized in that, The step of synchronizing incremental data in the target data table in the target database corresponding to the source data table, based on the incremental data tags included in the source data table, includes: Based on the incremental data markers added to the source data table during the synchronization period, data synchronization is performed on the target data table in the target database corresponding to the source data table for the corresponding period.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor is configured to execute a computer program stored in the memory, wherein when the computer program is executed, it implements the data synchronization method according to any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the data synchronization method according to any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the data synchronization method according to any one of claims 1-7.

Images