Data synchronization method, device, electronic device and storage medium

Abstract

Embodiments of the present application disclose a data synchronization method, device, electronic device and storage medium, the method comprising: obtaining first original data to be synchronized and a data type of the first original data, the data type comprising full data or incremental data; when the data type of the first original data is full data, creating a new storage partition, and storing the first original data in the new storage partition; different storage partitions store different full data; performing data incremental operations according to a data synchronization time point of the first original data, so as to store the incremental data corresponding to the data incremental operations in the new storage partition; synchronizing the data stored in the new storage partition to a data user, so as to avoid data loss during the full data synchronization process, and retaining historical data by creating a new storage partition.

Description

Technical Field

[0001] This application relates to a data synchronization technology, providing a data synchronization method, a data synchronization device, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Data synchronization is a common requirement in IT (Internet technology) system design. In related technologies, data synchronization solutions generally use a combination of full synchronization and incremental synchronization. If multiple full synchronizations are required during the synchronization process, the previous data will be overwritten during each full synchronization, and then incremental synchronization will be used for continuous updates. This results in the inability to trace back the overwritten historical data, and the synchronization time of full and incremental synchronization is not easy to coordinate. Data loss may occur during the full synchronization period that affects incremental synchronization. Summary of the Invention

[0003] The purpose of this application is to provide a data synchronization method, data synchronization device, electronic device, and computer-readable storage medium to avoid data loss during full data synchronization, achieve stable and efficient data synchronization, ensure data accuracy, and retain historical data for easy review.

[0004] This application proposes a data synchronization method, comprising: acquiring first raw data to be synchronized and the data type of the first raw data, wherein the data type includes full data or incremental data; when the data type of the first raw data is full data, creating a new storage partition and storing the first raw data in the new storage partition; storing different full data in different storage partitions; performing incremental data operation according to the data synchronization time point of the first raw data, storing the incremental data corresponding to the incremental data operation in the new storage partition; and synchronizing the data stored in the new storage partition to the data user.

[0005] Further, after obtaining the first raw data to be synchronized and the data type of the first raw data, the method further includes: writing the first raw data into an intermediate table and adding a first record of unprocessed full data to a log table; the intermediate table is used to cache the data to be synchronized, and the log table is used to record the processing status of the data to be synchronized; when the data type of the first raw data is full data, creating a new storage partition and storing the first raw data in the new storage partition includes: when it is determined by polling the log table that there is a first record of unprocessed full data, creating the new storage partition in a result table; the result table is used to store all synchronized data; storing the first raw data corresponding to the first record in the intermediate table in the new storage partition.

[0006] Further, the step of performing incremental data operation based on the data synchronization time point of the first original data to store the incremental data corresponding to the incremental data operation in the new storage partition includes: updating the log table to update the records of incremental data from the data synchronization time point of the first original data to the current time point as unprocessed; obtaining all incremental data from the intermediate table from the data synchronization time point of the first original data to the current time point, and appending all incremental data to the new storage partition.

[0007] Furthermore, after appending all incremental data to the new storage partition, the method further includes: updating the version number in the data version table to the latest version number corresponding to the new storage partition; the data version table is used to manage the version number of all data; and updating the log table to update the first record to the processed record.

[0008] Furthermore, before updating the log table to update the first record to a processed record, the method further includes: receiving a first data read request sent by a data user; obtaining data of a target storage partition from the result table according to the first data read request, and sending the data stored in the target storage partition to the data user, wherein the target storage partition is the storage partition corresponding to the previous version number of the latest version number.

[0009] Furthermore, after updating the log table to update the first record to a processed record, the method further includes: obtaining second raw data to be synchronized, the second raw data being incremental data; writing the second raw data into an intermediate table and adding a second record of unprocessed incremental data to the log table; when it is determined by polling the log table that there is a second record of unprocessed incremental data, obtaining the current version number in the data version table; storing the second raw data corresponding to the second record in the intermediate table in the storage partition corresponding to the current version number, and updating the second record to a processed record.

[0010] Furthermore, after updating the log table to update the first record to a processed record, the method further includes: sending the data version table to the data user and receiving a second data read request from the data user carrying a target version number; querying the data of the storage partition corresponding to the target version number from the result table, and sending the queried data to the data user so that the data user can process the queried data.

[0011] This application also proposes a data synchronization device, comprising: a data acquisition module, configured to acquire first raw data to be synchronized and the data type of the first raw data, wherein the data type includes full data or incremental data; a data storage module, configured to create a new storage partition when the data type of the first raw data is full data, and store the first raw data in the new storage partition; different storage partitions store different full data; an incremental operation module, configured to perform incremental data operation according to the data synchronization time point of the first raw data, so as to store the incremental data corresponding to the incremental data operation in the new storage partition; and a data sending module, configured to synchronize the data stored in the new storage partition to the data user.

[0012] This application also proposes an electronic device comprising: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the electronic device to perform the method described above.

[0013] This application also proposes a computer-readable storage medium storing computer-readable instructions thereon, which, when executed by a computer's processor, cause the computer to perform the method described above.

[0014] Compared with the prior art, this application has the following beneficial effects:

[0015] In the technical solution provided in this application, during the data synchronization process, if the data to be synchronized is full data, unlike the synchronization methods in related technologies, historical data is not overwritten or cleared. Instead, a new storage partition is created, and incremental operations are appended to this new storage partition to store the incremental data corresponding to the incremental operations in the new storage partition. This ensures that the data stored in the new storage partition is the latest data, avoiding data loss during full data synchronization, achieving stable and efficient data synchronization, ensuring data accuracy, and retaining historical data. This allows for convenient backtracking of historical data. Furthermore, different full data are located in different storage partitions, and historical data and current data do not interfere with each other. Historical data does not affect the query efficiency of current data. In this way, while ensuring query efficiency, historical data can be backtracked. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of one implementation environment involved in this application;

[0017] Figure 2 This is a flowchart illustrating a data synchronization method in an exemplary embodiment of this application;

[0018] Figure 3This is a flowchart illustrating another data synchronization method in an exemplary embodiment of this application;

[0019] Figure 4 for Figure 3 A flowchart of step S350 in the data synchronization method shown in an example;

[0020] Figure 5 This is a schematic diagram illustrating another data synchronization method as shown in an exemplary embodiment of this application;

[0021] Figure 6 This is an entity relationship diagram illustrating a data synchronization method in an exemplary embodiment of this application;

[0022] Figure 7 This is a flowchart illustrating a data synchronization method in another exemplary embodiment of this application;

[0023] Figure 8 This is a flowchart illustrating a data synchronization method in yet another exemplary embodiment of this application;

[0024] Figure 9 This is a schematic diagram illustrating the structure of a data synchronization device according to an exemplary embodiment of this application;

[0025] Figure 10 A schematic diagram of the structure of an electronic device suitable for implementing embodiments of this application is shown. Detailed Implementation

[0026] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0027] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0028] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0029] It should also be noted that "multiple" as mentioned in this application refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0030] Please see Figure 1 , Figure 1 This is a schematic diagram of an implementation environment related to this application. The implementation environment includes a data provider 10, a data synchronization device 20, and a data user 30, which communicate with each other via wired or wireless networks.

[0031] Data provider 10 is used to provide the data to be synchronized;

[0032] The data synchronization device 20 is used to store the data to be synchronized. Specifically, if the data to be synchronized is full data, a new storage partition is created and the first original data is stored in the new storage partition; an incremental data operation is performed according to the data synchronization time point of the first original data, so that the incremental data corresponding to the incremental data operation is stored in the new storage partition, and the data stored in the new storage partition is synchronized to the data user.

[0033] Data user 30 is used to read the data stored in the new storage partition in order to achieve data synchronization with the data provider.

[0034] Among them, the data provider 10 and the data synchronization device 20 can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. This document does not impose any restrictions on these.

[0035] The data user 30 can be a server or a terminal, and the terminal can be implemented in various forms. For example, the terminal described in this invention may include mobile terminals such as mobile phones, tablets, laptops, PDAs, personal digital assistants (PDAs), portable media players (PMPs), wearable devices, smart bracelets, etc., as well as fixed terminals such as digital TVs and desktop computers with dial-up access functions.

[0036] Please see Figure 2 , Figure 2 This is a flowchart illustrating a data synchronization method in an exemplary embodiment of this application. The data synchronization method comprises... Figure 1 The data synchronization device 20 shown executes the data synchronization method, which includes steps S110 to S140, as detailed below:

[0037] S110. Obtain the first raw data to be synchronized and the data type of the first raw data. The data type includes full data or incremental data.

[0038] It is understandable that data synchronization refers to the synchronization and sharing of data between one party and another party, so that the data of both parties maintains integrity and consistency. In this embodiment, the first original data to be synchronized refers to the data provided by the data provider, which has not yet been synchronized to the data user. The data provided by the data provider can be full data or incremental data. Full data refers to all data within a certain large period. Incremental data is based on full data, and incremental data can refer to updated data. Incremental data can also be all data within a certain small period, such as full data for each month and incremental data for each hour.

[0039] In one example of this embodiment, the data provider sends the first raw data to be synchronized to the data synchronization device, and also marks whether the first raw data is full data or incremental data, so that the data synchronization device can obtain the raw data and data type.

[0040] In another example of this embodiment, since the synchronization time points of full data and incremental data are different, the data synchronization device may periodically retrieve the original data from the data provider, and then determine whether the original data is full data or incremental data based on the retrieval time.

[0041] S120. When the data type of the first original data is full data, create a new storage partition and store the first original data in the new storage partition. Different storage partitions store different full data.

[0042] In this embodiment, different storage partitions store different full data. That is, a new storage partition is created each time full data is obtained, and this storage partition is used to store the full data each time. Since the data type of the first original data is full data, the first original data is stored in a new storage partition. Therefore, the technical solution provided in this embodiment will not overwrite or delete historical data. By retaining the full data of each time period in each storage partition, the historical data can be traced back.

[0043] It should be noted that each storage partition is physically independent, but logically it is a single table. Historical data will not affect the query efficiency of the current full data, and the migration, backup, or deletion of historical data will not interrupt the synchronization of the current full data.

[0044] In this embodiment, when the data type of the first original data is incremental data, the first original data is stored in the current storage partition.

[0045] S130. Perform incremental data operation based on the data synchronization time point of the first original data, so as to store the incremental data corresponding to the incremental data operation in the new storage partition.

[0046] It is worth noting that when the execution frequency of full data synchronization operations is lower than that of incremental data synchronization operations, full data and incremental data are not easily coordinated. Full data synchronization may affect incremental data synchronization. For example, if the full monthly data is transmitted to the data synchronization device at 3:00 AM on January 1st, incremental data needs to be continuously synchronized hourly between 0:00 AM and 3:00 AM to ensure the data is up-to-date. If the full data is applied directly at 3:00 AM, since the full data only extends to 0:00 AM on January 1st, the synchronization of incremental data between 0:00 AM and 3:00 AM may be lost. Therefore, in this embodiment, when the first original data is full data, the data is based on the data from the first original data... Incremental data operations are performed at synchronized time points to store the incremental data corresponding to the incremental operations in a new storage partition. In other words, the data stored in the new storage partition is appended to the latest data. For example, after storing the full data of January 1st at 00:00 in a new storage partition, an incremental data operation is performed based on the time point of 00:00. Since this incremental data operation corresponds to a batch of incremental data, the incremental data is appended to the first batch of original data. If the latest time point is January 1st at 03:00, the data stored in the new storage partition is appended to the data of January 1st at 03:00, thus avoiding data loss during the full data synchronization process.

[0047] S140. Synchronize the data stored in the new storage partition with the data user.

[0048] After the incremental data corresponding to the incremental data operation is stored in the new storage partition, that is, after the data stored in the new storage partition is appended to the latest version, the data stored in the new storage partition is synchronized to the data user. In other words, the full data and the appended incremental data are synchronized to the data user to realize the data synchronization between the data provider and the data user.

[0049] It is worth noting that the data synchronization device in this embodiment can also establish an intermediate table, a log table, a data version table, and a result table to ensure that the original data to be synchronized can be synchronized in an orderly and smooth manner. The intermediate table is used to cache the data to be synchronized without performing any other processing. The log table records the processing status of the data to be synchronized; that is, each time a batch of data to be synchronized is written to the intermediate table, a record of that data to be synchronized is written to the log table and marked as unprocessed. When the data to be synchronized has been processed and synchronized, it is marked as processed. The data version table manages the version numbers of all data; it can contain only one currently used data version number or all data version numbers. The result table stores all synchronized data; different storage partitions store different full data sets, and each full data set corresponds to one data version, therefore different storage partitions store different data versions of the data.

[0050] like Figure 3 As shown, Figure 3 This application illustrates another data synchronization method according to an exemplary embodiment, the data synchronization method comprising:

[0051] S310, Obtain the first raw data to be synchronized and the data type of the first raw data.

[0052] Please refer to S110 above for details on this step; it will not be repeated here.

[0053] S320. Write the first raw data into the intermediate table, and add a first record of unprocessed full data to the log table.

[0054] In this embodiment, since the intermediate table is used to cache the data to be synchronized, after the first raw data to be synchronized is obtained, the first raw data is cached in the intermediate table, and a first record is written to the log table. The first record is marked as the full data has not been processed.

[0055] Understandably, each time data to be synchronized is obtained, it needs to be written to an intermediate table and an unprocessed record is added to the log table.

[0056] S330. When it is determined by polling the log table that there is a first record with all data that has not been processed, a new storage partition is created in the result table.

[0057] It is understandable that there may be multiple records in the log table. In order to synchronize in an orderly manner, the log table is polled in turn according to the record time order of the multiple records to determine whether there is any full data that needs to be processed. When the polled log table determines that there is a first record with full data that has not been processed, a new storage partition is created in the result table to store the full data for this time.

[0058] In this embodiment, the result table contains multiple storage partitions, each corresponding to a full set of data once, and each storage partition is independent of the others in the result table.

[0059] S340. Store the first original data corresponding to the first record in the intermediate table in a new storage partition.

[0060] In this embodiment, since the first original data corresponding to the first record is cached in the intermediate table, after a new storage partition is created, the first original data corresponding to the first record is extracted from the intermediate table, and then the first original data is stored in the new storage partition.

[0061] Optionally, after the first original data is extracted from the intermediate table, the first original data is deleted from the intermediate table to reduce the storage resources of the intermediate table.

[0062] S350. Perform incremental data operation based on the data synchronization time point of the first original data, so as to store the incremental data corresponding to the incremental data operation in the new storage partition.

[0063] It is understandable that, such as Figure 4 As shown, Figure 4 for Figure 3 In a flowchart of an example example, step S350 of the data synchronization method shown includes performing an incremental data operation:

[0064] S410. Update the log table to update the records of incremental data from the data synchronization time point of the first original data to the current time point to unprocessed data.

[0065] Since the log table contains records of the processing status of each piece of data to be synchronized, the records of incremental data from the data synchronization time of the first original data to the current time are retrieved from the log table and updated to "unprocessed". For example, if the data synchronization time of the first original data is 0:00 on January 1st, and the current latest time is 3:30 on January 1st, then the records of incremental data between 0:00 and 3:30 are retrieved. Assuming there is one record of incremental data every hour, the records of incremental data at 1:00, 2:00, and 3:00 are updated to "unprocessed".

[0066] It is understandable that regardless of whether the incremental data records from the data synchronization time point of the first original data to the current time point are unprocessed, they will all be updated to unprocessed. For example, assuming that the incremental data record at point 1 is processed, when updating the log table, the record at point 1 will be updated to unprocessed so that the corresponding data can be accurately synchronized based on the unprocessed status of the record in the future.

[0067] S420. Obtain all incremental data from the data synchronization time point of the first original data to the current time point from the intermediate table, and append all incremental data to the new storage partition.

[0068] Continuing from the previous example, the incremental data for points 1, 2, and 3 are extracted from the intermediate table, and all the extracted incremental data is appended to the new storage partition, so that the data stored in the new storage partition is the latest, that is, the data stored in the new storage partition is the full data and the incremental data for points 1 to 3.

[0069] Optionally, after appending all incremental data to the new storage partition, the data synchronization method provided in this embodiment further includes: updating the version number in the data version table to the latest version number corresponding to the new storage partition; and updating the log table to update the first record to the processed record.

[0070] In this embodiment, the data version table contains version numbers corresponding to each full set of data, and each version number corresponds one-to-one with each storage partition. After the latest data is appended to the new storage partition, the version number in the data version table is updated, i.e., a new version number is added. The version number marks the version corresponding to the full set of data. The version number can be flexibly adjusted according to the actual situation. For example, the version number can be the time corresponding to the full set of data. Assuming the full set of data is January 2021, the version number would be 202101.

[0071] After the data version table is updated, the first record is updated to show as a processed record. It's understandable that the synchronization of the first original data is only complete once the first record in the log table is updated to show as processed, allowing subsequent data users to find the latest version number in the data version table.

[0072] In one example of this embodiment, when the first record in the log table is still an unprocessed record, if the data user needs to read the data, it will still read the data from the previous version.

[0073] Specifically, before updating the log table to update the first record to the processed record, the data synchronization method further includes: receiving a first data read request sent by the data user, obtaining the data of the target storage partition from the result table according to the first data read request, and sending the data of the target storage partition to the data user, wherein the target storage partition is the storage partition corresponding to the previous version number of the latest version number.

[0074] In one example of this embodiment, the first data read request defaults to the current version number of the data version table. Since the first record in the log table is still an unprocessed record, even if the data version table has been updated to the latest version number corresponding to this full data, the data user cannot read it yet. The data user can only access the data in the storage partition corresponding to the previous version number of the latest version number.

[0075] For example, if the version number of the data version table is updated to 202101, but the first record is still an unprocessed record, after receiving the first data read request from the data user, the current version number of the data version table that the data user can read is 202012. Then, the data of the unique storage partition corresponding to version number 202012 is sent from the result table to the data user.

[0076] In another embodiment, as long as the data version table is updated to the latest version number, the data user can read the data in the storage partition corresponding to the latest version number.

[0077] In this embodiment, the concept of data version is introduced to achieve a smooth transition between full synchronization and incremental synchronization. When applying incremental data based on the full data corresponding to the latest version number, the data user is still accessing the data of the previous version until the data synchronization is completed. Then, the data user can find the latest version number in the data version table and smoothly switch to the new data.

[0078] As mentioned earlier, incremental data is based on the full dataset. After synchronizing the initial raw data, the process also includes synchronizing the incremental data. Figure 5 As shown, after updating the log table to update the first record to the processed record, the method further includes:

[0079] S510: Obtain the second raw data to be synchronized. The second raw data is incremental data.

[0080] S520. Write the second raw data into the intermediate table, and add a second record of unprocessed incremental data to the log table.

[0081] S530. When it is determined by polling the log table that there is a second record with unprocessed incremental data, obtain the current version number from the data version table.

[0082] S540. Store the second original data corresponding to the second record in the intermediate table in the storage partition corresponding to the current version number, and update the second record to the processed record.

[0083] Since the base data for incremental data is the full data, when incremental data to be synchronized is obtained, the second original data is cached in an intermediate table, and a second record is written to the log table. This second record is marked as incremental data that has not been processed. When it is determined through polling the log table that there is a second record of incremental data that has not been processed, the current version number is obtained from the data version. For example, if the current version number is 202101, the second original data corresponding to the second record in the intermediate table is updated and stored in the storage partition corresponding to version 202101.

[0084] Optionally, in one example of this embodiment, since each storage partition stores all the data, this embodiment can also locate any version number based on the data version table. After updating the log table to update the first record to the processed record, the data synchronization method further includes:

[0085] Send the data version table to the data user and receive the second data read request from the data user carrying the target version number; query the data of the storage partition corresponding to the target version number from the result table and send the queried data to the data user so that the data user can process the queried data.

[0086] Understandably, the data version table includes multiple version numbers, each corresponding to a storage partition. The data version table is sent to the data user, who can then retrieve the various version numbers based on the table. The data synchronization device receives a second data read request from the data user, which carries the target version number, such as version number 202011. The data synchronization device then queries the result table for the data in the storage partition corresponding to the target version number. Subsequently, the data user can perform operations such as migration, backup, or deletion on the retrieved data.

[0087] In some embodiments, when a data user, data provider, or data synchronization device discovers an error in the data, the data synchronization device can locate the error through the data version table, determine the cause of the error, and then modify the erroneous data to ensure the accuracy of the data.

[0088] To facilitate understanding, this embodiment uses a more specific application scenario to illustrate the data synchronization method.

[0089] On November 27, 2019, China Telecom, China Mobile, and China Unicom officially launched the "number portability" service nationwide. Tens of millions of users have already used this service, resulting in a massive amount of number portability data that needs to be synchronized from the group to a specialized company. The group provides full data monthly and incremental data hourly. All data is uploaded to an FTP server (FileTransfer Protocol Server) for retrieval.

[0090] As agreed, the centralized business management system generates a full data packet up to the current date at the beginning of each month (before 3:30 AM on the 1st of each month). Hourly incremental data packets are generated every hour on the hour (0:00 AM to 0:00 PM).

[0091] However, due to the different frequencies of the two synchronization methods and the variable timing of monthly synchronization (before 3:30 AM on the 1st of each month), there is a risk of data errors. For example, suppose the full monthly data is transmitted to a professional company at 3:00 AM on January 1st. Between 0:00 AM and 3:00 AM, data needs to be continuously synchronized hourly to ensure it is up-to-date. If the monthly data is directly applied for cleaning at 3:00 AM, since the monthly data only goes up to 0:00 AM on January 1st, there is a risk of losing data synchronized between 0:00 AM and 3:00 AM.

[0092] To address the aforementioned issues, this embodiment provides an entity relationship diagram for a data synchronization method, such as... Figure 6 As shown, the FTP server establishes an intermediate table 610, a log table 620, a result table 630, and a data version table 640, along with two synchronization programs: an incremental data synchronization program 650 and a full data synchronization program 660. The group acts as the original data provider 670, and specialized companies act as data users 680. The result table 630 includes multiple storage partitions, each storing different full data. After receiving the original data uploaded by the data provider, the FTP server directly writes it to the intermediate table and simultaneously inserts a record into the log table. The synchronization program polls the log table; if it finds any unprocessed data, it begins processing the corresponding table and writes the processed data to the result table, updating the data version table as needed. Data users obtain the final data by combining the data version table and the result table.

[0093] like Figure 7 As shown, if "unprocessed" data is found in the log table, it is incremental data. The specific synchronization process is as follows:

[0094] S701, Polling Log Table.

[0095] S702. Determine if there is any incremental data that needs to be processed. If yes, proceed to S703; otherwise, proceed to S701.

[0096] S703. Obtain the current data version number from the data version table.

[0097] S704. Update the result table based on the corresponding data in the intermediate table and the current data version number.

[0098] Update the incremental data in the intermediate table to the storage partition in the result table that corresponds to the current data version number.

[0099] S705. Upon success, update the status of this record to "processed".

[0100] like Figure 8 As shown, the "unprocessed" data in the log table is the full data. The specific synchronization process is as follows:

[0101] S801, Polling Log Table.

[0102] S802. Determine if there is any full data that needs to be processed. If yes, proceed to S803; otherwise, proceed to S801.

[0103] S803. Create a partition in the result table to store the full data of this operation, and insert the corresponding full data from the intermediate table into the new partition.

[0104] S804, Start a transaction (commit).

[0105] S805. Update the log table, updating all incremental data records in the log table from the time of this full data release up to the current time to "unprocessed".

[0106] S806: Call the global incremental synchronization program to reapply all incremental data from the time of this full data release up to the current time to the new partition.

[0107] S807. Update the version number in the data version table to the latest version.

[0108] S808, Transaction complete.

[0109] S809. Update the log table and change the status of this record to "processed".

[0110] Since the three major telecom operators officially launched the "number portability" service nationwide on November 27, 2019, tens of millions of users have used this service, resulting in a large amount of number portability data that needs to be synchronized from the group to a professional company.

[0111] To improve service timeliness, professional companies must incrementally synchronize number portability data at the highest frequency (hourly) to ensure that users can enjoy full service as soon as they complete the porting process. To improve data accuracy, however, it is also necessary to maintain low-frequency (monthly) full data synchronization to correct errors that may occur during high-frequency synchronization and to ensure that the data is thoroughly cleaned every month.

[0112] The above data synchronization method efficiently achieves a smooth connection between hourly incremental data and monthly full data, ensuring that a complete number portability data synchronization is performed at the beginning of each month without affecting data timeliness, and minimizing errors during synchronization.

[0113] Furthermore, even if the full data synchronization at the beginning of the month arrives late, users continue to read the latest incremental data based on the full data of the previous month. Only after receiving the full data of the latest month will the incremental data be supplemented based on the latest full version, and then the data version table will be updated to smoothly switch to the data based on the latest full version.

[0114] Furthermore, because historical version data is retained, even if error messages occur, they can be located and the cause determined by reviewing historical versions, allowing for improvements. Since historical and current number portability data reside in different physical partitions, historical data will not affect the query efficiency of current data. Migrating, backing up, or deleting historical data will not interrupt online services.

[0115] The following describes an apparatus embodiment of this application, which can be used to execute the data synchronization method described above in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the data synchronization method described above in this application.

[0116] like Figure 9 As shown, Figure 9 This is a schematic diagram illustrating the structure of a data synchronization device according to an exemplary embodiment of this application, comprising:

[0117] Data acquisition module 910 is used to acquire the first raw data to be synchronized and the data type of the first raw data, including full data or incremental data;

[0118] The data storage module 920 is used to create a new storage partition when the data type of the first original data is full data, and to store the first original data in the new storage partition; different storage partitions store different full data.

[0119] The incremental operation module 930 is used to perform incremental data operation based on the data synchronization time point of the first original data, so as to store the incremental data corresponding to the incremental data operation in a new storage partition.

[0120] The data sending module 940 is used to synchronize the data stored in the new storage partition to the data user.

[0121] In some embodiments of this application, based on the aforementioned scheme, the data synchronization device further includes a writing module. Specifically, the writing module is used to write the first raw data into an intermediate table and add a first record of unprocessed full data to a log table. The intermediate table is used to cache the data to be synchronized, and the log table is used to record the processing status of the data to be synchronized. At this time, the data storage module 920 is specifically used to create a new storage partition in the result table when it determines, by polling the log table, that there is a first record of unprocessed full data. The result table is used to store all synchronized data; and the first raw data corresponding to the first record in the intermediate table is stored in the new storage partition.

[0122] In some embodiments of this application, based on the aforementioned scheme, the incremental operation module 930 is specifically used to update the log table to update the records of incremental data from the data synchronization time point of the first original data to the current time point to unprocessed; obtain all incremental data from the intermediate table from the data synchronization time point of the first original data to the current time point, and append all incremental data to a new storage partition.

[0123] In some embodiments of this application, based on the foregoing scheme, the data synchronization device further includes an update module, which is specifically used to update the version number in the data version table to the latest version number corresponding to the new storage partition; the data version table is used to manage the version number of all data; and the update log table is used to update the first record to the processed record.

[0124] In some embodiments of this application, based on the aforementioned scheme, the data synchronization device further includes a receiving module. Before the updating module updates the log table to update the first record to the processed record, the receiving module is used to receive a first data read request sent by the data user. At this time, the data sending module 940 is also used to obtain the data of the target storage partition from the result table according to the first data read request, and send the data stored in the target storage partition to the data user. The target storage partition is the storage partition corresponding to the previous version number of the latest version number.

[0125] In some embodiments of this application, based on the aforementioned scheme, after the update module updates the log table to update the first record to a processed record, the data acquisition module 910 is further used to acquire the second raw data to be synchronized. The second raw data is incremental data. The writing module is further used to write the second raw data into an intermediate table and add a second record of unprocessed incremental data to the log table. The data acquisition module 910 is further used to acquire the current version number in the data version table when it is determined by polling the log table that there is a second record of unprocessed incremental data. The data storage module 920 is further used to store the second raw data corresponding to the second record in the intermediate table in the storage partition corresponding to the current version number and update the second record to a processed record.

[0126] In some embodiments of this application, based on the foregoing scheme, after the update module updates the log table to update the first record to the processed record, the data sending module 940 is further configured to send the data version table to the data user, and the receiving module is further configured to receive the second data read request carrying the target version number from the data user; the data sending module 940 is further configured to query the data of the storage partition corresponding to the target version number from the result table, and send the queried data to the data user so that the data user can process the queried data.

[0127] It should be noted that the apparatus provided in the above embodiments and the method provided in the above embodiments belong to the same concept, and the specific way in which each module and unit performs operations has been described in detail in the method embodiments, and will not be repeated here.

[0128] In one exemplary embodiment, an electronic device includes one or more processors; and a storage device for storing one or more programs that, when executed by the one or more processors, cause the electronic device to perform the method described above.

[0129] Figure 10 This is a schematic diagram of the structure of an electronic device according to an exemplary embodiment.

[0130] It should be noted that this electronic device is merely an example adapted to this application and should not be construed as providing any limitation on the scope of use of this application. Furthermore, this electronic device should not be interpreted as requiring or depending on any specific feature. Figure 10 One or more components of the exemplary electronic device shown.

[0131] like Figure 10As shown, in an exemplary embodiment, the electronic device includes a processing component 1001, a memory 1002, a power supply component 1003, a multimedia component 1004, an audio component 1005, a processor 1006, a sensor component 1007, and a communication component 1008. Not all of these components are mandatory; the electronic device may add or remove other components according to its functional requirements, and this embodiment does not impose any limitations.

[0132] Processing component 1001 typically controls the overall operation of an electronic device, such as operations related to display, data communication, and log data synchronization. Processing component 1001 may include one or more processors 1006 to execute instructions to complete all or part of the aforementioned operations. Furthermore, processing component 1001 may include one or more modules to facilitate interaction between processing component 1001 and other components. For example, processing component 1001 may include a multimedia module to facilitate interaction between multimedia component 1004 and processing component 1001.

[0133] The memory 1002 is configured to store various types of data to support operation of the electronic device, examples of which include instructions for any application or method operating on the electronic device. The memory 1002 stores one or more modules configured to be executed by the one or more processors 1006 to perform all or part of the steps in the methods described in the above embodiments.

[0134] Power supply component 1003 provides power to various components of an electronic device. Power supply component 1003 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device.

[0135] Multimedia component 1004 includes a screen that provides an output interface between an electronic device and a user. In some embodiments, the screen may include a TP (Touch Panel) and an LCD (Liquid Crystal Display). If the screen includes a touch panel, the screen can be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation.

[0136] Audio component 1005 is configured to output and / or input audio signals. For example, audio component 1005 includes a microphone configured to receive external audio signals when the electronic device is in an operating mode, such as a call mode, a recording mode, or a voice recognition mode. In some embodiments, audio component 1005 also includes a speaker for outputting audio signals.

[0137] Sensor assembly 1007 includes one or more sensors for providing state assessments of various aspects of the electronic device. For example, sensor assembly 1007 can detect the on / off state of the electronic device and can also detect temperature changes in the electronic device.

[0138] The communication component 1008 is configured to facilitate wired or wireless communication between electronic devices and other devices. The electronic devices can access wireless networks based on communication standards, such as Wi-Fi (Wireless-Fidelity).

[0139] Understandable. Figure 10 The structure shown is for illustrative purposes only; the electronic device may include more than [the structure shown]. Figure 10 The more or fewer components shown, or having the same Figure 10 The different components are shown. Figure 10 Each component shown can be implemented using hardware, software, or a combination thereof.

[0140] In one exemplary embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the method described above. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not incorporated into the electronic device.

[0141] It should be noted that the computer-readable storage medium shown in the embodiments of this application can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0142] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0143] The above description is merely a preferred exemplary embodiment of this application and is not intended to limit the implementation of this application. Those skilled in the art can easily make corresponding modifications or alterations based on the main concept and spirit of this application. Therefore, the scope of protection of this application should be determined by the scope of protection claimed in the claims.

Claims

1. A data synchronization method, characterized in that: include: Acquire first original data to be synchronized and a data type of the first original data, where the data type includes full data or incremental data; When the data type of the first original data is the full amount of data corresponding to the first time period generated at a preset time point, the first original data is written into the intermediate table, and a first record of the full amount of data that has not been processed is added to the log table; the intermediate table is used to cache the data to be synchronized, and the log table is used to record the processing status of the data to be synchronized, and the first time period does not include the preset time point; When it is determined by polling the log table that there is a first record of unprocessed full data, a new storage partition is created in the result table; the result table is used to store all synchronized data; wherein, each time the full data is obtained, a storage partition is created, different storage partitions store the full data obtained at different times, and each storage partition retains the full data of each time period; storing the first original data corresponding to the first record in the intermediate table in the new storage partition; Updating the log table to update the record of incremental data corresponding to a second time period from a data synchronization time point of the first original data to a preset time point as unprocessed, wherein the data synchronization time point is the end point of the first time period, and the second time period is less than the first time period; Acquire all incremental data from the data synchronization time point of the first original data to a preset time point from the intermediate table, and append all incremental data to the new storage partition to make the data stored in the new storage partition up to date; The data stored in the new storage partition is synchronized to the data user.

2. The method according to claim 1, characterized in that After appending and storing all incremental data in the new storage partition, the method further includes: Update the version number in the data version table to the latest version number corresponding to the new storage partition; the data version table is used to manage the version number of the full amount of data; The log table is updated to update the first record to a processed record.

3. The method according to claim 2, characterized in that Before updating the log table to update the first record to a processed record, the method further includes: Receiving a first data reading request sent by a data user; According to the first data read request, data of the target storage partition is obtained from the result table, and the data stored in the target storage partition is sent to the data user, wherein the target storage partition is a storage partition corresponding to a previous version number of the latest version number.

4. The method according to claim 2, characterized in that: After updating the log table to update the first record to a processed record, the method further includes: Acquire second original data to be synchronized, where the second original data is incremental data; Writing the second original data into the intermediate table, and adding a second record of unprocessed incremental data into the log table; When it is determined by polling the log table that there is a second record of unprocessed incremental data, obtaining the current version number in the data version table; The second original data corresponding to the second record in the intermediate table is stored in the storage partition corresponding to the current version number, and the second record is updated to a processed record.

5. The method according to claim 2, characterized in that: After updating the log table to update the first record to a processed record, the method further includes: Sending the data version table to a data user, and receiving a second data read request carrying a target version number from the data user; The data of the storage partition corresponding to the target version number is queried from the result table, and the queried data is sent to the data user so that the data user processes the queried data.

6. A data synchronization device, characterized in that: include: A data acquisition module, used to acquire first original data to be synchronized and a data type of the first original data, where the data type includes full data or incremental data; a writing module, configured to write the first original data into an intermediate table and add a first unprocessed record of the full amount of data to a log table when the data type of the first original data is the full amount of data corresponding to the first time period generated at a preset time point; the intermediate table is used to cache the data to be synchronized, the log table is used to record the processing status of the data to be synchronized, and the first time period does not include the preset time point; A data storage module is used to create a new storage partition in the result table when it is determined by polling the log table that there is a first record of unprocessed full data; the result table is used to store all synchronized data; the first original data corresponding to the first record in the intermediate table is stored in the new storage partition; wherein, each time the full data is obtained, a storage partition is created, different storage partitions store the full data obtained at different times, and each storage partition retains the full data of each time period; An incremental operation module is used to update the log table to update the record of incremental data corresponding to the second time period from the data synchronization time point of the first original data to the preset time point as unprocessed, wherein the data synchronization time point is the end point of the first time period, and the second time period is less than the first time period; obtain all incremental data from the data synchronization time point of the first original data to the preset time point from the intermediate table, and append all incremental data to the new storage partition to make the data stored in the new storage partition up to date; The data sending module is used to synchronize the data stored in the new storage partition to the data user.

7. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: Computer-readable instructions are stored thereon, and when the computer-readable instructions are executed by a processor of a computer, the computer is caused to execute the method according to any one of claims 1 to 5.

Images