A data synchronization method and related products
By splitting MySQL data update statements into delete and insert statements and generating version numbers associated with timestamps and values, the problem of non-sequential version numbers in HBase databases is solved, achieving strong consistency and low loss rate in data synchronization.
Patent Information
- Application Number
- CN202510622105.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-14
- Publication Date
- 2026-11-17
AI Technical Summary
When synchronizing data from a MySQL database to an HBase database, errors in the timestamp field may cause the version number to not increment in sequence, resulting in data loss.
The update statement in the MySQL data is split into delete and insert statements. The version number of the target data is generated and is related to the latest timestamp and the target value. The target value increments or returns to zero within the same second to ensure the monotonically increasing nature of the version number.
This ensures proper synchronization of data version numbers, reduces the possibility of data loss, and guarantees strong consistency in data synchronization.
Smart Images

Figure CN123115269A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular to a data synchronization method and related products. Background Technology
[0002] HBase (Hadoop Database) is an open-source, non-relational, distributed database used to store massive amounts of data. Because HBase is built on the Hadoop Distributed File System (HDFS), it features high reliability, high performance, column-oriented storage, and easy scalability. Therefore, in practical applications, business systems can synchronize data from their original MySQL databases to HBase, continuing to provide services with the help of HBase.
[0003] However, when data changes, the timestamp field in the log used to record data change operations may contain errors, preventing it from continuously incrementing. Since this timestamp field is highly correlated with the data's version number, synchronizing data to the HBase database may cause the version number of the data obtained by the HBase database to not increment sequentially, resulting in data loss. Summary of the Invention
[0004] This application provides a data synchronization method and related products to ensure that data version numbers are synchronized normally, reduce the possibility of data loss, and achieve strong consistency in data synchronization.
[0005] The embodiments of this application disclose the following technical solutions:
[0006] In a first aspect, embodiments of this application provide a data synchronization method, including:
[0007] Retrieve MySQL data from a MySQL database;
[0008] The update statements in the MySQL data are split into delete and insert statements to obtain the target data;
[0009] A version number for the target data is generated, which is related to the latest timestamp and target value of the target data; wherein, if an operation on the target data is performed within the same second, the target value is obtained by incrementing the preset original value; if no operation on the target data is performed within the same second, the target value is reset to the original value.
[0010] Based on the version number, the target data is written to the HBase database.
[0011] Secondly, embodiments of this application provide a data synchronization device, comprising:
[0012] The data acquisition module is used to retrieve MySQL data from the MySQL database.
[0013] The statement splitting module is used to split the update statements in the MySQL data into delete statements and insert statements to obtain the target data.
[0014] A version number generation module is used to generate a version number for the target data. The version number is related to the latest timestamp and target value of the target data. If an operation on the target data is performed within the same second, the target value is obtained by incrementing the preset original value. If no operation on the target data is performed within the same second, the target value is reset to the original value.
[0015] The data synchronization module is used to write the target data into the HBase database based on the version number.
[0016] Thirdly, embodiments of this application provide an electronic device, the device including a processor and a memory:
[0017] The memory is used to store computer programs and to transfer the computer programs to the processor;
[0018] The processor is configured to execute the steps of the data synchronization method provided in the first aspect according to the instructions in the computer program.
[0019] Fourthly, embodiments of this application provide a computer-readable storage medium for storing program code for executing the steps of the data synchronization method provided in the first aspect.
[0020] Fifthly, embodiments of this application provide a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps of the data synchronization method provided in the first aspect.
[0021] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:
[0022] In this embodiment, after retrieving MySQL data from the MySQL database, since HBase database does not support direct update operations, the update statements in the MySQL data can be split into delete and insert statements, that is, the update operation can be split into delete and insert operations to obtain the target data. Next, a version number for the target data is generated. This version number is related to the latest timestamp of the target data and the target value. If the operation on the target data is performed within the same second, the target value is obtained by incrementing the pre-set original value; if the operation on the target data is not performed within the same second, the target value is reset to the original value. In this way, since the version number is not based on the latest timestamp of the target data, but rather by associating the target data's version number with both the latest timestamp and the target value, and the target value increments within the same second and resets to its original value at different times, monotonically increasing version numbers can be achieved every second. Furthermore, the counting restarts when a new second begins, ensuring that the order of version numbers is not disrupted by inconsistencies during master-slave failover, thus achieving normal synchronization of data version numbers. Subsequently, the target data can be written to the HBase database based on its version number, reducing the possibility of data loss and achieving strong consistency in data synchronization. Attached Figure Description
[0023] Figure 1 A flowchart illustrating a data synchronization method provided in an embodiment of this application;
[0024] Figure 2 A flowchart illustrating another data synchronization method provided in this application embodiment;
[0025] Figure 3 A flowchart of an initialization process provided in an embodiment of this application;
[0026] Figure 4 A flowchart illustrating a periodic disk-flush metadata process is provided as an embodiment of this application.
[0027] Figure 5 A flowchart illustrating a retry mechanism provided in an embodiment of this application;
[0028] Figure 6 A schematic diagram of an HBase table provided in an embodiment of this application;
[0029] Figure 7 A schematic diagram of the overall architecture of a system for applying a data synchronization method provided in an embodiment of this application;
[0030] Figure 8 This is a schematic diagram of the structure of a data synchronization device provided in an embodiment of this application;
[0031] Figure 9 This application provides a schematic diagram of the structure of a server according to an embodiment of the present application.
[0032] Figure 10 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation
[0033] For ease of understanding, the terminology that may be involved in the embodiments of this application will be introduced below.
[0034] MySQL is a relational database management system that uses Structured Query Language (SQL) as its standard database language. Relational databases store data in different tables, rather than putting all the data in one large repository, which increases speed and improves flexibility.
[0035] HBase is an open-source, non-relational, distributed database used to store massive amounts of data. Because HBase is built on top of HDFS, it features high reliability, high performance, columnar storage, and easy scalability.
[0036] Kafka is an open-source stream processing platform. As a high-throughput distributed publish-subscribe messaging system, it can process all action stream data from consumers on a website.
[0037] Furthermore, as mentioned earlier, for example, when data changes, if the primary and backup servers switch over at different times, the timestamp field in the log used to record the data change operation may not continuously increment; that is, the new timestamp field may be smaller than the old timestamp field. In this case, since this timestamp field is highly correlated with the data version number, synchronizing the data to the HBase database may cause the version number of the data obtained by the HBase database to not increment in sequence, resulting in data loss.
[0038] To address the aforementioned issues, this application provides a data synchronization method. Specifically, after retrieving MySQL data from a MySQL database, since HBase databases do not support direct update operations, the update statements in the MySQL data can be split into delete and insert statements, i.e., the update operation is broken down into delete and insert operations to obtain the target data. Next, a version number for the target data is generated. This version number is related to the latest timestamp and the target value of the target data. If operations on the target data are performed within the same second, the target value is incremented by a pre-set original value; if no operations are performed within the same second, the target value is reset to its original value. In this way, since the version number is not based on the latest timestamp of the target data, but rather by associating the target data's version number with both the latest timestamp and the target value, and the target value increments within the same second and resets to its original value at different times, monotonically increasing version numbers can be achieved every second. Furthermore, the counting restarts when a new second begins, ensuring that the order of version numbers is not disrupted by inconsistencies during master-slave failover, thus achieving normal synchronization of data version numbers. Subsequently, the target data can be written to the HBase database based on its version number, reducing the possibility of data loss and achieving strong consistency in data synchronization.
[0039] It should be noted that the embodiments of this application do not limit the executing entity of the technical solution of this application. For example, the data synchronization method provided in the embodiments of this application can be applied to a user terminal or a server, or it can be processed collaboratively by a user terminal and a server. As an example, a user terminal includes, but is not limited to, mobile phones, computers, smart voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc. The server can be a standalone server, a cluster server, or a cloud server.
[0040] To enable those skilled in the art to better understand the present invention, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0041] Figure 1 A flowchart illustrating a data synchronization method provided in an embodiment of this application. Figure 2A flowchart illustrating another data synchronization method provided in an embodiment of this application. (In conjunction with...) Figure 1 As shown, the data synchronization method provided in this application embodiment may include:
[0042] S101: Retrieve MySQL data from the MySQL database.
[0043] Here, MySQL data refers to the data to be synchronized stored in the MySQL database. In this embodiment, the MySQL data can first be synchronized from the MySQL database to the Kafka platform, and then further synchronized to the HBase database via the Kafka platform.
[0044] Based on this, in specific implementation, combined with Figure 2 As shown, firstly, a consumer thread in the Kafka platform can be started to pull raw data from a MySQL database. Next, a predefined MySQL regular expression is used to identify the pulled raw data, retaining only the data that matches the regular expression and deleting the data that does not. This reduces the amount of subsequent data processing, thereby improving data processing efficiency.
[0045] In practical applications, different business scenarios or different data to be synchronized may correspond to different processing rules and requirements. Therefore, combining... Figure 2 As shown, before starting the consumer thread to pull data, the database configuration information can be read first. The database configuration information can define corresponding configuration parameters, such as the regular expression mentioned above or the mapping relationship of HBase tables in the HBase database, thereby indicating the execution of the subsequent data synchronization scheme.
[0046] Next, combined Figure 2 As shown, after reading the corresponding database configuration information, the current data position marker and version number can be further loaded to improve data consistency. For example, offset is an identifier used in the Kafka platform to mark the data position. Loading the offset helps to start consuming data from the correct position, thereby reducing the possibility of duplicate consumption or missing data.
[0047] Based on this, in this embodiment of the application, an initialization scheme can also be provided to initialize the aforementioned database configuration information, thereby preparing for the task of data synchronization from the MySQL database to the HBase database. For ease of understanding, this initialization scheme is described below with reference to a possible implementation method.
[0048] As one possible implementation, the database configuration information can be initialized first.
[0049] Here, regarding the initialization process of database configuration information, in specific implementation, firstly, relevant metadata of the HBase database can be obtained, such as table structure and column family information, to understand the current state and available resources of the HBase database, providing a foundation for subsequent data writing. Then, combined with... Figure 3 As shown, the instance information table of the SQL database can be queried to identify multiple related MySQL instances, thus determining the range of MySQL instances for which data needs to be collected and providing a data source for subsequent data collection tasks. Next, the connection information of the HBase instances and the Zookeeper (ZK) configuration information in the HBase database are obtained. Since ZK can be used to manage the HBase database, a stable connection to the HBase database can be established for data operations. Then, the connection pool of the HBase database can be initialized so that subsequent data write operations can efficiently reuse connections. Next, the queried MySQL instances are associated with their corresponding collection paths to determine the data collection path and rules for each MySQL instance, thereby establishing a clear data collection process so that data from each MySQL instance can be collected and processed according to predetermined rules. Then, MySQL tables are filtered according to predefined MySQL regular expressions to identify tables that match a specific pattern, and the corresponding HBase tables are determined based on the aforementioned collection paths, thus establishing a mapping relationship between MySQL tables and HBase tables to ensure that data is accurately written to the HBase tables.
[0050] Next, after initializing the database configuration information, it's possible to check if there are preset configuration parameters for the current data synchronization task. These preset parameters can then be used to instruct the execution of the data synchronization method. Accordingly, if the database configuration information includes preset configuration parameters, they can be loaded; otherwise, a parameter creation task is performed. Finally, the preset configuration parameters can be persistently stored, ensuring their availability even after a system restart. For ease of understanding, Table 1 below provides an illustrative explanation of the preset configuration parameters.
[0051] Table 1
[0052]
[0053] It should be noted that Fqueue_size, one of the preset configuration parameters shown in Table 1 above, refers to the method area memory, with a default size of 512MB. This can be changed by modifying MaxPermSize in the startup script. Accordingly, Fqueue_size supports a maximum of approximately 20,000 entries; otherwise, it will cause the Java Virtual Machine (JVM) to malfunction.
[0054] For Fjvm_xmx_xms, its value is calculated as follows: Heap memory = 2 * Fmax_poll_records * size of each record (approximately 30KB) + other heap objects (approximately 200MB). This formula indicates that the size of the heap memory mainly depends on Fmax_poll_records and the approximate size of each record, while also including the memory occupied by other heap objects.
[0055] Regarding Fmax_poll_records, assuming there's a database instance with 4000 Transactions Per Second (TPS) and 4000 rows written per second, the current TPS is approximately (20000 rows / 13.889s) ≈ 1428 rows / s. Therefore, for a write rate of 4000 rows / s, Fmax_poll_records would need to be 20000 * (4000 / 1428) = 56000. However, in reality, many factors need to be considered, such as the time spent writing to the HBase database, the number of RegionServer (RS) connections, or the sparsity of actual valid data. Therefore, under normal circumstances, considering data processing efficiency and resource balance, the ratio of Fmax_poll_records to the current processing rate can be set to 1:1, corresponding to the current processing rate.
[0056] In addition, combined Figure 2 As shown, to improve data processing efficiency, in this embodiment, after obtaining MySQL data, the MySQL data can be divided into multiple batches, and then processed in a concurrent manner, thereby leveraging the advantages of concurrent execution to improve processing efficiency. Furthermore, the MySQL data within each batch can be processed sequentially through a traversal process, ensuring the correct processing order for data within a small scope and avoiding data chaos or errors caused by excessive concurrency.
[0057] S102: Split the update statement in the MySQL data into delete and insert statements to obtain the target data.
[0058] In this embodiment of the application, considering that the HBase database itself does not support direct update operations, therefore, in combination with Figure 1 and Figure 2 As shown, in practical implementation, the update operation in the MySQL database can be converted into a delete operation followed by an insert operation to obtain the target data, thereby achieving the purpose of updating the data.
[0059] S103: Generate the version number of the target data. The version number is related to the latest timestamp of the target data and the target value. If the operation on the target data is performed within the same second, the target value is obtained by incrementing the pre-set original value. If the operation on the target data is not performed within the same second, the target value is reset to the original value.
[0060] After the statement transformation, to improve the consistency and accuracy of data versions, appropriate version numbers must be regenerated for the new delete and insert statements. For ease of understanding, one possible implementation method is described below.
[0061] As one possible implementation, regarding step S103 above, firstly, the latest timestamp and the previous timestamp of the target data can be obtained. In practical applications, both the latest timestamp and the previous timestamp can be represented by the timestamp field begintime in binlog (a type of binary log). begintime can be used to record the time sequence of data changes. Therefore, the latest timestamp of the target data can indicate the time when the target data was last changed, while the previous timestamp of the target data can indicate the time when the change occurred before the latest timestamp.
[0062] Next, a target timestamp can be generated based on the latest timestamp and the previous timestamp, and a version number can be generated based on the target timestamp and the target value. The version number includes the target timestamp and the target value in sequence.
[0063] For example, version number = target timestamp + target value, where the target timestamp is represented by a 10-digit begintime, and the original value corresponding to the target value is a fixed 9-digit value of 100000000. Based on this, regarding the process of determining the target value, as mentioned earlier, if an operation on the target data is performed within the same second, the target value is obtained by incrementing the pre-set original value. The target value can be incremented according to a preset step size. For example, if the target data changes within one second, the original value can be incremented by a step size of 2 to obtain the target value, such as incrementing from 100000000 to 100000002, and then from 100000002 to 100000004, and so on.
[0064] If an operation on the target data is not performed within the same second, the target value is reset to its original value. In other words, if the target data changes within or after the next second, the target value can be reset to its original value, 100000000. This ensures the uniqueness and sequentiality of different operations or data versions within the same second, avoiding version number conflicts.
[0065] Furthermore, regarding the generation process of the target timestamp, in the specific implementation, if the latest timestamp is less than the previous timestamp, the latest timestamp can be processed to generate the target timestamp, which is greater than or equal to the previous timestamp. In this way, even if the latest timestamp becomes less than the previous timestamp due to an anomaly, this solution ensures that the target timestamp will not decrease along with the latest timestamp, effectively preventing data version chaos caused by timestamp anomalies and improving the stability and orderliness of version numbers. Conversely, if the latest timestamp is greater than the previous timestamp, it can be directly used as the target timestamp.
[0066] To facilitate understanding, based on the above content, the generation of the target data version number will be explained below with reference to Table 2.
[0067] Table 2
[0068]
[0069] As shown in Table 2 above, the final generated version number can be associated with begintime and the target value. The target value can maintain an increasing trend within the same second and return to the original value at different times. Therefore, the version number can be monotonically increased within each second, and the counting can start again when the time enters a new second. This ensures that the order of the version numbers will not be confused due to the time inconsistency during the master-slave switch, thus achieving normal synchronization of the data version numbers.
[0070] Furthermore, in this embodiment, in order to reduce the possibility of data loss after a process crashes and restarts, reduce the risk of data loss, and improve the continuity and reliability of data, a periodic disk flushing mechanism can also be provided to periodically persist the relevant metadata used to generate the above version number, so that in the event of a process failure and restart, the previously saved metadata can be successfully read and loaded into the version number generation algorithm provided in the above embodiment to continue the data processing process.
[0071] Specifically, combined Figure 4As shown, metadata can be persistently stored in a MySQL database using an asynchronous disk flushing thread. This metadata is used to generate version numbers. For example, metadata may include a Global Transaction Identifier (GTID), offset, version number, and begintime. Thus, if the process restarts during the generation of the version number for the target data, the metadata can be read from the MySQL database, and the version number can be generated based on this metadata. Therefore, periodic disk flushing allows the previous metadata state to be restored when the process restarts, enabling data processing to resume from the point of interruption, ensuring the continuity and consistency of data processing, and thus achieving high system availability.
[0072] S104: Write the target data to the HBase database based on the version number.
[0073] In this embodiment, the HBase database can store data in the form of HBase tables. Therefore, combined with Figure 2 As shown, to successfully write the target data to the HBase database, the target data is first converted to a format that matches the required format of the HBase table. Next, a queue can be created for each HBase table, and the converted data is placed into the corresponding queue. This allows for orderly management and scheduling of data across different tables, facilitating subsequent concurrent write operations and reducing interference between data writes from different tables. Then, the converted data can be written in batches using the concurrent execution method mentioned above; that is, the first write operation is performed based on the version number, and this first write operation is used to write the converted data to the HBase table.
[0074] Furthermore, during the execution of the first write operation, various reasons such as network problems or HBase database service anomalies may cause write failures. Therefore, in this embodiment, a combination of immediate retries (quickly retrying failed write operations) and delayed retries (retrying after a certain time interval for multiple failures) can be used to retrieve write operations, thereby improving the reliability and success rate of data writing. For ease of understanding, the retry mechanism is described below with reference to the accompanying drawings.
[0075] Combination Figure 5 As shown in this embodiment, firstly, if the first write operation fails, the first write operation can be re-executed until the number of executions of the first write operation reaches a first preset number. Then, the converted data is converted into a binary format data stream and saved in the MySQL database.
[0076] For example, the initial preset number of attempts can be set to four, meaning the first write operation can be retried three times immediately. If the write operation still fails after three immediate attempts, the transformed data is further converted into the aforementioned binary format data stream and stored in the MySQL database. This multiple immediate attempts handle potential temporary network issues or brief fluctuations in the HBase database service, improving the write success rate. Furthermore, in cases where multiple immediate attempts fail, the transformed data is temporarily stored in a reliable storage system, ensuring that the transformed data is not lost due to failed HBase database writes and allowing for subsequent retries.
[0077] Next, a data stream can be retrieved from the MySQL database, and a second write operation can be performed to write the data stream to an HBase table. Then, if the second write operation fails, it is re-executed at target time intervals until the number of executions reaches a second preset number. At this point, an alarm message is output, and the target time interval is incremented based on the number of second write operations executed.
[0078] For example, in practical implementation, a timed thread can be started to read and store the aforementioned data stream and perform a second write operation at target time intervals. The second preset interval can be 10 times, and the target time interval can increase with the number of times the second write operation is executed. That is, if the first second write operation fails, it can be retried after a 1-hour interval; if the second second write operation fails, it can be retried after a 2-hour interval; if the third second write operation also fails, it can be retried after a 3-hour interval, and so on, until the 10th second write operation fails, at which point an alarm message can be output. In this way, by gradually increasing the retry interval, frequent retries can be avoided when the HBase database service is still unstable, reducing the pressure on the system and giving the HBase database service time to recover.
[0079] Furthermore, it should be noted that since the HBase database itself does not support built-in indexes, in order to improve data retrieval efficiency, in this embodiment of the application, a corresponding index table can be designed and maintained in the HBase database, thereby improving data query efficiency with the help of the index table. For ease of understanding, the following description is provided in conjunction with the accompanying drawings and a possible implementation method.
[0080] As one possible implementation method, combined with Figure 6 As shown, the HBase tables mentioned above can include a main table and an index table.
[0081] The main table can be implemented as a bucketed table. For example, it can be bucketed according to the value of a certain column, and all data can be distributed and stored in different buckets.
[0082] Accordingly, the row key of the main table can include a first value, which is the last n digits of the value extracted from the partition column, where n is a positive integer. The partition column is used to partition the transformed data; the row value of the main table is used to store the transformed data.
[0083] For example, when n is 3, the first value is SUBSTR(partition column, -3). This allows us to use the last three digits of the partition column value as the basis for bucketing, distributing the data evenly across 1000 different buckets and avoiding data skew. The RowValue table in the main table stores all the transformed data, including complete data records. This data satisfies the business's requirements for data integrity, facilitating various business processing and data statistics operations.
[0084] In addition, the RowKey of this main table can also include a time column and a globally unique column. The time column represents the time attribute of the data, such as the creation or update time. Including time information in the RowKey facilitates data querying and management by time range. The globally unique column ensures that each record in the main table has a unique identifier, preventing duplicate data storage and achieving uniqueness for each record in the bucketed table.
[0085] Based on the above, the format of the RowKey in the main table can be represented as SUBSTR(partition column, -n)#time column#global unique column, where # is a concatenation operator used to join the various parts into a complete RowKey string.
[0086] Index tables, on the other hand, allow for fast queries based on the value of a specific column in the main table. For example, by establishing a mapping relationship between the index column and the RowKey in the main table, the corresponding record in the main table can be found quickly during a query without performing a full table scan, thus greatly improving query efficiency.
[0087] Accordingly, the row key of the index table includes a second value, which is the value obtained by reversing the index column, which is used to create the index; the row value of the index table is used to store the row key of the main table.
[0088] For example, the second value is REVERSE(index column), which reverses the value of the index column and uses it as part of the RowKey in the index table. The RowValue in the index table stores the RowKey from the main table. In this way, when the corresponding RowKey is found through the index table, data can be quickly retrieved from the main table using that RowKey, achieving efficient data retrieval.
[0089] In addition, the RowKey of this index table can also include a time column and a globally unique column, which serve a similar purpose to those in the main table. The time column is used to represent the time attribute of the data, and the globally unique column enables the data to have a unique identifier in the main table.
[0090] Based on the above, the RowKey format of this index can be represented as REVERSE(index column)#time column#globally unique column, where # is a concatenation operator used to join the various parts into a complete RowKey string.
[0091] In this way, by leveraging the main table and index tables, the data query performance of the non-relational HBase database can be significantly improved, thereby meeting various data access needs of the business and greatly enhancing the system's concurrent processing performance.
[0092] Furthermore, combined Figure 2 As shown, after writing the target data to the HBase database, key metadata, such as the offset, version number, and timestamp mentioned above, can be persistently stored. This facilitates subsequent data replay status recovery, data consistency checks, and troubleshooting through metadata. Then, after the entire data replay batch is complete, the offset can be committed, indicating that the data before that offset has been successfully processed. This updates the consumption progress, providing an accurate starting point for the next data replay and avoiding duplicate processing of successfully processed data.
[0093] Based on the above steps S101-S104, it is clear that in this embodiment, after obtaining MySQL data from the MySQL database, since the HBase database does not support direct update operations, the update statements in the MySQL data can be split into delete and insert statements, that is, the update operation can be split into delete and insert operations to obtain the target data. Next, a version number for the target data is generated. This version number is related to the latest timestamp and the target value of the target data. If the operation on the target data is performed within the same second, the target value is obtained by incrementing the pre-set original value; if the operation on the target data is not performed within the same second, the target value is reset to the original value. In this way, since the version number is not based on the latest timestamp of the target data, but rather by associating the target data's version number with both the latest timestamp and the target value, and the target value increments within the same second and resets to its original value at different times, monotonically increasing version numbers can be achieved every second. Furthermore, the counting restarts when a new second begins, ensuring that the order of version numbers is not disrupted by inconsistencies during master-slave failover, thus achieving normal synchronization of data version numbers. Subsequently, the target data can be written to the HBase database based on its version number, reducing the possibility of data loss and achieving strong consistency in data synchronization.
[0094] Based on the data synchronization method provided in the foregoing embodiments, this application can also provide a system that applies this data synchronization method. The overall architecture of the system will be described below with reference to the embodiments and accompanying drawings.
[0095] Figure 7 This is a schematic diagram of the overall architecture of a system providing an application data synchronization method according to an embodiment of this application. (Combined with...) Figure 7 As shown in this embodiment, the overall architecture of the system may include: a web front-end, an API layer, a management layer, an execution layer, and a storage layer.
[0096] The web front-end is primarily built upon Element UI (an open-source UI component library), webpack (a static module bundler), and ECMAScript (a scripting language). Element UI is used for page design, webpack for bundling, and ECMAScript for page scripting. The process of building a web front-end using these three tools can be implemented using any existing or future implementation method; no specific limitations are specified here.
[0097] The Application Programming Interface (API) layer provides stateless interfaces, Power2 access functionality, and load balancing capabilities. The stateless interfaces can be accessed externally, for example, for archiving and statistical purposes. Power2 access functionality enables permission management, enhancing system security and access control. Load balancing improves interface performance and availability, better handling multiple requests and processes, thus ensuring stable system operation.
[0098] The management team can provide the following functions:
[0099] (1) Consumption management function, that is, to perform batch operations and management on consumers, such as providing batch restart, migration or addition of consumers, to improve the efficiency and flexibility of consumption management.
[0100] (2) Database access function, which is to write metadata for new databases in the MySQL cluster, so as to update the metadata in a timely manner and enable the system to obtain and process the information of new databases in the MySQL cluster in a timely manner.
[0101] (3) Report detection function, which comprehensively detects the operating status of each node, such as providing detection information on consumption latency, JVM or offset level of each node, so as to provide important data support for system maintenance and optimization, so as to discover and solve problems in a timely manner.
[0102] (4) Node management function, which is to manage various information of the node in detail, such as the node's IP, port, version, registration status, etc., thereby helping to grasp the node status and carry out reasonable resource allocation and management.
[0103] (5) Access task function, that is, to manage and control tasks in a comprehensive manner, such as creating, viewing, starting, pausing or restarting tasks, thereby improving the flexibility and controllability of task execution.
[0104] (6) Service upgrade function, which provides upgrade functions for the Dts-action process (used to execute the data synchronization scheme) and the Aqua-agent process (used to collect data), so that the system's functions and performance can be continuously updated and optimized as needs and technologies develop.
[0105] (7) GRPC (a remote call framework) function, which provides a channel for the management layer to issue commands to the execution layer or for the execution layer to report information to the management layer, thereby efficiently realizing the information flow between the management layer and the execution layer, and enabling information to be transmitted and executed in a timely manner.
[0106] (8) High availability function, namely, high availability is ensured by ZK master selection, which improves the system's reliability and disaster recovery capabilities and reduces system downtime caused by failure.
[0107] (9) Metadatabase function, which is a storage center that stores all the metadata of the entire platform operation. It is an important foundation for the stable operation of the system and can realize data persistence and recoverability.
[0108] The execution layer can include processes such as Dts-action and Aqua-agent. The Dts-action process, acting as the executor of data synchronization, can be flexibly deployed as needed; multiple Dts-action processes can be deployed on the same server to improve data synchronization efficiency and performance. The Aqua-agent process, acting as a probe, can collect information on the server's CPU, memory, and disk usage, as well as collect and monitor server resource information and the status of the Dts-action processes, enabling fine-grained management and monitoring of the execution layer. Only one Aqua-agent process is deployed per server.
[0109] The storage layer can include a MySQL database and an HBase database, with the MySQL database serving as the source and the HBase database as the destination. Through the overall architecture provided by this system, data can be synchronized from the MySQL database to the HBase database.
[0110] Furthermore, based on the data synchronization method provided in the preceding embodiments, this application embodiment can also provide corresponding application scenarios, which will be described in detail below.
[0111] In practical applications, the above data synchronization methods can be applied to various business tables, such as Ma receipt table, Medi receipt table, bank POS table, Bank receipt table, transaction table, order table, or transaction log report, etc.
[0112] The Ma Payment Receipt table is used for payment-related transactions with the payment platform and records payment information received through this payment channel. Using the aforementioned data synchronization method, it can support a transaction volume of 4.6 billion records per day per transaction link for the Ma Payment Receipt table, totaling 9.2 billion records per day.
[0113] The Medi Payment Statement table is used for payment transactions related to the healthcare field, recording detailed accounting and management of healthcare institutions' revenue from healthcare services. Using the data synchronization method described above, it can support a transaction volume of 2.1 billion records per day per transaction link, totaling 4.2 billion records per day.
[0114] The bank POS table is used to record transactions made by POS machines. Using the data synchronization method described above, it can support a transaction volume of 1.8 billion records per day per transaction link, totaling 3.6 billion records per day.
[0115] The Bank Receipts Form can summarize and record relevant bank receipt transactions. Using the data synchronization method described above, it can support a transaction volume of 1.8 billion records per day per transaction link, totaling 3.6 billion records per day.
[0116] The transaction log can record various transaction behaviors. Using the data synchronization method described above, it can support a transaction log volume of 1.6 billion records per day per transaction link, totaling 3.2 billion records per day.
[0117] The volume of order tables and transaction log reports will not be elaborated here, but the total system capacity is approximately 52.4 billion records per day. This demonstrates that the data synchronization method provided by this solution offers efficient data storage, transmission, processing, and analysis capabilities for such a massive data volume, ensuring the smooth operation of all business processes. Furthermore, given the high requirements for consistency, concurrency, and availability in the financial sector, this solution precisely achieves these financial-grade strong consistency, high concurrency, and high availability.
[0118] Based on the data synchronization method provided in the foregoing embodiments, this application embodiment may also provide a data synchronization device. The data synchronization device will now be described in conjunction with the embodiments and accompanying drawings.
[0119] Figure 8 This is a schematic diagram of a data synchronization device provided in an embodiment of this application. Figure 8 As shown, the data synchronization device 800 provided in this application embodiment includes:
[0120] Data acquisition module 801 is used to acquire MySQL data from a MySQL database;
[0121] The statement splitting module 802 is used to split the update statements in the MySQL data into delete statements and insert statements to obtain the target data.
[0122] Version number generation module 803 is used to generate a version number of the target data, which is related to the latest timestamp and target value of the target data; wherein, if an operation on the target data is performed within the same second, the target value is obtained by incrementing the preset original value; if no operation on the target data is performed within the same second, the target value is reset to the original value.
[0123] The data synchronization module 804 is used to write the target data into the HBase database based on the version number.
[0124] Optionally, the version number generation module 803 includes:
[0125] The timestamp acquisition module is used to acquire the latest timestamp and the previous timestamp of the target data;
[0126] The timestamp generation module is used to generate a target timestamp based on the latest timestamp and the previous timestamp;
[0127] The version number generation submodule is used to generate the version number based on the target timestamp and the target value, wherein the version number includes the target timestamp and the target value in sequence.
[0128] Optionally, the timestamp generation module is specifically used for:
[0129] If the latest timestamp is less than the previous timestamp, the latest timestamp is processed to generate the target timestamp, and the target timestamp is greater than or equal to the previous timestamp;
[0130] If the latest timestamp is greater than the previous timestamp, the latest timestamp shall be used as the target timestamp.
[0131] Optionally, the HBase database stores data in the form of HBase tables;
[0132] The data synchronization module 804 is specifically used for:
[0133] The target data is converted to obtain converted data, and the format of the converted data matches the data format required by the HBase table.
[0134] A first write operation is performed based on the version number, the first write operation being used to write the transformed data into the HBase table.
[0135] Optionally, the HBase table includes a main table and an index table;
[0136] The row key of the main table includes a first value, which is the last n digits of the value extracted from the partition column, where n is a positive integer. The partition column is used to partition the transformed data. The row values of the main table are used to store the transformed data.
[0137] The row key of the index table includes a second value, which is a value obtained by reversing the index column, and the index column is used to create an index; the row values of the index table are used to store the row key of the main table.
[0138] Optionally, the data synchronization device 800 further includes:
[0139] The first retry module is used to re-execute the first write operation if the first write operation fails, until the number of executions of the first write operation reaches a first preset number, then convert the converted data into a binary format data stream and save it in the MySQL database.
[0140] The data stream acquisition module is used to acquire the data stream from the MySQL database and perform a second write operation, wherein the second write operation is used to write the data stream into the HBase table;
[0141] The second retry module is used to re-execute the second write operation according to a target time interval if the second write operation fails, until the number of executions of the second write operation reaches a second preset number, and then output an alarm message. The target time interval is incremented based on the number of executions of the second write operation.
[0142] Optionally, the data synchronization device 800 further includes:
[0143] A metadata storage module is used to store metadata in the MySQL database via an asynchronous disk flushing thread, and the metadata is used to generate the version number;
[0144] The version number generation module 803 is specifically used for:
[0145] If the process restarts, the metadata is read from the MySQL database;
[0146] The version number is generated based on the metadata.
[0147] Optionally, the data synchronization device 800 further includes:
[0148] The initialization module is used to initialize the database configuration information;
[0149] A data loading module is used to load the preset configuration parameters when the database configuration information includes preset configuration parameters, wherein the preset configuration parameters are used to indicate the execution of the data synchronization method;
[0150] The data storage module is used to persistently store the preset configuration parameters.
[0151] The structure of the control equipment for implementing the above data synchronization methods is described below for both server and terminal device configurations.
[0152] Figure 9This is a schematic diagram of a server structure provided in an embodiment of this application. The server 900 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 922 (e.g., one or more processors) and memory 932, and one or more storage media 930 (e.g., one or more mass storage devices) for storing application programs 942 or data 944. The memory 932 and storage media 930 can be temporary or persistent storage. The program stored in the storage media 930 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the server. Furthermore, the CPU 922 may be configured to communicate with the storage media 930 and execute the series of instruction operations in the storage media 930 on the server 900.
[0153] Server 900 may also include one or more power supplies 926, one or more wired or wireless network interfaces 950, one or more input / output interfaces 958, and / or one or more operating systems 941, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM etc.
[0154] The steps performed by the server in the above embodiments can be based on this Figure 9 The server structure shown.
[0155] The CPU 922 is used in the following steps:
[0156] Retrieve MySQL data from a MySQL database;
[0157] The update statements in the MySQL data are split into delete and insert statements to obtain the target data;
[0158] A version number for the target data is generated, which is related to the latest timestamp and target value of the target data; wherein, if an operation on the target data is performed within the same second, the target value is obtained by incrementing the preset original value; if no operation on the target data is performed within the same second, the target value is reset to the original value.
[0159] Based on the version number, the target data is written to the HBase database.
[0160] This application also provides another control device, such as... Figure 10 As shown, for ease of explanation, only the parts related to the embodiments of this application are shown. For specific technical details not disclosed, please refer to the method section of the embodiments of this application. The terminal can be any terminal device including mobile phones, tablets, personal digital assistants (PDAs), point-of-sale (POS) terminals, in-vehicle computers, etc. Taking a mobile phone as an example:
[0161] Figure 10 This is a block diagram illustrating a portion of the structure of a mobile phone related to the terminal provided in the embodiments of this application. (Reference) Figure 10 The mobile phone includes: a radio frequency (RF) circuit 1010, a memory 1020, an input unit 1030, a display unit 1040, a sensor 1050, an audio circuit 1060, a wireless fidelity (WiFi) module 1070, a processor 1080, and a power supply 1090, etc. Those skilled in the art will understand that... Figure 10 The mobile phone structure shown does not constitute a limitation on the mobile phone and may include more or fewer components than shown, or combine certain components, or have different component arrangements.
[0162] The following is combined Figure 10 A detailed introduction to each component of a mobile phone:
[0163] The RF circuit 1010 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and processes it with the processor 1080; additionally, it transmits uplink data to the base station. Typically, the RF circuit 1010 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc. Furthermore, the RF circuit 1010 can also communicate wirelessly with networks and other devices. The aforementioned wireless communications may use any communication standard or protocol, including but not limited to Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, and Short Messaging Service (SMS).
[0164] The memory 1020 can be used to store software programs and modules. The processor 1080 executes various mobile phone functions and data processing by running the software programs and modules stored in the memory 1020. The memory 1020 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 1020 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.
[0165] The input unit 1030 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of the mobile phone. Specifically, the input unit 1030 may include a touch panel 1031 and other input devices 1032. The touch panel 1031, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 1031), and drive the corresponding connection devices according to a pre-set program. Optionally, the touch panel 1031 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 1080, and can also receive and execute commands sent by the processor 1080. In addition, the touch panel 1031 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 1031, the input unit 1030 may also include other input devices 1032. Specifically, other input devices 1032 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.
[0166] The display unit 1040 can be used to display information input by the user or information provided to the user, as well as various menus of the mobile phone. The display unit 1040 may include a display panel 1041, which may optionally be configured as a Liquid Crystal Display (LCD), Organic Light-Emitting Diode (OLED), or similar display panel 1041. Furthermore, a touch panel 1031 may cover the display panel 1041. When the touch panel 1031 detects a touch operation on or near it, it transmits the information to the processor 1080 to determine the type of touch event. Subsequently, the processor 1080 provides corresponding visual output on the display panel 1041 based on the type of touch event. Although in Figure 10 In this embodiment, the touch panel 1031 and the display panel 1041 are two separate components to realize the input and output functions of the mobile phone. However, in some embodiments, the touch panel 1031 and the display panel 1041 can be integrated to realize the input and output functions of the mobile phone.
[0167] The mobile phone may also include at least one sensor 1050, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel 1041 according to the ambient light level, and the proximity sensor can turn off the display panel 1041 and / or the backlight when the phone is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity and can be used for applications that recognize the phone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometer, taps), etc. Other sensors that may be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.
[0168] The audio circuit 1060, speaker 1061, and microphone 1062 provide an audio interface between the user and the mobile phone. The audio circuit 1060 converts the received audio data into electrical signals and transmits them to the speaker 1061, where the speaker 1061 converts them into sound signals for output. On the other hand, the microphone 1062 converts the collected sound signals into electrical signals, which are then received by the audio circuit 1060, converted into audio data, and then processed by the processor 1080 before being transmitted via the RF circuit 1010 to, for example, another mobile phone, or the audio data can be output to the memory 1020 for further processing.
[0169] WiFi is a short-range wireless transmission technology. Through the WiFi module 1070, mobile phones can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access. Although Figure 10 The WiFi module 1070 is shown, but it is understood that it is not an essential component of a mobile phone and can be omitted as needed without changing the essence of the invention.
[0170] The processor 1080 is the control center of the mobile phone, connecting various parts of the phone through various interfaces and lines. It executes software programs and / or modules stored in the memory 1020 and calls data stored in the memory 1020 to perform various functions and process data, thereby collecting overall data and information from the phone. Optionally, the processor 1080 may include one or more processing units; preferably, the processor 1080 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 1080.
[0171] The mobile phone also includes a power supply 1090 (such as a battery) that supplies power to various components. Preferably, the power supply can be logically connected to the processor 1080 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.
[0172] Although not shown, mobile phones may also include a camera, Bluetooth module, etc., which will not be described in detail here.
[0173] In this embodiment of the application, the processor 1080 included in the terminal also has the following functions:
[0174] Retrieve MySQL data from a MySQL database;
[0175] The update statements in the MySQL data are split into delete and insert statements to obtain the target data;
[0176] A version number for the target data is generated, which is related to the latest timestamp and target value of the target data; wherein, if an operation on the target data is performed within the same second, the target value is obtained by incrementing the preset original value; if no operation on the target data is performed within the same second, the target value is reset to the original value.
[0177] Based on the version number, the target data is written to the HBase database.
[0178] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.
[0179] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0180] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.
[0181] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0182] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0183] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.
[0184] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. A data synchronization method, characterized in that, include: Retrieve MySQL data from a MySQL database; The update statements in the MySQL data are split into delete and insert statements to obtain the target data; A version number for the target data is generated, which is related to the latest timestamp and target value of the target data; wherein, if an operation on the target data is performed within the same second, the target value is obtained by incrementing the preset original value; if no operation on the target data is performed within the same second, the target value is reset to the original value. Based on the version number, the target data is written to the HBase database.
2. The data synchronization method according to claim 1, characterized in that, The version number used to generate the target data includes: Obtain the latest timestamp and the previous timestamp of the target data; Generate a target timestamp based on the latest timestamp and the previous timestamp; The version number is generated based on the target timestamp and the target value, wherein the version number includes the target timestamp and the target value in sequence.
3. The data synchronization method according to claim 2, characterized in that, The step of generating a target timestamp based on the latest timestamp and the previous timestamp includes: If the latest timestamp is less than the previous timestamp, the latest timestamp is processed to generate the target timestamp, and the target timestamp is greater than or equal to the previous timestamp; If the latest timestamp is greater than the previous timestamp, the latest timestamp shall be used as the target timestamp.
4. The data synchronization method according to claim 1, characterized in that, The HBase database stores data in the form of HBase tables; The step of writing the target data into the HBase database based on the version number includes: The target data is converted to obtain converted data, and the format of the converted data matches the data format required by the HBase table. A first write operation is performed based on the version number, the first write operation being used to write the transformed data into the HBase table.
5. The data synchronization method according to claim 4, characterized in that, The HBase tables include a main table and an index table; The row key of the main table includes a first value, which is the last n digits of the value extracted from the partition column, where n is a positive integer. The partition column is used to partition the transformed data. The row values of the main table are used to store the transformed data. The row key of the index table includes a second value, which is a value obtained by reversing the index column, and the index column is used to create an index; the row values of the index table are used to store the row key of the main table.
6. The data synchronization method according to claim 4, characterized in that, After performing the first write operation based on the version number, the method further includes: If the first write operation fails, the first write operation is re-executed until the number of executions of the first write operation reaches a first preset number. Then, the converted data is converted into a binary data stream and saved in the MySQL database. The data stream is obtained from the MySQL database, and a second write operation is performed to write the data stream to the HBase table. If the second write operation fails, the second write operation is re-executed according to the target time interval until the number of executions of the second write operation reaches the second preset number, at which point an alarm message is output. The target time interval is incremented based on the number of executions of the second write operation.
7. The data synchronization method according to any one of claims 1 to 6, characterized in that, The method further includes: Metadata is stored in the MySQL database via an asynchronous disk flushing thread, and the metadata is used to generate the version number; The version number used to generate the target data includes: If the process restarts, the metadata is read from the MySQL database; The version number is generated based on the metadata.
8. The data synchronization method according to any one of claims 1 to 6, characterized in that, Before retrieving MySQL data from the MySQL database, the method further includes: Initialize the database configuration information; If the database configuration information includes preset configuration parameters, the preset configuration parameters are loaded, and the preset configuration parameters are used to indicate the execution of the data synchronization method; The preset configuration parameters are persistently stored.
9. A data synchronization device, characterized in that, include: The data acquisition module is used to retrieve MySQL data from the MySQL database. The statement splitting module is used to split the update statements in the MySQL data into delete statements and insert statements to obtain the target data. A version number generation module is used to generate a version number for the target data. The version number is related to the latest timestamp and target value of the target data. If an operation on the target data is performed within the same second, the target value is obtained by incrementing the preset original value. If no operation on the target data is performed within the same second, the target value is reset to the original value. The data synchronization module is used to write the target data into the HBase database based on the version number.
10. An electronic device, characterized in that, The device includes a processor and a memory: The memory is used to store computer programs and to transfer the computer programs to the processor; The processor is configured to execute the steps of the data synchronization method according to any one of claims 1 to 8, based on instructions in the computer program.
11. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program that, when executed by a terminal device, implements the steps of the data synchronization method according to any one of claims 1 to 8.
12. A computer program product, characterized in that, It includes a computer program that, when executed by a terminal device, implements the steps of the data synchronization method according to any one of claims 1 to 8.
