Data synchronization method, electronic device, and computer-readable storage medium

Abstract

The application discloses a data synchronization method, an electronic device and a computer readable storage medium. The method comprises the following steps: a processing node acquires first data description information in a meta database in response to a data synchronization task trigger; the first data description information comprises second data description information of a master database and third data description information of a synchronization database; the processing node consumes data in the master database according to the second data description information; the processing node partitions the consumed data to obtain a temporary data set; and the processing node synchronizes the temporary data set to the synchronization database according to the third data description information. Through the above method, the technical problem that the expansibility is not strong due to different business rules of various business databases during database synchronization can be solved.

Description

Technical Field

[0001] This application relates to the field of data processing, and in particular to a data synchronization method, electronic device, and computer-readable storage medium. Background Technology

[0002] With the rapid development of business, various data synchronization scenarios have emerged. To enable data in the master database to be synchronized more quickly to various business databases to provide data support for their functions, the usual approach is to consume data from the master database and then synchronize it to the corresponding database. However, business databases specify relevant rules, resulting in limited business scalability. Furthermore, if a new table structure needs to be added to a business database, the relevant code needs to be modified, and real-time updates are not possible. Summary of the Invention

[0003] The main purpose of this application is to provide a data synchronization method, electronic device, and computer-readable storage medium that can solve the technical problem of poor scalability caused by different business rules of various business databases during database synchronization.

[0004] To address the aforementioned technical problems, the first technical solution adopted in this application is to provide a data synchronization method. This method is applied to a distributed system, which includes processing nodes, a metadata database, a master database, and a synchronization database. The method includes: a processing node, in response to a data synchronization task trigger, obtaining first data description information from the metadata database; the first data description information includes second data description information from the master database and third data description information from the synchronization database; the processing node consumes data from the master database according to the second data description information; the processing node partitions the consumed data to obtain a temporary dataset; and the processing node synchronizes the temporary dataset to the synchronization database according to the third data description information.

[0005] To solve the aforementioned technical problems, the second technical solution adopted in this application is to provide an electronic device. This electronic device includes a memory and a processor. The memory stores program data, which can be executed by the processor to implement the method described in the first technical solution.

[0006] To address the aforementioned technical problems, the third technical solution adopted in this application is to provide a computer-readable storage medium. This computer-readable storage medium stores program data and can be executed by a processor to implement the method described in the first technical solution.

[0007] The beneficial effects of this application are as follows: By constructing a metadata database to store data description information of the main database and synchronization data, when a data synchronization task is triggered, the second description information obtained from the main database and the third data description information obtained from the synchronization database, which are pre-stored in the metadata database, are retrieved. Data that has not yet been synchronized in the main database is read using the second data description information of the main database, and the read data is synchronized to the synchronization database using the third data description information of the synchronization database, thereby achieving data synchronization between the main database and the synchronization database. By constructing a metadata database to obtain data description information of downstream synchronization databases, this solution can simultaneously support data synchronization of multiple downstream synchronization databases and flexibly support the updating, expansion, and addition of downstream synchronization databases. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 This is a flowchart illustrating the first embodiment of the data synchronization method of this application;

[0010] Figure 2 This is a schematic diagram of the distributed system used in this application;

[0011] Figure 3 This is a flowchart illustrating the second embodiment of the data synchronization method of this application;

[0012] Figure 4 This is a flowchart illustrating the third embodiment of the data synchronization method of this application;

[0013] Figure 5 This is a flowchart illustrating the fourth embodiment of the data synchronization method of this application;

[0014] Figure 6 This is a schematic diagram of a specific implementation example of the data synchronization execution process;

[0015] Figure 7 This is a schematic diagram of the structure of the first embodiment of the electronic device of this application;

[0016] Figure 8 This is a schematic diagram of the structure of the first embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0018] The terms "first," "second," etc., used in this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0019] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0020] Reference Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the data synchronization method of this application. The method is applied to a distributed system. The distributed system includes processing nodes, a metadata database, a master database, and a synchronization database. Figure 2 As shown, Figure 2 This is a schematic diagram of the distributed system used in this application. The master database, metadata database, and synchronization database establish communication connections with the processing nodes. The method includes the following steps:

[0021] S11: The processing node responds to the data synchronization task trigger by obtaining the first data description information from the metadata database.

[0022] In response to a data synchronization task, the processing node requests services from the metadata database to obtain the first data description information. This first data description information includes the second data description information from the master database and the third data description information from the synchronization database. This data description information is equivalent to metadata in the database; metadata, also known as intermediary data or relay data, describes the data, primarily its attributes, and supports functions such as indicating storage location, historical data, resource lookup, and file records.

[0023] S12: The processing node consumes data from the main database based on the second data description information.

[0024] The processing node uses the acquired second data description information from the primary database to consume data from the primary database, thereby retrieving data from the primary database that has not yet been synchronized, in preparation for subsequent synchronization of the database. Consumption can be understood as an act of acquiring data.

[0025] S13: The processing node partitions the consumed data to obtain a temporary dataset.

[0026] The data in the main database is partitioned according to predetermined rules, and data from the same partition is treated as a temporary dataset. These temporary datasets are then synchronized to the main database.

[0027] S14: The processing node synchronizes the temporary dataset to the synchronization database according to the third data description information.

[0028] When synchronizing a temporary dataset, the temporary dataset is synchronized according to the third data description information of the synchronization database so that the data in the temporary dataset can conform to the data storage rules in the synchronization database.

[0029] In this embodiment, a metadata database is constructed to store data description information for the main database and synchronization data. When a data synchronization task is triggered, the second description information obtained from the main database and the third data description information obtained from the synchronization database, which are pre-stored in the metadata database, are retrieved. Data that has not yet been synchronized in the main database is read using the second data description information of the main database, and the read data is synchronized to the synchronization database using the third data description information of the synchronization database, thereby achieving data synchronization between the main database and the synchronization database. By constructing a metadata database to obtain data description information from downstream synchronization databases, this solution can simultaneously support data synchronization of multiple downstream synchronization databases and flexibly support the updating, expansion, and addition of downstream synchronization databases.

[0030] In one embodiment, the aforementioned distributed system utilizes Hadoop Zookeepe for cluster coordination during use, which can fully guarantee the smooth operation of the database cluster.

[0031] In one embodiment, before the processing node responds to the data synchronization task, a meta-database is constructed, and data description information of the master database and the synchronization database is stored in the meta-database.

[0032] Specifically, taking Kafka as the primary database and Hive and HBase as synchronization databases as an example, a metadata repository is established in the RDS database. Upon the first runtime of the program, data description information is retrieved from each database. Using Kafka consumers, metadata such as all topics, partitions, and offsets of each partition are obtained from Kafka. Parsing the source information requesting Kafka also yields metadata information such as the number of partitions and consumer groups. All this metadata information is written to the Kafka data description table in the metadata repository. Then, the schema information from the Hive database is retrieved and written to the Hive data description table in the metadata repository. Similarly, the schema information from the HBase database is retrieved and written to the HBase data description table in the metadata repository.

[0033] In one embodiment, after the processing node partitions the consumed data to obtain a temporary dataset, the process further includes: persisting the temporary dataset. The persistence operation caches data read from the main database in memory or on disk, or backs up multiple copies, thereby reducing the time consumed by task retries due to data loss or downstream task failures.

[0034] In another embodiment, after the processing node synchronizes the temporary dataset to the synchronization database according to the third data description information, the method further includes updating the second data description information according to the attribute information of the temporary dataset in response to the synchronization of the same batch of temporary datasets.

[0035] After a batch of temporary datasets is synchronized from the main database to the synchronization database, the second data description information in the main database is updated according to the partitions, offsets, and status information of the processed batch of data. This ensures that subsequent data synchronization processes can be handled based on the updated second data description information, thereby updating the task progress. If the second data description information is updated abnormally after a batch of temporary databases is synchronized, the next synchronization schedule may continue to synchronize data based on the outdated second data description information, resulting in duplicate data consumption.

[0036] When processing nodes perform data synchronization, they will retry if the process fails. Only after all data in the batch is successfully synchronized will the second data description information be updated, thereby ensuring strong data consistency and preventing data loss.

[0037] Reference Figure 3 , Figure 3 This is a flowchart illustrating a second embodiment of the data synchronization method of this application. The method is a further extension of step S12, and includes the following steps:

[0038] S21: The processing node obtains the starting offset of the preset topic in the second data description information within the processing node based on the preset topic.

[0039] When retrieving data from the main database that needs to be synchronized, the second description information is first obtained. The data offset in the second description information is used to determine the starting position for data reading. Since the main database stores various types of data, the corresponding data is first determined according to a preset topic. Then, the starting offset of the data in the processing node is obtained based on the preset topic, allowing the processing node to process the data under the preset topic based on this starting offset.

[0040] S22: Determine the data to be consumed based on the starting offset and the current offset range of the preset topic in the main database.

[0041] After determining the starting offset corresponding to the preset topic, the data to be consumed is determined based on the current offset of that topic in the main database. The current offset range of the main database is the data offset range of the main database at the current point in time.

[0042] In one embodiment, in response to the starting offset being within the current offset interval, the data corresponding to the maximum endpoint of the current offset interval from the starting offset is taken as the data to be consumed; in response to the starting offset being outside the current offset interval, the data corresponding to the current offset interval is taken as the data to be consumed or an anomaly alert is issued.

[0043] When the starting offset is within the current offset range, it means that the data has already been consumed. The offsets generated by the previously consumed data have been stored in the metadata database. In this case, only the data between the starting offset and the maximum endpoint of the range from the starting offset to the current offset will be consumed later.

[0044] When the starting offset is outside the current offset range, it indicates that data has expired or is abnormal. In this case, the data offset needs to be reset to ensure the normal operation of the data system. Therefore, the starting offset is reset to the minimum endpoint of the current offset range, and the data corresponding to the current offset range is consumed. Specifically, there are two scenarios: First, when the starting offset is less than the minimum endpoint of the current offset range, this is normal and usually indicates that data has expired due to prolonged inactivity in the main database. If data from the starting offset to the minimum endpoint of the current offset range is consumed now, the main database will throw an exception. Therefore, the data corresponding to the current offset range is consumed, or an exception is thrown depending on the actual situation. Second, when the starting offset is greater than the maximum endpoint of the current offset range, this is an abnormal situation. This usually indicates that the metadata database still contains uncleaned data offsets, or that data in the metadata database has been manually modified, leading to inconsistent data results. Therefore, the data corresponding to the current offset range is consumed, or an exception is thrown depending on the actual situation.

[0045] Furthermore, in response to the number of partitions corresponding to the starting offset being less than the number of partitions corresponding to the minimum endpoint in the current offset interval, all data from the newly added partitions in the main database, as well as the incremental data from the original partitions, are added to the data to be consumed. In response to the number of partitions corresponding to the starting offset being greater than the number of partitions corresponding to the minimum endpoint in the current offset interval, all data from all partitions in the main database are used as the data to be consumed.

[0046] When the number of partitions corresponding to the starting offset is less than the number of partitions corresponding to the minimum endpoint in the current offset interval, it indicates that the primary database has triggered an expansion mechanism, and the number of partitions in the primary database has increased. Since data will also flow into the expanded partitions, the expanded partitions, i.e., the newly added partitions, need to be consumed in full. However, for the previous partitions, the expanded portion is not considered, so the original partitions need to be consumed incrementally.

[0047] If the number of partitions corresponding to the starting offset is greater than the number of partitions corresponding to the maximum endpoint in the current offset range, it is an abnormal situation, indicating that there are still data offsets in the metadata database that have not been cleaned up. Therefore, it should be modified to be consistent with the number of partitions corresponding to the current offset range, and the data corresponding to the current offset range should be consumed, or an exception should be thrown depending on the actual situation.

[0048] Reference Figure 4 , Figure 4 This is a flowchart illustrating the third embodiment of the data synchronization method of this application. The method is a further extension of step S13, and includes the following steps:

[0049] S31: The processing node partitions the data according to the topic category.

[0050] After the processing node obtains the data to be consumed from the main database, since the obtained data is obtained according to a preset theme, the obtained data can be distinguished according to the theme category, and data of the same theme category can be treated as the same partition.

[0051] S32: Parse the data in the same partition to generate a temporary dataset.

[0052] Data within the same partition is processed uniformly, and the main database data within the same partition is parsed to generate a temporary dataset for use in subsequent tasks.

[0053] According to the above method, this embodiment stores metadata information such as the offset of the main database in the metadata database, and can process data under different topics at the same time, so that it can support consuming multiple topics and reading data under multiple topics when reading.

[0054] Reference Figure 5 , Figure 5 This is a flowchart illustrating the fourth embodiment of the data synchronization method of this application. The method is a further extension of step S14, and includes the following steps:

[0055] S41: The processing node maps the data in the temporary dataset to the first type of data according to the third data description information.

[0056] The data type of the first type of data is the same as the data type of the data in the synchronous database. The processing node changes the data of the temporary dataset to a data type suitable for storage in the synchronous database according to the third data description information.

[0057] In one embodiment, each computing node in the processing node receives broadcast third data description information, which includes metadata information related to the synchronization database. The computing nodes process the temporary dataset according to this received information.

[0058] S42: Synchronize the first type of data to the synchronization database.

[0059] After the data type change is completed in the temporary dataset, the data is stored in the corresponding synchronous database.

[0060] In one embodiment, the synchronization database includes multiple databases, and the first type of data includes multiple types of data. After the temporary dataset is processed, each type of data is stored in the corresponding synchronization database according to the corresponding relationship.

[0061] The following specific embodiment will be used to illustrate the technical solution of this application in detail.

[0062] In this embodiment, Kafka is the primary database, Hive and HBase are the synchronization databases, RDS is the metadata database, Spark is the processing node, and ZooKeeper is used for cluster coordination.

[0063] Before starting data synchronization, first create a metadata database in the RDS database.

[0064] The entire data consumption process can be divided into three parts: input, consumption, and output. Each end includes different components, and each component is responsible for maintaining different metadata at different stages. For the input component metadata, it's crucial to ensure that the same message is transmitted identically each time, guaranteeing message idempotency and preventing message loss, even in the event of message backlog. For the consumption component metadata, metadata needs to be read during consumption to maintain the consumption progress and ensure at-first-come, at-last-time semantics. For the output component metadata, data must be persisted, or the metadata's real-time and persistence must be guaranteed.

[0065] Then the program starts its first run to pull metadata information from the Kafka, Hive, and HBase databases and write it to the metadata database.

[0066] The metadata retrieval process for the Kafka database begins with obtaining the Kafka domain name via ZooKeeper. Since ZooKeeper handles Kafka's consistency service, each Kafka broker needs to record its own address in ZooKeeper, allowing the domain name information to be retrieved directly from ZooKeeper. Retrieving this information from ZooKeeper also reduces the number of configuration parameters. Next, the Kafka domain name is used to connect to Kafka, and successfully connected domain names are added to the valid set. If a domain name fails to connect, it is discarded. Then, the Kafka consumer retrieves all topics, partitions, and the offset of each partition, among other metadata. The source information obtained from the request to Kafka is parsed to obtain metadata such as the number of partitions and consumer groups. All this metadata information is written to the Kafka data description table in the metadata repository. The table structure of the Kafka data description table is shown in Table 1.

[0067] Table 1

[0068] name type length non-empty Notes topic varchar 128 √ kaka topic partitions int 16 √ Number of partitions partitionid int 16 √ Partition ID offset varchar 255 √ offset createtime datetime 0 √ Creation time group varchar 255 √ group name

[0069] Here, topic represents the Kafka topic, partitions indicates the total number of partitions in the current topic, partitionid indicates the number of partitions in the current topic, offset indicates the offset of the current topic, createtime indicates the time of insertion into the table, and group indicates the Kafka consumer group.

[0070] The metadata retrieval process for the Hive database involves obtaining the schema information of the tables in the Hive database and writing the schema information into the Hive data description information table in the metadata database. The table structure of the Hive data description information table is shown in Table 2.

[0071] Table 2

[0072] name type length non-empty Notes db varchar 255 √ Database name tb varchar 255 √ Table name fieldname varchar 255 √ Field Name fieldtype varchar 255 √ Field type

[0073] Where db represents the Hive database name, tb represents the Hive table name, fieldname represents the Hive table field name, and fieldtype represents the Hive table field type.

[0074] The metadata retrieval process for the HBase database involves obtaining the schema information of the tables in the HBase database and writing the schema information into the HBase data description information table in the metadata database. The table structure of the HBase data description information table is shown in Table 3.

[0075] Table 3

[0076]

[0077] Here, columnncluster represents the HBase database name, tb represents the HBase table name, fieldname represents the HBase table field name, and fieldtype represents the HBase table field type.

[0078] like Figure 6 As shown, Figure 6 This is a schematic diagram of a specific implementation of the data synchronization process. After the information in the data description table of the metadata database is obtained, data synchronization is performed. In response to the triggering of the data synchronization task, the processing node requests the RDS service to read the Kafka table description information table, the Hive table description information table, and the HBase table description information table to obtain metadata information such as table name, table fields, and topic offset.

[0079] Then, a Kafka consumer is created to consume all data in the topic from the offset recorded in RDS up to the current time point. Assume the starting offset record for the topic in RDS is `startOffset`, and the offset range of the data in Kafka at the current time point is `oldestOffset` and `endOffset`, where `oldestOffset` is the smallest offset in Kafka and `endOffset` is the largest offset in Kafka. Then, the following situations typically occur:

[0080] If `startOffset` is greater than `oldestOffset` but less than `endOffset`, this indicates that the data has already been consumed. The offset of previously consumed data is stored in RDS as `startOffset`. The corresponding approach is to consume the incremental data between `startOffset` and `endOffset`.

[0081] `StartOffset` being less than `oldestOffset` is normal. This typically occurs when Kafka data hasn't been consumed for a long time, causing data to expire. If you try to consume data between `startOffset` and `oldestOffset`, Kafka will throw an exception. Therefore, the appropriate approach is to consume all data from Kafka, or optionally throw an exception.

[0082] StartOffset greater than endOffset: This is an abnormal situation, generally occurring when RDS stores an old, uncleaned startOffset, or when RDS data has been manually modified, leading to data inconsistency. The corresponding handling is to consume all data from Kafka and record the largest offset for each partition in RDS. Alternatively, you can choose to throw an exception and stop the task, depending on the situation.

[0083] If the number of partitions in StartOffset is less than OldestOffset, it means Kafka has triggered its scaling mechanism. Under this mechanism, the number of Kafka partitions will increase. Data will also flow into the newly scaled partitions, so these newly scaled partitions need to be fully consumed, while older partitions need to be incrementally consumed.

[0084] If the number of partitions in startOffset is greater than the oldestOffset: This situation is the same as the third situation, caused by the existence of old, uncleaned startOffsets in RDS. You can either consume all the data in Kafka or, depending on the situation, choose to throw an exception and stop the task.

[0085] After reading data from Kafka, Spark partitions the Kafka data and performs batch cleaning, packaging, and generation of a temporary dataset. Spark can read data from a single Kafka topic or data from multiple Kafka topics. During data processing, data is retrieved from a specified offset position in Kafka to obtain information about the batch. The batch is then partitioned according to the topic. Data within the same partition is processed uniformly, and the Kafka data is parsed based on the partitions to generate a temporary dataset for subsequent tasks.

[0086] After generating a temporary dataset, Spark performs persistence operations on the data. The persistence operation caches the data read from Kafka into memory or disk, and can further back up multiple copies to reduce the time consumed by data loss or task retries due to downstream task failures.

[0087] After obtaining the temporary dataset, the table names and corresponding metadata information of HBase and Hive, along with the table field information, are broadcast as broadcast variables to all computing nodes. Simultaneously, a global variable is maintained in each executor; tasks only need to use this variable during execution, significantly reducing the memory overhead of the executor.

[0088] Next, each Spark compute node maps the dataset to an HBase data type and inserts it into the HBase database using broadcast variables, and also maps the dataset to a Hive data type and inserts it into the Hive database using broadcast variables.

[0089] After the data is stored in both HBase and Hive databases, the second data description information in the main database is updated based on the partitions, offsets, and status information of the processed batch of data. This second description information includes the offset information from the Kafka database. This allows subsequent data synchronization processes to be handled according to the updated offset information in the second data description information, thus updating the task progress. If the second data description information is updated abnormally after a batch of temporary databases has been synchronized, the next synchronization schedule may continue to synchronize data based on the outdated second data description information, resulting in duplicate data consumption.

[0090] like Figure 7 As shown, Figure 7 This is a schematic diagram of the structure of the first embodiment of the electronic device of this application.

[0091] The electronic device includes a processor 110 and a memory 120.

[0092] Processor 110 controls the operation of electronic devices. Processor 110 may also be referred to as a CPU (Central Processing Unit). Processor 110 may be an integrated circuit chip with signal sequence processing capabilities. Processor 110 may also be a general-purpose processor, a digital signal sequence processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor.

[0093] The memory 120 stores the instructions and program data required for the processor 110 to operate.

[0094] The processor 110 is used to execute instructions to implement the method provided in any of the first to third embodiments of the data synchronization method described above in this application and possible combinations thereof.

[0095] like Figure 8 As shown, Figure 8 This is a schematic diagram of the structure of the first embodiment of the computer-readable storage medium of this application.

[0096] One embodiment of the readable storage medium of this application includes a memory 210 that stores program data that, when executed, implements the method provided in any of the first to third embodiments of the data method of this application and possible combinations thereof.

[0097] The memory 210 may include a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or other media that can store program instructions. Alternatively, it may be a server that stores the program instructions, which can send the stored program instructions to other devices for execution or execute the stored program instructions itself.

[0098] In summary, by constructing a metadata database to store data description information for the master database and synchronization data, when a data synchronization task is triggered, the system retrieves the second description information pre-stored in the metadata database (obtained from the master database) and the third data description information (obtained from the synchronization database). It then uses the second data description information from the master database to read data that has not yet been synchronized, and uses the third data description information from the synchronization database to synchronize the read data to the synchronization database, thereby achieving data synchronization between the master and synchronization databases. By constructing a metadata database to obtain data description information from downstream synchronization databases, this solution can simultaneously support data synchronization with multiple downstream synchronization databases and flexibly support updates, expansions, and additions to downstream synchronization databases.

[0099] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0100] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0101] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0102] If the integrated units in the other embodiments described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0103] The above are merely embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A data synchronization method, characterized in that, Applied to a distributed system, the distributed system including processing nodes, a metadata database, a master database, and a synchronization database, the method includes: In response to the data synchronization task, the processing node obtains the first data description information from the metadata database; the first data description information includes the second data description information of the master database and the third data description information of the synchronization database. The processing node consumes data from the main database based on the second data description information; The processing node partitions the consumed data according to the topic category to obtain a temporary dataset; The processing node synchronizes the temporary dataset to the synchronization database according to the third data description information.

2. The method according to claim 1, characterized in that, The processing node consumes data from the main database based on the second data description information, including: The processing node obtains the starting offset of the preset topic in the second data description information within the processing node according to the preset topic; The data to be consumed is determined based on the starting offset and the current offset range of the preset topic in the main database.

3. The method according to claim 2, characterized in that, The step of determining the data to be consumed based on the starting offset and the current offset range of the preset topic in the main database includes: In response to the starting offset being located within the current offset interval, the data corresponding to the maximum endpoint of the current offset interval from the starting offset is taken as the data to be consumed. If the starting offset is outside the current offset range, the data corresponding to the current offset range is either consumed or an anomaly alert is issued.

4. The method according to claim 3, characterized in that, The step of determining the data to be consumed based on the starting offset and the current offset range of the preset topic in the main database includes: In response to the fact that the number of partitions corresponding to the starting offset is less than the number of partitions corresponding to the minimum endpoint in the current offset interval, all data of the newly added partition in the main database, as well as the incremental data of the original partition, are added to the data that needs to be consumed. In response to the fact that the number of partitions corresponding to the starting offset is greater than the number of partitions corresponding to the minimum endpoint in the current offset interval, all data in all partitions of the main database are used as the data to be consumed.

5. The method according to claim 2, characterized in that, The processing node partitions the consumed data according to topic categories to obtain a temporary dataset, including: The processing node partitions the data according to the topic category; The temporary dataset is generated by parsing the data from the same partition.

6. The method according to claim 1, characterized in that, After the processing node synchronizes the temporary dataset to the synchronization database according to the third data description information, it also includes: In response to the synchronization of the temporary datasets in the same batch, the second data description information is updated according to the attribute information of the temporary datasets.

7. The method according to claim 1, characterized in that, The processing node partitions the consumed data according to topic categories to obtain a temporary dataset, and then further includes: The temporary dataset is then persisted.

8. The method according to claim 1, characterized in that, The processing node synchronizes the temporary dataset to the synchronization database according to the third data description information, including: The processing node maps the data in the temporary dataset to a first type of data according to the third data description information; wherein, the data type of the first type of data is the same as the data type of the data in the synchronous database; The first type of data is synchronized to the synchronization database.

9. An electronic device, characterized in that, It includes a memory and a processor, the memory being used to store program data, the program data being executable by the processor to implement the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, It stores program data and can be executed by a processor to implement the method as described in any one of claims 1-8.

Images