A data synchronization method and related equipment

Abstract

The present application provides a method for data synchronization and related equipment. The method includes: the follower partition obtains an offset selection strategy; the follower partition determines the target offset in the leader partition of the topic to be synchronized based on the offset selection strategy; the follower partition starts synchronizing the data in the leader partition to the follower partition from the target offset determined by the leader partition. The above method can improve the flexibility of data synchronization, improve the efficiency of data synchronization, and shorten the data synchronization time.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a data synchronization method and related equipment. Background Art

[0002] The distributed publish-subscribe messaging system (Kafka) has good functional characteristics. It provides message persistence through a disk structure, can maintain long-term stable performance, has high throughput, can support millions of messages per second, and can support parallel data loading. Therefore, Kafka is generally used to process massive streaming data generated by application service systems or artificial intelligence (AI) platforms.

[0003] Kafka is the most commonly used distributed message queue system. It usually caches message data on the local disk. The objects of publishing and subscription in Kafka are categories (topics). Each topic represents a type of data. A topic is divided into multiple partitions (partitions), and each partition is an ordered queue. Different partitions of each topic are distributed on different disks or hosts. At the same time, in order to ensure reliability, Kafka has multiple replicas (replicas) for each partition. A master replica will be elected among multiple replicas. The master replica can also be called the leader partition (leader partition), and the remaining replicas are slave replicas, which can also be called follower partitions (follower partitions). Follower partitions will continuously synchronize the latest message data from the leader partition. Consumers (clients that subscribe to messages from topics) and producers (clients that publish messages to topics) interact directly with the leader partition when consuming and producing. When a node where a leader partition is located suddenly fails or loses power, the follower partition corresponding to the partition will elect a new leader partition to ensure service reliability and data security.

[0004] When a node in a Kafka cluster fails and cannot be recovered, or the storage capacity of the Kafka cluster needs to be expanded (i.e., the number of nodes needs to be increased), the replicas in the Kafka cluster need to be migrated to ensure the integrity of the replicas. Currently, the migration of replicas in the Kafka cluster is completed through the migration tool provided by Kafka itself, but this method is not flexible enough, the migration time is long, the efficiency is low, and it will continue to occupy central processing unit (CPU) resources and network bandwidth resources, affecting the normal operation of the business. Summary of the invention

[0005] The embodiment of the present invention discloses a data synchronization method and related equipment, wherein a follower partition can flexibly select data to be synchronized from a leader partition, thereby improving data synchronization efficiency, shortening data synchronization time, and reducing impact on services.

[0006] In a first aspect, the present application provides a method for data synchronization, comprising: a follower partition followerpartition obtains an offset selection strategy; the follower partition determines a target offset in a leader partition of a topic to be synchronized based on the offset selection strategy; the followerpartition synchronizes the data in the leader partition to the follower partition starting from the determined target offset of the leader partition.

[0007] In an embodiment of the present application, the follower partition can flexibly select the target offset to be synchronized as needed, and after determining the target offset, synchronize the data in the leader partition to the follower partition starting from the target offset determined in the leader partition, thereby improving data synchronization efficiency, shortening data synchronization time, and expanding applicable scenarios.

[0008] In a possible implementation, the follower partition sends a query request to the leader partition, where the query request is used to query the offset in the leader partition at the current moment; the follower partition calculates the difference between the offset in the leader partition at the current moment and the offset in the follower partition, and determines to synchronize the data in the leader partition to the follower partition when the difference is greater than a preset threshold.

[0009] In the embodiment of the present application, before performing data synchronization, the follower partition needs to query the offset in the leader partition at the current moment, and then calculate the difference between the offset in the leader partition and the offset in the follower partition, and perform data synchronization when the difference is greater than a preset threshold. In this way, frequent data synchronization can be avoided, and data synchronization is performed only when conditions are met, thereby improving system efficiency.

[0010] In a possible implementation, the offset selection strategy is an offset adaptive determination strategy, the follower partition obtains the average data writing speed of the leader partition; the follower partition calculates the target offset according to the average data writing speed of the leader partition.

[0011] In an embodiment of the present application, when the offset selection strategy of the follower partition is the offset adaptive determination strategy, it is necessary to obtain the average data writing speed of the leader partition, so as to calculate the target offset.

[0012] In a possible implementation, the offset selection strategy is an offset adaptive determination strategy, the follower partition obtains the starting offset of the leader partition and the average data writing speed of the leader partition; the follower partition calculates the target offset based on the starting offset of the leader partition and the average data writing speed of the leader partition.

[0013] In an embodiment of the present application, when the offset selection strategy is the offset adaptive determination strategy, the follower partition needs to obtain the starting offset and the average data writing speed of the leader partition, so as to ensure that the calculated target offset is more accurate.

[0014] In a possible implementation, when the average data write speed of the leader partition is greater than the data synchronization transmission speed of the leader partition, the target offset is the maximum offset of the leader partition at the current moment; when the average data write speed of the leader partition is less than or equal to the data synchronization transmission speed, the target offset is the starting offset of the leader partition.

[0015] In the embodiment of the present application, the target offset is determined by comparing the relationship between the average data writing speed and the data synchronization transmission speed of the leader partition, so that the determined target offset can be ensured to be more reasonable.

[0016] In a possible implementation, when the average data write speed of the leader partition is greater than the data synchronization transmission speed of the leader partition, the target offset is the maximum offset of the leader partition at the current moment; when the average data write speed of the leader partition is less than or equal to the data synchronization transmission speed, and the synchronization time required for the current total amount of data of the leader partition is less than a preset duration, the target offset is the starting offset of the leader partition.

[0017] In an embodiment of the present application, the relationship between the average data writing speed of the leader partition and the data synchronization transmission speed, as well as the relationship between the synchronization time required for the current total data amount of the leader partition and the preset duration are considered simultaneously to determine the target offset, which can further improve the rationality and accuracy of the determined target offset.

[0018] In a possible implementation, the offset selection strategy is a selection strategy based on data value, the follower partition obtains the consumption offsets of multiple consumer groups, and the follower partition determines the minimum consumption offset as the target offset; wherein the consumer group is used to consume the data in the topic to be synchronized, each consumer group includes at least one consumer, and the consumption offset is used to indicate the amount of data in the leader partition consumed by the consumer group.

[0019] In an embodiment of the present application, when the offset selection strategy of the follower partition is a selection strategy based on data value, it is necessary to obtain the consumption offsets of multiple consumer groups that consume data in the topic to be synchronized, so as to determine the target offset, that is, to determine the minimum consumption offset as the target offset.

[0020] In the second aspect, the present application provides a data synchronization device, including: an acquisition unit, used to obtain an offset selection strategy; a processing unit, used to determine the target offset in the leader partition of the topic to be synchronized based on the offset selection strategy; a synchronization unit, used to synchronize the data in the leader partition starting from the determined target offset of the leader partition.

[0021] In a possible implementation, the data synchronization device also includes a sending unit, which is used to send a query request to the leader partition, and the query request is used to query the offset in the leader partition at the current moment; the processing unit is also used to calculate the difference between the offset in the leader partition at the current moment and the offset in the data synchronization device, and determine to synchronize the data in the leader partition when the difference is greater than a preset threshold.

[0022] In a possible implementation, the offset selection strategy is an offset adaptive determination strategy, and the processing unit is specifically used to: obtain an average data writing speed of the leader partition; and calculate the target offset according to the average data writing speed of the leader partition.

[0023] In a possible implementation, the offset selection strategy is an offset adaptive determination strategy, and the processing unit is specifically used to: obtain the starting offset of the leader partition and the average data writing speed of the leader partition; and calculate the target offset based on the starting offset of the leader partition and the average data writing speed of the leader partition.

[0024] In a possible implementation, when the average data write speed of the leader partition is greater than the data synchronization transmission speed of the leader partition, the target offset is the maximum offset of the leader partition at the current moment; when the average data write speed of the leader partition is less than or equal to the data synchronization transmission speed, the target offset is the starting offset of the leader partition.

[0025] In a possible implementation, when the average data write speed of the leader partition is greater than the data synchronization transmission speed of the leader partition, the target offset is the maximum offset of the leader partition at the current moment; when the average data write speed of the leader partition is less than or equal to the data synchronization transmission speed, and the synchronization time required for the current total amount of data of the leader partition is less than a preset duration, the target offset is the starting offset of the leader partition.

[0026] In a possible implementation, the offset selection strategy is a selection strategy based on data value, and the processing unit is specifically used to: obtain consumption offsets of multiple consumer groups, and determine the minimum consumption offset as the target offset; wherein the consumer group is used to consume the data in the topic to be synchronized, each consumer group includes at least one consumer, and the consumption offset is used to indicate the amount of data in the leaderpartition consumed by the consumer group.

[0027] In a third aspect, the present application provides a computing device, comprising a processor and a memory, wherein the processor executes computer instructions stored in the memory, so that the computing device executes the above-mentioned first aspect and a method combined with any one of the implementation modes of the above-mentioned first aspect.

[0028] In a fourth aspect, the present application provides a computer storage medium storing a computer program, wherein the computer program, when executed by a computing device, implements the first aspect and a method combined with any one of the implementation methods of the first aspect.

[0029] In a fifth aspect, the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computing device, the computing device can execute the above-mentioned first aspect and a method combined with any one of the implementation methods of the above-mentioned first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0031] Figure 1A It is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0032] Figure 1B is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0033] Figure 1C is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0034] Figure 2 is a schematic diagram of data synchronization provided by an embodiment of the present application;

[0035] Figure 3 It is a flowchart of a data synchronization method provided in an embodiment of the present application;

[0036] Figure 4 It is a structural diagram of a data synchronization device provided in an embodiment of the present application;

[0037] Figure 5 It is a structural diagram of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the present application are described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0039] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0040] First, some of the terms and related technologies involved in this application are explained in conjunction with the accompanying drawings to facilitate understanding by those skilled in the art.

[0041] Kafka is a distributed, partitioned, multi-copy, zookeeper-coordinated distributed messaging system that can process large amounts of data in real time to meet various demand scenarios: such as hadoop-based batch processing systems, low-latency real-time systems, storm / spark streaming engines, access logs, messaging services, etc. Kafka's features include: high throughput, low latency, strong scalability, good persistence, high fault tolerance, support for high concurrency, and is widely used in log collection, user activity tracking, etc.

[0042] Zookeeper is a distributed, open source distributed application coordination service and an important component of Hadoop and Hbase. It is a software that provides consistency services for distributed applications, including configuration maintenance, domain name services, distributed synchronization, group services, etc. The goal of Zookeeper is to encapsulate complex and error-prone key services and provide users with simple and easy-to-use interfaces and efficient and stable systems.

[0043] Figure 1AA possible application scenario of an embodiment of the present application is shown. In this application scenario, node 110, node 120 and node 130 are node devices in the kafka cluster 100, such as servers, etc. Different partitions of different topics are stored on each node. For example, node 110 stores a leader partition 111 of the first partition, a follower partition 112 of the first partition, and a follower partition 113 of the second partition; node 120 stores a follower partition 121 of the first partition, a follower partition 122 of the second partition, and a leader partition 123 of the third partition; node 130 stores two follower partitions 131 and 132 of the third partition, and a leader partition of the second partition. The first partition, the second partition and the third partition can be different partitions under the same topic, or different partitions under different topics, and each partition is stored in three copies. At this time, the kafka cluster 100 needs to be expanded, that is, a new node 140 is added. At this time, there is no data stored on the new node 140, and the data distribution of the entire kafka cluster 100 is not balanced. It is necessary to use a migration tool to migrate part of the partitions stored on other nodes to the new node 140 to ensure balanced data distribution of the entire cluster. For example, the replica migration tool kafka-reassign-partitions provided by kafka is used to migrate follower partition 112 in node 110 to node 140, and to migrate follower partition 131 in node 130 to node 140.

[0044] Figure 1B Another possible application scenario is shown. Figure 1BAs shown, the node 110 stores the leader partition 111 of the first partition, a follower partition 112 of the first partition, and a follower partition 113 of the second partition; the node 120 stores a follower partition 121 of the first partition, a follower partition 122 of the second partition, and the leader partition 123 of the third partition; the node 130 stores two follower partitions 131 and 132 of the third partition and the leader partition of the second partition. At this time, the node 120 fails during operation, and the data stored in the node 120 needs to be migrated to the nodes 110 and 130 to achieve balanced data distribution of the entire cluster 100. The migration tool is used to migrate the follower partition 121 and the leader partition 123 in the node 120 to the node 110, and the follower partition 122 in the node 120 to the node 130.

[0045] Figure 1C Another possible application scenario is shown. Figure 1C As shown, the leader partition 111 of the first partition, a follower partition 112 of the first partition, and a follower partition 113 of the second partition are stored in the node 110; the follower partition 121 of the first partition and the leader partition 122 of the third partition are stored in the node 120; the leader partition 131 of the second partition is stored in the node 130. At this time, the data distribution of the cluster 100 is not balanced, and the migration tool needs to be used to migrate the follower partition 113 stored in the node 110 to the node 130 to achieve a more balanced data distribution of the cluster 100.

[0046] For the above Figure 1A , Figure 1B and Figure 1CIn the application scenarios shown, the migration tool is required to migrate the follower partition on a node to another node. The migration here does not mean moving data directly from one node to another, but pulling data from the leader partition corresponding to the follower partition in a synchronous manner, and finally achieving consistency with the data in the leader partition to complete the purpose of migration. Currently, the replica migration tool kafka-reassign-partitions provided by Kafka is used to complete the migration of follower partition (that is, data synchronization). When a new follower partition is created in a node, the replica synchronization thread will pull data from the leader partition corresponding to the follower partition until the data of the follower partition is synchronized with the data of the leader partition. However, in the above solution, the newly created follower partition will start synchronizing from the starting offset of the leader partition when synchronizing data. When the amount of cached data in the leader partition is large, it will take a lot of time to complete the data synchronization, and the CPU resources and network bandwidth resources of the node will continue to be occupied during the data synchronization process, affecting the normal operation of the business. In addition, in many scenarios, the value of the data cached in Kafka will gradually decrease over time, while the actual business or application is only interested in the latest data. For example, in the application scenario of counting the number of times a target appears within one minute, only the data within the current minute is valuable, while the previous data is of lower value. Therefore, it is meaningless to synchronize a large amount of historical cached data. The follower partition can only be synchronized from the starting offset of the leader partition, and the offset cannot be flexibly specified, that is, it is impossible to synchronize from any offset according to actual needs, and it is impossible to avoid synchronizing worthless historical cached data.

[0047] In order to solve the above problems, the present application provides a data synchronization method and related equipment, which can flexibly select the data to be synchronized from the leader partition during data synchronization, improve data synchronization efficiency, shorten data synchronization time, and reduce the impact on business.

[0048] See also Figure 2 , Figure 2 This is a schematic diagram of data synchronization provided by this application. Figure 2As shown, node 210 deploys the leader partition of a partition, and the leader partition caches the data written by the producer, including the historical written data and the latest written data. Node 220 deploys the follower partition corresponding to the same partition. The follower partition needs to synchronize data from the leader partition in node 210. The follower partition can flexibly select the data to be synchronized. For example, it can start data synchronization from the starting offset of the leader partition, that is, offset 1, and synchronize all data; it can also start synchronization from offset 7 and synchronize only the latest data written by the leader partition; or select any offset (such as offset 5, etc.) to start synchronization.

[0049] Combination Figure 1A , Figure 1B and Figure 1C The application scenarios shown, and Figure 2 See the data synchronization diagram shown in Figure 3 , Figure 3 Schematic diagram of a data synchronization method provided by an embodiment of the present application. Figure 3 As shown, the method includes but is not limited to the following steps:

[0050] S301: The follower partition obtains an offset selection strategy.

[0051] For example, users can access Zookeeper through the World Wide Web (Web) browser or other interfaces (such as RESTful interfaces, etc.) provided by the Kafka cluster, and set the offset selection strategy. The strategy is set in topic units, that is, the offset acquisition strategy is set for all partitions under the same topic. After the setting is completed, the set offset acquisition strategy can be saved as a file in JavaScript object notation (JSON) format, and the file is recorded in the specified directory of Zookeeper.

[0052] In addition, all nodes in the Kafka cluster will elect a management node, which uses the watch mechanism to monitor Zookeeper. Therefore, when the offset acquisition strategy is recorded in the directory specified by Zookeeper, the management node will sense the change in the Zookeeper directory and then send a synchronization request to the relevant node, that is, the node of the follower partition of the topic corresponding to the offset acquisition strategy. After receiving the synchronization request sent by the management node, the relevant node will start the replica synchronization thread. After the replica synchronization thread is started, the relevant node will read the offset acquisition strategy of the relevant topic on Zookeeper.

[0053] In a possible implementation, the follower partition sends a query request to the leader partition, wherein the query request is used to query the offset in the leader partition at the current moment; the follower partition calculates the difference between the maximum offset of the leader partition at the current moment and the maximum offset of the follower partition, and determines to synchronize the data in the leader partition to the follower partition when the difference is greater than a preset threshold.

[0054] Specifically, each log has an offset to uniquely mark a message. The offset value is an 8-byte number used to indicate its position in the partition. Therefore, each message (i.e., data) stored in the leader partition and follower partition corresponds to a unique offset. After starting the replica synchronization thread, the follower partition sends a query request to the leader partition to obtain the maximum offset of the leader partition at the current moment, and then calculates the difference with its own offset. If the difference exceeds the preset threshold, it means that there is a large amount of data in the leader partition that has not been synchronized to the follower partition. At this time, the follower partition needs to synchronize the data in the leader partition to the local. It should be understood that the preset threshold can be set as needed, and this application does not limit this.

[0055] S302: The follower partition determines the target offset in the leader partition of the topic to be synchronized based on the offset selection strategy.

[0056] Specifically, the follower partition needs to determine the offset selection strategy before performing data synchronization. After the offset selection strategy is determined, it needs to further determine the target offset in the leader partition of the topic to be synchronized, that is, the offset from which to start data synchronization.

[0057] In a possible implementation, the offset selection strategy is an offset adaptive determination strategy, and the follower partition obtains the average data writing speed of the leader partition; the follower partition calculates the target offset according to the average data writing speed of the leader partition.

[0058] Specifically, the follower partition obtains the average data writing speed of the leader partition through the interface provided by the Java management extensions (JMX) in the node where the leader partition is located, for example, the average data writing speed of the leader partition in the last hour is obtained through the interface provided by JMX. After obtaining the average data writing speed of the leader partition, the follower partition can calculate the target offset.

[0059] Optionally, after obtaining the average data writing speed of the leader partition, the follower partition can further send a query request to the leader partition to query the starting offset of the leader partition, and then calculate the target offset based on the starting offset of the leader partition obtained from the query and the average data writing speed of the leader partition.

[0060] For example, after obtaining the average data writing speed of the leader partition, the follower partition first determines whether a speed threshold is set. The speed threshold is used to limit the data synchronization transmission speed of the follower partition to avoid occupying too many CPU resources and network bandwidth resources during data synchronization, thereby avoiding affecting the business and causing the business to not run normally. If the speed threshold is set, the speed threshold is used as the data synchronization transmission speed; if the threshold is not set, the consumption speed of the follower partition is used as the data synchronization transmission speed.

[0061] For example, when the follower partition performs data synchronization, its essence is to start another consumption thread in the follower partition. The follower partition consumes the data in the leader partition as a consumer to achieve data synchronization. Therefore, if no speed threshold is set, the consumption speed of the follower partition is the data synchronization transmission speed.

[0062] Then, the follower partition determines the relationship between the data synchronization transmission speed and the average data writing speed of the leader partition. When the average data writing speed of the leader partition is greater than the data synchronization transmission speed, it means that even if the follower partition has been continuously synchronizing data, it cannot completely catch up with the leader partition, that is, the follower partition has been synchronizing the historical data of the leader partition. Therefore, in this case, only the latest data in the leader partition needs to be synchronized. The maximum offset of the leader partition at the current moment is used as the target offset, and the follower partition starts data synchronization from the maximum offset of the leader partition at the current moment. When the average data writing speed of the leader partition is less than or equal to the data synchronization transmission speed, it means that after enough time, the follower partition can definitely catch up with the leader partition, that is, the follower partition can synchronize all the data in the leader partition to the local. In this scenario, further judgment processing is required to determine the final target offset.

[0063] In one possible implementation, the follower partition calculates the time required to synchronize the current total amount of data of the leader partition. Optionally, the follower partition can obtain the current total amount of data while obtaining the average write speed of the leader partition through the interface provided by JMX, or can obtain the current total amount of data of the leader partition through other methods, which is not limited in this application. After obtaining the current total amount of data, the follower partition divides the total amount of data by the data synchronization transmission speed to obtain the synchronization time. When the synchronization time is less than or equal to the preset time, it means that the current total amount of data of the leader partition is not large, and the synchronization process will not cause excessive resource consumption. The follower partition uses the starting offset of the leader partition as the target offset, and the follower partition synchronizes all data in the leader partition to the local. The preset time can be set as needed, for example, it can be set to one hour, which is not limited in this application.

[0064] When the synchronization time is greater than the preset time, the target offset can be calculated according to the following formula 1:

[0065] ((y-y0) * n) / v1=((y1-y)*n) / (v2-v1) Formula 1

[0066] Among them, y0 represents the starting offset of the leader partition, y represents the target offset, y1 represents the maximum offset of the leader partition, n represents the average data size of a message, v1 represents the average data writing speed of the leader partition, and v2 represents the data synchronization transmission speed.

[0067] By further simplifying the above formula 1, the following formula 2 can be obtained:

[0068] y=(y1*v1 + (v2-v1)y0) / v2 Formula 2

[0069] It can be seen that after obtaining the starting offset and maximum offset of the leader partition, the average data writing speed of the leader partition, and the data synchronization transmission speed, the target offset can be calculated and the data in the leader partition can be synchronized starting from the target offset. This can not only avoid long-term occupation of resources such as CPU and network bandwidth during the data synchronization process, which affects the business, but also avoid synchronizing some expired and meaningless historical data, thereby improving synchronization efficiency and shortening synchronization time.

[0070] In another possible implementation, the offset selection strategy is a selection strategy based on data value, the follower partition obtains the consumption offsets of multiple consumer groups, and the follower partition determines the minimum consumption offset as the target offset, wherein the consumer group is used to consume the data in the topic to be synchronized, each consumer group includes at least one consumer, and the consumption offset is used to indicate the amount of data in the leader partition consumed by the consumer group.

[0071] Specifically, each consumer belongs to a specific group. You can specify a group name for each consumer. If you do not specify a group name, it belongs to the default group. When consuming data, the consumer group consumes data in topics. Different consumer groups can consume the same topic. For consumers in a consumer group, consumers in the same group cannot consume the same partition of the same topic, that is, different consumers cannot consume the same partition, but the same consumer can consume multiple partitions.

[0072] Therefore, for the topic to be synchronized, when there are multiple consumer groups consuming the data in the topic at the same time, find the consumer that consumes the leader partition to be synchronized from each consumer group, and obtain the consumer's consumption offset, that is, the amount of data in the leader partition that the consumer has currently consumed. Then compare the obtained consumption offsets and select the smallest consumption offset as the target offset.

[0073] It is easy to understand that using the minimum consumption offset as the target offset can ensure that when the node where the leader partition is located suddenly fails, all consumer groups can still fully consume the data in the topic to avoid affecting the business.

[0074] In yet another possible implementation, the offset selection strategy is an offset specification strategy, and the follower partition uses the specified offset as the target offset.

[0075] Specifically, after the follower partition determines that it needs to synchronize the data in the leader partition to the follower partition, it directly uses the specified offset as the target offset and starts data synchronization from the target offset. The specific value of the specified offset can be set according to actual needs, and this application does not limit this.

[0076] S303: The follower partition starts synchronizing the data in the leader partition to the follower partition from the determined target offset of the leader partition.

[0077] Specifically, after determining the target offset, the follower partition starts a consumer thread to consume the data in the leader partition from the target offset, that is, to synchronize the data in the leader partition to the local.

[0078] It should be understood that steps S301 to S303 involved in the above method embodiment are only schematic descriptions and should not constitute specific limitations. The steps involved can be increased, reduced or combined as needed.

[0079] The method of the embodiment of the present application is described in detail above. In order to facilitate better implementation of the above scheme of the embodiment of the present application, correspondingly, related equipment for cooperating in implementing the above scheme is also provided below.

[0080] See also Figure 4 , Figure 4 Schematic diagram of a data synchronization device provided by an embodiment of the present application. Figure 4 As shown, the data synchronization device 400 includes an acquisition unit 410, a processing unit 420 and a synchronization unit 430.

[0081] The acquisition unit 410 is used to acquire an offset selection strategy.

[0082] Specifically, the acquisition unit 410 is used to execute the aforementioned step S301, and optionally execute the optional methods in the aforementioned steps.

[0083] The processing unit 420 is used to determine the target offset in the leaderpartition of the topic to be synchronized based on the offset selection strategy.

[0084] Specifically, the processing unit 420 is used to execute the aforementioned step S302, and optionally execute the optional method in the aforementioned step.

[0085] The synchronization unit 430 is used to synchronize the data in the leader partition starting from the determined target offset of the leader partition.

[0086] Specifically, the synchronization unit 430 is used to execute the aforementioned step S303, and optionally execute the optional methods in the aforementioned steps.

[0087] In a possible implementation, the data synchronization device 400 also includes a sending unit 440, which is used to send a query request to the leader partition, and the query request is used to query the offset in the leader partition at the current moment; the processing unit 420 is also used to calculate the difference between the offset in the leader partition at the current moment and the offset in the data synchronization device, and determine to synchronize the data in the leader partition when the difference is greater than a preset threshold.

[0088] In a possible implementation, the offset selection strategy is an offset adaptive determination strategy, and the processing unit 420 is specifically used to: obtain an average data writing speed of the leader partition; and calculate the target offset according to the average data writing speed of the leader partition.

[0089] In a possible implementation, the offset selection strategy is an offset adaptive determination strategy, and the processing unit 420 is specifically used to: obtain the starting offset of the leader partition and the average data writing speed of the leader partition; and calculate the target offset based on the starting offset of the leader partition and the average data writing speed of the leader partition.

[0090] In a possible implementation, when the average data write speed of the leader partition is greater than the data synchronization transmission speed of the leader partition, the target offset is the maximum offset of the leader partition at the current moment; when the average data write speed of the leader partition is less than or equal to the data synchronization transmission speed, the target offset is the starting offset of the leader partition.

[0091] In a possible implementation, when the average data write speed of the leader partition is greater than the data synchronization transmission speed of the leader partition, the target offset is the maximum offset of the leader partition at the current moment; when the average data write speed of the leader partition is less than or equal to the data synchronization transmission speed, and the synchronization time required for the current total amount of data of the leader partition is less than a preset duration, the target offset is the starting offset of the leader partition.

[0092] In a possible implementation, the offset selection strategy is a selection strategy based on data value, and the processing unit 420 is specifically used to: obtain consumption offsets of multiple consumer groups, and determine the minimum consumption offset as the target offset; wherein the consumer group is used to consume the data in the topic to be synchronized, each consumer group includes at least one consumer, and the consumption offset is used to indicate the amount of data in the leaderpartition consumed by the consumer group.

[0093] It should be noted that the structure of the above-mentioned data synchronization device and the process of realizing data synchronization using the data synchronization device are only used as an example and should not constitute a specific limitation. The various units in the data synchronization device can be increased, reduced or merged as needed. In addition, the operations and / or functions of the various modules in the data synchronization device are to realize the above-mentioned Figure 3 For the sake of brevity, the corresponding process of the described method will not be repeated here.

[0094] See also Figure 5 , Figure 5 Schematic diagram of a computing device provided in an embodiment of the present application. Figure 5 As shown, the computing device 500 includes: a processor 510, a communication interface 520 and a memory 530, wherein the processor 510, the communication interface 520 and the memory 530 are interconnected via an internal bus 540. It should be understood that the computing device may be a library server.

[0095] The computing device 500 may be Figure 1A-Figure 1C , Figure 2 and Figure 3 The server in which the follower partition is deployed. Figure 1A-Figure 1C , Figure 2 and Figure 3 The functions performed by the server in are actually executed by the processor 510 of the server.

[0096] The processor 510 may be composed of one or more general-purpose processors, such as a central processing unit (CPU), or a combination of a CPU and a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0097] The bus 540 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus 540 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0098] The memory 530 may include a volatile memory, such as a random access memory (RAM); the memory 530 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); the memory 530 may also include a combination of the above types. The program code may be used to implement the functional units shown in the data synchronization device 400, or to implement Figure 3 In the method embodiment shown, the method steps are performed by using the follower partition as the execution subject.

[0099] The present application also provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the program can implement part or all of the steps of any one of the above method embodiments, and implement the above method. Figure 4 The functionality of any functional unit described.

[0100] The present application also provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the program can implement part or all of the steps of any one of the above method embodiments, and implement the above method. Figure 4 The functionality of any functional unit described.

[0101] The embodiment of the present application also provides a computer program product, which, when executed on a computer or processor, enables the computer or processor to perform one or more steps in any of the above methods. If the components of the above-mentioned devices are implemented in the form of software functional units and sold or used as independent products, they can be stored in the computer-readable storage medium.

[0102] In the above embodiments, the description of each embodiment has different emphases. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0103] It should also be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0104] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0105] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0106] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for data synchronization, characterized in that: The method comprises: Follower partition obtains the offset selection strategy; The follower partition determines the target offset in the leader partition of the topic to be synchronized based on the offset selection strategy; The follower partition starts synchronizing the data in the leader partition to the follower partition from the target offset determined by the leader partition; The follower partition determines the offset in the leader partition of the topic to be synchronized based on the offset selection strategy, including: The follower partition obtains the average data writing speed of the leader partition when the offset selection strategy is the offset adaptive determination strategy; The follower partition calculates the target offset according to the average data writing speed of the leader partition; when the average data writing speed of the leader partition is greater than the data synchronization transmission speed of the leader partition, the target offset is the maximum offset of the leader partition at the current moment.

2. The method according to claim 1, characterized in that The method further comprises: The follower partition sends a query request to the leader partition, where the query request is used to query the offset in the leader partition at the current moment; The follower partition calculates the difference between the offset in the leader partition and the offset in the follower partition at the current moment, and determines to synchronize the data in the leader partition to the follower partition when the difference is greater than a preset threshold.

3. The method according to claim 1 or 2, characterized in that The method further includes: the follower partition obtaining a starting offset of the leader partition; The follower partition calculates the target offset according to the average data writing speed of the leader partition, including: the follower partition calculates the target offset according to the starting offset of the leader partition and the average data writing speed of the leader partition.

4. The method according to claim 1, characterized in that When the average data writing speed of the leader partition is less than or equal to the data synchronization transmission speed, the target offset is the starting offset of the leader partition; or when the average data writing speed of the leader partition is less than or equal to the data synchronization transmission speed and the synchronization time required for the current total amount of data of the leader partition is less than the preset duration, the target offset is the starting offset of the leader partition.

5. The method according to claim 3, characterized in that When the average data writing speed of the leader partition is less than or equal to the data synchronization transmission speed, the target offset is the starting offset of the leader partition; or when the average data writing speed of the leader partition is less than or equal to the data synchronization transmission speed and the synchronization time required for the current total amount of data of the leader partition is less than the preset duration, the target offset is the starting offset of the leader partition.

6. The method according to claim 1 or 2, characterized in that: The follower partition determines the offset in the leader partition of the topic to be synchronized based on the offset selection strategy, and further includes: When the offset selection strategy is a selection strategy based on data value, the follower partition obtains consumption offsets of multiple consumer groups, and the follower partition determines the minimum consumption offset as the target offset; The consumer group is used to consume the data in the topic to be synchronized, each consumer group includes at least one consumer, and the consumption offset is used to indicate the amount of data in the leader partition consumed by the consumer group.

7. A data synchronization device, characterized in that: include: An acquisition unit, used to acquire an offset selection strategy; A processing unit, configured to determine a target offset in a leader partition of a topic to be synchronized based on the offset selection strategy; A synchronization unit, configured to synchronize data in the leader partition starting from a determined target offset of the leader partition; The processing unit is specifically used for: When the offset selection strategy is the offset adaptive determination strategy, the follower partition obtains the average data writing speed of the leader partition; The follower partition calculates the target offset according to the average data writing speed of the leader partition; When the average data writing speed of the leader partition is greater than the data synchronization transmission speed of the leader partition, the target offset is the maximum offset of the leader partition at the current moment.

8. The data synchronization device according to claim 7, characterized in that: The data synchronization device further includes a sending unit. The sending unit is used to send a query request to the leader partition, where the query request is used to query the offset in the leader partition at a current moment; The processing unit is further used to calculate the difference between the offset in the leader partition at the current moment and the offset in the data synchronization device, and determine to synchronize the data in the leader partition when the difference is greater than a preset threshold.

9. The data synchronization device according to claim 7 or 8, characterized in that: The processing unit is further specifically used for: Get the starting offset of the leader partition; The follower partition calculates the target offset according to the average data writing speed of the leader partition, including: calculating the target offset according to the starting offset of the leader partition and the average data writing speed of the leader partition.

10. The data synchronization device according to claim 7 or 8, characterized in that: When the average data writing speed of the leader partition is less than or equal to the data synchronization transmission speed, the target offset is the starting offset of the leader partition; or when the average data writing speed of the leader partition is less than or equal to the data synchronization transmission speed and the synchronization time required for the current total amount of data of the leader partition is less than the preset duration, the target offset is the starting offset of the leader partition.

11. The data synchronization device according to claim 9, characterized in that: When the average data writing speed of the leader partition is less than or equal to the data synchronization transmission speed, the target offset is the starting offset of the leader partition; or when the average data writing speed of the leader partition is less than or equal to the data synchronization transmission speed and the synchronization time required for the current total amount of data of the leader partition is less than the preset duration, the target offset is the starting offset of the leader partition.

12. The data synchronization device according to claim 7 or 8, characterized in that: The offset selection strategy is a selection strategy based on data value, and the processing unit is further used to: When the offset selection strategy is a selection strategy based on data value, obtaining consumption offsets of multiple consumer groups, and determining the minimum consumption offset as the target offset; The consumer group is used to consume the data in the topic to be synchronized, each consumer group includes at least one consumer, and the consumption offset is used to indicate the amount of data in the leader partition consumed by the consumer group.

13. A computing device, characterized in that: The computing device comprises a processor and a memory, and the processor executes computer instructions stored in the memory, so that the computing device executes the method according to any one of claims 1 to 6.

14. A computer storage medium, characterized in that: The computer storage medium stores a computer program, and when the computer program is executed by a computing device, the method according to any one of claims 1 to 6 is implemented.

15. A computer program product, comprising computer instructions, and when the computer instructions are executed by a computing device, the computing device can perform the method according to any one of claims 1 to 6.

Images