Data synchronization method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a data synchronization method and device, electronic equipment, a storage medium and a program product. The method comprises the following steps: in response to a data synchronization request, determining a first theme module from a first cluster, and determining a first unit from the first theme module; determining a second theme module from a second cluster, and determining a second unit from the second theme module; and acquiring first synchronous data from the first unit, and writing the first synchronous data into the second unit. By means of the method and device, data synchronization of multiple clusters in the unit dimension can be achieved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data synchronization method, device, electronic device, storage medium and program product. Background Art

[0002] With the rapid development of Internet technology, various applications are also continuously evolving and updating. The distributed publish-subscribe messaging system (Kafka) is often selected as the middleware for distributed messaging due to its excellent performance and reliability.

[0003] In order to improve the cross-regional disaster recovery capability of applications and enable applications to quickly switch to data centers deployed in other regions to continue to obtain data when a failure occurs in a certain region, the relevant technology uses the synchronization component provided by Kafka to create mirror data of the topic module (Topic) in the Kafka clusters (Kafka clusters) in different Internet data center computer rooms, thereby realizing message synchronization between multiple Kafka clusters. However, the relevant technology will double the amount of Topic data in the Kafka cluster, increasing the pressure on the Kafka cluster itself. Summary of the invention

[0004] The embodiments of the present application provide a data synchronization method, device, electronic device, storage medium and program product, which can realize data synchronization of multiple clusters in the unit dimension.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present invention provides a data synchronization method, which includes:

[0007] In response to the data synchronization request, determining a first subject module from a first cluster, and determining a first unit from the first subject module;

[0008] determining a second subject module from the second cluster, and determining a second unit from the second subject module;

[0009] Acquire first synchronization data from the first unit, and write the first synchronization data into the second unit.

[0010] The present application provides a data synchronization device, the device comprising:

[0011] a unit search module, configured to determine, in response to a data synchronization request, a first subject module from a first cluster and a first unit from the first subject module;

[0012] The determination module is further used to determine a second theme module from the second cluster, and to determine a second unit from the second theme module;

[0013] A data synchronization module is used to obtain first synchronization data from the first unit and write the first synchronization data into the second unit.

[0014] An embodiment of the present application provides an electronic device, the electronic device comprising:

[0015] A memory for storing computer executable instructions;

[0016] The processor is used to implement the data synchronization method provided in the embodiment of the present application when executing the computer executable instructions stored in the memory.

[0017] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the data synchronization method provided in the embodiment of the present application when executed by a processor.

[0018] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, the data synchronization method provided in the embodiment of the present application is implemented.

[0019] The embodiments of the present application have the following beneficial effects:

[0020] Determine a first theme module from the first cluster, and determine a first unit from the first theme module, determine a second theme module from the second cluster, and determine a second unit from the second theme module, synchronize the first synchronization data from the first unit to the second unit, and achieve fine-grained data synchronization between clusters through data synchronization between units. Compared with the solution of fully synchronizing data in a cluster in the related technology, the data synchronization method provided in the embodiment of the present application only performs data synchronization on the first unit in the first cluster, reducing unnecessary data storage requirements, streamlining the storage resources of each cluster, and improving the efficiency of data synchronization between clusters. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a structural diagram of the data synchronization system architecture provided by an embodiment of the present application;

[0022] Figure 2 is a structural diagram of an electronic device for data synchronization provided by an embodiment of the present application;

[0023] Figure 3 This is a first flow chart of the data synchronization method provided by an embodiment of the present application;

[0024] Figure 4 It is a second flow diagram of the data synchronization method provided in an embodiment of the present application;

[0025] Figure 5 It is a third flow chart of the data synchronization method provided in an embodiment of the present application;

[0026] Figure 6 is a fourth flow chart of the data synchronization method provided in an embodiment of the present application;

[0027] Figure 7 It is a first principle schematic diagram of the data synchronization method provided in an embodiment of the present application;

[0028] Figure 8 It is a second principle schematic diagram of the data synchronization method provided in an embodiment of the present application;

[0029] Fig. 9 is a third principle schematic diagram of the data synchronization method provided in an embodiment of the present application;

[0030] Fig.10 This is a fourth principle diagram of the data synchronization method provided in an embodiment of the present application.

[0031] It should be pointed out that the above-mentioned "first" and "second" are only used to distinguish different solutions, and do not represent the degree of superiority or inferiority of the solutions or the priority in the implementation process. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0033] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0034] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0035] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0036] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0037] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0038] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0039] 1) Kafka cluster: A Kafka cluster consists of multiple Kafka servers. Each Kafka server plays the role of producer, consumer, and agent, and is configured to communicate with each other. Kafka is a distributed publish-subscribe messaging system for building real-time data pipelines and streaming applications. Kafka servers are also called Kafka nodes, which are responsible for storing data, processing data streams, and maintaining the stability and reliability of the cluster.

[0040] 2) Producer: The producer is the publisher of messages and is used to send messages to one or more topic modules in the Kafka cluster.

[0041] 3) Consumer: A consumer is a message receiver that subscribes to one or more topic modules from the Kafka cluster and consumes messages from the subscribed topic modules.

[0042] 4) Topic module: A topic module is a logical concept used to classify data in a Kafka cluster according to specific topics or application areas. For example, the log data topic module is used to process log-type data in the cluster, and the user behavior data topic module is used to process user behavior-related data in the cluster.

[0043] 5) Unit: Unit is a logical concept, which is a further logical refinement of the subject module. A subject module includes at least one unit. The working logics of the units are independent of each other, that is, each unit can independently and completely perform data synchronization and storage tasks, and each unit can be tested and modified independently of other units, which helps to isolate system failures and minimize the impact on other units when modifying or upgrading a unit.

[0044] 6) Partition: Partition is the basic building block of data processing and storage in Kafka cluster. A unit can include at least one partition. Each partition is an ordered, immutable message queue, in which messages are sorted in the order of sending, and each message has a unique offset to uniquely identify its position in the partition.

[0045] 7) Multi-Active Disaster Recovery: It is an IT system architecture and strategy that ensures that critical business systems can continue to operate under any circumstances by deploying active system instances in multiple geographical locations.

[0046] 8) Internet Data Center (IDC): A building or facility specifically designed to house computer systems, network equipment, data storage equipment, and related equipment. IDC typically provides power, cooling, security, network connectivity, and other critical infrastructure services to ensure the safe and efficient operation of the equipment.

[0047] The embodiments of the present application provide a data synchronization method, device, electronic device, computer-readable storage medium, and computer program product, which can realize data synchronization of multiple clusters in the unit dimension. The following describes an exemplary application of the electronic device provided by the embodiment of the present application. The electronic device provided by the embodiment of the present application can be implemented as various types of terminals such as laptop computers, tablet computers, desktop computers, set-top boxes, smart phones, smart speakers, smart watches, smart TVs, and vehicle-mounted terminals, and can also be implemented as servers. Below, an exemplary application when the electronic device is implemented as a server will be described.

[0048] See also Figure 1 , Figure 1 It is a structural diagram of the data synchronization system architecture provided in an embodiment of the present application. In the data synchronization system 10, in order to support a data synchronization application, cluster 200-1 and cluster 200-2 are connected to server 100 through network 300, wherein network 300 can be a wide area network or a local area network, or a combination of the two.

[0049] In response to the data synchronization request, server 100 determines a first theme module from cluster 200-1 (representing the first cluster), and determines a first unit from the first theme module, determines a second theme module from cluster 200-2 (representing the second cluster), and determines a second unit from the second theme module; obtains first synchronization data from the first unit, and writes the first synchronization data into the second unit.

[0050] In some embodiments, clusters 200-1 and clusters 200-2 or server 100 can implement the data synchronization method provided in the embodiments of the present application by running a computer program. For example, the computer program can be a native program or software module in the operating system; it can be a native application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as an instant messaging application; it can also be a small program that can be embedded in any APP, that is, a program that can be run only by downloading it to a browser environment. In short, the above-mentioned computer program can be an application, module or plug-in in any form.

[0051] In some embodiments, cluster 200-1 and cluster 200-2 can be implemented as clusters of various types of electronic devices, such as laptop computers, tablet computers, desktop computers, set-top boxes, mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices, in-vehicle devices), etc.

[0052] In some embodiments, the server 100 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. Clusters 200-1 and clusters 200-2 and server 100 may be directly or indirectly connected via wired or wireless communications, which is not limited in the embodiments of the present application.

[0053] In some embodiments, multiple servers may form a blockchain, and server 100 is a node on the blockchain. Information connections may exist between each node in the blockchain, and information may be transmitted between nodes through the above information connections. Among them, the data related to the data synchronization method provided in the embodiment of the present application may be stored on the blockchain.

[0054] The structure of the electronic device for data synchronization provided by the embodiment of the present application is described below. Figure 2 , Figure 2 is a schematic diagram of the structure of an electronic device for data synchronization provided in an embodiment of the present application, taking the electronic device 500 as a server as an example, Figure 2 The electronic device 500 for data synchronization shown includes: at least one processor 510, a memory 550 and at least one network interface 520. The various components in the electronic device 500 are coupled together via a bus system 540. It is understood that the bus system 540 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 540 is not described in detail. Figure 2 Various buses are labeled as bus system 540 .

[0055] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0056] The memory 550 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550 described in the embodiment of the present application is intended to include any suitable type of memory. The memory 550 optionally includes one or more storage devices that are physically far away from the processor 510.

[0057] In some embodiments, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0058] Operating system 551, including system programs for processing various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0059] A network communication module 552, for reaching other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB);

[0060] In some embodiments, the data synchronization device provided in the embodiments of the present application can be implemented in software. For example, it can be the data synchronization service in the server mentioned above. Of course, it is not limited to this. The data synchronization device provided in the embodiments of the present application can be provided as various software embodiments, including various forms including applications, software, software modules, scripts or codes.

[0061] Figure 2 A data synchronization device 554 stored in a memory 550 is shown, which may be software in the form of a program and a plug-in, such as a data synchronization plug-in, and includes a series of modules, including a unit search module 5541, a data synchronization module 5542, and a unit division module 5543. These modules are logical, and therefore may be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0062] In other embodiments, the data synchronization device provided in the embodiments of the present application can be implemented in hardware. As an example, the data synchronization device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the data synchronization method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (Application Specific Integrated Circuit, ASIC), digital signal processors (Digital Signal Processor, DSP), programmable logic devices (Programmable Logic Device, PLD), complex programmable logic devices (Complex Programmable Logic Device, CPLD), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other electronic components.

[0063] See also Figure 3 , Figure 3 This is a first flow chart of the data synchronization method provided by the embodiment of the present application. Figure 3 The steps shown are explained, Figure 3 The execution subject of the step is the server.

[0064] In step 101, in response to a data synchronization request, a first subject module is determined from a first cluster, and a first unit is determined from the first subject module.

[0065] As an example, a data synchronization request is used to request synchronization of data in a first cluster to a second cluster. In response to the data synchronization request, synchronization configuration information is obtained, the synchronization configuration information including a first path of a first unit. Based on the first path of the synchronization configuration information, a first subject module is determined from the first cluster, and a first unit is determined from the first subject module.

[0066] In some embodiments, see Figure 4 , Figure 4 is a second flow chart of the data synchronization method provided in an embodiment of the present application, Figure 3 The illustrated step 101 may be implemented by following steps 1011 to 1014, which are described in detail below.

[0067] In step 1011 , a first cluster and a second cluster are determined, and the first cluster is monitored.

[0068] As an example, a first cluster and a second cluster are determined, and data update of the first cluster is monitored at the same time, wherein the data update includes data addition, data deletion and data modification, and the following mainly takes data addition as an example for explanation.

[0069] In step 1012, when it is monitored that the first cluster generates first synchronization data, an address of the first synchronization data is determined.

[0070] As an example, when it is monitored that the first cluster generates first synchronization data, the position of the first synchronization data in the first cluster is determined, and the position of the first synchronization data in the first cluster is determined as the address of the first synchronization data, wherein the address of the first synchronization data includes a primary path from the first cluster to the first subject module and a secondary path from the first subject module to the first unit.

[0071] For example, when it is monitored that the first synchronization data (first synchronization data) is generated in unit 1 of the theme module A of cluster 1 (first cluster), the address of the first synchronization data is "Cluster 1. Theme module A. Unit 1", and the first-level path of the address of the first synchronization data is "Cluster 1. Theme module A", that is, the path from cluster 1 (first cluster) to theme module A (first theme module), and the second-level path of the address of the first synchronization data is "Theme module A. Unit 1", that is, the path from theme module A to unit 1 (first unit).

[0072] In step 1013, based on the primary path, a first theme module is determined from the first cluster.

[0073] Continuing with the above example, based on the primary path "Cluster 1. Theme Module A", the theme module A (first theme module) is determined from cluster 1 (first cluster).

[0074] In step 1014, a first unit is determined from the first subject module based on the secondary path.

[0075] Continuing with the above example, based on the secondary path "Subject Module A. Unit 1", Unit 1 (first unit) is determined from Subject Module A (first Subject Module).

[0076] In step 102, a second subject module is determined from the second cluster, and a second unit is determined from the second subject module.

[0077] As an example, in response to a data synchronization request, synchronization configuration information is obtained, the synchronization configuration information including a first path of a first unit and a second path of a second unit, based on the second path of the synchronization configuration information, a second subject module is determined from a second cluster, and a second unit is determined from the second subject module.

[0078] In some embodiments, a first cluster and a second cluster are determined, and the first cluster is monitored. When the first cluster is monitored to generate first synchronization data, Figure 3 The illustrated step 102 may be implemented by the following steps: determining a second theme module from a plurality of theme modules included in the second cluster; and determining a second unit from a plurality of units included in the second theme module.

[0079] As an example, the second theme module is randomly determined from the plurality of theme modules included in the second cluster, and the second unit is randomly determined from the plurality of units included in the second theme module.

[0080] In some embodiments, the third cluster is any one of the first cluster and the second cluster. Figure 3 The step 101 or step 102 shown can be implemented by the following steps: acquiring synchronization configuration information; the primary path is used to determine the third cluster and determine the subject module from the third cluster; the secondary path is used to determine the unit from the subject module.

[0081] As an example, in response to a data synchronization request, synchronization configuration information is obtained, the synchronization configuration information is used to indicate that the data in the first unit is synchronized to the second unit, and the synchronization configuration information includes a first path of the first unit and a second path of the second unit, the first path includes a primary path from the first cluster to the first subject module, and a secondary path from the first subject module to the first unit, and the second path includes a primary path from the second cluster to the second subject module, and a secondary path from the second subject module to the second unit, wherein the primary path of the first path is used to determine the first cluster and determine the subject module from the first cluster, and the secondary path of the first path is used to determine the unit from the subject module. The primary path of the second path is used to determine the second cluster and determine the subject module from the second cluster, and the secondary path of the second path is used to determine the unit from the subject module.

[0082] For example, in response to a data synchronization request, the synchronization configuration information "first path = cluster 1. subject module A. unit 1; second path = cluster 2. subject module A. unit 2" carried in the data synchronization request is obtained, the first-level path of the first path is "cluster 1. subject module A", the first cluster is determined to be cluster 1, and the first subject module is determined to be subject module A from cluster 1, the second-level path of the first path is "subject module A. unit 1", and the first unit is determined to be unit 1 from subject module A; the second path is "cluster 2. subject module A. unit 2", the first-level path of the second path is "cluster 2. subject module A", based on the first-level path of the second path, subject module A is determined from cluster 2, the second-level path of the second path is "subject module A. unit 2", and based on the second-level path of the second path, unit 2 is determined from subject module A.

[0083] In some embodiments, the third cluster is any one of the first cluster and the second cluster. Figure 3 Before the steps 101 and 102 shown, the following steps may be performed: monitoring the network quality index of the third cluster; when the network quality index of the third cluster is greater than the index threshold, determining to execute the processing of determining the theme module from the third cluster and determining the unit from the theme module.

[0084] As an example, the network quality indicators of the first cluster are monitored in real time through a network monitoring tool. When it is monitored that the network quality indicators of the first cluster are greater than a first indicator threshold (indicating that the current network quality of the first cluster is good), a first theme module is determined from the first cluster, and a first unit is determined from the first theme module; or the network quality indicators of the second cluster are monitored in real time through a network monitoring tool. When it is monitored that the network quality indicators of the second cluster are greater than a second indicator threshold, a second theme module is determined from the second cluster, and a second unit is determined from the second theme module, wherein the first indicator threshold can be set according to actual business needs, and the monitoring of the network quality indicators of the third cluster can be achieved through open source tools such as NetDoctor, Ganglia, Nagios, Zabbix, or through professional network performance management solutions, which are not limited here.

[0085] In step 103, first synchronization data is obtained from the first unit, and the first synchronization data is written into the second unit.

[0086] As an example, based on the synchronization configuration information, a data synchronization task is created, where the data synchronization task is used to instruct the consumer to obtain the first synchronization data from the first unit and the producer to write the first synchronization data into the second unit.

[0087] For example, see Fig. 9 , Fig. 9 This is a third principle diagram of the data synchronization method provided in an embodiment of the present application. Data synchronization path 1 includes "first path = cluster 1. Subject module A. Unit 1; second path = cluster 2. Subject module A. Unit 2". Data synchronization task 1 is created based on data synchronization path 1. The consumer of data synchronization task 1 will obtain the first synchronization data from unit 1 of subject module A of cluster 1 by consumption, and the producer of data synchronization task 1 will write the first synchronization data into unit 2 in subject module A of cluster 2 by production.

[0088] The embodiment of the present application realizes data synchronization of the unit dimension between clusters, and at the same time realizes finer-grained control over data synchronization tasks between clusters. While simplifying the architecture design of the subscription message system and reducing operation and maintenance costs, it also improves the efficiency of data synchronization between clusters.

[0089] In some embodiments, Figure 3 The illustrated step 103 may be implemented by the following steps: acquiring synchronization position information; acquiring first synchronization data from the first unit based on the synchronization position information; and updating the synchronization position information in response to receiving confirmation information that the first synchronization data is written to the second unit.

[0090] As an example, the confirmation information is sent by the second cluster, indicating that the first synchronization data has been successfully written into the second unit, and the synchronization site information is obtained locally, wherein the synchronization site information is used to record the site information (i.e., the offset of the partition in the first unit during the last data synchronization), and based on the synchronization site information, the site information of the first synchronization data is determined, for example, the site information recorded in the synchronization site information is added by 1 to obtain the site information of the first synchronization data, and then based on the site information of the first synchronization data, the first synchronization data is obtained from the partition in the first unit, and after the first synchronization data is successfully written into the second unit, in response to receiving the confirmation information that the first synchronization data is written into the second unit, the synchronization site information in the built-in subject module is updated to the site information of the first synchronization data.

[0091] For example, see Fig.10 , Fig.10 This is a fourth principle diagram of the data synchronization method provided in an embodiment of the present application. The consumer obtains the synchronization site information of the current unit from unit 1 of cluster 1, and obtains the first synchronization data based on the synchronization site information. After the producer successfully writes the first synchronization data into unit 2 of cluster 2, the site information of the first synchronization data is written into a local built-in theme module for recording some status information during the synchronization process.

[0092] In an embodiment of the present application, by synchronizing the location information, it can be ensured that each synchronization starts from the correct position, avoiding duplicate data synchronization and data omissions. At the same time, after the synchronization task fails, the location information can be used to quickly restore data synchronization between clusters, reducing the fault recovery time and thereby improving the reliability of the subscription message system.

[0093] In some embodiments, the third theme module includes multiple partitions, and the third theme module is any one of the first theme module and the second theme module. Figure 3 Before step 101 shown, the following steps may also be performed: performing unit division processing on the multiple partitions to obtain multiple units included in the third subject module, each unit including at least one partition.

[0094] As an example, the third theme module includes multiple third partitions, each third partition is an ordered, immutable queue, and the multiple third partitions are divided into multiple units, so that the third theme module includes multiple units, and each unit of the third theme module includes at least one partition, wherein the third unit is any unit in the third theme module.

[0095] For example, see Figure 7 , Figure 7 1 is a first principle diagram of the data synchronization method provided by the embodiment of the present application. The subject module A includes partition 1, partition 2, partition 3, partition 4, partition 5 and partition 6. The above 6 partitions are divided into units to obtain unit 1 and unit 2. Unit 1 includes partition 1, partition 2 and partition 3, and unit 2 includes partition 4, partition 5 and partition 6. See Figure 8 , Figure 8 This is a second principle diagram of the data synchronization method provided in an embodiment of the present application. The subject module B includes partition 1, partition 2, partition 3, partition 4, partition 5, partition 6, partition 7, partition 8 and partition 9. The above 9 partitions are divided into units to obtain 3 units.

[0096] In the embodiment of the present application, a logical concept of a unit is introduced under the subject module within the cluster, and the partitions under the same subject module are grouped into units, thereby realizing the storage and synchronization of data with units as the dimension.

[0097] In some embodiments, the first cluster and the second cluster are deployed in a subscription message system, the subscription message system includes multiple clusters, and the first number of units included in the third theme module matches the second number of clusters included in the subscription message system; the first unit is used to store data generated by the first cluster, and the third unit included in the first theme module is used to store data synchronized from the third cluster included in the subscription message system, and the third cluster is different from the first cluster; the second unit is used to store data generated by the second cluster, and the fourth unit included in the second theme module is used to store data synchronized from the fourth cluster included in the subscription message system, and the fourth cluster is different from the second cluster.

[0098] As an example, a subscription message system is a distributed messaging system, including a second number of clusters deployed in multiple Internet data centers, performing unit division processing on multiple partitions of the third theme module, dividing the multiple partitions in the third theme module into a first number of multiple units, the first number and the second number being consistent.

[0099] The units included in the topic module correspond one-to-one to the clusters. The first topic module includes a first unit and a third unit. The first unit is used to store data produced by the first cluster, and the third unit is used to store data synchronized from the third cluster, wherein the third unit is any unit in the first topic module except the first unit, and the third cluster is any cluster in the subscription message system except the first cluster; the second topic module includes a second unit and a fourth unit. The second unit is used to store data produced by the second cluster, and the fourth unit is used to store data synchronized from the fourth cluster, wherein the fourth unit is any unit in the second topic module except the second unit, and the fourth cluster is any cluster in the subscription message system except the second cluster.

[0100] In an embodiment of the present application, the subscription message system can help achieve high availability and disaster recovery capabilities of the application through multiple data synchronized Kafka clusters in a multi-active disaster recovery scenario.

[0101] For example, see Figure 7 In the case where the subscription message system includes two clusters 1 and 2 located in different Internet data centers, the six partitions in the topic module A in cluster 1 (herein, it can be regarded as the first cluster) are divided into units to obtain two units (such as unit 1 and unit 2 in the topic module A). Figure 8 , when the subscription message system includes three clusters 1, cluster 2 and cluster 3 located in different Internet data centers, the three partitions in the topic module B in cluster 1 are divided into units to obtain three units (such as unit 1, unit 2 and unit 3 in the topic module B).

[0102] See also Figure 8, unit 1 in the subject module B in cluster 1 is used to store the data produced by cluster 1, and in response to a data synchronization request, the data in unit 1 in the subject module B in cluster 1 is synchronized to unit 2 in the subject module B in cluster 2 and unit 3 in the subject module B in cluster 3; unit 1 in the subject module B in cluster 2 is used to store the data produced by cluster 2, and in response to a data synchronization request, the data in unit 1 in the subject module B in cluster 2 is synchronized to unit 3 in the subject module B in cluster 1 and unit 2 in the subject module B in cluster 3; unit 1 in the subject module B in cluster 3 is used to store the data produced by cluster 3, and in response to a data synchronization request, the data in unit 1 in the subject module B in cluster 3 is synchronized to unit 2 in the subject module B in cluster 1 and unit 3 in the subject module B in cluster 2.

[0103] In an embodiment of the present application, based on the number of clusters of multiple clusters, multiple partitions are divided into units to obtain multiple units with the same number of clusters. The subscription message system can be simply linearly expanded as the number of clusters increases, thereby improving the scalability of the subscription message system; at the same time, based on the one-to-one correspondence between units and clusters, specific monitoring indicators and management strategies can be set for each unit and cluster, thereby reducing the operating costs of the subscription message system.

[0104] In some embodiments, see Figure 5 , Figure 5 This is the third flow chart of the data synchronization method provided in the embodiment of the present application. "Performing unit division processing on multiple partitions to obtain multiple units included in the third theme module" can be achieved through the following steps 201 to 202, which are described in detail below.

[0105] In step 201, based on the data of each partition, the similarity between any two partitions in the plurality of partitions is determined to obtain a plurality of similarities.

[0106] As an example, metadata of multiple partitions in the third theme module are collected, and feature extraction is performed on the collected metadata of the multiple partitions to obtain partition features of the multiple partitions, and similarity processing is performed on the partition features of the multiple partitions to obtain multiple similarities (similarity matrices) between the multiple partitions, wherein the metadata of the partition includes but is not limited to one or more of a partition identifier (ID), a partition size, a partition age, a partition status, and an in-sync replica set status (ISR, In-Sync Replicas). The similarity processing of the multiple partitions may be calculating the cosine similarity between the partition features of the multiple partitions, or calculating the Euclidean distance between the partition features of the multiple partitions, etc., which is not limited here.

[0107] In step 202, based on multiple similarities, multiple partitions are divided into units to obtain multiple units included in the third theme module; if a unit includes multiple partitions, the similarity between any two partitions in the same unit is greater than or equal to a preset threshold.

[0108] As an example, based on multiple similarities, multiple partitions are divided into units, and at least two partitions whose similarity between partitions is greater than or equal to a preset threshold are divided into the same unit. If a unit includes multiple partitions, the similarity between any two partitions in the same unit is greater than or equal to the preset threshold. The unit division process for multiple partitions can be based on a clustering algorithm, and the multiple partitions are clustered. The clustering algorithm includes a K-means algorithm, a hierarchical clustering algorithm, etc., which are not limited here.

[0109] For example, when the third theme module includes 6 partitions, that is, the third theme module includes {partition 1, partition 2...partition 6}, metadata of partitions 1 to 6 are collected respectively, and the collected metadata include {identity identifier, partition size, partition age}, feature extraction is performed on the collected metadata of partitions 1 to 6 respectively to obtain partition features of partitions 1 to 6 {partition feature 1, partition feature 2...partition feature 6}, the cosine similarity between each two of the partition features of partitions 1 to 6 is calculated to obtain a similarity set between partitions 1 to 6, and the similarity set between partitions 1 to 6 is clustered by the K-means algorithm to obtain two units included in the third theme module: unit 1 {partition 1, partition 2, partition 3} and unit 2 {partition 4, partition 5, partition 6}.

[0110] In an embodiment of the present application, similar partitions are divided into the same unit through similarity processing and clustering processing, so as to realize centralized management of units including similar partitions. For example, the same security policy, access policy and data encryption policy are implemented on all partitions of the same unit.

[0111] In some embodiments, see Figure 6 , Figure 6 This is the fourth process diagram of the data synchronization method provided in the embodiment of the present application. "Performing unit division processing on multiple partitions to obtain multiple units included in the third theme module" can be achieved through the following steps 301 to 302, which are described in detail below.

[0112] In step 301, the priority of each partition is determined based on the data of each partition.

[0113] As an example, the priority of each partition is determined according to the timeliness of data or the type of data in multiple partitions in the third theme module. For example, a partition with a higher priority is used to process data with a higher timeliness, and a partition with a lower priority is used to process data with a lower timeliness, or a partition with a higher priority is used to process data of a first data type, and a partition with a lower priority is used to process data of a second data type. For example, a partition with a higher priority is used to process financial data, and a partition with a lower priority is used to process log messages. The embodiments of the present application are not limited to determining the priority of partitions by timeliness or the type of data, and the priority can be set according to actual needs.

[0114] In step 302, based on the priority of each partition, multiple partitions are divided into units to obtain multiple units included in the third theme module; if a unit includes multiple partitions, the partitions in the same unit have the same priority.

[0115] As an example, based on the priority of each partition, partitions of the same priority are divided into the same unit in the third theme module, and multiple units included in the third theme module are obtained, where if a unit includes multiple partitions, the partitions belonging to the same unit have the same priority.

[0116] In an embodiment of the present application, partitions with similar priorities are divided into the same unit, more memory and cache space can be allocated to the units corresponding to the high-priority partitions, and data in the units corresponding to the high-priority partitions can be read preferentially, thereby reducing the reading delay of high-priority data or the risk of high-priority data loss and improving the reliability of the system.

[0117] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0118] In the related technology, data synchronization between two Kafka clusters is achieved through the Kafka cluster synchronization component (MirrorMaker) officially provided by Kafka. Kafka officially provides two versions of MirrorMaker, namely the first version of the synchronization component (MM1, MirrorMaker1) and the second version of the synchronization component (MM2, MirrorMaker2). Among them, the first version of the synchronization component MM1 realizes the one-way synchronization of messages (equivalent to the first synchronization data above) from the source Kafka to the target Kafka; based on MM1, the second version of the synchronization component MM2 introduces a backup topic module (Topic) to achieve two-way message synchronization between the two sets of Kafka clusters while avoiding the problem of message loop synchronization.

[0119] However, in the actual application process, the relevant technology still has the following technical problems:

[0120] 1) MM1 can only achieve one-way synchronization of messages between two Kafka clusters, and cannot guarantee the technical effect of the two Kafka clusters backing up each other.

[0121] 2) MM2 realizes the two-way synchronization of messages between two sets of Kafka clusters by creating a backup Topic. That is, when synchronizing messages between different Kafka clusters, it is necessary to create a backup for each Topic in the source Kafka cluster in the synchronization component and change the name of each Topic backup (add a fixed prefix before the Topic name of the source Kafka cluster). When the source Kafka cluster fails, the backup Kafka cluster cannot directly use the backup Topic because the name of the backup Topic is different from that of the original Topic. It is necessary to modify the compatibility of the client's consumption logic. Specifically, in a multi-active scenario, as different Internet Data Centers (IDCs) increase or decrease, it is necessary to continuously modify the client's production and consumption logic to achieve the purpose of multi-center disaster recovery. This makes client access and compatibility difficult for MM2, and the cost of use increases accordingly.

[0122] 3) As mentioned above, MM2 achieves bidirectional synchronization of messages between two sets of Kafka clusters by creating a backup Topic. Therefore, the number of Topics in the entire system will double (original Topic and backup Topic), which will increase the pressure on the Kafka cluster itself to a certain extent and cause certain performance loss to the Kafka cluster itself.

[0123] In order to solve the above problems, the embodiment of the present application proposes a data synchronization method, which realizes two-way message synchronization between multiple Kafka clusters; realizes message synchronization under the same Topic without modifying the production and consumption logic of the client; realizes a synchronization solution for multi-active synchronization control of Kafka based on units, making multi-active synchronization control more refined and more flexible; solves the problem of doubling the usage of Kafka cluster Topic resources in MM2's multi-active synchronization solution, and reduces resource consumption.

[0124] See also Figure 7Under the Topic inside the Kafka cluster, a logical concept of unit is introduced. The partitions under the same Topic in cluster 1 and cluster 2 are divided into unit 1 and unit 2. In the subsequent business processing, the message synchronization business can be grouped and processed according to the unit grouping, and data synchronization and data isolation of multiple Kafka clusters based on the unit dimension can be realized, so that the client connection partitions under the Topic can be managed more flexibly from the unit dimension.

[0125] The following will be combined Figure 7 , taking the message synchronization between two Kafka clusters as an example, the data synchronization method of this embodiment is described:

[0126] First, divide the partitions under a Topic into two units. For example, Topic1 can be divided into two units, which can be expressed as (Topic1{unit1:[partition1, partition2, partition3], unit2:[partition4, partition5, partition6]}).

[0127] Secondly, based on the address of the source Kafka cluster, the address of the target Kafka cluster, the authentication information of the source Kafka cluster, the authentication information of the target Kafka cluster and the synchronization configuration information, the data synchronization component (Connector) is configured, wherein the synchronization configuration information is used to specify the source unit and the destination unit in the message synchronization task. For example, the synchronization configuration information is: source.unit=kafka-cluster1.TopicA.unit1, target.unit=kafka-cluster2.TopicA.unit2, which specifies that the source unit in the message synchronization task is unit unit1 in topic TopicA in Kafka cluster kafka-cluster1, and the target unit is unit unit2 in topic TopicA in Kafka cluster kafka-cluster2. The message in unit1 of TopicA in kafka-cluster1 cluster is the message produced by the client in Internet Data Center 1 (IDC1); and the message in unit2 of TopicA in kafka-cluster1 cluster is the synchronization from the kafka-cluster2 cluster in Internet Data Center 2 (IDC2).

[0128] Again, create multiple parallel message synchronization instances (workers) in the Connector to perform message synchronization tasks. Each worker includes a consumer and a producer. Fig. 9 , create data synchronization task 1 (message synchronization instance 1) and data synchronization task 2 (message synchronization instance 2), each data synchronization task includes a consumer and a producer.

[0129] Next, when performing message synchronization tasks, see Fig. 9 , the consumer in message synchronization instance 1 (data synchronization task 1) will continuously pull the messages to be synchronized (equivalent to the first synchronized data above) of TopicA.unit1 (unit 1 in topic module A) in the source Kafka cluster kafka-cluster1 (cluster 1) in a round-robin manner, and pull the above messages to be synchronized into the local buffer.

[0130] Next, the synchronization location information of the message is synchronously recorded in the built-in Topic of the Connector to ensure that after an exception occurs (such as network interruption, machine failure, etc.), the Connector can also read the latest status information from these built-in Topics and continue the message synchronization task to ensure data consistency and integrity.

[0131] Finally, see Fig. 9 The producer of the corresponding message synchronization instance (data synchronization task 1) pushes the messages to be synchronized in this batch from the local buffer to TopicA.unit2 (unit 2 in topic module A) in kafka-cluster2 (cluster 2).

[0132] During the message synchronization process, by isolating the messages to specific units, the implementation of the multi-active synchronization solution is ensured, so that the two units in the Topic can perform their respective duties: one unit is dedicated to processing the messages produced by the current IDC, and the other unit is responsible for processing the messages synchronized from another IDC by the Connector. When synchronizing the consumer offset (consumer_offset) of two Kafka clusters, different units can also be used to achieve efficient synchronization of information in Kafka clusters in different IDCs.

[0133] See also Figure 8 The difference between message synchronization between three Kafka clusters and message synchronization between two Kafka clusters is:

[0134] First, for the three Kafka cluster message synchronization methods, the partitions under a Topic need to be divided into three units. For example, the topic module B can be divided into three units, which can be expressed as (TopicB{unit1:[partition1, partition2, partition3], unit2:[partition4, partition5, partition6], unit3:[partition7, partition8, partition9]}).

[0135] Secondly, select a unit from TopicB as the main unit to take charge of normal message writing of the current Kafka cluster. The main unit will synchronize the written messages to the other two units in the other two Kafka clusters. For example, unit1 in TopicB of Kafka cluster 1 is used to take charge of normal message writing of the current Kafka cluster, and is also responsible for synchronizing the newly added messages to unit2 in TopicB of cluster 2 and unit3 in TopicB of cluster 3.

[0136] The data synchronization method of the embodiment of the present application introduces a logical concept of a unit under the Topic within the Kafka cluster, and divides the partitions under the same Topic into unit groups, so that messages can be stored in different units; by implementing isolated transmission of unit-dimensional messages at the client level, repeated isolation operations at the Kafka cluster or Topic level can be avoided, thereby simplifying the architecture design, reducing operation and maintenance costs, and improving resource utilization; the message synchronization solution has no additional impact on the Kafka cluster itself, and there is no need to transform the client's production and consumption logic, making it easier for the client to implement a multi-cluster disaster recovery solution.

[0137] The following further describes an exemplary structure of the data synchronization device 554 provided in the embodiment of the present application implemented as a software module. Figure 2 As shown, the software modules stored in the data synchronization device 554 of the memory 550 may include:

[0138] The unit search module 5541 is used to determine the first theme module from the first cluster and the first unit from the first theme module in response to a data synchronization request; determine the second theme module from the second cluster and the second unit from the second theme module.

[0139] The data synchronization module 5542 is used to obtain first synchronization data from the first unit and write the first synchronization data into the second unit.

[0140] In some embodiments, the third theme module includes multiple partitions, and the third theme module is any one of the first theme module and the second theme module. The unit division module 5543 is used to perform unit division processing on the multiple partitions to obtain multiple units included in the third theme module, and each of the units includes at least one partition.

[0141] In some embodiments, the unit division module 5543 is also used to deploy the first cluster and the second cluster in a subscription message system, the subscription message system includes multiple clusters, the first number of units included in the third theme module matches the second number of clusters included in the subscription message system; the first unit is used to store data generated by the first cluster, and the third unit included in the first theme module is used to store data synchronized from the third cluster included in the subscription message system, and the third cluster is different from the first cluster; the second unit is used to store data generated by the second cluster, and the fourth unit included in the second theme module is used to store data synchronized from the fourth cluster included in the subscription message system, and the fourth cluster is different from the second cluster.

[0142] In some embodiments, the unit division module 5543 is also used to determine the similarity between any two of the multiple partitions based on the data of each of the partitions to obtain multiple similarities; based on the multiple similarities, the multiple partitions are subjected to unit division processing to obtain multiple units included in the third theme module; if the unit includes multiple partitions, the similarity between any two partitions belonging to the same unit is greater than or equal to a preset threshold.

[0143] In some embodiments, the unit division module 5543 is also used to determine the priority of each partition based on the data of each partition; based on the priority of each partition, the multiple partitions are subjected to unit division processing to obtain the multiple units included in the third theme module; if the unit includes multiple partitions, the partitions belonging to the same unit have the same priority.

[0144] In some embodiments, the data synchronization module 5542 is also used to obtain synchronization configuration information, wherein the synchronization configuration information is used to indicate that the data in the first unit is synchronized to the second unit, and the synchronization configuration information includes a first path of the first unit and a second path of the second unit, the first path includes a first-level path from the first cluster to the first subject module, and a second-level path from the first subject module to the first unit, and the second path includes a first-level path from the second cluster to the second subject module, and a second-level path from the second subject module to the second unit; the first-level path is used to determine the third cluster and determine the subject module from the third cluster; the second-level path is used to determine the unit from the subject module.

[0145] In some embodiments, the data synchronization module 5542 is also used to determine the first cluster and the second cluster, and monitor the first cluster, wherein the data synchronization request is used to request synchronization of data in the first cluster to the second cluster; when it is monitored that the first cluster generates the first synchronization data, the address of the first synchronization data is determined, wherein the address includes a primary path from the first cluster to the first subject module and a secondary path from the first subject module to the first unit; based on the primary path, the first subject module is determined from the first cluster; based on the secondary path, the first unit is determined from the first subject module.

[0146] In some embodiments, the third cluster is any one of the first cluster and the second cluster, and the unit search module 5541 is also used to monitor the network quality index of the third cluster; when the network quality index of the third cluster is greater than the index threshold, it is determined to execute the processing of determining the theme module from the third cluster and determining the unit from the theme module.

[0147] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the electronic device executes the data synchronization method described above in the embodiment of the present application.

[0148] The present application embodiment provides a computer-readable storage medium, in which computer executable instructions or computer programs are stored. When the computer executable instructions or computer programs are executed by a processor, the processor will execute the data synchronization method provided by the present application embodiment, for example, Figures 3 to 6 The data synchronization method is shown.

[0149] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0150] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0151] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0152] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0153] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A data synchronization method, characterized in that: The method comprises: In response to the data synchronization request, determining a first subject module from a first cluster, and determining a first unit from the first subject module; determining a second subject module from the second cluster, and determining a second unit from the second subject module; Acquire first synchronization data from the first unit, and write the first synchronization data into the second unit.

2. The method according to claim 1, characterized in that The third theme module includes a plurality of partitions, and the third theme module is any one of the first theme module and the second theme module; The method further comprises: The multiple partitions are divided into units to obtain multiple units included in the third theme module, each of which includes at least one partition.

3. The method according to claim 2, characterized in that The first cluster and the second cluster are deployed in a subscription message system, and the subscription message system includes a plurality of clusters; The first number of units included in the third topic module matches the second number of clusters included in the subscription message system; The first unit is used to store data generated by the first cluster, and the third unit included in the first theme module is used to store data synchronized from a third cluster included in the subscription message system, and the third cluster is different from the first cluster; The second unit is used to store data generated by the second cluster, and the fourth unit included in the second theme module is used to store data synchronized from a fourth cluster included in the subscription message system, and the fourth cluster is different from the second cluster.

4. The method according to claim 2, characterized in that: The unit division processing is performed on the multiple partitions to obtain multiple units included in the third theme module, including: Based on the data of each of the partitions, determine the similarity between any two of the multiple partitions to obtain multiple similarities; Based on the multiple similarities, the multiple partitions are divided into units to obtain multiple units included in the third theme module; if the unit includes multiple partitions, the similarity between any two partitions in the same unit is greater than or equal to a preset threshold.

5. The method according to claim 2, characterized in that: The unit division processing is performed on the multiple partitions to obtain multiple units included in the third theme module, including: Determining the priority of each partition based on the data of each partition; Based on the priority of each partition, the multiple partitions are divided into units to obtain multiple units included in the third theme module; if the unit includes multiple partitions, the partitions belonging to the same unit have the same priority.

6. The method according to any one of claims 1 to 5, characterized in that: The third cluster is any one of the first cluster and the second cluster; The method further comprises: Acquire synchronization configuration information, wherein the synchronization configuration information is used to indicate that data in the first unit is synchronized to the second unit, and the synchronization configuration information includes a first path of the first unit and a second path of the second unit, the first path includes a primary path from the first cluster to the first subject module, and a secondary path from the first subject module to the first unit, and the second path includes a primary path from the second cluster to the second subject module, and a secondary path from the second subject module to the second unit; The primary path is used to determine the third cluster, and determine the subject module from the third cluster; The secondary pathway is used to identify units from the subject module.

7. The method according to any one of claims 1 to 5, characterized in that: The step of determining a first theme module from the first cluster and determining a first unit from the first theme module comprises: determining the first cluster and the second cluster, and monitoring the first cluster, wherein the data synchronization request is used to request synchronization of data in the first cluster to the second cluster; In the case where it is monitored that the first cluster generates the first synchronization data, determining an address of the first synchronization data, wherein the address includes a primary path from the first cluster to the first subject module and a secondary path from the first subject module to the first unit; Based on the primary path, determining the first subject module from the first cluster; Based on the secondary path, the first unit is determined from the first subject module.

8. The method according to any one of claims 1 to 5, characterized in that: The third cluster is any one of the first cluster and the second cluster; The method further comprises: Monitoring network quality indicators of the third cluster; In a case where the network quality indicator of the third cluster is greater than the indicator threshold, it is determined to perform a process of determining a theme module from the third cluster and determining a unit from the theme module.

9. A data synchronization device, characterized in that: The device comprises: a unit search module, configured to determine, in response to a data synchronization request, a first subject module from a first cluster and a first unit from the first subject module; The unit search module is further used to determine a second theme module from the second cluster, and determine a second unit from the second theme module; A data synchronization module is used to obtain first synchronization data from the first unit and write the first synchronization data into the second unit.

10. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions; A processor, used to implement the data synchronization method described in any one of claims 1 to 8 when executing the computer executable instructions or computer programs stored in the memory.

11. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the data synchronization method according to any one of claims 1 to 8 is implemented.

12. A computer program product comprising computer executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the data synchronization method according to any one of claims 1 to 8 is implemented.

Images