Message synchronization method, message synchronization system, electronic equipment and storage medium

Abstract

The invention relates to a message synchronization method, a message synchronization system, electronic equipment and a storage medium. The message synchronization system comprises a management server and a mirror image service cluster, when the method is executed by the mirroring service cluster, the method comprises the steps that task configuration information is acquired, the task configuration information comprises configuration information of at least one message synchronization task, and the configuration information at least comprises a target area; the configuration information is generated by the management server; selecting a message synchronization task of which a target area is consistent with the area where the mirror image service cluster is located from the task configuration information as a target message synchronization task; and running the target message synchronization task based on the configuration information of the target message synchronization task. In this way, it is guaranteed that the message synchronization task is configurable and executable, message synchronization of the Kafka message queue in the same region cross-service and the different region cross-service can be achieved, and it is guaranteed that message transmission of the region cross-service and the region cross-service is stable and controllable.

Description

Technical Field

[0001] The present invention relates to the field of regional communication technology, and in particular to a message synchronization method, a message synchronization system, an electronic device and a storage medium. Background Art

[0002] Regions and availability zones (AZs) can both be used to describe the location of data centers. Users can create resources in specific regions and availability zones. Regions are divided based on geographic location and network latency. Public services such as elastic computing, block storage, object storage, VPC networks, elastic public IPs, and mirroring are shared within the same region. The purpose is to enable users to access nearby and reduce network latency. A region includes at least two or three availability zones (AZs), which are used to build a high-availability architecture. An AZ is a collection of one or more physical data centers with independent wind, fire, water, and electricity. Computing, network, storage, and other resources are logically divided into multiple clusters within the AZ.

[0003] With the global development of mobile application APP business, in order to improve the data processing efficiency of APP server and reduce the deployment cost, usually only one set of public services (such as payment and audit, etc.) is deployed in the same region, but there may be multiple APPs in the same region, and these APPs may be deployed in different AZs. Under this architecture, data communication problems between cross-business and cross-region gradually emerge.

[0004] In order to solve the problem of data communication across businesses and regions, in related technologies, public services and business services usually use their own Kafka clusters, message queue data is produced in the cluster where the Producer is located, and consumers communicate data across AZs. The advantage of this approach is that there is only one cluster instance, which is convenient for maintenance and monitoring. In addition, the network between consumers and clusters across AZs is implemented through the intranet, and the communication between consumers and clusters can directly use the intranet address. However, there are also the following shortcomings:

[0005] Unstable network: The cross-AZ network connection is unreliable, which affects the performance and availability of message transmission.

[0006] High maintenance cost: AZs are connected through the intranet, which complicates network management and increases dedicated line costs.

[0007] Isolation failure: Network interconnection between businesses may lead to direct connection of other data, resulting in high system coupling and affecting the security between businesses. Summary of the invention

[0008] In order to solve the above technical problems, the present invention provides a message synchronization method, a message synchronization system, an electronic device and a storage medium to solve the problem of data communication between cross-businesses and cross-regions.

[0009] In a first aspect, the present invention provides a message synchronization method, which is applied to a message synchronization system, wherein the message synchronization system comprises: a management server and a mirror service cluster;

[0010] The method is performed by the mirror service cluster, and includes:

[0011] Acquire task configuration information, wherein the task configuration information includes configuration information of at least one message synchronization task, and the configuration information includes at least a target area; the configuration information is generated by the management server;

[0012] Selecting a message synchronization task whose target area is consistent with the area where the mirroring service cluster is located from the task configuration information as the target message synchronization task;

[0013] Based on the configuration information of the target message synchronization task, the target message synchronization task is run.

[0014] Optionally, the message synchronization system further includes: a kafka cluster; the configuration information further includes: a source area, a source business, a target business, and a synchronization topic;

[0015] The running the target message synchronization task based on the configuration information of the target message synchronization task includes:

[0016] Read the synchronization message corresponding to the synchronization Topic from the Kafka cluster of the source business in the source area;

[0017] Write the synchronization message into the Kafka cluster of the target business.

[0018] Optionally, if the source area is the same as the target area, the source area and the target area use the same mirror service cluster, and the Kafka cluster of the source business and the Kafka cluster of the target business are in the same area;

[0019] If the source area is different from the target area, the source area and the target area respectively have corresponding mirror service clusters, and the Kafka cluster of the source business and the Kafka cluster of the target business are in different areas.

[0020] Optionally, the message synchronization system further includes a zookeeper cluster, and after the management server generates the task configuration information, the task configuration information is stored in the zookeeper cluster;

[0021] The obtaining of task configuration information includes:

[0022] When the image service cluster is started, task configuration information is read from the zookeeper cluster.

[0023] Optionally, after running the target message synchronization task, the method further includes:

[0024] The zookeeper cluster is monitored, and if the pause or stop configuration of the target message synchronization task is monitored, the target message synchronization task is paused or stopped.

[0025] Optionally, pausing or stopping the target message synchronization task includes:

[0026] If the target message synchronization task is suspended, the operation of the target message synchronization task is stopped but the configuration information of the target message synchronization task is not deleted;

[0027] If the target message synchronization task is stopped, the operation of the target message synchronization task is stopped and the configuration information of the target message synchronization task is deleted.

[0028] Optionally, the task configuration information is a task configuration generated by the management server after obtaining a required configuration input by a user.

[0029] In a second aspect, the present invention provides a message synchronization system, including a management server and a mirror service cluster;

[0030] The mirror service cluster includes:

[0031] A configuration acquisition module, used to acquire task configuration information, wherein the task configuration information includes configuration information of at least one message synchronization task, and the configuration information includes at least a target area; the configuration information is generated by the management server;

[0032] A target task determination module, used to select a message synchronization task whose target area is consistent with the area where the mirror service cluster is located from the task configuration information as a target message synchronization task;

[0033] The target task running module is used to run the target message synchronization task based on the configuration information of the target message synchronization task.

[0034] In a third aspect, the present invention provides an electronic device, the electronic device comprising:

[0035] at least one processor; and

[0036] a memory communicatively connected to the at least one processor; wherein,

[0037] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the message synchronization method described in the first aspect of the present invention.

[0038] In a fourth aspect, the present invention provides a computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a processor to implement the message synchronization method described in the first aspect of the present invention when executed.

[0039] This embodiment is based on the message synchronization system of the mirror service cluster, and cooperates with the management server to manage the message queue. The message synchronization system generates the configuration information of the message synchronization task through the management server. After the mirror service cluster obtains the configuration information of the message synchronization task, it compares the target area carried by the configuration information with the area where the mirror service cluster is located. If they are consistent, the current message synchronization task is used as the target message synchronization task and runs the target message synchronization task. In this way, the configurability and executableness of the message synchronization task are guaranteed, and the message synchronization of the Kafka message queue across businesses in the same region and across businesses in different regions can be realized, ensuring the stability and controllability of cross-region and cross-business message transmission, and ensuring the integrity and synchronization performance of the data.

[0040] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0042] Figure 1 is a structural diagram of a message synchronization system provided by an embodiment of the present invention;

[0043] Figure 2 is a flow chart of a message synchronization method provided by an embodiment of the present invention;

[0044] Figure 3 This is a schematic diagram of a cross-region and cross-Kafka cluster data synchronization architecture provided by an embodiment of the present invention;

[0045] Figure 4 This is a schematic diagram of an architecture for one-way synchronization of same-region messages provided by an embodiment of the present invention;

[0046] Figure 5 It is a schematic diagram of an architecture for bidirectional synchronization of same-region messages provided by an embodiment of the present invention;

[0047] Figure 6 This is a schematic diagram of an architecture for one-way synchronization of cross-region messages provided by an embodiment of the present invention;

[0048] Figure 7 It is a schematic diagram of an architecture for bidirectional synchronization of cross-region messages provided by an embodiment of the present invention;

[0049] Figure 8 is a flow chart of another message synchronization method provided by an embodiment of the present invention;

[0050] Fig. 9 It is a schematic diagram of the structure of the mirror service cluster provided by an embodiment of the present invention;

[0051] Fig.10 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0053] An embodiment of the present invention provides a method for message synchronization. This embodiment is applicable to scenarios where data is synchronized across regions and / or across businesses. The method can be executed by a message synchronization system, which can be implemented in the form of hardware and / or software.

[0054] In this embodiment, if Figure 1 As shown, the message synchronization system may include: a management server, a zookeeper cluster, a mirror service cluster, and a kafka cluster. In addition, the data synchronization system may also include a message producer and a message consumer (not shown in the figure).

[0055] In implementation, the management server may be MQ Admin, which is mainly responsible for the management of the Kafka cluster, the mirror service cluster, and the message synchronization task. Specifically, the management of the Kafka cluster may include, for example, the address management of each broker node in the Kafka cluster, cluster queue query, etc.; the management of the mirror service cluster may include, for example, the information management of the mirror service cluster, the addition of a mirror service cluster, and the status query of the mirror service cluster; the management of the message synchronization task may include, for example, the creation of a new message synchronization task, synchronization status query, and synchronization task control (start synchronization, pause synchronization, continue synchronization, and delete synchronization).

[0056] The Kafka cluster contains multiple Broker nodes. Broker is a server node in the Kafka cluster. Each Broker is responsible for managing multiple partitions and replicas, receiving messages from producers and providing messages to consumers. Multiple Brokers form a Kafka cluster, forming a highly available data storage and processing platform. Each Broker has a globally unique id.

[0057] The Mirror Service cluster can include multiple Mirror Service nodes. The Mirror Service node can be MirrorMaker2 (MM2 for short), which is responsible for receiving the dynamic configuration of the message synchronization task from MQ Admin (implemented through Zookeeper) and running / pausing / stopping the message synchronization task. Using MirrorMaker to synchronize data between different clusters can ensure the availability and reliability of the Kafka cluster.

[0058] Kafka uses Zookeeper to manage cluster metadata and controller elections.

[0059] Message consumers, or Kafka Consumers, are consumer clients of Kafka events (messages) and are used to subscribe to and receive messages from the Kafka cluster. Consumers can be organized in different consumer groups, and each consumer group can consume different partitions of the same Topic's messages. This can achieve load balancing and scalability for consumers.

[0060] The message producer is Kafka Producer, which is responsible for publishing (sending) messages to the Kafka cluster. It sends messages to the specified Topic and can choose to send messages to a specific partition. Producer is an application or service that sends messages to the Kafka cluster, such as a web server, mobile application, database change data capture (CDC) tool, etc.

[0061] The message synchronization system of this embodiment is implemented through an intranet environment, which can effectively isolate different business networks, reduce coupling, and ensure data security.

[0062] Figure 2 A flowchart of a message synchronization method provided by an embodiment of the present invention, the message synchronization method can be executed by a mirror service cluster, such as Figure 2 As shown, the message synchronization method includes:

[0063] Step 101: Obtain task configuration information, where the task configuration information includes configuration information of at least one message synchronization task, and the configuration information includes at least a target area.

[0064] In this embodiment, the task configuration information is generated by the management server. In one implementation, the task configuration information is generated by the management server according to the message synchronization requirements configured by the user. Specifically, the user can configure the requirement information through a configuration file or a configuration interface, and the MQ Admin can identify the message synchronization requirements from the requirement information.

[0065] Exemplarily, the message synchronization requirement may include synchronization direction, synchronization topic, synchronization topology, etc.

[0066] The synchronization direction is the direction of message transmission, which can include the source region Source Region, the target region TargetRegion, the source business Source BusinessEnv, and the target business Target BusinessEnv. Businesses can include APP business or public service business, such as payment, audit, etc. The source region and the target region can be the same or different, and the source business and the target business can be the same or different. For example, when a user purchases a product on the APP, the payment business in region A calls back the result to the payment Kafka cluster, and then synchronizes it to the Kafka cluster of the APP business in region B. In this scenario, the source region is region A, the target region is region B, the source business is the payment business, and the target business is the APP business.

[0067] Topic is a logical classification of messages. In Kafka, messages are published to one or more topics. Each topic can have multiple partitions, and each partition is replicated on different brokers to achieve load balancing and fault tolerance.

[0068] Synchronous topology is the topological structure of message transmission, including one-way synchronization and two-way synchronization. One-way synchronization means that the message transmission direction is one-way, and can only be transmitted from producer to consumer; two-way synchronization means that the message transmission direction is two-way, and both parties of the message transmission are two roles, which can be either producer or consumer.

[0069] After the message synchronization requirements are determined, the message synchronization scenario can be determined as one-way synchronization of messages in the same region, two-way synchronization of messages in the same region, one-way synchronization of messages across regions, or two-way synchronization of messages across regions.

[0070] After MQ Admin determines the message synchronization requirements, it can configure the message synchronization task according to the preset message synchronization task rules and generate the configuration information of the message synchronization task. The configuration information of the message synchronization task may include SourceRegion, Source BusinessEnv, Topic, Target Region and Target BusinessEnv.

[0071] For example, the configuration information of a message synchronization task can be expressed as:

[0072] ${source.region}.${source.business.env}.${topic}

[0073] ->${target.region}.${target.business.env}.${topic}

[0074] This embodiment manages and controls the message synchronization tasks through MQ Admin, which helps to improve the manageability of the system for message synchronization tasks, so as to better implement cross-region and cross-business message synchronization management.

[0075] After MQ Admin generates the configuration information of the message synchronization task, it can write the configuration information of the message synchronization task into the zookeeper cluster. The zookeeper cluster is responsible for managing the configuration information of each message synchronization task. The configuration information of all message synchronization tasks can be called task configuration information.

[0076] For the mirror service cluster, when it starts, it can read the task configuration information from the zookeeper cluster. This embodiment automatically pulls the synchronization configuration and starts the synchronization task when MM2 starts, which helps to improve the automation of the system during message synchronization.

[0077] Step 102 : Select a message synchronization task whose target area is consistent with the area where the mirror service cluster is located from the task configuration information as the target message synchronization task.

[0078] In this step, after the mirror service cluster obtains the task configuration information, it can further filter out the message synchronization tasks belonging to the mirror service cluster as the target message synchronization tasks. During implementation, the mirror service cluster can compare whether the target area of ​​each message synchronization task is consistent with the area where it is located. If consistent, the message synchronization task is retained as the target message synchronization task; if inconsistent, the message synchronization task is discarded.

[0079] Step 103: Run the target message synchronization task based on the configuration information of the target message synchronization task.

[0080] In this step, after the target message synchronization task is determined, the target message synchronization task can be started and run according to the configuration information of the target message synchronization task.

[0081] In one embodiment, the step of running the target message synchronization task based on the configuration information of the target message synchronization task may further include the following steps:

[0082] Read the synchronization message corresponding to the synchronization topic from the Kafka cluster of the source business in the source area; write the synchronization message to the Kafka cluster of the target business.

[0083] Specifically, the mirror service cluster determines the source region sourceregion, source business Source BusinessEnv, target business Target BusinessEnv, and synchronization Topic from the configuration information of the target message synchronization task, and then determines the source Kafka cluster based on the source region source region and source business Source BusinessEnv, and determines the target Kafka cluster based on the target business. Then, the synchronization message corresponding to the Topic of the current task is read from the Broker node of the source Kafka cluster and copied to the Broker node of the target Kafka cluster.

[0084] It should be noted that if the source area and the target area are the same, the source area and the target area use the same mirror service cluster, and the Kafka cluster of the source business and the Kafka cluster of the target business are in the same area; if the source area and the target area are different, the source area and the target area have corresponding mirror service clusters respectively, and the Kafka cluster of the source business and the Kafka cluster of the target business are in different areas.

[0085] For example, Figure 3The data synchronization architecture across regions and Kafka clusters is shown, including the core modules MQAdmin and MM2. The region where MQ Admin is located is the primary region, and the decision on which region to deploy MQ Admin in mainly depends on the convenience of the user (such as the administrator). Generally, the administrator will deploy MQ Admin in the region where he is located.

[0086] like Figure 3 As shown, MQ Admin relies on the data information of the MySQL storage system. After MQ Admin generates the configuration information of the message synchronization task, it writes it to the zookeeper cluster in the area (also called the main zookeeper cluster). The main zookeeper cluster then synchronizes the configuration information to the zookeeper clusters in other areas. When each MM2 starts, it can obtain and read the configuration information from the zookeeper cluster in its area, filter out the target message synchronization task from the configuration information, and run the target message synchronization task. Figure 3 In the example, assuming that the current MM2 is region2-MM2, after it determines the target message synchronization task, it determines that the target region of the target message synchronization task is region1, then it can read the synchronization message from the kafka broker of region1 and copy it to the kafka broker of region2. Alternatively, assuming that the current MM2 is region1-MM2, after it determines the target message synchronization task, it determines that the target region of the target message synchronization task is region2, then it can read the synchronization message from the kafka broker of region2 and copy it to the kafka broker of region1.

[0087] In order to enable those skilled in the art to better understand the embodiments of the present invention, Figure 3 Based on the architecture, the scenarios of one-way synchronization of messages in the same region, two-way synchronization of messages in the same region, one-way synchronization of messages across regions, and two-way synchronization of messages across regions are described separately:

[0088] refer to Figure 4The architecture diagram of one-way synchronization of messages in the same region is shown. The Producer of the source business pushes the synchronization message to a specific Topic (topic1) in the Kafka cluster, and the message of topic1 is stored in the Kafka cluster of the source business. After MM2 determines the target message synchronization task, it extracts the synchronization Topic as topic1, reads the synchronization message from the Kafka cluster of topic1, and then synchronizes the synchronization message to the Kafka cluster corresponding to the target business. The Consumer of the target business reads the synchronization message in topic1 through the Broker.

[0089] refer to Figure 5 The architecture diagram of bidirectional synchronization of messages in the same region is shown. The Producer of the source business pushes the synchronization message to a specific Topic (topic1) in the Kafka cluster, and the messages of topic1 are stored in the Kafka cluster of the source business. After MM2 determines the target message synchronization task, it extracts the synchronization Topic as topic1, reads the synchronization message from the Kafka cluster of topic1, and then synchronizes the synchronization message to the Kafka cluster corresponding to the target business. The Consumer of the target business reads the synchronization message in topic1 through the Broker. Similarly, in reverse communication, the Producer of the target business pushes the synchronization message to a specific Topic (topic2) in the Kafka cluster, and the messages of topic2 are stored in the Kafka cluster of the target business. After MM2 determines the target message synchronization task, it extracts the synchronization Topic as topic2, reads the synchronization message from the Kafka cluster of topic2, and then synchronizes the synchronization message to the Kafka cluster corresponding to the source business. The Consumer of the source business reads the synchronization message in topic2 through the Broker.

[0090] refer to Figure 6 The architecture diagram of cross-region message one-way synchronization is shown. Since this example focuses on the process of MM2 running message synchronization tasks, Figure 6 In order to facilitate the explanation of the cross-region communication process, the MQAdmin and zookeeper clusters are not shown in the figure. Figure 6In the example, there are two regions, region1 and region2. The Producer of the source business pushes the synchronization message to a specific Topic (topic3) in the Kafka cluster, and the message of topic3 ​​is stored in the Kafka cluster of the source business. After region1-MM2 determines the target message synchronization task and extracts the synchronization Topic as topic3, it reads the synchronization message from the Kafka cluster of topic3, and then copies the synchronization message to the Broker of the Kafka cluster of the public service. Next, region2-MM2 reads the synchronization message from the Kafka cluster of the public service, and copies the synchronization message to the Kafka cluster corresponding to the target business. The Consumer of the target business reads the synchronization message in topic3 ​​through the Broker.

[0091] refer to Figure 7 The architecture diagram of cross-region message bidirectional synchronization is shown. Since this example focuses on the process of MM2 running message synchronization tasks, Figure 7 In order to facilitate the explanation of the cross-region communication process, the MQAdmin and zookeeper clusters are not shown in the figure. Figure 7 In the example, there are two regions, region1 and region2. The Producer of the source business pushes the synchronization message to a specific Topic (topic1) in the Kafka cluster, and the message of topic1 is stored in the Kafka cluster of the source business. After region1-MM2 determines the target message synchronization task, it extracts the synchronization Topic as topic1, reads the synchronization message from the Kafka cluster of topic1, and then copies the synchronization message to the Kafka cluster of the public service. Next, region2-MM2 reads the synchronization message from the Broker of the Kafka cluster of the public service, and copies the synchronization message to the Kafka cluster corresponding to the target business. The Consumer of the target business reads the synchronization message in topic1 through the Broker. Similarly, in reverse communication, the Producer of the target business pushes the synchronization message to a specific Topic (topic2) in the Kafka cluster, and the message of topic2 is stored in the Kafka cluster of the target business. After Region2-MM2 determines the target message synchronization task, it extracts the synchronization topic as topic2, reads the synchronization message from the Kafka cluster of topic2, and then copies the synchronization message to the Kafka cluster of the public service. Next, region1-MM2 reads the synchronization message from the Kafka cluster of the public service, and copies the synchronization message to the Kafka cluster corresponding to the source business. The Consumer of the source business reads the synchronization message in topic2 through the Broker.

[0092] Through the above Figure 4-Figure 7 As can be seen from the example, the cross-region / cross-business message queue data synchronization can be achieved through the MM2 multi-instance cluster deployment, which can ensure low data latency and improve the reliability and availability of message synchronization. During the data synchronization process, Kafka Connect is used to realize the production and consumption of messages, achieve high availability and high reliability of data synchronization, and ensure that message synchronization data is not lost.

[0093] This embodiment is based on the message synchronization system of the mirror service cluster, and cooperates with the management server to manage the message queue. The message synchronization system generates the configuration information of the message synchronization task through the management server. After the mirror service cluster obtains the configuration information of the message synchronization task, it compares the target area carried by the configuration information with the area where the mirror service cluster is located. If they are consistent, the current message synchronization task is used as the target message synchronization task and runs the target message synchronization task. In this way, the configurability and executableness of the message synchronization task are guaranteed, and the message synchronization of the Kafka message queue across businesses in the same region and across businesses in different regions can be realized, ensuring the stability and controllability of cross-region and cross-business message transmission, and ensuring the integrity and synchronization performance of the data.

[0094] Figure 8 A flowchart of another message synchronization method provided by an embodiment of the present invention, wherein the message synchronization method is Figure 2 Based on the embodiment, the dynamic control scenario of the message synchronization task is described, such as Figure 8 As shown, the message synchronization method includes:

[0095] Step 201, when the image service cluster is started, task configuration information is read from the zookeeper cluster in the region, where the task configuration information includes configuration information of at least one message synchronization task, and the configuration information includes at least a target region.

[0096] The configuration information is generated by the management server.

[0097] Step 202 : Select a message synchronization task whose target area is consistent with the area where the mirror service cluster is located from the task configuration information as the target message synchronization task.

[0098] Step 203: Run the target message synchronization task based on the configuration information of the target message synchronization task.

[0099] Step 204, monitor the zookeeper cluster, and if the pause or stop configuration of the target message synchronization task is monitored, the target message synchronization task is paused or stopped.

[0100] In one implementation, MM2 can monitor Zookeeper based on the monitoring mechanism of Zookeeper. If the configuration change of Zookeeper is detected, the configuration information is read. And it is determined whether the target message synchronization task has changed according to the changed configuration information. If the configuration information of the target message synchronization task has changed, the target message synchronization task is dynamically controlled according to the change.

[0101] The configuration information change of the target message synchronization task includes pausing or stopping the target message synchronization task. Specifically, if the target message synchronization task is suspended, the running of the target message synchronization task is stopped but the configuration information of the target message synchronization task is not deleted. In implementation, when pausing a message synchronization task, for message synchronization tasks in the same region, the mirror service cluster in the region is used to stop the message synchronization task; for cross-regional message synchronization tasks, the mirror service cluster in the source region is used to stop the message synchronization task first, and then the mirror service cluster in the target region is used to stop the message synchronization task.

[0102] If the target message synchronization task is stopped, the target message synchronization task will be stopped and its configuration information will be deleted. In implementation, when stopping a message synchronization task, for a message synchronization task in the same region, the mirror service cluster in that region will be used to stop and remove the message synchronization task; for a message synchronization task across regions, the mirror service cluster in the source region will be used to stop and remove the message synchronization task first, and then the mirror service cluster in the target region will be used to stop and remove the message synchronization task.

[0103] In this embodiment, MQ Admin centrally manages the Kafka cluster address and MM2 cluster information, and dynamically controls the running status of the synchronization task. After obtaining the dynamic configuration of the message synchronization task through MM2, the message synchronization task is run / pause / stopped, thereby dynamically adjusting the synchronization task according to different business needs and improving the flexible management method of the system. Furthermore, a monitoring mechanism can also be built into the system to track the size of data synchronized per second, delay, etc., to ensure the efficiency and timeliness of synchronization.

[0104] Corresponding to the message synchronization method in the present invention, the present invention also provides a message synchronization system, including a management server and a mirror service cluster; Fig. 9 This is a schematic diagram of the structure of the image service cluster. Fig. 9 As shown, the image service cluster includes:

[0105] The configuration acquisition module 301 is used to acquire task configuration information, wherein the task configuration information includes configuration information of at least one message synchronization task, and the configuration information includes at least a target area; the configuration information is generated by the management server;

[0106] A target task determination module 302 is used to select a message synchronization task whose target area is consistent with the area where the mirror service cluster is located from the task configuration information as a target message synchronization task;

[0107] The target task running module 303 is used to run the target message synchronization task based on the configuration information of the target message synchronization task.

[0108] In one embodiment, the message synchronization system further includes: a kafka cluster; the configuration information further includes: a source area, a source business, a target business, and a synchronization topic;

[0109] The target task running module 303 is specifically used for:

[0110] Read the synchronization message corresponding to the synchronization Topic from the Kafka cluster of the source business in the source area;

[0111] Write the synchronization message into the Kafka cluster of the target business.

[0112] In one embodiment, if the source region is the same as the target region, the source region and the target region use the same mirror service cluster, and the Kafka cluster of the source service and the Kafka cluster of the target service are in the same region;

[0113] If the source area is different from the target area, the source area and the target area respectively have corresponding mirror service clusters, and the Kafka cluster of the source business and the Kafka cluster of the target business are in different areas.

[0114] In one embodiment, the message synchronization system further includes a zookeeper cluster, and after the management server generates the task configuration information, the task configuration information is stored in the zookeeper cluster;

[0115] The configuration acquisition module 301 is specifically used for:

[0116] When the image service cluster is started, task configuration information is read from the zookeeper cluster.

[0117] In one embodiment, the device further comprises:

[0118] A monitoring module, used for monitoring the zookeeper cluster after running the target message synchronization task;

[0119] The task processing module is used to pause or stop the target message synchronization task if the pause or stop configuration of the target message synchronization task is monitored.

[0120] In one embodiment, the task processing module is specifically used to:

[0121] If the target message synchronization task is suspended, the operation of the target message synchronization task is stopped but the configuration information of the target message synchronization task is not deleted;

[0122] If the target message synchronization task is stopped, the operation of the target message synchronization task is stopped and the configuration information of the target message synchronization task is deleted.

[0123] In one embodiment, the task configuration information is the task configuration generated by the management server after obtaining the required configuration input by the user.

[0124] The message synchronization system provided by the embodiment of the present invention can execute the message synchronization method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0125] Fig.10 A schematic diagram of an electronic device 40 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0126] like Fig.10 As shown, the electronic device 40 includes at least one processor 41, and a memory connected to the at least one processor 41, such as a read-only memory (ROM) 42, a random access memory (RAM) 43, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 41 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 42 or the computer program loaded from the storage unit 48 to the random access memory (RAM) 43. In the RAM 43, various programs and data required for the operation of the electronic device 40 can also be stored. The processor 41, the ROM 42, and the RAM 43 are connected to each other through a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0127] A number of components in the electronic device 40 are connected to the I / O interface 45, including: an input unit 46, such as a keyboard, a mouse, etc.; an output unit 47, such as various types of displays, speakers, etc.; a storage unit 48, such as a disk, an optical disk, etc.; and a communication unit 49, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 49 allows the electronic device 40 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0128] The processor 41 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 41 executes the various methods and processes described above, such as a message synchronization method.

[0129] In some embodiments, the method of message synchronization may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 48. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 40 via the ROM 42 and / or the communication unit 49. When the computer program is loaded into the RAM 43 and executed by the processor 41, one or more steps of the method of message synchronization described above may be performed. Alternatively, in other embodiments, the processor 41 may be configured to perform the method of message synchronization in any other appropriate manner (e.g., by means of firmware).

[0130] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0131] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0132] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0133] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0134] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0135] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0136] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0137] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A message synchronization method, characterized in that: Applied in a message synchronization system, the message synchronization system comprises: a management server and a mirror service cluster; The method is performed by the mirror service cluster, and includes: Acquire task configuration information, wherein the task configuration information includes configuration information of at least one message synchronization task, and the configuration information includes at least a target area; the configuration information is generated by the management server; Selecting a message synchronization task whose target area is consistent with the area where the mirroring service cluster is located from the task configuration information as the target message synchronization task; Based on the configuration information of the target message synchronization task, the target message synchronization task is run.

2. The method according to claim 1, characterized in that The message synchronization system also includes: a kafka cluster; the configuration information also includes: a source area, a source business, a target business, and a synchronization topic; The running the target message synchronization task based on the configuration information of the target message synchronization task includes: Read the synchronization message corresponding to the synchronization Topic from the Kafka cluster of the source business in the source area; Write the synchronization message into the Kafka cluster of the target business.

3. The method according to claim 2, characterized in that If the source region is the same as the target region, the source region and the target region use the same mirror service cluster, and the Kafka cluster of the source business and the Kafka cluster of the target business are in the same region; If the source area is different from the target area, the source area and the target area respectively have corresponding mirror service clusters, and the Kafka cluster of the source business and the Kafka cluster of the target business are in different areas.

4. The method according to claim 1, 2 or 3, characterized in that: The message synchronization system further includes a zookeeper cluster. After the management server generates task configuration information, the task configuration information is stored in the zookeeper cluster. The obtaining of task configuration information includes: When the image service cluster is started, task configuration information is read from the zookeeper cluster.

5. The method according to claim 4, characterized in that After running the target message synchronization task, the method further includes: The zookeeper cluster is monitored, and if the pause or stop configuration of the target message synchronization task is monitored, the target message synchronization task is paused or stopped.

6. The method according to claim 5, characterized in that The pausing or stopping the target message synchronization task includes: If the target message synchronization task is suspended, the operation of the target message synchronization task is stopped but the configuration information of the target message synchronization task is not deleted; If the target message synchronization task is stopped, the operation of the target message synchronization task is stopped and the configuration information of the target message synchronization task is deleted.

7. The method according to claim 1, 2 or 3, characterized in that: The task configuration information is the task configuration generated by the management server after obtaining the required configuration input by the user.

8. A message synchronization system, characterized in that: Includes management server and image service cluster; The mirror service cluster includes: A configuration acquisition module, used to acquire task configuration information, wherein the task configuration information includes configuration information of at least one message synchronization task, and the configuration information includes at least a target area; the configuration information is generated by the management server; A target task determination module, used to select a message synchronization task whose target area is consistent with the area where the mirror service cluster is located from the task configuration information as a target message synchronization task; The target task running module is used to run the target message synchronization task based on the configuration information of the target message synchronization task.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the message synchronization method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the message synchronization method according to any one of claims 1 to 7 when executed.

Images