Message queue system and operation method, device, equipment, medium and product thereof

Abstract

The application discloses a message queue system and a running method, device, equipment, medium and product thereof. The method comprises the following steps: deploying servers in different regions as a same message queue cluster; setting a plurality of copies for a topic to which a to-be-transmitted message belongs in the servers of the message queue cluster; wherein, each region in the message queue cluster is distributed with at least one copy; selecting one copy from the plurality of copies as a first copy, and taking a server where the first copy is located as a read-write service node, which is used for providing a read-write service of the to-be-transmitted message to a client. The embodiment of the application can avoid the risk of data inconsistency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of communication, and in particular to a message queue system and a running method, device, equipment, medium and product thereof. BACKGROUND

[0002] A message queue is a container for saving messages in the transmission process of the messages. In actual use, an upstream data source end sends data to a message queue cluster through a producer, and a downstream data application end pulls data from the message queue through a consumer.

[0003] In recent years, with the rapid development of computer networks and basic hardware, more possibilities are provided for the application requirements of cross-region data transmission. Cross-region refers to the producer and the consumer, which are usually distributed in different regions. For the cross-region transmission problem, the current main solution is to use a double-cluster solution: a set of message queue clusters are deployed in two regions, data synchronization is performed between the two sets of message queue clusters, and the downstream applications in each region preferentially access the message queue cluster in their own region. However, this approach has the following problems:

[0004] The production messages of the client of a normal message queue are all with an ack (Acknowledgment) mechanism, that is, for a message sent by the client, the server needs to ensure that there are enough copies written to complete the feedback to the client that the message is written successfully, so as to guarantee the consistency and reliability of the data. However, in the double-cluster solution, the first set of clusters exists as a client of the second set of clusters, so when producing messages, it cannot be guaranteed that the messages can be written to the second set of clusters, that is, there is no ack mechanism to confirm whether the messages pushed by the first set of clusters to the second set of clusters are successfully synchronized to the second set of clusters. This leads to the problem that data inconsistency is easily generated between the two sets of clusters, and further leads to the risk that the downstream applications also face data inconsistency. SUMMARY

[0005] The present application provides a message queue system and a running method, device, equipment, medium and product thereof to solve the problem that the existing double-cluster solution is prone to data inconsistency.

[0006] To achieve the above-mentioned purpose, the present application embodiment provides a running method of a message queue system, comprising:

[0007] deploying servers in different regions as the same message queue cluster;

[0008] setting multiple copies for a topic to which a to-be-transmitted message belongs in the servers of the message queue cluster; wherein each region in the message queue cluster is distributed with at least one copy.

[0009] select one of the plurality of replicas as a first replica, and select a server where the first replica is located as a read-write service node for providing a read-write service of the to-be-transmitted message to a client.

[0010] As an improvement of the above-mentioned solution, the method further comprises:

[0011] select at least one second replica from the plurality of replicas, and select a server where the second replica is located as a read-only service node for providing a read service of the to-be-transmitted message to a client; wherein the second replica is a replica in a different region from the first replica.

[0012] As an improvement of the above-mentioned solution, there is one second replica in each region where the first replica is not in the message queue cluster.

[0013] As an improvement of the above-mentioned solution, the method further comprises:

[0014] route a connection of a consumer client to a target service node in a same region as itself; wherein the target service node is the read-write service node or the read-only service node, and the consumer client is a client that needs to read the to-be-transmitted message from the message queue cluster.

[0015] As an improvement of the above-mentioned solution, the method comprises:

[0016] route a connection of a producer client to the read-write service node; wherein the producer client is a client that needs to write the to-be-transmitted message to the read-write service node.

[0017] As an improvement of the above-mentioned solution, the selecting one of the plurality of replicas as a first replica comprises:

[0018] for each of the plurality of replicas, count device information of a client that needs to request a service from a server where the replica is located; wherein the device information comprises at least one of the following: region information and performance information;

[0019] confirm a best region according to the device information;

[0020] select the first replica from the best region.

[0021] As an improvement of the above-mentioned solution, the confirming a best region according to the device information comprises:

[0022] confirm the best region according to a preset target function and the device information;

[0023] the target function comprises at least one of the following:

[0024] The first objective function aims to minimize the total network delay between all first target clients requiring service from the server where the same copy is located and the server where the copy is located; wherein the first target client is a producer client, or the first target client is the producer client and a consumer client; the consumer client is a client requiring to read the to-be-transmitted message from the message queue cluster, and the producer client is a client requiring to write the to-be-transmitted message to the read-write service node;

[0025] The second objective function aims to maximize the total flow of all second target clients in any region; wherein the second target client is the producer client and / or the consumer client;

[0026] The third objective function aims to maximize the total number of all third target clients in any region requiring cross-region service; the third target client is the producer client and the consumer client.

[0027] As an improvement of the above scheme, the first copy is selected from the optimal region, comprising:

[0028] The first copy is selected from the optimal region according to the distribution characteristics of the read-write service node and / or the computing resources of the server.

[0029] To achieve the above object, the embodiment of the present application further provides a running device of a message queue system, comprising:

[0030] A deployment module is configured to deploy servers in different regions as a same message queue cluster;

[0031] A setting module is configured to set a plurality of copies for a topic to which a to-be-transmitted message belongs in servers of the message queue cluster; wherein each region in the message queue cluster is distributed with at least one copy.

[0032] A first selection module is configured to select one copy from the plurality of copies as a first copy, and to select a server where the first copy is located as a read-write service node for providing read-write service of the to-be-transmitted message to a client.

[0033] To achieve the above object, the embodiment of the present application further provides a message queue system, comprising:

[0034] A server module is configured to deploy servers in different regions as a same message queue cluster; and to set a plurality of copies for a topic to which a to-be-transmitted message belongs in servers of the message queue cluster; wherein each region in the message queue cluster is distributed with at least one copy.

[0035] a decision module configured to select one of the multiple replicas as a first replica and a server where the first replica is located as a read-write service node;

[0036] a client module configured to request a read-write service of the to-be-transmitted message from the read-write service node;

[0037] a metadata management module configured to record and monitor data of the server module, the decision module and the client module.

[0038] To achieve the above object, the embodiment of the present application further provides a running device of a message queue system, comprising a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the running method of the message queue system as above when executing the computer program.

[0039] To achieve the above object, the embodiment of the present application further provides a computer readable storage medium, comprising a stored computer program; wherein the computer program controls the device where the computer readable storage medium is located to execute the running method of the message queue system as above when running.

[0040] To achieve the above object, the embodiment of the present application further provides a computer program product, comprising computer program / instruction, which implements the running method of the message queue system as above when executed by a processor.

[0041] Compared with the prior art, the embodiment of the present application provides a message queue system and a running method, device, equipment, medium and product thereof, which deploys servers in different regions as a same message queue cluster; sets multiple replicas for a topic to which a to-be-transmitted message belongs in the servers of the message queue cluster; wherein each region in the message queue cluster is distributed with at least one of the replicas; selects one of the multiple replicas as a first replica and a server where the first replica is located as a read-write service node, which is used to provide a read-write service of the to-be-transmitted message for a client. As can be seen, the embodiment of the present application only deploys a cross-regional message queue cluster, that is, adopts a single cluster solution, which can effectively ensure data consistency and avoid the risk of data inconsistency in the existing double cluster solution. Moreover, the embodiment of the present application only needs at least one replica in each region, which can save storage resources. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 is a whole structure diagram of a double cluster solution;

[0043] Figure 2is a flow chart of a running method of a message queue system provided by an embodiment of the present application;

[0044] Figure 3 is a structural block diagram of a running device of a message queue system provided by an embodiment of the present application;

[0045] Figure 4 is a structural block diagram of a message queue system provided by an embodiment of the present application;

[0046] Figure 5 is a structural block diagram of another message queue system provided by an embodiment of the present application;

[0047] Figure 6 is a structural block diagram of a running device of a message queue system provided by an embodiment of the present application. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the present application will be clearly and completely described with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0049] In the description of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0050] In the description of the present application, the term "exemplary" or "for example" is used to represent as an example, illustration or description. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the term "exemplary" or "for example" is intended to present the relevant concept in a specific manner.

[0051] In the description of the present application, the terms "first", "second", etc. are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices. The term "based on" is "at least partially based on". The term "according to" is "at least partially according to". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The term "and / or" means at least one of the connected objects, for example, A and / or B means three cases including A alone, B alone, and A and B both. Unless otherwise stated, the term "a plurality of" means two or more.

[0052] As Figure 1 , the existing dual-cluster solution is to deploy a set of message queue clusters in two regions respectively, and then perform real-time data synchronization between the two clusters (i.e. message queue clusters), and the downstream application preferentially accesses the message queue cluster in the same region; wherein each server is referred to as a broker. This solution mainly has the following three problems:

[0053] (1) Data consistency problem:

[0054] The normal message queue client's production message is with ack mechanism, that is, for a message sent by the client, the server needs to ensure that there are enough copies written to complete the feedback to the client to declare that the message is written successfully, so as to ensure the consistency and reliability of the data. However, in the dual-cluster solution, the situation is different: the first set of clusters exists as a client of the second set of clusters, so when producing messages, it cannot be guaranteed that the message can be written to the second set of clusters, that is, there is no ack mechanism to confirm whether the message pushed by the first set of clusters to the second set of clusters is successfully synchronized to the second set of clusters. This leads to the problem of data inconsistency between the two sets of clusters, and further leads to the risk of data inconsistency faced by the downstream application.

[0055] (2) Data reliability problem:

[0056] In the dual-cluster solution, only one complete copy is guaranteed for the data synchronized from the first set of clusters to the second set of clusters. If the data of the second set of clusters is lost after the data copy is completed and before the user uses the data, the existing solution does not restore the data.

[0057] (3) Resource waste problem:

[0058] 3.1 Storage resource waste:

[0059] In order to ensure the reliability of data, the message queue cluster usually stores data in multiple copies, that is, at least two copies of a piece of data are stored in a set of clusters, so at least four copies are required for the dual-cluster solution; for example, Figure 1 Each slave broker stores one copy, and there are four copies in total.

[0060] 3.2 Dual-cluster cannot share consumption offset:

[0061] When the downstream application consumes the message from the message queue cluster, the server records how much data the application has pulled, that is, the consumption offset of the data. In the dual-cluster solution, only the message content can be synchronized, and the consumption offset cannot be synchronized, which limits each client to access the message queue cluster in the corresponding region. When the message queue cluster in the local region fails, even if the message queue cluster in the other region stores the same data, the client cannot use it across regions. Therefore, although there are two clusters, high availability cannot be guaranteed.

[0062] That is, the data synchronization between the dual clusters is an independent task, and the timeliness and reliability of data synchronization cannot be guaranteed, so the user needs to confirm the status of data synchronization. In addition, only the message data is synchronized between the dual clusters, and the consumption of the client to the message is not synchronized, and the two sets of clusters are essentially used independently.

[0063] Therefore, the embodiments of the present application provide a running method of a message queue system, which deploys only one cross-region message queue cluster, that is, adopts a single-cluster solution, can effectively guarantee the consistency and reliability of data by means of the copy synchronization mechanism of the message queue cluster itself, and avoids the risk of data inconsistency in the existing dual-cluster solution. And only one copy needs to be distributed in each region, and only the consumption of the client to the data needs to be maintained, which effectively saves the storage space and guarantees the cross-region high availability of the server.

[0064] Referring to Figure 2 , Figure 2 is a flowchart of a running method of a message queue system provided by the embodiments of the present application, and the running method of the message queue system comprises:

[0065] S1, deploy servers in different regions as one message queue cluster;

[0066] It is worth noting that in the message queue cluster, each server is called a broker, and a message queue cluster is composed of multiple brokers. The number of regions is not limited in the embodiments of the present application. Assuming that the servers are distributed in two regions: region A and region B, the servers in region A and region B are deployed as one message queue cluster, and in the cluster, there are servers in region A and servers in region B. Assuming that the servers are distributed in three regions: region A, region B and region C, the servers in region A, region B and region C are deployed as one message queue cluster, and in the cluster, there are servers in region A, servers in region B and servers in region C.

[0067] The embodiments of the present application only deploy one cross-region message queue cluster, that is, a single cluster solution is adopted, which can effectively ensure data consistency and avoid the risk of data inconsistency in the existing double cluster solution.

[0068] In addition, before the message queue cluster is started, configuration information of the region is added to each broker in the cluster to indicate the region where the broker is located.

[0069] S2, set multiple replicas for a topic to which a to-be-transmitted message belongs in the servers of the message queue cluster; wherein each region in the message queue cluster is distributed with at least one replica of the topic;

[0070] It is worth noting that messages are grouped by topic, and the production and consumption of the client are all for the topic. In order to ensure data reliability, multiple replicas are set for each topic, and each replica is saved on a broker. Each broker can have multiple topics.

[0071] In order to ensure that data is evenly distributed in the cross-region message queue cluster as much as possible, while reducing resource waste, at least one replica of the same topic is distributed in each region in the message queue cluster, that is, at least one server in each region is randomly selected to distribute one replica, which can save storage resources.

[0072] For example, assuming that there are server 1 and server 2 in region A and server 3 in region B in a message queue cluster;

[0073] The to-be-transmitted message 1 and the to-be-transmitted message 2 belong to topic 1, multiple replicas of topic 1 are set, and each replica of topic 1 includes the to-be-transmitted message 1 and the to-be-transmitted message 2. The replicas of topic 1 are set in server 1 and / or server 2, and the replicas of topic 1 are set in server 3.

[0074] The to-be-transmitted message 3 belongs to topic 2, multiple replicas of topic 2 are set, and each replica of topic 2 includes the to-be-transmitted message 3. The replicas of topic 2 are set in server 1 and / or server 2, and the replicas of topic 2 are set in server 3.

[0075] S3, selecting one of the multiple replicas as a first replica, and taking a server where the first replica is located as a read-write service node for providing read-write services of the to-be-transmitted message to a client.

[0076] In the embodiments of the present application, in order to ensure the consistency of data, one of the multiple replicas of the same topic can provide read-write services of the topic (specifically, messages of the topic) to a client, and this replica is referred to as a first replica. The first replica can be a randomly selected replica, and a server where the first replica is located is a read-write service node. Depending on the data synchronization mechanism of the message queue cluster itself, the remaining replicas can synchronize data of the first replica from the read-write service node at regular intervals, and the remaining replicas do not provide services of the topic to the client, but only serve as data backup. This can avoid the phenomenon that only one complete replication is guaranteed in the existing double-cluster solution, and if data of the second set of clusters is lost after replication, the part of data cannot be recovered, thereby improving data reliability.

[0077] Further, the servers in the message queue cluster are also used to record consumption offsets of data, so that only one copy of consumption of data by a client needs to be maintained, storage space is effectively saved, and each client is prevented from being limited to accessing only the message queue cluster in a corresponding region, thereby ensuring high availability of the server across regions.

[0078] In an optional embodiment, the method further includes:

[0079] selecting at least one second replica from the multiple replicas, taking a server where the second replica is located as a read-only service node for providing read services of the to-be-transmitted message to a client, and wherein the second replica is a replica that is not in the same region as the first replica.

[0080] It is worth mentioning that in the scenario of setting only 1 read-write service node in the cross-region message queue cluster, if each client not in the same region as the first copy wants to access the first copy, data needs to be transmitted cross-region, and cross-region transmission often means greater network delay. Therefore, the embodiment of the application selects at least 1 second copy in the remaining regions except the region where the first copy is located, and takes the server where the second copy is located as a read-only service node for the client to consume data, which can reduce cross-region network transmission, reduce cross-region network pressure, thereby improving data transmission performance, and is especially suitable for scenarios where business is more sensitive to data delay.

[0081] The number of second copies is not limited in the embodiment of the application. Assuming that the copies of a topic are distributed in region A, region B and region C, and the first copy is in region A, then 1 second copy can be randomly selected in region B and region C, for example, only 1 second copy is selected in region B, and the server where the second copy is located is taken as a read-only service node; the clients in region B can consume data through the read-only service node; the clients in region A can consume data through the read-write service node; the clients in region C can consume data through the read-only service node or the read-write service node, and the specific decision can be made according to the client information in region C, for example, if the total network delay of all clients in region C and the read-write service node is less than the total network delay of all clients in region C and the read-only service node, the read-write service node is selected for the clients in region C to consume data. Only 1 second copy can also be selected in region C, and 1 second copy can also be selected in region B and region C at the same time, which will not be described here.

[0082] In an example, for a topic, assuming that the first copy is in region A, if multiple clients in region B want to consume the data of the topic, if only the read-write service node is set, each client in region B will be connected to the read-write service node, and each client in region B will perform cross-region data transmission. The embodiment of the application selects one copy not in the same region as the first copy as a second copy from multiple copies of the topic to provide read-only service for the client. When all clients in region B have the demand to consume the topic, they directly connect to the server where the second copy is located and consume data. In this way, for the data of the topic, only cross-region transmission is performed during copy synchronization, and cross-region transmission is not needed when the clients in region B consume data, which effectively reduces the occupation of cross-region network bandwidth. And since it is essentially the same set of message queue clusters, relying on the data synchronization mechanism of the message queue cluster itself, the consistency and reliability of the data can be effectively guaranteed.

[0083] In an alternative embodiment, one second copy is selected in each region where the first copy is not present in the message queue cluster.

[0084] In order to reduce cross-region data transmission to the greatest extent, in the embodiment of the application, one second copy of the same topic is selected in each region where the first copy of the same topic is not present, so that the second copy can be used for data consumption by a client in the same region as the read-only service node, and the remaining clients in other regions can use the service provided by the message queue without cross-region network, thereby reducing cross-region data transmission to the greatest extent.

[0085] Suppose that the copies of a topic are distributed in region A, region B and region C, and the first copy is in region A, then one second copy can be randomly selected in region B and one second copy can be randomly selected in region C, so that the second copy in region B is only used for data consumption by a client in region B, and the second copy in region C is only used for data consumption by a client in region C, thereby reducing cross-region data transmission to the greatest extent.

[0086] In an alternative embodiment, the method further comprises:

[0087] The connection of the consumer client is routed to a target service node in the same region as itself, wherein the target service node is the read-write service node or the read-only service node, and the consumer client is a client that needs to read the to-be-transmitted message from the message queue cluster.

[0088] It is worth noting that the consumer client can read the to-be-transmitted message from the read-write service node in the message queue cluster, or read the to-be-transmitted message from the read-only service node in the message queue cluster.

[0089] In the embodiment of the application, the connection of the consumer client is routed to the read-write service node in the same region as itself, or the read-only service node in the same region, so that when the consumer client needs to consume data, the data is consumed through the routing in the region of the consumer client, thereby reducing cross-region data transmission to the greatest extent.

[0090] In an alternative embodiment, the method comprises:

[0091] The connection of the producer client is routed to the read-write service node, wherein the producer client is a client that needs to write the to-be-transmitted message to the read-write service node.

[0092] It is worth noting that in the embodiment of the application, the connection of the producer client is routed to the read-write service node, so that when the producer client needs to produce data, the data is produced through the routing of the producer client, thereby reducing cross-region data transmission.

[0093] In an optional embodiment, the selecting one of the multiple copies as the first copy comprises:

[0094] For each of the multiple copies, statistics of device information of clients requiring service from a server where the copy is located are collected; wherein the device information comprises at least one of the following: region information and performance information;

[0095] According to the device information, an optimal region is determined;

[0096] The first copy is selected from the optimal region.

[0097] Embodiments of the present application determine an optimal region according to device information of clients, and then select a first copy from the optimal region, which can avoid blind selection and improve selection reliability.

[0098] In an optional embodiment, the determining the optimal region according to the device information comprises:

[0099] The optimal region is determined according to a preset target function and the device information;

[0100] The target function comprises at least one of the following:

[0101] A first target function aims to minimize total network delay between all first target clients requiring service from a server where a copy is located and the server where the copy is located; wherein the first target clients are producer clients, or the first target clients are the producer clients and consumer clients; the consumer clients are clients requiring to read the to-be-transmitted messages from the message queue cluster, and the producer clients are clients requiring to write the to-be-transmitted messages to the read-write service node;

[0102] A second target function aims to maximize total traffic of all second target clients in any region; wherein the second target clients are the producer clients and / or the consumer clients;

[0103] A third target function aims to maximize total number of all third target clients requiring cross-region service in any region; the third target clients are the producer clients and the consumer clients.

[0104] It is worth noting that the first target function is constructed according to performance information of target clients, and specific performance information is network delay between the first target clients and the server.

[0105] Specifically, the total network delay between all producer clients and consumer clients in the cluster and the server where the same replica is located is counted.

[0106] The first target client is the producer client and the consumer client, and the case of the consumer client is considered, the device information can be more comprehensively understood, the selection reliability is improved, and it is more suitable for a message queue cluster which is only provided with read-write service nodes. For example, the replicas of a topic in a message queue cluster include replica 1 and replica 2, replica 1 is distributed in region A, and replica 2 is distributed in region B. The following information is counted: the total network delay 1 between all first target clients (here including: producer clients and consumer clients) in the cluster and the server where replica 1 is located, and the total network delay 2 between all first target clients in the cluster and the server where replica 2 is located. If the total network delay 1 is less than the total network delay 2, it indicates that the clients in the region where replica 1 is located, i.e. region A, will produce less cross-region data transmission, and region A is confirmed as the best region to reduce cross-region transmission.

[0107] The first target client is the producer client, and the case of the consumer client does not need to be considered, and it is more suitable for a message queue cluster which is provided with read-write service nodes and read-only service nodes. For example, the replicas of a topic in a message queue cluster include replica 1 and replica 2, replica 1 is distributed in region A, and replica 2 is distributed in region B. The following information is counted: the total network delay 3 between all producer clients in the cluster and the server where replica 1 is located, and the total network delay 4 between all producer clients in the cluster and the server where replica 2 is located. If the total network delay 3 is less than the total network delay 4, it indicates that the clients in the region where replica 3 is located, i.e. region A, will produce less cross-region data transmission, and the region where replica 1 is located, i.e. region A, is confirmed as the best region to reduce cross-region transmission.

[0108] The second target function is constructed according to the performance information of the target client, and the specific performance information is the traffic of the second target client.

[0109] For example, the replicas of a topic include replica 1 and replica 2, replica 1 is distributed in region A, and replica 2 is distributed in region B. The following information is counted: the total traffic 1 of all second target clients (here including: producer clients and consumer clients) in region A, and the total traffic 2 of all second target clients in region B. If the total traffic 1 is less than the total traffic 2, it indicates that the clients in region B corresponding to the total traffic 2 have more traffic, and the transmission efficiency will be faster, and therefore region B is confirmed as the best region.

[0110] The third target function is constructed according to the region information of the target client, and the specific region information is the number of third target clients in any region.

[0111] For example, a topic has two replicas, replica 1 and replica 2, replica 1 is distributed in region A, and replica 2 is distributed in region B. The following information is counted: total number 1 of all third target clients (here including: producer clients and consumer clients) in region A that need cross-region services, and total number 2 of all third target clients in region B that need cross-region services. If total number 1 is less than total number 2, it means that the clients in region B will generate less cross-region data transmission, and region B is confirmed as the best region to reduce cross-region transmission.

[0112] In an optional embodiment, the first replica is selected from the best region according to the distribution characteristics of read-write service nodes and / or the computing resources of servers.

[0113] The first replica is selected from the best region according to the distribution characteristics of read-write service nodes and / or the computing resources of servers.

[0114] It is worth noting that the embodiments of the present application select the first replica from the best region according to the distribution characteristics of read-write service nodes and / or the computing resources of servers, taking into account the uniformity of read-write server distribution and the computing resources of servers, avoiding blind selection of the first replica, and improving selection reliability and transmission efficiency.

[0115] Specifically, the first replica is selected from the best region according to the distribution characteristics of read-write service nodes. This can make read-write service nodes as evenly distributed as possible, thereby avoiding server overload and improving transmission efficiency.

[0116] For example, the best region of topic 1 and topic 2 is region A, region A has server A and server B, and server A and server B each have their replicas; suppose server 1 is a read-write service node of topic 1, in order to make read-write service nodes as evenly distributed as possible, server 2 is selected as a read-write service node of topic 2, so that server 1 and server 2 each serve as a read-write service node of the corresponding topic. At this time, if the best region of topic 3 is region A, and server A and server B each have a replica of topic 3, regardless of whether server A or server B is selected as a read-write service node of topic 3, the distribution characteristics are the same, so server A or server B can be randomly selected as a read-write service node of topic 3; or the distribution characteristics of read-write service nodes are a preset distribution path, for example, the distribution path stipulates that server A, server B, and the like in region A are sequentially selected as read-write service nodes in order, so that server A is selected as a read-write service node of topic 3 according to the distribution path; or further in combination with the computing resources of servers, the server with more abundant computing resources is selected as a read-write service node of topic 3.

[0117] Specifically, according to the computing resources of the servers, several servers satisfying preset sufficient conditions from the optimal region are selected as a target server set; and the first copy is selected from the target server set. This can avoid selecting a server with overloaded computing resources, thereby avoiding affecting the use of read-write service nodes due to the overloaded computing resources, and further improving transmission efficiency.

[0118] Referring to Figure 3 , Figure 3 is a structural block diagram of a running device 10 of a message queue system provided by an embodiment of the present application, the running device 10 of the message queue system comprising:

[0119] a deployment module 11 configured to deploy servers in different regions as a same message queue cluster;

[0120] a setting module 12 configured to set a plurality of copies for a topic to which a to-be-transmitted message belongs in servers of the message queue cluster; wherein each region in the message queue cluster is distributed with at least one copy;

[0121] a first selection module 13 configured to select one of the plurality of copies as a first copy, and select a server where the first copy is located as a read-write service node configured to provide read-write services of the to-be-transmitted message to a client.

[0122] deploy servers in different regions as a same message queue cluster;

[0123] set a plurality of copies for a topic to which a to-be-transmitted message belongs in servers of the message queue cluster; wherein each region in the message queue cluster is distributed with at least one copy;

[0124] select one of the plurality of copies as a first copy, and select a server where the first copy is located as a read-write service node configured to provide read-write services of the to-be-transmitted message to a client.

[0125] Optionally, the device 10 further comprises:

[0126] a second selection module configured to select at least one second copy from the plurality of copies, and select a server where the second copy is located as a read-only service node configured to provide read services of the to-be-transmitted message to a client; wherein the second copy is a copy not in a same region as the first copy.

[0127] Optionally, each region not having the first copy in the message queue cluster is selected to have one second copy.

[0128] Optionally, the apparatus 10 further comprises:

[0129] a first routing module, configured to route a connection of a consumer client to a target service node in a same zone as itself; wherein the target service node is the read-write service node or the read-only service node, and the consumer client is a client that needs to read the to-be-transmitted message from the message queue cluster.

[0130] Optionally, the apparatus 10 further comprises:

[0131] a second routing module, configured to route a connection of a producer client to the read-write service node; wherein the producer client is a client that needs to write the to-be-transmitted message to the read-write service node.

[0132] Optionally, the selecting one of the multiple replicas as the first replica comprises:

[0133] For each of the multiple replicas, statistics of device information of clients that need to request services from a server where the replica is located are obtained; wherein the device information comprises at least one of the following: zone information and performance information;

[0134] The best zone is determined according to the device information.

[0135] The first replica is selected from the best zone.

[0136] Optionally, the determining the best zone according to the device information comprises:

[0137] The best zone is determined according to a preset target function and the device information.

[0138] The target function comprises at least one of the following:

[0139] A first target function, whose target is to minimize total network delay between all first target clients that need to request services from a server where a same replica is located and the server; wherein the first target client is a producer client, or the first target client is the producer client and a consumer client; the consumer client is a client that needs to read the to-be-transmitted message from the message queue cluster, and the producer client is a client that needs to write the to-be-transmitted message to the read-write service node.

[0140] A second target function, whose target is to maximize total flow of all second target clients in any zone; wherein the second target client is the producer client and / or the consumer client.

[0141] A third objective function aims to maximize the total number of all third target clients in any region that need to be served across regions; the third target clients are the producer clients and the consumer clients.

[0142] Optionally, the selecting the first copy from the optimal region comprises:

[0143] The first copy is selected from the optimal region according to the distribution characteristics of read-write service nodes and / or the computing resources of servers.

[0144] It is worth noting that the working processes of the various modules in the message queue system running device 10 described in the embodiments of the present application can refer to the working processes of the message queue system running method described in the above embodiments and achieve the same beneficial effects, which will not be repeated here.

[0145] Referring to Figure 4 , Figure 4 is a structural block diagram of a message queue system 20 provided by the embodiments of the present application, the message queue system 20 comprises:

[0146] The server module 21 is configured to deploy servers in different regions as one message queue cluster; a plurality of copies are set for a topic to which a to-be-transmitted message belongs in the servers of the message queue cluster; wherein each region in the message queue cluster is distributed with at least one copy.

[0147] The decision module 22 is configured to select one copy from the plurality of copies as a first copy, and select a server where the first copy is located as a read-write service node.

[0148] The client module 23 is configured to request read-write services of the to-be-transmitted message from the read-write service node.

[0149] The metadata management module 24 is configured to record and monitor data of the server module, the decision module and the client module.

[0150] It is worth noting that the server module 21 comprises various servers, the client module 23 comprises various clients, and the working processes of the server module 21, the decision module 22 and the client module 23 can refer to the working processes of the message queue system running method described in the above embodiments and achieve the same beneficial effects, which will not be repeated here.

[0151] For the metadata management module 24, whether it is the metadata of the message queue cluster itself, or the client information and the decision made by the decision module (i.e. the decision of selecting the first copy and the second copy), is recorded in the znode (node) of the zookeeper (an open source distributed coordination service), and the cooperation between the modules is realized by means of the watcher (listener) of the zookeeper.

[0152] Optionally, the decision module 22 is further configured to:

[0153] select at least one second copy from the plurality of copies, and take the server where the second copy is located as a read-only service node for providing read service of the to-be-transmitted message to the client; wherein the second copy is a copy not in the same region as the first copy.

[0154] Optionally, there is one second copy in each region where the first copy is not located in the message queue cluster.

[0155] Optionally, the client module 23 is further configured to:

[0156] route the connection of the consumer client to the target service node in the same region as itself; wherein the target service node is the read-write service node or the read-only service node, and the consumer client is a client that needs to read the to-be-transmitted message from the message queue cluster.

[0157] Optionally, the client module 23 is further configured to:

[0158] route the connection of the producer client to the read-write service node; wherein the producer client is a client that needs to write the to-be-transmitted message to the read-write service node.

[0159] Optionally, the selecting one copy from the plurality of copies as the first copy comprises:

[0160] For each of the plurality of copies, statistics the device information of the client that needs to request service from the server where the copy is located; wherein the device information comprises at least one of the following: region information and performance information;

[0161] confirm the best region according to the device information;

[0162] select the first copy from the best region.

[0163] Optionally, the confirming the best region according to the device information comprises:

[0164] According to the preset target function and the device information, the optimal region is determined;

[0165] The target function comprises at least one of:

[0166] The first target function aims to minimize the total network delay between all first target clients requiring service from the server where the same copy is located and the server where the copy is located; wherein the first target client is a producer client, or the first target client is the producer client and a consumer client; the consumer client is a client requiring to read the to-be-transmitted message from the message queue cluster, and the producer client is a client requiring to write the to-be-transmitted message to the read-write service node;

[0167] The second target function aims to maximize the total flow of all second target clients in any region; wherein the second target client is the producer client and / or the consumer client;

[0168] The third target function aims to maximize the total number of all third target clients in any region requiring cross-region service; the third target client is the producer client and the consumer client.

[0169] Optionally, the selecting the first copy from the optimal region comprises:

[0170] According to the distribution characteristics of the read-write service node and / or the computing resources of the server, the first copy is selected from the optimal region.

[0171] In an example, as Figure 5 In the message queue system, the copy of a topic is distributed in broker1, broker2 and broker3, broker2 is used as a read-write service node, and broker3 is used as a read-only service node. The broker2 synchronizes the messages of the topic for the broker1 and the broker3. The producer client 1 and the producer client 2 produce the messages of the topic for the broker2, the consumer client 1 consumes the messages of the topic through the broker2, and the consumer client 2 and the consumer client 3 consume the messages of the topic through the broker3.

[0172] In addition, the embodiment of the present application further provides a computer readable storage medium, which comprises a stored computer program; wherein the computer program controls the device where the computer readable storage medium is located to execute the running method of the message queue system according to any one of the above embodiments when running.

[0173] Further, the embodiment of the present application provides a computer program product, comprising computer programs / instructions, which, when executed by a processor, implement the running method of the message queue system according to any of the above embodiments.

[0174] Referring to Figure 6 , Figure 6 is a structural diagram of a message queue system running device 30 provided by the embodiment of the present application, the message queue system running device 30 comprises a processor 31, a memory 32, and a computer program stored in the memory 32 and executable on the processor 31. The processor 31 implements the steps in the running method embodiment of the message queue system when executing the computer program. Alternatively, the processor 31 implements the functions of each module / unit in each device embodiment when executing the computer program.

[0175] For example, the computer program can be divided into one or more modules / units, which are stored in the memory 32 and executed by the processor 31 to complete the present application. The one or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the message queue system running device 30.

[0176] The message queue system running device 30 can include, but is not limited to, the processor 31 and the memory 32. Those skilled in the art can understand that the schematic diagram is only an example of the message queue system running device 30, and does not limit the message queue system running device 30, which can include more or fewer components than the diagram, or combine certain components, or different components, for example, the message queue system running device 30 can also include an input / output device, a network access device, a bus, etc.

[0177] The processor 31 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The processor 31 is a control center of the message queue system running device 30, and connects various parts of the message queue system running device 30 through various interfaces and lines.

[0178] The memory 32 can be used to store computer programs and / or modules. The processor 31 realizes various functions of the message queue system running device 30 by running or executing computer programs and / or modules stored in the memory 32, and calling data stored in the memory 32. The memory 32 can mainly include a program storage area and a data storage area. The program storage area can store an operating system, at least one application program required for a function (such as a sound playing function, an image playing function, etc.), etc. The data storage area can store data created according to use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 32 can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0179] The modules / units integrated in the running device 30 of the message queue system can be stored in a computer readable storage medium if they are implemented in the form of software function units and sold or used as independent products. Based on this understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware. The computer program can be stored in a computer readable storage medium. When the computer program is executed by the processor 31, the steps of the above-mentioned various method embodiments can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms, etc. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0180] It should be noted that the apparatus embodiments described above are merely illustrative, and the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. In addition, the connection relationship between the modules in the apparatus embodiment provided by the present application indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines. Those of ordinary skill in the art can understand and implement it without creative labor.

[0181] The above is the preferred embodiment of the present application. It should be noted that those of ordinary skill in the art can make several improvements and refinements without departing from the principles of the present application, and these improvements and refinements are also considered within the protection scope of the present application.

Claims

1. A method for operating a message queue system, characterized in that, include: Deploy servers in different regions as the same message queue cluster; In the message queue cluster server, multiple replicas are set up for the topic to which the message to be transmitted belongs; wherein, each region in the message queue cluster has at least one replica. Select one of the multiple replicas as the first replica, and use the server where the first replica is located as a read / write service node to provide read / write services for the message to be transmitted to the client. The method further includes: At least one second replica is selected from the plurality of replicas, and the server where the second replica is located is used as a read-only service node to provide the client with the reading service of the message to be transmitted; wherein, the second replica is a replica that is not in the same region as the first replica.

2. The method for operating the message queue system as described in claim 1, characterized in that, In the message queue cluster, one second replica is selected for each region where the first replica is not present.

3. The method for operating the message queue system as described in claim 2, characterized in that, The method further includes: The connection of the consumer client is routed to the target service node located in the same region as itself; wherein, the target service node is the read-write service node or the read-only service node, and the consumer client is the client that needs to read the message to be transmitted from the message queue cluster.

4. The method for operating the message queue system as described in any one of claims 1 to 3, characterized in that, The method includes: The connection of the producer client is routed to the read / write service node; wherein, the producer client is the client that needs to write the message to be transmitted to the read / write service node.

5. The method for operating the message queue system as described in claim 1, characterized in that, The step of selecting one of the plurality of replicas as the first replica includes: For each of the plurality of replicas, the device information of the clients that need to request services from the server where that replica resides is collected; wherein, the device information includes at least one of the following: region information and performance information; Based on the device information, the optimal area is determined; Select the first copy from the optimal region.

6. The method for operating the message queue system as described in claim 5, characterized in that, The step of determining the optimal area based on the device information includes: The optimal region is determined based on the preset objective function and the device information; The objective function includes at least one of the following: The first objective function aims to minimize the total network latency between all first target clients that need to request services from the server where the same replica resides and the server where that replica resides; wherein, the first target client is a producer client, or, the first target client is the producer client and the consumer client; the consumer client is the client that needs to read the message to be transmitted from the message queue cluster, and the producer client is the client that needs to write the message to be transmitted to the read-write service node; The second objective function aims to maximize the total traffic of all second target clients within any given region; wherein the second target clients are the producer clients and / or the consumer clients. The third objective function aims to maximize the total number of all third target clients within any given region that require cross-regional services; the third target clients are the producer clients and the consumer clients.

7. The method for operating the message queue system as described in claim 5, characterized in that, Selecting the first copy from the optimal region includes: The first replica is selected from the optimal region based on the distribution characteristics of the read / write service nodes and / or the computing resources of the server.

8. An operating device for a message queue system, characterized in that, include: The deployment module is used to deploy servers in different regions as the same message queue cluster; The configuration module is used to configure multiple replicas for the topic to which the message to be transmitted belongs in the server of the message queue cluster; wherein, each region in the message queue cluster has at least one replica. The first selection module is used to select one of the multiple replicas as the first replica, and to use the server where the first replica is located as a read / write service node to provide read / write services for the message to be transmitted to the client. The second selection module is used to select at least one second replica from the plurality of replicas, and to use the server where the second replica is located as a read-only service node to provide the client with the reading service of the message to be transmitted; wherein the second replica is a replica that is not in the same region as the first replica.

9. A message queue system, characterized in that, include: The server-side module is used to deploy servers in different regions as the same message queue cluster; In the message queue cluster server, multiple replicas are set up for the topic to which the message to be transmitted belongs; wherein, each region in the message queue cluster has at least one replica. The decision module is used to select one of the multiple replicas as the first replica and designate the server where the first replica is located as the read / write service node. The client module is used to request the read / write service of the message to be transmitted from the read / write service node; The metadata management module is used to record and monitor the data of the server module, the decision module, and the client module; The decision module is also used for: At least one second replica is selected from the plurality of replicas, and the server where the second replica is located is used as a read-only service node to provide the client with the reading service of the message to be transmitted; wherein, the second replica is a replica that is not in the same region as the first replica.

10. An operating device for a message queue system, characterized in that, The system includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements a method for operating the message queue system as described in any one of claims 1 to 7.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program; wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the operation method of the message queue system as described in any one of claims 1 to 7.

12. A computer program product, characterized in that, It includes a computer program / instruction that, when executed by a processor, implements the method of operating the message queue system as described in any one of claims 1 to 7.

Images