Message distribution method, apparatus, device and product

Abstract

The application discloses a message distribution method and device, equipment and product, and relates to the technical field of communication. The message distribution method is applied to a non-central architecture publishing and subscribing system, the publishing and subscribing system comprises a subscriber cluster and an arbitration center, and the message distribution method comprises the following steps: determining the subscription priority of a plurality of subscribers in the subscriber cluster corresponding to a subscription topic based on a consistent hashing algorithm and the subscription topic; determining a first type of subscriber according to a preset lease mechanism based on the arbitration center and the subscription priority; and distributing the message of the subscription topic to the first type of subscriber, so that the first type of subscriber performs consumption. Through the above scheme, the efficiency of message distribution can be effectively improved, and efficient load balancing without centralization is realized.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to a message distribution method, apparatus, device and product. Background Technology

[0002] In distributed publish-subscribe systems, to improve availability and throughput, multiple subscriber instances are typically deployed horizontally to consume messages on the same topic. However, this creates a load balancing requirement.

[0003] Existing load balancing solutions have significant limitations: First, centralized load balancers pose a single point of failure risk, which will lead to service interruption once a failure occurs; second, client-side load balancing, where subscribers make decisions based on potentially delayed registration information, is prone to a "herding effect," where a large number of requests surge to a few low-load nodes, creating new imbalances.

[0004] Therefore, it is necessary to propose a solution that can achieve decentralized and efficient load balancing. Summary of the Invention

[0005] The main purpose of this application is to provide a message distribution method, apparatus, device and product, which aims to solve the technical problem of the lack of technical solutions in the industry to achieve decentralized and efficient load balancing.

[0006] To achieve the above objectives, this application proposes a message distribution method applied to a publish-subscribe system with a decentralized architecture. The publish-subscribe system includes a subscriber cluster and an arbitration center. The message distribution method includes:

[0007] Based on the consistent hashing algorithm and the subscription topic, the subscription priority of several subscribers in the subscriber cluster is determined;

[0008] Based on the arbitration center and the subscription priority, the first type of subscribers are determined according to the preset lease mechanism;

[0009] The message of the subscribed topic is distributed to the first type of subscribers so that the first type of subscribers can consume it.

[0010] In one embodiment, the step of determining the subscription priority of several subscribers in the subscriber cluster based on the consistent hashing algorithm and the subscription topic includes:

[0011] Based on the subscribed topic, several subscribers are selected from the subscriber cluster;

[0012] Based on the consistent hashing algorithm, several hash values ​​are calculated according to the identifiers and subscription information of the subscribers.

[0013] Based on the aforementioned hash values, the subscription priorities of several subscribers in the subscriber cluster are determined.

[0014] In one embodiment, the step of determining the subscription priority of several subscribers in the subscriber cluster based on the several hash values ​​includes:

[0015] Based on a preset modulus, a modulo operation is performed on the plurality of hash values ​​to obtain the modulus value corresponding to the plurality of subscribers;

[0016] The subscription priority of each subscriber is determined according to the order of the magnitude of the modulus.

[0017] In one embodiment, the preset lease mechanism includes a preset lease adjudication strategy, and the step of determining the first type of subscriber based on the arbitration center and the subscription priority according to the preset lease mechanism includes:

[0018] In response to several subscription requests, based on the arbitration center, the ownership of message subscription rights is adjudicated according to the subscription priority and a preset lease adjudication strategy, and a subscription lease for the subscription topic is established. The subscription lease is used to determine the message distribution object of the subscription topic.

[0019] Based on the aforementioned subscription lease, the first type of subscribers is identified.

[0020] In one embodiment, the subscription lease includes a lease term. Following the step of establishing a subscription lease for the subscription topic in response to several subscription requests, based on the arbitration center, and according to the subscription priority and a preset lease adjudication strategy, the method further includes:

[0021] During the lease term of the subscription agreement, obtain the heartbeat signals reported by the first type of subscribers according to a preset period;

[0022] If the heartbeat signal is normal, the subscription lease will be renewed based on the arbitration center.

[0023] If the heartbeat signal is abnormal, the subscription lease will be released based on the arbitration center; and / or

[0024] During the lease term of the subscription agreement, obtain message consumption information of the subscription topic reported by the first type of subscribers.

[0025] In one embodiment, after the step of releasing the subscription lease based on the arbitration center if the heartbeat signal is abnormal, the method further includes:

[0026] Based on the arbitration center, a lease release notice is sent to the plurality of subscribers to trigger the plurality of subscribers to send a subscription request to the arbitration center;

[0027] Execution steps: In response to several subscription requests, based on the arbitration center, the subscription rights of the messages are decided according to the subscription priority and a preset lease adjudication strategy, and a subscription lease for the subscription topic is established;

[0028] Based on the aforementioned subscription lease, the first type of subscribers is identified.

[0029] In one embodiment, the publish-subscribe system includes a message middleware, and the message distribution method further includes:

[0030] Based on the message middleware, messages are distributed to the first type of subscribers according to the message consumption information, so that the first type of subscribers can consume them; and / or

[0031] Based on the first type of subscriber, messages are obtained from the message middleware and consumed according to the message consumption information.

[0032] Furthermore, to achieve the above objectives, this application also proposes a message distribution device applied to a publish-subscribe system with a decentralized architecture, the publish-subscribe system including a subscriber cluster and an arbitration center, the message distribution device comprising:

[0033] The first determining module is used to determine the subscription priority of several subscribers in the subscriber cluster based on the consistent hashing algorithm and the subscription topic.

[0034] The second determination module is used to determine the first type of subscribers based on the arbitration center and the subscription priority, according to a preset lease mechanism.

[0035] The distribution module is used to distribute messages of the subscribed topic to the first type of subscribers so that the first type of subscribers can consume them.

[0036] In addition, to achieve the above objectives, this application also proposes a message distribution device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the message distribution method as described above.

[0037] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the message distribution method described above.

[0038] One or more technical solutions proposed in this application have at least the following technical effects:

[0039] The message distribution method, apparatus, device, and product proposed in this application specifically involve determining the subscription priorities of several subscribers in a subscriber cluster based on a consistent hashing algorithm and a subscription topic in a publish-subscribe system with a decentralized architecture; determining a first type of subscriber based on an arbitration center and the subscription priorities according to a preset lease mechanism; and distributing messages of the subscription topic to the first type of subscriber so that the first type of subscriber can consume them.

[0040] In a decentralized publish-subscribe system, this application uses a consistent hashing algorithm and a subscription topic to determine the subscription priorities of several subscribers in a subscriber cluster, enabling rapid determination of the message distribution range corresponding to the subscription topic. Then, based on an arbitration center and the subscription priorities, a first type of subscriber is determined according to a preset lease mechanism. Messages related to the subscription topic are distributed to this first type of subscriber for consumption. The arbitration center then redistributes message subscription rights according to the preset lease mechanism, achieving intelligent and even distribution of subscription load. This effectively avoids the load imbalance caused by the "herding effect," thus achieving decentralized and efficient load balancing. Attached Figure Description

[0041] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a flowchart illustrating the first embodiment of the message distribution method of this application;

[0044] Figure 2 This is a flowchart illustrating the second embodiment of the message distribution method of this application.

[0045] Figure 3 This is a flowchart illustrating the third embodiment of the message distribution method of this application;

[0046] Figure 4 This is a schematic diagram of the module structure of the message distribution device of this application;

[0047] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the message distribution method of this application embodiment.

[0048] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0049] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0050] The main solution of this application embodiment is as follows: In a publish-subscribe system with a decentralized architecture, based on a consistent hashing algorithm and a subscription topic, the subscription priority of several subscribers in the subscriber cluster is determined; based on an arbitration center and the subscription priority, a first type of subscriber is determined according to a preset lease mechanism; messages of the subscription topic are distributed to the first type of subscriber so that the first type of subscriber can consume them.

[0051] In existing digital information and communication technologies, the publish / subscribe pattern, as a classic asynchronous communication model, decouples publishers and subscribers at the time, space, and control flow levels. Publishers only need to push messages to designated topics without needing to worry about the number of subscribers or their online status; subscribers, on the other hand, subscribe to specific topics according to their own needs and receive matching messages. This pattern demonstrates significant advantages in scenarios such as event-driven architectures and real-time data distribution.

[0052] In distributed systems, to achieve high availability and high throughput, the subscribers to a topic are typically horizontally scaled, meaning multiple subscriber instances are deployed to consume messages from a single topic. This raises the need for load balancing, i.e., how to reasonably distribute the message flow under a topic among multiple subscribers to avoid some subscribers being overloaded while others are idle.

[0053] Traditional load balancing solutions have at least the following drawbacks:

[0054] First, centralized load balancers face the risk of single point of failure. They typically rely on independent load balancers (such as Nginx, HAProxy, etc., hardware or software) for message distribution. If this device fails, the entire message transmission will be interrupted. Second, client-side load balancing is prone to the "herding effect." When subscribers select nodes from the service registry, if they all make decisions based on the same, potentially lagging, load information, a large number of requests will flood into a few nodes with the lowest load in a short period of time, which will exacerbate the load imbalance.

[0055] Therefore, it is necessary to propose a solution that can achieve decentralized and efficient load balancing.

[0056] To address the aforementioned technical problems, this application proposes a solution in a decentralized publish-subscribe system. By using a consistent hashing algorithm and a subscription topic, the subscription priorities of several subscribers in the subscriber cluster can be determined, enabling rapid determination of the message distribution range corresponding to the subscription topic. Then, based on an arbitration center and the subscription priorities, a first type of subscriber is determined according to a preset lease mechanism. Messages related to the subscription topic are distributed to this first type of subscriber for consumption. The arbitration center then redistributes message subscription rights according to the preset lease mechanism, achieving intelligent and even distribution of subscription load. This effectively avoids the load imbalance caused by the "herding effect," thereby achieving decentralized and efficient load balancing.

[0057] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, personal computer, mobile phone, etc., or a message distribution device, publish-subscribe system, etc. that can realize the above functions.

[0058] Based on this, embodiments of this application provide a message distribution method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the message distribution method of this application. The message distribution method is applied to a publish-subscribe system with a decentralized architecture. The publish-subscribe system includes a subscriber cluster and an arbitration center. The message distribution method includes steps S110 to S130:

[0059] Step S110: Based on the consistent hashing algorithm and the subscription topic, determine the subscription priority of several subscribers in the subscriber cluster;

[0060] It should be noted that a decentralized publish-subscribe system refers to a communication system that does not rely on centralized middleware or coordinator, but achieves message publishing and subscription through direct interaction of distributed nodes. In the embodiments of this application, the decentralized publish-subscribe system includes a publisher cluster, a subscriber cluster, an arbitration center, and a message middleware.

[0061] Specifically, a decentralized publish-subscribe system transmits messages through distributed network nodes. The publisher cluster consists of several publishers responsible for publishing messages to one or more subscription topics. The subscriber cluster comprises several subscriber instances, which can collectively subscribe to messages on the same topic or messages on multiple different topics. The message middleware provides basic topic-based publish / subscribe capabilities. Here, a subscription topic refers to an identifier used in the publish-subscribe system for logically classifying messages. Publishers send messages by topic, and subscribers receive messages by topic. It should be understood that the subscriber instance in this application has the same meaning as a subscriber.

[0062] Since message transmission between the publisher and subscriber clusters occurs through distributed network nodes, both centralized load balancers and client-side load balancing solutions have shortcomings. Therefore, this application proposes a load balancing scheme based on an arbitration center, combined with a consistent hashing algorithm and a pre-defined lease mechanism. In this application, the arbitration center is a highly available distributed key-value store system supporting consistent consensus, responsible for message scheduling and determining the message distribution object for subscribed topics, preferably Etcd or Zookeeper. Etcd is a distributed key-value store system developed by the CoreOS team, initially designed for Kubernetes configuration and service discovery, and now responsible for storing all cluster state data. Zookeeper is software that provides distributed application coordination services and is a crucial cornerstone of many large-scale distributed systems such as Hadoop and HBase. Zookeeper ensures strong data consistency by using its proprietary Zab (Zookeeper AtomicBroadcast) consensus protocol, while providing a tree-like hierarchical namespace similar to a file system.

[0063] Specifically, this application is applied to a publish-subscribe system with a decentralized architecture. In order to achieve efficient load balancing in a decentralized manner, the publish-subscribe system needs to select a number of subscribers from the subscriber cluster through a subscription topic, and then use a consistent hashing algorithm to determine the subscription priority of the subscribers corresponding to the subscription topic. This determines the subscription priority of the subscriber instances associated with the subscription topic, so that the first type of subscriber can be further determined based on the priority, thereby achieving efficient message distribution.

[0064] Consistent hashing, in particular, maps the hash value space to a virtual ring structure. Specifically, it maps nodes and data to the same hash ring, with data distributed to the nearest node in a clockwise direction. When a node changes, only the data area from the failed node to the next node in the counter-clockwise direction is affected. This ensures that adding or removing nodes only affects adjacent nodes on the ring, thereby minimizing data migration and solving the problem of maintaining data mapping relationships when nodes are dynamically added or removed in a distributed system.

[0065] For example, the publish-subscribe system uses a consistent hashing algorithm to map multiple subscriber instances associated with the same subscription topic to the same hash ring. This allows for the rapid determination of the message distribution range corresponding to the subscription topic when a subscriber instance fails. Simultaneously, based on performance indicators such as the clockwise distance between the nodes corresponding to multiple subscriber instances on the hash ring, the current CPU (Central Processing Unit) load of the nodes, or the remaining memory, a priority score is generated using a weighted summation method. This score is then used to further determine the first type of subscriber, thereby achieving efficient message distribution and high-throughput message consumption.

[0066] In one feasible implementation, the step of determining the subscription priority of several subscribers in the subscriber cluster based on the consistent hashing algorithm and the subscription topic includes steps S1101~S1103:

[0067] Step S1101: Based on the subscribed topic, select several subscribers from the subscriber cluster;

[0068] Step S1102: Based on the consistent hashing algorithm, calculate the corresponding hash values ​​according to the identifiers and subscription information of the subscribers.

[0069] In this implementation, the publish-subscribe system first selects several subscriber instances from the subscriber cluster based on the subscription topic. Then, for each subscriber, a consistent hashing algorithm is used to calculate the corresponding hash value based on the subscriber's unique identifier and the information of the topic subscribed to. The identifier is a string or number that uniquely distinguishes the subscriber node, and can be an IP (Internet Protocol) address, instance ID, or a combination of IP address and port information. The subscription information can include the subscription topic name, subscription topic identifier field, etc.

[0070] Optionally, the publish-subscribe system takes a byte array concatenated with the subscriber's identifier and the topic name as input, and calculates a fixed-length hash value using a consistent hashing algorithm; this hash value corresponds to the subscriber. By performing the above process on several subscribers, several hash values ​​can be obtained for each subscriber.

[0071] Step S1103: Determine the subscription priority of several subscribers in the subscriber cluster based on the several hash values.

[0072] Then, the publish-subscribe system normalizes several hash values ​​to determine their order of magnitude, and then determines the subscriber's message subscription priority based on this order of magnitude.

[0073] Optionally, based on a preset modulus, a modulo operation is performed on the plurality of hash values ​​to obtain the modulus values ​​corresponding to the plurality of subscribers; the subscription priority of the plurality of subscribers is determined according to the order of the magnitude of the modulus values. The preset modulus is pre-set by relevant personnel based on actual needs or experience, and is typically the total number of subscribers associated with the subscription topic.

[0074] Specifically, based on a preset modulus, the remainders corresponding to several hash values ​​are obtained by performing a modulo operation on several hash values, which are the modulo values ​​corresponding to the several subscribers. The order of the modulo values ​​obtained from the several modulo values ​​is used to determine the message subscription priority of the subscribers.

[0075] Step S120: Based on the arbitration center and the subscription priority, determine the first type of subscribers according to the preset lease mechanism;

[0076] It should be noted that the pre-set lease mechanism refers to a distributed lock protocol based on time leases established between the arbitration center and subscribers. The lease holder has the exclusive right to consume the message stream of a certain subscribed topic.

[0077] Specifically, in a decentralized publish-subscribe system, the publish-subscribe system can rely on an arbitration center to dynamically allocate message subscription rights according to the subscription priorities of several subscribers and a preset lease mechanism. This ensures that healthy and high-priority subscribers become lease holders, i.e., first-class subscribers, thereby avoiding single points of failure and the "herd effect" and achieving decentralized and efficient load balancing.

[0078] Step S130: Distribute the message of the subscribed topic to the first type of subscribers so that the first type of subscribers can consume it.

[0079] Specifically, based on the first type of subscribers obtained in step S120, the publish-subscribe system only distributes real-time or historical messages corresponding to the subscribed topic to the subscriber nodes currently holding leases. Other nodes maintain their subscription relationships but will not receive messages for that subscribed topic until their leases expire. After receiving messages for the subscribed topic, the first type of subscribers consumes and processes the messages, specifically executing pre-defined business logic, such as updating local caches, writing to the database, or triggering downstream events.

[0080] In this embodiment, in a decentralized publish-subscribe system, the subscription priorities of several subscribers in the subscriber cluster can be determined through a consistent hashing algorithm and the subscription topic, so as to quickly determine the message distribution range corresponding to the subscription topic. Then, based on the arbitration center and the subscription priorities, a first type of subscriber is determined according to a preset lease mechanism. The messages of the subscription topic are distributed to the first type of subscribers so that they can consume them. The arbitration center redistributes the message subscription rights according to the preset lease mechanism, realizing intelligent and uniform distribution of subscription load, effectively avoiding the load unevenness problem caused by the "herd effect", thereby achieving decentralized and efficient load balancing.

[0081] Based on the above embodiments of this application, a second embodiment of this application is proposed. In the second embodiment of this application, content that is the same as or similar to the above embodiments can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 , Figure 2 This is a flowchart illustrating the second embodiment of the message distribution method of this application; as shown... Figure 2 As shown, the preset lease mechanism includes a preset lease adjudication strategy. The step of determining the first type of subscriber based on the arbitration center and the subscription priority according to the preset lease mechanism includes steps S210~S220:

[0082] Step S210: In response to several subscription requests, based on the arbitration center, the subscription rights of messages are decided according to the subscription priority and a preset lease adjudication strategy, and a subscription lease for the subscription topic is established. The subscription lease is used to determine the message distribution object of the subscription topic.

[0083] Step S220: Determine the first type of subscribers based on the subscription lease.

[0084] It should be noted that the preset lease ruling strategy refers to a set of rules pre-set for the subscription topic regarding the ownership of subscription rights, which is used to select a unique or limited number of lease holders from multiple candidate subscribers. This set of rules may include performance limits such as node CPU load, memory capacity and network latency, priority-based lease preemption rules, maximum node concurrency limit, lease validity period, and renewal conditions.

[0085] Specifically, several subscribers in a healthy state send subscription requests to the arbitration center based on atomic operations (such as Compare-And-Swap) to attempt to preempt the lease for the subscription topic. The arbitration center responds to these subscription requests by determining the subscription rights based on the subscription priority included in the requests, according to the priority-based lease preemption rules in the preset lease adjudication strategy, and other constraints. The center selects the subscriber with the highest priority who meets the constraints to win the subscription right and establishes a subscription lease for the subscription topic with that subscriber. The leaseholder of the subscription lease is the first type of subscriber.

[0086] Furthermore, to ensure the reliability of message consumption, this application also proposes a fault handling mechanism for publish-subscribe systems with a decentralized architecture, which realizes fault detection and handling through the heartbeat keep-alive mechanism of the first type of subscribers and the reporting mechanism of message consumption information.

[0087] The subscription lease includes a lease term. Following the step of responding to several subscription requests, and based on the arbitration center, adjudicating the subscription rights according to the subscription priority and a preset lease adjudication strategy, and establishing a subscription lease for the subscription topic, the system further includes steps A01-A04:

[0088] Step A01: During the lease term of the subscription lease, obtain the heartbeat signals reported by the first type of subscribers according to a preset period;

[0089] Specifically, during the subscription period, the first type of subscriber is required to periodically report heartbeat signals according to a preset cycle. This ensures that the arbitration center confirms that the first type of subscriber has not experienced any faults and can still consume messages from the subscribed topic, thus guaranteeing the reliability of message consumption. The heartbeat signal may include verification fields, timestamps, and the subscription topic ID, among other information.

[0090] Step A02: If the heartbeat signal is not abnormal, the subscription lease is renewed based on the arbitration center.

[0091] Step A03: If the heartbeat signal is abnormal, the subscription lease is released based on the arbitration center;

[0092] If no abnormality is found in the heartbeat signal, meaning that the heartbeat signals reported by the first type of subscriber are uploaded according to the preset cycle throughout the entire lease term, and the verification fields, timestamps, and subscription topic IDs included in each heartbeat signal match the subscription lease, the arbitration center can determine that the first type of subscriber has not encountered any fault problems, and at this time, the subscription lease will be renewed with the first type of subscriber.

[0093] When an abnormal heartbeat signal occurs, meaning that the heartbeat signal reported by the first type of subscriber is missing once or multiple times during the entire lease period, or that the verification fields, timestamps, subscription topic IDs, etc. included in a certain heartbeat signal do not match the subscription lease, the arbitration center can determine that the first type of subscriber has a fault. At this time, it is necessary to release the subscription lease with the first type of subscriber, specifically by marking the subscription lease as invalid and revoking the corresponding subscription rights, so as to find a healthy and high-priority subscriber at the current moment to continue message consumption.

[0094] And / or step A04, during the lease term of the subscription lease, obtain message consumption information of the subscription topic reported by the first type of subscriber.

[0095] In existing distributed publish-subscribe systems, the decoupling of publishers and subscribers at the time, space, and control flow levels makes it impossible for publishers to perceive the status of subscribers. If a subscriber encounters a network partition or abnormal crash while processing messages, and the system does not remove it from the consumer group in time, the relevant messages may neither be processed nor redelivered, resulting in message loss. This fails to meet the reliability requirements of message consumption in critical scenarios such as finance.

[0096] Specifically, during the subscription period, the first type of subscriber is required to report message consumption information for the subscribed topic. This can be done periodically according to a preset cycle, or synchronously after each successful message processing, so that the arbitration center can confirm the consumption progress of the message stream under that subscription topic. The message consumption information refers to the specific consumption information of the messages processed by the first type of subscriber, including processing operation, processing time, processing success or failure, the latest consumed message position, and failure reason code.

[0097] In this embodiment, the present application, through the above-described scheme, responds to several subscription requests and, based on the arbitration center, adjudicates the subscription rights of messages according to the subscription priority and a preset lease adjudication strategy, establishing a subscription lease for the subscription topic. The subscription lease is used to determine the message distribution object of the subscription topic. Based on the subscription lease, a first type of subscriber is determined. By deploying the arbitration center in a cluster mode, which has the characteristics of high availability, the single point of failure problem of traditional load balancers is avoided. At the same time, the preset lease adjudication strategy is used to adjudicate the subscription rights of messages, realizing intelligent and even distribution of subscription load, effectively avoiding the load unevenness problem caused by the "herd effect", thereby achieving decentralized and efficient load balancing.

[0098] Based on the above embodiments of this application, a third embodiment of this application is proposed. In this third embodiment, content that is the same as or similar to the above embodiments can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 , Figure 3 This is a flowchart illustrating the third embodiment of the message distribution method of this application; as shown... Figure 3 As shown, after the step of releasing the subscription lease based on the arbitration center if the heartbeat signal is abnormal, the method further includes steps S310~S330:

[0099] Step S310: Based on the arbitration center, send a lease release notice to the plurality of subscribers to trigger the plurality of subscribers to send a subscription request to the arbitration center;

[0100] Specifically, the publish-subscribe system relies on an arbitration center to send a lease release notification message to several subscribers who have subscribed to the subscription topic. This triggers healthy subscribers to send subscription requests to the arbitration center, requesting to seize the subscription rights for the subscription topic and achieve exclusive consumption of the message.

[0101] Step S320, execution steps: In response to several subscription requests, based on the arbitration center, according to the subscription priority and a preset lease adjudication strategy, the ownership of the subscription rights of the message is adjudicated, and a subscription lease for the subscription topic is established;

[0102] Step S330: Determine the first type of subscriber based on the subscription lease.

[0103] When the arbitration center receives several subscription requests from several subscribers, it re-executes the following steps: in response to the several subscription requests, based on the arbitration center, it adjudicates the subscription rights of the message according to the subscription priority and a preset lease adjudication strategy, and establishes a subscription lease for the subscription topic; based on the subscription lease, it determines the first type of subscribers, and implements the steps described in the second embodiment to re-determine the first type of subscribers.

[0104] This embodiment uses a preset lease mechanism to seamlessly and quickly transfer the message subscription rights of a failed subscriber instance to other healthy subscribers, thereby achieving high availability and high throughput message consumption.

[0105] Furthermore, to ensure the reliability of message consumption to address specific business domains, this application achieves zero message loss during failover through a pre-defined lease mechanism and message consumption location management, thereby realizing highly available, high-throughput, and reliable message consumption. The publish-subscribe system includes message middleware, and the message distribution method further includes steps B01-B02:

[0106] Step B01: Based on the message middleware, distribute messages to the first type of subscribers according to the message consumption information, so that the first type of subscribers can consume the messages; and / or

[0107] Step B02: Based on the first type of subscriber, messages are obtained from the message middleware and consumed according to the message consumption information.

[0108] Specifically, the decentralized publish-subscribe system actively distributes messages based on a message middleware. The message middleware only needs to push messages for the subscribed topic to the first type of subscribers based on the latest consumed message position and lease information in the message consumption information. Even when subscribers frequently go online and offline, the middleware can switch message distribution objects in seconds through lease information, ensuring that messages are not duplicated, not lost, and in the same order, and the push latency can be stabilized in the millisecond range.

[0109] Optionally, after the first type of subscribers are re-identified in the publish-subscribe system with a decentralized architecture, the first type of subscribers can actively pull messages from the message middleware for consumption based on the latest message position consumed in the message consumption information.

[0110] In this embodiment, the present application, through the above-described scheme, sends a lease release notice to several subscribers through the arbitration center when the first type of subscriber fails, triggering a new round of lease preemption for the subscription topic, and quickly handling the fault. This ensures that the failure of a single subscriber will not result in the subscription topic messages being unconsumed. At the same time, combined with the reliable storage of message consumption information, it ensures that messages will not be lost during the fault transfer process, thereby achieving decentralized and efficient load balancing, and ensuring that messages are never lost when the subscriber fails.

[0111] It should be noted that the above embodiments can be combined and implemented in a reasonable manner according to the actual situation, and this embodiment will not elaborate on this further.

[0112] This application also provides a message distribution device, please refer to... Figure 4 The message distribution device is applied to a publish-subscribe system with a decentralized architecture, the publish-subscribe system including a subscriber cluster and an arbitration center, and the message distribution device includes:

[0113] The first determining module 10 is used to determine the subscription priority of several subscribers in the subscriber cluster based on the consistent hashing algorithm and the subscription topic;

[0114] The second determining module 20 is used to determine the first type of subscribers based on the arbitration center and the subscription priority, according to a preset lease mechanism.

[0115] The distribution module 30 is used to distribute messages of the subscribed topic to the first type of subscribers so that the first type of subscribers can consume them.

[0116] The message distribution device provided in this application, employing the message distribution method described in the above embodiments, can solve the technical problem of the lack of a centralized and efficient load balancing solution in the industry. Compared with the prior art, the beneficial effects of the message distribution device provided in this application are the same as those of the message distribution method provided in the above embodiments, and other technical features in the message distribution device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0117] This application provides a message distribution device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the message distribution method described above.

[0118] The following is for reference. Figure 5 The diagram illustrates a structural schematic of a message distribution device suitable for implementing embodiments of this application. The message distribution device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The message distribution device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0119] like Figure 5As shown, the message distribution device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in the read-only memory 1002 or a program loaded from the storage device 1003 into the random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the message distribution device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the message distribution device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows message distribution devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0120] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0121] The message distribution device provided in this application, employing the message distribution method described in the above embodiments, can solve the technical problem of the lack of a centralized and efficient load balancing solution in the industry. Compared with the prior art, the beneficial effects of the message distribution device provided in this application are the same as those of the message distribution method provided in the above embodiments, and other technical features of the message distribution device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0122] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0123] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0124] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the message distribution method in the above embodiments.

[0125] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0126] The aforementioned computer-readable storage medium may be included in the message distribution device; or it may exist independently and not be assembled into the message distribution device.

[0127] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by a message distribution device, the message distribution device: in a decentralized publish-subscribe system, determines the subscription priorities of several subscribers in the subscriber cluster based on a consistent hashing algorithm and the subscription topic; determines a first type of subscriber based on an arbitration center and the subscription priorities according to a preset lease mechanism; and distributes messages of the subscription topic to the first type of subscriber so that the first type of subscriber can consume them.

[0128] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0129] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0130] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0131] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described message distribution method. This addresses the technical problem of the lack of a centralized, efficient load balancing solution in the industry. Compared with existing technologies, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the message distribution method provided in the above embodiments, and will not be elaborated upon here.

[0132] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the message distribution method described above.

[0133] The computer program product provided in this application can solve the technical problem of the lack of technical solutions in the industry for achieving decentralized and efficient load balancing. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the message distribution method provided in the above embodiments, and will not be repeated here.

[0134] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A message distribution method, characterized in that, The message distribution method is applied to a publish-subscribe system with a decentralized architecture, the publish-subscribe system including a subscriber cluster and an arbitration center, and the message distribution method includes: Based on the consistent hashing algorithm and the subscription topic, the subscription priority of several subscribers in the subscriber cluster is determined; Based on the arbitration center and the subscription priority, the first type of subscribers are determined according to the preset lease mechanism; Distribute messages of the subscribed topic to the first type of subscribers so that the first type of subscribers can consume them; The step of determining the subscription priority of several subscribers in the subscriber cluster based on the consistent hashing algorithm and the subscription topic includes: Based on the subscribed topic, several subscribers are selected from the subscriber cluster; For any subscriber, the hash value corresponding to the subscriber is calculated using a consistent hashing algorithm based on the subscriber's identifier and the byte array concatenated with the subscription topic name. Based on the hash values ​​corresponding to the subscribers, the subscription priority of the subscribers in the subscriber cluster is determined, specifically including: Based on a preset modulus, a modulo operation is performed on the plurality of hash values ​​to obtain the modulus value corresponding to the plurality of subscribers; The subscription priority of each subscriber is determined according to the order of the magnitude of the modulus. The preset lease mechanism includes a preset lease adjudication strategy. The step of determining the first type of subscriber based on the arbitration center and the subscription priority according to the preset lease mechanism includes: In response to several subscription requests, based on the arbitration center, the ownership of message subscription rights is adjudicated according to the subscription priority and a preset lease adjudication strategy, and a subscription lease for the subscription topic is established. The subscription lease is used to determine the message distribution object of the subscription topic. Based on the aforementioned subscription lease, the first type of subscribers is identified.

2. The message distribution method as described in claim 1, characterized in that, The subscription lease includes a lease term. Following the step of responding to several subscription requests, and based on the arbitration center, adjudicating the subscription rights according to the subscription priority and a preset lease adjudication strategy, and establishing a subscription lease for the subscription topic, the service further includes: During the lease term of the subscription agreement, obtain the heartbeat signals reported by the first type of subscribers according to a preset period; If the heartbeat signal is normal, the subscription lease will be renewed based on the arbitration center. If the heartbeat signal is abnormal, the subscription lease will be released based on the arbitration center; and / or During the lease term of the subscription agreement, obtain message consumption information of the subscription topic reported by the first type of subscribers.

3. The message distribution method as described in claim 2, characterized in that, Following the step of releasing the subscription lease based on the arbitration center if the heartbeat signal becomes abnormal, the method further includes: Based on the arbitration center, a lease release notice is sent to the plurality of subscribers to trigger the plurality of subscribers to send a subscription request to the arbitration center; Execution steps: In response to several subscription requests, based on the arbitration center, the subscription rights of the messages are decided according to the subscription priority and a preset lease adjudication strategy, and a subscription lease for the subscription topic is established; Based on the aforementioned subscription lease, the first type of subscribers is identified.

4. The message distribution method as described in claim 3, characterized in that, The publish-subscribe system includes a message middleware, and the message distribution method further includes: Based on the message middleware, messages are distributed to the first type of subscribers according to the message consumption information, so that the first type of subscribers can consume them; and / or Based on the first type of subscriber, messages are obtained from the message middleware and consumed according to the message consumption information.

5. A message distribution device, characterized in that, The message distribution device is applied to a publish-subscribe system with a decentralized architecture, the publish-subscribe system including a subscriber cluster and an arbitration center, and the message distribution device includes: The first determining module is used to determine the subscription priority of several subscribers in the subscriber cluster based on the consistent hashing algorithm and the subscription topic. The second determination module is used to determine the first type of subscribers based on the arbitration center and the subscription priority, according to a preset lease mechanism. The distribution module is used to distribute messages of the subscribed topic to the first type of subscribers so that the first type of subscribers can consume them; The first determining module is further configured to select several subscribers from the subscriber cluster based on the subscribed topic; For any subscriber, the hash value corresponding to the subscriber is calculated using a consistent hashing algorithm based on the subscriber's identifier and the byte array concatenated with the subscription topic name. Based on the hash values ​​corresponding to the subscribers, the subscription priority of the subscribers in the subscriber cluster is determined, specifically including: Based on a preset modulus, a modulo operation is performed on the plurality of hash values ​​to obtain the modulus value corresponding to the plurality of subscribers; The subscription priority of each subscriber is determined according to the order of the magnitude of the modulus. The preset lease mechanism includes a preset lease adjudication strategy. The second determining module is also used to respond to several subscription requests, based on the arbitration center, to adjudicate the subscription rights of messages according to the subscription priority and the preset lease adjudication strategy, and to establish a subscription lease for the subscription topic. The subscription lease is used to determine the message distribution object of the subscription topic. Based on the aforementioned subscription lease, the first type of subscribers is identified.

6. A message distribution device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the message distribution method as described in any one of claims 1 to 4.

7. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the message distribution method as described in any one of claims 1 to 4.

Images