Message consumption method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a message consumption method and device, computer equipment, a computer readable storage medium and a computer program product. The method is applied to a group coordination server and comprises the steps of selecting a main consumer from a target consumption group under the condition that message configuration information of the target consumption group changes; sending the message distribution strategy to the main consumer to indicate the main consumer to update the message distribution strategy according to a rebalance decision corresponding to the changed message configuration information to obtain a new message distribution strategy; based on the new message distribution strategy, determining a target consumer and a new subscription scheme to which the target consumer belongs, converting the consumption state of the target consumer into a heavy balance state, and continuously consuming the message in an available state by other consumers in the target consumption group according to the subscription scheme to which the target consumer belongs; and sending the new subscription scheme to the target consumer so as to indicate the target consumer to update the consumption state to the available state and consume the message according to the new subscription scheme, thereby improving the message consumption efficiency.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to a message consumption method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the development of Internet technology, open source stream processing platform components have become the mainstay in the field of big data processing. These platforms can efficiently process real-time data streams and support complex business requirements in modern applications. In these platforms, the rebalancing mechanism is a core concept that aims to dynamically adjust resource allocation in the system to adapt to load changes or hardware failures, ensuring system stability and efficiency.

[0003] In the related technology, when performing a corresponding rebalancing for a certain consumer group, all consumers in the consumer group are notified that rebalancing is required according to the heartbeat mechanism. At this time, consumers cannot consume messages until the redistribution is completed and all consumers enter a stable state, that is, an available state. Only then can all consumers in the consumer group consume messages normally.

[0004] However, in the related technology, once the consumer group is rebalanced, each consumer in the consumer group cannot consume messages from the open source stream processing platform. Since there are many nodes in the open source stream processing platform and each node has a large number of main partitions, the rebalancing will take a very long time. During this period, the consumer group is in an unconsumable state, resulting in low message consumption efficiency. Summary of the invention

[0005] Based on this, it is necessary to provide a message consumption method, apparatus, computer equipment, computer-readable storage medium and computer program product that can improve message consumption efficiency in response to the above technical problems.

[0006] On the one hand, the present application provides a message consumption method, applied to a group coordination server, comprising:

[0007] When it is verified that the message configuration information of the target consumer group has changed, a primary consumer is selected from the target consumer group for making a rebalancing decision;

[0008] Sending the message allocation strategy of the target consumer group to the main consumer to instruct the main consumer to update the message allocation strategy according to the rebalancing decision corresponding to the changed message configuration information to obtain a new message allocation strategy;

[0009] Based on the new message distribution strategy returned by the main consumer, determine the target consumer that needs to be rebalanced and the new subscription plan to which it belongs, and change the consumption state of the target consumer to the rebalanced state. The other consumers in the target consumer group except the target consumer continue to consume messages in an available state according to their subscription plan;

[0010] The new subscription plan is sent to the target consumer to instruct the target consumer to update the consumption state to an available state and consume the message in an available state according to the new subscription plan.

[0011] On the other hand, the present application also provides a message consumption device, including:

[0012] A selection module is used to select a main consumer from the target consumer group for making a rebalancing decision when it is verified that the message configuration information of the target consumer group has changed;

[0013] A first sending module is used to send the message allocation strategy of the target consumer group to the main consumer to instruct the main consumer to update the message allocation strategy according to the rebalancing decision corresponding to the changed message configuration information to obtain a new message allocation strategy;

[0014] A determination module is used to determine the target consumer that needs to be rebalanced and the new subscription plan it belongs to based on the new message distribution strategy returned by the main consumer, and to change the consumption state of the target consumer to the rebalanced state. Other consumers in the target consumer group except the target consumer continue to consume messages in an available state according to their subscription plan;

[0015] The second sending module is used to send the new subscription plan to the target consumer to instruct the target consumer to update the consumption state to the available state and consume the message in the available state according to the new subscription plan.

[0016] On the other hand, the present application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned message consumption method when executing the computer program.

[0017] On the other hand, the present application also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the above-mentioned message consumption method are implemented.

[0018] On the other hand, the present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned message consumption method when executed by a processor.

[0019] The above-mentioned message consumption method, device, computer equipment, computer-readable storage medium and computer program product, by verifying that the message configuration information of the target consumer group has changed, selects a main consumer from the target consumer group for making a rebalancing decision; sends the message allocation strategy of the target consumer group to the main consumer, so as to instruct the main consumer to update the message allocation strategy according to the rebalancing decision corresponding to the changed message configuration information, and obtains a new message allocation strategy. At this time, there is no need for the consumer to rejoin the target consumer group, which improves the convenience of the rebalancing process. Based on the new message allocation strategy returned by the main consumer, the target consumer that needs to be rebalanced and the new subscription plan to which it belongs are determined, and the consumption state of the target consumer is converted to the rebalancing state, and other consumers in the target consumer group except the target consumer continue to consume messages in an available state according to the subscription plan to which they belong. That is to say, the target consumer group has not entered an unavailable state, and other consumers that do not need to be rebalanced can continue to consume, and the target consumer that needs to be rebalanced is in an unavailable state. Then, by sending the new subscription plan to the target consumer, the target consumer is instructed to update the consumption state to an available state, and consume messages in an available state according to the new subscription plan. In this way, the low efficiency of message consumption caused by the unavailability of the target consumer group during the entire rebalancing process is avoided, and the efficiency of message consumption during the rebalancing process is effectively improved.

[0020] On the other hand, the present application provides a message consumption method, which is applied to a main consumer, including:

[0021] Obtaining a message distribution strategy sent by the group coordination server, and determining a subscription plan for each consumer involved in the message distribution strategy based on the message distribution strategy;

[0022] Based on the subscription plan of each consumer, count the number of partitions subscribed by each consumer, and determine the new message distribution strategy based on the rebalancing decision corresponding to the changed message configuration information based on the number of partitions subscribed by each consumer;

[0023] The new message distribution strategy is sent to the group coordination server to instruct the group coordination server to determine the target consumer that needs to be rebalanced and the new subscription plan it belongs to according to the new message distribution strategy, and to change the consumption state of the target consumer to the rebalanced state. Other consumers in the target consumer group except the target consumer continue to consume messages in an available state according to their subscription plan.

[0024] On the one hand, the present application also provides a message consumption device, including:

[0025] An acquisition module, used to acquire the message distribution strategy sent by the group coordination server, and determine the subscription plan of each consumer involved in the message distribution strategy based on the message distribution strategy;

[0026] The statistics module is used to count the number of partitions subscribed by each consumer based on the subscription plan of each consumer, and determine the new message distribution strategy based on the number of partitions subscribed by each consumer and the rebalancing decision corresponding to the changed message configuration information;

[0027] The third sending module is used to send the new message distribution strategy to the group coordination server to instruct the group coordination server to determine the target consumer that needs to be rebalanced and the new subscription plan it belongs to according to the new message distribution strategy, and to change the consumption state of the target consumer to a rebalanced state. Other consumers in the target consumer group except the target consumer continue to consume messages in an available state according to their subscription plan.

[0028] On the other hand, the present application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned message consumption method when executing the computer program.

[0029] On the other hand, the present application also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the above-mentioned message consumption method are implemented.

[0030] On the other hand, the present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned message consumption method when executed by a processor.

[0031] The above-mentioned message consumption method, device, computer equipment, computer-readable storage medium and computer program product, by verifying that the message configuration information of the target consumer group has changed, selects a main consumer from the target consumer group for making a rebalancing decision; sends the message allocation strategy of the target consumer group to the main consumer, so as to instruct the main consumer to update the message allocation strategy according to the rebalancing decision corresponding to the changed message configuration information, and obtains a new message allocation strategy. At this time, there is no need for the consumer to rejoin the target consumer group, which improves the convenience of the rebalancing process. Based on the new message allocation strategy returned by the main consumer, the target consumer that needs to be rebalanced and the new subscription plan to which it belongs are determined, and the consumption state of the target consumer is converted to the rebalancing state, and other consumers in the target consumer group except the target consumer continue to consume messages in an available state according to the subscription plan to which they belong. That is to say, the target consumer group has not entered an unavailable state, and other consumers that do not need to be rebalanced can continue to consume, and the target consumer that needs to be rebalanced is in an unavailable state. Then, by sending the new subscription plan to the target consumer, the target consumer is instructed to update the consumption state to an available state, and consume messages in an available state according to the new subscription plan. In this way, the low efficiency of message consumption caused by the unavailability of the target consumer group during the entire rebalancing process is avoided, and the efficiency of message consumption during the rebalancing process is effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0033] FIG1 is an application environment diagram of a message consumption method in one embodiment;

[0034] Figure 2 A schematic diagram of a flow chart of a message consumption method in an embodiment;

[0035] Figure 3 A schematic diagram of a flow chart of a message consumption method in another embodiment;

[0036] Figure 4 A schematic diagram of message consumption steps in the prior art;

[0037] Figure 5 A schematic diagram of a message consumption step in an embodiment;

[0038] Figure 6 is a structural block diagram of a message consumption device in one embodiment;

[0039] Figure 7 is a structural block diagram of a message consumption device in another embodiment;

[0040] FIG. 8 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0042] Before introducing the embodiments of the present application, the following terms are explained:

[0043] Kafka (an open source stream processing platform): is a high-throughput distributed publish-subscribe messaging system. In modern Internet big data, massive amounts of user behavior data, behavior logs, etc. are collected. The transmission and aggregation of these massive amounts of data can be completed through Kafka.

[0044] Broker: A node in a Kafka cluster.

[0045] Topic: A topic in Kafka, used for message storage.

[0046] Partition: There are multiple partitions in a Topic, each partition is used to store messages. The messages in each partition are stored in a Broker.

[0047] Consumer group: A consumer cluster composed of multiple consumers using the same id (identity). A consumer group can subscribe to at least one Topic, and consumers in the same consumer group cannot subscribe to the same partition.

[0048] The message consumption method provided in the embodiment of the present application can be applied in the application environment shown in Figure 1. Each consumer in the target consumer group 102 communicates with the server 104 through the network. Consumers C1 to Cm communicate with the server 104 respectively.

[0049] In some embodiments, when server 104 verifies that the message configuration information of target consumer group 102 has changed, server 104 selects a main consumer from the target consumer group for making a rebalancing decision; server 104 sends the message allocation policy of the target consumer group to the main consumer in target consumer group 102 to instruct the main consumer to update the message allocation policy according to the rebalancing decision corresponding to the changed message configuration information to obtain a new message allocation policy; based on the new message allocation policy returned by the main consumer, server 104 determines the target consumer that needs to be rebalanced and the new subscription plan to which it belongs, and changes the consumption state of the target consumer to the rebalancing state, and other consumers in target consumer group 102 except the target consumer continue to consume messages in an available state according to their subscription plan; the new subscription plan is sent to the target consumer in target consumer group 102 to instruct the target consumer to update the consumption state to an available state, and consume messages in an available state according to the new subscription plan.

[0050] Each consumer can be regarded as a terminal, wherein the terminal can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server 104 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. Exemplarily, the server 104 can be a group coordination server for storing partition subscription information submitted by the consumer's client for load calculation.

[0051] In an exemplary embodiment, Figure 2 As shown, a message consumption method is provided, which is described by taking the method applied to the server 104 in FIG. 1 (specifically, the server 104 is a group coordination server) as an example, including the following steps S202 to S208. Among them:

[0052] Step S202: When it is verified that the message configuration information of the target consumer group has changed, a main consumer is selected from the target consumer group for making a rebalancing decision.

[0053] The message configuration information includes at least the information of the consumers in the target consumer group and the partition information corresponding to the target consumer group. The main consumer refers to the consumer that makes the rebalancing decision and can be regarded as the leader consumer. The main consumer can be any consumer in the target consumer group. For example, the main consumer can be a consumer that has just joined the target consumer group or a consumer that has already joined the target consumer group. There is no specific limitation.

[0054] Optionally, when the group coordination server determines that the consumption configuration information of the target consumer group has changed at the current time (current moment), it starts a rebalancing operation and randomly selects a consumer from the target consumer group as the main consumer, which is the consumer used to make rebalancing decisions.

[0055] Exemplarily, when the group coordination server determines at least one of personnel change information and partition change information corresponding to the target consumer group at the current moment, it determines that the message configuration information of the target consumer group has changed. At this time, step S202 is continued.

[0056] The personnel change information may be information about a new consumer joining the target consumer group, or information about a consumer belonging to the target consumer group leaving the target consumer group. The partition change information may be information about a new partition being added.

[0057] Exemplarily, before executing step S202, upon receiving a heartbeat signal from each consumer involved in the current consumption allocation strategy, the group coordination server determines that the target consumer group is consuming messages normally.

[0058] Among them, the heartbeat signal is sent by the consumer to the group coordination server, which is used by the group coordination server to confirm whether the consumer is online normally. If it is not received, it means that the consumer may be offline or have a fault. The current consumption allocation strategy is explained later.

[0059] Step S204: Send the message allocation strategy of the target consumer group to the main consumer to instruct the main consumer to update the message allocation strategy according to the rebalancing decision corresponding to the changed message configuration information to obtain a new message allocation strategy.

[0060] The consumption allocation strategy includes the subscription plan of each consumer, and the subscription plan of each consumer indicates the partitions in which the consumer is allowed to consume messages. The new message allocation strategy at least includes the new subscription plan of the target consumer that needs to be rebalanced.

[0061] Exemplarily, the group coordination server counts the subscription plans of consumers who currently have subscription plans in the target consumer group, obtains the current message allocation strategy, and sends the message allocation strategy to the main consumer. The main consumer updates the message allocation strategy according to the rebalancing decision corresponding to the changed message configuration information to obtain a new message allocation strategy.

[0062] Step S206, based on the new message distribution strategy returned by the main consumer, determine the target consumer that needs to be rebalanced and the new subscription plan it belongs to, and change the consumption state of the target consumer to the rebalanced state. Other consumers in the target consumer group except the target consumer continue to consume messages in an available state according to their subscription plan.

[0063] Among them, the consumption state refers to the state of message consumption, which involves two categories: available state and unavailable state. The rebalancing (prepareRablance) state refers to the state in which partitions need to be reallocated. For consumers in the rebalancing state, messages cannot be consumed temporarily, that is, messages cannot be obtained from the corresponding partitions, which can also be understood as an unavailable state. The available state refers to the state in which messages can be consumed, that is, messages can be obtained from the corresponding partitions, which can be understood as an available state.

[0064] Optionally, the group coordination server obtains the new message distribution strategy returned by the main consumer, and determines the target consumer that needs to be rebalanced and the new subscription plan it belongs to based on the new message distribution strategy. The group coordination server changes the consumption status of the target consumer to the rebalanced state, and the consumption status of other consumers in the target consumer group (consumers that do not need to be rebalanced) remains in the available state, and continues to obtain messages from the corresponding partition according to the previous subscription plan.

[0065] Exemplarily, before the group coordination server receives the new message allocation strategy, it operates according to the current message allocation strategy, that is, each consumer involved in the current message allocation strategy is in an available state. After receiving the new consumer allocation strategy, it returns to the above step of determining the target consumer that needs to be rebalanced and the new subscription plan to which it belongs according to the new message allocation strategy and continues to execute.

[0066] For example, the target consumer group includes consumers C1, C2, and C3. Among them, C3, as a new consumer, joins the target consumer group, which causes changes in the message configuration information. The current message distribution strategy stipulates the subscription plans of C1 and C2, and C1 and C2 are in the available state. Therefore, the new message distribution strategy indicates that C3 and C1 are the target consumers, and the consumption status of C3 and C1 is determined to be the rebalanced state, and the consumption status of C2 is still in the available state.

[0067] Step S208: Send the new subscription plan to the target consumer to instruct the target consumer to update the consumption state to the available state and consume the message in the available state according to the new subscription plan.

[0068] Exemplarily, for each target consumer, the group coordination server sends the corresponding new subscription plan to the target consumer. After the target consumer receives the corresponding new subscription plan, the consumption status is automatically updated to the available status, and the message is consumed in the available status according to the new subscription plan.

[0069] In the above message consumption method, when the message configuration information of the target consumer group is changed, a main consumer is selected from the target consumer group for making a rebalancing decision; the message allocation strategy of the target consumer group is sent to the main consumer to instruct the main consumer to update the message allocation strategy according to the rebalancing decision corresponding to the changed message configuration information, and obtain a new message allocation strategy. At this time, there is no need for the consumer to rejoin the target consumer group, which improves the convenience of the rebalancing process. Based on the new message allocation strategy returned by the main consumer, the target consumer that needs to be rebalanced and the new subscription plan to which it belongs are determined, and the consumption state of the target consumer is converted to the rebalancing state, and other consumers in the target consumer group except the target consumer continue to consume messages in an available state according to the subscription plan to which they belong. In other words, the target consumer group has not entered an unavailable state, and other consumers that do not need to be rebalanced can continue to consume, and the target consumer that needs to be rebalanced is in an unavailable state. Then, the new subscription plan is sent to the target consumer to instruct the target consumer to update the consumption state to an available state, and consume messages in an available state according to the new subscription plan. In this way, the low efficiency of message consumption caused by the unavailability of the target consumer group during the entire rebalancing process is avoided, and the efficiency of message consumption during the rebalancing process is effectively improved.

[0070] In some embodiments, the method also includes: when an application message from a new consumer applying to join the target consumer group is received, or when it is detected that at least one consumer in the target consumer group exits the target consumer group, or when it is detected that at least one new partition is added, determining that the message configuration information of the target consumer group has changed.

[0071] For example, the current target consumer group includes consumers C1 to C2. If the group coordination server receives a join request from a new consumer C3, C3 joins the target consumer group. At this time, it is automatically determined that the message configuration information of the target consumer group has changed, that is, the new consumer joins the target consumer group.

[0072] For example, the current target consumer group includes consumers C1 to C3. If the group coordination server receives a withdrawal request from a new consumer C3, C3 withdraws from the target consumer group. At this time, it is automatically determined that the message configuration information of the target consumer group has changed, that is, the consumer withdraws from the target consumer group.

[0073] For example, the target consumer group involves N partitions, namely P1 to PN. If a new partition is added at the current moment, it is determined that the message configuration information of the target consumer group has changed.

[0074] In this embodiment, through the personnel change information and partition change information corresponding to the target consumer group, it is possible to timely check whether the consumption configuration information of the target consumer group has changed, that is, whether rebalancing is required, thereby being able to respond to the rebalancing process in a timely and rapid manner.

[0075] In some embodiments, sending a new subscription plan to a target consumer includes: verifying whether a heartbeat signal sent by each consumer in the target consumer group is received within a preset time interval; and sending the new subscription plan to the target consumer if a heartbeat signal sent by each consumer in the target consumer group is received.

[0076] It should be noted that when the group coordination server receives the heartbeat signal sent by each consumer in the target consumer group, it indicates that each consumer in the target consumer group is online normally, so that the new subscription plan can be sent to the target consumer.

[0077] Exemplarily, if at least one consumer in the target consumer group does not send a heartbeat signal, no new subscription plan is sent to the target consumer. At this time, the group coordination server continues to wait, and upon receiving the heartbeat signal sent by each consumer, each consumer re-enters the target consumer group and redetermines the message distribution strategy.

[0078] In this embodiment, after determining the target consumer and the corresponding new subscription plan, it is necessary to ensure that each consumer in the target consumer group is operating normally, so that the new subscription plan can be sent to the target consumer, ensuring the effectiveness and reliability of message consumption.

[0079] In some embodiments, the main consumer updates the message distribution strategy according to the rebalancing decision corresponding to the changed message configuration information to obtain a new message distribution strategy, including: obtaining the message distribution strategy sent by the group coordination server, and determining the subscription plan of each consumer involved in the message distribution strategy based on the message distribution strategy; based on the subscription plan of each consumer, counting the number of partitions subscribed by each consumer, and based on the number of partitions subscribed by each consumer, determining the new message distribution strategy according to the rebalancing decision corresponding to the changed message configuration information.

[0080] Exemplarily, after the main consumer obtains the message distribution strategy sent by the group coordination server, it parses the message distribution strategy and determines the subscription plan of each consumer involved in the message distribution strategy. For each consumer's subscription plan, the main consumer counts the number of subscribed partitions involved in the subscription plan. The main consumer selects a corresponding rebalancing decision from multiple preset rebalancing decisions based on the changed message configuration information, and determines a new message distribution strategy according to the corresponding rebalancing decision and the corresponding number of each consumer involved in the message distribution strategy. Among them, multiple preset rebalancing decisions are different from each other.

[0081] In this embodiment, after obtaining the message distribution strategy, the main consumer counts the number of subscribed partitions of each consumer involved in the message distribution strategy, and performs matching rebalancing based on each number according to the corresponding rebalancing decision, thereby improving the accuracy of the rebalancing process.

[0082] In some embodiments, when the changed message configuration information is a new consumer joining the target consumer group, each consumer involved in the message distribution strategy is taken as a first consumer, and based on the number of partitions subscribed by each consumer, a new message distribution strategy is determined according to the rebalancing decision corresponding to the changed message configuration information, including: determining the first consumer with the largest number based on the number of partitions subscribed by each first consumer, and taking the first consumer with the largest number and the new consumer as target consumers that need to be rebalanced; according to the number of target consumers, equally distributing the partitions subscribed by the first consumer with the largest number to obtain a new subscription plan to which each target consumer belongs; generating a new message distribution strategy based on the consumer identifier of each target consumer and the new subscription plan to which each target consumer belongs.

[0083] Exemplarily, the current target consumer group includes consumers C1 to C2. If the group coordination server receives a join request from a new consumer C3, C3 joins the target consumer group. It can be seen that the consumers involved in the current message distribution strategy are C1 and C2 (i.e., regarded as the first consumer).

[0084] At this time, the corresponding new message distribution strategy determination process is: from the number of subscribed partitions L1 of C1, the number of subscribed partitions L2 of C2; select the largest one from L1 and L2. If L1 is the largest, then the subscribed partitions of C1 are evenly divided, that is, L1 / 2 of the subscribed partitions of C1 are given to C3, which is the new subscription plan for C3, and the other L1 / 2 of the subscribed partitions of C1 are given to C1, which is the new subscription plan for C1.

[0085] Of course, in other embodiments, the numbers can be sorted from large to small, and the ranking of consumers involved in the message distribution strategy can be determined based on the sorting result, that is, the larger the number of consumers, the higher the ranking of consumers. The top-ranked consumers are used as target consumers (for example, the top X consumers), and the sum of the numbers of the top-ranked consumers is calculated, and the ratio of the sum to (X+1) is used as the partition number of each consumer in the X consumers and the new consumer. For example, the message distribution strategy involves 20 consumers C1 to C20. According to the number of partitions subscribed by each of these 20 consumers, they are sorted from large to small, namely C1C2C3...C20. The first 20% of consumers are taken as target consumers, namely C1 to C4. The sum of the number of subscribed partitions of these four consumers is calculated (such as 10). C21 is a new consumer, and the ratio of the sum to 5 is taken as the number of partitions for each of the five target consumers (C1 to C4, C21).

[0086] In this embodiment, when the changed message configuration information is a new consumer joining the target consumer group, the new consumer automatically becomes the target consumer, and the first consumer with the largest number of subscribed partitions is used as the target consumer. By equally allocating the largest number of subscribed partitions to each target consumer, the pressure on the first consumer with the largest number of subscribed partitions can be reduced, ensuring the efficiency of message consumption after subsequent rebalancing.

[0087] In some embodiments, when the changed message configuration information causes at least one consumer in the target consumer group to exit the target consumer group, a new message distribution strategy is determined based on the number of partitions subscribed by each consumer and the rebalancing decision corresponding to the changed message configuration information, including: treating the consumers in the target consumer group except the exited consumer as second consumers; determining the second consumer with the smallest number based on the number of subscribed allocations of each second consumer; treating the second consumer with the smallest number as the target consumer that needs to be rebalanced, and assigning the subscribed partitions of the exited consumer to the target consumer to generate a new subscription plan to which the target consumer belongs; and generating a new message distribution strategy based on the consumer identifier of the target consumer and the new subscription plan to which the target consumer belongs.

[0088] For example, there are 5 consumers in the target consumer group, namely C1 to C5. At the current moment, C5 exits the target consumer group. Then, the consumer with the smallest number of subscribed partitions is determined from the second consumers C1, C2, C3, and C4. For example, it is C1 (the target consumer). Then, the subscribed partitions of C5 are allocated to C1. That is, the subscribed partitions of C5 are added to the subscription plan of C1 to obtain a new subscription plan.

[0089] In this embodiment, when the changed message configuration information causes at least one consumer in the target consumer group to exit the target consumer group, the consumer with the smallest number of subscriptions is selected from the remaining consumers (the second consumers) as the target consumer, and the subscribed partitions of the exiting consumer are directly given to the smallest consumer. In this way, the pressure on the remaining consumers can be balanced, ensuring stable and effective message consumption by the entire target consumer group.

[0090] In some embodiments, when the changed message configuration information adds at least one new partition, based on the number of partitions subscribed by each consumer and according to the rebalancing decision corresponding to the changed message configuration information, a new message distribution strategy is determined, including: based on the number of partitions subscribed by each consumer, determining the consumer with the smallest number, and taking the consumer with the smallest number as the target consumer that needs to be rebalanced; allocating at least one new partition to the target consumer to generate a new subscription plan to which the target consumer belongs; generating a new message distribution strategy based on the consumer identifier of the target consumer and the new subscription plan to which the target consumer belongs.

[0091] For example, there are 5 consumers in the target consumer group, namely C1 to C5, involving N partitions, namely P1 to PN. If a new partition is added at the current moment, the consumer with the smallest number of subscribed partitions is selected from C1 and C5, for example, C1, and the new partition is allocated to C1. That is, a new C1 is added to C1's subscription plan to obtain a new subscription plan.

[0092] In this embodiment, when the changed message configuration information adds at least one new partition, the consumer with the smallest number of subscriptions is used as the target consumer, and the newly added partition is allocated to the target consumer. In this way, the pressure on the remaining consumers can be balanced, ensuring stable and effective message consumption by the entire target consumer group.

[0093] In an exemplary embodiment, Figure 3 As shown, a message consumption method is provided, and the method is applied to a target consumer as an example for explanation, including the following steps S302 to S306. Among them:

[0094] Step S302: Obtain the message distribution strategy sent by the group coordination server, and determine the subscription plan of each consumer involved in the message distribution strategy based on the message distribution strategy.

[0095] Step S304: based on the subscription plan of each consumer, count the number of partitions subscribed by each consumer, and determine a new message distribution strategy based on the number of partitions subscribed by each consumer and the rebalancing decision corresponding to the changed message configuration information.

[0096] In some embodiments, when the changed message configuration information is a new consumer joining the target consumer group, each consumer involved in the message distribution strategy is taken as a first consumer, and based on the number of partitions subscribed by each consumer, a new message distribution strategy is determined according to the rebalancing decision corresponding to the changed message configuration information, including: determining the first consumer with the largest number based on the number of partitions subscribed by each first consumer, and taking the first consumer with the largest number and the new consumer as target consumers that need to be rebalanced; according to the number of target consumers, equally distributing the partitions subscribed by the first consumer with the largest number to obtain a new subscription plan to which each target consumer belongs; generating a new message distribution strategy based on the consumer identifier of each target consumer and the new subscription plan to which each target consumer belongs.

[0097] In some embodiments, when the changed message configuration information causes at least one consumer in the target consumer group to exit the target consumer group, a new message distribution strategy is determined based on the number of partitions subscribed by each consumer and the rebalancing decision corresponding to the changed message configuration information, including: treating the consumers in the target consumer group except the exited consumer as second consumers; determining the second consumer with the smallest number based on the number of subscribed allocations of each second consumer; treating the second consumer with the smallest number as the target consumer that needs to be rebalanced, and assigning the subscribed partitions of the exited consumer to the target consumer to generate a new subscription plan to which the target consumer belongs; and generating a new message distribution strategy based on the consumer identifier of the target consumer and the new subscription plan to which the target consumer belongs.

[0098] In some embodiments, when the changed message configuration information adds at least one new partition, based on the number of partitions subscribed by each consumer and according to the rebalancing decision corresponding to the changed message configuration information, a new message distribution strategy is determined, including: based on the number of partitions subscribed by each consumer, determining the consumer with the smallest number, and taking the consumer with the smallest number as the target consumer that needs to be rebalanced; allocating at least one new partition to the target consumer to generate a new subscription plan to which the target consumer belongs; generating a new message distribution strategy based on the consumer identifier of the target consumer and the new subscription plan to which the target consumer belongs.

[0099] Step S306, sends the new message distribution strategy to the group coordination server to instruct the group coordination server to determine the target consumer that needs to be rebalanced and the new subscription plan it belongs to according to the new message distribution strategy, and changes the consumption state of the target consumer to the rebalanced state. Other consumers in the target consumer group except the target consumer continue to consume messages in an available state according to their subscription plan.

[0100] The above process is described in the previous article and will not be repeated here.

[0101] In the above message consumption method, when the message configuration information of the target consumer group is changed, a main consumer is selected from the target consumer group for making a rebalancing decision; the message allocation strategy of the target consumer group is sent to the main consumer to instruct the main consumer to update the message allocation strategy according to the rebalancing decision corresponding to the changed message configuration information, and obtain a new message allocation strategy. At this time, there is no need for the consumer to rejoin the target consumer group, which improves the convenience of the rebalancing process. Based on the new message allocation strategy returned by the main consumer, the target consumer that needs to be rebalanced and the new subscription plan to which it belongs are determined, and the consumption state of the target consumer is converted to the rebalancing state, and other consumers in the target consumer group except the target consumer continue to consume messages in an available state according to the subscription plan to which they belong. In other words, the target consumer group has not entered an unavailable state, and other consumers that do not need to be rebalanced can continue to consume, and the target consumer that needs to be rebalanced is in an unavailable state. Then, the new subscription plan is sent to the target consumer to instruct the target consumer to update the consumption state to an available state, and consume messages in an available state according to the new subscription plan. In this way, the low efficiency of message consumption caused by the unavailability of the target consumer group during the entire rebalancing process is avoided, and the efficiency of message consumption during the rebalancing process is effectively improved.

[0102] In a specific embodiment, the specific steps are as follows:

[0103] Step 1: When the group coordination server receives an application message from a new consumer to join the target consumer group, or detects that at least one consumer in the target consumer group has exited the target consumer group, or detects that at least one new partition has been added, it determines that the message configuration information of the target consumer group has changed.

[0104] Step 2: The group coordination server sends the message distribution policy of the target consumer group to the primary consumer.

[0105] Step 3: The load calculator in the main consumer obtains the message distribution strategy sent by the group coordination server, and based on the message distribution strategy, determines the subscription plan of each consumer involved in the message distribution strategy; based on the subscription plan of each consumer, counts the number of partitions subscribed by each consumer, and based on the number of partitions subscribed by each consumer, determines the new message distribution strategy according to the rebalancing decision corresponding to the changed message configuration information. Among them, the load calculator is used to obtain the subscription status of all consumers from the group coordination server and re-formulate the message distribution strategy.

[0106] Specifically, when the changed message configuration information is a new consumer joining the target consumer group, the load calculator in the main consumer determines the first consumer with the largest number based on the number of partitions subscribed by each first consumer, and uses the first consumer with the largest number and the new consumer as the target consumers that need to be rebalanced; according to the number of target consumers, the partitions subscribed by the first consumer with the largest number are equally distributed to obtain a new subscription plan to which each target consumer belongs; based on the consumer identifier of each target consumer and the new subscription plan to which each target consumer belongs, a new message distribution strategy is generated.

[0107] When the changed message configuration information causes at least one consumer in the target consumer group to exit the target consumer group, the load calculator in the main consumer uses the consumers in the target consumer group except the exited consumer as second consumers; determines the second consumer with the smallest number based on the number of subscribed allocations of each second consumer; uses the second consumer with the smallest number as the target consumer that needs to be rebalanced, and allocates the subscribed partitions of the exited consumer to the target consumer to generate a new subscription plan to which the target consumer belongs; generates a new message allocation strategy based on the consumer identifier of the target consumer and the new subscription plan to which the target consumer belongs.

[0108] In the case where the changed message configuration information adds at least one new partition, the load calculator in the main consumer determines the consumer with the smallest number based on the number of partitions subscribed by each consumer, and uses the consumer with the smallest number as the target consumer that needs to be rebalanced; allocates at least one new partition to the target consumer to generate a new subscription plan to which the target consumer belongs; and generates a new message distribution strategy based on the consumer identifier of the target consumer and the new subscription plan to which the target consumer belongs.

[0109] Step 4: Based on the new message distribution strategy returned by the main consumer, the group coordination server determines the target consumer that needs to be rebalanced and the new subscription plan it belongs to, and changes the consumption state of the target consumer to the rebalanced state. Other consumers in the target consumer group except the target consumer continue to consume messages in an available state according to their subscription plan.

[0110] Step 5: The group coordination server verifies whether a heartbeat signal sent by each consumer in the target consumer group is received within a preset time interval; if a heartbeat signal sent by each consumer in the target consumer group is received, the new subscription plan is sent to the target consumer to instruct the target consumer to update the consumption state to an available state and consume messages in an available state according to the new subscription plan. Specifically, this step can be performed by the rebalancing process control unit in the group coordination server.

[0111] To compare the embodiments of the present application with the related art, a new consumer joining the target consumer is used as an example to illustrate the message consumption process of the prior art. Figure 4 FIG. 1 is a schematic diagram of the message consumption steps of the prior art. Figure 4 , the target consumer group includes consumers C1 and C2. Consumer client 1 of consumer C1 and consumer client 2 of consumer C2 both send heartbeat signals to the group coordination server normally. After the group coordination server determines that the new consumer C3 joins the target consumer group and selects C3 as the Leader (target consumer), C1 and C2 need to rejoin the target consumer group and are both in a rebalancing state. After the Leader determines the new message distribution strategy, it sends it to the group coordination server and feeds back the rebalancing completion information to C1, C2, and C3. In other words, for every new consumer that joins, each existing consumer in the target consumer group needs to rejoin the target consumer group, and the entire target consumer group is in an unavailable state.

[0112] Next, the message consumption steps of the embodiment of the present application are introduced, such as Figure 5 FIG. 1 is a schematic diagram of a message consumption step in an embodiment. Figure 5 In the dotted box, before the new message distribution strategy is determined, C1 and C2 both subscribe to messages normally, and send their respective subscription plans to the group coordination server to generate a message distribution strategy, and send it to the Leader to generate a new message distribution strategy. For example, when it is determined that C1 and C3 need to be rebalanced, C2 subscribes to messages normally, and C1 and C3 are in an unavailable state. After C1 and C3 determine their respective new subscription plans, it is determined that the rebalancing is completed. During the whole process, every time a new consumer is added, each existing consumer in the target consumer group does not need to rejoin the target consumer group, and the entire target consumer group is not all in an unavailable state. Compared with the existing technology, message consumption is more efficient.

[0113] In this embodiment, when the message configuration information of the target consumer group is verified to have changed, a main consumer for making a rebalancing decision is selected from the target consumer group; the message allocation strategy of the target consumer group is sent to the main consumer to instruct the main consumer to update the message allocation strategy according to the rebalancing decision corresponding to the changed message configuration information, and obtain a new message allocation strategy. At this time, there is no need for the consumer to rejoin the target consumer group, which improves the convenience of the rebalancing process. Based on the new message allocation strategy returned by the main consumer, the target consumer that needs to be rebalanced and the new subscription plan to which it belongs are determined, and the consumption state of the target consumer is converted to a rebalancing state, and other consumers in the target consumer group except the target consumer continue to consume messages in an available state according to the subscription plan to which they belong. In other words, the target consumer group has not entered an unavailable state, and other consumers that do not need to be rebalanced can continue to consume, and the target consumer that needs to be rebalanced is in an unavailable state. Then, by sending the new subscription plan to the target consumer, the target consumer is instructed to update the consumption state to an available state, and consume messages in an available state according to the new subscription plan. In this way, the low efficiency of message consumption caused by the target consumer group being in an unavailable state during the entire rebalancing process is avoided, and the efficiency of message consumption during the rebalancing process is effectively improved. It should be understood that although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless clearly stated in this document, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the various embodiments described above may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0114] Based on the same inventive concept, the embodiment of the present application also provides a message consumption device for implementing the message consumption method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more message consumption device embodiments provided below can refer to the limitations on the message consumption method above, and will not be repeated here.

[0115] In an exemplary embodiment, Figure 6 As shown, a message consumption device 600 is provided, including: a selection module 602, a first sending module 604, a determination module 606 and a second sending module 608, wherein:

[0116] A selection module 602 is used to select a main consumer from the target consumer group for making a rebalancing decision when it is verified that the message configuration information of the target consumer group has changed;

[0117] The first sending module 604 is used to send the message allocation strategy of the target consumer group to the main consumer to instruct the main consumer to update the message allocation strategy according to the rebalancing decision corresponding to the changed message configuration information to obtain a new message allocation strategy;

[0118] The determination module 606 is used to determine the target consumer that needs to be rebalanced and the new subscription plan it belongs to based on the new message distribution strategy returned by the main consumer, and change the consumption state of the target consumer to the rebalanced state. The other consumers in the target consumer group except the target consumer continue to consume messages in an available state according to the subscription plan they belong to;

[0119] The second sending module 608 is used to send the new subscription plan to the target consumer to instruct the target consumer to update the consumption state to the available state and consume the message in the available state according to the new subscription plan.

[0120] In some embodiments, the selection module 602 is also used to determine that the message configuration information of the target consumer group has changed when an application message is received from a new consumer applying to join the target consumer group, or when it is detected that at least one consumer in the target consumer group has exited the target consumer group, or when it is detected that at least one new partition has been added.

[0121] In some embodiments, the second sending module 608 is used to verify whether a heartbeat signal sent by each consumer in the target consumer group is received within a preset time interval; if a heartbeat signal sent by each consumer in the target consumer group is received, a new subscription plan is sent to the target consumer.

[0122] Based on the same inventive concept, the embodiment of the present application also provides a message consumption device for implementing the message consumption method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more message consumption device embodiments provided below can refer to the limitations on the message consumption method above, and will not be repeated here.

[0123] In an exemplary embodiment, Figure 7 As shown, a message consumption device 700 is provided, including: an acquisition module 702, a statistics module 704 and a third sending module 706, wherein:

[0124] The acquisition module 702 is used to acquire the message distribution strategy sent by the group coordination server, and determine the subscription plan of each consumer involved in the message distribution strategy based on the message distribution strategy;

[0125] A statistics module 704 is used to count the number of partitions subscribed by each consumer based on the subscription plan of each consumer, and determine a new message distribution strategy based on the number of partitions subscribed by each consumer and the rebalancing decision corresponding to the changed message configuration information;

[0126] The third sending module 706 is used to send the new message distribution strategy to the group coordination server to instruct the group coordination server to determine the target consumer that needs to be rebalanced and the new subscription plan it belongs to according to the new message distribution strategy, and to change the consumption state of the target consumer to the rebalanced state. Other consumers in the target consumer group except the target consumer continue to consume messages in an available state according to their subscription plan.

[0127] In some embodiments, when the changed message configuration information is a new consumer joining the target consumer group, the statistical module 704 is also used to determine the first consumer with the largest number based on the number of partitions subscribed by each first consumer, and use the first consumer with the largest number and the new consumer as the target consumers that need to be rebalanced; according to the number of target consumers, the partitions subscribed by the first consumer with the largest number are equally distributed to obtain a new subscription plan to which each target consumer belongs; based on the consumer identifier of each target consumer and the new subscription plan to which each target consumer belongs, a new message distribution strategy is generated.

[0128] In some embodiments, when the changed message configuration information causes at least one consumer in the target consumer group to exit the target consumer group, the statistical module 704 is further used to treat the consumers in the target consumer group, except the exited consumer, as second consumers; determine the second consumer with the smallest number based on the number of subscribed allocations of each second consumer; treat the second consumer with the smallest number as the target consumer that needs to be rebalanced, and assign the subscribed partition of the exited consumer to the target consumer to generate a new subscription plan to which the target consumer belongs; and generate a new message distribution strategy based on the consumer identifier of the target consumer and the new subscription plan to which the target consumer belongs.

[0129] In some embodiments, when the changed message configuration information is the addition of at least one new partition, the statistical module 704 is also used to determine the consumer with the smallest number based on the number of partitions subscribed by each consumer, and use the consumer with the smallest number as the target consumer that needs to be rebalanced; allocate at least one new partition to the target consumer to generate a new subscription plan to which the target consumer belongs; and generate a new message distribution strategy based on the consumer identifier of the target consumer and the new subscription plan to which the target consumer belongs.

[0130] Each module in the above message consumption device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0131] In an exemplary embodiment, a computer device is provided, which can be a server or a terminal, and its internal structure diagram can be shown in Figure 8. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a message consumption method is implemented.

[0132] Those skilled in the art will understand that the structure shown in FIG. 8 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0133] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0134] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0135] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0136] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0137] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0138] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0139] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A message consumption method, characterized in that: Applied to a group coordination server, the method comprises: When it is verified that the message configuration information of the target consumer group has changed, a primary consumer is selected from the target consumer group for making a rebalancing decision; Sending the message allocation strategy of the target consumer group to the main consumer to instruct the main consumer to update the message allocation strategy according to the rebalancing decision corresponding to the changed message configuration information to obtain a new message allocation strategy; Based on the new message distribution strategy returned by the main consumer, determine the target consumer that needs to be rebalanced and the new subscription plan to which it belongs, and change the consumption state of the target consumer to the rebalanced state. The other consumers in the target consumer group except the target consumer continue to consume messages in an available state according to their subscription plan; The new subscription plan is sent to the target consumer to instruct the target consumer to update the consumption state to an available state and consume the message in an available state according to the new subscription plan.

2. The method according to claim 1, characterized in that The method further comprises: When an application message is received from a new consumer to join the target consumer group, or when it is detected that at least one consumer in the target consumer group exits the target consumer group, or when it is detected that at least one new partition is added, it is determined that the message configuration information of the target consumer group has changed.

3. The method according to claim 1, characterized in that The sending of the new subscription plan to the target consumer includes: Verify whether a heartbeat signal sent by each consumer in the target consumer group is received within a preset time interval; When a heartbeat signal sent by each consumer in the target consumer group is received, a new subscription plan is sent to the target consumer.

4. A message consumption method, characterized in that: Applied to the main consumer, the method includes: Obtaining a message distribution strategy sent by the group coordination server, and determining a subscription plan for each consumer involved in the message distribution strategy based on the message distribution strategy; Based on the subscription plan of each consumer, count the number of partitions subscribed by each consumer, and determine the new message distribution strategy based on the rebalancing decision corresponding to the changed message configuration information based on the number of partitions subscribed by each consumer; The new message distribution strategy is sent to the group coordination server to instruct the group coordination server to determine the target consumer that needs to be rebalanced and the new subscription plan it belongs to according to the new message distribution strategy, and to change the consumption state of the target consumer to the rebalanced state. Other consumers in the target consumer group except the target consumer continue to consume messages in an available state according to their subscription plan.

5. The method according to claim 4, characterized in that In the case where the changed message configuration information is a new consumer joining the target consumer group, each consumer involved in the message distribution strategy is taken as the first consumer, and the new message distribution strategy is determined based on the number of partitions subscribed by each consumer and the rebalancing decision corresponding to the changed message configuration information, including: Based on the number of partitions subscribed by each first consumer, determine the first consumer with the largest number, and use the first consumer with the largest number and the new consumer as target consumers that need to be rebalanced; According to the number of target consumers, the subscribed partitions of the first consumer with the largest number are equally distributed to obtain a new subscription plan for each target consumer; A new message distribution strategy is generated based on the consumer identifier of each target consumer and the new subscription scheme to which each target consumer belongs.

6. The method according to claim 4, characterized in that In the case where the changed message configuration information indicates that at least one consumer in the target consumer group exits the target consumer group, the determining of a new message allocation strategy based on the number of partitions subscribed by each consumer and the rebalancing decision corresponding to the changed message configuration information includes: The consumers in the target consumer group except the withdrawn consumer are used as second consumers; Determine the second consumer with the smallest number based on the number of allocations subscribed by each second consumer; The second consumer with the smallest number is used as the target consumer that needs to be rebalanced, and the subscribed partition of the exiting consumer is allocated to the target consumer to generate a new subscription plan to which the target consumer belongs; A new message distribution strategy is generated based on the consumer identifier of the target consumer and the new subscription scheme to which the target consumer belongs.

7. The method according to claim 4, characterized in that In the case where the changed message configuration information adds at least one new partition, the new message distribution strategy is determined based on the number of partitions subscribed by each consumer and the rebalancing decision corresponding to the changed message configuration information, including: Based on the number of partitions subscribed by each consumer, determine the consumer with the smallest number, and use the consumer with the smallest number as the target consumer that needs to be rebalanced; Allocating at least one new partition to the target consumer to generate a new subscription plan to which the target consumer belongs; A new message distribution strategy is generated based on the consumer identifier of the target consumer and the new subscription scheme to which the target consumer belongs.

8. A message consumption device, characterized in that: The device comprises: A selection module is used to select a main consumer from the target consumer group for making a rebalancing decision when it is verified that the message configuration information of the target consumer group has changed; A first sending module is used to send the message allocation strategy of the target consumer group to the main consumer to instruct the main consumer to update the message allocation strategy according to the rebalancing decision corresponding to the changed message configuration information to obtain a new message allocation strategy; A determination module is used to determine the target consumer that needs to be rebalanced and the new subscription plan it belongs to based on the new message distribution strategy returned by the main consumer, and to change the consumption state of the target consumer to the rebalanced state. Other consumers in the target consumer group except the target consumer continue to consume messages in an available state according to their subscription plan; The second sending module is used to send the new subscription plan to the target consumer to instruct the target consumer to update the consumption state to the available state and consume the message in the available state according to the new subscription plan.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images