Broadcast message processing method and system, server, storage medium and program product

Abstract

The invention discloses a broadcast message processing method and system, a server, a storage medium and a program product, and relates to the technical field of message processing. The broadcast message processing method comprises the following steps: under the condition that a Kafka cluster increases nodes based on expansion operation, binding newly increased nodes with a consumption group in a Kafka message queue; and consuming the broadcast message based on the consumption group in the Kafka message queue. In the broadcast consumption mode, the consumption groups are dynamically bound to the nodes added in the capacity expansion operation, so that efficient load balancing and resource utilization are ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of message processing, and in particular to a broadcast message processing method, system, server, storage medium and program product. Background Art

[0002] With the advent of the big data era, the importance of distributed systems in data processing and transmission has become increasingly prominent. As a high-throughput distributed messaging system, Kafka is widely used in a variety of business scenarios due to its efficient message delivery capabilities and flexible scalability. However, the traditional Kafka broadcast consumption model has many limitations in practical applications and is difficult to meet the growing business needs.

[0003] In the existing Kafka system, the broadcast consumption mode mainly relies on static manual configuration and cannot dynamically adapt to the rapid changes in the business. When the business load increases, the increase or expansion of nodes cannot be dynamically bound to the consumer group, resulting in uneven resource allocation, which in turn affects the load balancing and stability of the system. Summary of the invention

[0004] The present invention provides a broadcast message processing method, system, server, storage medium and program product to solve the problem that the static manual configuration mode in the traditional Kafka broadcast consumption mode cannot dynamically respond to the growing business needs.

[0005] In a first aspect, an embodiment of the present invention provides a broadcast message processing method, including:

[0006] When the Kafka cluster adds nodes based on the expansion operation, the newly added nodes are bound to the consumer groups in the Kafka message queue;

[0007] The broadcast message is consumed based on the consumer group in the Kafka message queue.

[0008] In a second aspect, an embodiment of the present invention provides a broadcast message processing method, including:

[0009] When a node is offline due to a scaling-down operation in the Kafka cluster, the consumer group or consumer corresponding to the offline node is removed from the Kafka message queue;

[0010] Consume broadcast messages based on consumer groups in the Kafka message queue.

[0011] In a third aspect, an embodiment of the present invention provides a broadcast message processing system, including:

[0012] The binding module is used to bind the newly added nodes to the consumer groups in the Kafka message queue when the Kafka cluster adds nodes based on the expansion operation;

[0013] The first consumption module is used to consume broadcast messages based on the consumption group in the Kafka message queue.

[0014] In a fourth aspect, an embodiment of the present invention provides a broadcast message processing system, including:

[0015] The first removal module is used to remove the consumer group or consumer corresponding to the offline node from the Kafka message queue when the Kafka cluster goes offline based on the scaling-down operation;

[0016] The second consumption module is used to consume broadcast messages based on the consumption group in the Kafka message queue.

[0017] In a fifth aspect, an embodiment of the present invention provides a server, wherein the server includes:

[0018] at least one processor; and

[0019] a memory communicatively connected to the at least one processor; wherein,

[0020] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the broadcast message processing method described in any embodiment of the present invention.

[0021] In a sixth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the broadcast message processing method described in any embodiment of the present invention when executed.

[0022] In a seventh aspect, an embodiment of the present invention provides a computer program product including a computer program, and when the computer program is executed by a processor, the broadcast message processing method described in any embodiment of the present invention is implemented.

[0023] The technical solution of the embodiment of the present invention is to bind the newly added nodes to the consumer groups in the Kafka message queue when the Kafka cluster adds nodes based on the expansion operation; and consume broadcast messages based on the consumer groups in the Kafka message queue. By dynamically binding the consumer groups to the nodes added by the expansion operation in the broadcast consumption mode, the static manual configuration method in the traditional Kafka broadcast consumption mode is solved, and the problem of being unable to dynamically respond to growing business needs is achieved, thereby achieving the beneficial effect of ensuring efficient load balancing and resource utilization.

[0024] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 A flowchart of a broadcast message processing method provided in Embodiment 1 of the present invention;

[0027] Figure 2 A flowchart of a broadcast message processing method provided in Embodiment 2 of the present invention;

[0028] Figure 3 An architecture diagram of a broadcast message processing system provided in Embodiment 3 of the present invention;

[0029] Figure 4 An architecture diagram of a broadcast message processing system provided in Embodiment 4 of the present invention;

[0030] Figure 5 A schematic diagram of the structure of a server for implementing the broadcast message processing method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] Embodiment 1

[0034] Figure 1 This is a flowchart of a broadcast message processing method provided in Embodiment 1 of the present invention. This embodiment is applicable to the case of consuming broadcast messages based on a Kafka cluster. The method can be executed by a broadcast message processing system. The broadcast message processing system can be implemented in the form of hardware and / or software. The broadcast message processing system can be configured in a server. Figure 1 As shown, the method includes:

[0035] S110. When the Kafka cluster adds nodes based on the expansion operation, the newly added nodes are bound to the consumer groups in the Kafka message queue.

[0036] Among them, the Kafka cluster is a high-performance messaging system based on a distributed publish-subscribe model. A single server in the Kafka cluster is responsible for receiving, storing, and forwarding messages. The producer sends messages to the specified Topic in the Kafka cluster, and the consumer who subscribes to the Topic can pull and consume messages from it. Multiple consumers can form a consumer group to achieve load balancing and sequential processing of messages. The Kafka cluster can perform elastic expansion or contraction operations based on business needs and load conditions, and dynamically adjust the cluster size to optimize resource utilization and system performance. The expansion operation can be understood as the operation of dynamically increasing the number of nodes in the Kafka cluster when the business load increases. The Kafka message queue relies on the distributed architecture of the Kafka cluster to efficiently transmit messages between producers and consumers.

[0037] Specifically, when the Kafka cluster adds nodes based on the expansion operation, the newly added nodes are bound to the consumer groups in the Kafka message queue, so that the newly added nodes become consumers in the bound consumer groups, thereby realizing dynamic binding of the consumer groups to the added nodes of the expansion operation in the broadcast consumption mode, ensuring efficient load balancing and resource utilization.

[0038] Exemplarily, the consumer group in the Kafka message queue can be identified by a unique identifier. Preferably, the setting rule of the unique identifier of each consumer group in the Kafka message queue can be to use a natural number sequence that increases in sequence from 0. The method of binding the newly added node to the consumer group in the Kafka message queue can be to bind the newly added node to the unique identifier of the consumer group in the Kafka message queue.

[0039] S120, consuming broadcast messages based on the consumer group in the Kafka message queue.

[0040] Specifically, in the Kafka message queue, all consumers in each consumer group jointly consume broadcast messages of one or more topics.

[0041] The technical solution of the embodiment of the present invention is to bind the newly added nodes to the consumer groups in the Kafka message queue when the Kafka cluster adds nodes based on the expansion operation; and consume broadcast messages based on the consumer groups in the Kafka message queue. In the broadcast consumption mode, the consumer groups are dynamically bound to the nodes added by the expansion operation to ensure efficient load balancing and resource utilization.

[0042] As an optional embodiment of the embodiment of the present invention, S110, binding the newly added node to the consumer group in the Kafka message queue, includes:

[0043] S111. When the Kafka cluster adds multiple nodes based on the capacity expansion operation, use the first distributed lock to lock the operations of each consumer group in the Kafka message queue;

[0044] S112: Bind the newly added node holding the first distributed lock to the consumer group, and release the first distributed lock after the binding is completed.

[0045] Among them, a distributed lock is a mechanism used in a distributed system to coordinate access to shared resources by multiple processes or threads. Its core purpose is to ensure that in a concurrent environment, only one process or thread can hold a lock and operate shared resources at the same time, thereby avoiding problems such as data inconsistency or race conditions. In the process of binding a new node to a consumer group, the first distributed lock is mainly used to prevent multiple new nodes from binding to a consumer group at the same time.

[0046] Specifically, when the Kafka cluster adds multiple nodes based on the expansion operation, that is, when the nodes are added concurrently based on the expansion operation, a distributed lock mechanism is introduced, and the first distributed lock is used to lock the operations of each consumer group in the Kafka message queue, and only one newly added node can hold the first distributed lock at a time. For the newly added node holding the first distributed lock, the newly added node is bound to the consumer group, so that the newly added node becomes a consumer in the bound consumer group.

[0047] This embodiment introduces a distributed lock mechanism when adding nodes concurrently, which can prevent multiple newly added nodes from occupying the same consumer group at the same time.

[0048] As an optional embodiment of any of the above embodiments, binding the newly added node to the consumer group in the Kafka message queue includes:

[0049] A1. When there are unoccupied consumer groups in the Kafka message queue, the newly added node is bound to one of the unoccupied consumer groups.

[0050] Among them, the unoccupied consumer group can be understood as a consumer group that has not started consuming messages, has not been assigned to any partition, or all consumers are in an idle state. Usually, the "unoccupied" state of a consumer group is short-lived. Once a new message arrives or a partition is reallocated, the consumer group will change from the "unoccupied" state to the "active" state and start consuming messages.

[0051] Specifically, check whether there is an unoccupied consumer group in the Kafka message queue. If there is an unoccupied consumer group, bind the newly added node to the unoccupied consumer group. If there are multiple unoccupied consumer groups, it can be determined based on the configuration information of the consumer group, or it can be randomly selected.

[0052] For example, when a natural number sequence is used as a unique identifier of a consumer group in a Kafka message queue, the consumer group management module is requested to sequentially traverse the unique identifiers corresponding to each consumer group in the Kafka message queue to find out whether there is a missing natural number in the natural number sequence; if so, the consumer group whose unique identifier is a missing natural number is considered to be an unoccupied consumer group; the newly added node is bound to the missing natural number as the unique identifier. After binding the node, the number of consumers in the recorded consumer group is increased by one.

[0053] B1. When there is no unoccupied consumer group in the Kafka message queue and the number of consumers in each consumer group is not the same, bind the newly added node to the consumer group with the least number of consumers.

[0054] Specifically, if there is no unoccupied consumer group in the Kafka message queue, the number of consumers in each consumer group is obtained; if the number of consumers in each consumer group is not the same, the consumer group with the least number of consumers is determined, and the newly added node is bound to the consumer group with the least number of consumers, so that the newly added node joins the consumer group with the least number of consumers, thereby achieving load balancing among each consumer group.

[0055] C1. When there is no unoccupied consumer group in the Kafka message queue, the number of consumers in each consumer group is the same, and the number of consumer groups in the Kafka message queue does not reach the maximum limit, a new consumer group is created and the newly added node is bound to the new consumer group.

[0056] Specifically, if there are no unoccupied consumer groups in the Kafka message queue and the number of consumers in each consumer group is the same, it means that the consumer groups in the Kafka message queue have achieved load balancing. If the number of consumer groups in the Kafka message queue does not reach the maximum limit, a new consumer group is created and the newly added node is bound to the new consumer group.

[0057] It should be noted that there are a limited number of consumer groups in the Kafka message queue. If the maximum number of consumer groups has been reached, a new consumer group cannot be created. In this case, the new node can be bound to any consumer group in the Kafka message queue.

[0058] For example, when a natural number sequence is used as the unique identifier of a consumer group in a Kafka message queue, the consumer group management module increments the unique identifier corresponding to the consumer group existing in the Kafka message queue according to the natural number arrangement to obtain the unique identifier of the newly created consumer group. The newly added node is bound to the unique identifier of the newly created consumer group.

[0059] It can be understood that the method of binding the newly added node and the consumer group described in the above steps A1 to A3 can be applied to a node added based on the expansion operation or each node in multiple nodes added concurrently.

[0060] This embodiment binds the added nodes to the consumer groups according to the occupancy of the consumer groups in the Kafka message queue and the number of consumers, realizes the automatic binding of the added nodes and the consumer groups generated under the capacity expansion operation, and ensures the load balancing of the consumer groups in the Kafka message queue.

[0061] As an optional embodiment of the embodiment of the present invention, B1, binding the newly added node to the consumer group with the least number of consumers, includes:

[0062] Use the second distributed lock to lock the operations of each consumer group in the Kafka message queue;

[0063] While holding the second distributed lock, determine the consumer group with the least number of consumers in the Kafka message queue;

[0064] Bind the newly added node to the unique identifier of the consumer group with the least number of consumers, and release the second distributed lock after the binding is completed.

[0065] In the process of determining the consumer group with the least number of consumers, the second distribution lock is mainly used to lock the number of consumer groups in the consumer group to remain unchanged.

[0066] Specifically, when there is no unoccupied consumer group in the Kafka message queue, the number of consumers in each consumer group is traversed to find the consumer group with the least number of consumers. In order to avoid the number of consumers in the consumer group changing during the traversal of each consumer group, the second distributed lock is used to lock the operation of each consumer group in the Kafka message queue. While holding the second distributed lock, the number of consumers in each consumer group is traversed to find the consumer group with the least number of consumers. The newly added node is bound to the unique identifier corresponding to the consumer group with the least number of consumers, and the second distributed lock is released after the binding is successful.

[0067] This embodiment introduces a distributed lock mechanism in the process of determining the consumer group with the least number of consumers, thereby preventing the number of consumers in the consumer group from changing during the traversal process.

[0068] As an optional embodiment of the embodiment of the present invention, the step of adding nodes to the Kafka cluster based on the capacity expansion operation includes:

[0069] Obtain the comprehensive resource usage of the Kafka cluster;

[0070] If the comprehensive resource usage rate is greater than the capacity expansion threshold, the capacity expansion operation is performed to increase the nodes in the Kafka cluster.

[0071] The comprehensive resource utilization rate can be considered as the degree of utilization of one or more resources by each node in the Kafka cluster, for example, it can be comprehensively determined based on at least one of the CPU utilization rate, memory utilization rate, GPU utilization rate, disk utilization rate and network bandwidth utilization rate of each node. The expansion threshold can be understood as the minimum comprehensive resource utilization required for the expansion of the Kafka cluster, which can be set according to business needs, and the embodiments of the present invention do not limit this.

[0072] Specifically, obtain the utilization rate of the Kafka cluster in various resource utilization indicators and calculate the comprehensive resource utilization rate; if the comprehensive resource utilization rate of the Kafka cluster is greater than the expansion threshold, it is considered that the comprehensive resources of the Kafka cluster are close to saturation, and it is necessary to add nodes through expansion to distribute the load to more nodes, improve the overall throughput of the cluster, and support larger data volumes.

[0073] This embodiment automatically expands capacity and adds nodes based on the comprehensive resource utilization rate of the Kafka cluster, which can automatically optimize the resources of the Kafka cluster and meet business needs.

[0074] Embodiment 2

[0075] Figure 2 This is a flowchart of a broadcast message processing method provided in Embodiment 2 of the present invention. This embodiment is applicable to the case of consuming broadcast messages based on a Kafka cluster. The method can be executed by a broadcast message processing system. The broadcast message processing system can be implemented in the form of hardware and / or software. The broadcast message processing system can be configured in a server. Figure 2 As shown, the method includes:

[0076] S210. When a node is offline in the Kafka cluster based on a scaling-down operation, remove the consumer group or consumer corresponding to the offline node from the Kafka message queue.

[0077] Among them, the scaling-down operation can be understood as the operation of dynamically reducing the number of nodes in the Kafka cluster when the business load decreases.

[0078] Specifically, when the Kafka cluster reduces nodes based on a scaling-down operation, some nodes are taken offline, and the consumer groups or consumers corresponding to the offline nodes are removed from the Kafka message queue to ensure effective management and release of system resources.

[0079] Exemplarily, a method of removing a consumer group or a consumer corresponding to a node that is offline from a Kafka message queue may be to trigger an offline hook event to remove a consumer group or a consumer corresponding to the node that is offline from a Kafka message queue.

[0080] S220, consuming broadcast messages based on the consumer group in the Kafka message queue.

[0081] Specifically, after removing the consumer group or consumer corresponding to the offline node, the broadcast message is consumed based on the remaining consumer groups in the Kafka message queue.

[0082] The technical solution of the embodiment of the present invention is to remove the consumer group or consumer corresponding to the offline node from the Kafka message queue when the Kafka cluster is offline due to a scaling-down operation; and to consume broadcast messages based on the consumer group in the Kafka message queue. In the broadcast consumption mode, the consumer group bound to the node offline due to the scaling-down operation is dynamically removed from the Kafka message queue to ensure the effective management and release of system resources.

[0083] As an optional embodiment of the embodiment of the present application, S210, removing the consumer group or consumer corresponding to the offline node from the Kafka message queue, includes:

[0084] S211. When the offline node is the only consumer in its consumer group, trigger an offline hook event to remove the consumer group to which the offline node belongs from the Kafka message queue;

[0085] Among them, the offline hook event can be understood as an event triggered when the system or service is about to stop running, which is used to perform some cleanup or notification operations to ensure that resources are correctly released or the service status is correctly updated.

[0086] Specifically, when the offline node is the only consumer in the bound consumer group, the offline hook event is triggered to remove the consumer corresponding to the offline node from the Kafka message queue, that is, to remove the consumer group to which the offline node belongs.

[0087] S212: When the offline node is a non-unique consumer in the consumer group, trigger an offline hook event to remove the consumer corresponding to the offline node from the consumer group in the Kafka message queue.

[0088] Specifically, when the offline node is not the only consumer in its consumer group, that is, there are multiple consumers in the consumer group where the offline node is located, the offline hook event is triggered to remove the consumer corresponding to the offline node from the consumer group where the offline node is located in the Kafka message queue, and retain other consumers in the consumer group.

[0089] This embodiment removes the consumers or consumer groups corresponding to the nodes that are offline due to the scaling down operation in the Kafka message queue through the offline hook event, which can ensure that the system can automatically complete the cleanup work when it stops running, avoiding resource leakage or inconsistent status problems.

[0090] As an optional embodiment of any of the above embodiments, after triggering the offline hook event, the method further includes:

[0091] When the offline hook event triggers an exception, query the number of nodes in the Kafka cluster based on the scheduled job inspection mechanism;

[0092] If the number of consumers in the Kafka message queue is greater than the number of nodes, idle consumers or consumer groups are removed from the Kafka message queue.

[0093] Among them, the scheduled job inspection mechanism is a mechanism for monitoring and managing scheduled tasks. Its core purpose is to ensure that scheduled tasks can run as expected, promptly discover and solve problems in task execution, and thus ensure the stability and reliability of the system. In this embodiment, the scheduled job inspection mechanism is used to regularly query whether the number of nodes in the Kafka cluster is consistent with the number of consumers.

[0094] Specifically, when the offline hook event triggers an exception and the consumer or consumer group in the Kafka message queue cannot be offline, the number of nodes registered in the Kafka cluster is queried based on the scheduled job inspection mechanism, and it is determined whether the number of consumers in the Kafka message queue is equal to the number of registered nodes. If the number of consumers in the Kafka message queue is greater than the number of registered nodes, it means that the node is offline but the corresponding consumer has not been removed from the Kafka message queue. Therefore, the idle consumers or consumer groups are removed from the Kafka message queue to ensure that the corresponding consumers or consumer groups can be released after the node is offline, ensuring that system resources are effectively managed and released.

[0095] As an optional embodiment of the embodiment of the present invention, the step of taking offline nodes of the Kafka cluster based on the scaling-down operation includes:

[0096] Obtain the comprehensive resource usage of the Kafka cluster;

[0097] If the comprehensive resource usage rate is less than the shrinking threshold, a shrinking operation is performed to take the nodes in the Kafka cluster offline.

[0098] The expansion threshold may be understood as the minimum comprehensive resource usage required for expansion of the Kafka cluster, and may be set according to business requirements, which is not limited in the embodiments of the present invention.

[0099] Specifically, obtain the utilization rate of the Kafka cluster in various resource utilization indicators and calculate the comprehensive resource utilization rate; if the comprehensive resource utilization rate of the Kafka cluster is less than the expansion threshold, it is considered that the comprehensive resources of the Kafka cluster are too low, and it is necessary to reduce the hardware, network and operation and maintenance costs and improve resource utilization by scaling down and taking offline redundant nodes.

[0100] This embodiment automatically scales down the capacity based on the comprehensive resource utilization of the Kafka cluster, which can automatically reduce the number of nodes or resource allocation in the cluster, optimize resource utilization and reduce operation and maintenance costs.

[0101] It can be understood that in a broadcast message processing system, the broadcast message processing methods provided in Example 1 and Example 2 can be combined to perform automatic expansion and contraction based on the comprehensive resource utilization rate of the Kafka cluster, automatically bind the newly added nodes to the corresponding consumer groups in the Kafka message queue, and automatically remove the corresponding consumer groups or consumers in the Kafka message queue for the nodes that are scaled down and taken offline. In addition, broadcast messages are consumed based on the consumer groups in the Kafka message queue, thereby achieving system performance optimization and load balancing according to changes in business load, dynamically responding to changing business needs, and improving system stability.

[0102] Embodiment 3

[0103] Figure 3 FIG. 1 is a schematic diagram of a broadcast message processing system provided in Embodiment 3 of the present invention. Figure 3 As shown, the system includes: a binding module 310 and a first consumption module 320;

[0104] The binding module 310 is used to bind the newly added node to the consumer group in the Kafka message queue when the Kafka cluster adds nodes based on the expansion operation;

[0105] The first consumption module 320 is used to consume the broadcast message based on the consumption group in the Kafka message queue.

[0106] Optionally, the binding module 310 includes:

[0107] A locking unit, used to lock the operation of each consumer group in the Kafka message queue using the first distributed lock when the Kafka cluster adds multiple nodes based on the expansion operation;

[0108] A binding unit is used to bind the newly added node holding the first distributed lock to the consumer group, and release the first distributed lock after the binding is completed.

[0109] Optionally, the binding module 310 includes:

[0110] A first binding unit is used to bind the newly added node to one of the unoccupied consumer groups when there is an unoccupied consumer group in the Kafka message queue;

[0111] A second binding unit is used to bind the newly added node to the consumer group with the least number of consumers when there is no unoccupied consumer group in the Kafka message queue and the number of consumers in each consumer group is not the same;

[0112] The third binding unit is used to create a new consumer group and bind the newly added node to the newly created consumer group when there is no unoccupied consumer group in the Kafka message queue, the number of consumers in each consumer group is the same, and the number of consumer groups in the Kafka message queue reaches the maximum limit.

[0113] Optionally, the second binding unit is specifically used to:

[0114] Use the second distributed lock to lock the operations of each consumer group in the Kafka message queue;

[0115] While holding the second distributed lock, determine the consumer group with the least number of consumers in the Kafka message queue;

[0116] Bind the newly added node to the unique identifier of the consumer group with the least number of consumers, and release the second distributed lock after the binding is completed.

[0117] Optionally, the steps for adding nodes to the Kafka cluster based on the expansion operation include:

[0118] Obtain the comprehensive resource usage of the Kafka cluster;

[0119] If the comprehensive resource usage rate is greater than the capacity expansion threshold, the capacity expansion operation is performed to increase the nodes in the Kafka cluster.

[0120] The broadcast message processing system provided by the embodiment of the present invention can execute the broadcast message processing method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0121] Embodiment 4

[0122] Figure 4 This is a schematic diagram of the structure of a broadcast message processing system provided by Embodiment 4 of the present invention. Figure 4 As shown, the system includes: a first removal module 410 and a second consumption module 420;

[0123] The first removal module 410 is used to remove the consumer group or consumer corresponding to the offline node from the Kafka message queue when the Kafka cluster takes the node offline based on the scaling-down operation;

[0124] The second consumption module 420 is used to consume the broadcast message based on the consumption group in the Kafka message queue.

[0125] Optionally, the first removal module 410 includes:

[0126] A first removal unit is used to trigger an offline hook event when the offline node is the only consumer in the consumer group, and remove the consumer group where the offline node is located from the Kafka message queue;

[0127] The second removal unit is used to trigger an offline hook event when the offline node is a non-unique consumer in the consumer group where it is located, and remove the consumer corresponding to the offline node from the consumer group where the offline node is located in the Kafka message queue.

[0128] Optionally, also include:

[0129] A patrol module, used to query the number of nodes in the Kafka cluster based on a scheduled job patrol mechanism when the offline hook event triggers an exception;

[0130] The second removal module is used to remove idle consumers or consumer groups from the Kafka message queue if the number of consumers in the Kafka message queue is greater than the number of nodes.

[0131] Optionally, the steps for taking a node offline based on a scaling-down operation in a Kafka cluster include:

[0132] Obtain the comprehensive resource usage of the Kafka cluster;

[0133] If the comprehensive resource usage rate is less than the shrinking threshold, a shrinking operation is performed to take the nodes in the Kafka cluster offline.

[0134] The broadcast message processing system provided by the embodiment of the present invention can execute the broadcast message processing method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0135] Embodiment 5

[0136] Figure 5 The schematic diagram of the structure of the server 10 that can be used to implement the embodiment of the present invention is shown. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0137] like Figure 5As shown, the server 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the server 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The input / output (I / O) interface 15 is also connected to the bus 14.

[0138] A number of components in the server 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the server 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0139] The processor 11 may be a variety of general and / or dedicated processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a broadcast message processing method.

[0140] In some embodiments, the broadcast message processing method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the server 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the broadcast message processing method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the broadcast message processing method in any other appropriate manner (e.g., by means of firmware).

[0141] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0142] In some embodiments, the broadcast message processing method may be implemented as a computer program, which is invisibly included in a computer program product. The computer program implements the broadcast message processing method of the present invention when executed by a processor. The computer program product can be understood as a software product that implements its solution mainly through a computer program. The computer program for implementing the method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program can be executed entirely on the machine, partially on the machine, partially on the machine as an independent software package and partially on a remote machine, or entirely on a remote machine or server.

[0143] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0144] To provide interaction with a user, the systems and techniques described herein may be implemented on a server having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the server. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0145] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0146] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0147] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0148] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A broadcast message processing method, characterized in that: The method comprises: When the Kafka cluster adds nodes based on the expansion operation, the newly added nodes are bound to the consumer groups in the Kafka message queue; The broadcast message is consumed based on the consumer group in the Kafka message queue.

2. The method according to claim 1, characterized in that The binding of the newly added node to the consumer group in the Kafka message queue includes: When the Kafka cluster adds multiple nodes based on the expansion operation, the first distributed lock is used to lock the operations of each consumer group in the Kafka message queue; Bind the newly added node holding the first distributed lock to the consumer group, and release the first distributed lock after the binding is completed.

3. The method according to claim 1 or 2, characterized in that: Binding the newly added node to the consumer group in the Kafka message queue includes: In the case where there are unoccupied consumer groups in the Kafka message queue, binding the newly added node to one of the unoccupied consumer groups; When there is no unoccupied consumer group in the Kafka message queue and the number of consumers in each consumer group is not the same, the newly added node is bound to the consumer group with the least number of consumers; When there is no unoccupied consumer group in the Kafka message queue, the number of consumers in each consumer group is the same, and the number of consumer groups in the Kafka message queue reaches the maximum limit, a new consumer group is created and the newly added node is bound to the new consumer group.

4. The method according to claim 3, characterized in that Bind the newly added node to the consumer group with the least number of consumers, including: Use the second distributed lock to lock the operations of each consumer group in the Kafka message queue; While holding the second distributed lock, determine the consumer group with the least number of consumers in the Kafka message queue; Bind the newly added node to the unique identifier of the consumer group with the least number of consumers, and release the second distributed lock after the binding is completed.

5. The method according to claim 1, characterized in that The steps for adding nodes to the Kafka cluster based on capacity expansion operations include: Obtain the comprehensive resource usage of the Kafka cluster; If the comprehensive resource usage rate is greater than the capacity expansion threshold, the capacity expansion operation is performed to increase the nodes in the Kafka cluster.

6. A broadcast message processing method, characterized in that: The method comprises: When a node is offline due to a scaling-down operation in the Kafka cluster, the consumer group or consumer corresponding to the offline node is removed from the Kafka message queue; Consume broadcast messages based on consumer groups in the Kafka message queue.

7. The method according to claim 6, characterized in that The removing of the consumer group or consumer corresponding to the offline node from the Kafka message queue includes: In the case where the offline node is the only consumer in the consumer group, triggering an offline hook event, and removing the consumer group where the offline node is located from the Kafka message queue; In the case where the offline node is a non-unique consumer in the consumer group, an offline hook event is triggered to remove the consumer corresponding to the offline node from the consumer group where the offline node is located in the Kafka message queue.

8. The method according to claim 7, characterized in that After the offline hook event is triggered, it also includes: When the offline hook event triggers an exception, query the number of nodes in the Kafka cluster based on the scheduled job inspection mechanism; If the number of consumers in the Kafka message queue is greater than the number of nodes, idle consumers or consumer groups are removed from the Kafka message queue.

9. The method according to claim 6, characterized in that The steps for taking nodes offline based on scaling-down operations in a Kafka cluster include: Obtain the comprehensive resource usage of the Kafka cluster; If the comprehensive resource usage rate is less than the shrinking threshold, a shrinking operation is performed to take the nodes in the Kafka cluster offline.

10. A broadcast message processing system, characterized in that: The system comprises: The binding module is used to bind the newly added nodes to the consumer groups in the Kafka message queue when the Kafka cluster adds nodes based on the expansion operation; The first consumption module is used to consume broadcast messages based on the consumption group in the Kafka message queue.

11. A broadcast message processing system, characterized in that: The system comprises: The first removal module is used to remove the consumer group or consumer corresponding to the offline node from the Kafka message queue when the Kafka cluster goes offline based on the scaling-down operation; The second consumption module is used to consume broadcast messages based on the consumption group in the Kafka message queue.

12. A server, characterized in that: The server comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the broadcast message processing method according to any one of claims 1 to 9.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the broadcast message processing method according to any one of claims 1 to 9 when executed.

14. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the computer program implements the broadcast message processing method according to any one of claims 1 to 9.

Images