Message consumption control methods, devices, equipment, and media

Abstract

The present application relates to a message consumption control method and its apparatus, device, and medium in the field of e-commerce technology. The method comprises: sending a message provided by a producer to a service agent so that consumers can pull messages from the service agent; in response to an abnormal event of any service agent, enabling a coordination thread pool so that consumers can pull messages from the coordination thread pool; pulling some messages from service agents that have not triggered an abnormal event to the coordination thread pool, so as to distribute the some messages to consumers through the coordination thread pool. The present application can coordinate the processing of messages from service agents that have not experienced an abnormality when an abnormality occurs in a service agent, effectively avoiding message backlogs and ensuring the robustness and availability of message consumption.

Description

Technical Field

[0001] This application relates to the field of e-commerce technology, and in particular to a message consumption control method and corresponding apparatus, computer equipment, and computer-readable storage medium. Background Technology

[0002] Message middleware is a common and crucial technology in e-commerce, used for payment, order processing, and other e-commerce operations. Message middleware clusters typically consist of multiple servers storing messages, and the occasional downtime of some servers can lead to message backlogs on the remaining surviving servers. Traditional solutions involve temporarily increasing queue resources by 20 times and allocating 20 times the number of machines to consume messages at 20 times the original speed. However, this solution requires extremely large queue and consumption resources, making it prohibitively expensive and uneconomical.

[0003] Given the shortcomings of traditional technologies, the applicant has long been engaged in research in related fields and has therefore explored alternative approaches to solve industry problems in the e-commerce sector. Summary of the Invention

[0004] The primary objective of this application is to solve at least one of the aforementioned problems by providing a message consumption control method and corresponding apparatus, computer equipment, and computer-readable storage medium.

[0005] To achieve the various objectives of this application, the following technical solution is adopted:

[0006] A message consumption control method provided for one of the purposes of this application includes the following steps:

[0007] The producer sends the message to the service broker so that the consumer can pull the message from the service broker;

[0008] In response to any abnormal event of the service broker, enable the coordination thread pool so that consumers can pull messages from the coordination thread pool;

[0009] A portion of messages is pulled from the service broker that has never triggered an exception event to the coordination thread pool, so that the portion of messages can be distributed to consumers through the coordination thread pool.

[0010] In a further embodiment, fetching a portion of messages from the service broker that has never triggered an exception event to the coordination thread pool includes the following steps:

[0011] Retrieve all overload messages from the service broker that have not triggered abnormal events based on the preset load threshold;

[0012] Using the unique identifier carried by the overload message, query the consumption status corresponding to the unique identifier from the message status table. When the consumption status indicates that the message has not been consumed, retain the overload message in the coordination thread pool.

[0013] When the consumption status indicates that the message has been consumed, the corresponding overload message is discarded.

[0014] In a further embodiment, after fetching a portion of messages from the service broker that has not triggered an abnormal event to the coordination thread pool, and distributing the portion of messages to consumers through the coordination thread pool, the process includes:

[0015] When all service proxies that triggered the abnormal event have recovered to normal, and the message status of all messages in the coordination thread pool indicates that they have been consumed, the operation of the coordination thread pool is stopped.

[0016] In a further embodiment, sending messages provided by the producer to a service broker for consumers to pull messages from the service broker includes the following steps:

[0017] Obtain system stability indicators, including the number of producers, the number of messages in the service broker, the number of consumers, and the performance information of the service broker;

[0018] Determine whether all the system stability indicators are lower than the corresponding preset stability thresholds;

[0019] When a system stability indicator exceeds a stability threshold, the consumer's message consumption is paused, and stability processing is performed based on the system stability indicator to bring it below the corresponding stability threshold.

[0020] When all system stability indicators are lower than the corresponding preset stability thresholds, the consumer consumption message is restored.

[0021] In a further embodiment, stability processing is performed based on the system stability index to make it lower than the corresponding stability threshold, including: adjusting the maximum number of messages pulled by a single consumer to a first threshold based on the number of messages in the service broker that exceeds the stability threshold, and adjusting the maximum pull time of the consumer to a second threshold.

[0022] In a further embodiment, the method further includes the following steps:

[0023] Using the unique identifier carried in the message provided by the producer, the corresponding sending status is queried from the message status table. When the sending status indicates that the message has been sent, the corresponding message is discarded. When the sending status indicates that the message has not been sent, the corresponding message is sent to the service broker.

[0024] The service agent stores the messages provided by the producer, inserts the unique identifier carried by the message into the message status table, and marks the consumption status corresponding to the unique identifier to indicate that the message is in an unconsumed state.

[0025] In a further embodiment, the method further includes:

[0026] After the consumer pulls a message from the service broker or the coordination thread pool and completes consumption, it updates the consumption status corresponding to the unique identifier of the consumed message in the message status table to indicate that the message has been consumed.

[0027] On the other hand, a message consumption control device provided to meet one of the purposes of this application includes a normal processing module, an exception response module, and a message scheduling module. The normal processing module is used to send messages provided by the producer to a service broker for consumers to pull messages from the service broker. The exception response module is used to respond to any exception event of a service broker and enable a coordination thread pool for consumers to pull messages from the coordination thread pool. The message scheduling module is used to pull a portion of messages from service brokers that have not triggered exception events to the coordination thread pool, so as to distribute the portion of messages to consumers through the coordination thread pool.

[0028] In a further embodiment, the message scheduling module includes: a message retrieval submodule, used to retrieve all overloaded messages from the service broker that have not triggered abnormal events according to a preset load threshold; a message addition submodule, used to query the consumption status corresponding to the unique identifier carried by the overloaded message from the message status table, and retain the overloaded message in the coordination thread pool when the consumption status indicates that it has not been consumed; and a message discarding submodule, used to discard the corresponding overloaded message when the consumption status indicates that it has been consumed.

[0029] In a further embodiment, after the message scheduling module, there is a thread pool stopping module, which is used to stop the operation of the coordination thread pool when all service proxies that triggered the abnormal event have recovered to normal and the message status of all messages in the coordination thread pool indicates that they have been consumed.

[0030] In a further embodiment, the normal processing module includes: an indicator acquisition submodule, used to acquire system stability indicators, including the number of producers, the number of messages in the service broker, the number of consumers, and the performance information of the service broker; a threshold judgment submodule, used to determine whether each of the system stability indicators is lower than a corresponding preset stability threshold; a stability control submodule, used to suspend the consumer's message consumption when there is a system stability indicator exceeding the stability threshold, and to perform stability processing based on the system stability indicator to make it lower than the corresponding stability threshold; and a consumption recovery submodule, used to resume the consumer's message consumption when all system stability indicators are lower than the corresponding preset stability threshold.

[0031] In a further embodiment, the stability control submodule includes: a special adjustment unit, used to adjust the maximum number of messages pulled by a single consumer to a first threshold and the maximum pull time of the consumer to a second threshold based on the number of messages in the service agent that exceeds the stability threshold.

[0032] In a further embodiment, the method further includes: a message sending submodule, used to query the sending status corresponding to the unique identifier carried by the message provided by the producer from the message status table; when the sending status indicates that the message has been sent, the corresponding message is discarded; when the sending status indicates that the message has not been sent, the corresponding message is sent to the service broker; and a message insertion submodule, used by the service broker to save the message provided by the producer, insert the unique identifier carried by the message into the message status table, and mark the consumption status corresponding to the unique identifier to indicate that the message is in an unconsumed state.

[0033] In a further embodiment, the method further includes: a message consumption insertion submodule, used to update the consumption status corresponding to the unique identifier of the consumed message in the message status table after the consumer pulls the message from the service agent or from the coordination thread pool and completes consumption, so as to indicate that the message is in a consumed state.

[0034] In another aspect, a computer device provided to suit one of the purposes of this application includes a central processing unit and a memory, the central processing unit being used to invoke and run a computer program stored in the memory to perform the steps of the message consumption control method described in this application.

[0035] On another aspect, a computer-readable storage medium is provided to suit another purpose of this application, which stores, in the form of computer-readable instructions, a computer program implemented according to the described message consumption control method, which, when invoked by a computer, executes the steps included in the method.

[0036] The technical solution of this application has many advantages, including but not limited to the following aspects:

[0037] This application sends messages provided by the producer to a service broker for consumers to pull messages from. In response to any service broker's exception event, a coordination thread pool is activated for consumers to pull messages from. A portion of messages is pulled from service brokers that did not trigger exception events and then distributed to consumers through the coordination thread pool. On one hand, when any service broker malfunctions, messages that should be processed by the malfunctioning service broker are distributed to other service brokers, causing message backlogs on those brokers. This application effectively coordinates the backlogged messages from these service brokers that did not trigger exception events using the coordination thread pool, distributing them separately to consumers for pulling and consumption, thus avoiding message backlogs caused by service brokers that did not trigger exception events and ensuring the robustness and availability of message consumption. On the other hand, it guarantees the continuity of message consumption; when a service broker malfunctions, message consumption is not interrupted until all service brokers recover. On the other hand, during the recovery process of an abnormal service broker, the coordination thread pool effectively supplements the role of the abnormal service broker, eliminates the adverse effects caused by the abnormal service broker, and decouples message consumption when an abnormal service broker occurs from the restart of the service broker. Attached Figure Description

[0038] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0039] Figure 1 This is a flowchart illustrating a typical embodiment of the message consumption control method of this application;

[0040] Figure 2 This is a schematic diagram illustrating the process of adding an overload message to the coordination thread pool in an embodiment of this application;

[0041] Figure 3 This is a schematic diagram of the process for maintaining the stability of a messaging system based on system stability indicators in an embodiment of this application.

[0042] Figure 4 This is a schematic diagram of the message sending process in an embodiment of this application;

[0043] Figure 5 This is a schematic block diagram of the message consumption control device of this application;

[0044] Figure 6 This is a schematic diagram of the structure of a computer device used in this application. Detailed Implementation

[0045] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0046] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0047] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0048] Those skilled in the art will understand that the terms "client," "terminal," and "terminal device" as used herein include both devices that receive wireless signals, devices that only possess wireless signal receiver capabilities without transmission capabilities, and devices with receiving and transmitting hardware, devices that have receiving and transmitting hardware capable of bidirectional communication over a bidirectional communication link. Such devices may include: cellular or other communication devices such as personal computers or tablets, having single-line displays, multi-line displays, or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service) that can combine voice, data processing, fax, and / or data communication capabilities; PDA (Personal Digital Assistant) that may include a radio frequency receiver, pager, internet / intranet access, web browser, notepad, calendar, and / or GPS (Global Positioning System) receiver; and conventional laptops and / or handheld computers or other devices that have and / or include radio frequency receivers. As used herein, "client," "terminal," and "terminal device" can be portable, transportable, installed in a means of transportation (air, sea, and / or land), or suitable and / or configured to operate locally and / or in a distributed manner, operating in any other location on Earth and / or in space. "Client," "terminal," and "terminal device" as used herein can also be a communication terminal, an internet access terminal, or a music / video playback terminal, such as a PDA, a MID (Mobile Internet Device), and / or a mobile phone with music / video playback capabilities, or a smart TV, set-top box, etc.

[0049] The hardware referred to by the names "server," "client," and "service node" in this application is essentially an electronic device with the equivalent capabilities of a personal computer. It is a hardware device with the necessary components revealed by the von Neumann architecture, such as a central processing unit (including an arithmetic logic unit and a control unit), memory, input devices, and output devices. The computer program is stored in its memory, and the central processing unit loads the program stored in the secondary storage into the main memory to run it, execute the instructions in the program, and interact with the input and output devices to complete specific functions.

[0050] It should be noted that the concept of "server" used in this application can also be extended to apply to server clusters. Based on network deployment principles as understood by those skilled in the art, the servers should be logically divided; physically, these servers can be independent yet accessible through interfaces, or they can be integrated into a single physical computer or a computer cluster. Those skilled in the art should understand this flexibility and should not use it to constrain the implementation of the network deployment method described in this application.

[0051] One or more of the technical features of this application, unless explicitly specified herein, can be deployed on a server and accessed by a client remotely calling the online service interface provided by the server, or can be directly deployed and run on a client for access.

[0052] Unless otherwise specified, all data involved in this application may be stored remotely on a server or on a local terminal device, as long as it is suitable for use by the technical solution of this application.

[0053] Those skilled in the art will understand that although the various methods in this application are described based on the same concept and thus present commonality among them, they can be performed independently unless otherwise specified. Similarly, the various embodiments disclosed in this application are all based on the same inventive concept; therefore, concepts expressed in the same way, as well as concepts that are appropriately changed for convenience but are expressed differently, should be understood equivalently.

[0054] Unless otherwise expressly stated, the various embodiments disclosed in this application can be combined in a cross-cutting manner to flexibly construct new embodiments, as long as such combination does not depart from the inventive spirit of this application and can meet the needs of the prior art or solve a certain deficiency in the prior art. Those skilled in the art should be aware of such modifications.

[0055] The message consumption control method of this application can be programmed into a computer program product and deployed on a client or server. For example, in an exemplary application scenario of this application, it can be deployed on the server of an e-commerce platform. In this way, the method can be executed by human-computer interaction with the process of the computer program product through a graphical user interface by accessing the interface opened after the computer program product is running.

[0056] Please see Figure 1 The message consumption control method of this application, in its typical embodiment, includes the following steps:

[0057] Step S1100: Send the message provided by the producer to the service broker so that the consumer can pull the message from the service broker;

[0058] This application uses Kafka as the message middleware. Kafka is a high-throughput distributed publish-subscribe messaging system that can process all action stream data of consumers on the e-commerce platform. These actions include web browsing, product searching, product purchasing, and other user actions.

[0059] A Kafka cluster consists of one or more servers, called brokers. Programs that send messages to the brokers are called producers. Programs that subscribe to and consume messages from the brokers are called consumers. Each message published to the broker has a category called a topic, and each message belongs to one and only one topic. Each topic has at least one partition, and different partitions within the same topic contain different messages. All brokers, producers, and consumers in the Kafka cluster rely on ZooKeeper to ensure system availability.

[0060] The producer sends a message to the specified topic in the service broker. Upon receiving the message, the service broker writes it to that topic to save the message. The consumer retrieves messages from the specified topic from the service broker and then performs business processing to consume the messages. In this process, the service broker acts as an intermediary, acting as a message relay station.

[0061] Step S1200: In response to any abnormal event of the service proxy, enable the coordination thread pool so that consumers can pull messages from the coordination thread pool;

[0062] It is understandable that in industrial settings, to maximize the effective use of resources, the service brokers in a Kafka cluster are generally operating at near-maximum capacity. However, it is inevitable that some service brokers in a Kafka cluster will crash. In this case, the messages that these crashed service brokers cannot save will be distributed to the surviving service brokers for saving, greatly increasing the load on the surviving service brokers and causing them to become overloaded, resulting in message backlog.

[0063] To ensure their own survival, service brokers in the Kafka cluster send heartbeat requests to ZooKeeper at preset time intervals to report their status. ZooKeeper receives the heartbeat packet carried in the request and updates the heartbeat timestamp of the service broker that sent the heartbeat packet in its list of running service brokers belonging to the Kafka cluster, recording the latest survival time of that service broker. The heartbeat packet contains the service broker ID, service broker address, and service broker name, so ZooKeeper can determine the corresponding service broker based on the heartbeat packet and update its heartbeat timestamp accordingly. As an example, the preset time interval is set to 30 seconds, but those skilled in the art can set it as needed.

[0064] By obtaining the heartbeat timestamps of each service agent in the Kafka cluster from the service agent list maintained by ZooKeeper, calculating the time difference between the timestamps and the real-time timestamps, and determining whether the time difference is greater than a preset time period, the liveness status of each service agent can be monitored in real time. If the time difference is greater than the preset time period, it indicates that the current liveness status of the corresponding service agent is down, indicating an abnormal service agent, and triggering an abnormal event for any service agent. Otherwise, it indicates that the current liveness status of each service agent is alive, there is no abnormal service agent, and no abnormal event for any service agent is triggered.

[0065] In response to any abnormal event of a service broker, enable the coordination thread pool to handle the backlog of messages caused by overload of surviving service brokers in the Kafka cluster.

[0066] Step S1300: Retrieve a portion of messages from the service broker that has not triggered an abnormal event to the coordination thread pool, so as to distribute the portion of messages to consumers through the coordination thread pool.

[0067] The service brokers that did not trigger abnormal events are the surviving service brokers in the Kafka cluster. Multiple threads in the coordination thread pool are responsible for pulling partial messages from each partition of each topic within each service broker. These partial messages are divided according to the total number of partitions contained in each service broker, identifying messages belonging to different partitions. Messages belonging to different partitions are then distributed to consumers. Messages belonging to the same partition can only be assigned to one consumer, ensuring that a consumer can consume messages from only one partition at a time. After all messages in a partition have been pulled and consumed by the consumer, a partial message from the corresponding partition is pulled again. This partial message consists of messages exceeding the load limit of the surviving service brokers in the Kafka cluster. The specific number of threads in the coordination thread pool can be set as needed by those skilled in the art; more threads mean more messages can be pulled.

[0068] It is understood that multiple threads in the coordination thread pool can quickly retrieve partial information from service proxies that have not triggered abnormal events, effectively resolve the backlog of messages from these service proxies, and distribute the partial information to consumers in an orderly manner.

[0069] As can be seen from the typical embodiments of this application, the technical solution of this application has many advantages, including but not limited to the following aspects:

[0070] This application sends messages provided by the producer to a service broker for consumers to pull messages from. In response to any service broker's exception event, a coordination thread pool is activated for consumers to pull messages from. A portion of messages is pulled from service brokers that did not trigger exception events and then distributed to consumers through the coordination thread pool. On one hand, when any service broker malfunctions, messages that should be processed by the malfunctioning service broker are distributed to other service brokers, causing message backlogs on those brokers. This application effectively coordinates the backlogged messages from these service brokers that did not trigger exception events using the coordination thread pool, distributing them separately to consumers for pulling and consumption, thus avoiding message backlogs caused by service brokers that did not trigger exception events and ensuring the robustness and availability of message consumption. On the other hand, it guarantees the continuity of message consumption; when a service broker malfunctions, message consumption is not interrupted until all service brokers recover. On the other hand, during the recovery process of an abnormal service broker, the coordination thread pool effectively supplements the role of the abnormal service broker, eliminates the adverse effects caused by the abnormal service broker, and decouples message consumption when an abnormal service broker occurs from the restart of the service broker.

[0071] Please see Figure 2 In a further embodiment, step S1300, pulling a portion of messages from the service broker that has not triggered an abnormal event to the coordination thread pool, includes the following steps:

[0072] Step S1310: Retrieve all overload messages from the service agent that has not triggered an abnormal event according to the preset load threshold;

[0073] Service proxies that do not trigger abnormal events are limited by their own hardware configuration, and the corresponding number of messages that can be processed has an upper limit, namely the load threshold, which shall be specifically set by those skilled in the art in accordance with the settings disclosed herein.

[0074] Multiple threads in the coordination thread pool pull all overload information exceeding the load threshold for each service broker that has not triggered an exception event, in order to resolve the backlog of messages in these service brokers. The overload information is stored in each partition of a topic within the service broker. The specific number of threads in the coordination thread pool can be set as needed by those skilled in the art; more threads mean more messages are pulled.

[0075] Step S1320: Using the unique identifier carried by the overload message, query the consumption status corresponding to the unique identifier from the message status table. When the consumption status indicates that the message has not been consumed, retain the overload message in the coordination thread pool.

[0076] It is understandable that when a consumer pulls messages from the service broker for consumption, there may be instances where the consumer crashes, restarts, or is forcibly terminated, resulting in the consumer's consumed offset not being committed. Alternatively, if the consumer's subscription is canceled before the consumer process is shut down, some offsets may not be committed. Or, the consumer may time out while consuming messages, triggering a rebalance, which may also result in uncommitted consumer offsets. All of these situations will cause the consumer to repeatedly consume messages with uncommitted offsets, severely compromising the idempotency of message consumption.

[0077] To ensure idempotency in message consumption, a unique identifier corresponding to each message provided by the producer is pre-generated. This generation can be implemented using a distributed ID algorithm, including UUID, database auto-incrementing ID algorithms, Snowflake, MD5, MongoDB's ObjectID, TinyID, Uidgenerator, Leaf, and Redis algorithms, etc. Those skilled in the art can choose one to implement as needed. Therefore, each overloaded message carries a corresponding unique identifier.

[0078] Secondly, maintain a message status table, which stores the consumption status of each message in all service brokers in the Kafka cluster. Specifically, the unique identifier corresponding to each message in the message status table is mapped to the corresponding consumption status, which indicates whether the message has been consumed or not.

[0079] Accordingly, the consumption status of each overloaded message is mapped to the corresponding unique identifier in the message status table. When the consumption status indicates that the message has not been consumed, it means that the message has not been consumed and needs to be further sent to the consumer for consumption. The overloaded message is then retained in the coordination thread pool for distribution to the consumer.

[0080] Step S1330: When the consumption status indicates that the message has been consumed, discard the corresponding overload message.

[0081] When the consumption status indicates that the message has been consumed, it means that the corresponding message has been consumed by the consumer and does not need to be consumed again. Therefore, the corresponding overloaded message is discarded from the coordination thread pool.

[0082] In this embodiment, by querying the message status table, it is determined whether the messages pulled by the coordination thread pool have been consumed and whether they are retained or discarded accordingly. This ensures that all messages in the coordination thread pool are unconsumed, guarantees the idempotency of message consumption, avoids duplicate message consumption, and provides high availability, high efficiency, and low latency.

[0083] In a further embodiment, after step S1300, which involves pulling a portion of the message from the service broker that has not triggered an abnormal event to the coordination thread pool, and then distributing the portion of the message to the consumer through the coordination thread pool, the process includes:

[0084] Step S1400: When all service proxies that triggered the abnormal event have returned to normal, and the message status of all messages in the coordination thread pool indicates that they have been consumed, stop the operation of the coordination thread pool.

[0085] During the coordination of the thread pool, all service proxies that triggered abnormal events, i.e. all service proxies that crashed, will be restarted and restored to normal working status.

[0086] When these service proxies restart, to ensure their survival, they send a heartbeat request to ZooKeeper every preset time interval to report their status. ZooKeeper receives the heartbeat packet carried in the heartbeat request and updates the heartbeat timestamp of the service proxies that sent the heartbeat packet in its list of running service proxies belonging to the Kafka cluster, recording the latest survival time of that service proxies. The heartbeat packet contains the service proxies ID, address, and name, so ZooKeeper can determine the corresponding service proxies based on the heartbeat packet and update their heartbeat timestamps accordingly. The preset time interval is typically set to 30 seconds, but those skilled in the art can set it as needed.

[0087] By obtaining the heartbeat timestamps of all service proxies that triggered the abnormal event from the service proxy list maintained by ZooKeeper, calculating the time difference between the timestamps and the real-time timestamps, and determining whether the time difference is greater than the preset time period, the liveness status of each service proxy can be monitored in real time. When the heartbeat timestamps of all service proxies are less than or equal to the preset time period, it indicates that all service proxies have recovered to normal.

[0088] Furthermore, based on the unique identifiers carried by all messages in the coordination thread pool, the message status table is retrieved to determine the consumption status associated with each unique identifier. When there is no consumption status indicating that the messages have not been consumed, it means that all messages have been consumed.

[0089] As described above, when it is determined that all service proxies that triggered the abnormal event have recovered to normal and all messages in the coordination thread pool have been consumed, it means that all service proxies in the current Kafka cluster are alive and can work normally as relay stations for storing messages, without message backlog. Moreover, the backlog of messages handled by the coordination thread pool has been consumed. Therefore, the coordination thread pool no longer needs to work and should be stopped.

[0090] In this embodiment, the coordination thread pool is stopped when all service proxies that triggered the abnormal event have recovered and all messages in the coordination thread pool have been consumed. This automation requires no human intervention and ensures that the coordination thread pool is stopped appropriately and promptly.

[0091] Please see Figure 3 In a further embodiment, step S1100, sending the message provided by the producer to the service broker so that the consumer can pull the message from the service broker, includes the following steps:

[0092] Step S1110: Obtain system stability indicators, which include the number of producers, the number of messages in the service broker, the number of consumers, and the performance information of the service broker.

[0093] It is understood that changes in any of the following—the number of producers, the number of messages in the service brokers, the number of consumers, or the performance information of each service broker in the Kafka cluster—may affect the stability of the message publish-subscribe system. Therefore, to ensure the stability of the message publish-subscribe system, each potentially changing data point is treated as a system stability indicator and acquired separately. Each system stability indicator is evaluated to determine whether it affects the stability of the message publish-subscribe system. The execution entity of the message publish-subscribe system includes the producers, all service brokers in the Kafka cluster, and the consumers. It is understood that the producers send messages to the service brokers, the service brokers store the information, and the consumers subscribe to and consume the information stored by the service brokers, thus constituting the message publish-subscribe system.

[0094] The performance information of the service agent includes any one or more of the following: CPU utilization, memory usage and / or disk usage and / or disk utilization.

[0095] Step S1120: Determine whether all the system stability indicators are lower than the corresponding preset stability thresholds;

[0096] It's easy to understand that the smaller the various system stability indicators, the more stable the corresponding message publishing and subscription system. Conversely, when these indicators become too large, they can negatively impact the system's stability. Therefore, to determine whether each system stability indicator is excessively high and thus affects the stability of the message publishing and subscription system, corresponding stability thresholds can be set for each indicator. Specifically, the system stability indicators at the system's performance limit can be pre-obtained and set as the corresponding stability thresholds for each indicator.

[0097] Step S1130: When there is a system stability index that exceeds the stability threshold, pause the consumer's message consumption and perform stability processing according to the system stability index to make it lower than the corresponding stability threshold.

[0098] When a system stability index exceeds a stability threshold, it indicates that the system stability index is too high, exceeding the performance limit of the message publish-subscribe system and affecting its stability. In this case, the operation of consumers consuming information in the message publish-subscribe system is suspended, and stability processing is performed according to the system stability index to bring it below the corresponding stability threshold, thereby restoring the stability of the message publish-subscribe system.

[0099] In one embodiment, when the system stability indicator exceeding the stability threshold is the number of producers, the corresponding stability handling may be to perform a temporary expansion operation, increasing the number of consumers and the number of partitions of the same topic in the service broker of the Kafka cluster. When the number of producers falls below the corresponding stability threshold, the original number of consumers and the number of partitions of the same topic in the service broker of the Kafka cluster are restored.

[0100] In one embodiment, when the system stability indicator exceeding the stability threshold is the number of messages handled by the service broker, the maximum number of messages pulled by a single consumer is adjusted to a first threshold, and the maximum pull time for the consumer is adjusted to a second threshold to reduce unnecessary rebalancing operations on the consumer. For example, the first threshold is 500 messages, and the second threshold is recommended to be 5 minutes. Those skilled in the art can set these thresholds according to the actual consumption rate and consumption situation.

[0101] In one embodiment, when the system stability indicator exceeding the stability threshold is the number of messages from the service broker or the performance information of the service broker, the coordination thread pool implemented in this application is enabled to pull messages from the service broker that exceed the stability threshold so that consumers can pull information from the coordination thread pool and consume it. When the number of messages from the service broker or the performance information of the service broker is lower than the corresponding stability threshold, and all messages in the coordination thread pool have been consumed, the coordination thread pool is stopped.

[0102] In one embodiment, when the system stability indicator exceeding the stability threshold is the number of consumers, and this causes an increase in the corresponding number of consumer groups, the number of partitions in the topic within the service agent is adjusted according to the current number of consumer groups, so that the number of partitions is greater than the number of consumer groups. Once the number of consumers falls below the corresponding stability threshold, the original number of partitions in the topic within the service agent is restored.

[0103] Step S1140: When all system stability indicators are lower than the corresponding preset stability thresholds, restore the consumer consumption message.

[0104] If all the system stability indicators are lower than the corresponding preset stability thresholds, it means that the consumer publish-subscribe system is not operating beyond its performance limit, i.e., the consumer publish-subscribe system is stable. Therefore, the consumer can be replied to with consumer messages.

[0105] In this embodiment, multiple system stability indicators affecting system stability are obtained and compared with their corresponding stability thresholds for judgment. When any system stability indicator exceeds the stability threshold, the consumer's message consumption is paused, and stability processing is performed based on the system stability indicators. When all system stability indicators are below their respective stability thresholds, the consumer resumes message consumption. This ensures system stability, guarantees message consumption when the system is stable, and effectively avoids errors.

[0106] Please see Figure 4 In a further embodiment, the method further includes the following steps:

[0107] Step S2100: Using the unique identifier carried by the message provided by the producer, query the message status table for the sending status corresponding to the unique identifier. When the sending status indicates that the message has been sent, discard the corresponding message. When the sending status indicates that the message has not been sent, send the corresponding message to the service agent.

[0108] To ensure the idempotency of message transmission, firstly, a unique identifier corresponding to each message provided by the producer is pre-generated. This generation can be implemented using a distributed ID algorithm, including UUID algorithm, database auto-incrementing ID algorithm, snowflake algorithm, MD5 algorithm, MongoDB's ObjectID algorithm, TinyID algorithm, Uidgenerator algorithm, Leaf algorithm, Redis algorithm, etc. Those skilled in the art can choose one to implement as needed.

[0109] The system checks if the unique identifier carried by the message provided by the producer exists in the message status table. If it exists, it indicates that the corresponding message has been sent and does not need to be sent again; the message is discarded. If it does not exist, it indicates that the corresponding message has not been sent and needs to be sent to the service agent. The system then sends the corresponding message to the service agent.

[0110] Step S2200: The service agent saves the message provided by the producer, inserts the unique identifier it carries into the message status table, and marks the consumption status corresponding to the unique identifier to indicate that the message is in an unconsumed state.

[0111] The service agent stores the message provided by the producer, indicating that the message's sending status should be "sent". It inserts the unique identifier it carries into the message status table. Furthermore, if the message has not been given to the consumer for consumption, it indicates that the message's consumption status should be "unconsumed". The unique identifier is marked with the associated consumption status to indicate that the message is in an unconsumed state.

[0112] In this embodiment, by querying the message status table, it is determined whether the message sent to the service agent has been sent and then sent or discarded accordingly. This ensures that all messages sent to the service agent are unsent, guarantees the idempotency of message sending, avoids duplicate message sending, and has high availability, high efficiency, and low latency.

[0113] In a further embodiment, the method further includes:

[0114] Step S2300: After the consumer pulls a message from the service agent or from the coordination thread pool and completes consumption, it updates the consumption status corresponding to the unique identifier of the consumed message in the message status table to indicate that the message has been consumed.

[0115] After a consumer consumes a message, it indicates that the message's consumption status should be "consumed". The consumer then retrieves the unique identifier of the message in the consumption status table and updates the associated consumption status to indicate that the message has been consumed.

[0116] In this embodiment, the update of the message status corresponding to the unique identifier of the message in the consumption status table is disclosed to ensure that the consumption status can be updated in a timely manner in response to the consumption of the corresponding message. This ensures high availability, high efficiency and low latency, and lays a solid foundation for the idempotency of message consumption.

[0117] Please see Figure 5 This application provides a message consumption control device to meet one of its objectives. It is a functional embodiment of the message consumption control method of this application. The device includes a normal processing module 1100, an exception response module 1200, and a message scheduling module 1300. The normal processing module 1100 sends messages provided by the producer to a service broker for consumers to pull messages from the service broker. The exception response module 1200 responds to any exception event of a service broker by activating a coordination thread pool for consumers to pull messages from the coordination thread pool. The message scheduling module 1300 pulls a portion of messages from service brokers that have not triggered exception events to the coordination thread pool, and distributes the partial messages to consumers through the coordination thread pool.

[0118] In a further embodiment, the message scheduling module 1300 includes: a message retrieval submodule, used to retrieve all overloaded messages from the service agent that have not triggered an abnormal event according to a preset load threshold; a message addition submodule, used to query the consumption status corresponding to the unique identifier carried by the overloaded message from the message status table, and retain the overloaded message in the coordination thread pool when the consumption status indicates that it has not been consumed; and a message discarding submodule, used to discard the corresponding overloaded message when the consumption status indicates that it has been consumed.

[0119] In a further embodiment, after the message scheduling module 1300, there is a thread pool stopping module, which is used to stop the work of the coordination thread pool when all service proxies that triggered the abnormal event have recovered to normal and the message status of all messages in the coordination thread pool indicates that they have been consumed.

[0120] In a further embodiment, the normal processing module 1100 includes: an indicator acquisition submodule, used to acquire system stability indicators, the system stability indicators including the number of producers, the number of messages in the service broker, the number of consumers, and the performance information of the service broker; a threshold judgment submodule, used to determine whether each of the system stability indicators is lower than a corresponding preset stability threshold; a stability control submodule, used to suspend the consumer's message consumption when there is a system stability indicator that exceeds the stability threshold, and to perform stability processing according to the system stability indicator to make it lower than the corresponding stability threshold; and a consumption recovery submodule, used to resume the consumer's message consumption when all system stability indicators are lower than the corresponding preset stability threshold.

[0121] In a further embodiment, the stability control submodule includes: a special adjustment unit, used to adjust the maximum number of messages pulled by a single consumer to a first threshold and the maximum pull time of the consumer to a second threshold based on the number of messages in the service agent that exceeds the stability threshold.

[0122] In a further embodiment, the method further includes: a message sending submodule, used to query the sending status corresponding to the unique identifier carried by the message provided by the producer from the message status table; when the sending status indicates that the message has been sent, the corresponding message is discarded; when the sending status indicates that the message has not been sent, the corresponding message is sent to the service broker; and a message insertion submodule, used by the service broker to save the message provided by the producer, insert the unique identifier carried by the message into the message status table, and mark the consumption status corresponding to the unique identifier to indicate that the message is in an unconsumed state.

[0123] In a further embodiment, the method further includes: a message consumption insertion submodule, used to update the consumption status corresponding to the unique identifier of the consumed message in the message status table after the consumer pulls the message from the service agent or from the coordination thread pool and completes consumption, so as to indicate that the message is in a consumed state.

[0124] To address the aforementioned technical problems, embodiments of this application also provide computer equipment. For example... Figure 6 The diagram shows the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. The computer-readable storage medium stores an operating system, a database, and computer-readable instructions. The database may store a sequence of control information. When the computer-readable instructions are executed by the processor, they enable the processor to implement a message consumption control method. The processor of the computer device provides computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When these computer-readable instructions are executed by the processor, they enable the processor to execute the message consumption control method of this application. The network interface of the computer device is used for communication with a terminal. Those skilled in the art will understand that… Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0125] In this embodiment, the processor is used to execute... Figure 5The specific functions of each module and its submodules are defined within the system. The memory stores the program code and various data required to execute these modules or submodules. The network interface is used for data transmission between the user terminal and the server. In this embodiment, the memory stores the program code and data required to execute all modules / submodules in the message consumption control device of this application. The server can call the server's program code and data to execute the functions of all submodules.

[0126] This application also provides a storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the message consumption control method of any embodiment of this application.

[0127] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. This computer program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0128] In summary, this application, on the one hand, can coordinate the processing of messages from service proxies that are not experiencing anomalies when such proxies malfunction, effectively avoiding message backlog and ensuring the robustness, availability, and continuity of message consumption. On the other hand, this application can guarantee the idempotency of message consumption and message sending, and has high availability, high efficiency, and low latency.

[0129] Those skilled in the art will understand that the steps, measures, and solutions in the various operations, methods, and processes discussed in this application can be alternated, modified, combined, or deleted. Furthermore, other steps, measures, and solutions in the various operations, methods, and processes discussed in this application can also be alternated, modified, rearranged, decomposed, combined, or deleted. Furthermore, steps, measures, and solutions in the prior art that are similar to those disclosed in this application can also be alternated, modified, rearranged, decomposed, combined, or deleted.

[0130] The above description is only a partial embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A message consumption control method, characterized in that, Includes the following steps: The producer sends the message to the service broker so that the consumer can pull the message from the service broker; In response to any abnormal event of the service broker, enable the coordination thread pool so that consumers can pull messages from the coordination thread pool; A portion of messages are retrieved from service brokers that have not triggered any abnormal events and sent to the coordination thread pool for distribution to consumers. This includes retrieving all overloaded messages from service brokers that have not triggered abnormal events based on a preset load threshold; querying the consumption status corresponding to the unique identifier carried by the overloaded message from the message status table; retaining the overloaded message in the coordination thread pool when the consumption status indicates that it has not been consumed; and discarding the corresponding overloaded message when the consumption status indicates that it has been consumed. When all service proxies that triggered the abnormal event have recovered to normal, and the message status of all messages in the coordination thread pool indicates that they have been consumed, the operation of the coordination thread pool is stopped.

2. The message consumption control method according to claim 1, characterized in that, Sending messages provided by the producer to a service broker for consumers to pull messages from the service broker includes the following steps: Obtain system stability indicators, including the number of producers, the number of messages in the service broker, the number of consumers, and the performance information of the service broker; Determine whether all the system stability indicators are lower than the corresponding preset stability thresholds; When a system stability indicator exceeds a stability threshold, the consumer's message consumption is paused, and stability processing is performed based on the system stability indicator to bring it below the corresponding stability threshold. When all system stability indicators are lower than the corresponding preset stability thresholds, the consumer consumption message is restored.

3. The message consumption control method according to claim 2, characterized in that, The system stability index is adjusted to make it lower than the corresponding stability threshold, including: adjusting the maximum number of messages pulled by a single consumer to a first threshold based on the number of messages in the service broker that exceeds the stability threshold, and adjusting the maximum pull time of the consumer to a second threshold.

4. The message consumption control method according to claim 1, characterized in that, This method also includes the following steps: Using the unique identifier carried in the message provided by the producer, the corresponding sending status is queried from the message status table. When the sending status indicates that the message has been sent, the corresponding message is discarded. When the sending status indicates that the message has not been sent, the corresponding message is sent to the service broker. The service agent stores the messages provided by the producer, inserts the unique identifier carried by the message into the message status table, and marks the consumption status corresponding to the unique identifier to indicate that the message is in an unconsumed state.

5. The message consumption control method according to claim 1, characterized in that, This method also includes: After the consumer pulls a message from the service broker or the coordination thread pool and completes consumption, it updates the consumption status corresponding to the unique identifier of the consumed message in the message status table to indicate that the message has been consumed.

6. A message consumption control device, characterized in that, include: The normal processing module is used to send messages provided by the producer to the service broker so that consumers can pull messages from the service broker; An exception response module is used to respond to exception events of any service proxy and enable a coordination thread pool for consumers to pull messages from the coordination thread pool. The message scheduling module is used to pull a portion of messages from the service broker that has not triggered any abnormal events to the coordination thread pool, so as to distribute the portion of messages to consumers through the coordination thread pool. This includes pulling all overloaded messages from the service broker that has not triggered any abnormal events according to a preset load threshold. Using the unique identifier carried by the overload message, query the message status table for the consumption status corresponding to the unique identifier. When the consumption status indicates that the message has not been consumed, retain the overload message in the coordination thread pool; when the consumption status indicates that the message has been consumed, discard the corresponding overload message. When all service proxies that triggered the abnormal event have recovered to normal, and the message status of all messages in the coordination thread pool indicates that they have been consumed, the operation of the coordination thread pool is stopped.

7. The message consumption control device according to claim 6, characterized in that, The normal processing module includes: an indicator acquisition submodule, used to acquire system stability indicators, including the number of producers, the number of messages in the service broker, the number of consumers, and the performance information of the service broker; a threshold judgment submodule, used to determine whether all the system stability indicators are lower than the corresponding preset stability thresholds; a stability control submodule, used to pause the consumer's message consumption when there are system stability indicators exceeding the stability thresholds, and to perform stability processing based on the system stability indicators to bring them below the corresponding stability thresholds; and a consumption recovery submodule, used to resume the consumer's message consumption when all the system stability indicators are lower than the corresponding preset stability thresholds.

8. The message consumption control device according to claim 7, characterized in that, The stability control submodule includes: a special adjustment unit, used to adjust the maximum number of messages pulled by a single consumer to a first threshold and the maximum pulling time of the consumer to a second threshold based on the number of messages in the service agent that exceeds the stability threshold.

9. A computer device comprising a central processing unit and a memory, characterized in that, The central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the method as described in any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that, It stores, in the form of computer-readable instructions, a computer program implemented according to any one of claims 1 to 5, which, when invoked by a computer, executes the steps included in the corresponding method.

Images