Message processing method, electronic device, storage medium, and program product

Abstract

This disclosure provides a message processing method, an electronic device, a readable storage medium, and a computer program product. The message processing method of this disclosure includes: obtaining the current unconsumed message volume and the current message throughput of a target partition managed by a target consumer; determining whether the target consumer needs to be expanded based on the current unconsumed message volume and the current message throughput; in response to the target consumer needing to be expanded, creating a slave consumer for the target consumer; in response to receiving a set of messages to be processed from the target partition, determining, based on matching rules and the content of each message in the set of messages to be processed, a consumer that matches each message in the set of messages to be processed from the target consumer and the slave consumer; processing the messages that match the target consumer, and sending the messages that match the slave consumer to the slave consumer; the slave consumer is used to process the received messages.

Description

Technical Field

[0001] This disclosure relates to a message processing method, an electronic device, a storage medium, and a program product. Background Technology

[0002] With the increasing prevalence of distributed systems and microservice architectures, message queues (such as RocketMQ and Kafka) are widely used in numerous fields, including the internet, finance, and e-commerce, as core components for achieving system decoupling, asynchronous communication, and traffic shaping. Message queues typically manage messages by categorizing them into topics, each of which can be divided into multiple partitions to enable parallel message processing and storage, significantly improving system throughput and scalability. In the message queue's operating mechanism, consumers typically consume multiple partitions within a topic through consumer groups. Each partition can only be consumed exclusively by one consumer within the group, ensuring message accuracy.

[0003] In real-world production environments, when a consumer's processing capacity is insufficient, its program malfunctions, or unexpected resource bottlenecks occur, message consumption in its partition may lag or even accumulate. This accumulation not only causes data delays but can also lead to serious consequences such as data loss and business process blockages, posing a significant threat to the stability and reliability of the entire system. The current mainstream solution is to use temporary topics. The specific steps include: for the target partition experiencing accumulation, suspending the original consumer's message retrieval or processing for that partition; creating a new temporary topic and setting the number of partitions in this temporary topic to N times the number of partitions in the target partition's topic (N being a positive integer greater than 1); forwarding messages from the target partition to the temporary topic; horizontally scaling the consumer cluster to N times its original size and mounting all consumers to the temporary topic for parallel message processing; and restoring the original message consumption architecture after all messages in the temporary topic have been processed.

[0004] However, because each stage of message processing is closely coupled with specific business operations and lacks a universal processing mechanism, existing solutions require ad-hoc development work each time they are dealing with message backlog, increasing the complexity and manpower costs of message processing. Summary of the Invention

[0005] This disclosure provides a message processing method, an electronic device, a storage medium, and a program product.

[0006] According to one aspect of this disclosure, a message processing method is provided, wherein the execution subject of the method is a target consumer, comprising: obtaining the current unconsumed message volume of a target partition under the responsibility of the target consumer and the current message throughput of the target consumer; determining whether the target consumer needs to be expanded based on the current unconsumed message volume and the current message throughput; in response to the target consumer needing to be expanded, creating a slave consumer for the target consumer; in response to receiving a set of messages to be processed sent by the target partition, determining, based on a matching rule and the content of each message in the set of messages to be processed, a consumer that matches each message in the set of messages to be processed from the target consumer and the slave consumer; processing the messages that match the target consumer, and sending the messages that match the slave consumer to the slave consumer; wherein the slave consumer is used to process the received messages.

[0007] According to one technical solution, decoupling the scaling from the consumer end from the message processing from the consumer end can avoid redundant development and maintenance costs.

[0008] According to at least one embodiment of the message processing method of this disclosure, after sending the message matching the slave consumer to the slave consumer, the method further includes: in response to the processing of messages by the target consumer and the slave consumer, monitoring the real-time unconsumed message volume of the target partition and the real-time message throughput of the target consumer and the slave consumer; determining whether the target consumer needs to be scaled down based on the real-time unconsumed message volume and the real-time message throughput; and in response to the need for the target consumer to be scaled down, deregistering the slave consumer.

[0009] According to the technical solution of this embodiment, redundant resources can be released and resource utilization can be improved.

[0010] According to at least one embodiment of the message processing method of this disclosure, the step of determining whether the target consumer needs to be expanded based on the current unconsumed message volume and the current message throughput includes: obtaining the message consumption delay time of the target partition based on the current unconsumed message volume and the current message throughput; and determining whether the target consumer needs to be expanded based on the consumption delay time.

[0011] According to the technical solution of this embodiment, excessive expansion can be prevented, resource waste can be avoided, and the effectiveness of expansion can be improved.

[0012] According to at least one embodiment of the message processing method of this disclosure, the step of obtaining the message consumption delay time of the target partition based on the current unconsumed message volume and the current message throughput includes: obtaining an unconsumed predicted value based on the current unconsumed message volume; and obtaining the message consumption delay time of the target partition based on the unconsumed predicted value and the current message throughput.

[0013] According to the technical solution of this embodiment, potential performance bottlenecks can be exposed in advance, enabling predictive capacity expansion.

[0014] According to at least one embodiment of the message processing method of this disclosure, the step of obtaining the unconsumed predicted value based on the current unconsumed message volume includes: obtaining a stacking smoothing value, a stacking trend value, and a prediction step number based on the current unconsumed message volume; multiplying the stacking trend value by the prediction step number to obtain a stacking compensation amount; and adding the stacking smoothing value to the stacking compensation amount to obtain the unconsumed predicted value.

[0015] According to the technical solution of this embodiment, noise interference can be eliminated, and prediction stability and efficiency can be improved.

[0016] According to at least one embodiment of the message processing method of this disclosure, the step of creating slave consumers for the target consumer based on the current unconsumed message volume and the current message throughput includes: performing elastic scaling calculation based on the current unconsumed message volume and the current message throughput to obtain the scaling quantity; and creating the scaling quantity of slave consumers for the target consumer.

[0017] According to the technical solution of this embodiment, the system's processing capacity during peak message accumulation periods can be significantly enhanced, achieving dynamic and elastic improvement in consumption capacity.

[0018] According to at least one embodiment of the message processing method of this disclosure, the step of performing elastic scaling calculation based on the current unconsumed message volume and the current message throughput includes: obtaining the current number of consumers in the target partition; dividing the current unconsumed message volume by the current message throughput to obtain the consumer demand; and subtracting the current number of consumers from the consumer demand.

[0019] According to the technical solution of this embodiment, a flexible expansion effect with rapid response and reasonable resource utilization can be achieved.

[0020] According to at least one embodiment of the message processing method of this disclosure, the step of determining the consumer end that matches each message in the message set to be processed from the target consumer end and the slave consumer end according to the matching rules and the content of each message in the message set to be processed includes: constructing an initial hash ring; mapping the target consumer end and the slave consumer end onto the initial hash ring to obtain a first hash ring; performing hash processing on the messages in the message set to be processed to obtain hash values; mapping the hash values ​​onto the first hash ring to obtain a second hash ring; and for any message in the target partition, matching the message with the hash value of the message at the first consumer end in a clockwise direction on the second hash ring.

[0021] According to the technical solution of this embodiment, load balancing can be achieved, thereby improving the overall throughput and resource utilization of the system.

[0022] According to at least one embodiment of the message processing method of this disclosure, the slave consumer is specifically used to detect whether the target consumer is abnormal; in response to the target consumer being normal, to process the received message; in response to the target consumer being abnormal, to re-execute the detection of whether the target consumer is abnormal until the target consumer is normal or a cutoff condition is reached, and in response to the cutoff condition being reached, to cancel the slave consumer.

[0023] According to the technical solution of this embodiment, a highly reliable, low-interference, and automated consumer fault-tolerant mechanism can be achieved.

[0024] According to at least one embodiment of the message processing method of this disclosure, the step of obtaining information from the target consumer is specifically used to detect whether the target consumer is abnormal based on the information from the target consumer.

[0025] According to the technical solution of this embodiment, a clear basis can be provided for judging anomalies on the target consumer side.

[0026] According to at least one embodiment of the message processing method of this disclosure, the information of the target consumer includes the IP address of the target consumer, metadata and statistical information for tracking consumption progress and status, and an ordered set of consumption progress; and / or the information of the target consumer is stored in a memory data storage system.

[0027] According to the technical solution of this embodiment, the abnormal state of the target consumer can be identified in a multi-dimensional and low-latency manner, and the system access speed, concurrent processing capability and real-time responsiveness can be significantly improved, thereby improving message processing speed.

[0028] According to at least one embodiment of the message processing method of this disclosure, after sending the message matching the slave consumer to the slave consumer, the method further includes: in response to receiving the message consumption progress of the target consumer and / or the slave consumer, sending the message consumption progress to the target partition.

[0029] The technical solution according to this embodiment can improve reliability.

[0030] According to at least one embodiment of the message processing method of this disclosure, sending a message matching the slave consumer to the slave consumer includes: detecting whether each of the slave consumers is available; and sending a message assigned to the slave consumer to an available slave consumer.

[0031] According to the technical solution of this embodiment, messages can be processed in a timely and correct manner, thereby improving the system's throughput and reliability.

[0032] According to another aspect of this disclosure, an electronic device is provided, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, causing the processor to perform a message processing method according to any embodiment of this disclosure.

[0033] According to another aspect of this disclosure, a readable storage medium is provided, wherein executable instructions are stored therein, which, when executed by a processor, are used to implement a message processing method according to any embodiment of this disclosure.

[0034] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements a message processing method according to any embodiment of this disclosure. Attached Figure Description

[0035] The accompanying drawings illustrate exemplary embodiments of the present disclosure and, together with the description thereof, serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0036] Figure 1 This is an illustrative interactive flow of a message processing method according to one embodiment of the present disclosure. Figure 1 .

[0037] Figure 2 This is an illustrative interactive flow of a message processing method according to one embodiment of the present disclosure. Figure 2 .

[0038] Figure 3 yes Figure 1The flowchart shows the expansion judgment method in the message processing method shown.

[0039] Figure 4 yes Figure 3 The illustrated interactive flow of the delayed acquisition method in the expansion judgment method is shown. Figure 1 .

[0040] Figure 5 yes Figure 3 The illustrated interactive flow of the delayed acquisition method in the expansion judgment method is shown. Figure 2 .

[0041] Figure 6 yes Figure 1 The diagram shows an illustrative interactive flowchart of the expansion method in the message processing method shown.

[0042] Figure 7 yes Figure 6 The diagram shows an illustrative interactive flowchart of the expansion calculation method.

[0043] Figure 8 yes Figure 1 The diagram shows an illustrative interactive flowchart of the message matching method in the message processing method shown.

[0044] Figure 9 This is an illustrative interactive flow of a message processing method according to one embodiment of the present disclosure. Figure 3 .

[0045] Figure 10 yes Figure 1 The diagram shows an illustrative interactive flowchart of the message sending method in the message processing method shown.

[0046] Figure 11 This is an illustrative interactive flowchart of a message processing method according to one embodiment of the present disclosure.

[0047] Figure 12 This is a schematic structural block diagram of a message processing apparatus according to one embodiment of the present disclosure.

[0048] Figure 13 This is a schematic structural block diagram of an electronic device according to one embodiment of the present disclosure. Detailed Implementation

[0049] The present disclosure will now be described in further detail with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the accompanying drawings.

[0050] It should be noted that, where there is no conflict, the embodiments and features described in this disclosure can be combined with each other. The technical solutions of this disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0051] Taking the order system of a large e-commerce platform as an example, during promotional activities, the surge in user order requests leads to a sudden surge in concurrent message flows within the order system and the order payment topic. One partition within this order payment topic, handling a large number of order requests from popular users, causes its associated consumer to be unable to keep up with the message backlog, resulting in a severe message backlog problem. To quickly alleviate the pressure on this partition, the operations team implemented the following approach, following existing mainstream methods: First, the existing consumer was paused from fetching and consuming messages from this partition via the consumption control interface; second, a temporary topic was created, with its number of partitions set to four times the number of partitions in the order payment topic; third, a forwarding program was deployed to continuously fetch historical backlog messages from this partition and forward them to the temporary topic; then, the number of consumer groups was expanded from the original 10 to 40, enabling them to consume all partitions of the temporary topic in parallel, accelerating message processing; finally, when all messages in the temporary topic were processed, the forwarding program was stopped, the number of consumer groups was reduced, and the original consumption structure of the order payment topic was restored. The above process relies on manual configuration and code deployment for multiple steps, and the processing logic needs to be redeveloped each time a similar problem is handled. This results in high development complexity and manpower costs for large e-commerce platforms when facing high-frequency hotspot scenarios, limiting their response efficiency.

[0052] To address this issue, the present disclosure proposes the following technical solution, which resolves the problem that existing solutions require ad-hoc development work each time they deal with message backlog due to the close coupling between each stage of message processing and specific business operations, and the lack of a universal processing mechanism, thereby increasing the complexity and manpower costs of message processing.

[0053] To facilitate description and make the technical solutions of this disclosure easier to understand, the terminology of this disclosure will be explained before describing the technical solutions of this disclosure.

[0054] The consumer is a component responsible for actively subscribing to (or fetching) messages from a specified topic and executing business processing logic on the messages.

[0055] Unconsumed messages refer to the number of messages in a message queue that have not been consumed in a timely manner within a specific topic or partition.

[0056] Message throughput refers to the number of messages successfully consumed by the consumer within a unit of time.

[0057] Consumption delay time refers to the time difference between when a message is produced and when it is processed.

[0058] The expansion capacity refers to the difference between the current number of consumers and the current number of consumers required to achieve the desired processing speed or clear the backlog of messages when the current system's consumption capacity is insufficient to handle message backlog.

[0059] A hash ring is a ring structure used in consistent hashing algorithms to map data to nodes.

[0060] Figure 1 A schematic flowchart of a message processing method M100 according to one embodiment of this disclosure is shown. Figure 1 The message processing method shown includes steps S110 to S150. The executing entity of this message processing method is the target consumer, such as an electronic device like a server.

[0061] In step S110, the current unconsumed message volume of the target partition and the current message throughput of the target consumer are obtained.

[0062] In some embodiments of this disclosure, the target partition in step S110 can be on a message broker server or node (commonly referred to as a Broker). The Broker is the core hub of the message queue system, acting as an intermediary layer between producers and consumers, responsible for coordinating message reception, storage, routing, and reliability assurance. A Broker can simultaneously store multiple partitions from multiple topics.

[0063] Step S110 can determine the current unconsumed message volume based on the difference between the current maximum offset of the target partition (i.e., the offset of the latest message currently written to the target partition) and the consumption offset of the consumer (i.e., the latest offset that the consumer has successfully consumed and committed). Step S110 can obtain the current message throughput based on monitoring metrics (such as how many messages the consumer pulls per second, the number of messages consumed per second, the average processing latency, etc.), logs, or code instrumentation.

[0064] Step S110 can be triggered by an instruction to obtain the amount of unconsumed messages and message throughput, or it can be obtained periodically or in real time.

[0065] In step S120, it is determined whether the target consumer needs to be expanded based on the current amount of unconsumed messages and the current message throughput.

[0066] In some embodiments of this disclosure, step S120 can determine whether there is a continuous backlog of unconsumed messages and insufficient consumer processing capacity based on the current unconsumed message volume and current message throughput. If yes, it indicates that the target consumer needs to be expanded, and step S130 is executed. If not, it indicates that the target consumer has the capacity to process the current unconsumed messages of the target partition. As the target consumer continues to consume, the current unconsumed message volume and current message throughput are continuously obtained through step S110 until the message processing of the target partition is completed or the target consumer needs to be expanded.

[0067] In step S130, in response to the target consumer needing to expand its capacity, a slave consumer is created for the target consumer.

[0068] In some embodiments of this disclosure, the consumer created through step S130 may specifically include one or more consumers. The number of consumers included may be related to the current amount of unconsumed messages and the current message throughput.

[0069] In step S140, in response to receiving the set of messages to be processed sent by the target partition, the consumer end that matches each message in the set of messages to be processed is determined from the target consumer end and from the consumers, according to the matching rules and the content of each message in the set of messages to be processed.

[0070] In some embodiments of this disclosure, a pull request can be sent to the target partition, causing the target partition to locate and return the corresponding set of messages to be processed. The pull request may include the topic name, partition number, current consumption progress, and the maximum amount of data to be pulled (e.g., maximum number of bytes). The set of messages to be processed in step S140 may include one or more messages.

[0071] In step S140, the matching rules can be random matching, weighted matching, keyword matching, or round-robin matching, etc.

[0072] In step S150, messages matching the target consumer are processed, and messages matching the slave consumer are sent to the slave consumer; the slave consumer is used to process the received messages.

[0073] In some embodiments of this disclosure, the executing entity for steps S110 to S150 can be a target consumer. That is, the target consumer can perform both message processing and message relay functions. Therefore, when a message matches a target consumer, it can be processed directly through step S150. When a message matches a consumer among the slave consumers, it can be sent to that consumer through step S140, and that consumer will process the message it receives.

[0074] The message processing method disclosed herein employs message processing software to scale up from the consumer end and performs message processing from the consumer end, thereby decoupling the scaling up from the consumer end from the message processing from the consumer end. This allows the message processing method to be applied to various businesses and scenarios, avoiding redundant development and maintenance costs. This method solves the problem in existing technologies where each stage of message processing is tightly coupled with specific business logic, lacking a universal processing mechanism. This results in existing solutions requiring ad-hoc development work each time they handle message backlogs, increasing the complexity of message processing and manpower costs.

[0075] Furthermore, the message processing method provided in this disclosure may further include, after step S150, the following steps: Figure 2 Steps S160 to S180 are shown.

[0076] In step S160, in response to the processing of messages by the target consumer and the slave consumer, the real-time unconsumed message volume of the target partition and the real-time message throughput of the target consumer and the slave consumer are monitored.

[0077] In some embodiments of this disclosure, step S160 can monitor the real-time unconsumed message volume and real-time message throughput based on indicator collection methods, monitoring components, log analysis methods, etc. Step S160 can also monitor the real-time unconsumed message volume and real-time message throughput based on scheduled tasks, event-driven methods, etc.

[0078] In step S170, it is determined whether the target consumer needs to be scaled down based on the real-time unconsumed message volume and the real-time message throughput.

[0079] In some embodiments of this disclosure, step S170 can determine whether there is redundancy between the target consumer and the slave consumer based on the real-time unconsumed message volume and the real-time message throughput. If redundancy exists, it indicates that the target consumer needs to be scaled down, and step S180 is executed. If there is no redundancy, the target consumer that has currently expanded its slave consumer does not need to be processed.

[0080] In step S180, the consumer is deregistered.

[0081] In some embodiments of this disclosure, step S170 can determine whether it is necessary to deregister the slave message end based on the real-time unconsumed message volume and real-time message throughput; if not, step S160 is re-executed; if so, step S180 is executed. Step S180 can employ a gradual deregistration strategy to deregister each consumer in the slave consumer end one by one. Step S180 can also employ a batch deregistration strategy to deregister all consumers in the slave consumer end in batches.

[0082] Steps S160 to S180 cancel the consumer based on the real-time unconsumed message volume and real-time message throughput, which can release redundant resources and improve resource utilization.

[0083] Regarding step S120, in some embodiments of this disclosure, it may include, for example... Figure 3 Steps S121 to S122 are shown.

[0084] In step S121, the consumption delay time of messages in the target partition is obtained based on the current unconsumed message volume and the current message throughput.

[0085] In some embodiments of this disclosure, step S121 may use the product of the current unconsumed message volume and the current message throughput as the consumption delay time. Step S121 may also obtain the message consumption delay time of the target partition based on the current unconsumed message volume and the current message throughput according to a preset calculation method, etc.

[0086] In step S122, it is determined whether the target consumer needs to be expanded based on the consumption delay time.

[0087] In some embodiments of this disclosure, step S122 can determine whether the target consumer needs to be expanded based on the consumption delay time and a preset delay threshold. When the consumption delay time is greater than the preset delay threshold, it can be determined that the target consumer needs to be expanded, and step S130 is executed. When the consumption delay time is not greater than the preset delay threshold, it can be determined that the target consumer does not need to be expanded, and the received messages sent by the target partition can be processed directly.

[0088] Steps S121 to S122 take determining whether expansion is needed as a prerequisite. Only when it is determined that expansion is needed will the consumer end be created, which can prevent excessive expansion, avoid resource waste, and improve the effectiveness of expansion.

[0089] Regarding step S121, in some embodiments of this disclosure, it may include, for example... Figure 4 Steps S1211 to S1212 are shown.

[0090] In step S1211, the unconsumed predicted value is obtained based on the current unconsumed message volume.

[0091] In some embodiments of this disclosure, step S1211 can obtain the unconsumed predicted value based on the amount of unconsumed messages using methods such as moving average, machine learning, and time series forecasting.

[0092] In step S1212, the consumption delay time of messages in the target partition is obtained based on the unconsumed predicted value and the current message throughput.

[0093] In some embodiments of this disclosure, step S1212 may determine the consumption delay time based on the product of the unconsumed forecast value and the message throughput.

[0094] Steps S1211 and S1212 determine the consumption delay time based on the unconsumed predicted value, which can expose potential performance bottlenecks in advance and achieve predictive capacity expansion.

[0095] Regarding step S1211, in some embodiments of this disclosure, it may include, for example... Figure 5 The following steps are shown.

[0096] The accumulation smoothing value, accumulation trend value, and prediction steps are obtained based on the current amount of unconsumed messages.

[0097] Multiply the stacking trend value by the number of prediction steps to obtain the stacking compensation amount.

[0098] Adding the stacking smoothing value to the stacking compensation amount yields the unconsumed prediction value.

[0099] In some embodiments of this disclosure, the heap smoothing value is a smoothed estimate of the amount of unconsumed messages in the target partition at the current time. The heap smoothing value = smoothing coefficient α × amount of unconsumed messages + (1-α) × (heap smoothing value at the previous time + heap trend value at the previous time).

[0100] The stacking trend value represents the direction and rate of change of the amount of unconsumed messages in the time series. The stacking trend value is calculated as: smoothing coefficient β × (current stacking smoothing value - previous stacking smoothing value) + (1-β) × previous stacking trend value.

[0101] Here, α and β are smoothing coefficients, ranging from 0 to 1, and their specific values ​​can be preset, such as α=0.8, β=0.5, etc. The initial stacked smoothing value can be determined based on the first value, the average value, the regression estimate, etc.

[0102] The prediction step size refers to the time step used when making predictions from the current point in time to the future; that is, how many steps into the future the unconsumed message volume needs to be predicted. This prediction step size is usually preset, and since predictions become less reliable the further into the future, it is usually set within a reasonable range (such as 10 minutes).

[0103] Determining the unconsumed forecast value based on the above steps can eliminate noise interference and improve forecast stability and efficiency.

[0104] Regarding step S130, in some embodiments of this disclosure, it may include, for example... Figure 6 Steps S131 to S132 are shown.

[0105] In step S131, elastic scaling is calculated based on the current amount of unconsumed messages and the current message throughput to obtain the scaling quantity.

[0106] In some embodiments of this disclosure, the elastic expansion calculation in step S131 can be implemented based on a preset expansion formula. The elastic expansion calculation in step S131 can also be implemented using a piecewise linear expansion method based on threshold rules, etc.

[0107] In step S132, an additional number of slave consumers are created for the target consumer.

[0108] By creating a consumer-side system based on elastic scaling calculations through steps S131 to S132, the system's processing capacity during peak message backlog periods can be significantly enhanced, achieving dynamic and elastic improvement in consumption capacity.

[0109] Regarding step S131, in some embodiments of this disclosure, it may include, for example... Figure 7 Steps S1311 to S1313 are shown.

[0110] In step S1311, the current number of consumers in the target partition is obtained.

[0111] In some embodiments of this disclosure, the current number of consumers in step S1311 is the target consumer of the target partition and the total number of consumers included in the current sub-consumer.

[0112] In step S1312, the current unconsumed message volume is divided by the current message throughput to obtain the consumer demand.

[0113] In step S1313, the current number of consumers is subtracted from the demand at the consumer end.

[0114] Steps S1311 to S1313 can achieve a rapid response and reasonable resource utilization for elastic scaling.

[0115] Regarding step S140, in some embodiments of this disclosure, it may include, for example... Figure 8 Steps S141 to S145 are shown.

[0116] In step S141, an initial hash ring is constructed.

[0117] In some embodiments of this disclosure, step S141 may use an ordered graph, pre-divided virtual node segments, or other methods to construct an initial hash structure that has not yet mapped any consumer or node.

[0118] In step S142, the target consumer and the consumer are mapped onto the initial hash ring to obtain the first hash ring.

[0119] In step S143, the messages in the message set to be processed are hashed to obtain hash values.

[0120] In step S144, the hash value is mapped onto the first hash ring to obtain the second hash ring.

[0121] In step S145, for any message in the target partition, the message is matched with the hash value of the message at the first consumer in a clockwise direction on the second hash ring.

[0122] Steps S141 to S145 use a hash ring to match messages with consumers, which can achieve load balancing and improve the overall throughput and resource utilization of the system.

[0123] In some embodiments of this disclosure, after sending the message matching the slave consumer to the corresponding slave consumer in step S150, the slave consumer is specifically used to detect whether the target consumer is abnormal; in response to the target consumer being normal, it processes the received message; in response to the target consumer being abnormal, it re-executes the detection of whether the target consumer is abnormal until the target consumer is normal or the cutoff condition is met, and in response to the cutoff condition being met, it cancels the slave consumer. The slave consumer processes messages based on the status of the master consumer, which can realize a highly reliable, low-interference, and automated consumption fault-tolerance mechanism.

[0124] Specifically, when detecting whether the target consumer is abnormal, information about the target consumer is obtained from the consumer itself; and the abnormality of the target consumer is detected based on this information. This method identifies the abnormal state of the target consumer based on its information, providing a clear basis for judging abnormalities in the target consumer.

[0125] The information of the target consumer may include its IP address, metadata and statistics for tracking consumption progress and status, and an ordered set of consumption progress data; and / or the target consumer's information may be stored in an in-memory data storage system. The IP address, metadata and statistics, and the ordered set of consumption progress data enable multi-dimensional and low-latency identification of abnormal states of the target consumer. The in-memory data storage system can specifically be Redis, Memcached, etc. Storing information in an in-memory data storage system such as Redis can significantly improve system access speed, concurrent processing capabilities, and real-time responsiveness, thereby improving message processing speed.

[0126] Furthermore, the message processing method provided in this disclosure may further include, after step S150, the following steps: Figure 9 The step S190 is shown.

[0127] In step S190, in response to receiving the message consumption progress from the target consumer and / or from the consumer, the message consumption progress is sent to the target partition.

[0128] In some embodiments of this disclosure, step S190 may send the message consumption progress to the target partition in a periodic, timed, batch, or asynchronous manner.

[0129] By synchronizing message consumption progress with the target partition in step S190, reliability can be improved.

[0130] Regarding step S150, in some embodiments of this disclosure, it may include, for example... Figure 10 Steps S151 to S152 are shown.

[0131] In step S151, it is detected whether each consumer in the consumer end is available.

[0132] In some embodiments of this disclosure, step S151 can determine which consumer(s) are available based on the information stored in the memory data storage system. A consumer is available when the consumer instance is in normal operating condition, capable of successfully pulling and processing messages, and free from faults or states that hinder consumption, such as downtime, timeout, or anomalies.

[0133] In step S152, the message assigned to the consumer is sent to the available consumer among the consumers.

[0134] Steps S151 to S152 only send messages to available consumers, which ensures that messages are processed in a timely and correct manner, thereby improving the system's throughput and reliability.

[0135] In some embodiments of this disclosure, the target consumer's information can be registered with the in-memory data storage system when the target consumer starts up. Similarly, the slave consumer's information can be registered with the in-memory data storage system when the slave consumer is created. This in-memory data storage system can set up a corresponding Stream message queue for each consumer, supporting streaming data storage, consumption, and other functions. The target consumer and slave consumers can be associated in the in-memory data storage system, indicating that these slave consumers were created for the target consumer.

[0136] The message processing method disclosed herein can dynamically calculate the expansion quantity based on the predicted unconsumed value of messages, thereby achieving expansion. It can also dynamically calculate the reduction quantity based on the predicted unconsumed value, thereby achieving reduction. This message processing method introduces a consumer-side mechanism to assist the target consumer in processing accumulated messages.

[0137] Figure 11 An exemplary flowchart based on the message processing method of this disclosure is shown.

[0138] Figure 11The flowchart shown may include steps S210 to S280.

[0139] In step S210, when the target consumer starts up, it registers its information with the Redis registry.

[0140] In some embodiments of this disclosure, the information of the target consumer registered in step S210 may include the IP address of the target consumer, metadata and statistical information for tracking consumption progress and status, and an ordered set of consumption progress. In step S210, the Redis registry center can be a pre-deployed Redis instance or cluster used to centrally store and maintain the registration information, status data, and status records of all consumers.

[0141] The process of registering the target consumer's information with the Redis registry center through step S210 is to write the target consumer's information under the corresponding key in the Redis registry center.

[0142] Furthermore, to improve system observability and maintainability, target consumers can also report environmental information such as their current running version, registration timestamp, and deployment region (e.g., availability zone ID) during registration. This information can be used for node health checks, traffic scheduling optimization, and canary deployment control in distributed deployment scenarios. When the registration center detects abnormally frequent restarts or registration failures of a certain type of target consumer, it can automatically trigger an alarm mechanism to prompt operations and maintenance personnel to intervene. To ensure the idempotency of the registration process, target consumers can use a unique identifier (such as a UUID or a Snowflake-based ID) as part of the registration key to ensure that information is not confused under network jitter or repeated startup scenarios.

[0143] In step S220, the current unconsumed message volume of the target partition and the current message throughput of the target consumer are obtained.

[0144] In some embodiments of this disclosure, step S220 may initiate a scheduled task to obtain the amount of unconsumed messages and the message throughput.

[0145] The amount of unconsumed messages can be calculated by the difference between the consumption progress and the current offset of the message queue, while message throughput can be statistically analyzed based on the number of messages processed by the target consumer per unit time. The system supports multiple statistical periods (such as 1 minute, 5 minutes, 15 minutes, etc.) to adapt to different business scenarios. Furthermore, to avoid misjudgments during capacity expansion due to sampling errors, a sliding window mechanism can be introduced to smooth the throughput. In addition, runtime metrics such as GC frequency, CPU utilization, and thread pool activity can be collected and used in conjunction with message processing metrics for system health assessment and resilience decision-making.

[0146] In step S230, the message consumption delay time of the target partition is obtained based on the current unconsumed message volume and the current message throughput.

[0147] In some embodiments of this disclosure, step S230 can make a prediction based on the current amount of unconsumed messages to obtain an unconsumed predicted value, and obtain the consumption delay time based on the unconsumed predicted value and message throughput. The unconsumed predicted value can be calculated based on a stacking smoothing value, a stacking trend value, and the number of prediction steps, etc.

[0148] To improve prediction accuracy, exponentially weighted moving average (EWMA), ARIMA models, or machine learning-based time series forecasting models can be used to fit trends to unconsumed messages. Model parameters can be dynamically adjusted during the prediction process to adapt to the data characteristics of different peak and off-peak periods. The calculation of consumption latency can take into account factors such as network jitter and changes in queue backlog speed, constructing a more robust latency estimation function.

[0149] In step S240, it is determined whether the target consumer needs to be expanded based on the consumption delay time.

[0150] In some embodiments of this disclosure, step S240 compares the consumption delay time with a delay threshold to determine whether the consumption delay time is greater than the delay threshold. If the consumption delay time is greater than the delay threshold, it indicates that capacity expansion is needed, and step S250 is executed. If the consumption delay time is not greater than the delay threshold, it indicates that capacity expansion is not needed, and messages can be processed directly through the target consumer.

[0151] The latency threshold can be configured, allowing for different tolerance ranges to be set for different business types. Certain high real-time business operations (such as financial risk control and log collection) may require a lower latency threshold, while operations with higher tolerance (such as asynchronous task processing) can have a more relaxed threshold. A jitter protection mechanism can also be introduced into the judgment logic, which involves maintaining observation for a period of time within the boundary value range of the latency threshold to avoid frequent scaling up and down due to short-term fluctuations, thus affecting system stability.

[0152] In step S250, elastic scaling calculation is performed based on the current amount of unconsumed messages and the current message throughput to obtain the scaling quantity.

[0153] In some embodiments of this disclosure, step S250 can perform elastic scaling calculations based on the amount of unconsumed messages, message throughput, and the current number of consumers in the target partition.

[0154] Elastic scaling calculations can be adjusted using non-linear scaling factors. For example, when the backlog significantly exceeds processing capacity, the system can exponentially increase the number of consumers to prevent processing capacity from lagging behind the message backlog rate. Simultaneously, a maximum scaling threshold and a minimum scaling granularity can be introduced to prevent resource waste caused by scaling up too many instances at once. This process can also be backtested using historical scaling performance data to dynamically adjust the scaling strategy.

[0155] In step S260, an additional number of slave consumers are created for the target consumer.

[0156] In some embodiments of this disclosure, when creating the consumer in step S260, the corresponding Stream message queue for the consumer can be created synchronously and dynamically.

[0157] The creation process of the consumer can be combined with containerization technologies (such as Docker / Kubernetes) to achieve deployment in seconds. During container scheduling, the node with the most sufficient resources can be automatically selected to deploy the new consumer instance based on the current node load. At the same time, to ensure message consistency and partition ordering, the consumer needs to synchronize the consumption position of the target consumer during the initialization phase and perform distributed lock coordination to avoid message duplication or omission.

[0158] In step S270, in response to receiving the set of messages to be processed sent by the target partition, the consumer end that matches each message in the set of messages to be processed is determined from the target consumer end and from the consumers, according to the matching rules and the content of each message in the set of messages to be processed.

[0159] In step S280, messages matching the target consumer are processed, and messages matching the slave consumer are sent to the slave consumer; the slave consumer is used to process the received messages.

[0160] In some embodiments of this disclosure, step S280 may send a message to the corresponding Stream message queue of the slave consumer, so that the slave consumer can pull messages from its corresponding Stream message queue and process them.

[0161] To improve concurrent processing efficiency, the system can employ an asynchronous task scheduling framework (such as Disruptor or Reactor pattern) to distribute messages and support a priority queue mechanism to ensure the real-time performance of high-priority services. Furthermore, the current load status of the consumers can be considered during message distribution to achieve load balancing and prevent excessive pressure on individual consumers.

[0162] This disclosure also provides a message processing apparatus (corresponding to a message processing method). Figure 12 A schematic diagram of the hardware implementation using the processing system is shown.

[0163] like Figure 13 As shown, the hardware architecture of electronic devices / devices can be implemented using a bus architecture. The bus architecture can include any number of interconnect buses and bridges, depending on the specific application of the hardware and overall design constraints. Bus 1100 connects various circuits including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400 such as peripherals, voltage regulators, power management circuits, external antennas, etc.

[0164] Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, this diagram uses only one connection line, but this does not imply that there is only one bus or one type of bus.

[0165] For ease of explanation, certain steps of the above method are described in relation to modules. It should be understood that the corresponding module performing one or more steps of the above method may be one or more hardware modules specifically configured to perform the corresponding step, or implemented by a processor configured to perform the corresponding step, or stored in a computer-readable medium for implementation by a processor, or implemented by some combination thereof.

[0166] like Figure 12 As shown, the message processing device may include a message acquisition module 1010, a judgment module 1020, an expansion module 1030, an object determination module 1040, and a message processing module 1050.

[0167] The message acquisition module 1010 is used to acquire the current unconsumed message volume of the target partition and the current message throughput of the target consumer.

[0168] The judgment module 1020 is used to determine whether the target consumer needs to be expanded based on the current amount of unconsumed messages and the current message throughput.

[0169] The expansion module 1030 is used to create a slave consumer for the target consumer in response to the need for expansion.

[0170] The object determination module 1040 is used to, in response to receiving a set of messages to be processed sent by the target partition, determine, from the target consumer and from the consumers, the consumer that matches each message in the set of messages to be processed, according to the matching rules and the content of each message in the set of messages to be processed.

[0171] The message processing module 1050 is used to process messages that match the target consumer and send messages that match the slave consumer to the slave consumer; the slave consumer is used to process the received messages.

[0172] The specific implementation of each module in the above-mentioned device can be referred to the implementation process of the corresponding steps in the above-mentioned method implementation method of this disclosure, and will not be repeated here.

[0173] This disclosure also provides a readable storage medium storing a computer program that, when executed by a processor, is used to implement the methods described above. A "readable storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples of a readable storage medium include: an electrical connection with one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM), etc.

[0174] This disclosure also provides a computer program product, the methods of which can be implemented wholly or partially through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented wholly or partially as a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed, all or part of the processes or functions of this disclosure are performed.

[0175] Computer programs or instructions can be stored in a readable storage medium or transferred from one readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The readable storage medium can be any available medium capable of access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; an optical medium, such as a digital video optical disc; or a semiconductor medium, such as a solid-state drive. The computer-readable storage medium can be a volatile or non-volatile storage medium, or it can include both volatile and non-volatile types of storage media.

[0176] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0177] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0178] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0179] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0180] In the description of this specification, the references to terms such as "one embodiment / mode," "some embodiments / modes," "example," "specific example," or "some examples," etc., refer to specific features, structures, or characteristics described in connection with that embodiment / mode or example, which are included in at least one embodiment / mode or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment / mode or example. Moreover, the specific features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments / modes or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments / modes or examples described in this specification, as well as the features of different embodiments / modes or examples.

[0181] Those skilled in the art should understand that the above embodiments are merely for illustrating the present disclosure and are not intended to limit the scope of the disclosure. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present disclosure.

Claims

1. A message processing method, characterized in that, The method is executed by the target consumer, including: Obtain the current unconsumed message volume of the target partition under the responsibility of the target consumer and the current message throughput of the target consumer; Determine whether the target consumer needs to be expanded based on the current unconsumed message volume and the current message throughput. In response to the need for expansion of the target consumer, a slave consumer is created for the target consumer. In response to receiving a set of messages to be processed from the target partition, based on matching rules and the content of each message in the set of messages to be processed, a consumer matching each message in the set of messages to be processed is determined from the target consumer and the slave consumers; and The message matching the target consumer is processed, and the message matching the slave consumer is sent to the slave consumer; the slave consumer is used to process the received message.

2. The message processing method as described in claim 1, characterized in that, After sending the message matching the consumer to the consumer, the method further includes: In response to the processing of messages by the target consumer and the slave consumer, monitor the real-time unconsumed message volume of the target partition and the real-time message throughput of the target consumer and the slave consumer; Based on the real-time unconsumed message volume and the real-time message throughput, determine whether the target consumer needs to be scaled down; and In response to the need for scaling down of the target consumer, the slave consumer is deregistered.

3. The message processing method as described in claim 1 or 2, characterized in that, The step of determining whether the target consumer needs to be expanded based on the current unconsumed message volume and the current message throughput includes: The consumption delay time of messages in the target partition is obtained based on the current unconsumed message volume and the current message throughput; and Determine whether the target consumer needs to be expanded based on the consumption delay time.

4. The message processing method as described in claim 3, characterized in that, The step of obtaining the message consumption delay time of the target partition based on the current unconsumed message volume and the current message throughput includes: Obtain the predicted value of unconsumed messages based on the current unconsumed message volume; and The consumption delay time of messages in the target partition is obtained based on the unconsumed prediction value and the current message throughput. The step of obtaining the unconsumed predicted value based on the current unconsumed message volume includes: Based on the current amount of unconsumed messages, obtain the stacking smoothing value, stacking trend value, and prediction steps; Multiplying the stacking trend value by the number of prediction steps yields the stacking compensation amount; and The unconsumed predicted value is obtained by adding the stacking smoothing value to the stacking compensation amount.

5. The message processing method as described in claim 1 or 2, characterized in that, Creating a slave consumer for the target consumer includes: Based on the current unconsumed message volume and the current message throughput, elastic scaling calculations are performed to obtain the scaling quantity; and Create the specified number of expanded slave consumers for the target consumer; The step of performing elastic scaling calculations based on the current unconsumed message volume and the current message throughput includes: Obtain the current number of consumers in the target partition; Divide the current unconsumed message volume by the current message throughput to obtain the consumer demand; and Subtract the current number of consumers from the consumer demand.

6. The message processing method as described in claim 1 or 2, characterized in that, The step of determining the consumer that matches each message in the message set to be processed from the target consumer and the secondary consumers, based on the matching rules and the content of each message in the message set to be processed, includes: Construct the initial hash ring; The target consumer and the consumer are mapped onto the initial hash ring to obtain the first hash ring; The messages in the set of messages to be processed are hashed to obtain hash values; Mapping the hash value onto the first hash ring yields the second hash ring; and For any message in the target partition, match the message with the hash value of the message at the first consumer in the clockwise direction on the second hash ring.

7. The message processing method as described in claim 1 or 2, characterized in that, The slave consumer is specifically used to detect whether the target consumer is abnormal; in response to the target consumer being normal, it processes the received message; in response to the target consumer being abnormal, it re-executes the detection of whether the target consumer is abnormal until the target consumer is normal or the cutoff condition is met, and in response to the cutoff condition being met, it cancels the slave consumer.

8. The message processing method as described in claim 7, characterized in that, Specifically, the step of obtaining information from the consumer end is used to: acquire information about the target consumer end; and detect whether the target consumer end is abnormal based on the information from the target consumer end. The information of the target consumer includes the IP address of the target consumer, metadata and statistical information for tracking consumption progress and status, and an ordered set of consumption progress; and / or, the information of the target consumer is stored in an in-memory data storage system.

9. The message processing method as described in claim 1 or 2, characterized in that, After sending the message matching the slave consumer to the slave consumer, the method further includes: in response to receiving the message consumption progress of the target consumer and / or the slave consumer, sending the message consumption progress to the target partition; And / or, sending a message matching the slave consumer to the slave consumer includes: detecting whether each of the slave consumers is available; and sending a message assigned to the slave consumer to the available slave consumer.

10. An electronic device, characterized in that, include: The memory stores execution instructions; as well as A processor that executes execution instructions stored in the memory, causing the processor to perform the message processing method according to any one of claims 1 to 9.

Images