Message backlog processing system and message backlog processing method

Abstract

A message backlog processing system and a message backlog processing method. The message backlog processing system comprises a storage apparatus and a processor, wherein the storage apparatus is configured to store a data detection module, a decision-making module and a computation module; and the processor is coupled to the storage apparatus, and is configured to execute the data detection module, the decision-making module and the computation module. The data detection module collects a production log, a consumption log, message queue data, and a processing data set in the processor. In response to a message backlog amount being greater than a message backlog threshold value, the decision-making module obtains a processing policy on the basis of a decision weight in a decision table and the processing data set. The computation module computes a suitable performance value on the basis of the processing policy, and updates message processing settings on the basis of the suitable performance value.

Description

Message backlog processing system and method

[0001] This application claims priority to the Chinese patent application No. 202411390743.4, filed on September 30, 2024, and entitled “Message backlog processing system and method”, the entire content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to a message processing technology, in particular to a message backlog processing system and method. BACKGROUND

[0003] When the user growth of a platform and the increase of platform usage, the data-driven task card issuance will be delayed or even lost due to message backlog. The message backlog production speed is greater than the consumption speed, which causes the messages in the message queue to be unable to be processed and consumed in time, resulting in too many messages accumulated in the queue. When there is too much message backlog, the messages cannot be consumed / processed in real time, which causes the tasks to be unable to be processed in time, and causes the consumer to have to process a large number of message tasks, resulting in system performance degradation, delay increase, and high resource consumption. Therefore, in order to solve the situation of message backlog, the processing capacity of the consumer can be improved or the speed of the producer to generate messages can be limited. However, excessive configuration of consumer resources will cause unnecessary resource waste, and excessive limitation of the production speed of the producer will affect the user satisfaction during actual use. SUMMARY

[0004] The present application is directed to a message backlog processing system and method, which can automatically obtain a processing strategy suitable for solving the current message backlog based on the processing data set of the system.

[0005] According to an embodiment of the present application, the message backlog processing system of the present application comprises a storage device and a processor. The storage device is used to store a data detection module, a decision module, and a calculation module. The processor is coupled to the storage device and is used to execute the data detection module, the decision module, and the calculation module. The data detection module collects production logs, consumption logs, message queue data, and processing data sets in the processor. In response to the message backlog being greater than a message backlog threshold, the decision module obtains a processing strategy according to a decision weight in a decision table and the processing data set. The calculation module calculates a suitable performance value according to the processing strategy, and updates the message processing settings according to the suitable performance value. The processing strategy is one of a limited production speed strategy, an increased thread number strategy, and an increased consumer instance number strategy. The suitable performance value is one of a suitable production speed, a suitable thread number, and a suitable consumer instance number.

[0006] According to an embodiment of the present application, the message backlog processing method comprises the following steps: collecting production logs, consumption logs, message queue data and a processing data set in the processor by executing a data detection module by the processor, wherein the processing data set comprises a message backlog amount; when the message backlog amount is greater than a message backlog threshold, obtaining a processing strategy according to a decision weight in a decision table and the processing data set by executing a decision module by the processor; calculating a suitable performance value according to the processing strategy by executing a calculation module by the processor, and updating message processing settings according to the suitable performance value; wherein the processing strategy is one of a limited production speed strategy, an increased thread number strategy and an increased consumer instance number strategy, and the suitable performance value is one of a suitable production speed, a suitable thread number and a suitable consumer instance number. The message processing settings are related to settings of message production, thread number control and message processing of the message processing system.

[0007] Based on the above, the message backlog processing system and the message backlog processing method of the present application can set the message backlog threshold according to the performance data and the scene importance, and adopt a suitable processing strategy according to the message backlog amount and the system performance index, so as to effectively and automatically process the message backlog condition.

[0008] In order to make the above features and advantages of the present application more obvious and easy to understand, the following embodiments are described in detail below, and the accompanying drawings are described as follows. BRIEF DESCRIPTION OF DRAWINGS

[0009] Fig. 1 is a schematic diagram of a message backlog processing system according to an embodiment of the present application;

[0010] Fig. 2 is a flowchart of a message backlog processing method according to an embodiment of the present application;

[0011] Fig. 3 is a flowchart of a message backlog processing method according to another embodiment of the present application;

[0012] Fig. 4 is a schematic diagram of a production end, a message queue, a consumption end and a database according to an embodiment of the present application;

[0013] Fig. 5 is a flowchart of a message backlog processing method according to another embodiment of the present application.

[0014] Reference signs: 100: message backlog processing system; 110: processor; 120: storage device; 401: user interaction; 402: system detection; 403: producer; 404: message queue; 405: consumer; 406: thread; 407: database; S210-S230, S310-S380, S501-S515: steps. DETAILED DESCRIPTION

[0015] Reference will now be made in detail embodiments of the application, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers are used in the drawings and the description to refer to the same or like parts.

[0016] FIG. 1 is a schematic diagram of a message backlog processing system according to an embodiment of the present application. Referring to FIG. 1, the message backlog processing system 100 includes a processor 110 and a storage device 120. The processor 110 is coupled to the storage device 120 and a message processing system. The message backlog processing system 100 can also include a communication interface or a data transmission interface with actual circuit components, so that the processor 110 can communicate or transmit data with an external message processing system. In this embodiment, the message backlog processing system 100 and the message processing system can be implemented in a cloud server or a private server in an enterprise, for example. The message processing system can also provide a plurality of application programming interfaces (APIs) so that the message backlog processing system 100 can call data of the message processing system, such as production logs, consumption logs, message queue data, processing data sets, and performance data. In an embodiment, the message processing system can be integrated into the message backlog processing system 100, and the present application should not be limited thereto.

[0017] In this embodiment, the processor 110 of the message backlog processing system 100 can include a central processing unit (CPU), or other programmable general purpose or special purpose microprocessors (Microprocessor), digital signal processors (DSP), application specific integrated circuits (ASIC), programmable logic devices (PLD), other similar processing circuits, or a combination thereof, for example.

[0018] The storage device 120 can also be a remote cloud storage service or a local data storage service. The storage device 120 can include a memory and / or a database, where the memory can be a non-volatile memory (NVM), for example. The storage device 120 can store related programs, modules, systems, or algorithms for implementing embodiments of the present application, so that the processor 110 can access and execute them to implement the related functions and operations described in embodiments of the present application. The storage device 120 can cache processing data, decision data, policy algorithms, and setting information such as settings described in embodiments of the present application.

[0019] FIG. 2 is a flowchart of a message backlog processing method according to an embodiment of the present disclosure. FIG. 3 is a flowchart of a message backlog processing method according to another embodiment of the present disclosure. Referring to FIGS. 1 and 2, the storage 120 can store a data detection module, a decision module, and a calculation module. The data detection module, the decision module, and the calculation module can be implemented, for example, in a program language such as JSON (JavaScript Object Notation), XML (Extensible Markup Language), or YAML, but the present disclosure is not limited thereto. The data detection module can be used, for example, to detect production logs, consumption logs, message queue data, and processing data sets of a processor of a message processing system, thereby obtaining various data related to message production, message consumption, and processing of messages. The decision module stores a decision table, a plurality of policy conditions, decision weights, parameter settings, weight values, and a policy algorithm. The calculation module dynamically calculates corresponding data, such as the number of threads, the number of instances, and the production speed, according to the current policy calculation and the policy algorithm. Also, the data detection module can access a database in the message processing system, for example, via HTTP (HyperText Transfer Protocol).

[0020] Referring to FIGS. 1 and 2, the message backlog processing system 100 performs steps S210-S230 as follows. In step S210, the processor 110 can execute the data detection module to collect production logs, consumption logs, message queue data, and processing data sets. The processing data sets include a message backlog amount. In this embodiment, the processor 110 executes the data detection module to cause the data detection module to perform a detection operation. In the detection operation, the data detection module uses a corresponding method to obtain the message backlog amount according to the type of the message queue, for example, an application programming interface or another interface to obtain the message backlog amount in real time, but the present disclosure is not limited thereto. In step S220, in response to the message backlog amount being greater than a message backlog threshold, the processor 110 executes the decision module to cause the decision module to obtain a processing policy according to decision weights in a decision table and the processing data sets. The processing data sets include the message backlog amount, the message backlog threshold, system performance indicators, and performance indicator thresholds. The decision table includes settings of different processing policies corresponding to the current message backlog amount and the system performance indicators.

[0021] At step S230, the computing module calculates the suitable performance value according to the processing strategy, and updates the message processing settings according to the suitable performance value. The processing strategy is one of the limiting production speed strategy, the increasing thread number strategy, and the increasing consumer instance number strategy. The suitable performance value is one of the suitable production speed, the suitable thread number, and the suitable consumer instance number. In this embodiment, the computing module calculates the suitable performance value according to the processing strategy provided by the decision module at step S220 and the calculation settings in the decision table. Then, the processor 110 updates the message processing settings of the message processing system for processing the message to the suitable performance value.

[0022] Referring to FIG. 1 and FIG. 3, the message backlog processing system 100 performs the following steps S210-S230. At step S310, the data detection module collects the production log, the consumption log, the message queue data, and the processing data set, and thus obtains the data related to processing the message and the processing settings. At step S320, the computing module calculates the current system performance index, the production speed, and the processing speed according to the data collected by the data detection module (e.g., the message backlog amount, the current thread number, the total amount of message generation, the total amount of message consumption, the processor occupancy rate, the memory occupancy rate, the disk occupancy rate, the bandwidth occupancy rate, etc.).

[0023] At step S330, the decision module sets different message backlog thresholds, danger thresholds, and system performance alarm indexes according to the plurality of scenarios. Then, the computing module sets different message backlog thresholds, danger thresholds, and performance index thresholds according to the plurality of scenarios. In this embodiment, the storage device 120 stores the weight values of the plurality of scenarios. In this way, the computing module calculates the performance index values of the scenarios according to the weight values, the processor occupancy rate, the memory occupancy rate, the disk occupancy rate, the bandwidth occupancy rate, and the corresponding weight values in the processing data set.

[0024] For example, the scenarios can be divided into important scenarios, less important scenarios, and general scenarios. The important scenarios can be the driving of task card issuance and core processes, the less important scenarios can be message notification and reminders, and the general scenarios can be the recording of operating system logs. In this way, the system designer or user can set the weight values of the scenarios according to the importance of the scenarios. Since the important scenarios have high real-time requirements, their message backlog thresholds, danger thresholds, and system performance alarm indexes are set to be low. The less important scenarios have lower real-time requirements than the important scenarios and can accept a certain delay, so the message backlog alarm thresholds, danger thresholds, and system performance alarm indexes of the less important scenarios are set to be higher than those of the important scenarios. When the data (i.e., messages) of the general scenarios are delayed or even lost, it will not affect the overall process, so the message backlog alarm thresholds, danger thresholds, and system performance alarm indexes of the general scenarios are usually set to a high level.

[0025] At step S340, when the message backlog is greater than the message backlog threshold of the current scenario, the processor 110 selects the corresponding strategy according to the settings of the decision table, and then executes the corresponding steps S350-S370. At step S350, the corresponding strategy is the production speed limiting strategy, so the processor 110 selects the production speed limiting strategy to reduce the message backlog. At step S360, the corresponding strategy is the thread number increasing strategy, so the processor 110 selects the thread number increasing strategy to solve the message backlog condition. At step S370, the corresponding strategy is the consumer instance number increasing strategy, so the processor 110 selects the consumer instance number increasing strategy to solve the message backlog condition.

[0026] At step S380, after the processor 110 adopts the corresponding processing strategy, the computing module dynamically calculates the suitable thread number, consumer instance number, or production speed of each scenario according to the decision table. Specifically, when the processing strategy is the thread number increasing strategy, the computing module can calculate the suitable thread number according to the processor core number in the processing data set, the current thread number, the current instance number, the message backlog threshold, the message backlog, the performance index value, and the performance index threshold.

[0027] In this embodiment, the computing module calculates the suitable thread number according to the following formula (1) and formula (2):

[0028] a max is the maximum performance index threshold of the system, a curr is the current system performance index, t curr is the current thread number, u is the CPU core number (i.e., the processor core number), m is the number of consumer ends (assuming that each consumer end instance is configured identically), c alert is the message backlog alarm threshold, c curr is the current message backlog, t pre is the suitable thread number, k is a compensation parameter between 0 and 1, and ceil is the ceiling function to ensure that the suitable thread number is an integer. In this way, when a curr a max , and there are messages to be processed in the message queue (c curr c alert ), the processor 110 can increase the thread number to improve the consumption capacity. Formula (2) is that when the suitable thread number (i.e., the recommended total thread number t pre ) is higher than the maximum thread number t max , the maximum thread number is taken as the suitable thread number t new , otherwise, the t pre of formula (1) is taken as the suitable thread number.

[0029] In an embodiment, the plurality of performance indicators of the system include CPU occupancy, memory occupancy, disk occupancy, and bandwidth occupancy, which are main factors affecting the performance of the system. In an embodiment, the computing module can obtain the performance indicators of the message processing system in real time through a shell script, and then calculate the performance indicator value of the message processing system through the following formula (3):

[0030] b1 is the average value of the CPU occupancy per unit time, b2 is the average value of the memory occupancy per unit time, b3 is the average value of the disk occupancy per unit time, and b4 is the occupancy of the bandwidth per unit time, and the corresponding weight values are w1, w2, w3, and w4, respectively. Among them, the weight values are between 0 and 1, that is, 0 ≦ w i ≦ 1, w1 + w2 + w3 + w4 = 1, and a is the performance indicator value after weighting calculation, 0 ≦ a ≦ 1. The larger the value of a is, the worse the performance of the system is.

[0031] In the present embodiment, the computing module calculates the suitable consumer instance number according to the current instance number in the processing data group, the average processing speed, the total production speed, and the speed multiple of the newly added instance and the current instance. First, the suitable instance number must satisfy that the total consumption speed is greater than or equal to the total production speed, so the following formula (4) can be obtained: new (m *e)+(m

[0032] *k*e)>=d……(4)

[0033] new m max is the number of instances to be supplemented, i is a compensation parameter, which ensures that i × m × e is the total speed of the current consumption. For example, in the current environment, the maximum total number of instances that can be accommodated is m max , then the maximum number of instances to be supplemented is m new_1 -m, then the number of instances that can be supplemented after considering the maximum total number of instances that can be accommodated (i.e. the suitable instance number) is m new_1 , and m new =m max -m).

[0034] In one embodiment, the calculation module calculates the suitable production speed based on the relevant scenario list, scenario weight value, scenario message backlog, maximum weight value, scenario speed reduction ratio, and minimum speed value in the message queue. The scenario message backlog is the total message backlog of all scenarios. The scenario speed reduction ratio is related to the ratio between the current backlog and the scenario message backlog, as well as the difference between the scenario weight value and the threshold weight. Specifically, this system sets different scenario weight values ​​for scenarios of different importance. For example, firstly, it obtains a list of all scenarios involving message queues in the current system, divides the scenario list according to importance levels (n levels in total), and assigns weight values ​​to the n levels, setting the scenario weight values ​​as m_w1, m_w2, m_w3, m_w4, ..., m_w n The corresponding message backlog amounts are m_c1, m_c2, m_c3, m_c4, ..., m_c n Where m_w1>m_w2>m_w3>...>m_w n And m_w1+m_w2+m_w3+...+m_w n =1.

[0035] Next, based on the current backlog m_c i and scene weights m_w i The restriction factor is calculated using the following formula (6): f_i=(m_c i / Σm_c)*(max_weight-m_w i ) / (max_weight-w_threshold)……(6)

[0036] w_threshold is the threshold, m_w i Here, Σm_c represents the scene weight value, which is the sum of the message backlog across all scenes, and max_weight is the threshold weight. i Let f_i be the backlog for the current scenario, a value between 0 and 1, representing the percentage by which the producer in scenario i should reduce its message generation rate, and this applies to each scenario i (where i ranges from 1 to n). The threshold weight is related to the largest weight value among all production scenarios in the current system. When calculating the threshold weight max_weight, a list of all scenarios involving message queues in the current system must first be obtained, and this list must be divided according to importance. For example, if the weight values ​​in the scenario list involve n levels, then the weight values ​​for these n levels are divided into m_w_1, m_w_2, m_w_3, m_w_4, ..., m_w_n, and the largest weight value among them is the threshold weight max_weight.

[0037] Then, the appropriate production rate of the current scenario is obtained according to the following formula (7): rate_new_i = max(min_rate, y_i * rate_original_i * (1 - f_i)) …… (7)

[0038] Assume that rate_original_i is the original message production rate, rate_new_i is the new appropriate production rate. min_rate is the minimum speed threshold, y_i is the current scenario compensation parameter, which can be flexibly adjusted according to the system environment, and 0 < y_i <= 1. In this way, the calculation module can calculate the appropriate production rate suitable for the current scenario, thereby reducing the message backlog, while maintaining the message flow of each business scenario through the minimum speed threshold.

[0039] Fig. 4 is a schematic diagram of a production end, a message queue, a consumer end and a database of an embodiment of the present application. Fig. 5 is a flow chart of a message backlog processing method of another embodiment of the present application. Referring to Fig. 4 first, when a user performs user interaction 401 through a system interaction interface, the user inputs instructions to the message producer 403 end in the message processing system. The producer 403 then transmits the message to the message consumer 405 end (i.e. the message processing end) through the message queue 404, and then processes the message through the consumer 405 end and its thread configuration. In this way, the message processing system processes the message into corresponding operation logic or program code, and stores it in the database 407. The message backlog processing system 100 performs system detection 402 through the data detection module, thereby obtaining the production log, the consumption log, the message queue type (i.e. the message queue data) and the processing data set. The processing data set is the data of the message processing system when processing the message. Referring to Figs. 1, 3, 4 and 5, the message backlog processing system 100 can perform the following steps S501-S515.

[0040] In step S501, when the data detection module detects that there is a message backlog condition in the process of detecting the data of the message processing system, it then performs steps S502-S504. In step S503, the decision module determines whether the message backlog amount is greater than the message backlog threshold. When the message backlog amount is greater than the message backlog threshold, the decision module then determines whether the message backlog amount is greater than the backlog danger threshold. In response to the message backlog amount being less than the backlog danger threshold, the decision module then determines whether the system performance index is lower than the performance index threshold. In response to the system performance index being lower than the performance index threshold, the decision module takes the increase thread number strategy as the current processing strategy according to the decision table, i.e. step S509 is performed.

[0041] At step S509, the decision module adopts the increasing thread number strategy, and then determines the system index threshold, the current system performance index, the current thread number, the processor core number, the instance number, the message backlog threshold and the message backlog amount through the collection of the data detection module and the calculation of the calculation module. At step S510, the calculation module calculates the suggested thread number (i.e. the suitable thread number) according to the above formula (1) and formula (2). Moreover, the calculation module inputs the suitable thread number to the message processing system, so that the message processing system adjusts the thread number to the suitable thread number.

[0042] At step S511, the data detection module collects the production log and the consumption log in the message processing system in real time, and then judges whether the current message processing state of the message processing system satisfies the production speed being less than the consumption speed. In response to the production speed being greater than the processing speed, and in response to the processing strategy being the increasing thread number strategy, the decision module updates the processing strategy to the increasing consumer instance number strategy. That is to say, when the production speed is still greater than the consumption speed in step S511, step S512 is executed next.

[0043] At step S504, the decision module judges whether the message backlog amount is greater than the message backlog threshold, and whether the performance index is greater than the performance index threshold. In response to the system performance index being higher than the performance index threshold, the decision module takes the increasing consumer instance number strategy as the processing strategy according to the decision table, i.e. step S512 is executed.

[0044] At step S512, the decision module adopts the increasing instance number strategy, and then determines the consumer instance number, the average consumption speed of a single consumer and the total production speed of the producer through the collection of the data detection module and the calculation of the calculation module. Then, step S513 is executed. At step S513, the calculation module calculates the suitable consumer instance number according to the processing data group, and then updates the consumer instance number in the message processing setting of the message processing system. At step S514, the data detection module collects the production log and the consumption log in the message processing system in real time, and then judges whether the current message processing state of the message processing system satisfies the production speed being less than the consumption speed. In response to the production speed being greater than the processing speed, and in response to the processing strategy being the increasing consumer instance number strategy, the decision module updates the processing strategy to the limiting production speed strategy. That is to say, when the production speed of the message processing system is still greater than the consumption speed after the processing setting of the message processing system is updated to the suitable consumer instance number in step S514, step S505 is executed next.

[0045] At step S502, in response to the message backlog being greater than the backlog danger threshold, the decision module determines whether the message backlog is greater than K times the backlog danger threshold, where K is a positive integer. Then, in response to the message backlog being greater than K times the backlog danger threshold, the decision module adopts the limit production rate strategy as the processing strategy, i.e., proceeds to step S505. At step S505, the decision module adopts the limit production rate strategy, and then determines the backlog of each scenario through the collection of the data detection module and the calculation of the calculation module, and calculates the limit factor. At step S506, the calculation module calculates the production rate of each scenario (i.e., the appropriate production rate of each scenario) based on the above formula (4) and formula (5) and according to the processing data set. Then, the message backlog processing system 100 inputs the appropriate production rate of each scenario to the message processing system, and then updates the production rate in the message processing setting of the message processing system.

[0046] At step S507, the data detection module collects the production log and the consumption log in the message processing system in real time, and then determines whether the current message processing state of the message processing system satisfies the production rate being less than the consumption rate. In other words, the data detection module detects the production log and the consumption log after the message processing setting is updated, to determine whether the production rate is less than the processing rate.

[0047] The processing strategy further includes a bottom-up strategy, wherein in response to the production rate being greater than the processing rate, and in response to the processing strategy being the limit production rate strategy, the decision module updates the processing strategy to the bottom-up strategy at step S508. At step S508, the decision module adopts the bottom-up strategy, and the calculation module proportionally reduces the production rate until the production rate is less than the processing rate according to the processing data set. And at step S508, when the production rate is less than the processing rate, then step S515 is performed. In the determination process of step S506, step S510, and step S513, when the production rate of the message processing system is less than the consumption rate, then step S515 is performed. At step S515, the flow ends.

[0048] In another embodiment, in response to the message backlog being greater than the danger value, while the message backlog is not greater than K times the backlog danger threshold, the decision module sequentially adopts the increase thread number strategy and the increase consumer instance number strategy as the processing strategy until the message backlog is less than the message backlog threshold. In other words, when the message backlog in the message processing system is greater than the danger value, but the message backlog is not greater than K times the backlog danger threshold, the decision module first adopts the increase thread number strategy. When the message backlog is still not lower than the danger value after the increase thread number strategy, the decision module then adopts the increase instance number strategy.

[0049] In summary, the message backlog processing system and method of the present application can effectively reduce the message backlog condition by setting multiple processing strategies according to different scenarios, different message backlog amounts and system performance indicators, while important scenarios / important instructions can be processed smoothly to effectively avoid the loss of important messages due to message backlog or production speed limitation. In other words, the message backlog processing system of the present application automatically increases or decreases the consumption resources (i.e. the number of threads, the number of consumption instances or the production speed) based on the real-time load condition of the message processing system / platform, the message backlog amount, the production speed and the consumption speed to ensure that the corresponding processing capacity can be provided during the task card issuance peak period (i.e. the peak period of message generation), while avoiding the idling of resources during the low peak period of message generation, thereby solving the problem of message backlog and achieving efficient use of resources.

[0050] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A message backlog processing system, characterized by, The system comprises: a storage device for storing a data detection module, a decision module and a calculation module; and a processor coupled to the storage device and used to execute the data detection module, the decision module and the calculation module, wherein the data detection module collects production logs, consumption logs, message queue data and a processing data group, wherein the processing data group comprises a message backlog, in response to the message backlog being greater than a message backlog threshold, the decision module obtains a processing strategy according to a decision weight in a decision table and the processing data group, wherein the calculation module calculates a suitable performance value according to the processing strategy and updates a message processing setting according to the suitable performance value, wherein the processing strategy is one of a limit production speed strategy, an increase thread number strategy and an increase consumer instance number strategy, and the suitable performance value is one of a suitable production speed, a suitable thread number and a suitable consumer instance number.

2. The message backlog processing system of claim 1, wherein, The storage device stores weight values of multiple scenarios, wherein the calculation module calculates a performance index value according to the weight values, processor occupancy, memory occupancy, disk occupancy, bandwidth occupancy and corresponding weight values in the processing data group, wherein the calculation module sets different message backlog thresholds, danger thresholds and performance index thresholds for the multiple scenarios respectively.

3. The message backlog processing system of claim 1, wherein, The calculation module calculates the suitable thread number according to the number of processor cores, the current thread number, the current instance number, the message backlog threshold, the message backlog, the performance index value and the performance index threshold in the processing data group.

4. The message backlog processing system of claim 1, wherein, When the message backlog is greater than the message backlog threshold, the decision module determines whether the message backlog is greater than a backlog danger threshold, in response to the message backlog being less than the backlog danger threshold, the decision module determines whether the system performance index is lower than a performance index threshold, in response to the system performance index being lower than the performance index threshold, the decision module sets the increase thread number strategy as the processing strategy according to the decision table, in response to the system performance index being higher than the performance index threshold, the decision module sets the increase consumer instance number strategy as the processing strategy according to the decision table, in response to the message backlog being greater than the backlog danger threshold, the decision module determines whether the message backlog is greater than K times of the backlog danger threshold, wherein K is a positive integer, in response to the message backlog not being greater than K times of the backlog danger threshold, the decision module sequentially sets the increase thread number strategy and the increase consumer instance number strategy as the processing strategy until the message backlog is less than the message backlog threshold, in response to the message backlog being greater than K times of the backlog danger threshold, the decision module sets the limit production speed strategy as the processing strategy.

5. The message backlog processing system of claim 1, wherein, The data detection module detects the production logs and the consumption logs after the message processing setting is updated to determine whether the production speed is less than the processing speed.

6. The message backlog processing system of claim 5, wherein, The processing strategy further includes a bottom-up strategy, wherein, in response to the production speed being greater than the processing speed and in response to the processing strategy being the limited production speed strategy, the decision module updates the processing strategy to the bottom-up strategy, wherein the calculation module proportionally reduces the production speed according to the processing data set until the production speed is less than the processing speed.

7. The message backlog processing system of claim 5, wherein, In response to the production speed being greater than the processing speed and in response to the processing strategy being the increased thread number strategy, the decision module updates the processing strategy to the increased consumer instance number strategy, wherein the calculation module calculates the suitable consumer instance number according to the processing data set, and updates the consumer instance number in the message processing setting.

8. The message backlog processing system of claim 7, wherein, The calculation module calculates the suitable consumer instance number according to the current instance number, the average processing speed, the total production speed, and the speed multiple of the new instance and the current instance in the processing data set.

9. The message backlog processing system of claim 5, wherein, In response to the production speed being greater than the processing speed and in response to the processing strategy being the increased consumer instance number strategy, the decision module updates the processing strategy to the limited production speed strategy, wherein the calculation module calculates the suitable production speed according to the processing data set, and updates the production speed in the message processing setting.

10. The message backlog processing system of claim 9, wherein, The calculation module calculates the suitable production speed according to the relevant scene list, the scene weight value, the scene message backlog, the maximum weight value, the scene speed reduction ratio, and the minimum speed value in the message queue, wherein the scene message backlog is the total message backlog of all scenes, wherein the scene speed reduction ratio is related to the ratio between the current backlog and the scene message backlog and the difference between the scene weight value and the threshold weight.

11. A message backlog processing method, characterized by, The method comprises: The processor executes a data detection module to collect production logs, consumption logs, message queue data, and processing data sets, wherein the processing data sets include message backlogs; When the message backlog is greater than a message backlog threshold, the processor executes a decision module to obtain a processing strategy according to decision weights in a decision table and the processing data sets; The processor executes a calculation module to calculate a suitable performance value according to the processing strategy, and updates message processing settings according to the suitable performance value; wherein the processing strategy is one of a limited production speed strategy, an increased thread number strategy, and an increased consumer instance number strategy, wherein the suitable performance value is one of a suitable production speed, a suitable thread number, and a suitable consumer instance number.

12. The method of claim 11, wherein, The storage device stores weight values of multiple scenes, and the method further comprises the following steps: The calculation module calculates performance indicator values according to the weight values, processor occupancy, memory occupancy, disk occupancy, bandwidth occupancy, and corresponding weight values in the processing data sets; and The calculation module sets different message backlog thresholds, danger thresholds, and performance indicator thresholds for the multiple scenes, respectively.

13. The method of claim 11, wherein, The method further comprises the following steps: The suitable thread number is calculated by the calculation module according to the processor core number, the current thread number, the current instance number, the message backlog threshold, the message backlog amount, the performance index value, and the performance index threshold in the processing data set.

14. The method of claim 11, wherein, The method comprises the following steps: When the message backlog amount is greater than the message backlog threshold, the decision module determines whether the message backlog amount is greater than a backlog danger threshold; In response to the message backlog amount being less than the backlog danger threshold, the decision module determines whether the system performance index is lower than a performance index threshold; In response to the system performance index being lower than the performance index threshold, the decision module determines the increase thread number strategy as the processing strategy according to the decision table; In response to the system performance index being higher than the performance index threshold, the decision module determines the increase consumer instance number strategy as the processing strategy according to the decision table; In response to the message backlog amount being less than the K times of the backlog danger threshold, the decision module sequentially determines the increase thread number strategy and the increase consumer instance number strategy as the processing strategy until the message backlog amount is less than the message backlog threshold; and In response to the message backlog amount being greater than the K times of the backlog danger threshold, the decision module determines the limit production speed strategy as the processing strategy. The step of updating the message processing settings further comprises:

15. The method of claim 11, wherein, The data detection module detects the production log and the consumption log after the message processing settings are updated to determine whether the production speed is less than the processing speed. The processing strategy further comprises a bottom strategy, 16. The method of claim 15, wherein, In response to the production speed being greater than the processing speed and in response to the processing strategy being the limit production speed strategy, the steps comprise: The decision module updates the processing strategy to the bottom strategy; and The calculation module proportionally reduces the production speed according to the processing data set until the production speed is less than the processing speed. In response to the production speed being greater than the processing speed and in response to the processing strategy being the increase thread number strategy, the steps comprise:

17. The method of claim 15, wherein, The decision module updates the processing strategy to the increase consumer instance number strategy; and The calculation module calculates the suitable consumer instance number according to the processing data set and updates the consumer instance number in the message processing settings. The method further comprises the following steps:

18. The method of claim 17, wherein, The calculation module calculates the suitable consumer instance number according to the current instance number, the average processing speed, the total production speed, and the speed multiple of the new instance and the current instance in the processing data set. In response to the production speed being greater than the processing speed and in response to the processing strategy being the increase consumer instance number strategy, the steps comprise:

19. The method of claim 15, wherein, The decision module updates the processing strategy to the limit production speed strategy; and The calculation module proportionally reduces the production speed according to the processing data set until the production speed is less than the processing speed. calculating, by the computing module, the appropriate production speed from the processing data set, and updating the production speed in the message processing settings.

20. The method of claim 19, wherein, The step of calculating the appropriate production speed from the processing data set comprises: calculating, by the computing module, the appropriate production speed from the processing data set, and updating the production speed in the message processing settings. wherein the scene message backlog is the total message backlog of all scenes, wherein the scene speed reduction ratio is related to the ratio between the current backlog and the scene message backlog and the difference between the scene weight value and the threshold weight value.

Images