Message backlog processing system and message backlog processing method

Abstract

The invention provides a message backlog processing system and a message backlog processing method. The message backlog processing system includes a storage device and a processor. The storage device is used for storing the processor, is coupled with the storage device, and is used for executing the data detection module, the decision module and the calculation module. The data detection module collects a production log, a consumption log, message queue data and a processing data set in the processor. In response to the fact that the message backlog amount is larger than the message backlog threshold value, the decision module obtains a processing strategy according to the decision weight in the decision table and the processing data set. The computing module computes a suitable performance value according to the processing policy and updates the message processing settings according to the suitable performance value.

Description

Technical Field

[0001] The invention relates to a message processing technology, in particular to a message backlog processing system and a message backlog processing method. Background Art

[0002] As the number of users on the platform grows and the usage of the platform increases, the issuance of data-driven task cards will be delayed or even lost due to message backlogs. The message backlog production speed is greater than the consumption speed, resulting in the inability to process and consume messages in the message queue in a timely manner, causing too many messages to accumulate in the queue. When there is too much message backlog, messages cannot be consumed / processed in real time, resulting in the inability to process tasks in a timely manner, and the consumer side must process a large number of message tasks, resulting in system performance degradation, increased latency, and excessive resource consumption. Therefore, in order to solve the message backlog situation, the processing capacity of the consumer side can be improved or the speed at which producers generate messages can be limited. However, over-allocation of consumer-side resources will lead to unnecessary waste of resources, and excessive restrictions on the production speed of producers will affect user satisfaction during actual use. Summary of the invention

[0003] The present invention is directed to a message backlog processing system and a message backlog processing method, which can automatically obtain a processing strategy suitable for solving the current message backlog based on a processing data group of the system.

[0004] According to an embodiment of the present invention, the message backlog processing system of the present invention includes a storage device and a processor. The storage device is used to store a data detection module, a decision module and a calculation module. The processor is coupled to the storage device and is used to execute the data detection module, the decision module and the calculation module. The data detection module collects production logs, consumption logs, message queue data and a processing data group in the processor. In response to the message backlog being greater than the message backlog threshold, the decision module obtains a processing strategy based on the decision weights in the decision table and the processing data group. The calculation module calculates a suitable performance value according to the processing strategy, and updates the message processing settings according to the suitable performance value. The processing strategy is one of a production speed restriction strategy, a thread number increase strategy and a consumer instance number increase strategy. The suitable performance value is one of a suitable production speed, a suitable number of threads and a suitable number of consumer instances.

[0005] According to an embodiment of the present invention, the message backlog processing method of the present invention includes the following steps: the processor executes a data detection module to collect production logs, consumption logs, message queue data and a processing data group in the processor, wherein the processing data group includes the message backlog; when the message backlog is greater than the message backlog threshold, the processor executes a decision module to obtain a processing strategy according to the decision weights in the decision table and the processing data group; the processor executes a calculation module to calculate a suitable performance value according to the processing strategy, and updates the message processing settings according to the suitable performance value; wherein the processing strategy is one of a production speed restriction strategy, a thread number increase strategy and a consumer instance number increase strategy, wherein the suitable performance value is one of a suitable production speed, a suitable thread number and a suitable consumer instance number. The message processing settings are about the message processing system producing messages, controlling the number of threads and processing messages.

[0006] Based on the above, the message backlog processing system and message backlog processing method of the present invention can set the message backlog threshold according to performance data and scene importance, and adopt appropriate processing strategies according to the message backlog amount and system performance indicators to effectively and automatically handle the message backlog situation.

[0007] In order to make the above features and advantages of the present invention more clearly understood, embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a schematic diagram of a message backlog processing system according to an embodiment of the present invention;

[0009] Figure 2 is a flow chart of a message backlog processing method according to an embodiment of the present invention;

[0010] Figure 3 is a flow chart of a message backlog processing method according to another embodiment of the present invention;

[0011] Figure 4 It is a schematic diagram of a production end, a message queue, a consumption end and a database of an embodiment of the present invention;

[0012] Figure 5 It is a flowchart of a message backlog processing method according to another embodiment of the present invention.

[0013] Description of Reference Numerals

[0014] 100: message backlog processing system;

[0015] 110: processor;

[0016] 120: storage device;

[0017] 401: User interaction;

[0018] 402: System detection;

[0019] 403: Producer;

[0020] 404: message queue;

[0021] 405: Consumer;

[0022] 406: thread;

[0023] 407: Database;

[0024] S210~S230, S310~S380, S501~S515: steps. DETAILED DESCRIPTION

[0025] Reference will now be made in detail to exemplary embodiments of the present invention, examples of which are illustrated in the accompanying drawings. Whenever possible, the same reference numerals are used in the drawings and the description to refer to the same or like parts.

[0026] Figure 1 FIG. 1 is a schematic diagram of a message backlog processing system according to an embodiment of the present invention. Figure 1 , the message backlog processing system 100 includes a processor 110 and a storage device 120. The processor 110 is coupled to the storage device 120 and the message processing system. The message backlog processing system 100 may also include a communication interface or a data transmission interface having actual circuit components, so that the processor 110 can communicate with an external message processing system or perform data transmission. In the present embodiment, the message backlog processing system 100 and the message processing system may be implemented, for example, on a cloud server or a private server within an enterprise, respectively. The message processing system may also provide a plurality of application programming interfaces (APIs) so that the message backlog processing system 100 can call the data of the message processing system, such as production logs, consumption logs, message queue data, processing data groups, and performance data. In one embodiment, the message processing system may be integrated into the message backlog processing system 100, but the present invention should not be limited to this.

[0027] In this embodiment, the processor 110 of the message backlog processing system 100 may include, for example, a central processing unit (CPU), or other programmable general-purpose or special-purpose microprocessors (Microprocessor), digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuits, ASIC), programmable logic devices (Programmable Logic Device, PLD), other similar processing circuits or a combination of these devices.

[0028] The storage device 120 may also implement a remote cloud storage service or a local data storage service. The storage device 120 may include a memory and / or a database, wherein the memory may be, for example, a non-volatile memory (NVM). The storage device 120 may store relevant programs, modules, systems or algorithms for implementing the various embodiments of the present invention, so that the processor 110 can access and execute them to implement the relevant functions and operations described in the various embodiments of the present invention. The storage device 120 may cache the processing data, decision data, strategy algorithms, settings and other setting information described in the various embodiments of the present invention.

[0029] Figure 2 The present invention is a flowchart of a message backlog processing method according to an embodiment of the present invention. Figure 3 FIG. 1 is a flowchart of a message backlog processing method according to another embodiment of the present invention. Figure 1 as well as Figure 2, the storage device 120 can store data detection modules, decision modules and calculation modules. The data detection module, decision module and calculation module can be implemented in programming languages ​​such as JSON (JavaScript Object Notation), Extensible Markup Language (XML) or YAML, but the present invention is not limited to this. The data detection module can be used to detect the production log, consumption log, message queue data of the database in the message processing system, and the processing data group of the processor of the message processing system, so as to obtain various data related to message production, message consumption and message processing. The decision module stores decision tables, multiple policy conditions, decision weights, parameter settings, weight values ​​and policy algorithms. The calculation module dynamically calculates the corresponding data according to the current policy calculation and policy algorithm, which can be processing settings such as the number of threads, the number of instances and the production speed. In addition, the data detection module can access the database in the message processing system, for example, through the HyperText Transfer Protocol (HTTP).

[0030] Refer to Figure 1 to Figure 2 , the message backlog processing system 100 executes the following steps S210 to S230. In step S210, the processor 110 may execute the data detection module to collect production logs, consumption logs, message queue data and process data groups. The process data group includes the message backlog. In this embodiment, the processor 110 executes the data detection module so that the data detection module performs a detection operation. In performing the detection operation, the data detection module uses a corresponding method to obtain the message backlog according to the type of the message queue. The acquisition method is, for example, an application program interface or other interfaces to obtain the message backlog in real time, and the present case should not be limited to this. In step S220, in response to the message backlog being greater than the message backlog threshold, the processor 110 executes the decision module so that the decision module obtains a processing strategy according to the decision weight in the decision table and the process data group. The process data group includes the message backlog, the message backlog threshold, the system performance index and the performance index threshold. The decision table includes settings for adopting different processing strategies according to the current message backlog and the system performance index.

[0031] In step S230, the calculation module calculates the suitable performance value according to the processing strategy, and updates the message processing settings according to the suitable performance value. The processing strategy is one of the production speed restriction strategy, the thread number increase strategy and the consumer instance number increase strategy. The suitable performance value is one of the production speed, the thread number and the consumer instance number. In this embodiment, the calculation module calculates the suitable performance value according to the processing strategy provided by the decision module in step S220 and the calculation settings in the decision table. Then, the processor 110 updates the message processing settings of the message processing system for processing messages to the suitable performance value.

[0032] Next reference Figure 1 as well as Figure 3 , the message backlog processing system 100 executes the following steps S210 to S230. In step S310, the data detection module collects production logs, consumption logs, message queue data and processing data groups, and then obtains data and processing settings related to message processing. In step S320, the calculation module calculates the current system performance indicators, production speed and processing speed based on the data collected by the data detection module (for example, message backlog, current number of threads, total message generation, total message consumption, processor occupancy, memory occupancy, disk occupancy, bandwidth occupancy, etc.).

[0033] In step S330, the decision module sets different message backlog thresholds, danger thresholds, and system performance alarm indicators according to multiple scenarios. Then, the calculation module sets different message backlog thresholds, danger thresholds, and performance indicator thresholds according to the multiple scenarios. In this embodiment, the storage device 120 stores weight values ​​of multiple scenarios. In this way, the calculation module calculates the performance indicator value of each scenario based on the weight value, the processor occupancy rate, memory occupancy rate, disk occupancy rate, bandwidth occupancy rate in the processing data group, and the corresponding weight value.

[0034] For example, scenarios can be divided into important scenarios, less important scenarios, and general scenarios. Important scenarios can be task card issuance and core process driving, less important scenarios can be message notifications and reminders, and general scenarios can be operating system log records. In this way, system designers or users can set the weight value of the scenario according to the importance of the scenario. Since important scenarios have high real-time requirements, their message backlog threshold, danger threshold, and system performance alarm index are set to be lower. Less important scenarios have slightly lower real-time requirements than important scenarios and can accept a certain degree of delay. Therefore, the message backlog alarm threshold, danger threshold, and system performance alarm index of less important scenarios are set higher than those of important scenarios. When the data (i.e., messages) of general scenarios are delayed or even lost, it will not affect the overall process. Therefore, the message backlog alarm threshold, danger threshold, and system performance alarm index of general scenarios are usually set at a higher level.

[0035] In step S340, when the message backlog is greater than the message backlog threshold of the current scenario, the processor 110 selects the corresponding strategy according to the setting of the decision table, and then executes the corresponding steps S350 to S370. In step S350, the corresponding strategy is the production speed restriction strategy, so the processor 110 selects the production speed restriction strategy to reduce the message backlog. In step S360, the corresponding strategy is the strategy of increasing the number of threads, so the processor 110 selects the strategy of increasing the number of threads to solve the message backlog. In step S370, the corresponding strategy is the strategy of increasing the number of consumer instances, so the processor 110 selects the strategy of increasing the number of consumer instances to solve the message backlog.

[0036] In step S380, after the processor 110 adopts the corresponding processing strategy, the calculation module dynamically calculates the appropriate number of threads, the number of consumer instances, or the production speed of each scenario according to the decision table. Specifically, when the processing strategy is to increase the number of threads, the calculation module can calculate the appropriate number of threads according to the number of processor cores, the current number of threads, the current number of instances, the message backlog threshold, the message backlog amount, the performance indicator value, and the performance indicator threshold in the processing data group.

[0037] In this embodiment, the calculation module calculates the number of suitable threads according to the following formula (1) and formula (2):

[0038]

[0039] a max is the maximum performance index threshold of the system, a curr is the current system performance index, t curr is the current number of threads, u is the number of CPU cores (i.e., the number of processor cores), m is the number of consumer terminals (assuming that each consumer terminal instance has the same configuration), c alert Message backlog alarm threshold, c curris the current message backlog, t pre For the number of threads, k is a compensation parameter between 0 and 1, and ceil is a round-up function to ensure that the number of threads is an integer. curr max When there are messages to be processed in the message queue (c curr >c alert ), the processor 110 can improve the consumption capacity by increasing the number of threads. Formula (2) is when the suitable number of threads (i.e. the recommended total number of threads t pre ) is higher than the maximum number of threads t max When , take the maximum number of threads as the appropriate number of threads t new , otherwise take t in formula (1) pre as appropriate for the number of threads.

[0040] In one embodiment, the multiple performance indicators of the system include CPU occupancy, memory occupancy, disk occupancy, and bandwidth occupancy, which are the main factors affecting system performance. In one embodiment, the calculation module can obtain the performance indicators of the message processing system in real time through a shell script, and then calculate the performance indicator value of the message processing system through the following formula (3):

[0041] a=w1×b1+w2×b2+w3×b3+w4×b4……(3)

[0042] b1 is the average CPU usage per unit time, b2 is the average memory usage per unit time, b3 is the average disk usage per unit time, b4 is the bandwidth usage per unit time, and the corresponding weight values ​​are w1, w2, w3, and w4. The weight values ​​are all between 0 and 1, that is, 0≦w i ≦1, w1+w2+w3+w4=1, a is the performance index value after weighted calculation, 0≦a≦1. The larger the a value, the worse the system performance.

[0043] In this embodiment, the calculation module calculates the number of suitable consumer instances based on the current number of instances in the processing data group, the average processing speed, the total production speed, and the speed multiple of the newly added instance and the current instance. First, the number of suitable instances must satisfy that the total consumption speed is greater than or equal to the total production speed, so the following formula (4) can be obtained:

[0044] (m*e)+(m new *k*e)>=d……(4)

[0045] ​m is the number of existing consumer terminals, e is the average consumption speed of a single consumer, d is the total production speed of producers, k is the multiple of the average consumption speed of the instance to be expanded, and n is the number of consumer terminals that need to be added (i.e., the number of consumer instances). Then, the following formula (5) can be obtained:

[0046]

[0047] m new is the number of instances that need to be supplemented, i is the compensation parameter, and i×m×e is the total speed of current consumption. For example, in the current environment, the maximum number of instances that can be accommodated is m max , then the maximum number of additional instances is m max -m, then the total number of instances that can be added (i.e. the number of suitable instances) is m after considering the maximum number of instances that can be accommodated. new_1 , and m new_1 =min(m new , m max -m).

[0048] In one embodiment, the calculation module calculates the suitable production speed based on the relevant scene list, scene weight value, scene message backlog, maximum weight value, scene speed reduction ratio and minimum speed value in the message queue. The scene message backlog is the total message backlog of all scenes, and the scene speed reduction ratio is related to the ratio between the current backlog and the scene message backlog and the difference between the scene weight value and the threshold weight. Specifically, the system sets different scene weight values ​​for scenes of different importance. For example, first obtain a list of all scenes involving the message queue in the current system, divide the scene list according to the importance level, involving a total of n levels, divide the n levels by weight value, and set the scene weight values ​​to m_w1, m_w2, m_w3, m_w4, ..m_w respectively. n , the corresponding message backlog is m_c1,m_c2,m_c3,m_c4,..m_c n , where m_w1>m_w2>m_w3>...>m_w n , and m_w1+m_w2+m_w3+...+m_w n =1.

[0049] Next, based on the current backlog m_c i and the scene weight m_w i The limiting factor is calculated using the following formula (6):

[0050] f_i=(m_c i / Σm_c)*(max_weight-m_w i) / (max_weight - w_threshold)……(6)

[0051] w_threshold is the threshold value, m_w i is the scene weight value, Σm_c is the sum of the message backlogs of all scenes, max_weight is the threshold weight, m_c i is the backlog of the current scene, f_i is a value between 0 and 1, representing the proportion by which the producer of scene i should reduce its message generation speed, and for each scene i (i ranging from 1 to n). The threshold weight is related to the maximum weight value among all production scenes in the current system. When calculating the threshold weight max_weight, it is necessary to first obtain the list of all scenes involving message queues in the current system and divide the scene list according to the importance level. For example, if the weight values in the scene list involve n levels, then the weight values of the n levels are divided, and the level weight values are set as m_w_1, m_w_2, m_w_3, m_w_4,..m_w_n respectively, and the largest weight value among them is the threshold weight max_weight.

[0052] Next, the suitable production speed for the current scene is obtained according to the following formula (7):

[0053] rate_new_i = max(min_rate, y_i * rate_original_i * (1 - f_i))……(7)

[0054] Assume that rate_original_i is the original message generation speed, rate_new_i is the new suitable production speed. min_rate is the minimum speed threshold, y_i is the current scene compensation parameter, which can be flexibly adjusted according to the system environment, 0 < y_i <= 1. In this way, the calculation module can calculate the suitable production speed for the current scene, thereby reducing the message backlog and maintaining the message flow of each business scene through the minimum speed threshold.

[0055] Figure 4 is a schematic diagram of the production end, message queue, consumption end, and database of an embodiment of the present invention. Figure 5 is a flowchart of the message backlog processing method of another embodiment of the present invention. First, refer to Figure 4When the user performs user interaction 401 through the system interaction interface, the instruction is input into the producer 403 end of the message in the message processing system. The producer 403 then passes the message to the consumer 405 end (i.e., the message processing end) of the message through the message queue 404, and then processes the message through the consumer 405 end and its thread configuration. In this way, the message processing system processes the message into corresponding operation logic or program code, and stores it in the database 407. The message backlog processing system 100 performs system detection 402 through the data detection module, and then obtains the production log, consumption log, message queue type (i.e., message queue data) and processing data group. The processing data group is the data of the message processing system when processing the message. Refer to Figure 1 , Figure 3 , Figure 4 as well as Figure 5 , the message backlog processing system 100 can execute the following steps S501 to S515.

[0056] In step S501, when the data detection module detects the data of the message processing system, if the data detection module detects that there is a message backlog, steps S502 to S504 are then executed. In step S503, the decision module determines whether the message backlog is greater than the message backlog threshold. When the message backlog is greater than the message backlog threshold, the decision module then determines whether the message backlog is greater than the backlog danger threshold. In response to the message backlog being less than the backlog danger threshold, the decision module then determines whether the system performance index is lower than the performance index threshold. In response to the system performance index being lower than the performance index threshold, the decision module uses the strategy of increasing the number of threads as the current processing strategy according to the decision table, i.e., executes step S509.

[0057] In step S509, the decision module adopts the strategy of increasing the number of threads, and then determines the system index threshold, the current system performance index, the current number of threads, the number of processor cores, the number of instances, the message backlog threshold and the message backlog through the data collection of the data detection module and the calculation of the calculation module. In step S510, the calculation module calculates the recommended number of threads (i.e., the suitable number of threads) according to the above formula (1) and formula (2). In addition, the calculation module inputs the suitable number of threads into the message processing system so that the message processing system adjusts the number of threads to the suitable number of threads.

[0058] In step S511, the data detection module collects the production log and consumption log in the message processing system in real time, and then determines whether the current message processing state of the message processing system satisfies the production speed less than the consumption speed. In response to the production speed being greater than the processing speed, and in response to the processing strategy being the strategy of increasing the number of threads, the decision module updates the processing strategy to the strategy of increasing the number of consumer instances. That is, in step S511, when the production speed is still greater than the consumption speed, step S512 is then executed.

[0059] In step S504, the decision module determines whether the message backlog is greater than the message backlog threshold and the performance index is greater than the performance index threshold. In response to the system performance index being greater than the performance index threshold, the decision module uses the strategy of increasing the number of consumer instances as the processing strategy according to the decision table, i.e., executing step S512.

[0060] In step S512, the decision module adopts the strategy of increasing the number of instances, and then determines the number of consumption instances, the average consumption speed of a single consumer, and the total production speed of the producer through the collection of the data detection module and the calculation of the calculation module. Then, step S513 is executed. In step S513, the calculation module calculates the number of suitable consumer instances based on the processed data group, and then updates the number of consumer instances in the message processing settings of the message processing system. In step S514, the data detection module collects the production log and consumption log in the message processing system in real time, and then determines whether the current message processing state of the message processing system satisfies the production speed less than the consumption speed. In response to the production speed being greater than the processing speed, and in response to the processing strategy being the strategy of increasing the number of consumer instances, the decision module updates the processing strategy to the strategy of limiting the production speed. That is, in step S514, when the processing settings of the message processing system are updated to be suitable for the number of consumer-end instances, if the production speed of its messages is still greater than the consumption speed, then step S505 is executed.

[0061] In step S502, in response to the message backlog being greater than the backlog danger threshold, the decision module determines whether the message backlog is greater than K times the backlog danger threshold (i.e., the backlog danger threshold), where K is a positive integer. Next, in response to the message backlog being greater than K times the backlog danger threshold, the decision module uses the production speed restriction strategy as the processing strategy, i.e., executes step S505. In step S505, the decision module adopts the production speed restriction strategy, and then determines the backlog of each scene through the collection of the data detection module and the calculation of the calculation module, and calculates the restriction factor. In step S506, the calculation module calculates the message production speed of each scene (i.e., the suitable production speed of each scene) based on the above formula (4) and formula (5) and according to the processing data group. Next, the message backlog processing system 100 inputs the suitable production speed of each scene into the message processing system, and then updates the production speed in the message processing setting of the message processing system.

[0062] In step S507, the data detection module collects the production log and consumption log in the message processing system in real time, and then determines whether the current message processing state of the message processing system satisfies the production speed being less than the consumption speed. In other words, the data detection module detects the production log and consumption log after the message processing settings are updated to determine whether the production speed is less than the processing speed.

[0063] The processing strategy further includes a fallback strategy, wherein in response to the production speed being greater than the processing speed, and in response to the processing strategy being a production speed limiting strategy, the decision module executes step S508 to update the processing strategy to a fallback strategy. In step S508, the decision module adopts the fallback strategy, and the calculation module proportionally reduces the production speed according to the processing data group until the production speed is less than the processing speed. And, in step S508, when the production speed is less than the processing speed, step S515 is then executed. In the judgment process of step S506, step S510, and step S513, when the production speed of the message processing system is less than the consumption speed, step S515 is then executed. In step S515, the process ends.

[0064] In another embodiment, in response to the message backlog being greater than the danger value, and the message backlog being less than K times the backlog danger threshold, the decision module sequentially uses the strategy of increasing the number of threads and the strategy of increasing the number of consumer instances as processing strategies until the message backlog is less than the message backlog threshold. In other words, when the message backlog in the message processing system is greater than the danger value, but the message backlog is not greater than K times the backlog danger threshold, the decision module first adopts the strategy of increasing the number of threads. After the strategy of increasing the number of threads is adopted, when the message backlog still cannot be lower than the danger value, the decision module then adopts the strategy of increasing the number of instances.

[0065] In summary, the message backlog processing system and message backlog processing method of the present invention can set a variety of processing strategies according to different scenarios, different message backlogs and system performance indicators to effectively reduce the message backlog situation, and at the same time, important scenarios / important instructions can be processed smoothly to effectively avoid the loss of important messages due to message backlog or production speed limitations. In other words, the message backlog processing system of the present invention automatically increases or decreases consumption resources (i.e., the number of threads, the number of consumption instances or the production speed) based on the real-time load of the message processing system / platform, the message backlog, the production speed, and the consumption speed to ensure that the corresponding processing capacity can be provided during the peak period of task card issuance (i.e., the peak period of message generation), while avoiding idle resources during the low peak period of message generation, thereby solving the problem of message backlog and realizing efficient use of resources.

[0066] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A message backlog processing system, characterized in that: include: A storage device for storing the data detection module, the decision module and the calculation module; as well as a processor, coupled to the storage device and configured to execute the data detection module, the decision module and the calculation module, The data detection module collects production logs, consumption logs, message queue data and processing data groups, wherein the processing data group includes message backlogs, In response to the message backlog being greater than the message backlog threshold, the decision module obtains a processing strategy according to the decision weights in the decision table and the processing data group, wherein the calculation module calculates a suitable performance value according to the processing strategy, and updates the message processing setting according to the suitable performance value, The processing strategy is one of a production speed restriction strategy, a thread number increase strategy, and a consumer instance number increase strategy, and the suitable performance value is one of a suitable production speed, a suitable thread number, and a suitable consumer instance number.

2. The message backlog processing system according to claim 1, characterized in that: The storage device stores weight values ​​of multiple scenes, The calculation module calculates the performance index value according to the weight value, the processor occupancy rate, the memory occupancy rate, the disk occupancy rate, the bandwidth occupancy rate in the processing data group and the corresponding weight value. The calculation module sets different message backlog thresholds, danger thresholds and performance indicator thresholds according to the multiple scenarios.

3. The message backlog processing system according to claim 1, characterized in that: The calculation module calculates the suitable number of threads according to the number of processor cores, the current number of threads, the current number of instances, the message backlog threshold, the message backlog amount, the performance indicator value, and the performance indicator threshold in the processing data group.

4. The message backlog processing system according to claim 1, characterized in that: When the message backlog is greater than the message backlog threshold, the decision module determines whether the message backlog is greater than a backlog danger threshold. In response to the message backlog being less than the backlog danger threshold, the decision module determines whether the system performance indicator is lower than a performance indicator threshold, In response to the system performance indicator being lower than the performance indicator threshold, the decision module uses the strategy of increasing the number of threads as the processing strategy according to the decision table, In response to the system performance indicator being higher than the performance indicator threshold, the decision module uses the strategy of increasing the number of consumer instances as the processing strategy according to the decision table, In response to the message backlog being greater than the backlog danger threshold, the decision module determines whether the message backlog is greater than K times the backlog danger threshold, where K is a positive integer. In response to the message backlog not being greater than K times the backlog danger threshold, the decision module sequentially uses the strategy of increasing the number of threads and the strategy of increasing the number of consumer instances as the processing strategy until the message backlog is less than the message backlog threshold. In response to the message backlog being greater than K times the backlog danger threshold, the decision module uses the production speed limiting strategy as the processing strategy.

5. The message backlog processing system according to claim 1, characterized in that: The data detection module detects the production log and the consumption log after the message processing setting is updated to determine whether the production speed is lower than the processing speed.

6. The message backlog processing system according to claim 5, characterized in that: The processing strategy further includes a fallback strategy, wherein in response to the production speed being greater than the processing speed, and in response to the processing strategy being the production speed limiting strategy, the decision module updates the processing strategy to the fallback strategy, wherein the calculation module reduces the production speed in proportion to the processing data group until the production speed is less than the processing speed.

7. The message backlog processing system according to claim 5, characterized in that: In response to the production speed being greater than the processing speed, and in response to the processing strategy being the strategy of increasing the number of threads, the decision module updates the processing strategy to the strategy of increasing the number of consumer instances, The calculation module calculates the number of suitable consumer instances according to the processing data group, and then updates the number of consumer instances in the message processing setting.

8. The message backlog processing system according to claim 7, characterized in that: The calculation module calculates the number of instances suitable for consumers based on the current number of instances in the processing data group, the average processing speed, the total production speed, and the speed multiple of the newly added instance and the current instance.

9. The message backlog processing system according to claim 5, characterized in that: In response to the production speed being greater than the processing speed, and in response to the processing strategy being the strategy of increasing the number of consumer instances, the decision module updates the processing strategy to the strategy of limiting the production speed, The calculation module calculates the suitable production speed according to the processing data group, and then updates the production speed in the message processing setting.

10. The message backlog processing system according to claim 9, characterized in that: The calculation module calculates the suitable production speed according to the relevant scene list, scene weight value, scene message backlog, maximum weight value, scene speed reduction ratio and minimum speed value in the message pair list. The scenario message backlog is the total message backlog of all scenarios. The scene speed reduction ratio is related to the ratio between the current backlog and the scene message backlog and the difference between the scene weight value and the threshold weight.

11. A method for processing message backlog, characterized in that: include: Executing a data detection module through a processor to collect production logs, consumption logs, message queue data, and a processing data group, wherein the processing data group includes a message backlog; When the message backlog is greater than the message backlog threshold, executing a decision module through the processor to obtain a processing strategy according to the decision weights in the decision table and the processing data group; Executing a calculation module by the processor to calculate a suitable performance value according to the processing strategy, and updating a message processing setting according to the suitable performance value; The processing strategy is one of a strategy for limiting production speed, a strategy for increasing the number of threads, and a strategy for increasing the number of consumer instances. The suitable performance value is one of a suitable production speed, a suitable number of threads, and a suitable number of consumer instances.

12. The method according to claim 11, characterized in that The storage device stores weight values ​​of multiple scenes, wherein the method further comprises the following steps: Calculating a performance index value by the calculation module according to the weight value, the processor occupancy rate, the memory occupancy rate, the disk occupancy rate, the bandwidth occupancy rate in the processing data group and the corresponding weight value; and Different message backlog thresholds, danger thresholds and performance indicator thresholds are set respectively according to the multiple scenarios through the calculation module.

13. The method according to claim 11, characterized in that The following steps are also included: The calculation module calculates the suitable number of threads according to the number of processor cores, the current number of threads, the current number of instances, the message backlog threshold, the message backlog amount, the performance indicator value, and the performance indicator threshold in the processing data group.

14. The method according to claim 11, characterized in that The following steps are involved: When the message backlog is greater than the message backlog threshold, the decision module determines whether the message backlog is greater than a backlog danger threshold; In response to the message backlog being less than the backlog danger threshold, determining, by the decision module, whether a system performance indicator is lower than a performance indicator threshold; In response to the system performance indicator being lower than the performance indicator threshold, using the strategy of increasing the number of threads as the processing strategy according to the decision table by the decision module; In response to the system performance indicator being higher than the performance indicator threshold, using the strategy of increasing the number of consumer instances as the processing strategy according to the decision table by the decision module; In response to the message backlog being less than the backlog danger threshold, determining by the decision module whether the message backlog is greater than K times the backlog danger threshold, where K is a positive integer; In response to the message backlog not being greater than K times the backlog danger threshold, the decision module sequentially uses the strategy of increasing the number of threads and the strategy of increasing the number of consumer instances as the processing strategy until the message backlog is less than the message backlog threshold; and In response to the message backlog being greater than K times the backlog danger threshold, the production speed limiting strategy is used as the processing strategy by the decision module.

15. The method according to claim 11, characterized in that The step of updating the message processing settings then includes: The production log and the consumption log after the message processing setting is updated are detected by the data detection module to determine whether the production speed is lower than the processing speed.

16. The method according to claim 15, characterized in that The processing strategy also includes a fallback strategy. In response to the production speed being greater than the processing speed, and in response to the processing strategy being the production speed limiting strategy, the steps include: Updating the processing strategy to the fallback strategy through the decision module; and The production speed is proportionally reduced by the calculation module according to the processing data set until the production speed is less than the processing speed.

17. The method according to claim 15, characterized in that In response to the production speed being greater than the processing speed, and in response to the processing strategy being the strategy of increasing the number of threads, the steps include: Updating the processing strategy to the strategy of increasing the number of consumer instances through the decision module; and The calculation module calculates the number of suitable consumer instances according to the processing data group, and then updates the number of consumer instances in the message processing setting.

18. The method according to claim 17, characterized in that The following steps are also included: The number of instances suitable for consumers is calculated by the calculation module according to the current number of instances in the processing data group, the average processing speed, the total production speed, and the speed multiple of the newly added instance and the current instance.

19. The method according to claim 15, characterized in that In response to the production speed being greater than the processing speed, and in response to the processing strategy being the strategy of increasing the number of consumer instances, the steps include: Updating the processing strategy to the production speed limiting strategy through the decision module; and The calculation module calculates the suitable production speed according to the processing data group, and then updates the production speed in the message processing setting.

20. The method according to claim 19, characterized in that The step of calculating the suitable production speed according to the processing data set comprises: The calculation module calculates the suitable production speed according to the scene list, scene weight value, scene message backlog, maximum weight value, scene speed reduction ratio and minimum speed value in the message pair list. The scenario message backlog is the total message backlog of all scenarios. The scene speed reduction ratio is related to the ratio between the current backlog and the scene message backlog and the difference between the scene weight value and the threshold weight.

Images