Message processing method, computer device, storage medium and program product

Abstract

The application discloses a message processing method, computer equipment, a storage medium and a program product, relates to the technical field of message processing, and the message processing method comprises the following steps: acquiring a message overdue time of a to-be-processed message in a plurality of priority queues, and adding the to-be-processed message with the message overdue time being less than or equal to a current time to an execution queue; calculating a message processing quota of the plurality of priority queues based on a historical queue depth of the execution queue, a current queue depth of the execution queue, current queue depths of the plurality of priority queues and weight configuration information of the plurality of priority queues; selecting to-be-processed messages meeting the message processing quota requirement from each priority queue, and taking out the to-be-processed messages from the execution queue and sending the to-be-processed messages to a processing node for processing in response to the to-be-processed messages meeting the message processing quota requirement being added to the execution queue. The technical problem of extensive message scheduling in the prior art is solved, and the technical effect of improving the reliability of message scheduling is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of message processing, and particularly relates to a message processing method, a computer device, a storage medium and a program product. BACKGROUND

[0002] In a distributed storage system, in order to ensure data redundancy and high availability, a multi-node deployment architecture is usually adopted. Data transmission between nodes is performed through high-speed physical links (such as PCIe, RDMA), and these transmissions include not only storage application business data (such as data mirroring, backup), but also storage system management data (such as node heartbeat, configuration synchronization, status information). There are mainly two implementation ways for inter-node communication: one is that various types of data share the same physical link channel, although resource reuse can be achieved, but different priority data transmission will interfere with each other; the other is to establish different dedicated channels for different types of data, although interference can be avoided, but it also brings the problems of unshareable resources, high cost and low utilization.

[0003] In the related art, when the same physical link channel is used for inter-node communication, most storage systems use a simple first-in-first-out (FIFO) or fixed weight round-robin strategy for message scheduling, and this way does not fully consider the urgency difference and its dynamic change of different business messages, and the message scheduling mechanism is extensive, thereby affecting the overall stability and reliability of the system. SUMMARY

[0004] The present application provides a message processing method, a computer device, a storage medium and a program product to solve the technical problem of extensive message scheduling in the related art.

[0005] The present application provides a message processing method applied to a signal analysis device, and the message processing method comprises the following steps:

[0006] Message overdue times of to-be-processed messages in a plurality of priority queues are acquired, and to-be-processed messages with message overdue times less than or equal to a current time are added to an execution queue; message processing quotas of the plurality of priority queues are calculated based on a historical queue depth of the execution queue, a current queue depth of the execution queue, current queue depths of the plurality of priority queues and weight configuration information of the plurality of priority queues; to-be-processed messages meeting the message processing quota requirements are selected from each priority queue, and the to-be-processed messages meeting the message processing quota requirements are added to the execution queue; in response to completion of adding the to-be-processed messages meeting the message processing quota requirements to the execution queue, the to-be-processed messages are taken out from the execution queue and sent to a processing node for processing.

[0007] The application further provides a computer device, comprising: a memory for storing a computer program; and a processor for implementing the steps of the message processing method in the embodiments when executing the computer program.

[0008] The message overdue time of the to-be-processed messages in the plurality of priority queues is obtained, and the to-be-processed messages with the message overdue time less than or equal to the current time are added to the execution queue; the message processing quota of the plurality of priority queues is calculated based on the historical queue depth of the execution queue, the current queue depth of the execution queue, the current queue depths of the plurality of priority queues and the weight configuration information of the plurality of priority queues; the to-be-processed messages meeting the message processing quota requirement are selected from each priority queue and added to the execution queue; and the to-be-processed messages are taken out from the execution queue and sent to the processing node for processing in response to the completion of adding the to-be-processed messages meeting the message processing quota requirement to the execution queue.

[0009] The application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the message processing method in the embodiments.

[0010] The message overdue time of the to-be-processed messages in the plurality of priority queues is obtained, and the to-be-processed messages with the message overdue time less than or equal to the current time are added to the execution queue; the message processing quota of the plurality of priority queues is calculated based on the historical queue depth of the execution queue, the current queue depth of the execution queue, the current queue depths of the plurality of priority queues and the weight configuration information of the plurality of priority queues; the to-be-processed messages meeting the message processing quota requirement are selected from each priority queue and added to the execution queue; and the to-be-processed messages are taken out from the execution queue and sent to the processing node for processing in response to the completion of adding the to-be-processed messages meeting the message processing quota requirement to the execution queue.

[0011] The application further provides a computer program product, comprising a computer program, and the computer program is executed by a processor to implement the steps of the message processing method in the embodiments.

[0012] acquire message overdue time of the pending messages in the plurality of priority queues, add the pending messages with the message overdue time less than or equal to the current time to the execution queue; calculate message processing quota of the plurality of priority queues based on the historical queue depth of the execution queue, the current queue depth of the execution queue, the current queue depths of the plurality of priority queues, and the weight configuration information of the plurality of priority queues; select the pending messages meeting the message processing quota requirement from each priority queue, and add the pending messages meeting the message processing quota requirement to the execution queue; and in response to completion of adding the pending messages meeting the message processing quota requirement to the execution queue, take the pending messages from the execution queue and send them to the processing node for processing.

[0013] The message processing method provided in the application can realize early scheduling of messages about to expire, ensure that the messages can be processed before the business timeout, calculate the message processing quota of the plurality of priority queues based on the historical queue depth of the execution queue, the current queue depth of the execution queue, the current queue depths of the plurality of priority queues, and the weight configuration information of the plurality of priority queues, select the pending messages meeting the message processing quota requirement from each priority queue, and add the pending messages meeting the message processing quota requirement to the execution queue, so that each priority queue can obtain more resources matching the weight, ensure high throughput and low delay of the high-priority queue, eliminate the risk of long-term shelving of low-priority messages, and ensure that the messages of each priority queue can be processed. In response to completion of adding the pending messages meeting the message processing quota requirement to the execution queue, the pending messages are taken from the execution queue and sent to the processing node for processing. In this way, without establishing a dedicated channel for different types of data, the application realizes differentiated services in one shared link, i.e., one execution queue, by combining the early scheduling mechanism of messages about to expire and the message processing quota calculation mechanism of the plurality of priority queues, so that the high-priority, highly relevant, and about-to-expire messages are preferentially transmitted, the technical problem of extensive message scheduling in the related art is solved, the reliability of message processing is improved, and the system performance is improved. BRIEF DESCRIPTION OF DRAWINGS

[0014] To more clearly illustrate the embodiments of the application, the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative effort.

[0015] Figure 1 A schematic diagram of a communication mode between nodes in the related art.

[0016] Figure 2 A flowchart of a message processing method provided by an embodiment of the present application is shown in FIG. 1.

[0017] Figure 3 A structure diagram of a message processing system provided by an embodiment of the present application is shown in FIG. 2.

[0018] Figure 4 A flowchart of a message processing method provided by another embodiment of the present application is shown in FIG. 3.

[0019] Figure 5 A flowchart of a message processing method provided by yet another embodiment of the present application is shown in FIG. 4.

[0020] Figure 6 A flowchart of a message processing method provided by still another embodiment of the present application is shown in FIG. 5.

[0021] Figure 7 A structure diagram of a message processing apparatus provided by an embodiment of the present application is shown in FIG. 6.

[0022] Figure 8 An internal structure diagram of a computer device provided by an embodiment of the present application is shown in FIG. 7. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, any other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0024] It should be noted that, in the description of the present application, the terms “comprise”, “contain” or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed, or includes elements inherent to such process, method, article or device. The terms “first”, “second” and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0025] In order for those skilled in the art to better understand the technical solutions of the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0026] In a storage system, in order to ensure redundancy, at least two nodes or multiple nodes are generally stored. Data is transmitted between nodes through channels such as PCIe (Peripheral Component Interconnect Express), RDMA (Remote Direct Memory Access), etc. There are data for storing application services, which are generally used for mirroring data transmission between nodes, and data for storing system management, which are generally used for synchronizing configuration information, management messages, state information, etc. of the storage system.

[0027] Please refer to Figure 1 At present, there are two main implementation methods for data transmission between nodes. In the first method of inter-node communication, different types of data between nodes use different link channels, such as application service data using a dedicated channel and management data using a different channel. Although this method can achieve that different types of data transmission do not affect each other, there is a problem of resource waste and inability to share resources. Each priority not only has its own waiting queue, but also has its own exclusive execution thread (execution queue). High-priority execution threads run fast, and low-priority execution threads run slow. It cannot achieve true resource sharing and balance. Even if the high-priority queue is idle, its execution thread will not help process the low-priority queue messages, resulting in resource waste.

[0028] In the second method of inter-node communication, various data in inter-node communication share link channels (including business data and management data), which can achieve resource sharing and reduce resource waste, but the transmission of different data affects each other. In this method, all priority messages eventually enter an execution queue. However, the strategy of entering the queue is very rough, such as: simple polling, taking one message from each high, medium, and low priority queue and putting it into the execution queue. This cannot guarantee that high-priority messages will get more resources. Or give the high-priority queue a fixed number of slots per cycle, but it cannot be dynamically adjusted according to real-time system load and cannot handle burst traffic.

[0029] When various data in inter-node communication share link channels, there are currently solutions for priority management of messages between nodes, but there are still the following main problems:

[0030] Message scheduling is rough: most storage systems use FIFO (First-In, First-Out) or simple polling strategies without considering business differences. For example: node fault detection messages and data backup messages are mixed and processed, emergency response is delayed. When the queue depth increases, low-priority messages are "starved", affecting system stability.

[0031] Message association processing: Distributed operations often require multiple message coordination (such as transaction submission requires "preparation-commit" message pair), but if the message association is not established, part of the associated messages will be executed in the wrong order due to different priorities.

[0032] Lack of dynamic optimization for overdue response: Traditional timeout detection only triggers retransmission or alarm, and does not dynamically improve processing priority. For example: cache synchronization messages that are about to expire are not scheduled in priority, causing data inconsistency.

[0033] To solve the above technical problems, as shown in Figure 2 An embodiment of the present application provides a message processing method, which specifically comprises the following steps:

[0034] Step 101: Obtain the message overdue time of the to-be-processed messages in the plurality of priority queues, and add the to-be-processed messages with the message overdue time less than or equal to the current time to the execution queue.

[0035] Please refer to Figure 3 , the storage node 1 sends the service messages received from the plurality of service modules (service module A, service module B, …, service module N) to other storage nodes (storage node 2, storage node 3, …, storage node N) after executing message processing through the message processing system. The message processing method of the present application is applied to the message processing system, the message processing system receives messages from the plurality of service modules, and the messages received from the plurality of service modules are scheduled by the entry queue, the plurality of priority queues (high-priority message queue, …, priority queue, low-priority message queue), the temporary queue and the execution queue set in the message processing system. The to-be-processed messages that need to be processed are scheduled to the execution queue, the to-be-processed messages scheduled to the execution queue are taken out, and are sent to the pre-configured processing node for processing. The message processing system receives the response result returned by the pre-configured processing node; and updates the state of the to-be-processed message corresponding to the serialized message data according to the response result.

[0036] Please refer to Figure 4 , in response to the message processing system receiving the messages sent by the plurality of service modules, a message structure body is constructed based on the information of the messages, the message overdue time of the messages is determined, and the message overdue time of the messages is updated to the message structure body; in response to obtaining an associated message having a preset association relationship with the message, the priority of the message is updated based on the priority of the associated message, and the message is written to the priority queue matching the priority of the message.

[0037] The message structure body is constructed based on the message information, and includes message information and a message information type.

[0038] The message identifier can be a message ID or a message unique identifier, which can be generated by the sender when creating the message, and can be generated by a business identifier, a timestamp, and a random number. The message sequence number is used to identify the order of the message in a certain context, which can be generated according to the order of message sending, for example, message A is the fourth message sent, and the message sequence number of message A can be 4. The message type is used to identify the business type or purpose of the message. The message sender identifier is used to identify the message sender, which can be the ID of a specific business module. The message receiver identifier is used to identify the intended recipient of the message, which can be the ID of the receiving storage node set by a certain business module. The associated message identifier is used to identify the ID of the message and other messages that have a predetermined logical association with the message, which is used to find the associated messages corresponding to the message. The message expiration time refers to the deadline for message scheduling, which can be presented in the form of a timestamp.

[0039] The message information type can be used as a field of the constructed message structure body, and the information corresponding to the message information type can be filled into the corresponding message structure body field to form the message structure body, so that the message information can be quickly obtained according to the message structure body.

[0040] Specifically, the message arrival system entry queue time, the preset business timeout duration, the preset message processing duration, and the message processing time threshold value can be obtained. The message expiration time of the message is calculated based on the message arrival system entry queue time, the preset business timeout duration, the preset message processing duration, the message processing time threshold value, and a message expiration time calculation formula.

[0041] The message expiration time calculation formula is as follows:

[0042] T E =T A +T O -T W -T M ;

[0043] T E represents the message expiration time, representing the scheduling warning time; T A represents the message arrival system entry queue time, representing the time when the system receives the message; T Orepresents a preset service timeout duration, representing the maximum allowed time for message processing; T W represents a preset message processing duration, representing the required time reserved by the system for actual processing, M represents a message processing time threshold, representing a state that the message processing is about to exceed the time limit.

[0044] The system entry queue time here refers to the time when the messages sent by multiple services reach the message processing system entry queue. The preset service timeout duration refers to the maximum allowed time for processing any message at the service level. For example, if the message is a payment message, the entire process from initiating payment to receiving the result must be completed within 10 seconds, and the 10 seconds here is the maximum allowed time for processing the payment message. The preset message processing duration (processing time window) refers to the required time reserved by the system for actual processing, which is an estimated empirical or statistical value. The message processing time threshold is the advance amount of the system for active intervention, which is a buffer time. In actual application, the system cannot wait until the message has reached the preset message processing duration before processing the message. The message processing time threshold defines the state that the message processing is about to exceed the time limit, and the system can perform early warning through the message processing time threshold. When the message enters the message processing time threshold, the highest priority intervention measure is triggered, such as directly adding it to the execution queue to preempt resources and avoid message processing timeout.

[0045] In this way, the message expiration time earlier than the absolute service timeout point is calculated as the scheduling warning time by using the message arrival system entry queue time, the preset service timeout duration, the preset message processing duration, and the message processing time threshold, and the message that is about to exceed the time limit is identified in advance. The scheduler can intervene before this time point, actively raise its priority or forcibly schedule it to the execution queue, changing post-repair to pre-prevention, fundamentally avoiding most business timeout failures, and greatly improving the reliability of the system.

[0046] The message expiration time of each message is calculated and updated to the message timeout time field in the message structure of the message, which is used for subsequent detection of messages that are about to exceed the time limit.

[0047] After updating the message expiration time of each message to the message timeout time field in the message structure of the message, the priority of the message needs to be determined, and the message is added to the priority queue matching the priority of the message according to the priority of the message.

[0048] Specifically, determining the priority of the message can include:

[0049] S1: Determine whether there is an associated message associated with the message in the system.

[0050] Specifically, the associated message identifier code of the message is found from the message structure body of the message; the message matching the associated message identifier code in the system is queried, and the message matching the associated message identifier code in the system is taken as the associated message.

[0051] S2: In response to the existence of the associated message having the association relationship with the message in the system, the associated message is added to the temporary queue, and the associated message state is marked; the associated message state includes: the message arrives at the system entry queue and the message is not processed, the message does not arrive at the system entry queue, and the message is processed.

[0052] S3: In response to the associated message state of the associated message being the message arriving at the system entry queue and the message not being processed, the priority of the associated message is obtained, the priority of the message is updated based on the highest priority corresponding to the associated message, and the message is added to the priority queue matching the updated priority according to the updated priority.

[0053] S4: In response to the non-existence of the associated message having the association relationship with the message in the system, the message type is obtained, the priority of the message is determined according to the message type, and the message is added to the priority queue matching the priority of the message according to the priority of the message.

[0054] In actual application, it is assumed that the plurality of priority queues include a high-priority queue, a medium-priority queue, and a low-priority queue, at this time, the messages in the high-priority queue are all high-priority, the messages in the medium-priority queue are all medium-priority, and the messages in the low-priority queue are all low-priority.

[0055] It is assumed that a low-priority message A depends on a high-priority associated message B. That is, the message B is the associated message of the message A, if the message A arrives at the message processing system first, and after being processed, the message A waits for B in the temporary queue. At this time, a large number of medium-priority messages rush into the system, which will occupy system resources. The high-priority B message cannot be processed in time, and thus is blocked. That is, the high-priority task B is blocked by the medium-priority task because of indirectly waiting for the low-priority task A. This will seriously damage the scheduling expectation of the system.

[0056] In the present application, when message A is waiting for message B, message A inherits the high priority of message B. In this way, A becomes a high-priority message in the view of the message processing system. When a medium-priority message attempts to preempt, it finds that "high-priority A" is running (in fact, message A is waiting for message B), thus causing the medium-priority message to yield resources, ensuring that high-priority message B can be processed as soon as possible. After message B is processed, message A is unlocked and continues to be executed at high priority. Moreover, if high-priority message B relates to a high-risk business such as fund safety, by having message A inherit the high priority of message B, it is also ensured that important business transactions can be given priority as a whole.

[0057] It can be understood that the temporary queue in the present application does not process messages, but only serves as a waiting function. For example, even if message B is an associated message of message A, when message B arrives at the system, message B also performs the above message processing steps as message A. The message processing system sets an initial priority for message B according to the message type of message B, and the system adds message B to the priority queue matching the initial priority of message B, regardless of whether message A is waiting for message B.

[0058] When message B is successfully received by the system, the associated message state of message B is changed to "message arrived at the system entry queue and message is not processed". At this time, it is triggered to check whether there is a temporary queue waiting for message B. If it is found that there is a temporary queue (such as the temporary queue created by message A) waiting for message B, the system adds the ID of message B to the "message arrived at the system entry queue and message is not processed" list of the temporary queue created by message A. It is worth noting that even if message B is still queuing or performing processing operations at this time, as long as the system adds the ID of message B to the "message arrived at the system entry queue and message is not processed" list of the temporary queue created by message A, message A can be unlocked to change from the waiting state to the execution state and enter the execution queue.

[0059] If there is no associated message associated with the message, the priority of the message is set according to the type of the message, and the message is added to the priority queue matching the priority of the message according to the priority of the message. For example, the type of the message is set to high priority if it is core business data transmission (such as I / O path data synchronization), and the type of the message is set to low priority if it is a configuration class asynchronous message (such as parameter update), thereby realizing differentiated scheduling of messages with different priorities between nodes. Under high system load, the processing capacity of messages with different priorities is dynamically adjusted according to factors such as the weight of different priorities, the task queue depth, and the message processing capacity. The priority of a message is also increased if it is associated with a high-priority dependent message. A timeout monitoring mechanism is introduced for the message, and when it is about to expire, the message task is processed in priority.

[0060] Referring to Figure 5 After determining the priority of the message, adding the message into the priority queue matching the priority of the message according to the priority of the message, the message can be traversed in the priority value of the plurality of priority queues from high to low, and the message overdue time of the pending message in the priority queue can be checked from the head of each priority queue. In response to the message overdue time of the pending message being less than or equal to the current time, the pending message is added to the execution queue. In response to the message overdue time of the pending message being greater than the current time, the priority queue in which the pending message with the message overdue time greater than the current time is stopped from being traversed.

[0061] Referring to Figure 6 In the present application, each priority queue is traversed, for example, from the highest priority queue to the middle priority queue to the lowest priority queue. The pending messages in each priority queue are traversed, and the message overdue time of the messages is checked in turn from the head of each priority queue to the tail of the queue. If the message overdue time is less than or equal to the current time, it means that the message is about to expire, and it is removed from the message queue and added to the execution queue. The number of remaining messages in the execution queue increases. If the message overdue time is greater than the current time, it means that the message has not expired. Since the messages are added to the priority queue in chronological order, the following messages have not expired, and the traversal of the priority queue is ended. Until the message traversal of each priority queue is completed.

[0062] Step 102: Calculate the message processing quota of the plurality of priority queues based on the historical queue depth of the execution queue, the current queue depth of the execution queue, the current queue depth of the plurality of priority queues, and the weight configuration information of the plurality of priority queues.

[0063] Specifically, the historical queue depth of the execution queue, the current queue depth of the execution queue, the current queue depth of the plurality of priority queues, and the weight value of the plurality of priority queues are obtained.

[0064] The historical queue depth of the execution queue refers to the total number of messages to be processed in the execution queue at the beginning of the last batch of message processing tasks, representing the load that the message processing system failed to process in time in the last scheduling round, and can be used as an index for measuring the historical pressure of the system. The current queue depth of the execution queue refers to the total number of messages accumulated in the execution queue that have not been taken away by the worker threads at the beginning of the current batch of message processing tasks, which can represent the real-time load pressure of the system. The current queue depth of each priority queue refers to the total number of messages in each priority queue at the beginning of the current batch of message processing tasks. The weight value of each priority queue refers to the weight value of each priority queue, which can be defined according to actual needs and used to represent the proportion of resource allocation configured for each priority queue. The specific value of the weight value of each priority queue is not limited in the present application.

[0065] The message processing quota of each priority queue is calculated based on the historical queue depth of the execution queue, the current queue depth of the execution queue, the current queue depth of each priority queue, the weight value of each priority queue, and a message processing quota calculation formula.

[0066] The message processing quota calculation formula is as follows:

[0067] ;

[0068] P(i) represents the message processing quota of priority queue i, L(i) represents the current queue depth of priority queue i, M(i) represents the weight value of priority queue i, C Total represents the historical queue depth of the execution queue, and C Remain represents the current queue depth of the execution queue.

[0069] In a specific example, assume that the plurality of priority queues include a first priority queue (priority 1, highest priority, weight 70%), a second priority queue (priority 2, second highest priority, weight 20%), and a third priority queue (priority 3, lowest priority, weight 10%). In the current time window, there are 100 messages to be processed in the message queue of priority 1, 150 messages to be processed in the message queue of priority 2, and 200 messages to be processed in the message queue of priority 3. If in the last time window, the message queue depth C Total was 80, and there were 10 messages in the processing queue C Remain . Then according to the message processing quota calculation formula, the message processing quota of the message queue of priority 1 in the next time window is 32, the message processing quota of the message queue of priority 2 is 14, and the message processing quota of the message queue of priority 3 is 9.

[0070] In the present application, (C Total -C Remain ) is a negative feedback loop. When the system is busy, C Remain increases, causing (C Total -C Remain ) to decrease, and the total quota for this allocation is automatically reduced, avoiding injecting more messages into an already congested system. When the system is idle, C Remain decreases (even to 0), causing (C Total -C Remain ) to increase, and the total quota for this allocation is automatically increased, fully digesting the backlog of messages and improving resource utilization, achieving dynamic scheduling of pending messages, and calculating message processing quotas for multiple priority queues based on historical queue depth of the execution queue, current queue depth of the execution queue, current queue depth of multiple priority queues, and weight configuration information of multiple priority queues. Select pending messages from each priority queue that meet the message processing quota requirement and add the pending messages that meet the message processing quota requirement to the execution queue. In this way, each priority queue can obtain more resources matching its weight, ensuring high throughput and low latency for high priority queues, and eliminating the risk of long-term shelving of low priority messages, ensuring that messages of each priority queue are processed.

[0071] In the present application, the weight configuration information of multiple priority queues is obtained from the configuration file, and the weight configuration information includes weight values corresponding to multiple priority queues and priority values of multiple priority queues. The weight values corresponding to each priority queue have a corresponding relationship with the priority values of each priority queue, and the priority values of each priority queue have a corresponding relationship with the priority values of the messages in each priority queue. It can be understood that higher priority queues are configured with higher weights to ensure high throughput and low latency for high priority queues.

[0072] Step 103: Select pending messages from each priority queue that meet the message processing quota requirement and add the pending messages that meet the message processing quota requirement to the execution queue.

[0073] Specifically, the same number of pending messages as the message processing quota corresponding to each priority queue is selected from each priority queue and added to the execution queue.

[0074] In a specific embodiment, if the queue of priority C is not empty, it is determined whether the number of messages to be processed E in the queue of priority C in this round of processing exceeds the message processing quota H of the queue of priority C. If E≤H, the first message at the head of the queue of priority C is taken out and added to the execution queue. Meanwhile, the number of messages to be processed E is incremented. If E>H, the queue of the next level of priority is processed. Until all the priority queues are traversed, the message processing count E of each priority queue is cleared.

[0075] Step 104: In response to the completion of adding the messages to be processed satisfying the message processing quota requirement to the execution queue, the messages to be processed are taken out from the execution queue and sent to the processing node for processing.

[0076] The messages to be processed are taken out from the execution queue. The messages to be processed taken out from the execution queue are serialized to obtain serialized message data. The serialized message data is sent to the pre-configured processing node through the network for processing, and the response result returned by the pre-configured processing node is received. The state of the message to be processed corresponding to the serialized message data is updated according to the response result, and the message processing task of this batch is completed.

[0077] Since network transmission and disk storage can only process byte sequences, they cannot directly process complex objects in memory. Here, the worker thread converts the messages to be processed taken out from the execution queue (usually objects in the structure of Java, Go, etc. in memory) into a byte stream format suitable for network transmission or persistence. The worker thread sends the serialized message data to a pre-configured processing node through network I / O. The pre-configured processing node obtains the information corresponding to the message receiving identifier field in the message structure of the message through the message receiving identifier field. The worker thread synchronously blocks or asynchronously waits for the response result returned by the processing node. The worker thread updates the final state of the message in the system according to the received response result and ends the life cycle of the message. In this way, the message processing task of this batch is completed.

[0078] In an embodiment, the message processing method provided by the present application further comprises: monitoring the queue depth information and the message processing delay information of each priority queue. The queue depth information includes the instantaneous queue depth of each priority queue at different time periods. The message processing delay information includes the average message processing delay and the system resource utilization rate. The queue depth change rate of each priority queue is calculated based on the instantaneous queue depth of each priority queue at different time periods.

[0079] The priority queues can be monitored, and the queue depth change rate of each priority queue can be calculated by analyzing the instantaneous queue depth of each priority queue at different time periods. Specifically, the instantaneous queue depth of each priority queue at the current moment can be taken as the minuend, the instantaneous queue depth at the last moment can be taken as the subtrahend, subtraction can be performed to obtain the instantaneous queue depth difference, the instantaneous queue depth difference can be taken as the divisor, the time interval between the current moment and the last moment can be taken as the dividend, division can be performed, and the quotient obtained by the division can be taken as the queue depth change rate of the priority queue.

[0080] In response to the queue depth change rate of any priority queue being greater than zero and the current instantaneous queue depth of the priority queue being continuously higher than the first depth threshold value, and the message processing delay of the priority queue being continuously higher than the first delay threshold value, the weight value of the priority queue is adjusted upward according to a first adjustment algorithm; the first adjustment algorithm is calculated based on the positive deviation value of the current queue depth from the first depth threshold value, and the upward adjustment amplitude is positively correlated with the size of the positive deviation value.

[0081] When the queue depth change rate of any priority queue is greater than zero (the queue depth is still growing) and the current instantaneous queue depth of the priority queue is continuously higher than the first depth threshold value (the queue depth is too long), and the message processing delay of the priority queue is continuously higher than the first delay threshold value (the processing efficiency is slow), these conditions are met at the same time, which indicates that the priority queue encounters a bottleneck and needs more resources. The purpose of the first adjustment algorithm is to adjust the weight of the priority queue with a large backlog upward. Here, the first adjustment algorithm can be:

[0082] ;

[0083] ;

[0084] wherein G(i) represents the weight value of the priority queue i after the adjustment, F(i) represents the weight value of the current priority queue i, Δw up represents the upward adjustment weight value of the priority queue i, kp1 represents a proportional coefficient, which is a constant, L i represents the current queue depth of the priority queue i, and y represents the first depth threshold value. In this way, the system overload can be responded quickly, and the stability of the system can be improved.

[0085] In response to the queue depth change rate of any priority queue being less than zero and the current instantaneous queue depth of the priority queue being continuously lower than the second depth threshold value, and the overall resource utilization rate of the system being lower than the resource saturation threshold value, the weight value of the priority queue is adjusted downward according to a second adjustment algorithm; the second adjustment algorithm is calculated based on the negative deviation value of the current queue depth from the second depth threshold value, and the downward adjustment amplitude is positively correlated with the size of the negative deviation value; and the weight resource obtained by the downward adjustment is allocated to the priority queue with the highest current instantaneous queue depth and / or the largest message processing delay.

[0086] In response to the queue depth change rate of any priority queue being less than zero (the queue depth becoming shorter) and its current instantaneous queue depth continuously being lower than the second depth threshold (the queue depth being shorter), while the overall resource utilization of the system is lower than the resource saturation threshold (there is a resource surplus), these conditions are met simultaneously, indicating that the priority queue resources are excessive and can be reduced. The second adjustment algorithm aims to reduce the weight of the idle priority queue. The second adjustment algorithm can be:

[0087] ;

[0088] ;

[0089] wherein Q(i) represents the weight value of the priority queue i after the reduction, F(i) represents the weight value of the current priority queue i, Δw down represents the reduced weight value of the priority queue i, kp2 represents a proportional coefficient, which is a constant, L i represents the current queue depth of the priority queue i, and Z represents the second depth threshold. In this way, the resources can be reasonably utilized, and the reliability of the system can be improved.

[0090] Here, after the weight of a queue is reduced, the released resources are immediately allocated to the queue that needs the resources most (the queue with the deepest depth or the largest delay), so as to maximize the utilization of the resources. The adjusted weight configuration information is persisted to a configuration file and / or updated in the system memory and used to execute the next batch of message processing tasks.

[0091] In the present application, the messages stored in the nodes are divided according to the priority, and the processable message quota of each priority is dynamically calculated according to the priority weight, the queue depth, the message processing capacity and other factors, so as to ensure that the high-priority messages are processed as soon as possible and the low-priority messages also have a chance to be processed. The association relationship between the messages is established, the priority of the associated messages is set according to the highest priority among them, the associated messages are ensured to have the same priority scheduling right, and the transaction integrity is ensured. The processing priority of the messages that are about to expire is improved, so as to avoid the problem that the low-priority messages are timed out due to the untimely scheduling.

[0092] Embodiments of the present application provide a message processing device, which is applied to a signal analysis equipment. The message processing device specifically as shown in Figure 7 includes an acquisition module 20, a calculation module 21, an allocation module 22 and a processing module 23.

[0093] The acquisition module 20 is used to acquire the message expiration time of the to-be-processed messages in the plurality of priority queues, and add the to-be-processed messages with the message expiration time less than or equal to the current time to an execution queue.

[0094] The computing module 21 is configured to calculate the message processing quota of the plurality of priority queues based on the historical queue depth of the execution queue, the current queue depth of the execution queue, the current queue depths of the plurality of priority queues, and the weight configuration information of the plurality of priority queues.

[0095] The allocating module 22 is configured to select the to-be-processed messages that meet the message processing quota requirement from each priority queue, and add the to-be-processed messages that meet the message processing quota requirement into the execution queue.

[0096] The processing module 23 is configured to, in response to the adding of the to-be-processed messages that meet the message processing quota requirement into the execution queue being completed, take the to-be-processed messages from the execution queue and send the to-be-processed messages to a processing node for processing.

[0097] As shown in Figure 8 The embodiment of the present application further provides a computer device, including a memory and a processor, the memory stores a computer program, and the processor is arranged to run the computer program to execute the steps in any of the message processing method embodiments.

[0098] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is arranged to execute the steps in any of the message processing method embodiments when running.

[0099] In an example embodiment, the computer readable storage medium described above can include, but is not limited to, a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0100] The skilled person can further realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in general terms in the above description. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0101] The above describes in detail the message processing method provided by the present application. The principle and implementation of the present application are described by applying specific examples, and the above description of the embodiments is only applicable to help understand the method of the present application and its core idea. It should be pointed out that, for ordinary skilled persons in the technical field, some improvements and modifications can be made to the present application without departing from the principle of the present application, and these improvements and modifications also fall within the protection scope of the present application.

Claims

1. A message processing method, characterized in that, The message processing method includes: Obtain the message expiration time of pending messages in multiple priority queues, and add pending messages with expiration times less than or equal to the current time to the execution queue; The message processing quotas for multiple priority queues are calculated based on the historical queue depth of the execution queue, the current queue depth of the execution queue, the current queue depth of multiple priority queues, and the weight configuration information of multiple priority queues. Select messages that meet the message processing quota requirements from each priority queue and add them to the execution queue. In response to the completion of adding pending messages that meet the message processing quota requirements to the execution queue, the pending messages are retrieved from the execution queue and sent to the processing node for processing; Monitor the queue depth information and message processing latency information of each priority queue. The queue depth information includes the instantaneous queue depth of each priority queue at different time periods, and the message processing latency information includes the average message processing latency and system resource utilization. The rate of change of queue depth for each priority queue is calculated based on the instantaneous queue depth of each priority queue at different time periods. If the rate of change of the queue depth of any priority queue is greater than zero and its current instantaneous queue depth is continuously higher than a first depth threshold, and its message processing latency is continuously higher than a first latency threshold, then the weight value of the priority queue is increased according to the first adjustment algorithm; the first adjustment algorithm is calculated based on the positive deviation value between the current queue depth and the first depth threshold, wherein the increase magnitude is positively correlated with the magnitude of the positive deviation value; If the rate of change of the queue depth of any priority queue is less than zero and its current instantaneous queue depth is continuously lower than the second depth threshold, while the overall system resource utilization is lower than the resource saturation threshold, then the weight value of the priority queue is reduced according to the second adjustment algorithm. The second adjustment algorithm is calculated based on the negative deviation between the current queue depth and the second depth threshold, wherein the reduction magnitude is positively correlated with the magnitude of the negative deviation value. The weighted resources obtained from the reduction are allocated to the priority queue with the highest current instantaneous queue depth and / or the largest message processing latency. The adjusted weight configuration information is persisted to the configuration file and / or updated in system memory for use in executing the next batch of message processing tasks.

2. The message processing method according to claim 1, characterized in that, Before the message expiration time for retrieving pending messages from multiple priority queues, the following is included: In response to a received message, a message structure is constructed based on the information in the message. Determine the message expiration time and update the message expiration time to the message structure; In response to the acquisition of a related message with a preset association, the priority of the message is updated based on the priority of the related message, and the message is written to a priority queue that matches the priority of the message.

3. The message processing method according to claim 2, characterized in that, The message structure built based on message information includes: Obtain message information and message information type, wherein the message information type includes message identifier code, message sequence number, message type, message sending identifier, message receiving identifier, associated message identifier code, and message expiration time; Based on the information in the message, a message structure corresponding to the message is constructed, and the message structure includes multiple message information type fields.

4. The message processing method according to claim 2, characterized in that, The message expiration time for the confirmed message includes: Get the message arrival time in the system entry queue, the preset business timeout duration, the preset message processing duration, and the message processing time threshold; The message expiration time is calculated based on the message arrival time in the system entry queue, the preset business timeout duration, the preset message processing duration, the message processing time threshold, and the message expiration time calculation formula. The formula for calculating message expiration time is as follows: T E =T A +T O -T W -T M ; Among them, T E Indicates the message expiration time, representing the scheduling warning time; T A Indicates the time it takes for a message to arrive at the system's entry queue, representing the time it takes for the system to receive the message; T O Indicates the preset service timeout duration, representing the maximum allowed time for message processing; T W T represents the preset message processing duration, indicating the time reserved by the system for actual message processing. M This represents the message processing time threshold, indicating that the message processing is about to expire.

5. The message processing method according to claim 2, characterized in that, The step of responding to the acquisition of an associated message with a preset association relationship with the message, updating the priority of the message based on the priority of the associated message, and writing the message into a priority queue that matches the priority of the message includes: Determine if there are related messages in the system that are associated with the message; In response to the existence of associated messages in the system, the associated messages are added to a temporary queue, and the status of the associated messages is marked. The associated message status includes: the message has arrived at the system entry queue and has not been processed, the message has not arrived at the system entry queue, and the message has been processed. In response to a message status of arriving at the system entry queue but not yet being processed, the priority of the associated message is obtained, the priority of the message is updated based on the highest priority corresponding to the associated message, and the message is added to the priority queue that matches the updated priority according to the updated priority. In response to the absence of a related message in the system, the message type is obtained, the message priority is determined based on the message type, and the message is added to a priority queue that matches the message priority.

6. The message processing method according to claim 5, characterized in that, The determination of whether there are associated messages in the system that are related to the message includes: Find the associated message identifier from the message structure of the message; The system queries messages that match the associated message identifier code and identifies those messages as associated messages.

7. The message processing method according to claim 1, characterized in that, The message processing method further includes: obtaining weight configuration information for multiple priority queues, wherein obtaining the weight configuration information for multiple priority queues includes: Obtain the weight configuration information of multiple priority queues from the configuration file. The weight configuration information includes the weight values ​​corresponding to the multiple priority queues and the priority values ​​of the multiple priority queues. The weight value corresponding to each priority queue is related to the priority value of each priority queue, and the priority value of each priority queue is related to the priority value of the message in each priority queue.

8. The message processing method according to claim 1, characterized in that, The step of obtaining the message expiration time of pending messages in multiple priority queues and adding pending messages with expiration times less than or equal to the current time to the execution queue includes: Traverse each priority queue from highest to lowest priority value, and check the message expiration time of each pending message in the priority queue from the head of the queue. If the message expiration time of the pending message is less than or equal to the current time, the pending message is added to the execution queue. In response to the message expiration time of the pending message being greater than the current time, the process of traversing the priority queue containing pending messages whose expiration time is greater than the current time will stop.

9. The message processing method according to claim 1, characterized in that, The calculation of message processing quotas for multiple priority queues based on the historical queue depth of the execution queue, the current queue depth of the execution queue, the current queue depth of multiple priority queues, and the weight configuration information of multiple priority queues includes: Get the historical queue depth of the execution queue, the current queue depth of the execution queue, the current queue depth of multiple priority queues, and the weight values ​​of multiple priority queues; The message processing quotas for multiple priority queues are calculated based on the historical queue depth of the execution queue, the current queue depth of the execution queue, the current queue depth of multiple priority queues, the weight values ​​of multiple priority queues, and the message processing quota calculation formula. The formula for calculating message processing quotas is as follows: ； P(i) represents the message processing quota of priority queue i, L(i) represents the current queue depth of priority queue i, M(i) represents the weight value of priority queue i, and C Total C represents the historical queue depth of the execution queue. Remain This indicates the current queue depth of the execution queue.

10. The message processing method according to claim 1, characterized in that, The step of selecting messages that meet the message processing quota requirements from each priority queue and adding them to the execution queue includes: The same number of messages to be processed as the message processing quota corresponding to each priority queue are selected from each priority queue and added to the execution queue in turn.

11. The message processing method according to claim 1, characterized in that, The step of retrieving messages from the execution queue and sending them to the processing node for processing includes: Retrieve messages to be processed from the execution queue; The messages to be processed retrieved from the execution queue are serialized to obtain serialized message data; The serialized message data is sent over the network to a pre-configured processing node for processing, and the response results returned by the pre-configured processing node are received. Update the status of the pending message corresponding to the serialized message data based on the response result.

12. A computer device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the message processing method as described in any one of claims 1 to 11.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the message processing method as described in any one of claims 1 to 11.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the message processing method as described in any one of claims 1 to 11.

Images