Micro-service thread queue accumulation processing method

Abstract

The application discloses a micro-service thread queue accumulation processing method, comprising the following steps: S1, establishing a micro-service thread queue model; S2, assigning a time flag and summarizing processing conditions to the queue model; S3, a method for detecting queue accumulation; S4, a recovery method for queue accumulation; and S5, a tracing method for queue accumulation. The application can solve the problems of difficult identification of thread queue accumulation in micro-service, difficult quick recovery and difficult accurate tracing to the message type causing the problem, accurately locate the message type causing the problem at the time, improve problem troubleshooting efficiency, quickly recover the queue and automatically locate the problem compared with the existing technical method, and therefore the application can solve the thread queue accumulation problem in the micro-service system, automatically recover the micro-service, improve the availability of the micro-service, automatically locate the problem message, and simplify the current complex governance work of the micro-service group.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of microservices, in particular to a microservice thread queue accumulation processing method. BACKGROUND

[0002] Unlike single applications, microservices call each other through a message mechanism, and each service has limited message processing capability. The service provider generally puts the received message into a queue first, and then multiple threads pull the message for processing.

[0003] If the message production rate of the service caller is continuously faster than the message consumption rate of the service provider, queue accumulation will occur. The service caller will therefore have a large number of timeouts and will not receive a response from the message provider. If the service caller is also dependent on other callers, a chain reaction will occur, resulting in a large number of service exceptions in the microservice system.

[0004] Based on this, the present application discloses a microservice thread queue accumulation processing method, which can identify that the queue has accumulated and quickly remove the accumulation to restore the processing capacity of the queue, and locate the faulty message. SUMMARY

[0005] The technical problem to be solved by the present application is to overcome the defects of the prior art and provide a microservice thread queue accumulation processing method.

[0006] The present application provides the following technical solutions:

[0007] The present application provides a microservice thread queue accumulation processing method, comprising the following steps:

[0008] S1, a microservice thread queue model is established:

[0009] A receiving queue is established inside each microservice, the queue is a bounded queue with a length of a, a>=10000, b consumer threads are started, b>=100, and the b consumer threads consume the messages in the queue fairly;

[0010] S2, the queue model is assigned a time flag and a summary processing condition:

[0011] (a) the service caller calls the message sending method of the service provider:

[0012] Specifically, the sent message packet must be attached with the following two key fields: timeout and msgType;

[0013] (a1) wait for the response timeout time timeout, which indicates that the service caller will not continue to wait after timeout seconds of waiting without receiving a response;

[0014] (a2) message type msgType, indicating the message type unique to the service provider, such as mobile phone login, sending login SMS, etc., and the message type set of the service provider is denoted as Q;

[0015] (b) Service provider receives message queuing method:

[0016] After the service provider receives the request message of the service invoker, the timeout and msgType are obtained from the message, and the time when the message request is received is recorded as the queuing time enQueueTime. The request content, enQueueTime, timeout and msgType are put into the queue as a data element;

[0017] (c) Service provider arrival frequency periodic summary method:

[0018] For each msgType, the arrival is calculated once per minute, and the arrival frequency of the message type is calculated;

[0019] Specifically, after the service provider receives the request, the number of msgType within 1 minute is accumulated for each request, and the arrival frequency arriveCount is calculated. The arrival frequency of msgType m at time t is denoted as g(m, t), and the arrival frequency of msgType m at time t-1 is denoted as g(m, t-1) by analogy.

[0020] (d) Service provider consumer data element dequeuing method, the service provider has b consumer threads, and if the queue is not empty, a consumer thread dequeues a data element from the queue;

[0021] (e) Service provider business processing method for the data element:

[0022] (e1) After a certain data element is dequeued, the current time is recorded as the request dequeuing time outQueueTime;

[0023] (e2) The message content in the data element is processed, and the processing time is denoted as handleTime;

[0024] (f) Service provider periodic summary method for processing efficiency:

[0025] For each msgType, the processing is calculated once per minute, and the average processing time avgHandleTime is calculated; the formula is as follows:

[0026] avgHandleTime(msgType) = handleTime(msgType) / count(msgType);

[0027] count(msgType) is the number of data elements of the same msgType processed in 1 minute;

[0028] ΣhandleTime(msgType) is the sum of the total processing time of data elements of the same msgType processed in 1 minute, and the average processing time avgHandleTime(msgType) is obtained;

[0029] The average processing time avgHandleTime(msgType, t) of msgType m at time t is simplified as f(m, t);

[0030] Similarly, the average processing time of msgType m at time t-1 is simplified as f(m, t-1);

[0031] S3, method for detecting queue accumulation:

[0032] If the following two conditions occur, it is determined that the queue is accumulated:

[0033] (1) The service provider receives a new message, but at this time the queue length has reached a, so it cannot continue to enter the queue, and the queue is full and accumulated, at this time the request is discarded, and the first accumulation time r is recorded, at this time the last time is not accumulated, if the last time is accumulated, it has been detected at the last time;

[0034] (2) After dequeuing from the queue, calculate outQueueTime, check outQueueTime-enQueueTime>timeout, so the request invoker no longer waits for its response result, and it is not necessary to continue processing, so the request is discarded; If there are c requests that need to be discarded in succession, it is determined that the queue is accumulated, and the first accumulation time r is recorded, at this time the last time is not accumulated, if the last time is accumulated, it has been detected at the last time;

[0035] S4, method for recovering queue accumulation:

[0036] Because the service provider has too many messages or the processing speed is lower than the sending speed of the service invoker, and the service invoker will keep resending requests to the service provider after perceiving the timeout; If the queue cannot be recovered in time, chain conduction will occur, and finally all services of the system will be at risk; The specific recovery method is:

[0037] (1) System recovery, the historical message invoker no longer waits, and it is not necessary to process the invalid messages, so all messages in the queue are emptied for system recovery;

[0038] (2) Automatic expansion, increase the consumption capacity of service providers, specifically: if cpu<30%, the number of threads increases by 100%, if 30%<cpu<50%, the number of threads increases by 30%;

[0039] S5, the tracing method of queue accumulation:

[0040] In the case of queue accumulation, find out which messages cause the accumulation; the specific method is:

[0041] (1) Trace the message type with too long average processing time;

[0042] For all message types msgType, m∈Q, if f(m,r)>α*f(m,r-1), then m is put into set E, E is the set of message types with long processing time, and α≥2, α is a preset threshold;

[0043] (2) Trace the message type with too fast arrival rate;

[0044] For all message types msgType, m∈Q, if g(m,r)>α*g(m,r-1), then m is put into set H, H is the set of message types with too fast arrival rate, and β≥2, β is a preset threshold;

[0045] Finally, at the accumulation moment r, find the message type sets E and H that cause the accumulation.

[0046] Compared with the prior art, the beneficial effects of the present application are as follows:

[0047] The present application provides a micro-service thread queue accumulation processing method to solve the problems of difficult identification of thread queue accumulation in micro-service, difficult quick recovery and difficult accurate tracing to the message type causing the problem. The method can accurately locate the message type causing the problem at the time by statistically summarizing the de-queue and enqueue time of the queue and the message input speed, improve the problem troubleshooting efficiency, compared with the existing technical method, can quickly recover the queue and automatically locate the problem;

[0048] In summary, the present application can solve the thread queue accumulation problem in the micro-service system, and can automatically recover the micro-service, improve the availability of the micro-service, and automatically locate the problem message, simplify the current complex governance of the micro-service group. BRIEF DESCRIPTION OF DRAWINGS

[0049] The accompanying drawings are used to provide a further understanding of the present application, and constitute a part of the specification, together with the embodiments of the present application, to explain the present application, and do not constitute a limitation on the present application. In the drawings:

[0050] Figure 1 is a flowchart of the present application. Detailed Implementation

[0051] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention. All identical reference numerals in the drawings refer to the same components.

[0052] Example 1

[0053] like Figure 1 This invention provides a method for handling thread queue backlog in microservices, comprising the following steps:

[0054] S1. Establish a microservice thread queue model:

[0055] Each microservice establishes a receiving queue, which is a bounded queue with a length of a, where a ≥ 10000. It starts b consumer threads, where b ≥ 100, and the b consumer threads consume messages from the queue fairly.

[0056] S2. Queue model allocation of time stamps and summary processing status:

[0057] (a) The service caller invokes the service provider's message sending method:

[0058] Specifically, the message to be sent must include the following two key fields: timeout and msgType;

[0059] (a1) Response timeout: This field indicates that if the service caller does not receive a response after waiting for timeout seconds, it considers the timeout to have occurred and will no longer continue waiting.

[0060] (a2) Message type msgType represents the unique message type of the service provider, such as mobile phone number login, sending login SMS, etc. The set of message types of the service provider is denoted as Q;

[0061] (b) Method for service providers to receive and enqueue messages:

[0062] After the service provider receives the request message from the service caller, it obtains the timeout and msgType from the message, records the time when the message request was received as the enqueue time, and puts the request content, enqueueTime, timeout and msgType into the queue as a data element.

[0063] (c) Method for periodically summarizing service provider arrival frequency:

[0064] Perform an arrival summary calculation for each msgType every minute to count the arrival times of the message type;

[0065] Specifically, the service provider receives the request, and for each requested msgType, accumulates the number of msgTypes within 1 minute to obtain the arrival count arriveCount. The arrival count of msgType m at time t is denoted as g(m, t), and the arrival count of msgType m at time t-1 is denoted as g(m, t-1).

[0066] (d) Service provider consumer data element dequeuing method. The service provider has b consumer threads. If the queue is not empty, a consumer thread dequeues a data element from the queue.

[0067] (e) Service provider business processing method for the data element:

[0068] (e1) After dequeuing a certain data element, record the current time as the request dequeuing time outQueueTime.

[0069] (e2) Process the message content in the data element, and the processing time is denoted as handleTime.

[0070] (f) Service provider periodic summary method for processing efficiency:

[0071] For each msgType, the processing situation is summarized once per minute to obtain the average processing time avgHandleTime. The formula is as follows:

[0072] avgHandleTime(msgType) = handleTime(msgType) / count(msgType);

[0073] count(msgType) is the number of data elements of the same msgType processed within 1 minute.

[0074] ΣhandleTime(msgType) is the sum of the total processing time of data elements of the same msgType processed within 1 minute, and the average processing time avgHandleTime(msgType) is obtained.

[0075] The average processing time of msgType m at time t is denoted as f(m, t).

[0076] Similarly, the average processing time of msgType m at time t-1 is denoted as f(m, t-1).

[0077] S3, Method for detecting queue accumulation:

[0078] If the following two conditions occur, determine that the queue is accumulated:

[0079] (1) The service provider receives a new message, but at this time the queue length has reached a, so it cannot continue to enter the queue, and the queue is full. The queue is accumulated at this time, the request is discarded, and the first accumulation time r is recorded. At this time, the last time was not accumulated, and if the last time was accumulated, it was detected at the last time.

[0080] (2) After dequeuing from the queue, calculate outQueueTime, check outQueueTime-enQueueTime>timeout, so the request caller is no longer waiting for its response result, and there is no need to continue processing, so the request is discarded. If there are c requests that need to be discarded in succession, it is determined that the queue is accumulated, and the first accumulation time r is recorded. At this time, the last time was not accumulated, and if the last time was accumulated, it was detected at the last time.

[0081] S4, queue accumulation recovery method:

[0082] Because the service provider has too many messages or the processing speed is lower than the service caller's sending speed, and the service caller will keep resending requests to the service provider after perceiving the timeout. If the queue cannot be recovered in time, chain transmission will occur, and eventually all services in the system will be at risk. The specific recovery method is:

[0083] (1) System recovery, the historical message caller is no longer waiting, and there is no need to process the invalid message, so empty the queue first to recover the system;

[0084] (2) Automatic expansion, increase the consumption capacity of the service provider, which can be specifically: if cpu<30%, increase the number of threads by 100%; if 30%<cpu<50%, increase the number of threads by 30%;

[0085] S5, queue accumulation tracing method:

[0086] In the case of queue accumulation, it is necessary to find which messages cause the accumulation. The specific method is:

[0087] (1) Trace the message type with too long average processing time;

[0088] For all message types msgType, m∈Q, if f(m,r)>α*f(m,r-1), then m is put into set E, E is the set of long processing time accumulation message types, α≥2, α is a preset threshold;

[0089] (2) Trace the message type with too fast arrival rate;

[0090] For all message types msgType, m∈Q, if g(m,r) > α*g(m,r-1), m is put into set H, H is the set of message types that arrive too fast, β≥2, β is a preset threshold;

[0091] Finally, at the accumulation moment r, the message type set E and H that cause the accumulation are found.

[0092] The application provides a micro-service thread queue accumulation processing method to solve the problems of difficult identification of thread queue accumulation, difficult quick recovery and difficult accurate tracing to the message type causing the problem in the micro-service. The method can accurately locate the message type causing the problem at the time by statistically collecting the de-queue and en-queue time of the queue and the message input speed, improves the problem troubleshooting efficiency, and can quickly recover the queue and automatically locate the problem compared with the existing technical method.

[0093] To sum up, the application can solve the thread queue accumulation problem in the micro-service system, automatically recover the micro-service, improve the availability of the micro-service, and automatically locate the problem message, and simplifies the current complex governance work of the micro-service group.

[0094] Finally, it should be noted that: the above only describes the preferred embodiments of the application and is not used to limit the application, although the application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions recorded in the foregoing embodiments or make equivalent replacement for some technical features. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the application shall be included in the protection scope of the application.

Claims

1.A method for processing microservice thread queue accumulation, characterized in that, The method comprises the following steps: S1, establishing a micro-service thread queue model: Establish a receiving queue in each micro-service, which is a bounded queue with a length of a (a≥10000), and start b consumer threads (b≥100), which fairly consume messages in the queue; S2, queue model allocation time flag and summary processing: (a) The service calling party calls the message sending method of the service provider: Specifically, the sent message packet must be attached with the following two key fields: timeout and msgType; (a1) Wait for the response timeout time timeout, which indicates that the service calling party will no longer continue to wait after timeout seconds without receiving a response; (a2) Message type msgType, indicating the unique message type of the service provider, including mobile phone login, sending login SMS, and the message type set of the service provider is denoted as Q; (b) Service provider receives message into queue method: After the service provider receives the request packet of the service calling party, obtain timeout and msgType from the packet, record the time of receiving the packet request as enQueueTime, and put request content, enQueueTime, timeout and msgType into the queue as a data element; (c) Service provider arrival frequency period summary method: Calculate the arrival frequency of each msgType every minute, and calculate the arrival frequency of the message type; Specifically, after the service provider receives the request, the number of msgType within 1 minute is accumulated for each request msgType, and the arrival frequency arriveCount is obtained, and the arrival frequency of msgType m at time t is denoted as g(m, t), and the arrival frequency of msgType m at time t-1 is denoted as g(m, t-1); (d) Service provider consumer data element dequeue method, the service provider has b consumer threads, if the queue is not empty, a consumer thread dequeues a data element from the queue; (e) Service provider business processing method for the data element: (e1) After a certain data element is dequeued, record the current time as the request dequeue time outQueueTime; (e2) Process the message content in the data element, and the processing time is denoted as handleTime; (f) Service provider processing efficiency periodic summary method: Calculate the processing of each msgType every minute, and calculate the average processing time avgHandleTime; the formula is as follows: avgHandleTime(msgType)=handleTime(msgType) / count(msgType); count(msgType) is the number of data elements of the same msgType processed within 1 minute; ΣhandleTime(msgType) is the sum of the total processing time of the same msgType data elements within 1 minute, and the average processing time avgHandleTime(msgType) is obtained; The average processing time of msgType m at time t is simplified as f(m, t); Similarly, the average processing time of msgType m at time t-1 is simplified as f(m, t-1); S3, the method for detecting queue accumulation: If the following two conditions occur, it is determined that the queue is accumulated: (1) The service provider receives a new message, but the queue length has reached a at this time, so it cannot continue to enter the queue, and the queue is full and accumulated, at this time the request is discarded, and the first accumulation time r is recorded, at this time the last time is not accumulated, if the last time is accumulated, it has been detected at the last time; (2) After dequeuing from the queue, calculate outQueueTime, check outQueueTime-enQueueTime>timeout, so the request caller no longer waits for its response result, and there is no need to continue processing, so the request is discarded; If c consecutive requests need to be discarded, it is determined that the queue has been accumulated, and the first accumulation time r is recorded, at this time the last time is not accumulated, if the last time is accumulated, it has been detected at the last time; S4, the recovery method of queue accumulation: Because the service provider has too many messages or the processing speed is lower than the service caller's sending speed, and the service caller perceives the timeout and repeatedly sends requests to the service provider; if the queue cannot be recovered in time, chain conduction will occur, and finally all services in the system will be at risk; the specific recovery method is: (1) System recovery, the historical message caller no longer waits, and there is no need to process the invalid message, first clear all messages in the queue for system recovery; (2) Automatic expansion, increase the consumption capacity of the service provider, specifically: if cpu<30%, increase the number of threads by 100%, if 30%<cpu<50%, increase the number of threads by 30%; S5, the tracing method of queue accumulation: In the case of queue accumulation, it is necessary to find which messages cause the accumulation; the specific method is: (1) Trace the message type with too long average processing time; For all message types msgType, m∈Q, if f(m, r)>α*f(m, r-1), m is put into set E, E is the set of long processing time accumulation message types, α≥2, α is a preset threshold; (2) Trace the message type with too fast arrival rate; For all message types msgType, m∈Q, if g(m, r)>β*g(m, r-1), m is put into set H, H is the set of fast arrival rate accumulation message types, β≥2, β is a preset threshold; Finally, at the accumulation time r, the message type sets E and H that cause the accumulation are found.

Images