Delay message queue scheduling method and system based on dynamic time window and storage medium

Abstract

The invention provides a delay message queue scheduling method and system based on a dynamic time window and a storage medium, which are used for a payment transaction system, and the method comprises the following steps: dividing delay messages in a delay message queue into N scheduling queues based on delay duration and delay message priority, dynamically dividing Mi time windows for the ith scheduling queue in the N scheduling queues based on the delay duration of the delay message, and constructing N linear storage structures as first-level storage areas to store the Mi time windows in the corresponding N scheduling queues, constructing a dynamic annular buffer area for each linear storage structure as a secondary storage area to store a delay message in a jth time window to be processed at present, generating N processing threads corresponding to N scheduling queues, and obtaining the number K of processing cores of a current processor, and executing the N processing threads in the K processing cores to process the delay messages in the N scheduling queues. The execution efficiency of the processing kernel is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing and task scheduling, and in particular to a delayed message queue scheduling method, system and storage medium based on a dynamic time window. Background Art

[0002] Delayed message queues in payment transaction systems Timely processing of delayed messages in distributed systems (order payment timeouts, scheduled tasks, etc.) plays a key role in system performance. The main existing delayed message processing methods are: (1) the time wheel algorithm, which uses a fixed time scale, resulting in waste of resources in sparse scenarios and low CPU utilization; (2) the implementation based on Redis ZSET: large serialization overhead and large cross-node synchronization delay. Therefore, an efficient scheduling and processing method for delayed message queues is urgently needed. Summary of the invention

[0003] The present invention proposes the following technical solutions to address one or more technical deficiencies in the above-mentioned prior art.

[0004] A delayed message queue scheduling method based on a dynamic time window, used in a payment transaction system, the method comprising: Processing steps: dividing the delayed messages in the delayed message queue into N scheduling queues based on the delay duration and the delayed message priority, where N≥2; The allocation step is to dynamically allocate M for the i-th scheduling queue among N scheduling queues based on the delay duration of the delayed message. i time windows, where M i ≥2, 1≤i≤N; Storage step: construct N linear storage structures as the first-level storage area to store the M corresponding to the N scheduling queues i time windows, a dynamic ring buffer is constructed for each linear storage structure as a secondary storage area to store the delayed messages in the jth time window currently to be processed, where 1≤j≤M i ; The scheduling step generates N processing threads corresponding to the N scheduling queues, obtains the number K of processing cores of the current processor, and executes the N processing threads on the K processing cores to process the delayed messages in the N scheduling queues, wherein K≥2.

[0005] Furthermore, the operation of constructing a dynamic ring buffer as a secondary storage area for each linear storage structure to store the delayed message in the j-th time window to be processed is: when j ≥ 2, when processing the delayed message in the j-1-th time window, obtain the number num of delayed messages in the j-th time window mes , based on num mesApply for a dynamic ring buffer of the corresponding size. After the delayed messages in the j-1th time window are processed, destroy the dynamic ring buffer corresponding to the j-1th time window. When j=1, obtain the number of delayed messages num in the jth time window. mes , based on num mes Apply for a dynamic ring buffer of the corresponding size.

[0006] Furthermore, the linear storage structure is an array storage structure.

[0007] Furthermore, the operation of dividing the delayed messages in the delayed message queue into N scheduling queues based on the delay duration and the delayed message priority is as follows: initializing N scheduling queues based on the number of delayed message priorities; reading each delayed message from the delayed message queue, and assigning it to the kth scheduling queue based on the delay duration and priority of the delayed message, until all delayed messages in the delayed message queue are assigned to obtain N scheduling queues, and the initial queue priority of the kth scheduling queue is k, where 1≤k≤N, and k is determined as follows: ;

[0008] Indicates the priority of the delayed message. Indicates the delay duration of the delayed message. Indicates the first time threshold.

[0009] Furthermore, the operation of the allocation step is: obtaining the delay duration of the delayed messages in the i-th scheduling queue, dividing the delayed messages with the same delay duration into a time window, and obtaining a total of M i ' time window, determine whether the number of delayed messages in each time window is greater than the second threshold, if so, split the time window into two time windows, and use the two split time windows to replace the original time window to obtain M i '' time window; then judge M i ''Whether the number of delayed messages in each time window of the time window is less than the third threshold value, if so, compare the number of delayed messages in the two time windows adjacent to the time window, and determine the time window with the smaller number of delayed messages as the target time window. If the time window is the first time window, the second time window is determined as the target time window. If the time window is the Mth time window, i '' time window, then the Mth i ''-1 time window is determined as the target time window, the delayed messages in the time window are merged into the target time window, and the time window is destroyed, and finally M is obtained. i time window.

[0010] Furthermore, the operation of the scheduling step is: the priorities of the N processing threads correspond to the initial queue priorities of the N scheduling queues, and the current state of each processing core in the K processing cores is obtained, and the current state represents the idle state of the processing core. If K≥N, N processing cores are selected from the K processing cores based on the current state and are sequentially allocated to the N processing threads, wherein the current state of the processing core allocated to the processing thread corresponding to the Nth scheduling queue is the idlest, and so on; if N K, calculate the current scheduling priority of each scheduling queue based on the total delay time of delayed messages in each scheduling queue and the initial queue priority of the scheduling queue, select the top K scheduling queues with the current scheduling priority, and assign K processing cores to the K processing threads corresponding to the top K scheduling queues with the current scheduling priority for parallel processing. Among them, the processing core assigned to the thread corresponding to the scheduling queue with the highest current scheduling priority is the most idle, and so on. For the remaining NK threads, when waiting for a processing core to be idle, they are assigned to the idle processing core based on the current scheduling priority of the NK threads. Among them, the current scheduling priority of the kth scheduling queue is The calculation method is: ; Where k represents the kth scheduling queue and the initial queue priority of the queue. Indicates the total delay duration of delayed messages in the kth scheduling queue. Indicates the median delay time of delayed messages in the kth scheduling queue.

[0011] The present invention also proposes a delayed message queue scheduling device based on a dynamic time window, which is used in a payment transaction system. The device includes: The processing unit divides the delayed messages in the delayed message queue into N scheduling queues based on the delay duration and the priority of the delayed messages, where N ≥ 2; The allocation unit dynamically allocates M for the i-th scheduling queue among N scheduling queues based on the delay duration of the delayed message i time windows, where M i ≥2, 1≤i≤N; Storage unit, build N linear storage structures as the first-level storage area to store the M corresponding to the N scheduling queues i time windows, a dynamic ring buffer is constructed for each linear storage structure as a secondary storage area to store the delayed messages in the jth time window currently to be processed, where 1≤j≤M i ; The scheduling unit generates N processing threads corresponding to the N scheduling queues, obtains the number K of processing cores of the current processor, and executes the N processing threads on the K processing cores to process the delayed messages in the N scheduling queues, wherein K≥2.

[0012] Furthermore, the operation of constructing a dynamic ring buffer as a secondary storage area for each linear storage structure to store the delayed message in the j-th time window to be processed is: when j ≥ 2, when processing the delayed message in the j-1-th time window, obtain the number num of delayed messages in the j-th time window mes , based on num mes Apply for a dynamic ring buffer of the corresponding size. After the delayed messages in the j-1th time window are processed, destroy the dynamic ring buffer corresponding to the j-1th time window. When j=1, obtain the number of delayed messages num in the jth time window. mes , based on num mes Apply for a dynamic ring buffer of the corresponding size.

[0013] Furthermore, the linear storage structure is an array storage structure.

[0014] Furthermore, the operation of dividing the delayed messages in the delayed message queue into N scheduling queues based on the delay duration and the delayed message priority is as follows: initializing N scheduling queues based on the number of delayed message priorities; reading each delayed message from the delayed message queue, and assigning it to the kth scheduling queue based on the delay duration and priority of the delayed message, until all delayed messages in the delayed message queue are assigned to obtain N scheduling queues, and the initial queue priority of the kth scheduling queue is k, where 1≤k≤N, and k is determined as follows: ; Indicates the priority of the delayed message. Indicates the delay duration of the delayed message. Indicates the first time threshold.

[0015] Furthermore, the operation of the allocation unit is: obtaining the delay duration of the delayed messages in the i-th scheduling queue, dividing the delayed messages with the same delay duration into a time window, and obtaining a total of M i ' time window, determine whether the number of delayed messages in each time window is greater than the second threshold, if so, split the time window into two time windows, and use the two split time windows to replace the original time window to obtain M i '' time window; then judge M i''Whether the number of delayed messages in each time window of the time window is less than the third threshold value, if so, compare the number of delayed messages in the two time windows adjacent to the time window, and determine the time window with the smaller number of delayed messages as the target time window. If the time window is the first time window, the second time window is determined as the target time window. If the time window is the Mth time window, i '' time window, then the Mth i ''-1 time window is determined as the target time window, the delayed messages in the time window are merged into the target time window, and the time window is destroyed, and finally M is obtained. i time window.

[0016] Furthermore, the operation of the scheduling step is: the priorities of the N processing threads correspond to the initial queue priorities of the N scheduling queues, and the current state of each processing core in the K processing cores is obtained, and the current state represents the idle state of the processing core. If K≥N, N processing cores are selected from the K processing cores based on the current state and are sequentially allocated to the N processing threads, wherein the current state of the processing core allocated to the processing thread corresponding to the Nth scheduling queue is the idlest, and so on; if N K, calculate the current scheduling priority of each scheduling queue based on the total delay time of delayed messages in each scheduling queue and the initial queue priority of the scheduling queue, select the top K scheduling queues with the current scheduling priority, and assign K processing cores to the K processing threads corresponding to the top K scheduling queues with the current scheduling priority for parallel processing. Among them, the processing core assigned to the thread corresponding to the scheduling queue with the highest current scheduling priority is the most idle, and so on. For the remaining NK threads, when waiting for a processing core to be idle, they are assigned to the idle processing core based on the current scheduling priority of the NK threads. Among them, the current scheduling priority of the kth scheduling queue is The calculation method is: ; Where k represents the kth scheduling queue and the initial queue priority of the queue. Indicates the total delay duration of delayed messages in the kth scheduling queue. Indicates the median delay time of delayed messages in the kth scheduling queue.

[0017] The present invention further provides a computer-readable storage medium, wherein the storage medium stores computer program code, and when the computer program code is executed by a computer, any of the above methods is executed.

[0018] The technical effect of the present invention is: a method, system and storage medium for scheduling delayed message queues based on a dynamic time window of the present invention, the method comprising: a processing step S101, dividing delayed messages in a delayed message queue into N scheduling queues based on delay duration and delay message priority, wherein N ≥ 2; an allocation step S102, dynamically dividing M scheduling queues into the i-th scheduling queue in the N scheduling queues based on the delay duration of the delayed message i time windows, where M i ≥2, 1≤i≤N; Storage step S103, constructing N linear storage structures as the first-level storage area to store the M corresponding to the N scheduling queues i time windows, a dynamic ring buffer is constructed for each linear storage structure as a secondary storage area to store the delayed messages in the jth time window currently to be processed, where 1≤j≤M i ; Scheduling step S104, generating N processing threads corresponding to N scheduling queues, obtaining the number of processing cores K of the current processor (i.e., the number of available processing cores), executing the N processing threads on the K processing cores to process the delayed messages in the N scheduling queues, wherein K≥2. The present invention creatively proposes to first divide the delayed message queue into multiple scheduling queues based on the delay duration and the delayed message priority. The initial scheduling priority of each scheduling queue is the same. Since both the delay duration and the delayed message priority of the delayed message are considered when allocating the delayed message to the corresponding scheduling queue, each delayed message can be efficiently processed. The present invention adopts a method of dynamically dividing the time window to ensure that the messages in each time window are neither too many nor too few, so that it can be well combined with the secondary memory structure proposed by the present invention. N linear storage structures are used as the primary storage area to store the corresponding M messages in the N scheduling queues. i The linear storage structure improves the query efficiency during window execution. The dynamic ring buffer is used as the secondary storage area to store the delayed messages in the jth time window to be processed, which can efficiently utilize the memory space and reduce the memory occupancy rate. In order to minimize the time wasted on locking and unlocking operations, the thread is matched with the processing kernel to reduce the lock operation, improve the execution efficiency of the processing kernel, improve the efficiency of delayed message processing, and greatly reduce the memory occupancy. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Other features, objects and advantages of the present application will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings.

[0020] Figure 1 The present invention is a flowchart of a delayed message queue scheduling method based on a dynamic time window according to an embodiment of the present invention.

[0021] Figure 2 It is a structural diagram of a delayed message queue scheduling device based on a dynamic time window according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0023] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0024] Figure 1 A delayed message queue scheduling method based on a dynamic time window of the present invention is shown, which is used in a payment transaction system. The method includes: Processing step S101, dividing the delayed messages in the delayed message queue into N scheduling queues based on the delay duration and the delayed message priority, where N≥2; In step S102, the i-th scheduling queue in the N scheduling queues is dynamically allocated M based on the delay time of the delayed message. i time windows, where M i ≥2, 1≤i≤N; Storage step S103, constructing N linear storage structures as the first-level storage area to store the M corresponding to the N scheduling queues i time windows, a dynamic ring buffer is constructed for each linear storage structure as a secondary storage area to store the delayed messages in the jth time window currently to be processed, where 1≤j≤M i ; Scheduling step S104, generating N processing threads corresponding to the N scheduling queues, obtaining the number of processing cores K of the current processor (ie, the number of available processing cores), executing the N processing threads on the K processing cores to process the delayed messages in the N scheduling queues, where K≥2.

[0025] In the present invention, in order to process delayed messages as quickly as possible, it is creatively proposed to first divide the delayed message queue into multiple scheduling queues based on the delay duration and the priority of the delayed message, and the initial scheduling priority of each scheduling queue is the same. Since both the delay duration and the priority of the delayed message are considered when allocating the delayed message to the corresponding scheduling queue, each delayed message can be efficiently processed. Furthermore, in order to improve the processing efficiency of delayed messages, the i-th scheduling queue among the N scheduling queues is dynamically divided into M based on the delay duration of the delayed message.i The present invention adopts a method of dynamically dividing the time window to ensure that the messages in each time window are neither too many nor too few, so that the secondary memory structure proposed by the present invention can be well combined, and N linear storage structures are used as the primary storage area to store the M messages in the corresponding N scheduling queues. i time windows, the linear storage structure improves the query efficiency during window execution, and the use of a dynamic ring buffer as a secondary storage area to store the delayed messages in the jth time window to be processed can efficiently utilize the memory space and reduce the memory occupancy rate. In order to minimize the time wasted on locking and unlocking operations, the lock operation is reduced by corresponding threads to processing cores, thereby improving the execution efficiency of the processing cores, that is, by generating N processing threads corresponding to N scheduling queues, and obtaining the number of processing cores K of the current processor, the N processing threads are executed on the K processing cores to process the delayed messages in the N scheduling queues. Through this scheduling method of the present invention, the efficiency of delayed message processing is improved, and the memory occupancy is greatly reduced. This is an important inventive concept of the present invention.

[0026] In one embodiment, the operation of constructing a dynamic ring buffer as a secondary storage area for each linear storage structure to store the delayed message in the j-th time window to be processed is: when j ≥ 2, when processing the delayed message in the j-1-th time window, obtain the number num of delayed messages in the j-th time window mes , based on num mes Apply for a dynamic ring buffer of the corresponding size. After the delayed messages in the j-1th time window are processed, destroy the dynamic ring buffer corresponding to the j-1th time window. When j=1, obtain the number of delayed messages num in the jth time window. mes , based on num mes Apply for a dynamic ring buffer of the corresponding size.

[0027] In the present invention, a method of pre-building a dynamic ring buffer is adopted. After the message processing in the j-1th (j≥2)th window is completed, the delayed message of the jth window has been loaded into the dynamic ring buffer, and the dynamic ring buffer corresponding to the j-1 time window is destroyed, that is, based on the number of delayed messages to be processed in each time window, a corresponding memory size is applied as a dynamic ring buffer, and the application is made in advance, thereby avoiding the processing core waiting for the loading of delayed messages, improving the overall efficiency of the system, and improving memory utilization. Through this secondary storage method, when processing delayed messages, the memory fragmentation rate is reduced by 72%, and the processing core pause time is shortened to within 3ms, which greatly improves the system performance.

[0028] In one embodiment, the linear storage structure is an array storage structure, and the query time of the array is O(n), which greatly improves the search speed of the time window.

[0029] In one embodiment, the operation of dividing the delayed messages in the delayed message queue into N scheduling queues based on the delay duration and the delayed message priority is as follows: initializing N scheduling queues based on the number of delayed message priorities; reading each delayed message from the delayed message queue, and assigning it to the kth scheduling queue based on the delay duration and priority of the delayed message, until all delayed messages in the delayed message queue are assigned to obtain N scheduling queues, and the initial queue priority of the kth scheduling queue is k, and the larger the k value, the higher the priority, wherein 1≤k≤N, and k is determined as follows: ;

[0030] Indicates the priority of the delayed message. Indicates the delay duration of the delayed message. Indicates the first time threshold.

[0031] An important link in realizing the present invention is how to divide the delayed messages in the delay queue into the corresponding scheduling queue. In the prior art, some are simply divided according to the delay duration, and some are divided according to the priority of the delayed message. Simply dividing the scheduling queue based on the delay duration or priority may result in low-priority delayed messages not being processed for a long time, or the number of delayed messages in each scheduling queue is quite different. In order to overcome these defects, the present invention proposes to initialize N scheduling queues based on the number of delayed message priorities, that is, at the beginning, each scheduling queue is empty, and a message is read from the delayed message queue, and the queue to which it is assigned is calculated based on the delay duration and its priority, so that the number of messages in each scheduling queue can be basically guaranteed to be consistent, and each scheduling queue is given a corresponding initial queue priority, that is, no matter what the original priority of the delayed message in the scheduling queue is, the subsequent processing is based on the initial queue priority of the queue, which improves the processing efficiency and avoids the defect that low-priority delayed messages are difficult to be processed. In addition, the present invention proposes a specific method for assigning delayed messages to scheduling queues, which improves the speed of delayed message division, which is another important inventive point of the present invention.

[0032] In one embodiment, the operation of the allocation step S102 is: obtaining the delay time of the delayed messages in the i-th scheduling queue, dividing the delayed messages with the same delay time into a time window, and obtaining a total of M i' time window, determine whether the number of delayed messages in each time window is greater than the second threshold, if so, split the time window into two time windows, and use the two split time windows to replace the original time window to obtain M i '' time window; then judge M i ''Whether the number of delayed messages in each time window of the time window is less than the third threshold value, if so, compare the number of delayed messages in the two time windows adjacent to the time window, and determine the time window with the smaller number of delayed messages as the target time window. If the time window is the first time window, the second time window is determined as the target time window. If the time window is the Mth time window, i '' time window, then the Mth i ''-1 time window is determined as the target time window, the delayed messages in the time window are merged into the target time window, and the time window is destroyed, and finally M is obtained. i time window.

[0033] Another important inventive concept of the present invention is to propose a method for dynamically dividing the scheduling queue into M i In order to improve the delayed message processing efficiency, the initial time window with too many delayed messages is split into two in the present invention. Of course, it can also be split into multiple times, but the time cost of splitting into multiple maintenance queues is high, so it is generally split into two, and then the number of messages in all time windows is scanned. If the number of messages in a time window is too small, the delayed messages therein are merged into the adjacent time window with a smaller number of delayed messages, and the original time window is deleted. In this way, the processing resources of the processing core can be fully utilized within one time window to avoid wasting processing resources. This is another important inventive point of the present invention.

[0034] In one embodiment, the operation of the scheduling step S104 is as follows: the priorities of the N processing threads correspond to the initial queue priorities of the N scheduling queues, and the current state of each processing core in the K processing cores is obtained, and the current state indicates the idle state of the processing core. If K≥N, N processing cores are selected from the K processing cores based on the current state and are sequentially allocated to the N processing threads, wherein the current state of the processing core allocated to the processing thread corresponding to the Nth scheduling queue is the idlest, and so on; if N K, calculate the current scheduling priority of each scheduling queue based on the total delay time of delayed messages in each scheduling queue and the initial queue priority of the scheduling queue, select the top K scheduling queues with the current scheduling priority, and assign K processing cores to the K processing threads corresponding to the top K scheduling queues with the current scheduling priority for parallel processing. Among them, the processing core assigned to the thread corresponding to the scheduling queue with the highest current scheduling priority is the most idle, and so on. For the remaining NK threads, when waiting for a processing core to be idle, they are assigned to the idle processing core based on the current scheduling priority of the NK threads. Among them, the current scheduling priority of the kth scheduling queue is The calculation method is: ;

[0035] Where k represents the kth scheduling queue and the initial queue priority of the queue. Indicates the total delay duration of delayed messages in the kth scheduling queue. Indicates the median delay time of delayed messages in the kth scheduling queue.

[0036] In the present invention, in order to solve the problem of poor system performance caused by frequent locks (locking and unlocking) caused by frequent switching of processing cores, the available processing cores are bound to the corresponding processing threads, that is, one processing core is responsible for processing the threads of a scheduling queue, and the idlest processing core processes the scheduling queue with the highest priority. It is proposed that when the number of processing cores is less than the scheduling queue, the current scheduling priority of each scheduling queue is calculated based on the total delay duration of delayed messages in each scheduling queue and the initial queue priority of the scheduling queue, and the threads of K scheduling queues are allocated to K processing cores for processing based on the current scheduling priority. The threads of the remaining NK scheduling queues wait for processing core resources in a polling manner, that is, this is a mixed scheduling method, and a specific method for calculating the current scheduling priority of each scheduling queue is proposed. Through this scheduling method, the processing speed of delayed messages is greatly improved, which is another important inventive concept of the present invention.

[0037] The method of the present invention is specially designed for payment transaction systems. Through actual tests in a certain e-commerce order system, an average of 230 million order payment timeout transactions are processed every day. The method of the present invention can reduce the fault recovery time from minutes to sub-seconds.

[0038] In a simulation test of 100 million delayed messages (time distribution conforms to the Poisson process) in a simulated payment transaction system, the performance is compared with the existing technology as follows:

[0039] It can be seen from the above table that compared with the existing methods, the present invention has obvious improvements in indicators such as throughput, delay time, and CPU utilization. It can be seen that the technical effect of the present invention is significantly better than the existing technology.

[0040] Figure 2 The invention shows a delayed message queue scheduling device based on a dynamic time window for a payment transaction system, comprising: The processing unit 201 divides the delayed messages in the delayed message queue into N scheduling queues based on the delay duration and the delayed message priority, where N≥2; The allocation unit 202 dynamically allocates M scheduling queues to the i-th scheduling queue in the N scheduling queues based on the delay duration of the delayed message. i time windows, where M i ≥2, 1≤i≤N; The storage unit 203 constructs N linear storage structures as a primary storage area to store the M corresponding to the N scheduling queues. i time windows, a dynamic ring buffer is constructed for each linear storage structure as a secondary storage area to store the delayed messages in the jth time window currently to be processed, where 1≤j≤M i ; The scheduling unit 204 generates N processing threads corresponding to the N scheduling queues, obtains the number of processing cores K of the current processor (ie, the number of available processing cores), and executes the N processing threads on the K processing cores to process the delayed messages in the N scheduling queues, where K≥2.

[0041] In the present invention, in order to process delayed messages as quickly as possible, it is creatively proposed to first divide the delayed message queue into multiple scheduling queues based on the delay duration and the priority of the delayed message, and the initial scheduling priority of each scheduling queue is the same. Since both the delay duration and the priority of the delayed message are considered when allocating the delayed message to the corresponding scheduling queue, each delayed message can be efficiently processed. Furthermore, in order to improve the processing efficiency of delayed messages, the i-th scheduling queue among the N scheduling queues is dynamically divided into M based on the delay duration of the delayed message. i The present invention adopts a method of dynamically dividing the time window to ensure that the messages in each time window are neither too many nor too few, so that the secondary memory structure proposed by the present invention can be well combined, and N linear storage structures are used as the primary storage area to store the M messages in the corresponding N scheduling queues. itime windows, the linear storage structure improves the query efficiency during window execution, and the use of a dynamic ring buffer as a secondary storage area to store the delayed messages in the jth time window to be processed can efficiently utilize the memory space and reduce the memory occupancy rate. In order to minimize the time wasted on locking and unlocking operations, the lock operation is reduced by corresponding threads to processing cores, thereby improving the execution efficiency of the processing cores, that is, by generating N processing threads corresponding to N scheduling queues, and obtaining the number of processing cores K of the current processor, the N processing threads are executed on the K processing cores to process the delayed messages in the N scheduling queues. Through this scheduling method of the present invention, the efficiency of delayed message processing is improved, and the memory occupancy is greatly reduced. This is an important inventive concept of the present invention.

[0042] In one embodiment, the operation of constructing a dynamic ring buffer as a secondary storage area for each linear storage structure to store the delayed message in the j-th time window to be processed is: when j ≥ 2, when processing the delayed message in the j-1-th time window, obtain the number num of delayed messages in the j-th time window mes , based on num mes Apply for a dynamic ring buffer of the corresponding size. After the delayed messages in the j-1th time window are processed, destroy the dynamic ring buffer corresponding to the j-1th time window. When j=1, obtain the number of delayed messages num in the jth time window. mes , based on num mes Apply for a dynamic ring buffer of the corresponding size.

[0043] In the present invention, a method of pre-building a dynamic ring buffer is adopted. After the message processing in the j-1th (j≥2)th window is completed, the delayed message of the jth window has been loaded into the dynamic ring buffer, and the dynamic ring buffer corresponding to the j-1 time window is destroyed, that is, based on the number of delayed messages to be processed in each time window, a corresponding memory size is applied as a dynamic ring buffer, and the application is made in advance, thereby avoiding the processing core waiting for the loading of delayed messages, improving the overall efficiency of the system, and improving memory utilization. Through this secondary storage method, when processing delayed messages, the memory fragmentation rate is reduced by 72%, and the processing core pause time is shortened to within 3ms, which greatly improves the system performance.

[0044] In one embodiment, the linear storage structure is an array storage structure, and the query time of the array is O(n), which greatly improves the search speed of the time window.

[0045] In one embodiment, the operation of dividing the delayed messages in the delayed message queue into N scheduling queues based on the delay duration and the delayed message priority is as follows: initializing N scheduling queues based on the number of delayed message priorities; reading each delayed message from the delayed message queue, and assigning it to the kth scheduling queue based on the delay duration and priority of the delayed message, until all delayed messages in the delayed message queue are assigned to obtain N scheduling queues, and the initial queue priority of the kth scheduling queue is k, and the larger the k value, the higher the priority, wherein 1≤k≤N, and k is determined as follows: ; Indicates the priority of the delayed message. Indicates the delay duration of the delayed message. Indicates the first time threshold.

[0046] An important link in realizing the present invention is how to divide the delayed messages in the delay queue into the corresponding scheduling queue. In the prior art, some are simply divided according to the delay duration, and some are divided according to the priority of the delayed message. Simply dividing the scheduling queue based on the delay duration or priority may result in low-priority delayed messages not being processed for a long time, or the number of delayed messages in each scheduling queue is quite different. In order to overcome these defects, the present invention proposes to initialize N scheduling queues based on the number of delayed message priorities, that is, at the beginning, each scheduling queue is empty, and a message is read from the delayed message queue, and the queue to which it is assigned is calculated based on the delay duration and its priority, so that the number of messages in each scheduling queue can be basically guaranteed to be consistent, and each scheduling queue is given a corresponding initial queue priority, that is, no matter what the original priority of the delayed message in the scheduling queue is, the subsequent processing is based on the initial queue priority of the queue, which improves the processing efficiency and avoids the defect that low-priority delayed messages are difficult to be processed. In addition, the present invention proposes a specific method for assigning delayed messages to scheduling queues, which improves the speed of delayed message division, which is another important invention point of the present invention.

[0047] In one embodiment, the operation of the allocation unit 202 is: obtaining the delay duration of the delayed messages in the i-th scheduling queue, dividing the delayed messages with the same delay duration into a time window, and obtaining a total of M i ' time window, determine whether the number of delayed messages in each time window is greater than the second threshold, if so, split the time window into two time windows, and use the two split time windows to replace the original time window to obtain M i '' time window; then judge M i''Whether the number of delayed messages in each time window of the time window is less than the third threshold value, if so, compare the number of delayed messages in the two time windows adjacent to the time window, and determine the time window with the smaller number of delayed messages as the target time window. If the time window is the first time window, the second time window is determined as the target time window. If the time window is the Mth time window, i '' time window, then the Mth i ''-1 time window is determined as the target time window, the delayed messages in the time window are merged into the target time window, and the time window is destroyed, and finally M is obtained. i time window.

[0048] Another important inventive concept of the present invention is to propose a method for dynamically dividing the scheduling queue into M i In order to improve the delayed message processing efficiency, the initial time window with too many delayed messages is split into two in the present invention. Of course, it can also be split into multiple, but the time cost of splitting into multiple maintenance queues is high, so it is generally split into two, and then the number of messages in all time windows is scanned. If the number of messages in a time window is too small, the delayed messages therein are merged into the adjacent time window with a smaller number of delayed messages, and the original time window is deleted. In this way, the processing resources of the processing core can be fully utilized within one time window to avoid wasting processing resources. This is another important invention point of the present invention.

[0049] In one embodiment, the operation of the scheduling unit 204 is as follows: the priorities of the N processing threads correspond to the initial queue priorities of the N scheduling queues, and the current state of each processing core in the K processing cores is obtained, and the current state indicates the idle state of the processing core. If K≥N, N processing cores are selected from the K processing cores based on the current state and are sequentially allocated to the N processing threads, wherein the current state of the processing core allocated to the processing thread corresponding to the Nth scheduling queue is the idlest, and so on; if N K, calculate the current scheduling priority of each scheduling queue based on the total delay time of delayed messages in each scheduling queue and the initial queue priority of the scheduling queue, select the top K scheduling queues with the current scheduling priority, and assign K processing cores to the K processing threads corresponding to the top K scheduling queues with the current scheduling priority for parallel processing. Among them, the processing core assigned to the thread corresponding to the scheduling queue with the highest current scheduling priority is the most idle, and so on. For the remaining NK threads, when waiting for a processing core to be idle, they are assigned to the idle processing core based on the current scheduling priority of the NK threads. Among them, the current scheduling priority of the kth scheduling queue is The calculation method is: ; Where k represents the kth scheduling queue and the initial queue priority of the queue. Indicates the total delay duration of delayed messages in the kth scheduling queue. Indicates the median delay time of delayed messages in the kth scheduling queue.

[0050] In the present invention, in order to solve the problem of poor system performance caused by frequent locks (locking and unlocking) caused by frequent switching of processing cores, the available processing cores are bound to the corresponding processing threads, that is, one processing core is responsible for processing the threads of a scheduling queue, and the idlest processing core processes the scheduling queue with the highest priority. It is proposed that when the number of processing cores is less than the scheduling queue, the current scheduling priority of each scheduling queue is calculated based on the total delay duration of delayed messages in each scheduling queue and the initial queue priority of the scheduling queue, and the threads of K scheduling queues are allocated to K processing cores for processing based on the current scheduling priority. The threads of the remaining NK scheduling queues wait for processing core resources in a polling manner, that is, this is a mixed scheduling method, and a specific method for calculating the current scheduling priority of each scheduling queue is proposed. Through this scheduling method, the processing speed of delayed messages is greatly improved, which is another important inventive concept of the present invention.

[0051] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0052] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a storage medium, such as ROM / RAM, a disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the devices described in each embodiment of the present application or some parts of the embodiments.

[0053] Finally, it should be noted that the above embodiments are only intended to illustrate rather than limit the technical solutions of the present invention. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the present invention can still be modified or replaced by equivalents. Any modification or partial replacement that does not depart from the spirit and scope of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A delayed message queue scheduling method based on dynamic time window, used in payment transaction system, characterized in that: The method includes: Processing steps: dividing the delayed messages in the delayed message queue into N scheduling queues based on the delay duration and the delayed message priority, where N≥2; The allocation step is to dynamically allocate M for the i-th scheduling queue among N scheduling queues based on the delay duration of the delayed message. i time windows, where M i ≥2, 1≤i≤N; Storage step: construct N linear storage structures as the first-level storage area to store the M corresponding to the N scheduling queues i time windows, a dynamic ring buffer is constructed for each linear storage structure as a secondary storage area to store the delayed messages in the jth time window currently to be processed, where 1≤j≤M i ; The scheduling step generates N processing threads corresponding to the N scheduling queues, obtains the number K of processing cores of the current processor, and executes the N processing threads on the K processing cores to process the delayed messages in the N scheduling queues, wherein K≥2.

2. The method according to claim 1, characterized in that The linear storage structure is an array storage structure.

3. The method according to claim 2, characterized in that The operation of dividing the delayed messages in the delayed message queue into N scheduling queues based on the delay duration and the priority of the delayed messages is as follows: initializing the N scheduling queues based on the number of delayed message priorities; reading each delayed message from the delayed message queue, and assigning it to the kth scheduling queue based on the delay duration and priority of the delayed message, until all delayed messages in the delayed message queue are assigned to obtain N scheduling queues, and the initial queue priority of the kth scheduling queue is k, wherein 1≤k≤N, and k is determined as follows: ； Indicates the priority of the delayed message. Indicates the delay duration of the delayed message. Indicates the first time threshold.

4. The method according to claim 3, characterized in that The operation of the allocation step is: obtaining the delay time of the delayed messages in the i-th scheduling queue, dividing the delayed messages with the same delay time into a time window, and obtaining a total of M i ' time window, determine whether the number of delayed messages in each time window is greater than the second threshold, if so, split the time window into two time windows, and use the two split time windows to replace the original time window to obtain M i '' time window; then judge M i ''Whether the number of delayed messages in each time window of the time window is less than the third threshold value, if so, compare the number of delayed messages in the two time windows adjacent to the time window, and determine the time window with the smaller number of delayed messages as the target time window. If the time window is the first time window, the second time window is determined as the target time window. If the time window is the Mth time window, i '' time window, then the Mth i ''-1 time window is determined as the target time window, the delayed messages in the time window are merged into the target time window, and the time window is destroyed, and finally M is obtained. i time window.

5. The method according to claim 4, characterized in that The operation of the scheduling step is: the priorities of the N processing threads correspond to the initial queue priorities of the N scheduling queues, and the current state of each processing core in the K processing cores is obtained, and the current state indicates the idle state of the processing core. If K≥N, N processing cores are selected from the K processing cores based on the current state and are sequentially allocated to the N processing threads, wherein the current state of the processing core allocated to the processing thread corresponding to the Nth scheduling queue is the idlest, and so on; if N K, calculate the current scheduling priority of each scheduling queue based on the total delay time of delayed messages in each scheduling queue and the initial queue priority of the scheduling queue, select the top K scheduling queues with the current scheduling priority, and assign K processing cores to the K processing threads corresponding to the top K scheduling queues with the current scheduling priority for parallel processing. Among them, the processing core assigned to the thread corresponding to the scheduling queue with the highest current scheduling priority is the most idle, and so on. For the remaining NK threads, when waiting for a processing core to be idle, they are assigned to the idle processing core based on the current scheduling priority of the NK threads. Among them, the current scheduling priority of the kth scheduling queue is The calculation method is: ； Where k represents the kth scheduling queue and the initial queue priority of the queue. Indicates the total delay duration of delayed messages in the kth scheduling queue. Indicates the median delay time of delayed messages in the kth scheduling queue.

6. A delayed message queue scheduling device based on dynamic time window, used in payment transaction system, characterized in that: The device includes: The processing unit divides the delayed messages in the delayed message queue into N scheduling queues based on the delay duration and the priority of the delayed messages, where N ≥ 2; The allocation unit dynamically allocates M for the i-th scheduling queue among N scheduling queues based on the delay duration of the delayed message i time windows, where M i ≥2, 1≤i≤N; Storage unit, build N linear storage structures as the first-level storage area to store the M corresponding to the N scheduling queues i time windows, a dynamic ring buffer is constructed for each linear storage structure as a secondary storage area to store the delayed messages in the jth time window currently to be processed, where 1≤j≤M i ; The scheduling unit generates N processing threads corresponding to the N scheduling queues, obtains the number K of processing cores of the current processor, and executes the N processing threads on the K processing cores to process the delayed messages in the N scheduling queues, wherein K≥2.

7. The device according to claim 6, characterized in that The linear storage structure is an array storage structure.

8. The device according to claim 7, characterized in that The operation of dividing the delayed messages in the delayed message queue into N scheduling queues based on the delay duration and the priority of the delayed messages is as follows: initializing the N scheduling queues based on the number of delayed message priorities; reading each delayed message from the delayed message queue, and assigning it to the kth scheduling queue based on the delay duration and priority of the delayed message, until all delayed messages in the delayed message queue are assigned to obtain N scheduling queues, and the initial queue priority of the kth scheduling queue is k, wherein 1≤k≤N, and k is determined as follows: ； Indicates the priority of the delayed message. Indicates the delay duration of the delayed message. Indicates the first time threshold.

9. The device according to claim 8, characterized in that The operation of the allocation unit is: obtaining the delay time of the delayed messages in the i-th scheduling queue, dividing the delayed messages with the same delay time into a time window, and obtaining a total of M i ' time window, determine whether the number of delayed messages in each time window is greater than the second threshold, if so, split the time window into two time windows, and use the two split time windows to replace the original time window to obtain M i '' time window; then judge M i ''Whether the number of delayed messages in each time window of the time window is less than the third threshold value, if so, compare the number of delayed messages in the two time windows adjacent to the time window, and determine the time window with the smaller number of delayed messages as the target time window. If the time window is the first time window, the second time window is determined as the target time window. If the time window is the Mth time window, i '' time window, then the Mth i ''-1 time window is determined as the target time window, the delayed messages in the time window are merged into the target time window, and the time window is destroyed, and finally M is obtained. i time window.

10. A computer-readable storage medium, wherein a computer program code is stored on the storage medium, and when the computer program code is executed by a computer, the method according to any one of claims 1 to 5 is executed.

Images