Task processing method and device

Abstract

Embodiments of the present specification provide a task processing method and device, wherein the task processing method comprises: in the process of task scheduling, a distributed scheduling system first reads the task processing time length of each sampling point according to the task scheduling frequency of the distributed process, and calculates the task grabbing parameter by combining the task processing time length, the number of key threads in the thread pool configured by the task processing module of the distributed process, and the task scheduling frequency; secondly, the tasks obtained by task grabbing according to the task grabbing parameter are scheduled to the thread pool for task processing, so as to calculate the task grabbing parameter before task grabbing, and realize dynamic adjustment of task grabbing and task scheduling.

Description

Technical Field

[0001] This document relates to the field of data processing technology, and in particular to a task processing method and apparatus. Background Technology

[0002] With the continuous development and promotion of Internet technology, the application scope of online services based on Internet technology is becoming wider and wider, covering many service fields. As users' needs for online services become more and more diversified, the complexity of online services is also increasing. Distributed architecture provides a feasible means for efficient processing of online services. Distributed architecture breaks down a function into multiple independent modules and deploys them on different nodes. However, with the further increase in user needs, service providers also face certain challenges in the application of distributed architecture. Summary of the Invention

[0003] This specification provides one or more embodiments of a task processing method applied to a distributed scheduling system. The method includes: reading the task processing duration of each sampling point according to the task scheduling frequency of the distributed process; calculating task capture parameters based on the task processing duration, the number of key threads in the thread pool configured in the task processing module of the distributed process, and the task scheduling frequency; capturing tasks from the unprocessed task set of the distributed process according to the task capture parameters, and scheduling the captured tasks to the thread pool for task processing. The task processing duration is obtained by the task processing module collecting the task duration of the target tasks corresponding to each sampling point.

[0004] This specification provides one or more embodiments of a task processing device that operates in a distributed scheduling system. The device includes: a duration reading module configured to read the task processing duration of each sampling point according to the task scheduling frequency of the distributed process; a parameter calculation module configured to calculate task capture parameters based on the task processing duration, the number of key threads in the thread pool configured in the task processing module of the distributed process, and the task scheduling frequency; and a task scheduling module configured to capture tasks from the set of tasks to be processed in the distributed process according to the task capture parameters and schedule the captured tasks to the thread pool for task processing. The task processing duration is obtained by the task processing module collecting the task duration of the target tasks corresponding to each sampling point.

[0005] This specification provides one or more embodiments of a task processing device, including: a processor; and a memory configured to store computer-executable instructions, which, when executed, cause the processor to: read the task processing duration of each sampling point according to the task scheduling frequency of a distributed process; calculate task capture parameters based on the task processing duration, the number of key threads in the thread pool configured in the task processing module of the distributed process, and the task scheduling frequency; capture tasks from the set of tasks to be processed in the distributed process according to the task capture parameters, and schedule the captured tasks to the thread pool for task processing. The task processing duration is obtained by the task processing module collecting the task duration of the target tasks corresponding to each sampling point.

[0006] This specification provides one or more embodiments of a computer-readable storage medium for storing computer-executable instructions, which, when executed, perform the following steps: reading the task processing duration of each sampling point according to the task scheduling frequency of a distributed process; calculating task capture parameters based on the task processing duration, the number of key threads in the thread pool configured in the task processing module of the distributed process, and the task scheduling frequency; capturing tasks from the set of tasks to be processed in the distributed process according to the task capture parameters, and scheduling the captured tasks to the thread pool for task processing. The task processing duration is obtained by the task processing module collecting the task duration of the target task corresponding to each sampling point. Attached Figure Description

[0007] To more clearly illustrate the technical solutions in one or more embodiments of this specification or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0008] Figure 1 A schematic diagram of an implementation environment for a task processing method provided in one or more embodiments of this specification;

[0009] Figure 2 A flowchart illustrating a task processing method provided in one or more embodiments of this specification;

[0010] Figure 3 A schematic diagram of a ring-shaped memory cell provided for one or more embodiments of this specification;

[0011] Figure 4A flowchart illustrating a task processing method for a distributed scenario provided in one or more embodiments of this specification;

[0012] Figure 5 A schematic diagram of one or more embodiments of a task processing device provided in this specification;

[0013] Figure 6 This is a schematic diagram of the structure of a task processing device provided for one or more embodiments of this specification. Detailed Implementation

[0014] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this document.

[0015] The task processing method provided in one or more embodiments of this specification is applicable to a distributed process implementation environment. (Refer to...) Figure 1 The implementation environment includes at least:

[0016] A distributed scheduling system 101; in addition, the implementation environment may also include a distributed process 102;

[0017] Among them, there can be one or more distributed processes, distributed process 102 can be any one of the multiple distributed processes, distributed scheduling system can be one or more, distributed scheduling system 101 can be the distributed scheduling system corresponding to distributed process 102 in multiple distributed scheduling systems, and each distributed process can have its own corresponding distributed scheduling system.

[0018] The distributed scheduling system 101 includes a task detection module 103, a task capture module 104, and a task processing module 105. The distributed process 102 and the distributed scheduling system 101 can be deployed on a server, which can be one or more servers, a server cluster composed of several servers, or a cloud server of a cloud computing platform. The distributed scheduling system 101 is used to calculate task capture parameters based on the task processing time of each sampling point, the number of key threads in the thread pool, and the task scheduling frequency, capture tasks from the set of tasks to be processed according to the task capture parameters, and schedule the captured tasks to the thread pool for task processing.

[0019] In addition, the implementation environment may also include a task triggering system 106, which is used to generate task scheduling messages according to the task scheduling frequency and send them to the task capture module 104.

[0020] In this implementation environment, the distributed scheduling system 101 reads the task processing time of each sampling point according to the task scheduling frequency of the distributed process 102. Combining the task processing time of each sampling point, the number of key threads in the thread pool configured by the task processing module 105 of the distributed process 102, and the task scheduling frequency, it calculates the task capture parameters to obtain the task capture parameters. According to the task capture parameters, it captures tasks from the set of tasks to be processed in the distributed process 102 and schedules the captured tasks to the thread pool for task processing. In this way, the task capture parameters are calculated before task capture, so as to realize the dynamic adjustment of task capture and task scheduling.

[0021] The task processing time for each sampling point is obtained by the task processing module 105 collecting the task duration of the target task corresponding to each sampling point.

[0022] One or more embodiments of a task processing method provided in this specification are as follows:

[0023] Reference Figure 2 The task processing method provided in this embodiment can be applied to a distributed scheduling system, specifically including steps S202 to S206.

[0024] Step S202: Read the task processing time of each sampling point according to the task scheduling frequency of the distributed process.

[0025] The distributed process described in this embodiment refers to a process that performs distributed processing on a target service. The target service can be a service in any domain, such as a credit service, resource service, government service, payment service, shopping service, travel service, or food delivery service. The distributed process can be multiple processes, and each process can be used to process one or more service nodes of the target service. The service node refers to an independent branch service, service function, or service unit obtained by splitting the target service. For example, if the target service is a credit service, the service nodes of the credit service include file download and upload nodes, data parsing nodes, and contract signing nodes. Each process in the distributed process can connect to one or more service nodes of the target service. The task scheduling frequency of the distributed process includes the scheduling frequency of task scheduling for the distributed process, which can be the task scheduling frequency.

[0026] Each sampling point refers to a sampling point used by the task processing module of the distributed process to collect task processing time during task processing; each sampling point includes time sampling points and / or task sampling points; the task processing time of each sampling point refers to the processing time of the corresponding task at each sampling point; optionally, each sampling point is configured according to a preset time interval or the task order of the task queue of the thread pool. For example, the time interval between adjacent sampling points is x seconds, which means that each sampling point is configured according to the preset time interval x seconds. Another example is setting the first sampling point as the first task in the task queue of the thread pool, the second sampling point as the second task in the task queue, and so on until the last sampling point is set; the number of sampling points for each sampling point can be a preset number, which can be determined based on the number of storage slots in the circular storage unit. Specifically, the number of sampling points for each sampling point can be equal to the number of storage slots in the circular storage unit; the circular storage unit can be replaced with a storage unit, for example... Figure 3 The circular storage unit shown, RT1 to RT7 to RTn, can represent the various storage slots of the circular storage unit. The storage slots are used to store the task processing time.

[0027] In specific implementation, the task processing time of each sampling point is read. Specifically, the task processing time of each sampling point can be read according to the task scheduling frequency of the distributed process. More specifically, the task detection module of the distributed process can read the task processing time of each sampling point according to the task scheduling frequency of the distributed process. In this embodiment, the task processing can be a batch processing scenario.

[0028] Optionally, the task processing duration is obtained by the task processing module collecting the task duration of the target task corresponding to each sampling point. The target task corresponding to any sampling point may include tasks in the thread pool of the distributed process that are being executed and have not been marked for collection at the sampling time corresponding to any sampling point. The collection mark may include a start timestamp based on the execution start time of the task. Specifically, after the sampling time corresponding to each sampling point expires, the task processing module of the distributed process may determine the tasks in the thread pool configured by the task processing module of the distributed process that are being executed and have not been marked for collection as target tasks. The start timestamp of the target task may be marked based on the execution start time of the target task, and the termination timestamp of the target task may be marked based on the execution termination time of the target task after the termination of the target task is detected. Furthermore, the task processing duration of the target task may be calculated based on the start timestamp and termination timestamp of the target task. The sampling time corresponding to any sampling point may be determined based on a preset time interval or the execution start time of each task in the task queue of the thread pool. For example, if the preset time interval is y seconds, a sampling point is reached every y seconds, or a sampling point is reached whenever a task in the task queue starts execution.

[0029] In practical applications, the number of target tasks corresponding to any sampling point may be one or more, and the number of target tasks corresponding to any sampling point is not fixed. To facilitate the subsequent calculation of task capture parameters based on task processing time, the same amount of task processing time can be set for target tasks with different numbers of tasks. Optionally, the task processing time of any sampling point is calculated based on the number of target tasks corresponding to that sampling point. Specifically, in one optional implementation of this embodiment, the following operations are performed during the time calculation based on the number of target tasks corresponding to any sampling point:

[0030] If the number of tasks corresponding to any sampling point is equal to 1, the task processing time of the target task corresponding to any sampling point is taken as the task processing time of any sampling point.

[0031] If the number of tasks is greater than 1, the task processing time of any sampling point is calculated based on the task processing time of the target task corresponding to any sampling point.

[0032] The target task corresponding to any sampling point may include tasks that are being executed by the thread pool of the distributed process at the sampling time corresponding to any sampling point and have not been marked for collection; the task processing time of the target task can be calculated based on the start timestamp and end timestamp of the target task.

[0033] Specifically, in the process of calculating the task processing time of any sampling point based on the task processing time of the target task corresponding to any sampling point, the task processing time of any sampling point can be obtained by averaging the task processing times of the target task corresponding to any sampling point; or the task processing time of the target task corresponding to any sampling point can be calculated by weighting the task processing time to obtain the weighted task processing time, and the task processing time of any sampling point can be obtained by averaging the weighted task processing time; wherein, the weight of each task in the target task corresponding to any sampling point can be determined based on the execution start order, task type and / or task priority of each task.

[0034] In specific implementation, to improve the calculation efficiency of task capture parameters and reduce the amount of data involved in calculating task capture parameters, the task processing time of each sampling point can be updated. Optionally, the task processing time of any sampling point can be updated based on the newly added task processing time. Specifically, the task processing time of each sampling point can be stored in the circular storage unit of the task processing module of the distributed process, the circular storage unit of the distributed process, the circular storage unit of the task detection module, and / or the circular storage unit of the task capture module. When a candidate task processing time is detected, it can be checked whether the storage slot corresponding to the allocation position value of the candidate task processing time in the circular storage unit is consistent with the storage slot corresponding to the task processing time of any sampling point. If they are consistent, the candidate task processing time can be used as the newly added task processing time to update the task processing time of any sampling point. That is, the task processing time of any sampling point stored in the storage slot of the circular storage unit can be overwritten based on the newly added task processing time, or the task processing time stored in the storage slot of the circular storage unit and the newly added task processing time can be used as the task processing time currently stored in the storage slot.

[0035] In practical application scenarios, since the thread pool can continuously process tasks, the task processing module can continuously collect the processing time of candidate tasks. In order to store the processing time of candidate tasks in a suitable storage slot and reduce the storage amount of time, optionally, a new task processing time can be added. If the allocation position value of the candidate task processing time is consistent with the storage slot corresponding to the task processing time in the circular storage unit, it is set as the candidate task processing time.

[0036] In the specific execution process, the allocation position value can be calculated for the processing time of the candidate task, and the storage slot corresponding to the processing time of the candidate task in the ring storage unit can be determined according to the allocation position value. In an optional implementation of this embodiment, the storage slot corresponding to the allocation position value of the processing time of the candidate task in the ring storage unit can be determined in each storage slot of the ring storage unit based on the cumulative allocation value corresponding to the processing time of the candidate task and the number of sampling points of each sampling point. Specifically, it can be determined in the following way:

[0037] The allocation position value is calculated based on the cumulative allocation value corresponding to the candidate task processing time and the number of sampling points at each sampling point.

[0038] Based on the assigned location value, determine the storage slot corresponding to the processing time of the candidate task among the storage slots contained in the ring storage unit.

[0039] Among them, the cumulative allocation value refers to the cumulative value or cumulative quantity of storage allocated to the ring storage unit for the processing time of the candidate task. For example, the cumulative allocation value 'a' represents the cumulative value of storage allocated to the ring storage unit for the processing time of the candidate task. The allocation location value refers to the location value at which the processing time of the candidate task is allocated to the ring storage unit for storage. For example, the allocation location value is the storage slot identifier of the ring storage unit.

[0040] Specifically, in the process of calculating the allocation position value based on the cumulative allocation value corresponding to the candidate task processing time and the number of sampling points of each sampling point, the number of sampling points of each sampling point can be moduloed based on the cumulative allocation value corresponding to the candidate task processing time, and the modulo result (remainder) can be used as the allocation position value; in the process of determining the storage slot corresponding to the candidate task processing time in each storage slot contained in the ring storage unit according to the allocation position value, the allocation position value can be matched with the storage slot identifier of each storage slot contained in the ring storage unit, and the corresponding storage slot can be determined based on the matching result.

[0041] To improve the service processing efficiency of the target service, a distributed process and a distributed scheduling system are used for service processing. Optionally, the distributed process, the distributed scheduling system, and the service node of the credit service within the application correspond one-to-one. There can be one or more distributed processes, and each distributed process corresponds to its own distributed scheduling system and / or the service node of the credit service within the application. In this embodiment, the credit service can be replaced by the target service. The distributed scheduling system may include a task detection module, a task capture module, and / or a task processing module of the distributed process.

[0042] The credit service or target service within the application may correspond to a task triggering system, which may be a centrally managed task triggering system. Optionally, the task scheduling frequency is synchronized to the task detection module in the distributed scheduling system through the task triggering system of the credit service. Specifically, since the distributed scheduling system may correspond to the service nodes of the target service within the application, the task scheduling frequency synchronized by the task triggering system may be the same or different for each service node.

[0043] In the actual execution process, if the task capture parameters are large when the credit service is first started, it may lead to task backlog, thereby reducing task processing efficiency. In order to avoid the task processing efficiency being reduced by large task capture parameters when the credit service is first started, on the basis of the one-to-one correspondence between the above-mentioned distributed process, distributed scheduling system and service nodes of the credit service in the application, optionally, the credit service captures tasks according to the preset task capture parameters within a preset time period after startup, or captures tasks according to each task capture parameter in the task capture parameter sequence, with each task capture parameter increasing in proportion to the task capture.

[0044] Among them, the preset time period refers to the time period set in advance; the task capture ratio can be the task capture growth ratio. Each task capture parameter in the task capture parameter sequence can increase according to the preset attribute task capture growth ratio. The preset attribute can include a fixed task capture growth ratio or a monotonically increasing task capture growth ratio.

[0045] It should be noted that the operation of reading the task processing time of each sampling point according to the task scheduling frequency of the distributed process can be replaced by reading the task processing time of each task stored in the storage unit of the distributed process, or it can be replaced by reading the task processing time of each sampling point; and it can be combined with other processing steps provided in this embodiment to form a new implementation method.

[0046] Step S204: Based on the task processing time, the number of key threads in the thread pool configured in the task processing module of the distributed process, and the task scheduling frequency, calculate the task capture parameters to obtain the task capture parameters.

[0047] The task processing time of each sampling point is read according to the task scheduling frequency of the distributed process. In this step, the task capture parameters are calculated by combining the task processing time of each sampling point, the number of key threads in the thread pool configured by the task processing module of the distributed process, and the task scheduling frequency. Specifically, the task capture parameters can be calculated by the task detection module of the distributed process based on the task processing time, the number of key threads in the thread pool configured by the task processing module of the distributed process, and the task scheduling frequency.

[0048] The task processing module of the distributed process described in this embodiment can be included in a distributed scheduling system; the key thread in the thread pool can be an active thread in the thread pool, specifically the core thread in the thread pool, or more specifically, a continuously active thread; the number of threads refers to the number of key threads in the thread pool. Optionally, the number of threads can be synchronized to the task detection module through the task processing module of the distributed process.

[0049] Task crawling parameters refer to the crawling parameters used to crawl tasks. Task crawling parameters may include the task crawling quantity, which can be the number of tasks to be crawled.

[0050] In practical applications, network jitter and other factors may cause instability in task processing time, making pre-set fixed task capture parameters inadequate to handle fluctuations. To address this, and to improve the flexibility and stability of task capture, task capture parameters can be calculated in real time. Using these real-time calculated parameters effectively addresses fluctuations in task processing time. In one optional implementation of this embodiment, during the calculation of task capture parameters based on task processing time, the number of key threads in the thread pool configured in the distributed process's task processing module, and the task scheduling frequency, the following operations are performed:

[0051] The duration distribution index is obtained by calculating the duration distribution index based on the task processing time of each sampling point.

[0052] The task crawling volume is calculated based on the duration distribution index, the number of threads, and the task scheduling frequency.

[0053] Among them, the duration distribution index can be a distribution index that characterizes the distribution of task processing time at each sampling point; the duration distribution index may include the mean of task processing time and / or the discrete value of task processing time.

[0054] Based on this, in the process of calculating the duration distribution index according to the task processing time of each sampling point to obtain the duration distribution index, in an optional implementation of this embodiment, the mean task processing time and / or the discrete value of task processing time can be calculated according to the task processing time of each sampling point. Specifically, the following operations can be performed:

[0055] Based on the task processing time of each sampling point and the number of sampling points, the average task processing time is calculated, and the discrete value of task processing time is calculated based on the task processing time.

[0056] The mean task processing time can be the average of the task processing time at each sampling point; the discrete value of task processing time can be the standard deviation or variance of the task processing time at each sampling point.

[0057] Specifically, in calculating the average task processing time based on the task processing time of each sampling point and the number of sampling points at each sampling point, the sum of the task processing times of each sampling point can be calculated, and then the ratio of the sum of the task processing times of each sampling point to the number of sampling points at each sampling point can be used as the average task processing time. In calculating the discrete value of task processing time based on the task processing time, the standard deviation and / or variance of the task processing time can be calculated based on the task processing time of each sampling point and / or the average task processing time. In calculating the task crawling volume based on the duration distribution index, the number of threads, and the task scheduling frequency, the product of the task scheduling frequency, the preset value, and the number of threads can be calculated, and the sum of the average task processing time and the standard deviation or variance of the task processing time can be calculated. The ratio of the product and the sum can be used as the task crawling volume.

[0058] Furthermore, in the process of calculating the average task processing time based on the task processing time of each sampling point and the number of sampling points at each sampling point, the task processing time of each sampling point and the number of sampling points at each sampling point can be input into the averaging algorithm to calculate the average task processing time. In the process of calculating the task crawling volume based on the duration distribution index, the number of threads, and the task scheduling frequency, the duration distribution index, the number of threads, and the task scheduling frequency can be input into the crawling volume algorithm to calculate the task crawling volume.

[0059] For example, the mean algorithm includes:

[0060]

[0061] in, This represents the average processing time for representative tasks; Representing the Task processing time for each sampling point; This represents the number of sampling points at each sampling point.

[0062] For example, crawl volume algorithms include:

[0063]

[0064] in, Represents the amount of data crawled by the task; Represents the frequency of task scheduling; This represents the number of critical threads in the thread pool. This represents the standard deviation or variance of task processing time.

[0065] In specific implementation, the task capture module of the distributed process can obtain task capture parameters from the task detection module of the distributed process in real time before capturing tasks according to the task scheduling frequency, and then perform task capture using the obtained task capture parameters; optionally, the task capture parameters can be obtained by the task capture module of the distributed process calling the detection interface of the task detection module of the distributed process; optionally, the interface call is executed after the task capture module detects the task scheduling message generated by the task triggering system according to the task scheduling frequency; specifically, after detecting the task scheduling message generated by the task triggering system according to the task scheduling frequency, the task capture module of the distributed process can call the detection interface of the task detection module and obtain the task capture parameters returned by the interface call.

[0066] It should be noted that the above operation of calculating task capture parameters based on task processing time, the number of key threads in the thread pool configured by the task processing module of the distributed process, and task scheduling frequency can be replaced by calculating task capture parameters based on task processing time, the number of key threads in the thread pool configured by the task processing module of the distributed process, and / or task scheduling frequency, and forming a new implementation method with other processing steps provided in this embodiment.

[0067] Step S206: Fetch tasks from the set of tasks to be processed in the distributed process according to the task fetching parameters, and schedule the fetched tasks to the thread pool for task processing.

[0068] The task fetching parameters are calculated by combining the task processing time, the number of key threads in the thread pool configured in the task processing module of the distributed process, and the task scheduling frequency. In this step, tasks are fetched from the set of tasks to be processed in the distributed process according to the task fetching parameters, and the fetched tasks are scheduled to the thread pool for task processing. Specifically, the task fetching module of the distributed process can fetch tasks from the set of tasks to be processed in the distributed process according to the task fetching parameters, and schedule the fetched tasks to the thread pool configured in the task processing module of the distributed process for task processing. The set of tasks to be processed in the distributed process refers to a set of one or more tasks to be processed.

[0069] In practical implementation, to improve the flexibility of task fetching, avoid task backlog during subsequent task processing, and improve task processing efficiency, this embodiment provides an optional implementation method in which the following operations are performed during task fetching from the set of tasks to be processed in the distributed process according to the task fetching parameters:

[0070] The task capture module captures tasks that meet the task capture parameters from the set of pending tasks stored in the task database of the distributed process.

[0071] Optionally, the task crawling parameters include the task crawling quantity, the task crawling type, and / or the expected time unit for performing task crawling.

[0072] Among them, task capture volume refers to the number of tasks captured; task capture type refers to the type of tasks captured; and the expected time unit for task capture can be the expected time period or the expected time point for task capture.

[0073] Specifically, the task fetching module of the distributed process can fetch tasks from the set of pending tasks stored in the task database of the distributed process according to the task fetching volume, task fetching type and / or the expected time unit for task fetching.

[0074] In addition, based on the task capture parameters obtained by the task capture module of the distributed process through the interface call of the detection interface of the task detection module of the distributed process, the task capture module can also perform the above-mentioned task capture operation that captures tasks that meet the task capture parameters from the set of unprocessed tasks stored in the task database of the distributed process.

[0075] Furthermore, based on the above-mentioned task capture module retrieving tasks that meet the task capture parameters from the task database stored in the distributed process, in an optional implementation of this embodiment, the following operations are performed during the process of scheduling the captured tasks to the thread pool for task processing:

[0076] The task capture module schedules tasks to a task queue in memory, and the thread pool executes the tasks in the task queue.

[0077] It should be noted that the user data obtained in this manual is authorized by the user and does not involve user privacy.

[0078] The above-described operation of fetching tasks from the set of tasks to be processed in the distributed process according to the task fetching parameters and scheduling the fetched tasks to the thread pool for task processing can be replaced by fetching tasks according to the task fetching parameters and scheduling the fetched tasks to the thread pool for task processing, and forming a new implementation method with other processing steps provided in this embodiment.

[0079] It should be added that each optional implementation method and each feasible execution method in steps S202 to S206 provided in this embodiment can be executed independently as needed, or they can be combined and referenced with each other. At the same time, each specific execution step in each optional implementation method or each feasible execution method can also be executed independently or combined as needed. The execution conditions of "if" or "under what circumstances" involved in each step or operation can be directly deleted, and subsequent operations can be executed. This embodiment does not make specific limitations on this.

[0080] In summary, the one or more task processing methods provided in this embodiment firstly generate task scheduling messages according to the task scheduling frequency of the distributed process through the task triggering system, and send the task scheduling messages to the task capture component (i.e., the task capture module) of the distributed process. The task capture component of the distributed process makes an interface call to the detection interface of the task detection module of the distributed process and obtains the task capture parameters returned by the interface call. Secondly, the task capture component of the distributed process captures tasks from the set of tasks to be processed in the distributed process according to the task capture parameters, and schedules the captured tasks to the thread pool configured by the task processing module of the distributed process for task processing. In this way, the task triggering system centrally manages the task scheduling of the distributed process to improve the convenience of task scheduling, improves the efficiency of task processing through the distributed process, and improves the flexibility of task capture by calculating the task capture parameters in real time. This can effectively cope with system fluctuations and adjust the task capture parameters in a timely manner in response to system fluctuations, thereby improving system stability.

[0081] In the process of calculating task capture parameters, the task processing time of each sampling point can be read through the task detection module of the distributed process, and the task capture parameters can be calculated based on the task processing time of each sampling point, the number of key threads in the thread pool configured by the task processing module of the distributed process, and the task scheduling frequency.

[0082] The following description uses the application of a task processing method provided in this embodiment in a distributed scenario as an example to further illustrate the task processing method provided in this embodiment. (See also...) Figure 4 A task processing method for distributed scenarios is proposed, which can be applied to a distributed scheduling system for credit services. The method includes the following steps.

[0083] Step S402: In response to the interface call of the task capture module of the distributed process to the detection interface, the task processing time of each sampling point is read through the task detection module of the distributed process.

[0084] Optionally, the task capture module can call the detection interface according to the task scheduling frequency of the distributed process.

[0085] Step S404: The task detection module calculates the average task processing time based on the task processing time of each sampling point and the number of sampling points at each sampling point, and calculates the discrete value of the task processing time based on the task processing time.

[0086] Step S406: The task capture volume is calculated by the task detection module based on the average task processing time, the discrete value of task processing time, the number of key threads in the thread pool configured by the task processing module of the distributed process, and the task scheduling frequency, and the task capture volume is obtained.

[0087] Step S408: Using the task capture module, tasks are captured from the set of unprocessed tasks stored in the task database of the distributed process according to the task capture volume returned by the interface call.

[0088] Step S410: The captured tasks are scheduled to the task queue in memory by the task capture module, and the tasks in the task queue are executed by the thread pool of the task processing module of the distributed process.

[0089] It should be noted that any one or more of steps S402 to S410 can be replaced by the corresponding technical means provided in steps S202 to S206 as needed for implementation and deployment. Any one or more of steps S402 to S410 can also be combined into a new implementation method as needed for implementation and deployment. Furthermore, any one or more of steps S402 to S410 can also be combined with one or more of the steps provided in steps S202 to S206 to form a new implementation method, or combined with one or more optional implementation methods provided in steps S202 to S206 to form a new implementation method, as needed for actual deployment. These will not be elaborated on here.

[0090] This specification provides an embodiment of a task processing device as follows:

[0091] In the above embodiments, a task processing method is provided, and correspondingly, a task processing device is also provided, which will be described below with reference to the accompanying drawings.

[0092] Reference Figure 5 The diagram illustrates an embodiment of a task processing device provided in this embodiment.

[0093] Since the apparatus embodiments correspond to the method embodiments, the descriptions are relatively simple. For relevant parts, please refer to the corresponding descriptions of the method embodiments provided above. The apparatus embodiments described below are merely illustrative.

[0094] This embodiment provides a task processing device that runs on a distributed scheduling system. The device includes:

[0095] The duration reading module 502 is configured to read the task processing duration of each sampling point according to the task scheduling frequency of the distributed process.

[0096] The parameter calculation module 504 is configured to calculate task capture parameters based on the task processing duration, the number of key threads in the thread pool configured by the task processing module of the distributed process, and the task scheduling frequency.

[0097] The task scheduling module 506 is configured to retrieve tasks from the set of tasks to be processed in the distributed process according to the task retrieval parameters, and schedule the retrieved tasks to the thread pool for task processing.

[0098] The task processing time is obtained by the task processing module collecting the task duration of the target task corresponding to each sampling point.

[0099] For ease of description, the above devices are described by dividing them into various modules or units based on their functions. Of course, when implementing one or more of these specifications, the functions of each module or unit can be implemented in the same or different software and / or hardware, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0100] This specification provides an example of a task processing device as follows:

[0101] Corresponding to the task processing method described above, based on the same technical concept, one or more embodiments of this specification also provide a task processing device for executing the task processing method provided above. Figure 6 This is a schematic diagram of the structure of a task processing device provided for one or more embodiments of this specification.

[0102] This embodiment provides a task processing device, including:

[0103] like Figure 6As shown, task processing devices can vary significantly due to differences in configuration or performance. They may include one or more processors 601 and memory 602, with memory 602 storing one or more application programs or data. Memory 602 can be temporary or persistent storage. The application programs stored in memory 602 may include one or more modules (not shown), each module including a series of computer-executable instructions from the task processing device. Furthermore, processor 601 may be configured to communicate with memory 602, executing the series of computer-executable instructions stored in memory 602 on the task processing device. The task processing device may also include one or more power supplies 603, one or more wired or wireless network interfaces 604, one or more input / output interfaces 605, one or more keyboards 606, etc.

[0104] In one specific embodiment, the task processing device includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer-executable instructions for the task processing device, and is configured to be executed by one or more processors. The one or more programs include computer-executable instructions for performing the following:

[0105] The task processing time of each sampling point is read according to the task scheduling frequency of the distributed process.

[0106] Based on the task processing time, the number of key threads in the thread pool configured in the task processing module of the distributed process, and the task scheduling frequency, the task fetching parameters are calculated to obtain the task fetching parameters.

[0107] Tasks are retrieved from the set of tasks to be processed in the distributed process according to the task retrieval parameters, and the retrieved tasks are scheduled to the thread pool for task processing.

[0108] The task processing time is obtained by the task processing module collecting the task duration of the target task corresponding to each sampling point.

[0109] This specification provides an embodiment of a computer-readable storage medium as follows:

[0110] Corresponding to the task processing method described above, and based on the same technical concept, one or more embodiments of this specification also provide a computer-readable storage medium.

[0111] The computer-readable storage medium provided in this embodiment is used to store computer-executable instructions, which, when executed, perform the following steps:

[0112] The task processing time of each sampling point is read according to the task scheduling frequency of the distributed process.

[0113] Based on the task processing time, the number of key threads in the thread pool configured in the task processing module of the distributed process, and the task scheduling frequency, the task fetching parameters are calculated to obtain the task fetching parameters.

[0114] Tasks are retrieved from the set of tasks to be processed in the distributed process according to the task retrieval parameters, and the retrieved tasks are scheduled to the thread pool for task processing.

[0115] The task processing time is obtained by the task processing module collecting the task duration of the target task corresponding to each sampling point.

[0116] It should be noted that the embodiments of a computer-readable storage medium described in this specification and the embodiments of a task processing method described in this specification are based on the same inventive concept. Therefore, the specific implementation of this embodiment can be referred to the implementation of the corresponding method described above, and the repeated parts will not be described again.

[0117] This specification provides an example of a computer program product as follows:

[0118] Corresponding to the task processing method described above, based on the same technical concept, one or more embodiments of this specification also provide a computer program product.

[0119] A computer program product includes a computer program / instructions that, when executed by a processor, perform the following steps:

[0120] The task processing time of each sampling point is read according to the task scheduling frequency of the distributed process.

[0121] Based on the task processing time, the number of key threads in the thread pool configured in the task processing module of the distributed process, and the task scheduling frequency, the task fetching parameters are calculated to obtain the task fetching parameters.

[0122] Tasks are retrieved from the set of tasks to be processed in the distributed process according to the task retrieval parameters, and the retrieved tasks are scheduled to the thread pool for task processing.

[0123] The task processing time is obtained by the task processing module collecting the task duration of the target task corresponding to each sampling point.

[0124] It should be noted that the embodiments of a computer program product described in this specification and the embodiments of a task processing method described in this specification are based on the same inventive concept. Therefore, the specific implementation of this embodiment can be referred to the implementation of the corresponding method described above, and the repeated parts will not be described again.

[0125] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments. For example, the device embodiment, equipment embodiment and computer-readable storage medium embodiment are all similar to the method embodiment, so the description is relatively simple. When reading the relevant content of the device embodiment, equipment embodiment and computer-readable storage medium embodiment, please refer to the description of the method embodiment.

[0126] Although one or more embodiments of this specification provide method steps as described in the embodiments or flowcharts, it is understood that the order of steps listed in the embodiments or flowcharts is only one of many possible execution orders and does not represent the only execution order. Therefore, when the claims involve method steps, any changes or adjustments to the order of such steps, or the parallelism between steps, are also within the scope of protection of the claims.

[0127] This specification uses specific terms to describe embodiments thereof. Terms such as "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described herein, as well as the features of those different embodiments or examples, without contradiction.

[0128] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0129] In the 1930s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many improvements to the methodology today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that an improvement to the methodology cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must also be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0130] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0131] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0132] For ease of description, the above apparatus is described by dividing it into various functional units. Of course, when implementing the embodiments of this specification, the functions of each unit can be implemented in one or more software and / or hardware.

[0133] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0134] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable test processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable test processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0135] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable test processing equipment to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0136] These computer program instructions can also be loaded onto a computer or other programmable test processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0137] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0138] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0139] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0140] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of features includes not only those features but also other features not expressly listed, or features inherent to such process, method, article, or apparatus. Without further limitations, a feature defined by the phrase "comprising one..." does not exclude the presence of other identical features in the process, method, article, or apparatus that includes said feature.

[0141] One or more embodiments of this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0142] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0143] The above description is merely an embodiment of this document and is not intended to limit the scope of this document. Various modifications and variations can be made to this document by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this document should be included within the scope of the claims of this document.

Claims

1. A task processing method applied to a distributed scheduling system, the method comprising: reading task processing durations of each sampling point according to a task scheduling frequency of a distributed process; calculating a duration distribution index according to the task processing durations to obtain the duration distribution index, inputting the duration distribution index, a number of key threads in a thread pool configured by a task processing module of the distributed process, and the task scheduling frequency into a task grabbing quantity algorithm to calculate a task grabbing quantity; and grabbing tasks from a set of to-be-processed tasks of the distributed process according to the task grabbing quantity, and scheduling the grabbed tasks to the thread pool for task processing. The task processing durations are obtained by collecting task durations of target tasks corresponding to the sampling points by the task processing module. 2.The task processing method of claim 1, wherein the task grabbing quantity algorithm comprises: wherein, represent a task grabbing amount, represent a task scheduling frequency, represent a thread number of a key thread in a thread pool, represent a task processing time length standard deviation or variance in a time length distribution index, represent a task processing time length mean in a time length distribution index. 3.The task processing method of claim 1, wherein the calculating a duration distribution index according to the task processing durations to obtain the duration distribution index comprises: calculating a task processing duration mean value according to the task processing durations of the sampling points and a number of the sampling points, and calculating a task processing duration dispersion value according to the task processing durations. 4.The task processing method of claim 1, wherein the task grabbing quantity is obtained by an interface call of a detection interface of a task detection module of the distributed process by a task grabbing module of the distributed process. The interface call is executed after the task grabbing module detects a task scheduling message generated by a task triggering system according to the task scheduling frequency. 5.The task processing method of claim 4, wherein the grabbing tasks from the set of to-be-processed tasks of the distributed process according to the task grabbing quantity comprises: grabbing tasks satisfying the task grabbing quantity from the set of to-be-processed tasks stored in a task database of the distributed process by the task grabbing module. 6.The task processing method of claim 5, wherein the scheduling the grabbed tasks to the thread pool for task processing comprises: scheduling the tasks to a task queue in a memory by the task grabbing module, and performing and processing the tasks in the task queue by the thread pool. 7.The task processing method of claim 1, wherein the sampling points are configured according to a preset time interval or a task arrangement position of a task queue of the thread pool. The task processing duration of any sampling point is calculated based on a number of target tasks corresponding to the any sampling point. 8.The task processing method of claim 7, wherein the calculating the task processing duration based on the number of target tasks corresponding to the any sampling point comprises: if the number of target tasks corresponding to the any sampling point is equal to 1, taking a task processing duration of the target task corresponding to the any sampling point as the task processing duration of the any sampling point. If the number of tasks is greater than 1, the task processing time length of any sampling point is calculated based on the task processing time length of the target task corresponding to the sampling point.

9. The task processing method of claim 1, wherein the task processing time length of any sampling point in the sampling points is updated based on a new task processing time length. The new task processing time length is set as the candidate task processing time length in the case that the allocation position value of the candidate task processing time length corresponds to the storage slot corresponding to the task processing time length in the ring-shaped storage unit.

10. The task processing method of claim 9, wherein the storage slot corresponding to the allocation position value of the candidate task processing time length in the ring-shaped storage unit is determined in the following manner: The allocation position value is calculated based on the allocation cumulative value corresponding to the candidate task processing time length and the number of sampling points in the sampling points, and the allocation position value is obtained. The storage slot corresponding to the candidate task processing time length is determined in each storage slot included in the ring-shaped storage unit according to the allocation position value.

11. The task processing method of claim 1, wherein the distributed process, the distributed scheduling system, and the service node of the application program internal credit service correspond to each other one by one. The task scheduling frequency is synchronized to the task detection module in the distributed scheduling system by the task triggering system of the credit service, and the number of threads is synchronized to the task detection module by the task processing module.

12. The task processing method of claim 11, wherein the credit service performs task grabbing according to preset task grabbing parameters within a preset time period after starting, or according to each task grabbing parameter in a task grabbing parameter sequence, and the task grabbing parameters are incremented according to a task grabbing ratio.

13. A task processing device running in a distributed scheduling system, the device comprising: A time length reading module configured to read the task processing time length of each sampling point according to the task scheduling frequency of a distributed process; A parameter calculation module configured to calculate a time length distribution index according to the task processing time length to obtain a time length distribution index, input the time length distribution index, the number of threads of a key thread in a thread pool configured by a task processing module of the distributed process, and the task scheduling frequency into a grabbing amount algorithm to calculate a task grabbing amount according to the grabbing amount algorithm to obtain a task grabbing amount; A task scheduling module configured to perform task grabbing from a set of tasks to be processed of the distributed process according to the task grabbing amount, and schedule the grabbed tasks to the thread pool for task processing; Wherein, the task processing time length is obtained by the task processing module collecting the task time length of the target task corresponding to each sampling point.

14. A task processing device comprising: A processor; And a memory configured to store computer executable instructions, the computer executable instructions, when executed, causing the processor to: Read the task processing time length of each sampling point according to the task scheduling frequency of a distributed process; According to the task processing duration, a duration distribution index is calculated to obtain a duration distribution index, and the duration distribution index, the number of key threads in the thread pool configured by the task processing module of the distributed process, and the task scheduling frequency are input into a task grabbing quantity algorithm to calculate a task grabbing quantity; According to the task grabbing quantity, tasks are grabbed from the set of tasks to be processed of the distributed process, and the grabbed tasks are scheduled to the thread pool for task processing; The task processing duration is obtained by collecting the task duration of the target task corresponding to each sampling point by the task processing module.

15. A computer readable storage medium for storing computer executable instructions, which when executed implement the steps of the method of claim 1.

Images