Task processing method and device, electronic equipment and computer program product

Abstract

The invention relates to a task processing method and device, electronic equipment and a computer program product, relates to the technical field of computers, and can be applied to a distributed task scheduling processing scene. The method comprises the steps of obtaining a to-be-executed task, wherein the to-be-executed task is obtained by executing task scanning operation based on a pre-configured task scheduling period; segmenting the to-be-executed task and adding the to-be-executed task to a task scheduling set, wherein the task scheduling set comprises a plurality of fragmented tasks; and executing the fragmented task, and updating the task scheduling set according to a task execution result of the fragmented task so as to add a new fragmented task in the updated task scheduling set. In the task execution process, scheduling of the next batch can be carried out without waiting for execution of all tasks of the current task batch, the problem of large task blocking can be effectively solved, the task processing efficiency is improved, and the problem of machine downtime caused by high machine memory occupation due to simultaneous execution of a large number of tasks is effectively avoided.

Description

Background Art

[0002] In task processing, for some time-consuming operations, asynchronous execution is generally used for processing, such as import and export functions, batch data refresh, batch bill synthesis and other functions. Various distributed scheduling platforms are usually used to process these tasks that require asynchronous processing. The processing methods are mainly serial batch processing and parallel processing. For tasks that do not use distributed tasks and directly start child threads to execute, this method cannot save the success of the task, and cannot be retried if it fails. This method is usually not used for some slightly important functions. Therefore, this execution method is generally not considered.

[0003] The above two task execution modes include: (1) serial batch processing. When tasks are distributed to each machine, each machine uses batch serial processing. For example, 10 tasks are queried for execution each time, and the next batch is executed after all 10 tasks are completed. (2) parallel processing. When tasks are distributed to each machine, each machine executes as many tasks as it scans to be executed.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The purpose of the present disclosure is to provide a task processing method, a task processing device, an electronic device, a computer-readable storage medium and a computer program product, thereby at least to a certain extent overcoming the problem of large task blocking or CPU surge and system crash in related task scheduling execution schemes.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present invention.

[0007] According to a first aspect of the present disclosure, a task processing method is provided, comprising: obtaining a task to be executed, wherein the task to be executed is obtained by performing a task scanning operation based on a preconfigured task scheduling period; splitting the task to be executed and adding it to a task scheduling set, wherein the task scheduling set includes a plurality of sliced ​​tasks; executing the sliced ​​tasks, and updating the task scheduling set according to the task execution results of the sliced ​​tasks, so as to add new sliced ​​tasks to the updated task scheduling set.

[0008] In an exemplary embodiment of the present disclosure, obtaining the task to be executed includes: obtaining a pre-configured task scheduling cycle, scanning and processing a table of tasks to be executed based on the task scheduling cycle to obtain the task to be executed; obtaining a task information table corresponding to the task to be executed, and the task information table is used to execute the task to be executed.

[0009] In an exemplary embodiment of the present disclosure, the table of tasks to be executed includes one or more of the following fields: a primary key task identifier, an associated identifier associated with the task information table, the current number of executions, the task status, the task pending execution time and the shard task identifier; the task information table includes one or more of the following fields: task query conditions, task query content, task processing method, file name, operating user information and task function information.

[0010] In an exemplary embodiment of the present disclosure, the task scheduling set has a specified initial capacity, and the splitting of the to-be-executed tasks and adding them to the task scheduling set includes: splitting the to-be-executed tasks to obtain a plurality of the shard tasks and a shard task identifier of each of the shard tasks; obtaining a pre-configured first total scheduling number; determining a task scheduling identifier corresponding to each of the shard tasks based on the first total scheduling number and the shard task identifier; and adding each of the shard tasks to the task scheduling set based on the task scheduling identifier and the specified initial capacity, wherein the task scheduling identifiers contained in the task scheduling set are unique.

[0011] In an exemplary embodiment of the present disclosure, each of the slice tasks is added to the task scheduling set based on the task scheduling identifier and the specified initial capacity, including: determining the number of idle schedules in the task scheduling set based on the specified initial capacity; when the number of idle schedules is not zero and the task scheduling set does not include the task scheduling identifier, adding the slice task to the idle schedule of the task scheduling set.

[0012] In an exemplary embodiment of the present disclosure, the task scheduling set is updated according to the task execution result of the shard task, including: when the task execution result is successful execution, the task status of the shard task is updated to successful execution; when the task execution result is failed execution, the task status is updated to failed execution, and the current execution count of the shard task with the task status of failed execution is updated; and the specified shard task in the task scheduling set is deleted, and the specified shard task is a shard task with the task status of successful execution or failed execution.

[0013] In an exemplary embodiment of the present disclosure, the method further includes: in response to a device update instruction of a task scheduling system, determining a newly added device from the task scheduling system, wherein the device update instruction is generated based on a device abnormality event or a device new event; determining a second total scheduling number corresponding to the newly added device; and based on the second total scheduling number, allocating the sharded tasks in the task scheduling set to the newly added device.

[0014] In an exemplary embodiment of the present disclosure, allocating the sharded task in the task scheduling set to the newly added device includes: obtaining basic task information of the sharded task; based on the basic task information, locking the sharded task to obtain a locked sharded task; and adding the locked sharded task to the task scheduling set in the newly added device.

[0015] According to a second aspect of the present disclosure, a task processing device is provided, including: a task acquisition module, used to acquire tasks to be executed, wherein the tasks to be executed are acquired by performing a task scanning operation based on a preconfigured task scheduling period; a task slicing module, used to slice the tasks to be executed and add them to a task scheduling set, wherein the task scheduling set includes multiple sliced ​​tasks; a task execution module, used to execute the sliced ​​tasks, and update the task scheduling set according to the task execution results of the sliced ​​tasks, so as to add new sliced ​​tasks to the updated task scheduling set.

[0016] In an exemplary embodiment of the present disclosure, the task acquisition module includes a task acquisition unit, which is used to: obtain a pre-configured task scheduling cycle, scan and process a task table to be executed based on the task scheduling cycle to obtain the task to be executed; obtain a task information table corresponding to the task to be executed, and the task information table is used to execute the task to be executed.

[0017] In an exemplary embodiment of the present disclosure, the task scheduling set has a specified initial capacity, and the task slicing module includes a task slicing unit, which is used to: split the task to be executed to obtain a plurality of slicing tasks and a slicing task identifier of each slicing task; obtain a pre-configured first total scheduling number; determine the task scheduling identifier corresponding to each slicing task based on the first total scheduling number and the slicing task identifier; and add each slicing task to the task scheduling set based on the task scheduling identifier and the specified initial capacity, and the task scheduling identifier contained in the task scheduling set is unique.

[0018] In an exemplary embodiment of the present disclosure, the task slicing unit includes a task adding sub-unit, which is used to: determine the number of idle schedules in the task scheduling set based on the specified initial capacity; when the number of idle schedules is not zero and the task scheduling identifier is not included in the task scheduling set, add the slicing task to the idle schedule of the task scheduling set.

[0019] In an exemplary embodiment of the present disclosure, the task execution module includes a set updating unit, which is used to: when the task execution result is successful execution, update the task status of the shard task to successful execution; when the task execution result is failed execution, update the task status to failed execution, and update the current execution count of the shard task with the task status of failed execution; delete the specified shard task in the task scheduling set, and the specified shard task is the shard task with the task status of successful execution or failed execution.

[0020] In an exemplary embodiment of the present disclosure, the task processing device also includes a device addition module, which is used to: determine a new device from the task scheduling system in response to a device update instruction of the task scheduling system, wherein the device update instruction is generated based on a device abnormal event or a device addition event; determine a second total scheduling number corresponding to the new device; and based on the second total scheduling number, allocate the sliced ​​tasks in the task scheduling set to the new device.

[0021] In an exemplary embodiment of the present disclosure, the device new addition module includes a task reallocation unit, which is used to: obtain basic task information of the shard task; based on the basic task information, lock the shard task to obtain a locked shard task; and add the locked shard task to the task scheduling collection in the new device.

[0022] According to a third aspect of the present disclosure, there is provided an electronic device, comprising: a processor; and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the task processing method according to any one of the above items is implemented.

[0023] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the task processing method according to any one of the above items is implemented.

[0024] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements any one of the above-mentioned task processing methods when executed by a processor.

[0025] The technical solution provided by the present disclosure may have the following beneficial effects:

[0026] The task processing method in the exemplary embodiment of the present disclosure, on the one hand, executes tasks through a task scheduling set, and updates the task scheduling set according to the task execution results, so that during the task execution process, there is no need to wait for all tasks of the current task batch to be completed before scheduling the next batch, which can effectively solve the problem of large task blocking and improve task processing efficiency. On the other hand, the task scheduling set usually has a specified initial capacity, which can effectively avoid the machine downtime caused by high machine memory usage due to the simultaneous execution of a large number of tasks during the parallel execution of tasks.

[0027] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0029] Figure 1 A flowchart schematically shows a task processing method according to an exemplary embodiment of the present disclosure;

[0030] Figure 2 A flowchart of distributed task scheduling using a fixed-capacity anti-duplicate set according to an exemplary embodiment of the present disclosure is schematically shown;

[0031] Figure 3 A flowchart of adding a task to be executed to an anti-weight container for execution is schematically shown according to an exemplary embodiment of the present disclosure;

[0032] Figure 4 A schematic diagram schematically shows a method of updating a task scheduling set according to a task execution result according to an exemplary embodiment of the present disclosure;

[0033] Figure 5 A block diagram schematically shows a task processing device according to an exemplary embodiment of the present disclosure;

[0034] Figure 6 A block diagram schematically shows an electronic device according to an exemplary embodiment of the present disclosure;

[0035] Figure 7 A schematic diagram of a computer-readable storage medium according to an exemplary embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0036] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0037] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known structures, methods, devices, implementations, materials or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.

[0038] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or these functional entities or parts of functional entities may be implemented in one or more software hardened modules, or these functional entities may be implemented in different networks and / or processor devices and / or microcontroller devices.

[0039] In task processing, serial batch processing and parallel processing are the two most commonly used processing methods, especially the first method. However, there are some problems with the above two task processing methods. Taking the export task as an example, the problems of these two methods are analyzed respectively:

[0040] (1) Serial batch processing: This method requires that one batch must be completed before the next batch can be executed. At this time, if a batch encounters a large export task with millions of rows or even more, a single export task may take 10-30 minutes to execute. Because this task has not been completed in the batch, the scheduling task is always in a waiting state. Even if there is a particularly small export task later, it is necessary to wait for all tasks in the current batch to be completed before scheduling the next batch, which affects the user experience.

[0041] (2) Parallel processing, that is, all export tasks are performed at the same time. Although this method can avoid the blocking situation of method 1 in most cases, once multiple people are performing export tasks at the same time, it is easy to use up the machine memory and cause frequent global garbage collection (Full Garbage Collection, Full GC) operations, which in turn causes the CPU to soar and even application crashes. Therefore, this method is generally not used in production environments.

[0042] Based on this, in this example embodiment, a task processing method is first provided. The task processing method of the present disclosure can be implemented using a server, or the method described in the present disclosure can be implemented using a terminal device, wherein the terminal described in the present disclosure may include mobile terminals such as mobile phones, tablet computers, laptops, PDAs, personal digital assistants (PDAs), and fixed terminals such as desktop computers. Figure 1 The following is a schematic diagram showing a task processing method flow according to some embodiments of the present disclosure. Figure 1 , the task processing method may include the following steps:

[0043] Step S110, obtaining the tasks to be executed, where the tasks to be executed are obtained by performing a task scanning operation based on a pre-configured task scheduling period.

[0044] According to some exemplary embodiments of the present disclosure, the to-be-executed task may be a task waiting to be executed. The task scheduling period may be a pre-configured time period for periodically scanning the task table to obtain the to-be-executed task. The task scanning operation may be a periodic scanning operation for scanning the to-be-executed task table to obtain the to-be-executed task.

[0045] Before task processing, a to-be-executed task list is created in advance. The to-be-executed task list is scanned regularly according to a pre-configured task scheduling cycle. A task scanning operation is performed on the to-be-executed task list to determine the to-be-executed tasks.

[0046] Step S120, splitting the task to be executed and adding it to a task scheduling set, wherein the task scheduling set includes a plurality of split tasks.

[0047] According to some exemplary embodiments of the present disclosure, a task scheduling set may be a container in a task execution device for carrying a shard task, and the container may carry a specified number of shard tasks. A shard task may be a task generated after a task to be executed is sharded. The sharding process may be a process of sharding a large task to be executed into multiple small tasks.

[0048] After obtaining the tasks to be executed, for some large tasks to be executed, which are tasks that take a long time to execute, in order to improve the efficiency of task execution, the tasks to be executed can be split into multiple shard tasks. Then, the generated multiple shard tasks are added to the task scheduling collection to obtain and execute the shard tasks from the task scheduling collection.

[0049] Step S130, executing the slice task, and updating the task scheduling set according to the task execution result of the slice task, so as to add the new slice task to the updated task scheduling set.

[0050] According to some exemplary embodiments of the present disclosure, the task execution result may be the result of whether the shard task is successfully executed, and the task execution result may include execution success and execution failure.

[0051] For the shard tasks that have been added to the task scheduling set, execute the added shard tasks and determine the task execution results of the shard tasks. For the tasks in the task scheduling set whose task execution results are successful or failed, it is necessary to delete the identifier corresponding to the shard task from the task scheduling set and update the above task scheduling set. If a shard task in the task scheduling set has been executed, an idle position will be generated in the updated task scheduling set.

[0052] For successfully executed sharded tasks, the results of successful task execution can be written back to the pre-created task table. For failed sharded tasks, when there are free positions in the task scheduling collection, they can be added to the task scheduling collection again through the next task scanning operation; if there are currently no free positions in the task scheduling collection, you can continue to wait for the next task scanning operation to add it to the task scheduling collection. Through the above task scheduling method, during the task scheduling process, there is no need to wait for all tasks in the current task round to be executed before scheduling the next batch of tasks, which can effectively avoid blocking of large tasks.

[0053] According to the task processing method in this example embodiment, on the one hand, tasks are executed through a task scheduling set, and the task scheduling set is updated according to the task execution results, so that during the task execution process, it is not necessary to wait for all tasks of the current task batch to be completed before scheduling the next batch, which can effectively solve the problem of large task blocking and improve task processing efficiency. On the other hand, the task scheduling set usually has a specified initial capacity, which can effectively avoid the machine downtime caused by high machine memory usage due to the simultaneous execution of a large number of tasks during the parallel execution of tasks.

[0054] Below, the task processing method in this example embodiment will be further explained. The data export function is a common blocking problem scenario. In the data export function, each person has different data permissions, and the query conditions for each person's export operation are also different. For users with higher permissions, one export may involve tens of millions of data; other users may only have permissions for a few data. In addition, at certain specific time points, such as the billing period, there may be hundreds or even thousands of export tasks running at the same time. This disclosure uses the common data export function as an example to explain the task processing method in detail.

[0055] In an exemplary embodiment of the present disclosure, for step S110, obtaining the task to be executed includes: obtaining a pre-configured task scheduling cycle, scanning and processing the task table to be executed based on the task scheduling cycle to obtain the task to be executed; obtaining a task information table corresponding to the task to be executed, and the task information table is used to execute the task to be executed.

[0056] The to-be-executed task table may be a data table used to record and store the to-be-executed tasks. The task information table may be a data table used to record relevant execution information of the to-be-executed tasks.

[0057] refer to Figure 2 , Figure 2 The flowchart of distributed task scheduling using a fixed-capacity anti-duplicate set according to an exemplary embodiment of the present disclosure is schematically shown. Before performing task scheduling, a table of tasks to be executed and a table of task information are created. In order to perform task scanning operations regularly, the present disclosure pre-configures a task scheduling cycle. For example, the task scheduling cycle can be configured as the current batch task scanning operation time + waiting scheduling time, such as the waiting scheduling time can be configured to 10 seconds.

[0058] In distributed task scheduling, each individual scheduler needs to wait until all batch tasks are completed and the system feedbacks to the scheduling center that the current task scheduling has been completed before entering the next scheduling cycle. For example, 10 seconds can be configured as a task scheduling cycle. The 10 seconds here does not mean scheduling once every 10 seconds, but the next task scheduling will be performed 10 seconds after the last scheduling. This solution is to immediately feedback to the scheduling center that the current task has been completed after the current scheduling scans the task and adds it to the task scheduling collection, and it can prepare to enter the next scheduling cycle.

[0059] The distributed task scheduling framework (such as the elastic-job framework) scans the pre-created task table to be executed according to the pre-configured task scheduling cycle to obtain the tasks to be executed; in addition, the task information table corresponding to the tasks to be executed can also be obtained, and specific task operations will be performed based on the task information table in the future. Taking the data export function as an example, the task information table can be an export information table, which records the relevant query conditions, query content, export method, file name, operator, and other information related to the export function itself. By regularly scanning the task table to be executed according to the task scheduling cycle, the tasks to be executed can be obtained in batches, and distributed scheduling can be performed on different batches of tasks to be executed.

[0060] In an exemplary embodiment of the present disclosure, the table of tasks to be executed includes one or more of the following fields: primary key task identifier, an associated identifier associated with the task information table, current number of executions, task status, task pending execution time and shard task identifier; the task information table includes one or more of the following fields: task query conditions, task query content, task processing method, file name, operating user information and task function information.

[0061] Among them, the primary key task identifier can be a unique identifier of a task to be executed, which is used to distinguish different tasks to be executed in the task scheduling system. The association identifier associated with the task information table can be a unique identifier used to indicate the association relationship with the task information table, which can be recorded as export_id. The current execution count can indicate the number of times a task has been executed, which can be recorded as execute_count. A task can be a shard task included in the task to be executed. In addition to recording the number of times the current task has been executed, the current execution count can also set the maximum number of task executions to prevent abnormal tasks from being executed all the time.

[0062] The task status can be the specific status of a task, which can be recorded as task_status. The task status can include three status values: initial status, successful execution, and failed execution. The task is in the initial status at the beginning, updated to the successful execution status after successful execution, and updated to the failed execution status after failed execution. The reason why there is no executing status is that if the task status is updated to executing during task execution, if a downtime or restart occurs at this time, the task status will always be in the executing status and will not be scanned and executed again.

[0063] The task waiting time can be understood as the time when the task is scanned, which can be recorded as job_time. The task waiting time can be used to indicate that the task will be executed at a specified time in the future after other conditions are met. The shard task identifier can be a unique identifier representing a shard task, which can be recorded as queue_id. The main purpose of the shard task identifier is to break up the tasks so that different machines can scan different tasks during distributed scheduling. Its value can be set to a range according to the actual situation, such as 0-63. When writing, the value can be written in sequence, or a random number between 0-63 can be written, as long as the purpose of evenly dividing the data can be achieved.

[0064] Task query conditions may refer to various conditions used when querying tasks, which can help users accurately find the required task information. Task query content may be the specific content based on which tasks are queried. Task processing method may be the specific method of processing a task. File name may be the file name corresponding to the task to be executed. Operation user information may be relevant information of the operation user corresponding to the task to be executed, and the operation user information may include basic user information and user permissions. Task function information may be the specific function corresponding to the task to be executed.

[0065] The present disclosure does not merge the task information table and the pending task table into one table because the pending task table will be frequently scanned during the task scheduling process, and the successfully executed tasks can be cleaned up regularly to avoid the slow SQL problem caused by the large amount of data, while the task information table may be retained for a long time for user query; wherein, slow SQL, also known as slow query, refers to a Structured Query Language (SQL) statement query that exceeds a specified time. By defining specific fields of the task information table and the pending task table, task scheduling operations can be performed later based on the above table fields.

[0066] In an exemplary embodiment of the present disclosure, for step S120, the task to be executed is split and added to a task scheduling set, including: splitting the task to be executed to obtain multiple shard tasks and a shard task identifier of each shard task; obtaining a pre-configured first total scheduling number; determining a task scheduling identifier corresponding to each shard task based on the first total scheduling number and the shard task identifier; adding each shard task to the task scheduling set based on the task scheduling identifier and the specified initial capacity, and the task scheduling identifier contained in the task scheduling set is unique.

[0067] Among them, the first total scheduling number can be the number of devices in the task scheduling system that can be used to process the tasks to be executed. The task scheduling identifier, also known as the current scheduling sequence number, can be the device identifier that actually executes a certain slice task. The task scheduling set can be a fixed-capacity anti-duplicate set. The fixed-capacity anti-duplicate set means that the task scheduling has a specified initial capacity and can only accommodate a specified number of tasks. The task scheduling identifiers contained in the task scheduling set are all different, and there are no duplicate tasks in the task scheduling set. The specified initial capacity can be the maximum capacity of the task scheduling set. For example, the specified initial capacity can be 10, which means that the task scheduling set can accommodate 10 tasks at the same time.

[0068] The task scheduling of this disclosure can be implemented based on the timed scheduling framework elastic-job. Figure 2 , the elastic-job framework is used to periodically scan the task table, obtain the tasks to be executed, and add the tasks to be executed to the task scheduling collection. If the addition is successful, the successfully added tasks can be executed later, and the task execution results are written back to the task table; if the addition fails, the task can be scanned again during the next task scheduling.

[0069] The task adding process can be performed based on the total number of schedules currently in the task scheduling system (i.e., the first total number of schedules). For example, elastic-job can evenly distribute the number of schedules executed by each machine based on the total number of machines and the total number of schedules. Assuming that the current task scheduling system has 8 machines and the total number of schedules is set to 8, each machine is assigned a schedule, and each schedule can have a corresponding schedule number. If the total number of schedules remains unchanged, the schedule number will not change, and the machine corresponding to the schedule number will not change. When scanning tasks, you can scan the tasks to be executed in this batch based on the total number of task schedules (the first total number of schedules) and the current schedule number, combined with the queue_id mentioned above. The specific query method is as follows.

[0070] Assuming that the task scheduling identifier (that is, the current scheduling sequence number) is 3, queue_id is assumed to be in the range of 0-63 as previously assumed, and the 64 numbers 0-63 are used to modulo 8 (the total number of schedulings). The results are 3 for 3, 11, 19, 27, 35, 43, 51, and 59. The current scheduling query is the pending tasks whose job_time is less than or equal to the current time and whose execution status is initial or failed (the current execution count of the failed shard task needs to be less than the specified maximum execution count) and whose queue_id is (3, 11, 19, 27, 35, 43, 51, 59), and the specified number of shard tasks are queried.

[0071] In the present disclosure, a task scheduling set is initialized in memory when each application instance is started, wherein the task scheduling set can be an anti-duplicate container, that is, the shard task identifiers in the task scheduling set all have different task identifiers. Taking the HashSet set in Java as an example, the specified initial capacity of the HashSet set is 10, which means that the current application instance can execute up to 10 export tasks at the same time. The specific data can be set according to the actual situation. The 10 to-be-executed task IDs queried in the previous step are added to the HashSet before they are actually executed. In other exemplary embodiments of the present disclosure, the specified initial capacity of the task scheduling set can also be set to other values, such as 8, 16, etc., according to the actual situation.

[0072] In addition, since the present disclosure deals with distributed tasks, and the anti-duplicate set is a memory set of a local application, whether machine a will scan the task id being executed by machine b in the task scheduling set will usually not occur in the present disclosure. As mentioned earlier, when the total scheduling number remains unchanged, the machine corresponding to the scheduling number does not change. The previous task uses queue_id to scatter the data, and the queue_id corresponding to each task id does not change. According to the query method mentioned above, a queue_id value will only be scanned by the machine corresponding to the fixed scheduling number, so this situation usually does not occur.

[0073] The present invention adopts a fixed-capacity anti-duplication collection to avoid problems such as CPU surge and crash caused by executing multiple tasks at the same time, and can also avoid repeated execution of the same task at the same time. For example, when the task id is scanned during the first scheduling, it is added to the anti-duplication collection. However, due to the long execution time of the task, the task is still being executed during the second scheduling. According to the above steps, the task is first added to the anti-duplication collection. At this time, the anti-duplication collection still has the task id, so the addition fails, and the task will not be repeatedly executed at the same time.

[0074] In an exemplary embodiment of the present disclosure, each shard task is added to a task scheduling set based on a task scheduling identifier and a specified initial capacity, including: determining the number of idle schedules in the task scheduling set based on the specified initial capacity; when the number of idle schedules is not zero and the task scheduling set does not include a task scheduling identifier, adding the shard task to the idle schedule of the task scheduling set.

[0075] The number of idle schedules may be the number of unoccupied schedules in the task schedule set. The idle schedule may be a schedule in an idle state.

[0076] refer to Figure 3 , Figure 3The flowchart of adding tasks to be executed to the anti-heavy container according to an exemplary embodiment of the present disclosure is schematically shown. When scanning the tasks to be executed, splitting the tasks to be executed and adding them to the task scheduling set HashSet, it is necessary to first determine the number of idle schedulings in the task scheduling set. For example, if the specified initial capacity of the task scheduling set HashSet is 10, if the current number of elements in the HashSet set is already 10 (indicating that the set is full), new tasks are not allowed to be executed, the current scheduling is completed, and the next scheduled task is waiting.

[0077] If the number of elements added to the current HashSet is less than 10, and the task scheduling collection does not include the task scheduling identifier that currently needs to be added to the HashSet, the current task id to be executed can be added successfully and the task will be executed subsequently; if the number of elements added to the current HashSet is less than 10, and the task scheduling collection includes the task scheduling identifier that currently needs to be added to the HashSet, at this time, the current id to be executed cannot be added to the HashSet collection, and an addition failure prompt message is returned, indicating that the current task id has been added before and is being executed, then continue to wait for the next scheduled task to be executed.

[0078] The task scheduling set disclosed in the present invention has a specified initial capacity. The purpose of setting the specified initial capacity is to avoid the same machine from executing too many export tasks at the same time, which may lead to excessive memory usage, frequent FullGC or even high CPU, system downtime, etc. This specified initial capacity value can be adjusted according to the actual task situation and combined with the machine configuration. The present invention does not impose any special restrictions on the specific value of the specified initial capacity.

[0079] In an exemplary embodiment of the present disclosure, updating a task scheduling set according to a task execution result of a shard task includes: when the task execution result is successful execution, updating the task status of the shard task to successful execution; when the task execution result is failed execution, updating the task status to failed execution, and updating the current execution count of the shard task whose task status is failed execution; deleting a specified shard task in the task scheduling set, wherein the specified shard task is a shard task whose task status is successful execution or failed execution. The specified shard task may be a shard task whose task status is a specified status value.

[0080] In the present disclosure, after obtaining the task to be executed, the task status of the shard task is assigned to the "initial status". When processing the shard task based on the task scheduling set, if the task is executed successfully, the task status is updated to "executed successfully", and the subsequent scanning task will no longer scan the task; if the task fails to execute (for example, a query of an interface fails in the middle), the task status is updated to "executed failed", and the current execution count is increased by 1. When the task execution count is greater than or equal to the set maximum execution count (for example, 6), the subsequent scanning task will no longer scan the task. This avoids the repeated execution of a task due to some abnormal reasons, which occupies resources. In addition, for the specified shard task in the task scheduling set, that is, the shard task whose task status is successful execution or failed execution, the corresponding task id in the task set should be deleted when changing the status, so that the failed task can be scanned again later.

[0081] The above-mentioned task scheduling collection update method can achieve the purpose of preventing large tasks from blocking. For details, refer to Figure 4 , Figure 4 The schematic diagram of updating the task scheduling set according to the task execution results according to the exemplary implementation of the present disclosure is schematically shown. Assuming that 10 tasks are scanned at a time, 2 of which are large tasks, the 10 tasks are added to the HashSet and then executed. Among them, 8 tasks are quickly completed and will be cleared from the HashSet, thus freeing up 8 positions to add new tasks. Later, the new tasks to be executed can be added and executed when the scheduling scan is performed. Subsequent scheduling will not be unable to proceed because the 2 large tasks have not been completed. Therefore, the task scheduling set can achieve the anti-blocking effect of large tasks.

[0082] In an exemplary embodiment of the present disclosure, in response to a device update instruction of a task scheduling system, a newly added device is determined from the task scheduling system, the device update instruction being generated based on a device abnormality event or a device new addition event; a second total scheduling number corresponding to the newly added device is determined; and based on the second total scheduling number, sharded tasks in the task scheduling set are allocated to the newly added device.

[0083] The task scheduling system may be a system for executing distributed task scheduling. The device update instruction may be an instruction for instructing the task scheduling system to generate a new device. The new device may be a device newly added to the task scheduling system for task execution. The device abnormality event may be an event in which an existing device in the task scheduling system has an abnormality. The device new event may be an event in which a new device is added to the task scheduling system. The second total scheduling number may be the total scheduling number of tasks corresponding to the generation of the new device in the task scheduling system.

[0084] In a distributed task scheduling system, if an application crashes or restarts, the HashSet will be cleared. This disclosure regards the crash or restart of an application as an abnormal device event. When an abnormal device event occurs in the task scheduling system, the newly added devices in the device scheduling system will be determined to assign the unexecuted shard tasks to the newly added devices. As mentioned above, the pending task table does not have an executing status. If a machine crashes, its scheduled tasks will be automatically transferred to another machine (a function of elastic-job itself). The machine is regarded as a newly added device, and the unexecuted tasks will be scanned and executed by the newly added device.

[0085] In addition, when a system is released online, applications are usually started one by one. In the task scheduling system, starting a new application is considered a new device event. During the startup process, the elastic-job scheduling task number will be reallocated as the number of machines released increases. The newly started application can be considered a new device.

[0086] In the case of newly added devices due to application downtime or restart or new application startup, tasks can be assigned according to the second total scheduling number of the newly added devices. For example, when only one machine is started, all the 8 schedules above will be assigned to this machine. When another machine is started, 4 of the 8 schedules will be transferred to the new machine, and so on. Before all machines are started, the scheduling sequence number is being reallocated. Through the above task scheduling method, unsuccessful tasks can be assigned to newly added devices to complete all pending tasks.

[0087] In an exemplary embodiment of the present disclosure, a shard task in a task scheduling set is allocated to a newly added device, including: obtaining basic task information of the shard task; based on the basic task information, locking the shard task to obtain a locked shard task; and adding the locked shard task to the task scheduling set in the newly added device.

[0088] The basic task information may be specific information based on which the task is executed. The locked shard task may be a distributed task after adding an execution lock.

[0089] In the process of reallocating sharding tasks to newly added devices, since a memory anti-duplicate set is used, it is necessary to combine the basic information of the task to determine how to ensure that the same task ID is not scanned and executed by multiple machines. Based on the basic information of the task, determine whether the task itself has idempotency and other actual conditions, and determine whether the sharding task needs additional processing such as distributed locks. For example, the export task mentioned above does not need to consider this issue because the export task has idempotency. It is nothing more than exporting once more, which does not affect the final result.

[0090] For some scenarios that may require distributed locks for additional processing, HashSet collections can be replaced by some distributed collection tools, such as redis. However, the specific implementation method of using distributed locks is to use redis distributed locks before adding tasks to execute. If the lock can be added, it will be executed. If it cannot be added, it means that an application is executing the task, so the task will not be executed again. After the task is completed, the redis lock will be cleared.

[0091] The above locking scheme has the following problems: one is that the lock time is difficult to control. If the lock time is too long, it will not be automatically unlocked. If the lock time is too short, the purpose of locking cannot be achieved, which may cause repeated execution of tasks. The second most critical thing is that if a task is locked with redis and the machine crashes, the lock will be locked all the time (until the lock time is up), and the task may be suspended for a long time without any machine executing it. Therefore, in response to the above problems with distributed locks, relevant technicians can determine whether to lock a task based on the basic information of the task. If it is determined that a shard task needs to be locked, after obtaining the locked shard task, the locked shard task is added to the task scheduling collection in the newly added device so that the locked shard task can be scheduled and executed later. Locking the shard tasks that need to be locked according to the basic information of the task can avoid the situation where the same task is executed by multiple machines.

[0092] It should be noted that the terms "first", "second", etc. used in the present disclosure are only used to distinguish different total task scheduling numbers and should not impose any limitations on the present disclosure.

[0093] It is easy for those skilled in the art to understand that the distributed task scheduling framework in the present disclosure can also be implemented using other frameworks, such as the clover framework, the easy-job framework, the xxl-job framework, etc. Similarly, the anti-duplication collection can use the anti-duplication collection of java itself, or the anti-duplication collection of a third party, or the anti-duplication collection can be implemented by itself. The present disclosure does not make any special limitation on this.

[0094] In summary, the task processing method disclosed in the present invention obtains tasks to be executed, and the tasks to be executed are obtained by performing task scanning operations based on a pre-configured task scheduling cycle; the tasks to be executed are segmented and added to a task scheduling set, and the task scheduling set includes multiple shard tasks; the shard tasks are executed, and the task scheduling set is updated according to the task execution results of the shard tasks, so as to add new shard tasks to the updated task scheduling set. On the one hand, tasks are executed through the task scheduling set, and the task scheduling set is updated according to the task execution results, so that during the task execution process, there is no need to wait for all tasks of the current task batch to be executed before scheduling the next batch, which can effectively solve the problem of large task blocking and improve task processing efficiency. On the other hand, the task scheduling set usually has a specified initial capacity. During the parallel execution of tasks, it can effectively avoid the problem of machine downtime caused by the high memory usage of the machine due to the simultaneous execution of a large number of tasks. On the other hand, when there are new devices in the task scheduling system, unexecuted tasks can also be assigned to the new devices for subsequent scheduling processing.

[0095] It should be noted that, although the steps of the method of the present invention are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0096] In addition, in this exemplary embodiment, a task processing device is also provided. Figure 5 The task processing device 500 may include: a task acquisition module 510, a task segmentation module 520 and a task execution module 530.

[0097] Specifically, the task acquisition module 510 is used to obtain tasks to be executed, and the tasks to be executed are obtained by performing a task scanning operation based on a pre-configured task scheduling cycle; the task slicing module 520 is used to split the tasks to be executed and add them to a task scheduling set, which includes multiple slicing tasks; the task execution module 530 is used to execute slicing tasks, and update the task scheduling set according to the task execution results of the slicing tasks, so as to add new slicing tasks to the updated task scheduling set.

[0098] In an exemplary embodiment of the present disclosure, the task acquisition module 510 includes a task acquisition unit, which is used to: obtain a pre-configured task scheduling cycle, scan and process the task table to be executed based on the task scheduling cycle to obtain the task to be executed; obtain the task information table corresponding to the task to be executed, and the task information table is used to execute the task to be executed.

[0099] In an exemplary embodiment of the present disclosure, a task scheduling set has a specified initial capacity, and the task slicing module 520 includes a task slicing unit, which is used to: split the task to be executed to obtain multiple slicing tasks and slicing task identifiers of each slicing task; obtain a pre-configured first total scheduling number; determine the task scheduling identifier corresponding to each slicing task based on the first total scheduling number and the slicing task identifier; based on the task scheduling identifier and the specified initial capacity, add each slicing task to the task scheduling set, and the task scheduling identifier contained in the task scheduling set is unique.

[0100] In an exemplary embodiment of the present disclosure, the task slicing unit includes a task adding sub-unit, which is used to: determine the number of idle schedules in the task scheduling set based on the specified initial capacity; when the number of idle schedules is not zero and the task scheduling set does not include a task scheduling identifier, add the slicing task to the idle schedule of the task scheduling set.

[0101] In an exemplary embodiment of the present disclosure, the task execution module 530 includes a collection update unit, which is used to: when the task execution result is successful execution, update the task status of the shard task to successful execution; when the task execution result is failed execution, update the task status to failed execution, and update the current execution count of the shard task with the task status of failed execution; delete the specified shard task in the task scheduling collection, and specify the shard task as a shard task with a task status of successful execution or failed execution.

[0102] In an exemplary embodiment of the present disclosure, the task processing device 500 also includes a device addition module, which is used to: determine a new device from the task scheduling system in response to a device update instruction of the task scheduling system, the device update instruction is generated based on a device abnormal event or a device addition event; determine a second total scheduling number corresponding to the new device; based on the second total scheduling number, allocate the sliced ​​tasks in the task scheduling set to the new device.

[0103] In an exemplary embodiment of the present disclosure, the device addition module includes a task reallocation unit, which is used to: obtain basic task information of the shard task; based on the basic task information, lock the shard task to obtain a locked shard task; and add the locked shard task to the task scheduling collection in the newly added device.

[0104] The specific details of the virtual modules of each task processing device mentioned above have been described in detail in the corresponding task processing method. The undisclosed details can be found in the implementation method content of the method part, so they will not be repeated here.

[0105] It should be noted that although several modules or units of the task processing device are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0106] In addition, in an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0107] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware embodiments, complete software embodiments (including firmware, microcode, etc.), or embodiments combining hardware and software aspects, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0108] The exemplary embodiments of the present disclosure also provide a computer program product, which includes a computer program, and when the computer program is executed by a processor, the task processing method described above is implemented.

[0109] In one embodiment, the computer program product may be a tangible product containing the computer program, such as a computer-readable storage medium storing the computer program. Figure 6 , Figure 6 The schematic diagram of a computer-readable storage medium according to an exemplary embodiment of the present disclosure is schematically shown. The readable storage medium may be a storage medium based on electrical, magnetic, optical, electromagnetic, infrared, and other signals, including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory (Flash), mechanical hard disk (HDD), solid state drive (SSD), etc. Exemplarily, the computer program product may be implemented as a non-volatile storage medium storing a computer program, such as a read-only memory, a NAND flash memory (Nand Flash), etc.

[0110] In one embodiment, the computer program product may be an intangible product including a computer program. Exemplarily, the computer program product may be implemented as a virtual digital product, such as a digital file storing an executable file, an installation package, etc. of the computer program.

[0111] The code of the computer program can be written in one or more programming languages. Programming languages ​​such as C language, Java, C++, etc. The program code can be executed entirely on the user computing device, or partially on the user computing device, or as a separate software package, or partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, such as a local area network (LAN), a wide area network (WAN), etc., or can be connected to an external computing device (e.g., an Internet connection provided by an operator).

[0112] The computer program may be carried or transmitted via electrical, magnetic, optical, electromagnetic, infrared, or other signals. The electronic device may convert the signal carrying the computer program into a digital signal, thereby running the computer program. When the computer program is run on an electronic device, its code is used to enable the electronic device to execute (more specifically, enable the processor of the electronic device to execute) the method steps of various exemplary embodiments of the present disclosure, such as the task processing method described above.

[0113] The exemplary embodiments of the present disclosure also provide an electronic device, which may include a processor and a memory. The memory stores executable instructions of the processor, such as a computer program. The processor executes the method steps of various exemplary embodiments of the present disclosure by executing the executable instructions. In addition, the electronic device may also include a display for displaying a graphical user interface.

[0114] Reference below Figure 7 , the electronic device is exemplarily described in the form of a general-purpose computing device. It should be understood that Figure 7 The electronic device 700 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0115] like Figure 7 As shown, the electronic device 700 may include: a processor 710 , a memory 720 , a bus 730 , an I / O (input / output) interface 740 , a network adapter 750 , and a display 760 .

[0116] The memory 720 may include a volatile memory, such as a RAM 721, a cache unit 722, and may also include a non-volatile memory, such as a ROM 723. The memory 720 may also include one or more program modules 724, such program modules 724 include but are not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or a combination thereof may include the implementation of a network environment. For example, the program module 724 may include each module in the above-mentioned device.

[0117] The processor 710 may include one or more processing units. For example, the processor 710 may include an AP (Application Processor), a modem processor, a GPU (Graphics Processing Unit), an ISP (Image Signal Processor), a controller, an encoder, a decoder, a DSP (Digital Signal Processor), a baseband processor and / or an NPU (Neural-Network Processing Unit), etc.

[0118] The processor 710 may be used to execute executable instructions stored in the memory 720, such as executing the above-mentioned task processing method.

[0119] The bus 730 is used to realize the connection between different components of the electronic device 700, and may include a data bus, an address bus, and a control bus.

[0120] The electronic device 700 can communicate with one or more external devices 800 (eg, a keyboard, a mouse, an external controller, etc.) through the I / O interface 740 .

[0121] The electronic device 700 can communicate with one or more networks through the network adapter 750. For example, the network adapter 750 can provide mobile communication solutions such as 3G / 4G / 5G, or wireless communication solutions such as wireless LAN, Bluetooth, near field communication, etc. The network adapter 750 can communicate with other modules of the electronic device 700 through the bus 730.

[0122] The electronic device 700 may display a graphical user interface through the display 760 , such as an interface for displaying successfully executed tasks.

[0123] although Figure 7 Not shown, other hardware and / or software modules may also be provided in the electronic device 700, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0124] As can be seen from the above, the technical solution of the present disclosure can be implemented as a method, an apparatus, a system, a computer program product, a storage medium, an electronic device, etc. Those skilled in the art can understand that various aspects of the present disclosure can be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software, such as being respectively referred to as a "circuit", "module" or "system".

[0125] It should be understood that the present disclosure is not limited to the specific method steps or structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from the scope thereof. Those skilled in the art will easily think of other embodiments based on the specific embodiments provided by the present disclosure. Therefore, the specific embodiments provided by the present disclosure are only exemplary, and the scope and spirit of the present disclosure are indicated by the claims, and any variations, uses or adaptive changes of the present disclosure should be covered, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the technical field that are not disclosed in the present disclosure.

Claims

1. A task processing method, characterized in that: include: Acquire the tasks to be executed, wherein the tasks to be executed are acquired by performing a task scanning operation based on a pre-configured task scheduling period; Splitting the task to be executed and adding it to a task scheduling set, wherein the task scheduling set includes a plurality of split tasks; The slice task is executed, and the task scheduling set is updated according to the task execution result of the slice task, so as to add the new slice task to the updated task scheduling set.

2. The method according to claim 1, characterized in that The obtaining of tasks to be executed includes: Obtaining a pre-configured task scheduling cycle, scanning and processing a task table to be executed based on the task scheduling cycle, and obtaining the task to be executed; A task information table corresponding to the task to be executed is obtained, where the task information table is used to execute the task to be executed.

3. The method according to claim 2, characterized in that The to-be-executed task table includes one or more of the following fields: a primary key task identifier, an associated identifier associated with the task information table, a current execution count, a task status, a task to-be-executed time, and a shard task identifier; The task information table includes one or more of the following fields: task query conditions, task query content, task processing method, file name, operating user information and task function information.

4. The method according to claim 1, characterized in that: The task scheduling set has a specified initial capacity, and the splitting of the to-be-executed tasks and adding the splitting of the to-be-executed tasks to the task scheduling set includes: Splitting the task to be executed to obtain a plurality of slice tasks and a slice task identifier of each slice task; Obtain a preconfigured first total scheduling number; Determine the task scheduling identifier corresponding to each of the slice tasks according to the first total scheduling number and the slice task identifier; Based on the task scheduling identifier and the specified initial capacity, each of the slice tasks is added to the task scheduling set, and the task scheduling identifier included in the task scheduling set is unique.

5. The method according to claim 4, characterized in that The adding each of the shard tasks to the task scheduling set based on the task scheduling identifier and the specified initial capacity includes: Based on the specified initial capacity, determining the number of idle schedules in the task schedule set; When the idle schedule quantity is not zero and the task scheduling set does not include the task scheduling identifier, the slice task is added to the idle schedule of the task scheduling set.

6. The method according to claim 1, characterized in that The updating of the task scheduling set according to the task execution result of the shard task includes: When the task execution result is successful, the task status of the shard task is updated to successful execution; When the task execution result is execution failure, the task status is updated to execution failure, and the current execution count of the shard task whose task status is execution failure is updated; Delete the specified slice task in the task scheduling set, where the specified slice task is a slice task whose task status is successful execution or failed execution.

7. The method according to claim 1, characterized in that The method further comprises: In response to a device update instruction of a task scheduling system, determining a newly added device from the task scheduling system, wherein the device update instruction is generated based on a device abnormality event or a device newly added event; Determine a second total scheduling number corresponding to the newly added device; Based on the second total scheduling number, the slice tasks in the task scheduling set are allocated to the newly added device.

8. The method according to claim 7, characterized in that The allocating the slice tasks in the task scheduling set to the newly added device includes: Obtaining basic task information of the shard task; Based on the basic information of the task, the shard task is locked to obtain a locked shard task; Add the locked shard task to the task scheduling set in the newly added device.

9. A task processing device, characterized in that: include: A task acquisition module, used to acquire tasks to be executed, wherein the tasks to be executed are acquired by performing a task scanning operation based on a pre-configured task scheduling period; A task slicing module is used to slice the task to be executed and add it to a task scheduling set, wherein the task scheduling set includes multiple slice tasks; The task execution module is used to execute the slice task and update the task scheduling set according to the task execution result of the slice task, so as to add the new slice task to the updated task scheduling set.

10. An electronic device, characterized in that: include: processor; as well as A memory having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by the processor, implement the task processing method as claimed in any one of claims 1 to 8.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the task processing method according to any one of claims 1 to 8 is implemented.

Images