A task distribution system for big data task crawling

Abstract

The present invention discloses a task distribution system for big data task capture, which relates to the field of information technology, including: a task aggregator, which aggregates pending tasks uploaded from various task request terminals and attaches task type identifiers; a task type statistics module, which extracts the task type identifiers of pending tasks, counts the number and proportion of each task type, and delivers the pending tasks to a message queue; a message queue, which provides a buffer for temporarily storing pending tasks; a task execution device, which executes pending tasks; a performance testing module, which tests the processing capabilities of each task execution device for different task types, and classifies the task execution devices into corresponding device types according to equal proportions; a load balancer, which distributes pending tasks in the message queue to task execution devices of corresponding types. The present invention distinguishes the processing capabilities of different types of tasks based on the task processing device, so that the performance of the overall system is optimized.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on application number 2021105075964, application date: May 10, 2021, and the name of the invention is: A task distribution system based on big data analysis. Technical Field

[0003] The present invention relates to the field of information technology, and in particular to a task distribution system for capturing big data tasks. Background Art

[0004] With the rapid development of mobile Internet technology and the continuous upgrading of the network, when faced with large amounts of data task processing, if the serial task processing method is used, the task processing time will be prolonged, so the task is generally processed in parallel. In order to improve the effect of task distribution processing, higher requirements are also placed on parallel processing capabilities.

[0005] For example, the invention patent application with the publication number CN103186418A discloses a task distribution method and system, which sets multiple task processing devices to receive tasks only when they are idle. When the number of tasks is greater than the number of task processing devices, each task processing device is processing tasks, and there is no idle task processing device. Therefore, it can be considered that the distribution of tasks meets the requirements of load balancing for each task processing device. Although this method meets the requirements of load balancing in principle, there is only one task to be processed in its task processor at a time. The same task processor must interact with the message directory between any two adjacent task processings. In this process, task processing will inevitably be interrupted, resulting in a reduction in the overall efficiency of task processing. In addition, the existing technology only considers the overall performance of the task processing device, but those skilled in the art know that tasks are divided into different types. For example, in the field of smart parks, different types of tasks such as face recognition, billing, navigation, statistics, etc. are included, and the processing capabilities of different types of tasks are different for different task processing devices. Therefore, in a distributed system, if you want to optimize the overall performance of the task distribution system, you are in urgent need of a task distribution system that can consider the processing capabilities of each task processing device for different types. Summary of the invention

[0006] The purpose of the present invention is to provide a task distribution system for big data task capture, which distinguishes the processing capabilities of different types of tasks based on the task processing device, so as to optimize the performance of the overall system.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] A task distribution system for big data task capture, including

[0009] A task aggregator collects pending tasks uploaded from various task requesting terminals and attaches a task type identifier to the pending tasks according to the task requesting terminals;

[0010] The task type statistics module extracts the task type identifiers of pending tasks, counts the number and proportion of each task type, and delivers pending tasks to the message queue;

[0011] Message queue, which provides a buffer for temporarily storing pending tasks;

[0012] Task execution devices, a number of which are used to execute the tasks to be processed;

[0013] The performance test module tests the processing capability of each task execution device for different task types, and classifies the task execution devices into corresponding device types in equal proportions based on the priority of the task type and the statistical proportion of each task type;

[0014] The load balancer distributes the pending tasks in the message queue to the task execution devices of the corresponding types.

[0015] Furthermore, in the performance test module, the test method for the processing capability of different task categories is as follows:

[0016] For each task type, a test task set containing a certain number of test tasks is generated, and the test task set is distributed to each task execution device respectively. The time taken by each task execution device to complete the test task set is counted, and the amount of tasks of this task type completed by each task execution device per unit time is calculated as the processing capacity of the task execution device for this task type.

[0017] Furthermore, in the performance test module, the classification method of the task execution device is to perform the following processing in order of the priority of the task type from high to low:

[0018] Calculate the product of the proportion of task types of the current priority and the total number of task execution devices as the number k of corresponding device types, take the top k task execution devices with the strongest processing capabilities for the task type of this priority among the task execution devices with unidentified device types, and attach the corresponding device type identification.

[0019] Furthermore, the performance testing module allocates at least one spare task processing device when classifying the device types.

[0020] Furthermore, the generation method of the backup task processing device is as follows:

[0021] KX1, rounding when solving for the quantity k;

[0022] KX2, after all device types are assigned, there are a number of unassigned task execution devices left;

[0023] KX3, compares the unassigned task execution devices with the preset number of spare task processing devices; if the unassigned task execution devices are greater than the preset number of spare task processing devices, enter KX4; if the unassigned task execution devices are less than the preset number of spare task processing devices, enter KX5;

[0024] KX4, in descending order of priority of task type, sequentially polls the unassigned task execution devices with the strongest processing capability of the corresponding task type and attaches the corresponding device type identifier until the number of unassigned task execution devices is equal to the preset number of standby task processing devices;

[0025] KX5, in descending order of priority of task type, polls the task execution devices of the marked device type in turn, selects the device with the weakest processing capability of the corresponding task type, and removes its identification until the number of unassigned task execution devices is equal to the preset number of spare task processing devices;

[0026] KX6, attach spare identification to unassigned mission execution devices.

[0027] Furthermore, the allocation steps of the load balancer are as follows:

[0028] FP1, for the same device type, calculate the proportion of the processing capacity of each task execution device to the total processing capacity of the task execution devices of the device type;

[0029] FP2, for the task type corresponding to the device type, the product of the number of the task type in the tasks to be processed and the proportion obtained in FP1 is used as the upper limit of the task volume of the task execution device to process the batch of tasks;

[0030] FP3, after the load balancer extracts the pending tasks from the message queue, it will poll and assign them in turn to different task execution devices of the corresponding device type and which have not reached the upper limit of the task volume according to the task type. When a task execution device reaches the upper limit of the task volume, it will be removed from the polling allocation queue.

[0031] Furthermore, after processing each task, the task execution device sends feedback to the message queue via the load balancer, and after receiving the feedback, the message queue deletes the corresponding task to be executed.

[0032] Furthermore, the load balancer monitors the completion status of each task execution device, and when a task execution device completes all pending tasks, it grabs pending tasks from task execution devices of the same device category.

[0033] Furthermore, the method for grabbing pending tasks from task execution devices of the same device category is as follows: calculate the estimated remaining completion time in the task execution devices of the same device category that have pending tasks, compare the estimated remaining completion time with the set first time threshold, and if there is one or several task execution devices whose estimated remaining completion time is greater than the first time threshold, then transfer the pending tasks in the corresponding task execution devices to the task execution devices that have completed all pending tasks in order of the estimated remaining completion time from large to small, until the estimated remaining completion time of the task execution devices that have completed all pending tasks reaches the first time threshold.

[0034] Compared with the prior art, the beneficial effect of the present invention is that when performing load balancing of tasks, the present invention fully considers the processing capabilities of different task execution devices for different types of tasks, and selects the optimal balancing solution from high to low according to the set task priority, thereby avoiding performance waste of the task execution device and further improving the overall performance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a system framework diagram of the present invention.

[0036] Figure 2 This is a flow chart of a method for generating a backup task processing device according to an embodiment of the present invention.

[0037] Figure 3 The present invention is a flowchart of a method for allocating pending tasks in a message queue according to an embodiment of the present invention.

[0038] Figure 4 The present invention is a flowchart of a method for capturing tasks to be processed according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] See also Figure 1 The present invention provides a task distribution system for big data task capture, comprising

[0041] The task aggregator collects pending tasks uploaded from each task request end, and attaches a task type identifier to the pending tasks according to the task request end. For example, when applied to a smart park, a face recognition, fingerprint recognition and other clock-in machines, a license plate recognition camera are set at the entrance of the park, and a corresponding vehicle parking metering system is set at the exit of the park, etc., which will generate a variety of tasks. Generally speaking, the task types generated by each task request end are of the same type. Therefore, the present invention distinguishes the types of pending tasks according to the task request end and attaches the corresponding task type identifier.

[0042] The task type statistics module extracts the task type identifier of the pending tasks, counts the number and proportion of tasks of each task type, and delivers the pending tasks to the message queue. In one embodiment, the same batch of pending tasks includes 1000 {R1, R2, ...RN}, where N is the total number of pending tasks, and in this embodiment, N=1000. There are 5 types of tasks, and the task type identifiers are A, B, C, D, and E, respectively. The corresponding number of tasks is 150, 200, 250, 100, and 300, respectively; then their proportions ZBj are 0.15, 0.2, 0.25, 0.1, and 0.3, respectively; where j represents the task type, j∈(A, B, C, D, E).

[0043] Message queue, providing a buffer for temporarily storing pending tasks {R1, R2, ...RN};

[0044] Task execution devices, a number of which include {Z1, Z2, ...ZM}, where M is the total number of task execution devices, used to execute the tasks to be processed;

[0045] The performance test module tests the processing capabilities of each task execution device for different task types, and classifies the task execution devices into corresponding device types in equal proportions based on the priority of the task type and the statistical proportion of each task type.

[0046] Among them, the testing method for the processing capability of different task categories is as follows:

[0047] For each task type (A, B, C, D, E), a test task set containing a certain number of test tasks is generated, and the test task set is distributed to each task execution device respectively. The time Tij for each task execution device to complete the test task set is counted, where i represents the serial number of the task execution device, i=[1,M], j∈(A, B, C, D, E). The amount of tasks of the task type j completed by each task execution device i in unit time (generally the number of tasks is used as the task amount) is calculated as the processing capacity of the task execution device i for the task type j, recorded as NLij.

[0048] The classification method of the task execution device is to perform the following processing in order of the priority of the task type from high to low; it is worth mentioning that the priority of the task type is preset manually.

[0049] The product of the proportion ZBj of the current priority task type j and the total number M of task execution devices is calculated as the number k of the corresponding device type, and a preliminary balance is achieved based on the proportion of task types. There are differences in priority between different tasks. For example, some real-time tasks, such as identification and billing, need to be carried out as soon as possible, otherwise it will affect the entry and exit of personnel, while some background tasks, such as report statistics, are relatively less demanding in terms of real-time performance. Therefore, in order to ensure that tasks with higher real-time performance requirements can be processed and completed in the first time, this embodiment takes the top k task execution devices with the strongest processing capabilities for the task type of the priority among the task execution devices with unidentified device types, and attaches the corresponding device type identification; the device type identification corresponds one-to-one to the task type identification.

[0050] It is well known in the art that the number k must be an integer, so when actually calculating the number k, this effect can be achieved by rounding off or rounding off.

[0051] Furthermore, the task execution device will inevitably fail during the task execution process, and the remaining tasks cannot be carried out. Therefore, it is necessary to set up a small number of spare task processing devices, which do not affect the overall progress and are only used when one or several task processing devices have accidental failures. Therefore, the present invention allocates at least one spare task processing device when the performance test module classifies the device type. Please refer to Figure 2 , the generation method of the backup task processing device is as follows:

[0052] KX1, rounding is performed when solving the quantity k; in the actual operation process, there will be at least one task execution server remaining.

[0053] KX2, after all device types are assigned, there are a number of unassigned task execution devices left;

[0054] KX3 compares the unassigned task execution devices with the preset number of spare task processing devices (determined by the staff according to the actual situation); if the unassigned task execution devices are greater than the preset number of spare task processing devices, enter KX4; if the unassigned task execution devices are less than the preset number of spare task processing devices, enter KX5;

[0055] KX4, this requires attaching device types to the redundant unassigned task execution devices and using them as corresponding task processing to further improve the overall performance of the system. In order of the priority of the task type from high to low, the unassigned task execution devices with the strongest processing capabilities corresponding to the task type are polled in sequence and attached with the corresponding device type identifier until the number of unassigned task execution devices equals the pre-set number of spare task processing devices. The device with the strongest processing capabilities corresponding to its task type is attached with a device type, and accordingly, the weaker performance is used as a backup, and the overall processing performance is improved.

[0056] KX5, in order of priority of task type from high to low, polls the task execution devices of the marked device type in sequence, selects the one with the weakest processing capability of the corresponding task type, removes its identification until the number of unassigned task execution devices is equal to the pre-set number of backup task processing devices. Contrary to KX4, when a part of the task execution devices of the marked device type needs to be removed as backup, it is preferred to select the one with weaker processing capability, so as to ensure the overall processing performance.

[0057] KX6 attaches a standby mark to the unassigned task execution devices. When one or several task processing devices fail to complete tasks for a long time, it is considered that they have crashed, and the tasks in progress are transferred to the standby task execution devices to ensure that the tasks can proceed normally.

[0058] The load balancer distributes the pending tasks in the message queue to the task execution devices of the corresponding type. Figure 3 , the specific allocation steps are as follows:

[0059] FP1, for the same device type, calculate the proportion of the processing capacity of each task execution device to the total processing capacity of the task execution devices of the device type;

[0060] FP2, for the task type corresponding to the device type, the product of the number of the task type in the tasks to be processed and the proportion obtained in FP1 is used as the upper limit of the task volume of the task execution device to process the batch of tasks;

[0061] FP3, after the load balancer extracts the pending tasks from the message queue, it will poll and assign them in turn to different task execution devices of the corresponding device type and which have not reached the upper limit of the task volume according to the task type. When a task execution device reaches the upper limit of the task volume, it will be removed from the polling allocation queue.

[0062] The allocation method can ensure that, after the initial allocation, the completion time of each task execution device for the allocated tasks to be processed is approximately equal.

[0063] It is worth mentioning that after the task execution device completes processing each task, it sends feedback to the message queue through the load balancer. After receiving the feedback, the message queue deletes the corresponding task to be executed.

[0064] People in this field are well aware that in the process of task processing devices executing tasks, their execution environment will change, such as temperature, humidity, voltage, etc. Therefore, after the initial allocation, the actual time for the task execution devices to complete the tasks is not exactly the same as the completion time we expected, which may cause one or several task execution devices to be faster while other task execution devices are slower.

[0065] Based on the above application, this application proposes a new capture mechanism: the load balancer monitors the completion status of each task execution device, and when a task execution device completes all pending tasks, it captures pending tasks from task execution devices of the same device category. The method for capturing pending tasks is as follows: calculate the estimated remaining completion time in the task execution devices of the same device category that have pending tasks, and compare the estimated remaining completion time with the set first time threshold. The first time threshold is the time allowed by the staff. Within the first time threshold, it does not affect the overall task processing progress. If there is one or several task execution devices whose estimated remaining completion time is greater than the first time threshold, the pending tasks in the corresponding task execution devices are transferred to the task execution devices that have completed all pending tasks in the order of the estimated remaining completion time from large to small, until the estimated remaining completion time of the task execution device that has completed all pending tasks reaches the first time threshold.

[0066] For example: set the first time threshold to 10s,

[0067] When there are task execution devices of the same device category, one of them completes all pending tasks first;

[0068] Then, the estimated remaining completion time of other task execution devices of the same device category is calculated; the estimated remaining completion time is calculated as: the remaining tasks to be processed divided by the processing capacity of the corresponding task type;

[0069] Grab a certain amount of pending tasks from the task execution device with the largest estimated remaining completion time and transfer them to the task execution device that has completed all pending tasks; please refer to Figure 4 , the amount of crawling is calculated by:

[0070] ZQ1, calculate the expected amount of tasks pi that can be completed by each task execution device within the next first time threshold period; the difference between the actual unfinished pending task amount qi and the task amount pi is taken as the overflow value si of the task execution device, where i is the serial number of the task execution device;

[0071] ZQ2, sort the overflow values ​​si of each task execution device;

[0072] ZQ3, calculate the task volume p0 that the task execution device that has completed all pending tasks is expected to complete in the next first time threshold period; compare the task volume p0 with the maximum value smax of the overflow value si, if the task volume p0 is less than smax, then only grab the pending tasks in the task execution device corresponding to the maximum value smax of the overflow value si to the task execution device that has completed all pending tasks, and the grabbed volume is p0;

[0073] ZQ4, if the task volume p0 is greater than smax, grab the overflowed pending tasks in the task execution device corresponding to the maximum value smax of the overflow value si and transfer them to the task execution device that has completed all pending tasks.

[0074] ZQ5, repeat steps ZQ2 to ZQ3 until the total amount of captured tasks reaches the amount of tasks p0 that the task execution device that has completed all pending tasks is expected to complete in the next first time threshold time period.

[0075] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, the embodiments should be considered exemplary and non-restrictive in all respects, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims be included in the present invention.

Claims

1. A task distribution system for big data task capture, characterized in that: include A task aggregator collects pending tasks uploaded from various task requesting terminals and attaches a task type identifier to the pending tasks according to the task requesting terminals; The task type statistics module extracts the task type identifiers of pending tasks, counts the number and proportion of each task type, and delivers pending tasks to the message queue; Message queue, which provides a buffer for temporarily storing pending tasks; Task execution devices, a number of which are used to execute the tasks to be processed; The performance test module tests the processing capability of each task execution device for different task types, and classifies the task execution devices into corresponding device types in equal proportions based on the priority of the task type and the statistical proportion of each task type; The load balancer distributes the pending tasks in the message queue to the task execution devices of the corresponding types; The load balancer monitors the completion status of each task execution device, and when a task execution device completes all pending tasks, it grabs pending tasks from task execution devices of the same device category; The method for grabbing pending tasks from task execution devices of the same device category is as follows: calculating the estimated remaining completion time of the task execution devices of the same device category that have pending tasks, comparing the estimated remaining completion time with a set first time threshold, and if there is one or more task execution devices whose estimated remaining completion time is greater than the first time threshold, transferring the pending tasks in the corresponding task execution devices to the task execution devices that have completed all pending tasks in descending order of the estimated remaining completion time, until the estimated remaining completion time of the task execution devices that have completed all pending tasks reaches the first time threshold; In the performance test module, the test method for the processing capability of different task categories is as follows: For each task type, a test task set containing a certain number of test tasks is generated, and the test task set is distributed to each task execution device respectively, and the time taken by each task execution device to complete the test task set is counted, and the amount of tasks of the task type completed by each task execution device per unit time is calculated as the processing capacity of the task execution device for the task type; In the performance test module, the classification method of the task execution device is to perform the following processing in order of the priority of the task type from high to low: Calculate the product of the proportion of the task type of the current priority and the total number of task execution devices as the number k of the corresponding device type, select the top k task execution devices with the strongest processing capability of the task type of the priority among the task execution devices without device type identification, and attach the corresponding device type identification; The performance testing module allocates at least one spare task processing device when classifying the device type; The generation method of the standby task processing device is as follows: KX1, rounding when solving for the quantity k; KX2, after all device types are assigned, there are a number of unassigned task execution devices left; KX3, compares the unassigned task execution devices with the preset number of spare task processing devices; if the unassigned task execution devices are greater than the preset number of spare task processing devices, enter KX4; if the unassigned task execution devices are less than the preset number of spare task processing devices, enter KX5; KX4, in descending order of priority of task type, sequentially polls the unassigned task execution devices with the strongest processing capability of the corresponding task type and attaches the corresponding device type identifier until the number of unassigned task execution devices is equal to the preset number of standby task processing devices; KX5, in descending order of priority of task type, polls the task execution devices of the marked device type in turn, selects the device with the weakest processing capability of the corresponding task type, and removes its identification until the number of unassigned task execution devices is equal to the preset number of spare task processing devices; KX6, attach spare identification to unassigned mission execution devices; The allocation steps of the load balancer are as follows: FP1, for the same device type, calculate the proportion of the processing capacity of each task execution device to the total processing capacity of the task execution devices of the device type; FP2, for the task type corresponding to the device type, the product of the number of the task type in the tasks to be processed and the proportion obtained in FP1 is used as the upper limit of the task volume of the task execution device; FP3, after the load balancer extracts each task to be processed from the message queue, it will poll and allocate it in turn to different task execution devices of the corresponding device type and which have not reached the upper limit of the task volume according to the task type. When a task execution device reaches the upper limit of the task volume, it will be removed from the polling allocation queue; After the task execution device completes processing each task, it sends feedback to the message queue via the load balancer. After receiving the feedback, the message queue deletes the corresponding task to be executed.

Images