Big data task processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a big data task processing method and device, electronic equipment and a storage medium, and relates to the technical field of big data tasks. The method comprises the steps of detecting whether a current to-be-executed task meets a preset shunting condition or not; each current to-be-executed task needs to use the same data table during processing; and when the current to-be-executed task meets the preset shunting condition, determining the to-be-processed task which needs to be processed currently from the current to-be-executed tasks, processing the currently determined to-be-processed task, and returning to execute the step of detecting whether the current to-be-executed task meets the preset shunting condition or not. The current to-be-executed task does not meet the preset shunting condition; the determined number of the to-be-processed tasks needing to be processed currently is smaller than the total number of the to-be-executed tasks currently; when the current to-be-executed tasks do not meet the preset shunting condition, the current to-be-executed tasks are processed at the same time, and the efficiency of processing the multiple to-be-executed tasks can be improved.

Description

Technical Field

[0001] The present invention relates to the field of big data task technology, and in particular to a big data task processing method, device, electronic device and storage medium. Background Art

[0002] The server can perform statistical analysis and other processing on the acquired big data, and the analysis results can be recorded in the data table. Later, the server can use the data table to process big data offline tasks.

[0003] For example, a user can request a video from a server through an application installed on a mobile device. In the server that provides video services, big data may include user behavior data; data tables obtained by analyzing user behavior data may include: a play list that records videos that users have viewed, an interaction list that records user likes, forwarding, and dislikes on videos, a click list that records user clicks on advertisements provided by the server, and a startup list that records the time and duration of user application startups. Big data offline tasks may include: a retention analysis task that analyzes user retention rates, a preference analysis task that analyzes user preferences, and an advertising analysis task that analyzes the effects of advertising delivered to users. When processing retention analysis tasks, play lists and startup lists are required; when processing preference analysis tasks, play lists and interaction lists are required; when processing advertising analysis tasks, play lists and click lists are required.

[0004] As can be seen from the above example, the playlist is required when processing multiple big data offline tasks. Assuming that all data tables except the playlist have been generated, the multiple big data offline tasks need to wait for the playlist to be generated; after the playlist is generated, the multiple big data offline tasks start processing.

[0005] However, the computing resources used to process big data offline tasks on the server are limited. When the server processes multiple big data offline tasks at the same time, the limited computing resources will be allocated to multiple big data offline tasks, and each big data offline task will be allocated less computing resources. This may cause the processing speed of each big data offline task to be extremely slow, and then it will take a long time to complete the multiple big data offline tasks, that is, the efficiency of processing the multiple big data offline tasks is too low. Summary of the invention

[0006] The purpose of the embodiments of the present invention is to provide a method, device, electronic device and storage medium for processing big data tasks to improve the efficiency of processing multiple tasks to be executed. The specific technical solution is as follows:

[0007] In the first aspect of the implementation of the present invention, a big data task processing method is first provided, the method comprising: detecting whether the current task to be executed meets the preset diversion condition; wherein each current task to be executed needs to use the same data table when processing; when the current task to be executed meets the preset diversion condition, determining the current task to be processed from the current tasks to be executed, processing the currently determined task to be processed at the same time, and returning to execute the step of detecting whether the current task to be executed meets the preset diversion condition, until the current task to be executed does not meet the preset diversion condition; wherein the number of tasks to be processed that are determined to be currently required to be processed is less than the total number of tasks to be executed; when the current task to be executed does not meet the preset diversion condition, processing the current tasks to be executed at the same time.

[0008] Optionally, the detecting whether the current task to be executed satisfies the preset diversion condition includes: based on the current number of tasks to be executed, detecting whether the current task to be executed satisfies the number diversion condition, and based on the current number of categories of tasks to be executed, detecting whether the current task to be executed satisfies the category diversion condition; wherein, tasks to be executed that have the same number of data tables used during execution belong to the same category; if the current task to be executed satisfies the number diversion condition and satisfies the category diversion condition, it is determined that the current task to be executed satisfies the preset diversion condition; if the current task to be executed does not satisfy the number diversion condition and / or does not satisfy the category diversion condition, it is determined that the current task to be executed does not satisfy the preset diversion condition.

[0009] Optionally, based on the current number of tasks to be executed, detecting whether the current tasks to be executed meet the number diversion condition includes: detecting whether the current number of tasks to be executed is greater than a congestion threshold; if the current number of tasks to be executed is greater than the congestion threshold, determining that the current tasks to be executed meet the number diversion condition; if the current number of tasks to be executed is not greater than the congestion threshold, determining that the current tasks to be executed do not meet the number diversion condition.

[0010] Optionally, the detecting whether the current tasks to be executed meet the number diversion condition based on the current number of tasks to be executed includes: obtaining the allocatable resource amount of the current allocatable computing resources, the current number of tasks to be executed, and the designated resource amount corresponding to each current task to be executed; wherein the designated resource amount corresponding to a task to be executed indicates: the resource amount required when the processing speed of the task to be executed is greater than the speed threshold; calculating the estimated resource amount allocated to each current task to be executed in the allocatable resource amount according to the weight corresponding to each current task to be executed; for each current task to be executed, calculating the ratio of the estimated resource amount allocated to the task to be executed and the designated resource amount corresponding to the task to be executed, and obtaining the resource ratio corresponding to the task to be executed; from the current tasks to be executed, determining the task to be executed whose corresponding resource ratio is less than the first ratio as the expected congestion task; if the proportion of each expected congestion task in all the current tasks to be executed is greater than the second ratio, determining that the current task to be executed meets the number diversion condition; if the proportion of each expected congestion task in all the current tasks to be executed is not greater than the second ratio, determining that the current task to be executed does not meet the number diversion condition.

[0011] Optionally, the detecting whether the current task to be executed meets the category diversion condition based on the number of categories of the current task to be executed includes: detecting whether the number of categories of the current task to be executed is not less than a diversion threshold; if the number of categories of the current task to be executed is not less than the diversion threshold, determining that the current task to be executed meets the category diversion condition; if the number of categories of the current task to be executed is less than the diversion threshold, determining that the current task to be executed does not meet the category diversion condition.

[0012] Optionally, before detecting whether the current task to be executed meets the preset diversion conditions, the method also includes: for each data table needed to be used when executing the big data task, when the expected generation time of the data table is reached but the data table has not been generated, when the data table is generated, from the major data tasks, determine the big data task that needs to use the data table for processing and all the data tables required for processing except the data table have been generated, as the task to be executed.

[0013] Optionally, before detecting whether the current task to be executed meets the preset diversion conditions, the method also includes: when the expected generation time of the data table is reached and the data table has been generated, when the data table is generated, from the major data tasks, determine the big data tasks that need to use the data table for processing and in which all other data tables required for processing except the data table have been generated, and process the determined big data tasks at the same time.

[0014] Optionally, determining the pending tasks that currently need to be processed from the current pending tasks includes: dividing the current pending tasks into a specified number of pending task groups; or, according to the categories of the current pending tasks, treating pending tasks of the same category as a pending task group; wherein pending tasks with the same number of data tables used during execution belong to the same category; and determining the pending tasks included in a pending task group as pending tasks that currently need to be processed.

[0015] Optionally, in the case where multiple pending task groups are obtained according to the categories of each pending task, determining the pending tasks included in a pending task group as the pending tasks that currently need to be processed includes: determining the processing order corresponding to each pending task group according to the size relationship of the number of tables corresponding to each pending task group; and determining the pending tasks included in the first pending task group in the processing order as the pending tasks that currently need to be processed.

[0016] Optionally, there are multiple pending tasks that need to be processed currently determined from the current pending tasks; the simultaneous processing of the currently determined pending tasks includes: allocating currently available computing power resources to the currently determined pending tasks, and processing the pending tasks by using the computing power resources allocated to each pending task; when there are pending tasks that have been processed among the currently determined pending tasks, allocating the computing power resources allocated to the pending tasks that have been processed to the unprocessed pending tasks, until all the currently determined pending tasks have been processed.

[0017] Optionally, the allocating of currently allocable computing power resources to each currently determined task to be processed includes: for each currently determined task to be processed, calculating the computing power resources allocated to the task to be processed in the currently allocable computing power resources according to the weight corresponding to the task to be processed, and allocating the calculated computing power resources to the task to be processed.

[0018] In a second aspect of the present invention, a big data task processing device is provided, the device comprising:

[0019] A detection module is used to detect whether the current pending tasks meet the preset diversion conditions; wherein each current pending task needs to use the same data table when processing;

[0020] The first processing module is used to determine the pending tasks that need to be processed currently from the current pending tasks when the current pending tasks meet the preset diversion condition, process the currently determined pending tasks at the same time, and return to execute the step of detecting whether the current pending tasks meet the preset diversion condition until the current pending tasks do not meet the preset diversion condition; wherein the number of the currently determined pending tasks that need to be processed is less than the total number of the current pending tasks;

[0021] The second processing module is used to simultaneously process the current tasks to be executed when the current tasks to be executed do not meet the preset diversion condition.

[0022] Optionally, the detection module is specifically used to: detect whether the current task to be executed satisfies the number diversion condition based on the current number of tasks to be executed, and detect whether the current task to be executed satisfies the category diversion condition based on the current number of categories of tasks to be executed; wherein, tasks to be executed that have the same number of data tables used during execution belong to the same category; if the current task to be executed satisfies the number diversion condition and satisfies the category diversion condition, determine that the current task to be executed satisfies the preset diversion condition; if the current task to be executed does not satisfy the number diversion condition and / or does not satisfy the category diversion condition, determine that the current task to be executed does not satisfy the preset diversion condition.

[0023] Optionally, the detection module is specifically used to: detect whether the current number of tasks to be executed is greater than a congestion threshold; if the current number of tasks to be executed is greater than the congestion threshold, determine that the current tasks to be executed meet the number diversion condition; if the current number of tasks to be executed is not greater than the congestion threshold, determine that the current tasks to be executed do not meet the number diversion condition.

[0024] Optionally, the detection module is specifically used to: obtain the allocatable resource amount of the currently allocatable computing power resources, the current number of tasks to be executed, and the designated resource amount corresponding to each current task to be executed; wherein the designated resource amount corresponding to a task to be executed indicates: the resource amount required when the processing speed of the task to be executed is greater than a speed threshold; calculate the estimated resource amount allocated to each current task to be executed in the allocatable resource amount according to the weights corresponding to each current task to be executed; for each current task to be executed, calculate the ratio of the estimated resource amount allocated to the task to be executed and the designated resource amount corresponding to the task to be executed, and obtain the resource ratio corresponding to the task to be executed; from the current tasks to be executed, determine the task to be executed whose corresponding resource ratio is less than the first ratio as the estimated congestion task; if the proportion of each expected congestion task in all current tasks to be executed is greater than the second ratio, determine that the current task to be executed meets the number diversion condition; if the proportion of each expected congestion task in all current tasks to be executed is not greater than the second ratio, determine that the current task to be executed does not meet the number diversion condition.

[0025] Optionally, the detection module is specifically used to: detect whether the number of categories of the current tasks to be executed is not less than the diversion threshold; if the number of categories of the current tasks to be executed is not less than the diversion threshold, determine that the current tasks to be executed meet the category diversion conditions; if the number of categories of the current tasks to be executed is less than the diversion threshold, determine that the current tasks to be executed do not meet the category diversion conditions.

[0026] Optionally, the device also includes: a third processing module, which is used to execute each data table needed to be used when executing the big data task before the detection module detects whether the current task to be executed meets the preset diversion conditions. When the expected generation time of the data table is reached but the data table has not been generated, when the data table is generated, from the major data tasks, determine the big data task that needs to use the data table for processing and all the data tables required for processing except the data table have been generated, as the task to be executed.

[0027] Optionally, the device also includes: a fourth processing module, which is used to execute, before the detection module executes to detect whether the current task to be executed meets the preset diversion condition, when the expected generation time of the data table is reached and the data table has been generated, determine from the major data tasks when the data table needs to be used for processing and all other data tables required for processing except the data table have been generated, and simultaneously process the determined big data tasks.

[0028] Optionally, the first processing module is specifically used to: divide each current task to be executed into a specified number of task groups to be processed; or, according to the category of each current task to be executed, group the tasks to be executed of the same category into a task group to be processed; wherein, tasks to be executed that have the same number of data tables used during execution belong to the same category; and determine the tasks to be executed included in a task group to be processed as the tasks to be processed that currently need to be processed.

[0029] Optionally, in the case where multiple pending task groups are obtained according to the categories of each pending task, the first processing module is specifically used to: determine the processing order corresponding to each pending task group according to the size relationship of the number of tables corresponding to each pending task group; determine the pending tasks included in the first pending task group in the processing order as the pending tasks that currently need to be processed.

[0030] Optionally, there are multiple pending tasks that need to be processed currently determined from the current pending tasks; the first processing module is specifically used to: allocate currently allocable computing power resources to each currently determined pending task, and process the pending task by using the computing power resources allocated to each pending task; when there are pending tasks that have been processed among the currently determined pending tasks, allocate the computing power resources allocated to the pending tasks that have been processed to the uncompleted pending tasks, until all the currently determined pending tasks have been processed.

[0031] Optionally, the first processing module is specifically used to: for each task to be processed currently determined, calculate the computing power resources allocated to the task to be processed from the currently allocable computing power resources according to the weight corresponding to the task to be processed, and allocate the calculated computing power resources to the task to be processed.

[0032] In a third aspect of the present invention, there is provided an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0033] Memory, used to store computer programs;

[0034] The processor is used to implement any step of the big data task processing method described in the first aspect when executing the program stored in the memory.

[0035] In another aspect of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any of the above-mentioned big data task processing methods is implemented.

[0036] In another aspect of the present invention, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute any of the above-mentioned big data task processing methods.

[0037] An embodiment of the present invention provides a big data task processing method, in which an electronic device detects whether a current task to be executed meets a preset diversion condition; wherein each current task to be executed needs to use the same data table when processing; when the current task to be executed meets the preset diversion condition, the task to be processed that currently needs to be processed is determined from the current tasks to be executed, the currently determined task to be processed is processed, and the step of detecting whether the current task to be executed meets the preset diversion condition is returned to execute until the current task to be executed does not meet the preset diversion condition; wherein the number of tasks to be processed that are determined to currently need to be processed is less than the total number of tasks to be executed currently; when the current task to be executed does not meet the preset diversion condition, each current task to be executed is processed.

[0038] Based on the above processing, when the current tasks to be executed meet the preset diversion conditions, the electronic device will divert the current tasks to be executed, and then process the diverted tasks to be processed, which reduces the number of tasks to be processed at the same time. In the case of limited computing resources, reducing the number of tasks to be processed at the same time can allocate more resources to the tasks being processed. Based on more resources, the situation where the processing speed of tasks is extremely slow can be avoided, that is, the tasks can be executed normally. Compared with processing all tasks at the same time, which results in an extremely slow processing speed for each task, that is, each task is not running normally, when the tasks can be executed normally, it is natural that the total time required to complete each task to be executed in batches according to the diversion results can be reduced, that is, the efficiency of processing the multiple tasks to be executed can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0040] Figure 1 A first flow chart of a method for processing a big data task provided by an embodiment of the present invention;

[0041] Figure 2 A second flow chart of the big data task processing method provided by an embodiment of the present invention;

[0042] Figure 3 A third flow chart of the big data task processing method provided by an embodiment of the present invention;

[0043] Figure 4 A fourth flow chart of the big data task processing method provided by an embodiment of the present invention;

[0044] Figure 5 A structural diagram of a big data task processing device provided by an embodiment of the present invention;

[0045] Figure 6 A structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0047] In the prior art, since the computing power resources of the server are limited, when the server processes multiple big data offline tasks at the same time, the limited computing power resources will be allocated to multiple big data offline tasks, and each big data offline task will be allocated less computing power resources. This will cause the processing speed of each big data offline task to be extremely slow, and then it will take a very long time to complete the multiple big data offline tasks, that is, the efficiency of processing the multiple big data offline tasks is too low. Processing multiple big data offline tasks based on limited computing power resources can also be called processing multiple big data offline tasks under a single queue.

[0048] In order to solve the above problems, the present invention provides a big data task processing method, which is applied to an electronic device in a server, and the electronic device can divert the big data tasks (i.e., tasks to be executed) that currently need to be processed, and process the big data tasks. For example, the electronic device may include a diversion thread and a task processing thread. The diversion thread can divert the current tasks to be executed based on the big data task processing method provided by an embodiment of the present invention to determine the tasks to be processed that currently need to be processed; the task thread can process the tasks to be processed that currently need to be processed determined by the diversion thread. The big data tasks in the present invention can be big data offline tasks, that is, big data tasks that can be processed periodically, such as data analysis tasks, data warehousing tasks, etc.

[0049] Each current task to be executed needs to use the same data table when processing. The electronic device first detects whether the current task to be executed meets the preset diversion condition. When the current task to be executed meets the preset diversion condition, the electronic device will divert the current task to be executed, and at the same time process the diverted task to be processed, and then continue to detect whether the current task to be executed meets the preset diversion condition until the current task to be executed does not meet the preset diversion condition. When the current task to be executed does not meet the preset diversion condition, the electronic device directly processes the current task to be executed. In the case of limited computing resources, reducing the number of tasks to be processed at the same time can allocate more resources to the tasks being processed. Based on more resources, the situation where the processing speed of the task is extremely slow can be avoided, that is, the task can be executed normally. Compared with the method of processing all tasks at the same time, which causes the processing speed of each task to be extremely slow, that is, each task is not running normally, when the task can be executed normally, it is natural to make the total time required to complete each task to be executed in batches according to the diversion result less, that is, the efficiency of processing the multiple tasks to be executed can be improved.

[0050] For example, a big data task may be an analysis task of user data of users who need to browse videos on the server side providing video services. Based on the above example, the current tasks to be executed may be retention analysis tasks, preference analysis tasks, and advertising analysis tasks; the data tables may be play tables, start tables, interaction tables, and click tables. Accordingly, the same data table used to process the major data tasks is the play table.

[0051] Exemplarily, a big data task may be an analysis task of the exercise habits of residents in various places on the exercise recording platform. The current tasks to be performed may include: a duration analysis task to analyze the residents' exercise duration, a relationship analysis task to analyze the relationship between the residents' exercise volume and the weather, and a route analysis task to analyze the residents' exercise routes; the data tables to be used when processing the tasks to be performed include: a timetable to record the exercise time of each resident, a weather table to record the weather in various places, a map table to record maps of various places, etc. A timetable is required when processing a duration analysis task, a timetable and a map table are required when processing a route analysis task, and a timetable and a weather table are required when processing a relationship analysis task. Correspondingly, the same data table to be used to process each of the major data tasks at this time is the timetable.

[0052] See also Figure 1 , Figure 1 A first flow chart of a method for processing a big data task provided by an embodiment of the present invention, the method may include the following steps:

[0053] S101: Detect whether the current task to be executed meets the preset diversion condition.

[0054] Among them, each current task to be executed needs to use the same data table when processing.

[0055] S102: When the current task to be executed meets the preset diversion condition, determine the task to be processed that needs to be processed from the current tasks to be executed, process the currently determined task to be processed, and return to the step of detecting whether the current task to be executed meets the preset diversion condition until the current task to be executed does not meet the preset diversion condition.

[0056] The number of tasks to be processed that is determined to be currently required to be processed is less than the total number of tasks to be executed currently.

[0057] S103: When the current tasks to be executed do not meet the preset diversion conditions, the current tasks to be executed are processed simultaneously.

[0058] Based on the big data task processing method provided by the present invention, when each current task to be executed meets the preset diversion condition, the electronic device will divert each current task to be executed, and then process the diverted tasks to be processed, that is, the number of tasks to be processed at the same time is reduced. In the case of limited computing resources, reducing the number of tasks to be processed at the same time can allocate more resources to the tasks being processed. Based on more resources, the situation where the processing speed of tasks is extremely slow can be avoided, that is, the tasks can be executed normally. Compared with the method of processing all tasks at the same time, which results in an extremely slow processing speed for each task, that is, each task is not running normally, when the tasks can be executed normally, it is natural that the total time required to complete each task to be executed in batches according to the diversion results can be reduced, that is, the efficiency of processing the multiple tasks to be executed can be improved.

[0059] Regarding step S101, it can be understood that before processing the big data task, it is necessary to generate the data table required for processing the big data task, that is, the generation of the data table is located upstream of the big data task processing. The device for generating the data table can be an electronic device for processing the big data task, or it can be other devices in the server, and the present invention is not limited to this. Obviously, there are multiple data tables.

[0060] Usually, when there are no abnormalities in the data table generation process, the data table will be generated when the estimated generation time of a data table is reached, that is, the data table is generated on time. At this time, the big data tasks that need to use the data table can also be executed in time. It is understandable that the order of generating data tables is sequential, and the data tables to be used when executing each major data task will also be different. This means that under normal circumstances, the major data tasks will not be executed at the same time, but will be executed in sequence according to the order in which all the data tables required for execution are generated.

[0061] However, when an abnormal situation occurs in the data table generation process, such as the device used to generate the data table uses less computing resources to generate a data table involving a large amount of data, this will cause the data table to consume more time to generate, that is, when the expected generation time of the data table is reached, the data table may not be generated, that is, the data table cannot be generated on time. At this time, if other data tables have been generated, this will cause all big data tasks that need to use the data table during processing to wait for the generation of the data table. The major data tasks that need to use the data table during processing and the other data tables that need to be used have been generated are the tasks to be executed. In the case of multiple tasks to be executed, multiple tasks to be executed may be executed simultaneously, which may lead to the problem of low efficiency in processing the multiple tasks to be executed. This problem can also be called task congestion.

[0062] Specifically, for each big data task, the relationship curve between the amount of computing power resources allocated to the big data task when processing the big data task and the processing speed of the big data task is S-shaped, that is, the two are nonlinear. Specifically, due to the limitations of the underlying processing logic of the big data task, based on Amdahl's law, it can be known that when processing the big data task, the electronic device will divide the computing power resources allocated to the big data task into serial resources for serial calculation and parallel resources for parallel calculation. The processing speed of the big data task will be limited by serial calculation, which will result in a nonlinear increase in the processing speed of the big data task even if the computing power resources allocated to the big data task are linearly increased. When increasing the computing power resources allocated to the big data task, the processing speed of the big data task has an upper limit due to limited memory and limited network bandwidth; and after the allocated computing power resources are increased, the complexity of scheduling more computing power resources and communication between computing power resources is also increased, that is, additional processing processes will be added, which will also affect the change in the processing speed of the big data task. Furthermore, it is understandable that when the number of data tables used in processing big data tasks is different, the processing speed based on the same computing power resources will also be different. Computing power resources include the memory of electronic devices, CPU (Central Processing Unit), the number of electronic devices, etc.

[0063] It can be seen that when fewer computing resources are allocated to big data tasks, the processing speed of big data tasks may be extremely slow, that is, task congestion problems will occur; but as long as more computing resources are allocated to big data tasks, the processing speed of big data tasks may be significantly improved.

[0064] Obviously, if a large number of big data tasks are executed at the same time, the computing resources allocated to each big data task may be less, resulting in extremely slow processing speed of the big data task, and the multiple big data tasks will have task congestion problems; but after reducing the big data tasks processed at the same time, the processing speed of the big data tasks processed at the same time can be significantly improved, so that the big data tasks can be processed normally.

[0065] For ease of understanding, we can compare the task processing process to the process of horses pulling goods. For example, there are 10 pallets of goods (i.e., tasks to be executed) and 10 horses (i.e., computing resources), and the horses need to pull the goods to the destination (i.e., complete the tasks to be executed through the allocated computing resources). If 1 horse pulls 1 pallet of goods, it may take dozens of days to reach the destination. Even if all 10 pallets of goods are pulled by horses, it will take dozens of days to pull all the goods to the destination. If 2 horses pull 1 pallet of goods, it will take 1 day to reach the destination, and then it will take 2 days to pull all the goods to the destination.

[0066] Correspondingly, it is understandable that when there are multiple tasks to be executed, task congestion does not necessarily occur. For example, the electronic device currently has a lot of computing resources available to process tasks to be executed, but there are only two tasks to be executed. At this time, even if these two tasks to be executed are executed at the same time, there will be no task congestion problem.

[0067] Therefore, the electronic device can detect whether the current task to be executed needs to be diverted according to whether the current task to be executed meets the preset diversion condition. When the current task to be executed meets the preset diversion condition, that is, if processing the current task to be executed at the same time will cause task congestion, the current task to be executed needs to be diverted later; when the current task to be executed does not meet the preset diversion condition, that is, if processing the current task to be executed at the same time may not cause task congestion, the electronic device can directly process the current task to be executed later. The method of detecting whether the current task to be executed meets the preset diversion condition can be found in the detailed description of the subsequent embodiments, which will not be repeated here.

[0068] With respect to step S102, when the current pending tasks meet the preset diversion conditions, the electronic device needs to divert the current pending tasks to avoid the task congestion problem, that is, to determine a part of the tasks from the current pending tasks as the pending tasks that need to be processed currently. Since the number of pending tasks that need to be processed currently determined is less than the number of the current pending tasks, that is, a smaller number of pending tasks are processed at the same time, more computing power resources can be allocated to each pending task, which can alleviate the task congestion problem to a certain extent compared to processing a larger number of pending tasks at the same time.

[0069] Moreover, it is understandable that the computing power resources available for processing tasks to be executed by the electronic device will change. For example, if the computing power resources of the electronic device are 10 points, and 5 points of computing power resources are used to process other tasks, then only 5 points of computing power resources can be allocated to the current task to be executed. Based on 5 points of computing power resources, processing this number of tasks to be executed may cause task congestion. However, if the current number of tasks to be executed remains unchanged, after other tasks are processed, 10 points of computing power resources can be allocated to the current task to be executed. Based on 10 points of computing power resources, processing this number of tasks to be executed may not cause task congestion. Obviously, if the electronic device still performs diversion when 10 points of computing power resources are available, as analyzed above, there is an upper limit to the processing speed of big data tasks. If the amount of computing power resources required when the processing speed of a big data task reaches the upper limit is called the maximum required resource amount, a computing power resource far greater than the maximum required resource amount of the task to be executed may be allocated to a task to be executed, which will cause part of the computing power resources allocated to the task to be executed to be wasted. That is, computing resources cannot be fully utilized, resulting in low efficiency in processing multiple tasks to be executed.

[0070] Therefore, in order to improve the efficiency of processing multiple tasks to be executed and make full use of the computing resources available for processing the tasks to be executed, the electronic device can detect whether the current task to be executed still meets the preset diversion condition after each diversion of the current task to be executed and processing the diverted task to be processed that needs to be processed. When the current task to be executed meets the preset diversion condition, the task to be processed that needs to be processed currently will continue to be diverted from the current task to be executed, and the determined task to be processed currently needs to be processed will be processed.

[0071] In the case where the electronic device includes a diversion thread and a task processing thread, in order to avoid diversion failure, the task processing thread can generate a checkdone (processing completed) message indicating that the current pending tasks have been processed after processing the current pending tasks; after detecting the checkdone message, the diversion thread can further detect whether the current pending tasks still meet the preset diversion conditions.

[0072] With respect to step S103, when the electronic device detects that the current task to be executed does not meet the preset diversion condition, it means that if the current task to be executed is processed at the same time, task congestion may not occur. Therefore, the electronic device can directly process the current task to be executed.

[0073] It is understandable that when the electronic device first detects whether the current task to be executed meets the preset diversion condition, it may obtain a detection result that the current task to be executed does not meet the preset diversion condition. Accordingly, the electronic device can directly process all tasks to be executed.

[0074] The electronic device may also detect that the current task to be executed does not meet the preset diversion condition after performing one or more diversion processes. Accordingly, the electronic device can process the current tasks to be executed, that is, the tasks to be executed that have not been diverted out.

[0075] In some embodiments, before step S101, the method may further include the following steps: for each data table needed to be used when executing a big data task, when the expected generation time of the data table is reached but the data table has not been generated, when the data table is generated, from among the major data tasks, determine the big data task that needs to use the data table for processing and all the data tables required for processing except the data table have been generated, as the task to be executed.

[0076] When a data table is generated, table generation information indicating that the data table has been generated may be generated. For example, the table generation information may be checkdone information including the identifier of the data table, the generation time of the data table, etc. Accordingly, after detecting the table generation information, the electronic device may determine that the data table has been generated.

[0077] For each data table, the electronic device can also obtain the estimated generation time of the data table. For example, the time after the estimated generation time of the data table starting from the generation time of the data table can be used as the estimated generation time. The estimated generation time can be the quotient of the amount of data designed when generating the data table and the generation speed of the data table.

[0078] When the estimated generation time of the data table is reached, but the data table is not generated, that is, an abnormality occurs in the generation process of the data table, after the data table is generated, the electronic device will determine the big data tasks that need to use the data table during processing, and all the data tables required for processing except the data table have been generated. These are all big data tasks that can be processed at present. If these big data tasks are processed at the same time, task congestion may occur. Therefore, the electronic device can use these big data tasks as the current tasks to be executed, and then detect whether the current tasks to be executed meet the preset diversion conditions.

[0079] In some embodiments, before step S101, the method may further include the following steps: when the expected generation time of the data table is reached and the data table has been generated, when the data table is generated, from the major data tasks, determine the big data tasks that need to use the data table for processing and in which all other data tables required for processing except the data table have been generated, and process the determined big data tasks at the same time.

[0080] When the estimated generation time of the data table is reached and the data table has been generated, that is, there is no abnormality in the generation process of the data table, correspondingly, after the data table is generated, the big data task that needs to use the data table during processing and all the data tables required for processing have been generated can also start processing. Therefore, after receiving the table generation information, the electronic device can determine the big data task that needs to use the data table during processing and all the data tables required for processing have been generated, and then simultaneously process the determined big data tasks.

[0081] Based on the above processing, the electronic device can pre-determine the situation where task blocking problem may occur based on the data table of abnormal generation process. Then, before the task blocking problem occurs, the current tasks to be executed can be diverted first, and then the diverted tasks to be processed can be directly processed, so as to avoid the occurrence of task blocking problem.

[0082] In some embodiments, the preset diversion conditions include a number diversion condition and a category diversion condition.

[0083] Accordingly, in Figure 1 Based on Figure 2 , step S101 may include the following steps:

[0084] S1011: Based on the current number of tasks to be executed, detecting whether the current tasks to be executed meet a number diversion condition, and based on the current number of categories of tasks to be executed, detecting whether the current tasks to be executed meet a category diversion condition.

[0085] Among them, tasks to be executed that have the same number of data tables used during execution belong to the same category.

[0086] S1012: If the current task to be executed meets the number diversion condition and the category diversion condition, it is determined that the current task to be executed meets the preset diversion condition; if the current task to be executed does not meet the number diversion condition and / or does not meet the category diversion condition, it is determined that the current task to be executed does not meet the preset diversion condition.

[0087] The number diversion condition can indicate whether there will be a task congestion problem when the current tasks to be executed are executed simultaneously; the category diversion condition can indicate whether the current tasks to be executed can be diverted. When the current tasks to be executed meet the number diversion condition, it can indicate that there will be a task congestion problem when the current tasks to be executed are executed simultaneously, which means that the current tasks to be executed need to be diverted; when the current tasks to be executed meet the category diversion condition, it can indicate that the current tasks to be executed can be diverted later.

[0088] Therefore, the electronic device needs to detect whether the current task to be executed meets the number diversion condition and whether it meets the category diversion condition. Only when both conditions are met, that is, when the preset diversion condition is met, can the electronic device divert the current task to be executed. If any of these two conditions is not met, the electronic device does not need to divert the current task to be executed, that is, it is determined that the current task to be executed does not meet the preset diversion condition, and the electronic device does not divert the current task to be executed, and can directly process the current task to be executed at the same time later.

[0089] There are many ways for an electronic device to detect whether a task congestion problem will occur when executing current tasks to be executed simultaneously.

[0090] In some embodiments, step S1011 may include the following steps: detecting whether the current number of tasks to be executed is greater than a congestion threshold; if the current number of tasks to be executed is greater than the congestion threshold, determining that the current tasks to be executed meet the number diversion condition; if the current number of tasks to be executed is not greater than the congestion threshold, determining that the current tasks to be executed do not meet the number diversion condition.

[0091] The technicians can determine how many big data tasks are processed simultaneously in actual scenarios according to their own work experience when task congestion occurs, and determine the number of big data tasks processed simultaneously when task congestion occurs as the congestion threshold. For example, the congestion threshold can be set to 6.

[0092] Accordingly, when the electronic device detects that the number of tasks to be executed currently is greater than the congestion threshold, it can be determined that if the tasks to be executed currently are simultaneously blocked, the task congestion problem may occur, that is, it is determined that the tasks to be executed currently meet the number diversion condition. When the electronic device detects that the number of tasks to be executed currently is not greater than the congestion threshold, it can be determined that if the tasks to be executed currently are simultaneously blocked, the task congestion problem will not occur, that is, it is determined that the tasks to be executed currently do not meet the number diversion condition.

[0093] Based on the above processing, the electronic device can directly detect whether the current task to be executed meets the number diversion condition according to the set congestion threshold, which reduces the time consumption of detection, that is, the diversion efficiency is higher.

[0094] In some embodiments, Figure 2 Based on Figure 3 , step S1011 may include the following steps:

[0095] S10111: Obtain the currently allocatable resource amount of the computing power resources that can be allocated, the current number of tasks to be executed, and the designated resource amount corresponding to each current task to be executed.

[0096] The designated resource amount corresponding to a task to be executed indicates: the amount of resources required when the processing speed of the task to be executed is greater than a speed threshold.

[0097] S10112: Calculate the estimated amount of resources to be allocated to each of the current tasks to be executed in the allocatable amount of resources according to the weights corresponding to the current tasks to be executed.

[0098] S10113: For each current task to be executed, calculate the ratio of the estimated amount of resources allocated to the task to be executed and the specified amount of resources corresponding to the task to be executed, and obtain the resource ratio corresponding to the task to be executed.

[0099] S10114: From among the current tasks to be executed, determine the tasks to be executed whose corresponding resource ratios are less than the first ratio as the expected blocking tasks.

[0100] S10115: If the proportion of each expected congested task in all current tasks to be executed is greater than the second ratio, it is determined that the current tasks to be executed meet the number diversion condition; if the proportion of each expected congested task in all current tasks to be executed is not greater than the second ratio, it is determined that the current tasks to be executed do not meet the number diversion condition, and based on the category of the current tasks to be executed, it is detected whether the current tasks to be executed meet the category diversion condition.

[0101] As mentioned above, the computing resources (i.e., the amount of allocable resources) of the electronic device that can be used to process the tasks to be executed will change, and under different allocable resource amounts, the upper limit of the number of tasks to be executed that can be processed at the same time while avoiding the problem of task congestion is also different. Therefore, in order to improve the efficiency of processing multiple tasks to be executed and make full use of the computing resources that can be used to process the tasks to be executed, when the electronic device detects whether it is necessary to divert the current tasks to be executed, it can detect in real time whether the current tasks to be executed will be blocked if they are executed at the same time based on the computing resources that are currently available to process the tasks to be executed. The amount of allocable resources can be the remaining resources in the device cluster on the server side.

[0102] Specifically, the electronic device can estimate: if the current tasks to be executed are processed at the same time, the computing resources (i.e., the estimated resource amount) allocated to each task to be executed. If most of the tasks to be executed are processed based on the estimated resource amount, their own processing speed will be less than the speed threshold, and task congestion may occur, and the current tasks to be executed need to be diverted.

[0103] It is understandable that for each big data task, the execution of the big data task needs to be tested before the electronic device processes the big data task, so as to avoid problems such as task execution errors when the electronic device actually processes the big data task. When the execution of the big data task is tested, the relationship between the computing power resources allocated to the big data task and the processing speed of the big data task can be obtained. If the processing speed of the big data task is set to reach the specified speed, the big data task is considered to be in a smooth and fast running state, then according to the obtained relationship, the amount of computing power resources required when the big data task is in a smooth and fast running state can be determined, that is, the specified amount of resources corresponding to the big data task is obtained. When the amount of computing power resources allocated to the big data task is not less than the specified amount of resources, it can be considered that the processing speed of the big data task is not less than the specified speed; when the amount of computing power resources allocated to the big data task is not less than the product of the specified amount of resources and the first ratio, it can be considered that the processing speed of the big data task is not less than the speed threshold. The device for testing can be an electronic device or other device, and the present invention is not limited to this. It is also understandable that the test can be performed multiple times, that is, the obtained relationships may be multiple, and the determined specified resource amounts may also be multiple. At this time, the statistical values ​​of the multiple specified resource amounts can be determined as the specified resource amounts corresponding to the big data task, and the statistical values ​​can be the maximum value, the minimum value, the mean value, etc.

[0104] The electronic device can obtain the amount of allocatable resources (which can be recorded as M), the current number of tasks to be executed (which can be recorded as x), and the designated amount of resources corresponding to each current task to be executed (which can be recorded as n). If the current tasks to be executed are processed at the same time, the estimated amount of resources allocated to each task to be executed (which can be recorded as m) can be expressed as: m=M / x. Furthermore, for each current task to be executed, the resource ratio corresponding to the task to be executed can be obtained, that is, m / n. If m / n is less than the first ratio, that is, when the task to be executed is processed based on the estimated amount of resources allocated, the processing speed of the task to be executed will be less than the speed threshold, that is, the processing speed is low. It can be understood that the n corresponding to different tasks to be executed may also be different. Tasks to be executed whose m / n is less than the first ratio are estimated blocked tasks, that is, tasks that cannot be processed normally. For example, the first ratio can be 1 / 2.

[0105] When the proportion of the estimated blocked tasks in all the current tasks to be executed is greater than the second ratio, that is, most tasks cannot be processed normally, it can be considered that if the current tasks to be executed are executed at the same time, there will be a problem of task blocking, that is, the number diversion condition is met. Correspondingly, when most tasks can be processed normally, it can be considered that the number diversion condition is not met. For example, the second ratio can be 4 / 5.

[0106] Based on the above processing, the electronic device can detect in real time whether the current tasks to be executed meet the number diversion conditions based on the current amount of allocatable resources, and can make full use of the computing power resources available for processing the tasks to be executed, thereby improving the efficiency of processing multiple tasks to be executed.

[0107] In some embodiments, step S1011 may include the following steps: detecting whether the number of categories of the current tasks to be executed is not less than the diversion threshold; if the number of categories of the current tasks to be executed is not less than the diversion threshold, determining that the current tasks to be executed meet the category diversion conditions; if the number of categories of the current tasks to be executed is less than the diversion threshold, determining that the current tasks to be executed do not meet the category diversion conditions.

[0108] Since the subsequent embodiments of the present invention will divert the current tasks to be executed based on their categories, it is obvious that only when the category of the current tasks to be executed is not less than the diversion threshold, can the current tasks to be executed be diverted based on their categories. Therefore, when the category of the current tasks to be executed is not less than the diversion threshold, it can be determined that the current tasks to be executed meet the category diversion condition. Correspondingly, when the category of the current tasks to be executed is less than the diversion threshold, that is, the current tasks to be executed cannot be diverted based on their categories, therefore, it can be determined that the current tasks to be executed do not meet the category diversion condition.

[0109] Based on the above processing, the electronic device can directly detect whether the current task to be executed meets the category diversion condition according to the set diversion threshold, which reduces the time consumption of detection, that is, the diversion efficiency is higher.

[0110] In some embodiments, step S102 may include the following steps: dividing each current task to be executed into a specified number of task groups to be processed; or, according to the category of each current task to be executed, treating the tasks to be executed of the same category as a task group to be processed; wherein, tasks to be executed that have the same number of data tables used during execution belong to the same category; and determining the tasks to be executed included in a task group to be processed as the tasks to be processed that currently need to be processed.

[0111] The specified number can be a number determined based on the current number of tasks to be executed, the current amount of allocatable resources, etc., or can be a fixed number set by the technician based on work experience. For example, if it is determined based on the current amount of allocatable resources, there will be a task congestion problem when four big data tasks are executed simultaneously. If the current number of tasks to be executed is 9, the specified number can be set to 3; if the current number of tasks to be executed is 6, the specified number can be set to 2. But it is not limited to this.

[0112] Furthermore, the electronic device can randomly divide the current tasks to be executed into a specified number of pending task groups. For example, if the current tasks to be executed include: Task 1, Task 2, Task 3, Task 4, Task 5, and Task 6, then Task 1, Task 2, and Task 3 can be divided into one pending task group, and Task 4, Task 5, and Task 6 can be divided into one pending task group; or Task 1, Task 3, and Task 5 can be divided into one pending task group, and Task 2, Task 4, and Task 6 can be divided into one pending task group, but the present invention is not limited thereto.

[0113] Alternatively, the electronic device may also group the current tasks to be executed according to the attribute "the number of data tables used during execution" possessed by each task to be executed. Specifically, the electronic device may determine that the tasks to be executed that have the same number of data tables used during processing belong to the same category, and then treat the tasks to be executed that belong to the same category as a task group to be processed, that is, the tasks to be executed included in a task group to be processed will use the same number of data tables during processing. If grouping is performed based on an attribute that is possessed by all, the subsequent grouping may continue to be performed according to the attribute, that is, the grouping will be performed in a unified manner, which is convenient for the management and maintenance of the electronic device, and further processing may be performed based on the attribute to further fully utilize the computing power resources that can be used to process the tasks to be executed, and further improve the efficiency of processing multiple tasks to be executed.

[0114] In some embodiments, when multiple pending task groups are obtained according to the categories of each pending task, the above steps of determining the pending tasks included in a pending task group as the pending tasks that currently need to be processed may include the following steps: determining the processing order corresponding to each pending task group according to the size relationship of the number of tables corresponding to each pending task group; determining the pending tasks included in the first pending task group in the processing order as the pending tasks that currently need to be processed.

[0115] The processing order determined according to the size relationship of the number of tables corresponding to each pending task group can be: the arrangement order of the size of the specified number of tables, such as the arrangement order of the size of the specified number of tables can be: the order of the number of tables from large to small, the order of the number of tables from small to large, etc., and the present invention is not limited to this. It can be understood that as long as the arrangement order of the size of the number of tables is specified, and the processing order corresponding to each pending task group can be determined in accordance with the specified order, it can be convenient to manage and maintain the electronic device. Furthermore, the electronic device can directly determine the pending tasks included in the first pending task group in the processing order as the pending tasks that need to be processed currently.

[0116] For example, based on the above example, when processing task 1, table a and table m1 need to be used; when processing task 2, table a and table m2 need to be used; when processing task 3, table a, table m3 and table n3 need to be used; when processing task 4, table a, table m4 and table n4 need to be used; when processing task 5, table a, table m5, table n5 and table p5 need to be used; when processing task 6, table a, table m6, table n6 and table p6 need to be used. Then, according to the categories of the tasks to be executed, these tasks can be divided into pending task group 1 including task 1 and task 2, pending task group 2 including task 3 and task 4, and pending task group 3 including task 5 and task 6. The number of tables corresponding to pending task group 1 is 2, the number of tables corresponding to pending task group 2 is 3, and the number of tables corresponding to pending task group 3 is 4. If the order of the size of the specified number of tables is: the order of the number of tables from small to large, then the first pending task group in the processing order is pending task group 1. Accordingly, the electronic device can determine that Task 1 and Task 2 are pending tasks that need to be processed currently. Subsequently, after Task 1 and Task 2 are processed, the electronic device can use a similar method to determine that Task 3 and Task 4 are pending tasks that need to be processed currently; after Task 3 and Task 4 are processed, the electronic device can use a similar method to determine that Task 5 and Task 6 are pending tasks that need to be processed currently.

[0117] In some embodiments, the number of pending tasks currently to be processed determined from the current pending tasks is multiple; Figure 1 Based on Figure 4 , the above step S102 may include the following steps:

[0118] S1021: When the current pending tasks meet the preset diversion conditions, determine the pending tasks that need to be processed from the current pending tasks, allocate the currently allocatable computing resources to the currently determined pending tasks, and process the pending tasks using the computing resources allocated to each pending task.

[0119] S1022: When there are completed tasks to be processed among the currently determined tasks to be processed, the computing power resources allocated to the completed tasks to be processed are allocated to the uncompleted tasks to be processed until all the currently determined tasks to be processed have been processed, and then the process returns to the step of detecting whether the current tasks to be executed meet the preset diversion conditions until the current tasks to be executed do not meet the preset diversion conditions.

[0120] The currently allocable computing power resources are the allocable resource amounts in the aforementioned embodiments. After the electronic device determines the pending tasks that need to be processed currently, it can process the pending tasks through the allocable computing power resources, that is, allocate the allocable computing power resources to each pending task. Accordingly, the pending tasks can be processed based on the allocated computing power resources.

[0121] It is understandable that the processing time of different pending tasks may be different, such as the number of tables of data tables used when processing different pending tasks may be different, the amount of data recorded in the data tables used when processing different pending tasks may be different, etc., which will lead to different processing times required for processing different pending tasks. When there are pending tasks that have been processed and completed among the currently determined pending tasks, the pending tasks that have been processed and completed do not need to continue to use the computing power resources allocated to them. Therefore, in order to make full use of computing power resources, the electronic device can allocate the computing power resources allocated to the pending tasks that have been processed and completed to the pending tasks that have not been processed and completed. Accordingly, these unprocessed pending tasks can be processed based on more computing power resources, which can make the processing speed of these unprocessed pending tasks faster, that is, the processing efficiency of each task to be executed can be further improved.

[0122] In some embodiments, the above step S1022 may include the following steps: for each pending task currently determined, calculate the computing power resources allocated to the pending task in the currently allocable computing power resources according to the weight corresponding to the pending task, and allocate the calculated computing power resources to the pending task.

[0123] Each currently determined task to be processed may have the same weight, that is, the electronic device will evenly distribute the currently allocable computing resources to each task to be processed. Alternatively, the electronic device may also determine the weight of each task to be processed according to the attributes of each task to be processed, such as the attribute of the task to be processed may be the category of the task to be processed, the amount of data involved in processing the task to be processed, etc.

[0124] Moreover, as analyzed above, when the number of data tables used in processing big data tasks is different, the processing speed based on the same computing power resources will also be different. If the currently determined pending tasks belong to the same category, then when the computing power resources are evenly distributed among the pending tasks belonging to the same category, the processing speeds of the respective pending tasks can be made closer. Compared with evenly distributing computing power resources among pending tasks of different categories, if the same computing power resources are allocated to task A that uses 2 data tables for processing and task B that uses 20 data tables for processing, part of the computing power resources allocated to task A may be wasted, that is, the computing power resources cannot be fully utilized. Therefore, if the currently determined pending tasks belong to the same category and evenly distribute computing power resources among the pending tasks belonging to the same category, the computing power resources can be fully utilized.

[0125] Based on the same inventive concept as the above-mentioned big data task processing method, the embodiment of the present invention also provides a big data task processing device. Figure 5 , Figure 5 A structural diagram of a big data task processing device provided by an embodiment of the present invention, the device includes:

[0126] The detection module 501 is used to detect whether the current task to be executed meets the preset diversion condition; wherein each current task to be executed needs to use the same data table when processing;

[0127] The first processing module 502 is used to determine the pending tasks that need to be processed from the current pending tasks when the current pending tasks meet the preset diversion condition, process the currently determined pending tasks at the same time, and return to execute the step of detecting whether the current pending tasks meet the preset diversion condition until the current pending tasks do not meet the preset diversion condition; wherein the number of the currently determined pending tasks that need to be processed is less than the total number of the current pending tasks;

[0128] The second processing module 503 is used to simultaneously process the current tasks to be executed when the current tasks to be executed do not meet the preset diversion condition.

[0129] Optionally, the detection module 501 is specifically used to: detect whether the current tasks to be executed meet the number diversion condition based on the current number of tasks to be executed, and detect whether the current tasks to be executed meet the category diversion condition based on the current number of categories of tasks to be executed; wherein, tasks to be executed that have the same number of data tables used during execution belong to the same category; if the current tasks to be executed meet the number diversion condition and meet the category diversion condition, determine that the current tasks to be executed meet the preset diversion condition; if the current tasks to be executed do not meet the number diversion condition and / or do not meet the category diversion condition, determine that the current tasks to be executed do not meet the preset diversion condition.

[0130] Optionally, the detection module 501 is specifically used to: detect whether the current number of tasks to be executed is greater than a congestion threshold; if the current number of tasks to be executed is greater than the congestion threshold, determine that the current tasks to be executed meet the number diversion condition; if the current number of tasks to be executed is not greater than the congestion threshold, determine that the current tasks to be executed do not meet the number diversion condition.

[0131] Optionally, the detection module 501 is specifically used to: obtain the allocatable resource amount of the currently allocatable computing power resources, the current number of tasks to be executed, and the designated resource amount corresponding to each of the current tasks to be executed; wherein the designated resource amount corresponding to a task to be executed indicates: the resource amount required when the processing speed of the task to be executed is greater than the speed threshold; and calculate the estimated resource amount allocated to each of the current tasks to be executed in the allocatable resource amount according to the weights corresponding to the current tasks to be executed;

[0132] For each current task to be executed, calculate the ratio of the estimated amount of resources allocated to the task to be executed and the specified amount of resources corresponding to the task to be executed, and obtain the resource ratio corresponding to the task to be executed; from the current tasks to be executed, determine the task to be executed whose corresponding resource ratio is less than the first ratio as the estimated congestion task; if the proportion of each estimated congestion task in all the current tasks to be executed is greater than the second ratio, determine that the current task to be executed meets the number diversion condition; if the proportion of each estimated congestion task in all the current tasks to be executed is not greater than the second ratio, determine that the current task to be executed does not meet the number diversion condition.

[0133] Optionally, the detection module 501 is specifically used to: detect whether the number of categories of the current tasks to be executed is not less than the diversion threshold; if the number of categories of the current tasks to be executed is not less than the diversion threshold, determine that the current tasks to be executed meet the category diversion conditions; if the number of categories of the current tasks to be executed is less than the diversion threshold, determine that the current tasks to be executed do not meet the category diversion conditions.

[0134] Optionally, the device further comprises:

[0135] The third processing module is used to execute each data table needed to be used when executing the big data task before the detection module 501 detects whether the current task to be executed meets the preset diversion conditions. When the expected generation time of the data table is reached but the data table has not been generated, when the data table is generated, from the major data tasks, determine the big data task that needs to use the data table for processing and all the data tables required for processing except the data table have been generated, as the task to be executed.

[0136] Optionally, the device also includes: a fourth processing module, which is used to execute, before the detection module 501 detects whether the current task to be executed meets the preset diversion condition, when the expected generation time of the data table is reached and the data table has been generated, determine from the major data tasks when the data table needs to be used for processing and all other data tables required for processing except the data table have been generated, and simultaneously process the determined big data tasks.

[0137] Optionally, the first processing module 502 is specifically used to: divide each current task to be executed into a specified number of task groups to be processed; or, according to the category of each current task to be executed, group the tasks to be executed of the same category as a task group to be processed; wherein, tasks to be executed that have the same number of data tables used during execution belong to the same category; and determine the tasks to be executed included in a task group to be processed as the tasks to be processed that currently need to be processed.

[0138] Optionally, in the case where multiple pending task groups are obtained according to the categories of each pending task, the first processing module 502 is specifically used to: determine the processing order corresponding to each pending task group according to the size relationship of the number of tables corresponding to each pending task group; determine the pending tasks included in the first pending task group in the processing order as the pending tasks that currently need to be processed.

[0139] Optionally, there are multiple pending tasks that need to be processed currently determined from the current pending tasks;

[0140] The first processing module 502 is specifically used to: allocate currently allocable computing power resources to each currently determined task to be processed, and process each task to be processed using the computing power resources allocated to the task to be processed; when there are completed tasks to be processed among the currently determined tasks to be processed, allocate the computing power resources allocated to the completed tasks to be processed to the uncompleted tasks to be processed, until all the currently determined tasks to be processed have been processed.

[0141] Optionally, the first processing module 502 is specifically used to: for each task to be processed currently determined, calculate the computing power resources allocated to the task to be processed from the currently allocable computing power resources according to the weight corresponding to the task to be processed, and allocate the calculated computing power resources to the task to be processed.

[0142] Based on the big data task processing device provided by the present invention, when each current task to be executed meets the preset diversion condition, the electronic device will divert each current task to be executed, and then process the diverted tasks to be processed, that is, the number of tasks to be processed at the same time is reduced. In the case of limited computing resources, reducing the number of tasks to be processed at the same time can allocate more resources to the tasks being processed. Based on more resources, the situation where the processing speed of tasks is extremely slow can be avoided, that is, the tasks can be executed normally. Compared with the method of processing all tasks at the same time, which results in an extremely slow processing speed for each task, that is, each task is not running normally, when the tasks can be executed normally, it is natural that the total time required to complete each task to be executed in batches according to the diversion results can be reduced, that is, the efficiency of processing the multiple tasks to be executed can be improved.

[0143] The embodiment of the present invention further provides an electronic device, such as Figure 6 As shown, it includes a processor 601, a communication interface 602, a memory 603 and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604.

[0144] Memory 603, used for storing computer programs;

[0145] The processor 601 is used to implement the steps of the big data task processing method described in any of the above-mentioned embodiments when executing the program stored in the memory 603.

[0146] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0147] The communication interface is used for communication between the above electronic device and other devices.

[0148] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0149] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0150] In another embodiment provided by the present invention, a computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is executed by a processor, the big data task processing method described in any of the above embodiments is implemented.

[0151] In another embodiment provided by the present invention, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any of the big data task processing methods described in the above embodiments.

[0152] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0153] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0154] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for devices, electronic devices, computer-readable storage media, and computer program products, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0155] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for processing big data tasks, characterized in that: The method comprises: Detect whether the current pending tasks meet the preset diversion conditions; wherein each current pending task needs to use the same data table when processing; When the current pending tasks meet the preset diversion condition, determine the pending tasks that need to be processed currently from the current pending tasks, process the currently determined pending tasks at the same time, and return to execute the step of detecting whether the current pending tasks meet the preset diversion condition until the current pending tasks do not meet the preset diversion condition; wherein the number of the currently determined pending tasks that need to be processed is less than the total number of the current pending tasks; When the current tasks to be executed do not meet the preset diversion condition, the current tasks to be executed are processed simultaneously.

2. The method according to claim 1, characterized in that The detecting whether the current task to be executed satisfies the preset diversion condition includes: Based on the current number of tasks to be executed, detecting whether the current tasks to be executed meet the number diversion condition, and based on the current number of categories of the tasks to be executed, detecting whether the current tasks to be executed meet the category diversion condition; wherein, tasks to be executed that have the same number of data tables used during execution belong to the same category; If the current task to be executed satisfies the number diversion condition and the category diversion condition, it is determined that the current task to be executed satisfies the preset diversion condition; If the current task to be executed does not satisfy the number diversion condition and / or does not satisfy the category diversion condition, it is determined that the current task to be executed does not satisfy the preset diversion condition.

3. The method according to claim 2, characterized in that The detecting, based on the current number of tasks to be executed, whether the current tasks to be executed meet the number diversion condition includes: Check whether the number of tasks to be executed is greater than the congestion threshold; If the current number of tasks to be executed is greater than the congestion threshold, it is determined that the current tasks to be executed meet the number diversion condition; if the current number of tasks to be executed is not greater than the congestion threshold, it is determined that the current tasks to be executed do not meet the number diversion condition.

4. The method according to claim 2, characterized in that: The detecting, based on the current number of tasks to be executed, whether the current tasks to be executed meet the number diversion condition includes: Obtain the currently allocable resource amount of the computing power resources that can be allocated, the current number of tasks to be executed, and the designated resource amount corresponding to each of the current tasks to be executed; wherein the designated resource amount corresponding to a task to be executed indicates: the resource amount required when the processing speed of the task to be executed is greater than the speed threshold; Calculate the estimated amount of resources to be allocated to each of the currently pending tasks in the allocatable amount of resources according to the weights corresponding to the currently pending tasks; For each current task to be executed, calculate the ratio of the estimated amount of resources allocated to the task to be executed to the specified amount of resources corresponding to the task to be executed, and obtain the resource ratio corresponding to the task to be executed; From among the current tasks to be executed, determine the tasks to be executed whose corresponding resource ratio is less than the first ratio as the expected blocking tasks; If the proportion of each expected congested task in all current tasks to be executed is greater than the second ratio, it is determined that the current tasks to be executed meet the number diversion condition; if the proportion of each expected congested task in all current tasks to be executed is not greater than the second ratio, it is determined that the current tasks to be executed do not meet the number diversion condition.

5. The method according to claim 2, characterized in that: The detecting, based on the number of categories of the current tasks to be executed, whether the current tasks to be executed meet the category diversion condition includes: Check whether the number of categories of the current tasks to be executed is not less than the diversion threshold; If the number of categories of the current tasks to be executed is not less than the diversion threshold, it is determined that the current tasks to be executed meet the category diversion condition; if the number of categories of the current tasks to be executed is less than the diversion threshold, it is determined that the current tasks to be executed do not meet the category diversion condition.

6. The method according to claim 1, characterized in that Before detecting whether the current task to be executed meets the preset diversion condition, the method further includes: For each data table needed to execute a big data task, when the expected generation time of the data table is reached but the data table has not been generated, when the data table is generated, from the major data tasks, determine the big data task that needs to use the data table for processing and all the data tables required for processing except the data table have been generated, as the task to be executed.

7. The method according to claim 6, characterized in that Before detecting whether the current task to be executed meets the preset diversion condition, the method further includes: When the expected generation time of the data table is reached and the data table has been generated, when the data table is generated, from the major data tasks, the big data tasks that need to use the data table for processing and in which all other data tables required for processing except the data table have been generated are determined, and the determined big data tasks are processed simultaneously.

8. The method according to claim 1, characterized in that The step of determining the currently processed task to be processed from the currently executed tasks includes: Divide the current tasks to be executed into a specified number of task groups to be processed; or, according to the categories of the current tasks to be executed, treat the tasks to be executed of the same category as one task group to be processed; wherein, the tasks to be executed that have the same number of data tables used during execution belong to the same category; The to-be-executed tasks included in a to-be-processed task group are determined as the to-be-processed tasks that need to be processed currently.

9. The method according to claim 8, characterized in that In the case where a plurality of to-be-processed task groups are obtained according to the categories of the to-be-performed tasks, determining the to-be-performed tasks included in one to-be-processed task group as the to-be-processed tasks that currently need to be processed includes: According to the size relationship of the number of tables corresponding to each task group to be processed, the processing order corresponding to each task group to be processed is determined; The to-be-executed tasks included in the first to-be-processed task group in the processing sequence are determined as the to-be-processed tasks that currently need to be processed.

10. The method according to claim 1, characterized in that It is determined that there are multiple pending tasks that need to be processed currently from the current pending tasks; The simultaneously processing of the currently determined tasks to be processed includes: Allocate currently allocable computing resources to each currently determined task to be processed, and process each task to be processed using the computing resources allocated to the task to be processed; When there are completed tasks to be processed among the currently determined tasks to be processed, the computing power resources allocated to the completed tasks to be processed are allocated to the uncompleted tasks to be processed until all the currently determined tasks to be processed have been completed.

11. The method according to claim 10, characterized in that The allocating currently allocable computing resources to each currently determined task to be processed includes: For each currently determined pending task, the computing power resources allocated to the pending task in the currently allocable computing power resources are calculated according to the weight corresponding to the pending task, and the calculated computing power resources are allocated to the pending task.

12. A big data task processing device, characterized in that: The device comprises: A detection module is used to detect whether the current pending tasks meet the preset diversion conditions; wherein each current pending task needs to use the same data table when processing; The first processing module is used to determine the pending tasks that need to be processed currently from the current pending tasks when the current pending tasks meet the preset diversion condition, process the currently determined pending tasks at the same time, and return to execute the step of detecting whether the current pending tasks meet the preset diversion condition until the current pending tasks do not meet the preset diversion condition; wherein the number of the currently determined pending tasks that need to be processed is less than the total number of the current pending tasks; The second processing module is used to simultaneously process the current tasks to be executed when the current tasks to be executed do not meet the preset diversion condition.

13. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1-11 when executing a program stored in a memory.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 11 are implemented.

Images