A cluster task scheduling method, device, computer equipment and storage medium
Through pre-trained task execution time-consuming prediction model and cluster equipment status information, selecting appropriate allocation strategies for task allocation, solving the problem of low task allocation efficiency in the existing technology, and achieving efficient utilization of cluster resources and timely processing of tasks .
Patent Information
- Application Number
- CN202011167392.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-10-28
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2040-11-22
AI Technical Summary
When allocating processing tasks, existing server clusters cannot effectively ensure the processing efficiency of each processing task, resulting in low resource utilization and the allocation strategy is too one-sided to meet the overall processing needs of the cluster.
By obtaining multiple allocation policies, determine the status information of each device, use the pre-trained task execution time-consuming prediction model to predict the expected time-consuming of each device under different policies, and select the most appropriate allocation strategy for the task based on the expected time-consuming time-consuming. Assign, comprehensively considering equipment status and task requirements.
It realizes timely processing of each task, meets the current task processing needs to the greatest extent, and improves task processing efficiency and resource utilization.
Smart Images

Figure CN112286658B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular to a cluster task scheduling method, device, computer equipment and storage medium. Background Art
[0002] Through a server cluster composed of multiple processing devices, multiple tasks can be received and executed. In order to improve the task processing efficiency, it is necessary to reasonably allocate each task to improve the task processing efficiency and cluster resource utilization.
[0003] Generally, when allocating processing tasks, processing devices with more resources such as the currently remaining processors are selected. This allocation method is somewhat one-sided and cannot guarantee the overall processing efficiency of the cluster for each processing task. Summary of the invention
[0004] The embodiments of the present disclosure at least provide a cluster task scheduling method, apparatus, computer equipment and storage medium.
[0005] In a first aspect, an embodiment of the present disclosure provides a cluster task scheduling method, including:
[0006] Acquire multiple allocation strategies for the tasks to be allocated; the allocation strategies are used to indicate that the tasks to be allocated are allocated to designated devices in the cluster, and the designated devices under different allocation strategies are different;
[0007] For each of the multiple allocation strategies, determine the status information of each device in the cluster under each allocation strategy;
[0008] Based on the status information of each device in the cluster under each allocation strategy, the expected time required for each device to complete the task undertaken under the allocation strategy is predicted through a pre-trained task execution time prediction model;
[0009] According to the expected time taken by each device under each allocation strategy to complete the task undertaken under the allocation strategy, an allocation strategy whose expected time meets preset conditions is selected from multiple allocation strategies as the target allocation strategy corresponding to the task to be allocated;
[0010] Based on the target allocation strategy, the to-be-allocated task is allocated to a designated device under the target allocation strategy.
[0011] In a possible implementation manner, the status information of each designated device under each allocation strategy includes:
[0012] The task type, task execution progress, and resource usage of the specified device are specified under the allocation strategy.
[0013] In a possible implementation manner, the status information further includes any one of the following information:
[0014] Hard disk utilization, network information, and user interaction information.
[0015] In a possible implementation, based on the status information of each device in the cluster under each allocation strategy, the expected time taken by each device to complete the task undertaken under the allocation strategy is predicted by a pre-trained task execution time prediction model, including:
[0016] For each allocation strategy, based on the state information of each device in the cluster under the allocation strategy, a state matrix corresponding to the allocation strategy is generated;
[0017] The state matrix corresponding to each allocation strategy is input into the task execution time prediction model to obtain the expected time required for each device to complete the task undertaken under the allocation strategy.
[0018] In a possible implementation manner, for each allocation strategy, based on the state information of each device in the cluster under the allocation strategy, generating a state matrix corresponding to the allocation strategy includes:
[0019] For each allocation strategy, based on the status information of each device in the cluster under the allocation strategy and the arrangement method corresponding to each status information, the status information of each device under the allocation strategy is arranged to generate a state matrix corresponding to the allocation strategy.
[0020] In a possible implementation, according to the expected time taken by each device under each allocation strategy to complete the task undertaken under the allocation strategy, an allocation strategy whose expected time meets preset conditions is selected from multiple allocation strategies as the target allocation strategy corresponding to the task to be allocated, including:
[0021] For each allocation strategy, the expected time required for each device under the allocation strategy to complete the tasks undertaken under the allocation strategy is weighted and summed according to the preset weights corresponding to each task to obtain the total time required for each allocation strategy;
[0022] The allocation strategy with the shortest total time consumption is selected as the target allocation strategy.
[0023] In a possible implementation manner, after allocating the to-be-allocated task to a designated device under the target allocation strategy based on the target allocation strategy, the method further includes:
[0024] Obtain the actual time taken by each device to complete the task undertaken under the target allocation strategy;
[0025] The task execution time consumption prediction model is optimized based on the actual time consumption and the state information of each device in the cluster under the target allocation strategy.
[0026] In a second aspect, the present disclosure also provides a cluster task scheduling device, including:
[0027] An acquisition module, used for acquiring a plurality of allocation strategies for the tasks to be allocated; the allocation strategies are used for indicating that the tasks to be allocated are allocated to designated devices in the cluster, and the designated devices under different allocation strategies are different;
[0028] A determination module, configured to determine, for each of the multiple allocation strategies, respective status information of each device in the cluster under each of the allocation strategies;
[0029] A prediction module, configured to predict the expected time required for each device to complete the task undertaken under each allocation strategy based on the status information of each device in the cluster under each allocation strategy and through a pre-trained task execution time prediction model;
[0030] A selection module is used to select an allocation strategy whose expected time consumption meets preset conditions from multiple allocation strategies as the target allocation strategy corresponding to the task to be allocated according to the expected time consumption of each device under each allocation strategy to complete the task undertaken under the allocation strategy;
[0031] The allocation module is used to allocate the to-be-allocated task to a designated device under the target allocation strategy based on the target allocation strategy.
[0032] In a possible implementation manner, the status information of each designated device under each allocation strategy includes:
[0033] The task type, task execution progress, and resource usage of the specified device are specified under the allocation strategy.
[0034] In a possible implementation manner, the status information further includes any one of the following information:
[0035] Hard disk utilization, network information, and user interaction information.
[0036] In a possible implementation, the prediction module, based on the status information of each device in the cluster under each allocation strategy, predicts the expected time taken by each device to complete the task undertaken under the allocation strategy through a pre-trained task execution time prediction model, and is used to:
[0037] For each allocation strategy, based on the state information of each device in the cluster under the allocation strategy, a state matrix corresponding to the allocation strategy is generated;
[0038] The state matrix corresponding to each allocation strategy is input into the task execution time prediction model to obtain the expected time required for each device to complete the task undertaken under the allocation strategy.
[0039] In a possible implementation manner, the prediction module, when generating a state matrix corresponding to each allocation strategy based on the state information of each device in the cluster under the allocation strategy for each allocation strategy, is used to:
[0040] For each allocation strategy, based on the status information of each device in the cluster under the allocation strategy and the arrangement method corresponding to each status information, the status information of each device under the allocation strategy is arranged to generate a state matrix corresponding to the allocation strategy.
[0041] In a possible implementation manner, the selection module, when selecting an allocation strategy whose expected time consumption meets preset conditions from multiple allocation strategies as the target allocation strategy corresponding to the task to be allocated according to the expected time consumption of each device under each allocation strategy to complete the task undertaken under the allocation strategy, is used to:
[0042] For each allocation strategy, the expected time required for each device under the allocation strategy to complete the tasks undertaken under the allocation strategy is weighted and summed according to the preset weights corresponding to each task to obtain the total time required for each allocation strategy;
[0043] The allocation strategy with the shortest total time consumption is selected as the target allocation strategy.
[0044] In a possible implementation manner, the apparatus further includes a training module, and after allocating the to-be-allocated tasks to designated devices under the target allocation strategy based on the target allocation strategy, the training module is configured to:
[0045] Obtain the actual time taken by each device to complete the task undertaken under the target allocation strategy;
[0046] The task execution time consumption prediction model is optimized based on the actual time consumption and the state information of each device in the cluster under the target allocation strategy.
[0047] In a third aspect, an optional implementation of the present disclosure further provides a computer device, a processor, and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is used to execute the machine-readable instructions stored in the memory, and when the machine-readable instructions are executed by the processor, the machine-readable instructions perform the steps of the above-mentioned first aspect, or any possible implementation of the first aspect.
[0048] In a fourth aspect, an optional implementation of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed, the steps of the above-mentioned first aspect, or any possible implementation of the first aspect are executed.
[0049] The cluster task scheduling method, apparatus, computer equipment and storage medium provided by the embodiments of the present disclosure can predict the expected time required for each device to complete the task undertaken under the allocation strategy through a pre-trained task execution time prediction model and the status information of multiple processing devices in the cluster, and then select a target allocation strategy based on the expected time. Because the task execution time prediction model comprehensively considers the status information of multiple processing devices in the cluster and the processing requirements of each task being executed when predicting the expected time, the target allocation result selected based on the expected time can meet the processing requirements of each current task to the greatest extent, thereby realizing timely processing of each task.
[0050] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.
[0052] Figure 1 A flow chart of a cluster task scheduling method provided by an embodiment of the present disclosure is shown;
[0053] Figure 2 A flowchart showing the optimization process of the task execution time consumption prediction model in the cluster task scheduling method provided by the embodiment of the present disclosure is shown;
[0054] Figure 3 A schematic diagram of a cluster task scheduling device provided by an embodiment of the present disclosure is shown;
[0055] Figure 4 A schematic diagram of the structure of a computer device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0056] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure is not intended to limit the scope of the present disclosure claimed for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.
[0057] Research has found that a server cluster consisting of multiple processing devices can receive and execute multiple tasks. In order to improve task processing efficiency, it is necessary to reasonably allocate each task to improve task processing efficiency and cluster resource utilization.
[0058] Generally, when allocating processing tasks, processing devices with more resources such as the currently remaining processors are selected. This allocation method is somewhat one-sided and cannot guarantee the overall processing efficiency of the cluster for each processing task.
[0059] Based on the above research, the present disclosure provides a cluster task scheduling method, apparatus, computer equipment and storage medium. Through a pre-trained task execution time prediction model and the status information of multiple processing devices in the cluster, it is possible to predict the expected time required for each device to complete the task undertaken under the allocation strategy, and then select a target allocation strategy based on the expected time. Since the task execution time prediction model comprehensively considers the status information of multiple processing devices in the cluster and the processing requirements of each task being executed when predicting the expected time, the target allocation result selected based on the expected time can meet the processing requirements of each current task to the greatest extent, thereby realizing timely processing of each task.
[0060] The defects existing in the above solutions are the results obtained by the inventor after practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed by the present disclosure for the above problems below should be the contributions made by the inventor to the present disclosure during the disclosure process.
[0061] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.
[0062] To facilitate understanding of this embodiment, a cluster task scheduling method disclosed in the embodiment of the present disclosure is first introduced in detail. The execution subject of the cluster task scheduling method provided in the embodiment of the present disclosure is generally a computer device with certain computing capabilities, and the computer device includes, for example: a terminal device or a server or other processing device, and the terminal device can be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the cluster task scheduling method can be implemented by a processor calling a computer-readable instruction stored in a memory.
[0063] It should be noted that the “processing device”, “designated device” and “device” mentioned below all refer to the designated device in the cluster.
[0064] See also Figure 1 FIG. 1 is a flow chart of a cluster task scheduling method provided by an embodiment of the present disclosure, wherein the method includes steps S101 to S105, wherein:
[0065] S101: Acquire multiple allocation strategies for tasks to be allocated; the allocation strategies are used to instruct to allocate the tasks to be allocated to designated devices in a cluster, and different designated devices under different allocation strategies are different.
[0066] S102: For each allocation strategy among the multiple allocation strategies, determine the status information of each device in the cluster under each allocation strategy.
[0067] S103: Based on the status information of each device in the cluster under each allocation strategy, the expected time required for each device to complete the task undertaken under the allocation strategy is predicted by a pre-trained task execution time prediction model.
[0068] S104: According to the expected time taken for each device under each allocation strategy to complete the task undertaken under the allocation strategy, an allocation strategy whose expected time meets preset conditions is selected from multiple allocation strategies as the target allocation strategy corresponding to the task to be allocated.
[0069] S105: Allocate the to-be-allocated task to a designated device under the target allocation strategy based on the target allocation strategy.
[0070] Here are detailed instructions for the above steps:
[0071] In one possible implementation, the multiple allocation strategies for the tasks to be assigned may be randomly generated allocation strategies, for example, the tasks to be assigned and the processing devices that can execute the tasks to be assigned are arranged and combined to obtain multiple allocation strategies for the tasks to be assigned, wherein the processing devices that can execute the tasks to be assigned may be processing devices that currently have no tasks being processed, or processing devices whose processor usage is less than a preset value, such as 50%.
[0072] Exemplarily, when the tasks to be assigned are task A and task B, and the processing devices that can execute the tasks to be assigned are device A and device B, the following four assignment strategies can be obtained by performing the permutations and combinations:
[0073] Task A is assigned to device A, and task B is assigned to device B; Task B is assigned to device A, and task A is assigned to device B; Both task A and task B are assigned to device A; Both task A and task B are assigned to device B.
[0074] In another possible implementation, the task to be assigned may be a feature task, and the acquired assignment strategy for the task to be assigned may be a specific assignment strategy set for a specific task.
[0075] The specific task may be a task that corresponds to a processing device, for example, a type A device performs a type 1 task, and a type B device performs a type 2 task, where the type A device and the type B device may include multiple devices.
[0076] In a possible implementation manner, under each of the allocation strategies, the status information of each designated device includes:
[0077] The task type, task execution progress, and resource usage of the specified device are specified under the allocation strategy.
[0078] Here, the designated device under each allocation strategy may be a device in the cluster to which the task to be allocated is to be allocated.
[0079] The resource occupancy level of the designated device may be a processor usage rate, which is used to characterize the usage status of the processor in the processing device. The value thereof is usually a percentage less than 1. For example, the processor usage rate may be 60%.
[0080] Under each allocation strategy, the task type of the task undertaken by the designated device may be the task type of the task running on the designated device; for example, the tasks currently running on processing device A are task A and task B, the task to be assigned is task C, and the allocation strategy is to assign task C to processing device A. Then the task types of the tasks undertaken by processing device A may include task A, task B, and task C.
[0081] In a specific implementation, different numbers may be set in advance for tasks of different task types, and then after determining the task type assigned to the processing device, the task type of the undertaken task is represented by the pre-set number.
[0082] Continuing with the above example, if the number corresponding to task A is 1001, the number corresponding to task B is 1002, and the number corresponding to task C is 1003, then the task types of the tasks undertaken by processing device A may include 1001, 1002, and 1003.
[0083] In a possible implementation manner, the status information further includes any one of the following information:
[0084] Hard disk utilization, network information, and user interaction information.
[0085] The hard disk utilization rate includes read bandwidth utilization rate, write bandwidth utilization rate, and space utilization rate, wherein the read bandwidth utilization rate can represent the percentage of bandwidth occupied when reading hard disk data to the total bandwidth; the write bandwidth utilization rate can represent the percentage of bandwidth occupied when writing hard disk data to the total bandwidth; the space utilization rate can represent the percentage of stored data volume to the total hard disk space. The values are usually percentages less than 1, for example, the space utilization rate can be 40%.
[0086] The network information includes flow information, used network bandwidth, and remaining network bandwidth. The flow information of a processing device may represent information about the flow consumed by the processing device in the process of executing a task assigned to the processing device. The flow information may include current flow information, average flow information, minimum flow information, maximum flow information, and total flow information. In practical applications, a network flow real-time monitoring tool nload may be used to monitor the flow information.
[0087] The interaction information includes the initiation time and / or initiating user information of each task assigned to the processing device. The initiation time can represent the specific initiation time of each assigned task, and is used to record the computing resources required to be consumed by the task according to the specific initiation time. In different time backgrounds, the computing resources required to be consumed by the task are different. For example, on special dates such as product release dates, weekends, Christmas, traditional holidays, or different time periods in the morning, afternoon, and evening of each day, the same task may need to consume different computing resources.
[0088] The status information of the processing device will affect the accuracy of the subsequent prediction of the expected time consumption of the undertaken task. The more status information of the processing device, the higher the accuracy of predicting the expected time consumption of the undertaken task in S103. However, in order to improve the calculation speed, part of the above status information can be selected for subsequent calculation.
[0089] Among them, when the expected time required for each device to complete the task undertaken under the allocation strategy is predicted based on the task type, task execution progress, and resource occupancy rate through a pre-trained task execution time prediction model, the accuracy can meet user needs and the amount of calculation is small; if the expected time required for each device to complete the task undertaken under the allocation strategy is predicted based on the task type, task execution progress, resource occupancy rate, and at least one of the hard disk utilization rate, network information, and user interaction information through a pre-trained task execution time prediction model, the accuracy is high, but the amount of calculation is large.
[0090] In practical applications, the status information of the selected processing equipment can be dynamically adjusted according to the accuracy requirements of different users.
[0091] Each processing device may be assigned multiple tasks. The expected time for the device to complete the tasks undertaken under the allocation strategy may include the time required for the processing device to complete each task assigned to the processing device. The processing device may execute the tasks assigned to the processing device in parallel.
[0092] In one possible implementation, for each of the allocation strategies, based on the respective status information of each device in the cluster under each of the allocation strategies, a pre-trained task execution time prediction model is used to predict the expected time required for each device to complete the tasks undertaken under the allocation strategy. A state matrix corresponding to the allocation strategy can be generated based on the respective status information of each device in the cluster under the allocation strategy; and then the state matrix corresponding to each allocation strategy is input into the task execution time prediction model to obtain the expected time required for each device to complete the tasks undertaken under the allocation strategy.
[0093] In actual applications, since the allocation strategy often allocates more than one task to be assigned, and more than one type of status information needs to be input, when determining the state matrix corresponding to the allocation strategy, for each allocation strategy, based on the status information of each device in the cluster under the allocation strategy and the arrangement method corresponding to each status information, the status information of each device under the allocation strategy can be arranged to generate the state matrix corresponding to the allocation strategy.
[0094] For the task types of the tasks undertaken, assuming that the task types of tasks 1 to 4 are 10005, 10005, 10007, and 10008 respectively, the state matrix of the task types of the tasks undertaken is:
[0095]
[0096] Regarding the processor usage, assuming that the processor usage of processing device A is 40% and the processor usage of processing device B is 20%, the state matrix of the processor usage of the assigned tasks is:
[0097]
[0098] For the task processing progress of the undertaken tasks, assuming that the progress of task 1 is 10%, the progress of task 2 is 50%, the progress of task 3 is 60%, and the task to be assigned is task 4, then the status matrix of the task processing progress of the undertaken tasks is:
[0099]
[0100] Among them, 0.0 means that the task to be assigned has just been assigned to the corresponding processing device, so the corresponding task processing progress is 0.0.
[0101] In a possible implementation manner, the status information of each processing device under any allocation strategy may further include the distribution relationship of the allocated tasks on the processing device.
[0102] For example, assuming that task 1 is on processing device A, task 2 is on processing device A, task 3 is on processing device B, the task to be assigned is task 4, and the assignment strategy is to assign task 4 to processing device B, then the distribution relationship matrix of the assigned tasks on the processing devices is:
[0103]
[0104] In the above matrix, “1” represents that the task is assigned to processing device A, and “2” represents that the task is assigned to processing device B.
[0105] In another possible implementation, multiple state information may be combined to generate a state matrix. For example, the distribution relationship of the undertaken tasks on the processing devices and the task processing progress of the undertaken tasks may be combined to generate a state matrix. The state matrix may be represented by an M*N-order matrix. The M*N-order matrix represents the distribution relationship of M tasks on N processing devices. If the T-th task is assigned to the N-1-th processing device, the value at the corresponding position is the task processing progress of the T-th task, and other values are 0.
[0106] For example, if task 1 is on processing device A, task 2 is on processing device A, task 3 is on processing device B, the task to be assigned is task 4, the assignment strategy is to assign task 4 to processing device B, the progress of task 1 is 10%, the progress of task 2 is 50%, and the progress of task 3 is 60%, then the corresponding state matrix can be:
[0107]
[0108] The first column represents processing device A, the second column represents processing device B, and "-1" indicates that the task is not on the corresponding processing device.
[0109] By inputting the above matrices into the task execution time prediction model, the expected time of at least one task undertaken under the allocation strategy can be obtained. When the task execution time prediction model outputs the expected time, it can also be represented by a time matrix. For example, the expected time of four tasks is 20 seconds, 40 seconds, 5 seconds, and 500 seconds respectively. The task execution time prediction model can directly output the time matrix corresponding to the expected time of the task undertaken under the allocation strategy:
[0110]
[0111] After predicting the expected time it takes for each device to complete the task under the allocation strategy, an allocation strategy with an expected time that meets preset conditions can be selected from multiple allocation strategies as the target allocation strategy corresponding to the task to be assigned based on the expected time it takes for each device under each allocation strategy to complete the task under the allocation strategy.
[0112] Exemplarily, for each allocation strategy, multiple expected time consumptions corresponding to the allocation strategy may be summed, and the allocation strategy with the smallest corresponding summed result may be selected as the target allocation strategy.
[0113] In practical applications, the priority of task execution varies due to the different importance of tasks. The ideal task execution situation is to ensure that important tasks are completed as much as possible with the minimum total time spent on all tasks, so as to achieve timely processing of each task.
[0114] In one possible implementation, based on the expected time taken for each device under each allocation strategy to complete the task undertaken under the allocation strategy, an allocation strategy with an expected time consumption that meets preset conditions is selected from multiple allocation strategies as the target allocation strategy corresponding to the task to be assigned. For each allocation strategy, the expected time taken for each device under the allocation strategy to complete the task undertaken under the allocation strategy can be weightedly summed according to the preset weights corresponding to each task to obtain the total time consumption corresponding to each allocation strategy; then, the allocation strategy with the shortest total time consumption is selected as the target allocation strategy.
[0115] Among them, the weight corresponding to each task can be pre-set or determined based on the task type of each task. Specifically, different types of tasks may have different importances. According to the importance of the tasks, the levels of the tasks can be divided, for example, they can be divided into level A, level B, level C, and level D. Each level corresponds to at least one task type, and the importance of each level decreases successively. In order to facilitate quantitative calculation and standard unification, the initial weight of each level of tasks can be set to 0.4, 0.3, 0.2, and 0.1 respectively. Based on the task type of any task, the level of the task can be determined, and then based on the level of the task, the weight corresponding to the task can be determined.
[0116] Furthermore, the to-be-allocated task may be allocated to a designated device under the target allocation strategy based on the target allocation strategy.
[0117] In a possible implementation, the optimization process of the task execution time prediction model can be as follows: Figure 2 As shown, it includes the following steps:
[0118] S201, obtaining the actual time taken by each device to complete the task undertaken under the target allocation strategy.
[0119] S202: Optimize the task execution time consumption prediction model based on the actual time consumption and the status information of each device in the cluster under the target allocation strategy.
[0120] Specifically, the sample state information of multiple processing devices and the sample processing time required for each processing device to complete at least one task assigned to the processing device can be obtained first; then, a state matrix is determined based on the sample state information of the multiple processing devices. The state matrix is then input into a task execution time prediction model to obtain the predicted processing time required for each processing device to complete at least one task assigned to the processing device; finally, the task execution time prediction model is optimized based on the sample processing time and the predicted processing time.
[0121] In practical applications, after the training of the task execution time prediction model is completed, it can also be optimized multiple times during the application process. Specifically, after the target allocation strategy is determined and assigned to the corresponding processing device for processing, the actual time taken to complete each task under the target allocation strategy can be obtained, and then the status information of multiple processing devices in the cluster and the actual time taken to complete each task under the target allocation strategy are used as the sample status information and sample processing time respectively to improve the sample data; then based on the improved sample data, the task execution time prediction model is retrained to achieve the optimization of the task execution time prediction model.
[0122] The cluster task scheduling method provided by the embodiment of the present disclosure can predict the expected time required for each device to complete the task undertaken under the allocation strategy through a pre-trained task execution time prediction model and the status information of multiple processing devices in the cluster, and then select a target allocation strategy based on the expected time. Because the task execution time prediction model comprehensively considers the status information of multiple processing devices in the cluster and the processing requirements of each task being executed when predicting the expected time, the target allocation result selected based on the expected time can meet the processing requirements of each current task to the greatest extent, thereby realizing timely processing of each task.
[0123] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.
[0124] Based on the same inventive concept, a cluster task scheduling device corresponding to the cluster task scheduling method is also provided in the embodiment of the present disclosure. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned cluster task scheduling method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.
[0125] Reference Figure 3 , which is a schematic diagram of a cluster task scheduling device provided by an embodiment of the present disclosure, the device includes: an acquisition module 301, a determination module 302, a prediction module 303, a selection module 304, and an allocation module 305; wherein,
[0126] The acquisition module 301 is used to acquire multiple allocation strategies for the tasks to be allocated; the allocation strategies are used to indicate that the tasks to be allocated are allocated to designated devices in the cluster, and different designated devices under different allocation strategies are different;
[0127] A determination module 302 is used to determine, for each allocation strategy among the multiple allocation strategies, respective status information of each device in the cluster under each allocation strategy;
[0128] A prediction module 303 is used to predict the expected time required for each device to complete the task undertaken under each allocation strategy based on the status information of each device in the cluster under each allocation strategy and through a pre-trained task execution time prediction model;
[0129] The selection module 304 is used to select an allocation strategy whose expected time consumption meets the preset conditions from multiple allocation strategies according to the expected time consumption of each device under each allocation strategy to complete the task undertaken under the allocation strategy as the target allocation strategy corresponding to the task to be allocated;
[0130] The allocation module 305 is used to allocate the to-be-allocated task to a designated device under the target allocation strategy based on the target allocation strategy.
[0131] In a possible implementation manner, the status information of each designated device under each allocation strategy includes:
[0132] The task type, task execution progress, and resource usage of the specified device are specified under the allocation strategy.
[0133] In a possible implementation manner, the status information further includes any one of the following information:
[0134] Hard disk utilization, network information, and user interaction information.
[0135] In a possible implementation, the prediction module 303, based on the status information of each device in the cluster under each allocation strategy, predicts the expected time taken by each device to complete the task undertaken under the allocation strategy through a pre-trained task execution time prediction model, and is used to:
[0136] For each allocation strategy, based on the state information of each device in the cluster under the allocation strategy, a state matrix corresponding to the allocation strategy is generated;
[0137] The state matrix corresponding to each allocation strategy is input into the task execution time prediction model to obtain the expected time required for each device to complete the task undertaken under the allocation strategy.
[0138] In a possible implementation manner, the prediction module 303, when generating a state matrix corresponding to each allocation strategy based on the state information of each device in the cluster under the allocation strategy for each allocation strategy, is used to:
[0139] For each allocation strategy, based on the status information of each device in the cluster under the allocation strategy and the arrangement method corresponding to each status information, the status information of each device under the allocation strategy is arranged to generate a state matrix corresponding to the allocation strategy.
[0140] In a possible implementation manner, the selection module 304, when selecting an allocation strategy whose expected time consumption meets a preset condition from multiple allocation strategies as the target allocation strategy corresponding to the task to be allocated according to the expected time consumption of each device under each allocation strategy to complete the task undertaken under the allocation strategy, is used to:
[0141] For each allocation strategy, the expected time required for each device under the allocation strategy to complete the tasks undertaken under the allocation strategy is weighted and summed according to the preset weights corresponding to each task to obtain the total time required for each allocation strategy;
[0142] The allocation strategy with the shortest total time consumption is selected as the target allocation strategy.
[0143] In a possible implementation manner, the apparatus further includes a training module 306. After allocating the to-be-allocated task to a designated device under the target allocation strategy based on the target allocation strategy, the training module 306 is configured to:
[0144] Obtain the actual time taken by each device to complete the task undertaken under the target allocation strategy;
[0145] The task execution time consumption prediction model is optimized based on the actual time consumption and the state information of each device in the cluster under the target allocation strategy.
[0146] The cluster task scheduling device provided by the embodiment of the present disclosure can predict the expected time required for each device to complete the task undertaken under the allocation strategy through a pre-trained task execution time prediction model and the status information of multiple processing devices in the cluster, and then select a target allocation strategy based on the expected time. Because the task execution time prediction model comprehensively considers the status information of multiple processing devices in the cluster and the processing requirements of each task being executed when predicting the expected time, the target allocation result selected based on the expected time can meet the processing requirements of each current task to the greatest extent, thereby realizing timely processing of each task.
[0147] Based on the same technical concept, the embodiment of the present disclosure also provides a computer device. Figure 4As shown, it is a schematic diagram of the structure of a computer device 400 provided in an embodiment of the present disclosure, including a processor 401, a memory 402, and a bus 403. Among them, the memory 402 is used to store execution instructions, including a memory 4021 and an external memory 4022; the memory 4021 here is also called an internal memory, which is used to temporarily store the operation data in the processor 401, and the data exchanged with the external memory 4022 such as a hard disk. The processor 401 exchanges data with the external memory 4022 through the memory 4021. When the computer device 400 is running, the processor 401 communicates with the memory 402 through the bus 403, so that the processor 401 executes the following instructions:
[0148] Acquire multiple allocation strategies for the tasks to be allocated; the allocation strategies are used to indicate that the tasks to be allocated are allocated to designated devices in the cluster, and the designated devices under different allocation strategies are different;
[0149] For each of the multiple allocation strategies, determine the status information of each device in the cluster under each allocation strategy;
[0150] Based on the status information of each device in the cluster under each allocation strategy, the expected time required for each device to complete the task undertaken under the allocation strategy is predicted through a pre-trained task execution time prediction model;
[0151] According to the expected time taken by each device under each allocation strategy to complete the task undertaken under the allocation strategy, an allocation strategy whose expected time meets preset conditions is selected from multiple allocation strategies as the target allocation strategy corresponding to the task to be allocated;
[0152] Based on the target allocation strategy, the to-be-allocated task is allocated to a designated device under the target allocation strategy.
[0153] In a possible implementation manner, in the instruction executed by the processor 401, the status information of each designated device under each allocation strategy includes:
[0154] The task type, task execution progress, and resource usage of the specified device are specified under the allocation strategy.
[0155] In a possible implementation manner, in the instruction executed by the processor 401, the status information further includes any one of the following information:
[0156] Hard disk utilization, network information, and user interaction information.
[0157] In a possible implementation manner, in the instructions executed by the processor 401, based on the status information of each device in the cluster under each allocation strategy, predicting the expected time taken by each device to complete the task undertaken under the allocation strategy through a pre-trained task execution time prediction model, includes:
[0158] For each allocation strategy, based on the state information of each device in the cluster under the allocation strategy, a state matrix corresponding to the allocation strategy is generated;
[0159] The state matrix corresponding to each allocation strategy is input into the task execution time prediction model to obtain the expected time required for each device to complete the task undertaken under the allocation strategy.
[0160] In a possible implementation manner, in the instructions executed by the processor 401, for each allocation strategy, based on the respective state information of each device in the cluster under the allocation strategy, generating a state matrix corresponding to the allocation strategy includes:
[0161] For each allocation strategy, based on the status information of each device in the cluster under the allocation strategy and the arrangement method corresponding to each status information, the status information of each device under the allocation strategy is arranged to generate a state matrix corresponding to the allocation strategy.
[0162] In a possible implementation manner, in the instructions executed by the processor 401, the expected time taken for each device under each allocation strategy to complete the task undertaken under the allocation strategy, and selecting an allocation strategy whose expected time meets a preset condition from multiple allocation strategies as the target allocation strategy corresponding to the task to be allocated, include:
[0163] For each allocation strategy, the expected time required for each device under the allocation strategy to complete the tasks undertaken under the allocation strategy is weighted and summed according to the preset weights corresponding to each task to obtain the total time required for each allocation strategy;
[0164] The allocation strategy with the shortest total time consumption is selected as the target allocation strategy.
[0165] In a possible implementation manner, the instructions executed by the processor 401, after allocating the to-be-allocated task to a designated device under the target allocation strategy based on the target allocation strategy, further include:
[0166] Obtain the actual time taken by each device to complete the task undertaken under the target allocation strategy;
[0167] The task execution time consumption prediction model is optimized based on the actual time consumption and the state information of each device in the cluster under the target allocation strategy.
[0168] The embodiment of the present disclosure also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the cluster task scheduling method described in the above method embodiment are executed. The storage medium can be a volatile or non-volatile computer-readable storage medium.
[0169] The computer program product of the cluster task scheduling method provided in the embodiment of the present disclosure includes a computer-readable storage medium storing a program code, and the instructions included in the program code can be used to execute the steps of the cluster task scheduling method described in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here. The computer program product can be implemented in hardware, software or a combination thereof. In an optional embodiment, the computer program product is specifically embodied as a computer storage medium. In another optional embodiment, the computer program product is specifically embodied as a software product, such as a software development kit (Software Development Kit, SDK) and the like.
[0170] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.
[0171] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0172] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0173] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0174] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.
Claims
1. A cluster task scheduling method for a server cluster, characterized in that: include: Obtaining a plurality of randomly generated allocation strategies for the tasks to be assigned or a plurality of allocation strategies specifically assigned to the task types of the tasks to be assigned; The allocation strategy is used to indicate that the task to be allocated is allocated to a designated device in the cluster. The designated devices under different allocation strategies are different. The designated device under each allocation strategy refers to the device in the cluster to which the task to be allocated is to be allocated. For each of the multiple allocation strategies, determine the status information of each device in the cluster under each allocation strategy; Based on the status information of each device in the cluster under each allocation strategy, the expected time required for each device to complete the task undertaken under the allocation strategy is predicted through a pre-trained task execution time prediction model; According to the expected time taken by each device under each allocation strategy to complete the task undertaken under the allocation strategy, an allocation strategy whose expected time meets preset conditions is selected from multiple allocation strategies as the target allocation strategy corresponding to the task to be allocated; Based on the target allocation strategy, the to-be-allocated task is allocated to a designated device under the target allocation strategy.
2. The method according to claim 1, characterized in that The status information of each designated device under each allocation strategy includes: The task type, task execution progress, and resource usage of the specified device are specified under the allocation strategy.
3. The method according to claim 2, characterized in that The status information also includes any one of the following information: Hard disk utilization, network information, and user interaction information.
4. The method according to claim 1, characterized in that: The predicting of the expected time taken by each device to complete the task undertaken under the allocation strategy based on the status information of each device in the cluster under each allocation strategy by using a pre-trained task execution time prediction model includes: For each allocation strategy, based on the state information of each device in the cluster under the allocation strategy, a state matrix corresponding to the allocation strategy is generated; The state matrix corresponding to each allocation strategy is input into the task execution time prediction model to obtain the expected time required for each device to complete the task undertaken under the allocation strategy.
5. The method according to claim 4, characterized in that For each allocation strategy, based on the state information of each device in the cluster under the allocation strategy, a state matrix corresponding to the allocation strategy is generated, including: For each allocation strategy, based on the status information of each device in the cluster under the allocation strategy and the arrangement method corresponding to each status information, the status information of each device under the allocation strategy is arranged to generate a state matrix corresponding to the allocation strategy.
6. The method according to claim 1, characterized in that The method of selecting, according to the expected time taken by each device under each allocation strategy to complete the task undertaken under the allocation strategy, an allocation strategy whose expected time meets a preset condition from multiple allocation strategies as the target allocation strategy corresponding to the task to be allocated includes: For each allocation strategy, the expected time required for each device under the allocation strategy to complete the tasks undertaken under the allocation strategy is weighted and summed according to the preset weights corresponding to each task to obtain the total time required for each allocation strategy; The allocation strategy with the shortest total time consumption is selected as the target allocation strategy.
7. The method according to any one of claims 1 to 6, characterized in that: After allocating the to-be-allocated task to a designated device under the target allocation strategy based on the target allocation strategy, the method further includes: Obtain the actual time taken by each device to complete the task undertaken under the target allocation strategy; The task execution time consumption prediction model is optimized based on the actual time consumption and the state information of each device in the cluster under the target allocation strategy.
8. A cluster task scheduling device for a server cluster, characterized in that: include: An acquisition module, used for acquiring a plurality of allocation strategies randomly generated for the tasks to be assigned and a plurality of allocation strategies specifically assigned for the task types of the tasks to be assigned; The allocation strategy is used to indicate that the task to be allocated is allocated to a designated device in the cluster. The designated devices under different allocation strategies are different. The designated device under each allocation strategy refers to the device in the cluster to which the task to be allocated is to be allocated. A determination module, configured to determine, for each of the multiple allocation strategies, respective status information of each device in the cluster under each of the allocation strategies; A prediction module, configured to predict the expected time required for each device to complete the task undertaken under each allocation strategy based on the status information of each device in the cluster under each allocation strategy and through a pre-trained task execution time prediction model; A selection module is used to select an allocation strategy whose expected time consumption meets preset conditions from multiple allocation strategies as the target allocation strategy corresponding to the task to be allocated according to the expected time consumption of each device under each allocation strategy to complete the task undertaken under the allocation strategy; The allocation module is used to allocate the to-be-allocated task to a designated device under the target allocation strategy based on the target allocation strategy.
9. A computer device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is used to execute the machine-readable instructions stored in the memory. When the machine-readable instructions are executed by the processor, the processor executes the steps of the cluster task scheduling method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program. When the computer program is executed by a computer device, the computer device executes the steps of the cluster task scheduling method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Task completion time prediction method based on BP (Back Propagation) neural network
CN104239194A
BP neural network time prediction-based task scheduling method in mixed cloud
CN108009023A
Intelligent scheduling method and device
CN109634714A
Methods and systems for detection in industrial internet of things data collection environment with large data sets
CN110073301A
Big data cluster self-adaptive resource scheduling method based on cloud platform
CN110390345A
