Calculation task allocation method and device, electronic equipment and storage medium
By obtaining the amount of data to be calculated and the amount of resources available for the calculation node, the target computing node is determined to allocate tasks to balance the problem of uneven allocation of computing tasks, and improving the utilization rate and efficiency of computing resources.
Patent Information
- Application Number
- CN202411226886.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-02
- Publication Date
- 2024-12-06
AI Technical Summary
In the prior art, uneven allocation of computing tasks leads to low utilization of computing resources, affecting computing efficiency.
By acquiring the amount of data to be calculated in the data processing system and the amount of resources available to the calculation node, the target computing node is determined to allocate tasks evenly and improve resource utilization.
It improves the utilization rate and computing efficiency of computing resources, and ensures balanced and reasonable allocation of data processing.
Smart Images

Figure CN119088567A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of resource scheduling, and in particular to a computing task allocation method, device, electronic device and storage medium. Background Art
[0002] With the continuous development of computer technology, in order to improve processing efficiency, data processing systems usually distribute tasks to be processed to multiple computing nodes included in a server cluster for execution.
[0003] In the related art, when computing tasks are assigned, computing tasks are generally evenly distributed to various computing node devices to perform data calculations. However, different node devices have different conditions and their operating conditions are also different. When tasks are assigned based on the above method, some node devices may have completed the calculation while some devices are still in the calculation state, which will prolong the calculation time and affect the overall calculation efficiency of the computing center. Summary of the invention
[0004] The present invention provides a computing task allocation method, device, electronic device and storage medium to achieve the effect of balanced and reasonable allocation of the system's data volume to be calculated based on the available resources of computing nodes on the basis of improving computing resource utilization, thereby improving the computing efficiency of the data volume to be calculated.
[0005] According to one aspect of the present invention, a method for allocating computing tasks is provided, the method comprising:
[0006] Obtain the amount of data to be calculated corresponding to the data processing system at the current moment;
[0007] Acquire available resource amounts of multiple computing nodes associated with the data processing system within a preset time period after the current moment, and determine a target available resource amount from the multiple available resource amounts;
[0008] When the amount of data to be calculated is less than a preset resource amount threshold, the target computing node corresponding to the data processing system at the current moment is determined based on the amount of data to be calculated and other available resource amounts among the multiple available resource amounts except the target available resource amount, wherein the target computing node is used to calculate the amount of data to be calculated corresponding to the data processing system at the current moment.
[0009] According to another aspect of the present invention, there is provided a computing resource allocation device, the device comprising:
[0010] A data volume acquisition module is used to acquire the amount of data to be calculated corresponding to the data processing system at the current moment;
[0011] An available resource amount determination module, used to obtain available resource amounts of multiple computing nodes associated with the data processing system within a preset time period after the current moment, and determine a target available resource amount from the multiple available resource amounts;
[0012] A computing node determination module is used to determine the target computing node corresponding to the data processing system at the current moment based on the amount of data to be calculated and other available resource amounts among the multiple available resource amounts except the target available resource amount when the amount of data to be calculated is less than a preset resource amount threshold, wherein the target computing node is used to calculate the amount of data to be calculated corresponding to the data processing system at the current moment.
[0013] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:
[0014] at least one processor; and
[0015] a memory communicatively connected to the at least one processor; wherein,
[0016] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the computing task allocation method described in any embodiment of the present invention.
[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the computing task allocation method described in any embodiment of the present invention when executed.
[0018] The technical solution of this embodiment obtains the amount of data to be calculated corresponding to the data processing system at the current moment, and further obtains the available resource amounts of multiple computing nodes associated with the data processing system within a preset time period after the current moment, and determines the target available resource amount from the multiple available resource amounts. Furthermore, when the amount of data to be calculated is less than a preset resource amount threshold, the target computing node corresponding to the data processing system at the current moment is determined based on the amount of data to be calculated and the available resource amounts other than the target available resource amount among the multiple available resource amounts, thereby solving the problems of unbalanced distribution of computing tasks, reduced resource utilization and affected computing efficiency existing in the related technologies, and achieving the effect of balanced and reasonable distribution of the system's amount of data to be calculated based on the available resource amounts of computing nodes on the basis of improving computing resource utilization, thereby improving the computing efficiency of the amount of data to be calculated.
[0019] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0021] Figure 1 is a flowchart of a computing task allocation method provided according to Embodiment 1 of the present invention;
[0022] Figure 2 is a structural diagram of a computing task allocation device provided according to Embodiment 2 of the present invention;
[0023] Figure 3 It is a structural schematic diagram of an electronic device for implementing the computing task allocation method according to an embodiment of the present invention. DETAILED DESCRIPTION
[0024] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0026] Embodiment 1
[0027] Figure 1This is a flow chart of a computing task allocation method provided in the first embodiment of the present invention. This embodiment is applicable to the case where the amount of data to be calculated in the system is allocated to the corresponding computing nodes. The method can be executed by a computing task allocation device, which can be implemented in the form of hardware and / or software, and can be configured in a terminal and / or a server. Figure 1 As shown, the method includes:
[0028] S110, obtaining the amount of data to be calculated corresponding to the data processing system at the current moment.
[0029] Among them, the data processing system can be understood as a data center in the cluster system. The data processing system is used to distribute the amount of data to be processed to each computing node, and receive the data processing results fed back by each computing node. In addition, the data processing system can also analyze the received data processing results to obtain data processing rules. The amount of data to be calculated can be understood as the amount of resources required to perform the tasks to be calculated. In this embodiment, during the operation of the cluster system, the data processing system has tasks to be calculated at each moment, and the amount of resources required to perform these tasks to be calculated can be used as the amount of data to be calculated. In general, the amount of data to be calculated can be distributed to different computing node devices for data calculation to improve data calculation efficiency.
[0030] In this embodiment, the data processing system can be tested for data volume to obtain the amount of data to be calculated corresponding to each data processing moment, and the obtained amount of data to be calculated can be stored. Furthermore, when allocating tasks to be processed by the data processing system at the current moment, the amount of data to be calculated corresponding to the current moment can be obtained from the stored amount of data to be calculated.
[0031] S120, obtaining available resource amounts of multiple computing nodes associated with the data processing system within a preset time period after the current moment, and determining a target available resource amount from the multiple available resource amounts.
[0032] The computing node may be any server in the cluster system; or, the computing node may be any data processing unit in any server. The preset time period may be a time interval of any length, optionally 1 minute, 5 minutes or 10 minutes, etc. The available resources may be understood as the amount of resources that can be provided to the process or task in the current state and have not yet been allocated. The available resources may be hardware resources, such as CPU space, memory space, disk space, input and output (I / O) devices, etc., or software resources, such as the number of database connections, network bandwidth, etc. The available resources may be used to determine whether the computing node can receive new processes or tasks, and the amount of resources that can be provided to these processes or tasks. The target available resources may be the available resources that meet the resource allocation requirements. The target available resources may be any available resources among multiple available resources, optionally, the available resources corresponding to the highest value of the resources among multiple available resources.
[0033] In this embodiment, in order to determine whether the amount of data to be calculated by the data processing system at the current moment can be allocated to multiple computing nodes associated with it, and in order to determine whether the computing nodes can complete the processing of the allocated amount of data to be calculated within a time period after the current moment, the available resources of the multiple computing nodes associated with the data processing system within a preset time period after the current moment can be determined.
[0034] It should be noted that, for each computing node, the available resource amount of the computing node within a preset time period after the current moment can be determined according to a preset resource amount determination method. The resource amount determination method can be any method, and optionally, the available resource amount corresponding to the computing node is predicted according to a resource amount prediction model.
[0035] Optionally, on the basis of the above-mentioned technical solutions, it also includes: for multiple computing nodes associated with the data processing system, obtaining resource usage data corresponding to the computing nodes at the current moment; performing feature extraction on the resource usage data to obtain predicted correlation features; inputting the predicted correlation features into a resource quantity prediction model to obtain the available resource quantity of the computing nodes within a preset time period after the current moment.
[0036] Among them, the resource usage data can be used to indicate the resource usage of the computing node. Optionally, the resource usage data includes CPU utilization, memory usage, disk I / O, network bandwidth, etc. The prediction-related features can be used to indicate the features in the resource usage data that are related to the prediction of available resource quantities. Optionally, the prediction-related features include task type, algorithm complexity, parallelism, etc. The resource quantity prediction model is obtained by training the time series model based on the historical resource usage data of the computing node. The resource quantity prediction model can be a time series model of any model structure, and optionally, it is an autoregressive integrated moving average model (ARIMA), a seasonal decomposition model (STL), or a long short-term memory network (LSTM).
[0037] In this embodiment, before applying the resource quantity prediction model provided by this embodiment, the time series model can be trained first. Before training the model, multiple training samples can be constructed to train the model based on the training samples. In order to improve the prediction accuracy of the resource quantity prediction model, as many and rich training samples as possible can be constructed. Optionally, the training process of the resource quantity prediction model can be: obtaining multiple training samples, wherein the training samples include historical correlation features and the actual available resources corresponding to the prediction time period; for each training sample, the historical correlation features in the training sample are input into the pre-constructed time series model, and the predicted available resources corresponding to the prediction time period are obtained; based on the predicted available resources and the actual available resources in the training sample, the loss value is determined; based on the loss value, the model parameters in the time series model are corrected, and the convergence of the loss function in the time series model is used as the training goal, so that the trained time series model is used as the resource quantity prediction model.
[0038] As an optional implementation of this embodiment, for multiple computing nodes associated with the data processing system, the resource usage data corresponding to the computing node at the current moment can be determined and obtained according to the resource detection unit set on the computing node. Further, feature extraction can be performed on the acquired resource usage data to extract features related to the prediction of available resource quantity and obtain prediction-related features. Further, the prediction-related features can be input into the resource quantity prediction model to process the prediction-related features based on the resource quantity prediction model and predict the available resource quantity of the computing node within a preset time period after the current moment.
[0039] In this embodiment, when available resource amounts corresponding to multiple computing nodes are obtained, a target available resource amount can be determined from the multiple available resource amounts.
[0040] Optionally, determining a target available resource amount from a plurality of available resource amounts includes: determining an available resource amount corresponding to a highest value of the resource amount from the plurality of available resource amounts, and using the determined available resource amount as the target available resource amount.
[0041] As an optional implementation of this embodiment, when multiple available resource amounts are obtained, the multiple available resource amounts can be sorted in descending order, and the available resource amount ranked first is the available resource amount corresponding to the highest resource amount value. Further, the available resource amount ranked first can be used as the target available resource amount.
[0042] S130. When the amount of data to be calculated is less than a preset resource amount threshold, determine the target computing node corresponding to the data processing system at the current moment according to the amount of data to be calculated and other available resource amounts among multiple available resource amounts except the target available resource amount.
[0043] Among them, the resource amount threshold can be a predetermined resource amount for limiting the computing node to which the data amount is allocated. In this embodiment, the resource amount threshold can be any resource amount. Optionally, the sum of the available resource amounts other than the target available resource amount is used as the resource amount threshold; or, a resource amount with a larger value determined based on historical resource usage data is used as the resource amount threshold, etc. The other available resource amounts can be the available resource amounts remaining after the target available resource amount is eliminated from the multiple available resource amounts. The target computing node is used to process the amount of data to be calculated corresponding to the data processing system at the current moment. The target computing node can be any computing node among the multiple computing nodes. The number of target computing nodes can be one or more.
[0044] It should be noted that in order to improve the resource utilization of computing nodes and the rationality and balance of resource allocation, when allocating the amount of data to be calculated in the data processing system to each computing node, if the amount of data to be calculated is less than the preset resource threshold, the computing node corresponding to the highest value of available resources can be eliminated first, and the target computing node corresponding to the amount of data to be calculated can be determined from the remaining computing nodes.
[0045] As an optional implementation of this embodiment, after determining the target available resource amount from multiple available resource amounts, the target available resource amount can be eliminated from the multiple available resource amounts to obtain at least one other available resource amount. Afterwards, the at least one other available resource amount can be summed, and the sum of the obtained resource amounts can be used as a preset resource amount threshold. Further, the amount of data to be calculated can be compared with a preset resource amount threshold, and when it is determined that the amount of data to be calculated is less than the preset resource amount threshold, the amount of data to be calculated can be compared with at least one other available resource amount. Furthermore, the target computing node corresponding to the data processing system at the current moment can be determined based on the comparison result.
[0046] Optionally, based on the amount of data to be calculated and other available resources except the target available resource amount among multiple available resource amounts, determining the target computing node corresponding to the data processing system at the current moment includes: arranging the other available resources except the target available resource amount among the multiple available resource amounts in order from small to large to obtain the sorted other available resource amounts; determining the other available resource amounts that are at the last position from the sorted other available resource amounts, and using the determined other available resource amounts as the resource amounts to be compared; determining the target computing node corresponding to the data processing system at the current moment based on the resource amounts to be compared, the sorted other available resource amounts and the amount of data to be calculated.
[0047] As an optional implementation of this embodiment, after determining the target available resource amount, the target available resource amount can be eliminated from multiple available resource amounts, and other available resource amounts other than the target available resource amount can be obtained. Further, the other available resource amounts can be sorted in order from small to large in terms of resource amount to obtain the sorted other available resource amounts. Further, the other available resource amounts at the last position can be determined from the sorted other available resource amounts, and the other available resource amounts can be used as the resource amounts to be compared, and the resource amounts to be compared are the other available resource amounts with the highest resource amount value. Further, the target computing node corresponding to the data processing system at the current moment can be determined based on the resource amounts to be compared, the amount of data to be calculated, and the sorted other available resource amounts.
[0048] It should be noted that the benefit of determining the other available resource amounts at the last position as the resource amounts to be compared is that the amount of data to be calculated can be compared with the amount of resources to be compared, and then, the number of target computing nodes can be determined to be one or more based on the comparison result. In other words, when the amount of data to be calculated is less than the amount of resources to be compared, it can be shown that the amount of data to be calculated is less than all other available resources, and the computing node corresponding to a single other available resource can process the amount of data to be calculated, and the number of target computing nodes can be determined to be one. When the amount of data to be calculated is greater than or equal to the amount of resources to be compared, it can be shown that the amount of data to be calculated is greater than or equal to all other available resources, and the computing node corresponding to only a single other available resource cannot process the amount of data to be calculated, and the number of target computing nodes can be determined to be multiple. Furthermore, the efficiency of determining the computing nodes used to process the amount of data to be calculated is improved, the efficiency of allocating data processing tasks is improved, and the resource utilization rate of computing nodes is improved.
[0049] In this embodiment, the number of target computing nodes may be one or more. When the number of target computing nodes is one, it can be explained that the available resources corresponding to one computing node can process the amount of data to be calculated corresponding to the data processing system at the current moment. When the number of target computing nodes is multiple, it can be explained that the sum of the available resources of multiple computing nodes can process the amount of data to be calculated corresponding to the data processing system at the current moment.
[0050] Optionally, the number of target computing nodes is one; based on the amount of resources to be compared, the sorted amount of other available resources and the amount of data to be calculated, determine the target computing node corresponding to the data processing system at the current moment, including: when the amount of data to be calculated is smaller than the amount of resources to be compared, compare the sorted amount of other available resources with the amount of data to be calculated in order of arrangement, and when the amount of data to be calculated is smaller than the current amount of available resources, use the computing node corresponding to the current amount of available resources as the target computing node.
[0051] The currently available resource amount may be other available resource amounts that are currently being compared with the data to be processed.
[0052] As an optional implementation of this embodiment, the amount of resources to be compared can be compared with the amount of data to be calculated. When the amount of data to be calculated is less than the amount of resources to be compared, it can be shown that the computing node corresponding to a single other available resource amount can process the amount of data to be calculated. Further, the sorted other available resource amounts can be compared with the amount of data to be calculated in the order of arrangement. Furthermore, when it is determined that the amount of data to be calculated is less than the current amount of available resources, the computing node corresponding to the current amount of available resources can be used as the target computing node.
[0053] For example, it is assumed that the sorted other available resource amounts are resource amount A of 50, resource amount B of 70, resource amount C of 75, resource amount D of 80, and resource amount E of 100, and the amount of data to be calculated is 60. Further, the sorted other available resource amounts can be compared with the amount of data to be calculated in turn, that is, the resource amount A is compared with the amount of data to be calculated, and it can be determined that the amount of data to be calculated is greater than the resource amount A. After that, the resource amount B can be compared with the amount of data to be calculated, and it can be determined that the amount of data to be calculated is less than the resource amount B. Furthermore, the computing node corresponding to the resource amount B can be used as the target computing node.
[0054] Optionally, there are multiple target computing nodes; based on the amount of resources to be compared, the sorted amount of other available resources and the amount of data to be calculated, determine the target computing node corresponding to the data processing system at the current moment, including: when the amount of data to be calculated is greater than or equal to the amount of resources to be compared, determine one other available resource from the other available resources according to a preset resource amount determination rule; determine the difference between the amount of data to be calculated and the determined other available resources to obtain the resource amount difference, and determine the target computing node based on the resource amount difference, the determined other available resources and the sorted other available resources.
[0055] The preset resource amount determination rule can be used to determine one of the multiple computing nodes that can process the amount of data to be calculated when one computing node cannot process the amount of data to be calculated. The preset resource amount determination rule can be any rule, and optionally, it can be determined randomly, or other available resources ranked in a preset position can be determined, etc. The preset order can be any position, and optionally, it can be the first or last position, etc.
[0056] As an optional implementation of this embodiment, when the amount of data to be calculated is greater than or equal to the amount of resources to be compared, it can be indicated that a computing node corresponding to a single other available resource amount cannot process the amount of data to be calculated, and multiple computing nodes corresponding to other available resource amounts are required to process the amount of data to be calculated. Furthermore, an other available resource amount can be determined from other available resource amounts according to a preset resource amount determination rule. Further, the difference between the amount of data to be calculated and the determined other available resource amounts can be determined, and the difference can be used as the resource amount difference. Furthermore, the target computing node can be determined based on the resource amount difference, the determined other available resource amounts, and the sorted other available resource amounts.
[0057] Optionally, the target computing node is determined based on the resource quantity difference, other determined available resources and sorted other available resources, including: when the resource quantity difference is greater than or equal to the resource quantity to be compared, determine one other available resource from the remaining other available resources according to a preset resource quantity determination rule, and determine the difference between the resource quantity difference and the determined other available resources, and use the difference as the resource quantity difference; compare the resource quantity difference with the resource quantity to be compared, and when it is determined that the resource quantity difference is greater than or equal to the resource quantity to be compared, repeat the steps of determining one other available resource from the remaining other available resources, determining the resource quantity difference and comparing the resource quantity difference with the resource quantity to be compared until the resource quantity difference is less than the resource quantity to be compared; when the resource quantity difference is less than the resource quantity to be compared, compare the remaining sorted other available resources with the resource quantity difference in order of arrangement; when the resource quantity difference is less than the current available resources, use the computing nodes corresponding to the determined other available resources and the computing nodes corresponding to the current available resources as the target computing nodes.
[0058] The remaining other available resources are the remaining amounts after the determined other available resources are screened out.
[0059] As an optional implementation of this embodiment, after obtaining the resource quantity difference, the resource quantity difference can be compared with the resource quantity to be compared. Furthermore, when the resource quantity difference is greater than or equal to the resource quantity to be compared, one other available resource quantity can be determined from the remaining other available resource quantities according to the preset resource quantity determination rule, and the difference between the resource quantity difference and the other available resource quantity can be determined, and the difference can be used as the resource quantity difference. After that, the resource quantity difference can be compared with the resource quantity to be compared, and when it is determined that the resource quantity difference is greater than or equal to the resource quantity to be compared, the steps of determining to determine one other available resource quantity from the remaining other available resource quantities, determining the resource quantity difference, and comparing the resource quantity difference with the resource quantity to be compared are repeated until the resource quantity difference is less than the resource quantity to be compared. Further, when the resource quantity difference is less than the resource quantity to be compared, the remaining sorted other available resource quantities can be compared with the resource quantity difference in order of arrangement. Furthermore, when it is determined that the resource amount difference is less than the current available resource amount, the computing node corresponding to the current available resource amount and all the computing nodes corresponding to other determined available resource amounts may be used as target computing nodes.
[0060] Exemplarily, assume that the other available resources after sorting are resource amount A is 50, resource amount B is 70, resource amount C is 80, resource amount D is 90, and resource amount E is 100, the amount of data to be calculated is 300, and the amount of resources to be compared is resource amount E. The other available resource amount determined according to the preset resource amount determination rule is resource amount C, and then, the difference between the amount of data to be calculated and resource amount C is determined, and the resource amount difference is 210. At this time, the resource amount difference is greater than the resource amount to be compared. Then, another available resource amount can be determined from resource amount A, resource amount B, resource amount D, and resource amount E again, and the other available resource amount determined at this time is resource amount D. Afterwards, it can be determined that the difference between the resource amount difference and resource amount D is 120, and the difference is used as the resource amount difference. At this time, the resource amount difference is still greater than the resource amount to be compared. Then, another available resource is determined from resource A, resource B and resource E. At this time, the determined resource is resource B. After that, it can be determined that the difference between the resource difference and resource B is 40, and the difference is used as the resource difference. At this time, the resource difference is less than the resource to be compared. Further, the remaining sorted resource A and resource E can be compared with the resource difference. The resource difference is first compared with resource A, and resource A is used as the current available resource. It can be determined that the resource difference is less than resource A. Then, the computing nodes corresponding to resource A, resource C, resource D and resource B can be used as target computing nodes, and the number of target computing nodes is 4.
[0061] It should be noted that, in the process of using the target computing node to process the amount of data to be calculated corresponding to the data processing system at the current moment, the computing nodes other than the target computing node among the multiple computing nodes can be used as spare computing nodes. Furthermore, when it is detected that the processing efficiency of any target computing node is reduced or processing is stopped, it can be determined that the target computing node is faulty. At this time, the spare computing node can be used to obtain the amount of data to be calculated in the failed target computing node for continued processing. Furthermore, it avoids the failure of a target computing node during the calculation process, and requires another target computing node to complete the calculation of the assigned task before the data processing in the failed target calculation can be performed, which is more efficient.
[0062] It should also be noted that, when the amount of data to be calculated is greater than or equal to a preset resource amount threshold, all computing nodes associated with the data processing system may be used as target computing nodes.
[0063] The technical solution of this embodiment obtains the amount of data to be calculated corresponding to the data processing system at the current moment, and further obtains the available resource amounts of multiple computing nodes associated with the data processing system within a preset time period after the current moment, and determines the target available resource amount from the multiple available resource amounts. Furthermore, when the amount of data to be calculated is less than a preset resource amount threshold, the target computing node corresponding to the data processing system at the current moment is determined based on the amount of data to be calculated and the available resource amounts other than the target available resource amount among the multiple available resource amounts, thereby solving the problems of unbalanced distribution of computing tasks, reduced resource utilization and affected computing efficiency existing in the related technologies, and achieving the effect of balanced and reasonable distribution of the system's amount of data to be calculated based on the available resource amounts of computing nodes on the basis of improving computing resource utilization, thereby improving the computing efficiency of the amount of data to be calculated.
[0064] Embodiment 2
[0065] Figure 2 Schematic diagram of a computing task allocation device provided by Embodiment 2 of the present invention. Figure 2 As shown, the device includes: a data volume acquisition module 210, an available resource volume determination module 220 and a computing node determination module 230.
[0066] Among them, the data amount determination module 210 is used to obtain the amount of data to be calculated corresponding to the data processing system at the current moment; the available resource amount determination module 220 is used to obtain the available resource amounts of multiple computing nodes associated with the data processing system within a preset time period after the current moment, and determine the target available resource amount from the multiple available resource amounts; the computing node determination module 230 is used to determine the target computing node corresponding to the data processing system at the current moment according to the amount of data to be calculated and other available resource amounts among the multiple available resource amounts except the target available resource amount when the amount of data to be calculated is less than a preset resource amount threshold, wherein the target computing node is used to calculate the amount of data to be calculated corresponding to the data processing system at the current moment.
[0067] The technical solution of this embodiment obtains the amount of data to be calculated corresponding to the data processing system at the current moment, and further obtains the available resource amounts of multiple computing nodes associated with the data processing system within a preset time period after the current moment, and determines the target available resource amount from the multiple available resource amounts. Furthermore, when the amount of data to be calculated is less than a preset resource amount threshold, the target computing node corresponding to the data processing system at the current moment is determined based on the amount of data to be calculated and the available resource amounts other than the target available resource amount among the multiple available resource amounts, thereby solving the problems of unbalanced distribution of computing tasks, reduced resource utilization and affected computing efficiency existing in the related technologies, and achieving the effect of balanced and reasonable distribution of the system's amount of data to be calculated based on the available resource amounts of computing nodes on the basis of improving computing resource utilization, thereby improving the computing efficiency of the amount of data to be calculated.
[0068] Optionally, the available resource amount determination module 220 is specifically configured to determine the available resource amount corresponding to the highest resource amount value from the multiple available resource amounts, and use the determined available resource amount as the target available resource amount.
[0069] Optionally, the computing node determination module 230 includes: a resource quantity sorting submodule, a to-be-compared resource quantity determination submodule and a computing node determination submodule.
[0070] A resource quantity sorting submodule, used for arranging the other available resource quantities among the multiple available resource quantities except the target available resource quantity in ascending order to obtain the sorted other available resource quantities;
[0071] A submodule for determining the amount of resources to be compared, used to determine the other available resource amount located at the last position from the sorted other available resource amounts, and use the determined other available resource amount as the amount of resources to be compared;
[0072] The computing node determination submodule is used to determine the target computing node corresponding to the data processing system at the current moment according to the amount of resources to be compared, the sorted amount of other available resources and the amount of data to be calculated.
[0073] Optionally, the number of the target computing node is one; the computing node determination submodule includes: a resource quantity comparison unit.
[0074] A resource quantity comparison unit is used to compare the sorted other available resource quantities with the amount of data to be calculated in order of arrangement when the amount of data to be calculated is less than the amount of resources to be compared, and to use the computing node corresponding to the current available resource quantity as the target computing node when the amount of data to be calculated is less than the current available resource quantity.
[0075] Optionally, there are multiple target computing nodes; the computing node determination submodule includes: a resource quantity determination unit, a resource quantity difference determination unit and a computing node determination unit.
[0076] A resource amount determination unit, configured to determine one of the other available resource amounts from the other available resource amounts according to a preset resource amount determination rule when the amount of data to be calculated is greater than or equal to the amount of resources to be compared;
[0077] A resource quantity difference determination unit, used to determine the difference between the to-be-calculated data quantity and the determined other available resource quantities, to obtain the resource quantity difference;
[0078] A computing node determination unit is used to determine a target computing node according to the resource quantity difference, the determined other available resource quantities and the sorted other available resource quantities.
[0079] Optionally, the computing node determination unit includes: a resource quantity difference updating subunit, a step repetition execution subunit, a resource quantity comparison subunit and a computing node determination subunit.
[0080] A resource quantity difference updating subunit is used to determine one of the other available resource quantities from the remaining other available resource quantities according to the preset resource quantity determination rule when the resource quantity difference is greater than or equal to the resource quantity to be compared, and determine the difference between the resource quantity difference and the determined other available resource quantity, and use the difference as the resource quantity difference; wherein the remaining other available resource quantity is the remaining after the determined other available resource quantity is screened out;
[0081] The step repeating subunit is used to compare the resource quantity difference with the resource quantity to be compared, and when it is determined that the resource quantity difference is greater than or equal to the resource quantity to be compared, repeatedly performing the steps of determining one of the other available resource quantities from the remaining other available resource quantities, determining the resource quantity difference, and comparing the resource quantity difference with the resource quantity to be compared, until the resource quantity difference is less than the resource quantity to be compared;
[0082] A resource quantity comparison subunit, configured to compare the remaining sorted other available resource quantities with the resource quantity difference in order of arrangement when the resource quantity difference is less than the resource quantity to be compared;
[0083] The computing node determination subunit is used to use the computing nodes corresponding to the other available resource amounts and the computing nodes corresponding to the current available resource amounts as target computing nodes when the resource amount difference is less than the current available resource amounts.
[0084] Optionally, the device further includes: a data acquisition module, a feature extraction module and an available resource quantity determination module.
[0085] A data acquisition module, used for acquiring resource usage data corresponding to a plurality of computing nodes associated with the data processing system at a current moment;
[0086] A feature extraction module, used to extract features from the resource usage data to obtain prediction-related features;
[0087] The available resource quantity determination module is used to input the prediction association characteristics into the resource quantity prediction model to obtain the available resource quantity of the computing node within a preset time period after the current moment.
[0088] The computing task allocation device provided in the embodiment of the present invention can execute the computing task allocation method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.
[0089] Embodiment 3
[0090] Figure 3 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.
[0091] like Figure 3 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.
[0092] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0093] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as a computing task allocation method.
[0094] In some embodiments, the computing task allocation method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the computing task allocation method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the computing task allocation method in any other appropriate manner (e.g., by means of firmware).
[0095] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0096] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0097] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0098] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).
[0099] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.
[0100] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.
[0101] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.
[0102] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A computing task allocation method, characterized in that: include: Obtain the amount of data to be calculated corresponding to the data processing system at the current moment; Acquire available resource amounts of multiple computing nodes associated with the data processing system within a preset time period after the current moment, and determine a target available resource amount from the multiple available resource amounts; When the amount of data to be calculated is less than a preset resource amount threshold, the target computing node corresponding to the data processing system at the current moment is determined based on the amount of data to be calculated and other available resource amounts among the multiple available resource amounts except the target available resource amount, wherein the target computing node is used to calculate the amount of data to be calculated corresponding to the data processing system at the current moment.
2. The computing task allocation method according to claim 1, characterized in that: The step of determining a target available resource amount from the plurality of available resource amounts comprises: The available resource amount corresponding to the highest value of the resource amount is determined from the multiple available resource amounts, and the determined available resource amount is used as the target available resource amount.
3. The computing task allocation method according to claim 1, characterized in that: The step of determining a target computing node corresponding to the data processing system at a current moment according to the amount of data to be calculated and other available resource amounts among the plurality of available resource amounts except the target available resource amount comprises: Arrange the other available resource amounts except the target available resource amount in the plurality of available resource amounts in ascending order to obtain the sorted other available resource amounts; Determine the other available resource amount at the last position from the sorted other available resource amounts, and use the determined other available resource amount as the resource amount to be compared; A target computing node corresponding to the data processing system at the current moment is determined according to the amount of resources to be compared, the sorted amounts of other available resources, and the amount of data to be calculated.
4. The computing task allocation method according to claim 3, characterized in that: The number of the target computing node is one; determining the target computing node corresponding to the data processing system at the current moment according to the amount of resources to be compared, the sorted amount of other available resources, and the amount of data to be calculated, includes: When the amount of data to be calculated is smaller than the amount of resources to be compared, the sorted other available resources are compared with the amount of data to be calculated in order of arrangement, and when the amount of data to be calculated is smaller than the current amount of available resources, the computing node corresponding to the current amount of available resources is used as the target computing node.
5. The computing task allocation method according to claim 3, characterized in that: The number of the target computing nodes is multiple; determining the target computing node corresponding to the data processing system at the current moment according to the amount of resources to be compared, the sorted amount of other available resources, and the amount of data to be calculated includes: In the case where the amount of data to be calculated is greater than or equal to the amount of resources to be compared, determining one of the other available resource amounts from the other available resource amounts according to a preset resource amount determination rule; Determine the difference between the amount of data to be calculated and the determined amount of other available resources to obtain the resource amount difference, and determine the target computing node according to the resource amount difference, the determined amount of other available resources and the sorted amount of other available resources.
6. The computing task allocation method according to claim 5, characterized in that: The determining of the target computing node according to the resource amount difference, the determined other available resource amounts, and the sorted other available resource amounts includes: In the case where the resource quantity difference is greater than or equal to the resource quantity to be compared, one of the other available resource quantities is determined from the remaining other available resource quantities according to the preset resource quantity determination rule, and the difference between the resource quantity difference and the determined other available resource quantity is determined, and the difference is used as the resource quantity difference; wherein the remaining other available resource quantity is the remaining after the determined other available resource quantity is screened out; Comparing the resource quantity difference with the resource quantity to be compared, and when it is determined that the resource quantity difference is greater than or equal to the resource quantity to be compared, repeatedly performing the steps of determining one of the other available resource quantities from the remaining other available resource quantities, determining the resource quantity difference, and comparing the resource quantity difference with the resource quantity to be compared, until the resource quantity difference is less than the resource quantity to be compared; When the resource quantity difference is less than the resource quantity to be compared, the remaining sorted other available resource quantities are compared with the resource quantity difference in order of arrangement; In the case where the resource amount difference is less than the current available resource amount, the computing node corresponding to the other determined available resource amount and the computing node corresponding to the current available resource amount are used as the target computing node.
7. The computing task allocation method according to claim 1, characterized in that: Also includes: For multiple computing nodes associated with data processing, obtaining resource usage data corresponding to the computing nodes at a current moment; Extracting features from the resource usage data to obtain prediction-related features; The prediction association feature is input into a resource quantity prediction model to obtain the available resource quantity of the computing node within a preset time period after the current moment.
8. A computing task allocation device, characterized in that: include: A data volume acquisition module is used to acquire the amount of data to be calculated corresponding to the data processing system at the current moment; An available resource amount determination module, used to obtain available resource amounts of multiple computing nodes associated with the data processing system within a preset time period after the current moment, and determine a target available resource amount from the multiple available resource amounts; A computing node determination module is used to determine the target computing node corresponding to the data processing system at the current moment based on the amount of data to be calculated and other available resource amounts among the multiple available resource amounts except the target available resource amount when the amount of data to be calculated is less than a preset resource amount threshold, wherein the target computing node is used to calculate the amount of data to be calculated corresponding to the data processing system at the current moment.
9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the computing task allocation method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the computing task allocation method according to any one of claims 1 to 7 when executed.
