An algorithm power allocation system and method of a fusion data processing platform

Abstract

The application relates to the technical field of computing power allocation, and discloses a computing power allocation system and method of a data processing platform, which comprises a task analysis module, a matching analysis module and an allocation module. The task analysis module is used for analyzing a to-be-allocated computing power task, and obtaining a work flow of the to-be-allocated computing power task, wherein the work flow comprises data types and processing modes of the to-be-allocated computing power task at each stage. The matching analysis module is used for matching and analyzing the work flow of the to-be-allocated computing power task with each computing power center, and obtaining a matching analysis result. The allocation module is used for allocating the to-be-allocated computing power task to a corresponding computing power center according to the matching analysis result. In the matching analysis process of the computing power center, an adaptive selection of computing power resources can meet a current to-be-allocated task and a low-impact allocation scheme on the available computing power of the global computing power center, that is, the goal of accurately allocating the computing power task according to the states of different computing power resources of the computing power center is achieved.

Description

Technical Field

[0001] This application relates to the technical field of computing power allocation, and in particular to a computing power allocation system and method for a converged data processing platform. Background Technology

[0002] Data fusion refers to the integration of data from different sources into a unified and coherent data set after processing. The processing of data fusion requires the computing center to make reasonable allocation of heterogeneous computing resources in order to achieve optimal global efficiency and lowest cost.

[0003] Existing methods for allocating computing power for fused data primarily rely on the degree to which different computing resources are involved in the fused data processing and the current idle percentage of different computing resources in a computing center. For example, when fused data processing mainly involves model training, its demand for GPU computing resources is high. Therefore, it is allocated to computing centers with higher GPU resource idleness to meet its GPU computing resource needs, improve the efficiency of computing task processing, and ensure reasonable global allocation. However, for fused data processing tasks in actual operation, the processes involved are diverse and complex, and the idle status of computing resources changes as the task processing progresses. Therefore, existing methods for allocating computing power for fused data cannot accurately match computing tasks based on the status of different computing resources in a computing center, thus reducing the rationality of computing power allocation. Therefore, how to accurately allocate computing tasks based on the status of different computing resources in a computing center is the fundamental problem that this invention aims to solve. Summary of the Invention

[0004] In order to accurately allocate computing tasks according to the status of different computing resources in the computing center, this application provides a computing power allocation system and method for a converged data processing platform.

[0005] Firstly, this application provides a computing power allocation system for an integrated data processing platform, employing the following technical solution:

[0006] A computing power allocation system integrating a data processing platform includes:

[0007] The task analysis module is used to analyze the computing power tasks to be allocated and obtain the workflow of the computing power tasks to be allocated. The workflow includes the data type and processing method of the computing power tasks to be allocated at each stage.

[0008] The matching analysis module is used to perform matching analysis between the workflow of the computing power tasks to be assigned and each computing power center, and to obtain the matching analysis results.

[0009] The allocation module is used to allocate computing power tasks to the corresponding computing power centers based on the matching analysis results.

[0010] Optionally, the matching analysis module performs the matching analysis process including:

[0011] Based on the workflow, determine the preset computing power curves for the tasks to be allocated across various computing resources, and assess whether all computing centers meet the preset computing power curves for the tasks to be allocated across various computing resources:

[0012] If no computing center meets the preset computing power curve of the computing power task to be allocated in various computing power resources, then calculate the processing time of the computing power task to be allocated for each computing center, and select the matching analysis result of the computing power task to be allocated based on the processing time.

[0013] If a computing center satisfies the preset computing power curve of the computing power task to be allocated in various types of computing power resources, then the idle occupancy curve of the corresponding computing center in various types of computing power resources is compared with the preset computing power curve of the computing power task to be allocated in various types of computing power resources, and the matching analysis result of the computing power task to be allocated is selected based on the comparison result.

[0014] Optionally, the process of determining the preset computing power curve includes:

[0015] Estimate the computational workload of each stage in various computing resources based on the data type and processing method of each stage. Determine the pre-allocated computing power corresponding to each type of computing resource based on the range of computational workload. Estimate the computation time of each type of computing resource in each stage based on the pre-allocated computing power. Select the longest computation time among all types of computing resources as the predicted time for each stage. Determine the preset computing power curve for each type of computing resource in each stage based on the predicted time and the pre-allocated computing power. Combine the preset computing power curves of all stages to obtain the preset computing power curve for each type of computing resource.

[0016] Optionally, the process of determining whether a computing center meets the preset computing power curves for the tasks to be allocated among various types of computing power resources includes:

[0017] Through equations Calculate the judgment factor of the computing power task to be allocated relative to the x-th computing power center on the i-th type of computing power resource. ;

[0018] When determining t∈[t1, t2] Is it always greater than zero?

[0019] If yes, then determine that the x-th computing center satisfies the computing power task to be allocated on the i-th type of computing power resource;

[0020] Otherwise, determine that the x-th computing center does not meet the requirements for the computing power task to be allocated on the i-th type of computing power resource;

[0021] When the xth computing center satisfies the assigned computing task on all types of computing resources, it is determined that the computing center satisfies the preset computing power curve of the assigned computing task in all types of computing resources.

[0022] in, Let x be the percentage of idle computing power in the x-th computing center on the ith type of computing resources. The preset computing power curve for the tasks to be allocated computing power on the i-th type of computing power resources. Let t1 and t2 be the real-time available computing power of the x-th computing center on the i-th type of computing resources, and let t1 and t2 be the start and end points of the preset computing power curve, respectively.

[0023] Optionally, when no computing center satisfies the preset computing power curves of the tasks to be allocated among various types of computing power resources, the process of selecting the matching analysis results includes:

[0024] Through equations Calculate the time tp used by the x-th computing center on the i-th type of computing resources in each stage, add up the maximum value of the time tp used by all types of computing resources in each stage, and obtain the processing time of the x-th computing center for the computing tasks to be allocated.

[0025] The computing center with the shortest processing time was selected as the matching analysis result.

[0026] Where ts is the start time of each stage, and te is the end time of each stage.

[0027] Optionally, when a computing center satisfies the preset computing power curves of the tasks to be allocated among various computing power resources, the process of selecting the matching analysis results includes:

[0028] Through equations Calculate the remaining available computing power of the x-th computing center on the i-th type of computing resources. ;

[0029] According to a preset fixed time interval Select n time points and use the equation Calculate the remaining available computing power alignment of the x-th computing power center for the i-th type of computing power resource. Based on all computing resources and Select the corresponding computing power center as the matching analysis result;

[0030] Where, j∈[1,n], This represents the j-th time point. For all The mean.

[0031] Optionally, based on all computing resources and The process of selecting the corresponding computing center as the matching analysis result includes:

[0032] Through equations Calculate the selection priority value of the x-th computing center ;

[0033] Select the maximum selection priority value The corresponding computing center is used as the matching analysis result;

[0034] Where m is the number of types of computing resources, i∈[1,m], Represents the time interval [t1, t2]. The maximum value, Let be the reference value for the i-th type of computing power resource. This is a reference value for the alignment of the remaining available computing power of the i-th type of computing power resource. To adjust the coefficient, is the weight coefficient of the i-th type of computing power resource.

[0035] Secondly, this application provides a method for allocating computing power in a converged data processing platform, employing the following technical solution:

[0036] A method for allocating computing power across a converged data processing platform, comprising:

[0037] The task analysis module analyzes the computing power tasks to be allocated and obtains the workflow of the computing power tasks to be allocated. The workflow includes the data type and processing method of the computing power tasks to be allocated at each stage.

[0038] The matching analysis module performs matching analysis between the workflow of the computing power tasks to be assigned and each computing power center to obtain the matching analysis results.

[0039] The allocation module allocates computing power tasks to the corresponding computing power centers based on the matching analysis results.

[0040] In summary, this application includes at least one of the following beneficial technical effects:

[0041] This invention employs a workflow acquisition method for acquiring computing power tasks to be allocated. By matching and analyzing the workflow with each computing power center, the optimal computing power center is selected for allocation. Since the workflow includes the data type and processing method of the computing power tasks to be allocated at each stage, it is possible to make a preliminary judgment on the computing power resource consumption and computation time at each stage. Furthermore, during the computing power center matching and analysis process, it is possible to adaptively select an allocation scheme that meets the current computing power tasks to be allocated and has a low impact on the available computing power of the global computing power center. In other words, it achieves the goal of accurately allocating computing power tasks according to the status of different computing power resources in different computing power centers. Attached Figure Description

[0042] Figure 1 This is a logical diagram of the computing power allocation system of the integrated data processing platform in this invention.

[0043] Figure 2 This is a flowchart of the steps of the computing power allocation method of the integrated data processing platform in this invention. Detailed Implementation

[0044] The embodiments of this application are described in detail below, and examples of the embodiments are shown in the accompanying drawings.

[0045] In the description of this specification, the references to "certain embodiments," "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples" refer to specific features, structures, materials, or characteristics described in connection with the described embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0046] This application discloses a computing power allocation system for an integrated data processing platform, referring to... Figure 1The system includes a task analysis module, a matching analysis module, and an allocation module. The task analysis module analyzes the computing power tasks to be allocated, obtaining their workflows. These workflows include the data types and processing methods of the tasks at each stage. The matching analysis module matches the workflows with each computing power center, obtaining the matching analysis results. The allocation module allocates the tasks to the corresponding computing power centers based on the matching analysis results. As can be seen from the above scheme, this embodiment obtains the workflows of the tasks to be allocated, matches them with each computing power center, and then selects the optimal computing power center for allocation. Since the workflows include the data types and processing methods of the tasks at each stage, a preliminary judgment can be made on the computing power resource usage and computation time at each stage. Therefore, during the computing power center matching analysis process, an allocation scheme can be adaptively selected that satisfies the current tasks while having a low impact on the available computing power of the global computing power centers. This achieves the goal of accurately allocating computing power tasks based on the status of different computing power resources in different computing power centers.

[0047] The matching analysis module performs the following process: First, based on the workflow, it determines the preset computing power curves for the tasks to be allocated across various computing resources. The process of determining the preset computing power curves includes: estimating the computational workload of each stage across various computing resources based on the data type and processing method of each stage; determining the pre-allocated computing power for each type of computing resource based on the range of computational workload; setting a standard for the correspondence between computational workload and pre-allocated computing power based on empirical data; and estimating the computational workload of each stage across various computing resources according to the pre-allocated computing power. Since the completion time of each stage is based on the latest completion time among all types of computing resources, the longest computation time among all types of computing resources is selected as the predicted time for each stage. Based on the predicted time and pre-allocated computing power, the preset computing power curves for each type of computing resource in each stage are determined. It should be noted that since the pre-allocated computing power is determined based on the computational load, the preset computing power curve in each stage represents a fully loaded state relative to the pre-allocated computing power. The computational boost and debunking processes are relatively short and are not considered here. Therefore, the shape of the preset computing power curve in each stage is close to a rectangle. Combining the preset computing power curves of all stages yields the predicted time for each type of computing resource. The system uses a preset computing power curve to determine whether all computing centers meet the preset computing power curves for the tasks to be allocated across various computing resources. If no computing center meets the preset computing power curves for the tasks to be allocated across various computing resources, the processing time for the tasks to be allocated is calculated for each computing center, and the matching analysis results for the tasks to be allocated are selected based on the processing time. If a computing center meets the preset computing power curves for the tasks to be allocated across various computing resources, the idle occupancy curve of the corresponding computing center in various computing resources is compared with the preset computing power curves for the tasks to be allocated across various computing resources, and the results are determined based on the comparison. The results of the matching analysis of the tasks to be allocated computing power are selected. Through the above judgment process, the optimal computing power center can be selected for allocation from three aspects: whether the computing power center meets the pre-allocated computing power, processing time, and reasonable allocation. When none of the computing power centers meet the pre-allocated computing power, the matching analysis results selected based on processing time are used. If there is a computing power center that meets the pre-allocated computing power, the computing power center that meets the pre-allocated computing power is further compared, and then the optimal computing power center can be selected for allocation. The process of judging whether the computing power center meets the preset computing power curve of the tasks to be allocated among various computing power resources includes: through equations Calculate the judgment factor of the computing power task to be allocated relative to the x-th computing power center on the i-th type of computing power resource. ;in, Let x be the percentage of idle computing power in the x-th computing center on the ith type of computing resources. The preset computing power curve for the tasks to be allocated computing power on the i-th type of computing power resources. For the x-th computing center, the real-time available computing power on the i-th type of computing resources, when the judgment factor When the value is always greater than zero, it indicates that the proportion of idle computing power in the x-th computing center on the i-th type of computing power resource always meets the demand for computing power tasks to be allocated. Therefore, when judging t∈[t1, t2]... Whether it is always greater than zero, t1 and t2 are the start and end points of the preset computing power curve, that is, the total time taken by the computing power task to be allocated according to the preset computing power. If it is, then it is determined that the x-th computing power center meets the requirements of the computing power task to be allocated on the i-th type of computing power resource; otherwise, it is determined that the x-th computing power center does not meet the requirements of the computing power task to be allocated on the i-th type of computing power resource. Since the computing power task to be allocated needs to consider whether multiple computing power resources are satisfied, when the x-th computing power center meets the requirements of the computing power task to be allocated on all types of computing power resources, it is determined that the computing power center meets the preset computing power curve of the computing power task to be allocated in all types of computing power resources. Therefore, through the above process, it is possible to accurately determine whether the computing power center meets the preset computing power curve of the computing power task to be allocated in all types of computing power resources, thus providing a basis for subsequent judgments.

[0048] It should be noted that in this real-time example, the preset computing power curve approaches a horizontal straight line at each stage, but it will change at different stages. Therefore, it is named as a curve from an overall perspective.

[0049] When no computing center meets the preset computing power curves of the tasks to be allocated among various computing power resources, the process of selecting the matching analysis results includes: through equations The time tp of the x-th computing center on the i-th type of computing resources in each stage is calculated, where ts is the start time of each stage and te is the end time of each stage. The maximum value of the time tp of all types of computing resources in each stage is added to obtain the processing time of the x-th computing center for the computing task to be allocated. The computing center with the shortest processing time is selected as the matching analysis result. Through the above process, when none of the computing centers meet the pre-allocated computing power, a relatively efficient computing center can be selected for allocation, ensuring the processing efficiency of the computing task to be allocated. When there is a computing center that meets the preset computing power curve of the computing task to be allocated in various types of computing resources, the matching analysis result selection process includes: through equations... Calculate the remaining available computing power of the x-th computing center on the i-th type of computing resources. Due to the remaining available computing power The more stable the change state, the greater the overall remaining computing power, and the more conducive it is to handling more computing power tasks. Therefore, according to a preset fixed time interval... Select n time points, preset fixed time intervals based on empirical data, and use equations to... Calculate the remaining available computing power alignment of the x-th computing power center for the i-th type of computing power resource. Where j∈[1, n], This represents the j-th time point. For all The average value, therefore the remaining available computing power alignment. The smaller the value, the more stable the idle computing power of the x-th computing center is in the i-th type of computing power resource. Therefore, based on all computing power resources... and Selecting the corresponding computing center as the matching analysis result includes: through equations Calculate the selection priority value of the x-th computing center Where m is the number of computing resource types, i∈[1,m], Represents the time interval [t1, t2]. The maximum value, Let be the reference value for the i-th type of computing power resource. This is a reference value for the alignment of the remaining available computing power of the i-th type of computing power resource. and All are determined based on the type of computing power resource of type i and the test data, and are used for... and Perform data normalization. The adjustment coefficients are set based on a fit to the test data. Let be the weight coefficient for the i-th type of computing power resource. This coefficient is used to adjust the weight of different computing power resources based on their priority settings. Therefore, the selection priority value of the x-th computing power center is obtained through calculation. It can determine the remaining available computing power of the x-th computing center after running the currently assigned computing power task by its size, and select the priority value. The larger the priority value, the more computing power the center can handle after running the current tasks to be allocated, and the better it matches the workflow of the current tasks to be allocated. Therefore, the maximum priority value is selected. The corresponding computing center serves as the matching analysis result, achieving the goal of accurately allocating computing tasks based on the status of different computing resources in the computing center.

[0050] This application also discloses a method for allocating computing power in a converged data processing platform, referring to... Figure 2 The process includes: analyzing the computing power tasks to be allocated through a task analysis module to obtain the workflow of the computing power tasks to be allocated, wherein the workflow includes the data type and processing method of the computing power tasks to be allocated at each stage; performing matching analysis on the workflow of the computing power tasks to be allocated with each computing power center through a matching analysis module to obtain the matching analysis results; and allocating the computing power tasks to be allocated to the corresponding computing power centers through an allocation module based on the matching analysis results.

[0051] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A computing power allocation system integrating a data processing platform, characterized in that, include: The task analysis module is used to analyze the computing power tasks to be allocated and obtain the workflow of the computing power tasks to be allocated. The workflow includes the data type and processing method of the computing power tasks to be allocated at each stage. The matching analysis module is used to perform matching analysis between the workflow of the computing power tasks to be assigned and each computing power center, and to obtain the matching analysis results. The allocation module is used to allocate computing power tasks to the corresponding computing power centers based on the matching analysis results. The matching analysis module performs the following process: Based on the workflow, determine the preset computing power curves for the tasks to be allocated across various computing resources, and assess whether all computing centers meet the preset computing power curves for the tasks to be allocated across various computing resources: If no computing center meets the preset computing power curve of the computing power task to be allocated in various computing power resources, then calculate the processing time of the computing power task to be allocated for each computing center, and select the matching analysis result of the computing power task to be allocated based on the processing time. If a computing center satisfies the preset computing power curve of the computing power task to be allocated in various types of computing power resources, then the idle occupancy curve of the corresponding computing center in various types of computing power resources is compared with the preset computing power curve of the computing power task to be allocated in various types of computing power resources, and the matching analysis result of the computing power task to be allocated is selected based on the comparison result. The process of determining the preset computing power curve includes: Based on the data type and processing method of each stage, estimate the computational workload of each stage in various computing resources. Determine the pre-allocated computing power corresponding to each type of computing resource based on the range of computational workload. Estimate the computation time of each type of computing resource in each stage according to the pre-allocated computing power. Select the longest computation time among all types of computing resources as the predicted time for each stage. Determine the preset computing power curve of each type of computing resource in each stage based on the predicted time and the pre-allocated computing power. Combine the preset computing power curves of all stages to obtain the preset computing power curve of each type of computing resource. When a computing center meets the preset computing power curves of the tasks to be allocated among various computing power resources, the process of selecting the matching analysis results includes: Through equations Calculate the remaining available computing power of the x-th computing center on the i-th type of computing resources. , The real-time available computing power of the x-th computing center on the i-th type of computing resources. A preset computing power curve for the computing power task to be allocated on the i-th type of computing power resource; According to a preset fixed time interval Select n time points and use the equation Calculate the remaining available computing power alignment of the x-th computing power center for the i-th type of computing power resource. Based on all computing resources and Select the corresponding computing power center as the matching analysis result; Where, j∈[1,n], This represents the j-th time point. For all The mean; Based on all computing resources and The process of selecting the corresponding computing center as the matching analysis result includes: Through equations Calculate the selection priority value of the x-th computing center ; Select the maximum selection priority value The corresponding computing center is used as the matching analysis result; Where m is the number of types of computing resources, i∈[1,m], Represents the time interval [t1, t2]. The maximum value, Let be the reference value for the i-th type of computing power resource. This is a reference value for the alignment of the remaining available computing power of the i-th type of computing power resource. To adjust the coefficient, is the weight coefficient of the i-th type of computing power resource.

2. The computing power allocation system for a converged data processing platform according to claim 1, characterized in that, When no computing center meets the preset computing power curves of the tasks to be allocated among various computing power resources, the process of selecting the matching analysis results includes: Through equations Calculate the time tp used by the x-th computing center on the i-th type of computing resources in each stage, add up the maximum value of the time tp used by all types of computing resources in each stage, and obtain the processing time of the x-th computing center for the assigned computing tasks. The computing center with the shortest processing time was selected as the matching analysis result. Where ts is the start time of each stage, and te is the end time of each stage.

3. A method for allocating computing power across a converged data processing platform, characterized in that, The method employs a computing power allocation system for a fusion data processing platform as described in any one of claims 1-2, comprising: The task analysis module analyzes the computing power tasks to be allocated and obtains the workflow of the computing power tasks to be allocated. The workflow includes the data type and processing method of the computing power tasks to be allocated at each stage. The matching analysis module performs matching analysis between the workflow of the computing power tasks to be assigned and each computing power center to obtain the matching analysis results. The allocation module allocates computing power tasks to the corresponding computing power centers based on the matching analysis results.

Images