Resource allocation method and system based on pipeline parallelism

Abstract

The invention relates to the field of data processing, in particular to a resource allocation method and system based on pipeline parallelism, and solves the problems of how to reasonably allocate computing resources of a power grid system and how to carry out pipeline parallelism processing on part of data so as to ensure that data calculation is completed within a time period. In order to solve the problem, the invention provides a resource allocation method, which comprises the following steps of: calculating the load capacity of a data set to be calculated; acquiring a calculation rate and a maximum load, and calculating actual calculation time; distributing a to-be-calculated data set to obtain an initial distribution plan; obtaining a parallel data set and dividing assembly line modules; acquiring initial communication efficiency, and calculating initial calculation time; correcting the initial communication efficiency to obtain theoretical calculation time; and when the initial distribution plan needs to be modified, carrying out pipeline parallel processing on part of the to-be-calculated data set to obtain a final distribution plan.

Description

Technical Field

[0001] The technology of the present invention relates to the field of data processing, and in particular, to a resource allocation method and system based on pipeline parallelism. Background Art

[0002] With the rapid development of information technology and the growth of global energy demand, the power system is facing the problems of improving efficiency, reducing costs and enhancing system reliability. In order to solve these problems, artificial intelligence technology has begun to be widely used in the field of power grids. More and more power systems have begun to use deep learning models to predict future power demand, and then formulate corresponding power consumption strategies and optimize the configuration of power resources. However, due to the particularity of the power system, the distribution of accidents in different operating states or fault types is usually extremely unbalanced, which may affect the training effect and generalization ability of the deep learning model, thereby consuming a large amount of computing resources and occupying a large amount of storage space. Furthermore, due to the high real-time requirements and high computational complexity of the power system, in the case of insufficient computing resources, the deep learning model often takes a long time to reason, which makes it difficult to meet the real-time response requirements of the power system. Therefore, how to reasonably allocate computing resources according to the computing resource status of the power system and maximize the computing efficiency of the data to be calculated under limited computing resources is one of the problems that technicians in this field urgently need to solve. Summary of the invention

[0003] The problem solved by the present invention is: how to reasonably allocate computing resources of a power grid system and perform pipeline parallel processing on part of the data to ensure that data computing is completed within a time limit.

[0004] To solve the above problems, an embodiment of the present invention provides a resource allocation method based on pipeline parallelism, which includes: dividing the data to be calculated into multiple data sets to be calculated, and calculating the load of each data set to be calculated according to the ideal calculation time; testing the computing device to obtain the calculation rate and maximum load of the computing device, and calculating the actual calculation time according to the calculation rate and the data set to be calculated; recording all the data sets to be calculated whose actual calculation time is less than or equal to the ideal calculation time as the original group, and allocating the data sets to be calculated in the original group to obtain an initial allocation plan; according to the initial allocation plan, obtaining all the unallocated data sets to be calculated to obtain parallel data sets, and performing pipeline parallel processing on the parallel data sets to obtain a pipeline module; obtaining the initial communication efficiency of the pipeline module according to the parallel mode of the pipeline module, and calculating the initial calculation time of the parallel data set according to the initial communication efficiency; correcting the initial communication efficiency according to the initial calculation time to obtain the theoretical calculation time of the parallel data set; judging whether the initial allocation plan needs to be modified according to the theoretical calculation time; when the initial allocation plan does not need to be modified, obtaining the final allocation plan in a parallel manner, and obtaining the final allocation plan according to the theoretical calculation time and the initial allocation plan.

[0005] Compared with the prior art, the technical effects achieved by adopting this technical solution are as follows: by dividing the data to be calculated into multiple data sets to be calculated and reasonably allocating the calculation tasks, the difficulty of data processing caused by excessive data volume and complex data types is avoided; by calculating the maximum load, it is ensured that the data sets to be calculated can be reasonably allocated to each of the computing devices, thereby improving the utilization rate of computing resources; by pipeline parallel processing, the allocation of computing resources is optimized, and the calculation time of the data sets to be calculated that cannot be calculated in time is reduced. The acquisition and correction of the initial communication efficiency ensure that the theoretical calculation time can effectively parallel the actual calculation time of the data set, and improve the rationality of the subsequent allocation of computing resources; by judging whether the initial allocation plan needs to be modified, the risk of some parallel data sets failing to complete the calculation within the ideal calculation time is avoided, and the problem of limited computing resources and high real-time requirements in the power system is effectively solved.

[0006] In one embodiment of the present invention, the data to be calculated is divided into multiple data sets to be calculated, and the load of each data set to be calculated is calculated according to the ideal calculation time, which specifically includes: acquiring all data that needs to be calculated to obtain the data to be calculated; dividing the data to be calculated into multiple data sets to be calculated according to the data type and calculation requirements; setting the ideal calculation time for the data set to be calculated according to the processing period of the data in the data set to be calculated; and calculating the load generated when the data set to be calculated is processed in the computing device according to the data type and data volume of the data set to be calculated to obtain the load.

[0007] Compared with the existing technology, the technical effect achieved by adopting this technical solution is as follows: by setting the ideal computing time, the subsequent computing tasks can be better allocated, ensuring that all data can be calculated within the processing period, improving the rationality of the system's computing resource allocation, and obtaining the load through calculation, which can provide data support for the subsequent allocation of data sets to be calculated, thereby improving the utilization rate of computing resources.

[0008] In one embodiment of the present invention, all data sets to be calculated whose actual calculation time is less than or equal to the ideal calculation time are recorded as original groups, and the data sets to be calculated in the original groups are allocated to obtain an initial allocation plan, which specifically includes: screening out all devices to be calculated whose actual calculation time is less than or equal to the ideal calculation time, and the corresponding computing devices to obtain the original group; allocating the data sets to be calculated in the original group according to the load and the maximum load to obtain candidate plans; calculating the calculation time of each candidate plan, and selecting the candidate plan with the shortest calculation time as the initial allocation plan.

[0009] Compared with the existing technology, the technical effect achieved by adopting this technical solution is as follows: by obtaining the original group, it is possible to quickly screen out allocation methods that can directly perform allocation and complete calculations within the time limit, thereby improving the efficiency of computing resource allocation; by obtaining the candidate plans and the initial allocation plan, it is possible to quickly screen out the allocation plan with the shortest calculation time, thereby improving the computing efficiency of the data to be calculated and the rationality of computing resource allocation.

[0010] In one embodiment of the present invention, according to the initial allocation plan, all unallocated computing data sets are obtained to obtain parallel data sets, and the parallel data sets are pipelined and parallelized to obtain pipeline modules, which specifically includes: according to the initial allocation plan, all unallocated data sets to be calculated are obtained to obtain parallel data sets; according to the data type and hierarchical structure information of the parallel data sets, the parallel data sets are split into multiple pipeline modules, the load is calculated, and the priority is divided; according to the initial allocation plan, the remaining load of each computing device is calculated; according to the load, priority and remaining load, the pipeline modules are allocated to obtain a parallel mode.

[0011] Compared with the existing technology, the technical effect achieved by adopting this technical solution is as follows: by obtaining priority, the task scheduling and resource allocation of the computing device in the computing process can be made more reasonable, ensuring that key data can be processed first, and the calculation of the remaining load can provide data support for the subsequent computing resources to be calculated normally according to the parallel strategy of the pipeline module when the computing resources are tight, thereby optimizing the allocation logic of computing resources.

[0012] In one embodiment of the present invention, the initial communication efficiency of the parallel data set is obtained according to the parallel mode of the pipeline module, and the initial calculation time of the parallel data set is calculated according to the initial communication efficiency, specifically including: according to the parallel mode, calculating the average value of the transmission speed between the computing devices to obtain the initial communication efficiency; according to the calculation rate, calculating the actual calculation time of each pipeline module; according to the initial communication efficiency and the actual calculation time, calculating the initial calculation time of the parallel data set.

[0013] Compared with the prior art, the technical effect achieved by adopting this technical solution is as follows: by calculating the initial communication efficiency, the specific situation of data transmission between various computing devices can be obtained, avoiding the discrepancy between the prediction of the initial calculation time of the parallel data set and the actual situation due to the low communication efficiency, and improving the accuracy of the initial calculation time calculation. By obtaining the initial calculation time, it can be clear whether the parallel data set can complete the calculation within the time limit, thereby improving the rationality of computing resource allocation.

[0014] In one embodiment of the present invention, the initial communication efficiency is corrected according to the initial calculation time to obtain the theoretical calculation time of the parallel data set, specifically including: obtaining the change of the initial communication efficiency over time according to historical data to obtain the actual communication efficiency; and calculating the theoretical calculation time according to the actual communication efficiency and the initial calculation time.

[0015] Compared with the existing technology, the technical effect achieved by adopting this technical solution is as follows: by correcting the initial communication efficiency, it is possible to avoid the impact of the decrease in communication efficiency caused by long-term data transmission on the allocation of computing resources, ensuring that the calculation time of parallel data sets can be accurately estimated, and the theoretical calculation time can be obtained, providing data support for subsequent judgments on whether the initial allocation plan needs to be modified, thereby improving the rationality of computing resource allocation.

[0016] In one embodiment of the present invention, when the initial allocation plan does not need to be modified, the final allocation plan is obtained through a parallel method. When the initial allocation plan needs to be modified, part of the data sets to be calculated in the original group are pipelined and processed in parallel according to the theoretical calculation time to obtain the final allocation plan, which specifically includes: if the initial allocation plan does not need to be modified, the final allocation plan is obtained according to the initial allocation plan and the parallel method; if the initial allocation plan needs to be modified, the difference between the theoretical calculation time and the ideal calculation time is calculated to obtain the overrun time; the theoretical calculation time of each data set to be calculated in the original group after pipeline parallel processing is estimated, and the difference between the theoretical calculation time and the actual calculation time is calculated to obtain the surplus time; according to the overrun time and the surplus time, part of the data sets to be calculated are selected for pipeline parallel processing, and the corresponding parallel method is calculated; the initial allocation plan is adjusted according to the parallel method to obtain the final allocation plan.

[0017] Compared with the existing technology, the technical effect achieved by adopting this technical solution is as follows: by calculating the overrun time, it is possible to clarify how much extra time is specifically required for the power grid system, which can provide a basis for selecting the appropriate data set to be calculated for pipeline parallel processing; by calculating the surplus time, it is possible to clarify the time benefit that can be obtained if each data to be calculated is processed in parallel through pipeline, thereby improving the rationality of subsequent computing resource allocation.

[0018] In one embodiment of the present invention, a resource allocation system based on pipeline parallelism is also provided. The resource allocation method based on pipeline parallelism recorded in the above embodiment is applied to the resource allocation system. The resource allocation system includes: a storage module, the storage module is used to store the data set to be calculated and the pipeline module; the processing module, the processing module is used to allocate the data set to be calculated and the pipeline module; the calculation module, the calculation module is used to calculate the actual calculation time, the initial communication efficiency and the theoretical calculation time; the allocation module, the allocation module is used to execute the final allocation plan. The resource allocation system has all the technical features of the above resource allocation method, which will not be repeated here one by one. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 One of the flow charts of the resource allocation method based on pipeline parallelism of the present invention; Figure 2 One of the flow charts of the resource allocation method based on pipeline parallelism of the present invention; Figure 3 One of the flow charts of the resource allocation method based on pipeline parallelism of the present invention; Figure 4 One of the flow charts of the resource allocation method based on pipeline parallelism of the present invention; Figure 5 A system schematic diagram of a resource allocation system based on pipeline parallelism of the present invention; Description of reference numerals: 100 - resource allocation system; 110 - storage module; 120 - processing module; 130 - computing module; 140 - allocation module. DETAILED DESCRIPTION

[0020] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0021] [First embodiment] See also Figure 1 In a specific embodiment, the present invention provides a resource allocation method based on pipeline parallelism, the resource allocation method comprising: S100, dividing the data to be calculated into a plurality of data sets to be calculated, and calculating the load of each data set to be calculated according to the ideal calculation time; S200, testing the computing device to obtain the computing rate and maximum load of the computing device, and calculating the actual computing time according to the computing rate and the data set to be calculated; S300, record all the data sets to be calculated whose actual calculation time is less than or equal to the ideal calculation time as the original group, and allocate the data sets to be calculated in the original group to obtain an initial allocation plan; S400, according to the initial allocation plan, obtaining all unallocated data sets to be calculated, obtaining parallel data sets, and performing pipeline parallel processing on the parallel data sets to obtain pipeline modules; S500, obtaining an initial communication efficiency of the pipeline module according to the parallel mode of the pipeline module, and calculating an initial calculation time of the parallel data set according to the initial communication efficiency; S600, correcting the initial communication efficiency according to the initial calculation time to obtain the theoretical calculation time of the parallel data set; S700: Obtain a final allocation plan based on the theoretical calculation time and the initial allocation plan.

[0022] In step S100, since the computing resources in the power grid system are limited and the real-time requirements are high, the types of data to be calculated are complex and the data volume is large, including measurement data, power grid operation data, distribution network data and other power system related data. If the data to be calculated is calculated directly, a large amount of computing resources will be consumed, the computing efficiency will be low, and a large amount of computing load will be placed on the computing equipment. Therefore, in order to improve the computing efficiency, the data to be calculated is divided into multiple data sets, namely, the data sets to be calculated, which are used to reasonably allocate computing tasks.

[0023] It should be noted that the load of a single data set to be calculated should be less than the maximum load of a single designed calculation device.

[0024] The ideal computing time refers to the remaining time in which the data set to be calculated can be processed in a timely manner according to the needs of the power grid system. For example, in the power grid system, the data set to be calculated obtained at 8:00 a.m. on November 4, 2024 includes meteorological data for the weather forecast on November 5, 2024. The ideal computing time of the data set to be calculated can be set to 10 hours or 12 hours. The specific ideal computing time setting can be adjusted according to actual needs.

[0025] Load refers to the computing device load generated by processing the data set to be calculated within the ideal computing time. It should be noted that due to the aging, replacement and expansion of computing equipment, the computing performance of each computing device varies. Therefore, for the same data set to be calculated, the load generated by processing it in different computing devices may also be different, and need to be calculated separately.

[0026] In step S200, normally, all computing devices should be of the same model. If there are computing devices of different models, the computing power needs to be converted according to the hardware of the computing devices. The computing devices are tested and the average computing speed of the computing devices during the test time is calculated to obtain the computing rate. The maximum load is obtained by calculating the average load size when the computing devices are running at full power during the test time. It should be noted that the test duration for calculating the computing rate and the maximum load should be the same.

[0027] In step S300, since pipeline parallelism requires multiple computing devices to perform collaborative computing, the communication efficiency requirement is relatively high. If all data sets to be calculated are processed in pipeline parallel, the communication burden between computing devices will be greatly increased, which may lead to the inability to continue communication and affect the processing efficiency of the data sets to be calculated, wasting a lot of computing resources. Therefore, for the data sets to be calculated that can be completed within the ideal computing time even without pipeline parallel processing, they are directly allocated and calculated to obtain an initial allocation plan.

[0028] In step S400, since only part of the data sets to be calculated are allocated in the initial allocation plan, the remaining unallocated calculation data sets cannot be processed in time because the actual calculation time is greater than the ideal calculation time. Therefore, pipeline parallel processing is used to shorten the calculation time. A parallel data set refers to a data set to be calculated that needs to be processed in parallel by pipeline. A pipeline module refers to each module obtained by splitting a parallel data set after pipeline parallel processing. It should be noted that the ideal calculation time of a parallel data set is the same as that of its corresponding data set to be calculated.

[0029] In step S500, since the pipeline modules may be assigned to different computing devices for calculation in pipeline parallelism, the communication efficiency between computing devices will affect the calculation efficiency of the parallel data set. Due to the performance differences of computing devices and the data types in the parallel data sets, the communication efficiency between different computing devices is different, and the communication efficiency between different pipeline modules on the same computing device is also different, and they need to be calculated separately. The initial communication efficiency is the communication efficiency when the data of the pipeline module is transmitted between computing devices under normal circumstances. For example, for pipeline module A, if the calculation result is calculated in computing device A and transmitted to computing device B, the initial communication efficiency may be 90%. If the calculation result is calculated in computing device B and transmitted to computing device C, the initial communication efficiency may be 75%. For pipeline module B, if the calculation result is calculated in computing device A and transmitted to computing device B, the communication efficiency may be 63%.

[0030] The initial computation world refers to the time required to compute a parallel data set in a pipelined parallel manner, taking into account the impact of initial communication efficiency.

[0031] In step S600, in actual use of the computing device, as time goes by, the computing device may be affected by temperature and other factors, making it difficult to maintain the initial communication efficiency for a long time. Therefore, for a parallel data set with a long initial calculation time, the communication efficiency may continue to decrease with time during the calculation process, resulting in the initial calculation time being unable to reflect the actual calculation time of the parallel data set. In order to ensure that it can accurately determine whether the parallel data set can complete the calculation within the ideal calculation time, it is necessary to correct the initial communication efficiency according to the initial calculation time, and recalculate the time required for the parallel data set calculation to obtain the theoretical calculation time.

[0032] In step S700, if the theoretical calculation time of at least one parallel data set is greater than the ideal calculation time, the initial allocation calculation needs to be modified; if the theoretical calculation time of all parallel data sets is less than or equal to the ideal calculation time, the initial allocation plan does not need to be modified.

[0033] If the initial allocation plan does not need to be modified, it means that all the data sets to be calculated can be calculated within the ideal computing time. Therefore, the initial allocation plan and the parallel method are combined to clarify the processing and allocation methods of each data set to be calculated, and the final allocation plan is obtained.

[0034] If there is at least one parallel data set whose theoretical calculation time is greater than the ideal calculation time, it means that there is a parallel data set that cannot complete the calculation within the time limit. Therefore, it is necessary to perform pipeline parallel processing on some of the data sets to be calculated in the initial allocation plan to reduce the calculation time, so that the parallel data sets that cannot complete the calculation in time can start processing in advance to ensure that the calculation can be completed within the time limit. For example, according to the initial allocation plan, the data set A to be calculated is calculated in device A and is expected to be completed at 3 pm. The parallel data set A is calculated after the data set A to be calculated. The theoretical calculation time is 3h, and the ideal calculation time is 2h. The calculation is expected to be completed at 6 pm and needs to be completed before 5 pm. It is judged that the calculation cannot be completed in time. Therefore, the data set A to be calculated is processed in pipeline parallel, which reduces the calculation time and completes the calculation in advance at 2 pm. Furthermore, the parallel data set A can start calculating in advance at 2 pm and is expected to be completed at 5 pm, so that the calculation can be completed in time.

[0035] By dividing the data to be calculated into multiple data sets to be calculated and reasonably allocating the computing tasks, the difficulty of data processing caused by excessive data volume and complex data types is reduced. By calculating the maximum load, it is ensured that the data sets to be calculated can be reasonably allocated to various computing devices in subsequent work, which improves the utilization rate of computing resources. Through pipeline parallel processing, the allocation of computing resources is optimized, and the computing time of the data sets to be calculated that cannot be calculated in time is reduced. The acquisition and correction of the initial communication efficiency ensures that the theoretical computing time can effectively parallel the actual computing time of the data set, improves the rationality of the subsequent computing resource allocation, and by judging whether the initial allocation plan needs to be modified, it avoids the risk that some parallel data sets cannot complete the calculation within the ideal computing time, effectively solving the problem of limited computing resources and high real-time requirements in the power system.

[0036] [Second embodiment] See also Figure 2 In a specific embodiment, the data to be calculated is divided into a plurality of data sets to be calculated, and the load of each data set to be calculated is calculated according to the ideal calculation time, which specifically includes: S110, acquiring all data that needs to be calculated, and obtaining data to be calculated; S120, dividing the data to be calculated into multiple data sets to be calculated according to data types and calculation requirements; S130, setting an ideal calculation time for the data set to be calculated according to the processing period of the data in the data set to be calculated; S140. Calculate the load generated when the data set to be calculated is processed in the computing device according to the data type and data amount of the data set to be calculated, and obtain the load amount.

[0037] In step S110 to step S120, since the power grid system is complex and real-time, the data type, data acquisition method and data acquisition time of different system structures are different. Therefore, the data to be calculated needs to be acquired in real time and periodically sorted according to actual needs. For example, for a certain power grid system, the distribution data is updated every five minutes and the meteorological data is updated every fifteen minutes. However, due to the different computing requirements of the two, the distribution data is sorted every half hour to obtain the data to be calculated, and the meteorological data is sorted every 1 hour to obtain the data to be calculated.

[0038] Computational requirements refer to the necessary computing conditions that must be met when processing the data to be calculated. According to the calculation, the data to be calculated is divided into multiple data sets to be calculated. For example, in a certain power grid system, it is necessary to predict the weather conditions in area A on September 25 based on the meteorological data of areas A and B on September 24. Then, all meteorological data of areas A and B on September 24 need to be divided into the same data set to be calculated for subsequent calculations.

[0039] In step S130, since it is impossible to ensure that all data sets to be calculated will be calculated within the expected time during the actual calculation process, some margin should be set according to the processing period when setting the ideal calculation time to avoid accidents that may prevent the calculation from being completed within the processing period. For example, in the power grid system, a data set to be calculated acquired at 8:00 a.m. on November 4, 2024 includes meteorological data for weather forecast on November 5, 2024. The data set to be calculated has 16 hours available for calculation, but in order to avoid increased calculation time due to accidents, the ideal calculation is set to 12 hours.

[0040] In step S140, the load includes parameters such as CPU usage, memory occupancy and disk I / O rate. Since the data types of the data sets to be calculated are different, and different computing devices have different capabilities to process different data types, different computing devices generate different loads when calculating the same data set to be calculated. For example, a data set A to be calculated occupies more CPU processing during the calculation process. The memory performance of computing device A is better, but the CPU performance is worse, while the memory performance of computing device B is worse, but the CPU performance is better. In this case, the load of computing data set A in computing device A for calculation is greater than that of computing device B.

[0041] By setting the ideal computing time, the subsequent computing tasks can be better allocated to ensure that all data can be calculated within the processing period, which improves the rationality of the system's computing resource allocation. The load obtained by calculation can provide data support for the subsequent allocation of data sets to be calculated, thereby improving the utilization rate of computing resources.

[0042] [Third embodiment] In a specific embodiment, all the data sets to be calculated whose actual calculation time is less than or equal to the ideal calculation time are recorded as the original group, and the data sets to be calculated in the original group are allocated to obtain an initial allocation plan, which specifically includes: S310, filter out all data sets to be calculated whose actual calculation time is less than or equal to the ideal calculation time, and obtain corresponding calculation devices to obtain an original group; S320, allocating the data sets to be calculated in the original group according to the load amount and the maximum load to obtain a candidate solution; S330, calculating the calculation time of each candidate solution, and selecting the candidate solution with the shortest calculation time as the initial allocation plan.

[0043] In step S310, since the time required to calculate the data set to be calculated in different computing devices is different, it is necessary to screen out the data sets to be calculated and the corresponding computing devices whose actual computing time is less than or equal to the ideal computing time, and record them as the original group. For example, for data set A to be calculated, the ideal computing time is 30 minutes. When calculating in computing device A, the actual computing time is 1 hour. When calculating in computing device B, the actual computing time is 20 minutes. When calculating in computing device B, the actual computing time of data set A to be calculated is less than or equal to the ideal computing time. Therefore, device A to be calculated and computing device B are recorded as the original group.

[0044] In step S320, the data sets to be calculated included in all the original groups are obtained. For the same data set to be calculated, the corresponding data set to be calculated is allocated according to the original group with the shortest actual calculation time. During the allocation process, for the same computing device, the total load of the allocated data sets to be calculated shall not be greater than the maximum load of the computing device. Furthermore, if the computing device corresponding to the original group with the shortest actual calculation time for a certain data set to be calculated does not have sufficient remaining load for the calculation of the data set to be calculated, then the data set to be calculated is allocated according to the original group with the shortest calculation time among the remaining original groups of the data set to be calculated, and so on. If the data set to be calculated cannot be allocated to any corresponding computing device according to the original group, then the data set to be calculated will not be allocated.

[0045] For example, for the data set A to be calculated, there are two corresponding original groups. The original group A is calculated in the computing device A, and the actual calculation time is 30 minutes. The original group B is calculated in the computing device B, and the actual calculation time is 45 minutes. Then the data set A to be calculated is preferentially allocated to the computing device A. If the remaining load of the computing device A is less than the load of the data set A to be calculated, the data set A to be calculated is allocated to the computing device B. If the remaining load of the computing device B is also less than the load of the data set A to be calculated, the data set A to be calculated will not be allocated.

[0046] In step S330, since there are multiple data groups to be calculated, there may be multiple alternative plans due to differences in the allocation order. The calculation time of each alternative plan is calculated, and the alternative plan with the shortest calculation time is selected as the initial allocation plan. The calculation time refers to the time required to complete the calculation of all allocated data sets to be calculated.

[0047] For example, if the alternative plan A allocates the data set A to be calculated to computing device A, and the data set B to be calculated to computing device B, and the calculation time is 3 hours, and the alternative plan B allocates the data set A to be calculated to computing device B, and the data set B to be calculated to computing device A, and the calculation time is 2.5 hours, then the alternative plan B is selected as the initial allocation plan.

[0048] By obtaining the original group, we can quickly screen out allocation methods that can directly allocate and complete the calculation within the time limit, thereby improving the efficiency of computing resource allocation. By obtaining the candidate plans and the initial allocation plan, we can quickly screen out the allocation plan with the shortest calculation time, thereby improving the calculation efficiency of the data to be calculated and the rationality of computing resource allocation.

[0049] [Fourth embodiment] See also Figure 3 In a specific embodiment, according to the initial allocation plan, all unallocated computing data sets are obtained to obtain parallel data sets, and pipeline parallel processing is performed on the parallel data sets to obtain a pipeline module, which specifically includes: S410, according to the initial allocation plan, acquiring all unallocated data sets to be calculated to obtain parallel data sets; S420, splitting the parallel data set into multiple pipeline modules according to the data type and hierarchical structure information of the parallel data set, calculating the load, and dividing the priorities; S430, calculating the remaining load of each computing device according to the initial allocation plan; S440, allocating pipeline modules according to load, priority and remaining load to obtain a parallel mode.

[0050] In step S410, in the power grid system, due to the performance limitation of the computing equipment, there may be some data sets to be calculated that cannot be processed by direct calculation within the time limit. Therefore, the initial allocation plan may not allocate all data sets to be calculated. The remaining unallocated data sets to be calculated need to be processed in parallel through the pipeline to complete the calculation within the ideal computing time. Therefore, these data sets to be calculated are marked as parallel data sets.

[0051] In step S420, usually, the data type of the parallel data set is complex and the data volume is large. If these data are calculated directly, it will generate too much load on the computing device. Therefore, the same type or related data in the parallel data set are integrated and processed and split into multiple pipeline modules.

[0052] It is worth noting that the data volume of the pipeline module is set according to the performance of the computing device. If the performance of the computing device is relatively good, the data volume of the pipeline module can be set relatively large. If the performance of the computing device is relatively poor, the data volume of the pipeline module can be set relatively small. The specific standard depends on the actual situation, and the data volume of each pipeline module may vary greatly.

[0053] Since pipeline parallelism distributes different pipeline modules of data to different computing devices to form a pipeline for parallel computing, there may be a subordinate relationship between pipeline modules in actual computing. For example, pipeline module B needs to be calculated based on the calculation results of pipeline module A. Therefore, priority should be set for each sub-module according to the subordinate relationship between each sub-module to ensure that when there are too many sub-modules and the computing device cannot calculate all sub-modules at the same time, the more basic sub-modules can be calculated first to ensure that the data can be processed normally.

[0054] In step S430, the remaining load refers to the remaining load of the computing device that can be used to process the data set to be calculated. According to the initial allocation plan, the load of the data set to be calculated allocated to each computing device is known, and the difference between the total load of the data set to be calculated and the maximum load of the corresponding computing device is calculated to obtain the remaining load.

[0055] It should be noted that the remaining load does not refer to a single performance parameter of the computing device, but includes parameters such as CPU usage, memory occupancy, and disk I / O rate.

[0056] In step S440, the parallel mode refers to a specific allocation plan for all pipeline modules, which needs to ensure that the total load of the computing device does not exceed its maximum load based on the remaining load of the computing device and the load of the pipeline module. Furthermore, in order to ensure the computing efficiency of the pipeline modules with subordinate relationships, the pipeline modules with high priority should be allocated first, and the pipeline modules with low priority should be allocated later. For example, if a parallel data set is divided into two pipeline modules, namely pipeline module A and pipeline module B, the calculation of pipeline module B requires the calculation result data support of pipeline module A, so pipeline module A should be allocated first to ensure that pipeline module B can be calculated in a timely manner in a parallel manner.

[0057] By obtaining the priority, the task scheduling and resource allocation of the computing device during the computing process can be made more reasonable, ensuring that key data can be processed first. The calculation of the remaining load can provide data support for the subsequent calculation according to the parallel strategy of the pipeline module when the computing resources are tight, thereby optimizing the allocation logic of computing resources.

[0058] [Fifth embodiment] In a specific embodiment, obtaining the initial communication efficiency of the parallel data set according to the parallel mode of the pipeline module, and calculating the initial calculation time of the parallel data set according to the initial communication efficiency specifically includes: S510, calculating the average value of the transmission speed between the computing devices in a parallel manner to obtain an initial communication efficiency; S520, calculating the actual calculation time of each pipeline module according to the calculation rate; S530: Calculate the initial computing time of the parallel data set according to the initial communication efficiency and the actual computing time.

[0059] In step S510, the initial communication efficiency refers to the ratio of the actual transmission data volume per unit time between the calculation groups to the expected transmission data volume, and the calculation formula is: H c =C e / C a ×100%,H c is the initial communication efficiency, C e is the actual transmission rate, C a is the expected transmission rate.

[0060] For example, if the expected transmission rate is 30 gbps and the actual transmission rate is 27 gbps, then the initial communication efficiency H c =C e / C a×100%=(27gbps / 30gbps)×100%. =90%.

[0061] In step S520 to step S530, the actual calculation time of the pipeline module is calculated in the same way as the data set to be calculated. Since the time for the pipeline module to complete the calculation may be affected by the communication efficiency, in order to ensure the accuracy of the initial calculation time of the parallel data set, when calculating the initial calculation time of the parallel data set by the actual calculation time of the pipeline module, the influence of the communication efficiency on the calculation efficiency needs to be taken into account. For example, a parallel data set is divided into three pipeline modules: pipeline module A, pipeline module B, and pipeline module C. Block A can be calculated directly, and the actual calculation time is 20 minutes. The calculation of pipeline module B requires the calculation result of pipeline module A as data support. The initial communication efficiency is 85%, and the actual calculation time is 30 minutes. The calculation of pipeline module C requires the calculation result of pipeline module B as data support. The initial communication efficiency is 90%, and the actual calculation time is 40 minutes. The initial calculation time of the parallel data set can be calculated as = 20 minutes + 30 minutes ÷ 85% + 40 minutes ÷ 90% ≈ 99.74 minutes.

[0062] By calculating the initial communication efficiency, we can get the specific situation of data transmission between various computing devices, avoid the prediction of the initial computing time of the parallel data set being inconsistent with the actual situation due to too low communication efficiency, and improve the accuracy of the initial computing time calculation. By getting the initial computing time, we can clarify whether the parallel data set can complete the calculation within the time limit, thereby improving the rationality of computing resource allocation.

[0063] [Sixth embodiment] In a specific embodiment, the initial communication efficiency is corrected according to the initial calculation time to obtain the theoretical calculation time of the parallel data set, which specifically includes: S610, obtaining the change of the initial communication efficiency over time according to the historical data, and obtaining the actual communication efficiency; S620: Calculate theoretical calculation time according to actual communication efficiency and initial calculation time.

[0064] In step S610 to step S620, as the communication efficiency of the computing device may decrease with the increase of transmission time, the initial communication efficiency needs to be corrected according to the change of communication efficiency, and the required time needs to be recalculated to obtain the theoretical calculation time. For example, when the technical results of pipeline module A need to be transmitted from computing device A to computing device B, the initial calculation time is 2h, and the initial communication efficiency is maintained at 95% for the first 15mins. From 15min to 40min, the communication efficiency decreases to 80% of the initial communication efficiency. After 40min, the communication efficiency is maintained at 75% of the initial communication efficiency. Then, it can be calculated that the theoretical calculation time = 15mins + 25mins ÷ 80% + 80mins ÷ 75% ≈ 152.92mins.

[0065] By correcting the initial communication efficiency, the impact of decreased communication efficiency due to long-term data transmission on computing resource allocation can be avoided, ensuring that the computing time of parallel data sets can be accurately estimated and the theoretical calculation time can be obtained. This provides data support for subsequent judgments on whether the initial allocation plan needs to be modified, thereby improving the rationality of computing resource allocation.

[0066] [Seventh embodiment] See also Figure 4 In a specific embodiment, the final allocation plan is obtained according to the theoretical calculation time and the initial allocation plan, specifically including: S710, if the initial allocation plan does not need to be modified, then obtain a final allocation plan according to the initial allocation plan and the parallel method; S720, if the initial allocation plan needs to be modified, the difference between the theoretical calculation time and the ideal calculation time is calculated to obtain the overspending time; S730, estimating the theoretical calculation time of each to-be-calculated data set in the original group after pipeline parallel processing, and calculating the difference between the theoretical calculation time and the actual calculation time to obtain the surplus time; S740, selecting part of the data set to be calculated for pipeline parallel processing according to the overspending time and the surplus time, and calculating a corresponding parallel mode; S750: Adjust the initial allocation plan in a parallel manner to obtain a final allocation plan.

[0067] In step S710, if the initial allocation plan does not need to be modified, it means that all the data sets to be calculated can be calculated within the ideal calculation time, so each data set to be calculated can be directly allocated and calculated according to the initial allocation plan and in a parallel manner.

[0068] In steps S720 to S730, the overrun time refers to how much time the computing device needs to spend beyond the ideal computing time to complete the calculation of all data sets to be calculated. The surplus time refers to how much time can be spent in advance of the actual calculation time to complete the calculation of the data set to be calculated if the data set to be calculated is calculated in a pipeline parallel manner. If the initial allocation plan needs to be modified, it means that there are data sets to be calculated that cannot be calculated within the ideal computing time.

[0069] In steps S740 to S750, due to the performance limitation of the computing device, if there is a data set to be calculated that is calculated in a pipeline parallel manner, it may cause excessive communication pressure between the computing devices, thereby causing the communication efficiency to be too low, affecting the overall computing efficiency. At the same time, too many pipeline modules will increase the difficulty of the system to allocate computing tasks, so the sum of the surplus time of the selected data set to be calculated should be as close to the overrun time as possible. Further, in order to avoid fluctuations in computing time caused by other factors, a certain margin should be set for the surplus time. For example, if a power grid system allocates a batch of data to be calculated, the overrun time is 5h, if the data module A to be calculated, the data module B to be calculated, and the data module C to be calculated are pipelined in parallel, the sum of the surplus time is 5h, and the data module B to be calculated, the data module C to be calculated, and the data module D to be calculated are pipelined in parallel, the sum of the surplus time is 6h, then the data module B to be calculated, the data module C to be calculated, and the data module D to be calculated should be selected for pipeline parallel processing. The specific reserved amount can be adjusted according to actual needs and actual conditions.

[0070] By calculating the overrun time, we can determine how much extra time the power grid system needs, which can provide a basis for selecting the appropriate data set to be calculated for pipeline parallel processing. By calculating the surplus time, we can determine the time benefit that can be obtained if each data set to be calculated is processed in pipeline parallel, thereby improving the rationality of subsequent computing resource allocation.

[0071] [Eighth embodiment] See also Figure 5In a specific embodiment, the present invention also provides a resource allocation system 100 based on pipeline parallelism. The resource allocation method based on pipeline parallelism recorded in the above embodiment is applied to the resource allocation system 100. The resource allocation system 100 includes: a storage module 110, the storage module 110 is used to store the data set to be calculated and the pipeline module; the processing module 120, the processing module 120 is used to allocate the data set to be calculated and the pipeline module; the calculation module 130, the calculation module 130 is used to calculate the actual calculation time, the initial communication efficiency and the theoretical calculation time; the allocation module 140, the allocation module 140 is used to execute the final allocation plan. The resource allocation system 100 has all the technical features of the above resource allocation method, which will not be repeated here one by one.

[0072] Although the present invention is disclosed as above, the present invention is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the scope defined by the claims.

Claims

1. A resource allocation method based on pipeline parallelism, characterized in that: The resource allocation method comprises: Dividing the data to be calculated into a plurality of data sets to be calculated, and calculating the load of each of the data sets to be calculated according to the ideal calculation time; Testing the computing device to obtain a computing rate and a maximum load of the computing device, and calculating an actual computing time according to the computing rate and the data set to be calculated; Recording all the data sets to be calculated whose actual calculation time is less than or equal to the ideal calculation time as an original group, and allocating the data sets to be calculated in the original group to obtain an initial allocation plan; According to the initial allocation plan, all unallocated data sets to be calculated are acquired to obtain parallel data sets, and pipeline parallel processing is performed on the parallel data sets to obtain pipeline modules; Acquire an initial communication efficiency of the pipeline module according to the parallel mode of the pipeline module, and calculate an initial calculation time of the parallel data set according to the initial communication efficiency; Correcting the initial communication efficiency according to the initial calculation time to obtain a theoretical calculation time of the parallel data set; A final allocation plan is obtained according to the theoretical calculation time and the initial allocation plan.

2. The resource allocation method based on pipeline parallelism according to claim 1, characterized in that: The step of dividing the data to be calculated into a plurality of data sets to be calculated, and calculating the load of each data set to be calculated according to the ideal calculation time, specifically includes: Obtain all data that needs to be calculated, and obtain the data to be calculated; Dividing the data to be calculated into a plurality of data sets to be calculated according to data types and calculation requirements; According to the processing period of the data in the data set to be calculated, setting an ideal calculation time for the data set to be calculated; According to the data type and data amount of the data set to be calculated, the load generated when the data set to be calculated is processed in the computing device is calculated to obtain the load amount.

3. The resource allocation method based on pipeline parallelism according to claim 2, characterized in that: The step of recording all the data sets to be calculated whose actual calculation time is less than or equal to the ideal calculation time as an original group, and allocating the data sets to be calculated in the original group to obtain an initial allocation plan specifically includes: Filter out all the data sets to be calculated whose actual calculation time is less than or equal to the ideal calculation time, and obtain the corresponding calculation devices to obtain the original group; Allocating the data sets to be calculated in the original group according to the load amount and the maximum load to obtain a candidate solution; The calculation time of each of the candidate solutions is calculated, and the candidate solution with the shortest calculation time is selected as the initial allocation plan.

4. The resource allocation method based on pipeline parallelism according to claim 3, characterized in that: The method of acquiring all unallocated data sets to be calculated according to the initial allocation plan to obtain parallel data sets, and performing pipeline parallel processing on the parallel data sets to obtain pipeline modules specifically includes: According to the initial allocation plan, all unallocated data sets to be calculated are acquired to obtain parallel data sets; According to the data type and hierarchical structure information of the parallel data set, split the parallel data set into a plurality of pipeline modules, calculate the load, and divide the priorities; Calculating the remaining load of each computing device according to the initial allocation plan; The pipeline modules are allocated according to the load, the priority and the remaining load to obtain the parallel mode.

5. The resource allocation method based on pipeline parallelism according to claim 4, characterized in that: The obtaining the initial communication efficiency of the pipeline module according to the parallel mode of the pipeline module, and calculating the initial calculation time of the parallel data set according to the initial communication efficiency specifically includes: According to the parallel method, an average value of the transmission speed between the computing devices is calculated to obtain an initial communication efficiency; According to the calculation rate, the actual calculation time of each of the pipeline modules is calculated; The initial computing time of the parallel data set is calculated according to the initial communication efficiency and the actual computing time.

6. The resource allocation method based on pipeline parallelism according to claim 5, characterized in that: The correcting the initial communication efficiency according to the initial calculation time to obtain the theoretical calculation time of the parallel data set specifically includes: According to the historical data, the change of the initial communication efficiency over time is obtained to obtain the actual communication efficiency; The theoretical calculation time is calculated according to the actual communication efficiency and the initial calculation time.

7. The resource allocation method based on pipeline parallelism according to claim 6, characterized in that: The obtaining of the final allocation plan according to the theoretical calculation time and the initial allocation plan specifically includes: If the initial allocation plan does not need to be modified, the final allocation plan is obtained according to the initial allocation plan and the parallel method; If the initial allocation plan needs to be modified, the difference between the theoretical calculation time and the ideal calculation time is calculated to obtain the overrun time; Predicting the theoretical calculation time of each of the to-be-calculated data sets in the original group after pipeline parallel processing, and calculating the difference between the theoretical calculation time and the actual calculation time to obtain surplus time; According to the overspending time and the surplus time, select part of the data set to be calculated for pipeline parallel processing, and calculate the corresponding parallel mode; The initial allocation plan is adjusted according to the parallel method to obtain the final allocation plan.

8. A resource allocation system based on pipeline parallelism, characterized in that: The resource allocation method according to any one of claims 1 to 7 is applied to the resource allocation system, and the resource allocation system comprises: A storage module, the storage module is used to store the data set to be calculated and the pipeline module; A processing module, the processing module is used to allocate the data set to be calculated and the pipeline module; A calculation module, the calculation module is used to calculate the actual calculation time, the initial communication efficiency and the theoretical calculation time; An allocation module is used to execute the final allocation plan.

Images