Parallel programming method for diversified computing task requirements

By analyzing the processor memory access cost and task complexity, optimizing data transmission paths and load balancing, and using PID controllers to adjust processor frequency, the efficiency of task allocation and energy management in multi-core or multi-processor systems is solved, and the efficiency of task allocation and energy management in multi-core or multi-processor systems is improved. System performance and efficiency.

CN119065643APending Publication Date: 2024-12-03GUANGXI POWER GRID CORP
View PDF 0 Cites 6 Cited by

Patent Information

Application Number
CN202411095551.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-09
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

In multi-core or multi-processor systems, the prior art usually only considers the task-to-processor mapping, ignores the memory access cost, task complexity and time constraints of real-time tasks, resulting in the task allocation results not being optimized, and increases the complexity of task scheduling. and computational overhead.

Method used

By analyzing the processor's memory access cost and task complexity, the data transmission path between processors is optimized, the load balancing and data storage location is adjusted using greedy algorithms, the fault tolerance mechanism is activated, and the processor frequency is adjusted using the PID controller to Control energy consumption.

Benefits of technology

More effective task scheduling, resource management and energy control are achieved, system performance and efficiency are improved, and task allocation optimization and rational use of energy are ensured.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119065643A_ABST
    Figure CN119065643A_ABST
Patent Text Reader

Abstract

The invention provides a parallel programming method for diversified computing task requirements, which comprises the following steps of: quantifying the memory access cost of each processor by analyzing the specification and capability of multiple cores or multiple processors in a system; according to the memory access cost of the processor and the complexity of the task, performing preliminary mapping from the task to the processor; analyzing the time constraint of the real-time task, and modifying the preliminary mapping scheme according to the time constraint; a data transmission path between the processors is optimized so as to reduce the predicted communication cost; a greedy algorithm is adopted, adjustment is carried out according to the communication cost and the processing capacity between processors, and load balancing is optimized; according to a load balancing result, optimizing a data storage position so as to improve data locality; analyzing results of load balancing and data locality optimization, and starting a predetermined fault-tolerant mechanism; the energy consumption of the processor is predicted, and the frequency of the processor is adjusted by utilizing a PID (Proportion Integration Differentiation) controller so as to control the energy consumption.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a parallel programming method for meeting diverse computing task requirements. Background Art

[0002] With the popularity of multi-core or multi-processor systems, task scheduling, resource management, and energy control have become particularly important. However, existing technologies often only focus on one aspect, while ignoring the overall performance and efficiency of the entire system. Task scheduling, resource management, and energy control is a complex system engineering problem involving multiple factors and variables. Comprehensively considering these factors and optimizing them is a difficult task that requires a lot of computing and data processing. Therefore, in order to simplify the problem and improve feasibility, researchers often focus on one aspect and ignore the others. Existing methods usually only consider the mapping of tasks to processors, while ignoring the memory access cost, task complexity, and time constraints of real-time tasks. Diversified computing task requirements In multi-core or multi-processor systems, different processors have different costs for accessing different memory areas. How to effectively allocate tasks in such an environment is an unsolved problem. Considering more factors will increase the complexity and computational overhead of task scheduling. In practical applications, in order to ensure the feasibility and efficiency of task scheduling, simplified models or heuristic methods are often used for task allocation and scheduling. Such simplification may ignore some details, such as memory access cost and task complexity, resulting in the final task allocation result may not be optimal. Summary of the invention

[0003] The present invention provides a parallel programming method for diversified computing task requirements, which mainly includes:

[0004] By analyzing the specifications and capabilities of the multi-core or multi-processor in the system, the memory access cost of each processor is quantified; a list of diverse computing tasks to be executed is obtained, and the complexity of each task is quantified according to the time and resource amount required by the task; a preliminary mapping of tasks to processors is performed according to the memory access cost of the processor and the complexity of the task; the time constraints of the real-time task are analyzed, and the preliminary mapping scheme is modified according to the time constraints; the data transmission path between processors is optimized to reduce the estimated communication cost; a greedy algorithm is used to adjust according to the communication cost and processing power between processors to optimize load balancing; according to the results of load balancing, the data storage location is optimized to improve data locality; the results of load balancing and data locality optimization are analyzed, and a predetermined fault tolerance mechanism is started; the energy consumption of the processor is predicted, and the processor frequency is adjusted using a PID controller to control energy consumption.

[0005] In some embodiments, analyzing the specifications and capabilities of multiple cores or multiple processors in the system to quantify the memory access cost of each processor includes:

[0006] The processor specifications and multi-core or multi-processor configuration information are obtained through the hardware abstraction layer; it is determined whether the obtained multi-core or multi-processor configuration information meets the system specification requirements; if so, the operating status and computing power data of each processor are collected using CPU-Z; based on the collected operating status and computing power data, the data dependency between the processors is calculated and analyzed, and if the data dependency is higher than the preset data dependency threshold, resource scheduling is performed through the stochastic gradient descent algorithm; based on resource scheduling, the memory access cost of each processor is quantified through the Pearson correlation coefficient.

[0007] In some embodiments, obtaining a list of diverse computing tasks to be performed and quantifying the complexity of each task according to the time and amount of resources required by the task includes:

[0008] Obtain a list of pending computing tasks from the task management system or database, and use query statements to filter out pending tasks; for each pending task, obtain task complexity assessment information, including the time and resources required for the task; and quantitatively assess the complexity of each task based on the time and resources required for the task.

[0009] In some embodiments, performing preliminary mapping of tasks to processors according to the memory access cost of the processor and the complexity of the tasks includes:

[0010] According to the processor's memory access cost and the complexity of the task, the mapping score of each task is calculated; all processors are traversed and the processor with the highest mapping score is selected as the mapping processor of the current task; it is determined whether the selected processor has sufficient computing resources and memory capacity to execute the current task. If not, the next processor with the highest score is selected; it is checked whether the current task has dependencies. If so, it is checked whether the dependent task has been mapped to the processor; if the dependent task has not been mapped, it is waited for the dependent task to be completed before mapping; the load of the selected processor is updated and the mapping of tasks to processors is repeated until all tasks are mapped to processors.

[0011] In some embodiments, analyzing the time constraint of the real-time task and modifying the preliminary mapping scheme according to the time constraint includes:

[0012] According to the time constraints in real-time tasks, the processor's processing speed, data transmission speed and memory capacity parameters are evaluated; the mapping scheme is modified to optimize the processor's processing speed and data transmission speed to the highest level; the mapping scheme is tested to evaluate whether the time it takes for the task unit to complete meets the time constraints of the real-time task; if the time it takes for the task unit to complete does not meet the time constraints of the real-time task, the mapping scheme is readjusted.

[0013] In some embodiments, the optimizing the data transmission path between processors to reduce the estimated communication cost includes:

[0014] The data transmission path between processors is analyzed through the Bellman-Ford algorithm, and the shortest path estimate of each node is updated step by step until it converges to the optimal solution, thereby obtaining the estimated communication cost on the path; based on the estimated communication cost, it is determined whether optimization is required; if optimization is required, the path is reconstructed based on the estimated communication cost to obtain the amount of data under the new path; based on the amount of data under the new path, the transmission rate of the path is calculated; based on the calculated transmission rate, it is determined whether the path meets the delay limit; if the delay limit is met, a network flow check is performed on the new path for data integrity; through the network flow check, it is determined whether the new path has redundancy; if so, the redundant path is deleted.

[0015] In some embodiments, the greedy algorithm is used to adjust according to the communication cost and processing capacity between processors to optimize load balancing, including:

[0016] The communication costs between processors are sorted by a greedy algorithm to form a basic processor usage order; the correlation between the communication cost between processors and the data flow is determined, and the basic processor usage order is adjusted according to the new data flow demand; according to the new processor usage order, the communication protocol is calculated and adjusted to adapt to the new data flow demand; based on the new communication protocol, the communication cost between processors is re-evaluated; the correlation between the computing density of the processor and the newly evaluated communication cost is determined, and the communication cost is reduced through adjustment; on the basis of reducing the communication cost, the processing capacity of the processor is sorted to obtain the optimized processor usage order.

[0017] In some embodiments, optimizing the data storage location according to the load balancing result to improve data locality includes:

[0018] Analyze the load balancing results to obtain hotspot data; use the K-means algorithm to perform data locality analysis on the hotspot data to determine whether there is room for optimization; determine which hotspot data can be moved to the same or adjacent storage locations based on data dependency and data consistency; use the greedy algorithm to calculate the weight ratio of storage cost and access speed; perform quantitative evaluation of I / O bandwidth and system response time. If the I / O bandwidth is lower than the preset bandwidth threshold or the system response time is higher than the preset response time threshold, recommend data migration based on the obtained weight ratio and storage location; after data migration, verify data availability and data integrity.

[0019] In some embodiments, analyzing the results of load balancing and data locality optimization and initiating a predetermined fault tolerance mechanism includes:

[0020] Use K-means algorithm to classify the load of the current system, use Euclidean distance as the similarity measure, and obtain the classification result through iterative calculation; adjust the load balancing parameters according to the classification results of the K-means algorithm; perform task allocation and generate task execution report according to the adjusted load balancing parameters; use greedy algorithm to analyze task execution report and data locality, perform resource scheduling, and generate performance report after resource scheduling; analyze the performance report to determine whether the system stability is lower than the predetermined stability threshold, and execute the predetermined fault tolerance mechanism; analyze the performance report after the fault tolerance mechanism is executed, and if there is still a problem, trigger the predetermined fault tolerance mechanism again.

[0021] In some embodiments, predicting the energy consumption of the processor and adjusting the processor frequency using a PID controller to control the energy consumption includes:

[0022] The historical energy consumption data and corresponding input features of the processor are obtained, and the input features are normalized, wherein the input features include load, temperature and voltage; the energy consumption of the processor is used as the target variable, and the input features are used as the input of the model; the LSTM network structure is used for training, and the data is divided into a training set and a validation set, and mean square error and Adam model optimization are used at the same time; the real-time input features are used as input, and the trained LSTM model is used to predict the energy consumption of the processor; the prediction result is compared with the actual energy consumption data to calculate the prediction error; the prediction error is used as a feedback signal and input into the PID controller; a control signal, i.e., the frequency adjustment amount, is calculated according to the size and change rate of the prediction error; the control signal is applied to the processor to adjust the operating frequency accordingly to control the energy consumption; new actual energy consumption data is continuously collected, and prediction and control are performed.

[0023] The technical solution provided by the embodiment of the present invention may have the following beneficial effects:

[0024] The present invention discloses a technology that integrates multi-core or multi-processor specifications and capability analysis, by quantifying the processor's memory access cost and the complexity of the task, and mapping the task to the processor. Next, the mapping scheme is adjusted according to the time constraints of the real-time task, and the data transmission path between the processors is optimized to reduce the communication cost. Subsequently, a greedy algorithm is used to optimize the load balancing, and adjustments are made according to the communication cost and processing power between the processors. Then, the data storage location is optimized according to the results of the load balancing to improve data locality. After analyzing the results of the load balancing and data locality optimization, a predetermined fault-tolerant mechanism is started. Finally, the processor's energy consumption is predicted and the processor frequency is adjusted using a PID controller to control energy consumption. Combining the above technologies, the present invention provides an effective task scheduling, resource management and energy control method that can improve the performance and efficiency of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 The present invention is a flowchart of a parallel programming method for meeting diverse computing task requirements.

[0026] Figure 2 A schematic diagram of a parallel programming method for diverse computing task requirements according to the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0028] A parallel programming method for diversified computing task requirements in this embodiment may specifically include:

[0029] Step S101 , quantifying the memory access cost of each processor by analyzing the specifications and capabilities of the multi-cores or multi-processors in the system.

[0030] The processor specifications and multi-core or multi-processor configuration information are obtained through the hardware abstraction layer. It is determined whether the obtained multi-core or multi-processor configuration information meets the system specification requirements. If it meets the requirements, CPU-Z is used to collect the operating status and computing power data of each processor. Based on the collected operating status and computing power data, the data dependency between each processor is calculated and analyzed. If the data dependency is higher than the preset data dependency threshold, the resource scheduling is performed through the stochastic gradient descent algorithm. Based on the resource scheduling, the memory access cost of each processor is quantified by the Pearson correlation coefficient. For example, there is a system with two processors: processor A and processor B. Now it is necessary to obtain the specifications and configuration information of these two processors and determine whether they meet the system specification requirements. Through the hardware abstraction layer, it can be obtained that the specifications of processor A are Intel Core i7-9700K, with 8 cores and 16 threads. The specifications of processor B are AMD Ryzen 9 3900X, with 12 cores and 24 threads. According to the system specification requirements, the system needs to have at least 8 cores and 16 threads. Since the configuration information of processor A and processor B meet the requirements respectively, they both meet the system specification requirements. Next, the CPU-Z software can be used to collect the operating status and computing power data of each processor. Through CPU-Z, it can be obtained that the operating status of processor A is working at a frequency of 6GHz, the load is 50%, and the temperature is 60 degrees Celsius. The operating status of processor B is working at a frequency of 2GHz, the load is 70%, and the temperature is 65 degrees Celsius. At the same time, it can be obtained that the computing power of processor A is 8GFLOPS, while the computing power of processor B is 10GFLOPS. Next, the data dependency between the processors can be calculated and analyzed based on the collected data. There is a task in the system that needs to run on processor A and processor B at the same time, and the calculation result of processor A will be used as the input data of processor B. By analyzing the data dependency of the calculation task, the degree of dependence of processor B on the calculation result of processor A can be determined. If the data dependency is higher than the preset data dependency threshold, that is, processor B strongly depends on the calculation result of processor A, then the stochastic gradient descent algorithm can be used for resource scheduling to ensure that the calculation task of processor A is performed before processor B. Based on resource scheduling, the Pearson correlation coefficient can be used to quantify the memory access cost of each processor. For example, the memory access cost of processor A is 0.8, while the memory access cost of processor B is 0.6. By comparing the Pearson correlation coefficients of the two, it can be concluded that processor A has a higher memory access cost than processor B.

[0031] Step S102 , obtaining a list of diverse computing tasks to be executed, and quantifying the complexity of each task according to the time and resources required by the task.

[0032] Get a list of pending computing tasks from the task management system or database, and use query statements to filter out pending tasks. For each pending task, get the complexity assessment information of the task, including the time and resources required for the task. Quantitatively evaluate the complexity of each task based on the time and resources required for the task. For example, get a list of all pending computing tasks from the task management system or database, and use the following query statement to filter out pending tasks: SELECT * FROM tasks WHERE status = 'pending'; the query results include the following two pending tasks: Task 1 is expected to take 10 hours and require 5 CPU cores; Task 2 is expected to take 5 hours and require 3 CPU cores. Based on the time and resources required for the task, the complexity of each task can be quantitatively evaluated. Multiply the time and resources of the task to get a complexity assessment index. The complexity evaluation index of Task 1 is 10 hours * 5 CPU cores = 50, and the complexity evaluation index of Task 2 is 5 hours * 3 CPU cores = 15. By quantitatively evaluating the complexity evaluation index of tasks, we can compare the complexity of different tasks and provide a reference for task priority sorting or resource allocation. The higher the complexity evaluation index of Task 1, the more complex it is, which means that the task is more complex than Task 2 and may require more time and resources to execute.

[0033] Step S103, performing preliminary mapping of tasks to processors according to the memory access cost of the processor and the complexity of the tasks.

[0034] Calculate the mapping score of each task based on the processor's memory access cost and the complexity of the task. Traverse all processors and select the processor with the highest mapping score as the mapping processor for the current task. Determine whether the selected processor has sufficient computing resources and memory capacity to execute the current task. If not, select the next processor with the highest score. Check whether the current task has dependencies. If so, check whether the dependent task has been mapped to the processor. If the dependent task has not been mapped, wait until the dependent task is completed before mapping. Update the load of the selected processor and repeat the mapping of tasks to processors until all tasks are mapped to processors. For example, there are the following three tasks that need to be mapped to two processors. The processor memory access cost of task 1 is 10 and the task complexity is 5. The processor memory access cost of task 2 is 8 and the task complexity is 3. The processor memory access cost of task 3 is 6 and the task complexity is 7. First, calculate the mapping score of each task. Task 1 mapping score = memory access cost * task complexity = 10*5 = 50. Task 2 mapping score = 8*3 = 24, and task 3 mapping score = 6*7 = 42. Next, traverse the processors and select the processor with the highest mapping score as the mapping processor for the current task. If the computing resources and memory capacity of processor 1 are sufficient to execute task 1, task 1 will be mapped to processor 1. Then, check whether task 2 has any dependencies. Task 2 depends on the completion of task 1. Since task 1 has been mapped to processor 1, task 2 can be mapped to processor 1. Finally, check whether task 3 has any dependencies. Task 3 depends on the completion of task 2. Since task 2 has been mapped to processor 1, task 3 can be mapped to processor 1. Update the load of processor 1 and repeat the above steps to select the processor with the highest mapping score and check the dependencies of the tasks until all tasks are mapped to processors.

[0035] Step S104: analyzing the time constraints of the real-time tasks and modifying the preliminary mapping scheme according to the time constraints.

[0036] According to the time constraints in real-time tasks, the processing speed, data transmission speed and memory capacity parameters of the processor are evaluated. The mapping scheme is modified to optimize the processing speed and data transmission speed of the processor to the highest. The mapping scheme is tested to evaluate whether the time when the task unit is completed meets the time constraints of the real-time task. If the time when the task unit is completed does not meet the time constraints of the real-time task, the mapping scheme is readjusted. For example, there is a real-time task system that contains 3 task units, named task 1, task 2 and task 3. These task units need to be completed under strict time constraints, task 1 is completed within 100 milliseconds, task 2 is completed within 200 milliseconds, and task 3 is completed within 150 milliseconds. First, the processing speed, data transmission speed and memory capacity parameters of the processor need to be evaluated. The processing speed of the processor is 1GHz, the data transmission speed is 100MB / s, and the memory capacity is 1GB. Next, the mapping scheme needs to be modified to optimize the processing speed and data transmission speed of the processor. Use a multi-core processor and map task 1 to processor 1, task 2 to processor 2, and task 3 to processor 3. In addition, a cache is used to increase the data transmission speed, and the input data and output data of the task are stored in the cache. Then, the mapping scheme is tested to evaluate whether the time for the task unit to complete meets the time constraints of the real-time task. Task 1 needs to read 100MB of data from the memory and write the results to the memory after processing. The whole process takes 50 milliseconds. Task 2 needs to read 200MB of data from the memory and write the results to the memory after processing. The whole process takes 100 milliseconds. Task 3 needs to read 150MB of data from the memory and write the results to the memory after processing. The whole process takes 75 milliseconds. Based on the above data, it can be calculated whether the time for each task unit to complete meets the time constraints of the real-time task. Task 1 completes in 50 milliseconds, which meets the requirements; Task 2 completes in 100 milliseconds, which also meets the requirements; Task 3 completes in 75 milliseconds, which also meets the requirements. If the time for the task unit to complete does not meet the time constraints of the real-time task, the mapping scheme needs to be readjusted. You can try to increase the processor's processing speed, data transmission speed, or memory capacity, or reallocate the mapping scheme of the task unit to meet the time constraint requirements.

[0037] Step S105, optimizing the data transmission path between processors to reduce the estimated communication cost.

[0038] The data transmission path between processors is analyzed by the Bellman-Ford algorithm, and the shortest path estimate of each node is updated step by step until it converges to the optimal solution, and the estimated communication cost on the path is obtained. According to the estimated communication cost, it is determined whether optimization is required. If optimization is required, the path is reconstructed based on the estimated communication cost to obtain the amount of data under the new path. According to the amount of data under the new path, the transmission rate of the path is calculated. According to the calculated transmission rate, it is determined whether the path meets the delay limit. If the delay limit is met, the network flow verification of the data integrity of the new path is performed. Through the network flow verification, it is determined whether the new path has redundancy. If so, the redundant path is deleted. For example, there is a processor network consisting of 5 nodes, each node represents a processor, and data is transmitted between nodes through communication links. It is necessary to determine the shortest path from node A to node E and evaluate the estimated communication cost on the path. First, the Bellman-Ford algorithm is used to iteratively update the shortest path estimate of each node. Initially, the shortest path estimate of other nodes except node A is infinite, and the shortest path estimate of node A is 0. Then, it starts to iteratively update the shortest path estimate of each node until it converges to the optimal solution. In the first iteration, it is found that the distance from node A to node B is 2, and the shortest path estimate of node B is updated to 2. Next, it is found that the distance from node A to node C is 4, and the shortest path estimate of node C is updated to 4. Continue iterating, and it may be found that the shortest path from node A to node E is 6, and the shortest path estimate of node E is updated to 6. Finally, the shortest path estimate of node E is 6, indicating that the estimated communication cost of the shortest path from node A to node E is 6. According to the estimated communication cost, it can be determined whether the path needs to be optimized. Set the threshold to 5. If the estimated communication cost is greater than 5, optimization is required. Decide to reconstruct the path and find the amount of data under the new path. It may be found that the amount of data on the new path is 10. According to the amount of data under the new path, the transmission rate can be calculated. Assuming the transmission rate is 2, it means that 2 units of data can be transmitted per unit time. Next, it is necessary to determine whether the path meets the delay constraint. The delay limit is 3, which means that the data transmission needs to be completed within 3 units of time. Since the transmission rate is 2, the transmission can be completed within 5 units of time, meeting the delay limit. Then, the network flow check of the data integrity of the new path is performed to ensure that the data is not lost or damaged. Through the network flow check, it is determined that there is no data redundancy on the new path.

[0039] Step S106, using a greedy algorithm, adjusting according to the communication cost and processing capacity between processors to optimize load balancing.

[0040] The communication costs between processors are sorted by a greedy algorithm to form a basic processor usage order. The correlation between the communication cost between processors and the data flow is determined, and the basic processor usage order is adjusted according to the new data flow requirements. According to the new processor usage order, the communication protocol is calculated and adjusted to adapt to the new data flow requirements. Based on the new communication protocol, the communication cost between processors is re-evaluated. The correlation between the computing density of the processor and the newly evaluated communication cost is determined, and the communication cost is reduced by adjustment. On the basis of reducing the communication cost, the processing capacity of the processors is sorted to obtain the optimized processor usage order. For example, there is a computing task that needs to be run on 4 processors, namely A, B, C and D. The existing communication costs are as follows: the communication cost between A and B is 2, the communication cost between A and C is 5, the communication cost between A and D is 3, the communication cost between B and C is 4, the communication cost between B and D is 6, and the communication cost between C and D is 1. Now the communication costs between processors can be sorted by a greedy algorithm to form a basic processor usage order. First, select the processor pair with the lowest communication cost, namely C and D, and add them to the processor usage order in order. Then, select the processor with the lowest communication cost with the selected processor, namely A and D, and add them to the processor usage order. Next, select the processor with the lowest communication cost with the selected processor, namely A and B, and add them to the processor usage order. Finally, select the processor with the lowest communication cost with the selected processor, namely A and C, and add them to the processor usage order. Based on the above processor usage order, the correlation between the communication cost between processors and the data traffic can be determined. If the data traffic of a processor is large, the processor with a lower communication cost with the processor may be preferred to reduce the overall communication cost. Next, if there is a new data traffic demand, the communication protocol can be calculated and adjusted according to the new processor usage order to adapt to the new data traffic demand. If the data traffic of a processor increases, it may be necessary to optimize the communication protocol between it and other processors to a protocol with higher bandwidth or lower latency. Based on the new communication protocol, the communication cost between processors can be re-evaluated. Under the new protocol, the communication cost between A and B is reduced to 1, the communication cost between A and C is reduced to 4, the communication cost between A and D is reduced to 2, the communication cost between B and C is reduced to 3, the communication cost between B and D is reduced to 5, and the communication cost between C and D is reduced to 1. Next, the correlation between the computing density of the processor and the newly evaluated communication cost can be determined. If a processor has a higher computing density, it may have a higher communication cost with other processors. Finally, on the basis of reducing the communication cost, the processing power of the processors can be sorted to obtain the optimized processor usage order. If A has a higher computing power, it can be placed at the front of the processor usage order.

[0041] Step S107, optimizing the data storage location according to the load balancing result to improve data locality.

[0042] Analyze the load balancing results to obtain hot data. Use the K-means algorithm to perform data locality analysis on hot data to determine whether there is room for optimization. Determine which hot data can be moved to the same or adjacent storage locations based on data dependency and data consistency. Use the greedy algorithm to calculate the weight ratio of storage cost and access speed. Quantitatively evaluate the I / O bandwidth and system response time. If the I / O bandwidth is lower than the preset bandwidth threshold or the system response time is higher than the preset response time threshold, recommend data migration based on the obtained weight ratio and storage location. After data migration, verify data availability and data integrity. For example, there is a load balancing system that contains 100 storage nodes, each of which stores 1,000 data blocks. Identify hot data by analyzing the load balancing results, and optimize system performance through data migration. First, the hot data can be determined by counting the access frequency of each data block. Some data blocks have a significantly higher access frequency than other data blocks, and these data blocks can be marked as hot data. Next, the K-means algorithm can be used to perform data locality analysis on the hot data to determine whether there is room for optimization. The hot data is divided into two clusters, where cluster 1 contains 60 data blocks and cluster 2 contains 40 data blocks. The K-means algorithm can be used to determine the center positions of clusters 1 and 2, thereby determining whether there is data locality. Based on data dependency and data consistency, it can be determined which hot data can be moved to the same or adjacent storage locations. It is found that there is a strong data dependency between the data blocks of cluster 1, and there is also a strong data dependency between the data blocks of cluster 2. Therefore, the data blocks of cluster 1 can be moved to the same storage location, and the data blocks of cluster 2 can be moved to adjacent storage locations. Next, the weight ratio of storage cost and access speed needs to be calculated to make a decision on data migration. The weight ratio of storage cost and access speed is 3:7, indicating that access speed is more important. Based on this ratio, the optimal data migration strategy can be selected by comprehensively considering storage cost and access speed. Then, the I / O bandwidth and system response time can be quantitatively evaluated. The preset I / O bandwidth threshold is 100MB / s, and the preset system response time threshold is 10ms. If the actual I / O bandwidth is lower than 100MB / s or the system response time is higher than 10ms, data migration can be performed based on the obtained weight ratio and storage location recommendation. After performing data migration, the data availability and integrity need to be verified. The appropriate data migration strategy is selected and the hotspot data is successfully moved to the new storage location. The data availability and integrity can be verified by checking whether the data read and write operations are normal and whether the data consistency is guaranteed.

[0043] Step S108, analyzing the results of load balancing and data locality optimization, and starting a predetermined fault tolerance mechanism.

[0044] Use the K-means algorithm to classify the load of the current system, use the Euclidean distance as the similarity measure, and obtain the classification results through iterative calculation. According to the classification results of the K-means algorithm, adjust the load balancing parameters. According to the adjusted load balancing parameters, perform task allocation and generate a task execution report. Use the greedy algorithm to analyze the task execution report and data locality, perform resource scheduling, and generate a performance report after resource scheduling. Analyze the performance report to determine whether the system stability is lower than the predetermined stability threshold and execute the predetermined fault tolerance mechanism. Analyze the performance report after the fault tolerance mechanism is executed. If there is still a problem, trigger the predetermined fault tolerance mechanism again. For example, to classify the load of a system and adjust the load balancing parameters according to the classification results. There are 4 nodes in the system, and the K-means algorithm is used to divide the load into 2 categories. Initially, node 1 and node 2 each bear 50% of the load, and node 3 and node 4 each bear 50% of the load. First, randomly select two loads as the initial class centroids, and select the loads of node 1 and node 4 as the initial class centroids. Then, the distance between each load and the class centroid is calculated according to the Euclidean distance, and the load is assigned to the class with the closest distance. Through iterative calculation, the classification results are finally obtained as follows: Class 1 is the load of node 1 and node 3, and Class 2 is the load of node 2 and node 4. According to the classification results, the load balancing parameters can be adjusted to adjust the load balancing ratio between node 1 and node 3 to 60% and 40%. Next, the tasks are assigned according to the adjusted load balancing parameters. There are 10 tasks to be executed. The greedy algorithm is used to assign the tasks to the appropriate nodes according to the task execution time and node load. According to the greedy algorithm, the task execution report and data locality are analyzed to obtain the statistical information of the execution time and node load of the tasks on each node. 3 tasks are executed on node 1 with an average execution time of 10 seconds, 2 tasks are executed on node 2 with an average execution time of 15 seconds, 4 tasks are executed on node 3 with an average execution time of 8 seconds, and 1 task is executed on node 4 with an execution time of 20 seconds. Based on the performance report of resource scheduling, the advantages and disadvantages of execution time and node load can be analyzed. If the system stability is lower than the predetermined stability threshold, the predetermined fault tolerance mechanism can be executed to add nodes or adjust the task allocation strategy. According to the performance report after the fault tolerance mechanism is executed, the execution time and node load are analyzed again to determine whether there is a problem. If there is still a problem, the predetermined fault tolerance mechanism can be triggered again to add more nodes or adjust the load balancing parameters.

[0045] Step S109, predicting the energy consumption of the processor, and using a PID controller to adjust the processor frequency to control the energy consumption.

[0046] The historical energy consumption data of the processor and the corresponding input features are obtained, and the input features are normalized, and the input features include load, temperature and voltage. The energy consumption of the processor is used as the target variable, and the input features are used as the input of the model. The LSTM network structure is used for training, and the data is divided into a training set and a validation set, and mean square error and Adam model optimization are used at the same time. The real-time input features are used as input, and the trained LSTM model is used to predict the energy consumption of the processor. The prediction result is compared with the actual energy consumption data to calculate the prediction error. The prediction error is used as a feedback signal and input into the PID controller. According to the size and change rate of the prediction error, a control signal, that is, the frequency adjustment amount, is calculated. The control signal is applied to the processor so that it adjusts the operating frequency accordingly to control energy consumption. New actual energy consumption data is continuously collected, and prediction and control are performed. For example, the historical energy consumption data of the processor and the corresponding input features are obtained, where load, temperature and voltage are used as input features. These features are normalized, and then trained using the LSTM network structure. The data is divided into a training set and a validation set, and the model is trained using mean square error and Adam optimizer. The training set contains 100 samples, each with a 10-time-step input feature sequence and the corresponding energy consumption. Use the trained LSTM model to predict the processor energy consumption. Now, there is a real-time input feature sequence, including the current load, temperature, and voltage values. Take these real-time features as input and use the trained LSTM model to predict the energy consumption. The predicted energy consumption is 100 watts, while the actual energy consumption is 90 watts. The prediction error can be calculated to be 10 watts. Then, the prediction error is input into the PID controller as a feedback signal. According to the size and rate of change of the prediction error, the PID controller can calculate a control signal, that is, the frequency adjustment amount. The PID controller calculates the frequency adjustment amount to be 5GHz. Finally, the frequency adjustment amount is applied to the processor so that it adjusts the operating frequency accordingly to control the energy consumption. The processor currently operates at 5GHz, and after applying the frequency adjustment amount, the operating frequency of the processor will be adjusted to 0GHz. By continuously collecting new actual energy consumption data, and performing prediction and control, the energy consumption of the processor can be continuously optimized.

[0047] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A parallel programming method for diverse computing task requirements, It is characterized in that The method comprises: By analyzing the specifications and capabilities of the multi-core or multi-processor in the system, the memory access cost of each processor is quantified; a list of diverse computing tasks to be executed is obtained, and the complexity of each task is quantified according to the time and resource amount required by the task; a preliminary mapping of tasks to processors is performed according to the memory access cost of the processor and the complexity of the task; the time constraints of the real-time task are analyzed, and the preliminary mapping scheme is modified according to the time constraints; the data transmission path between processors is optimized to reduce the estimated communication cost; a greedy algorithm is used to adjust according to the communication cost and processing power between processors to optimize load balancing; according to the results of load balancing, the data storage location is optimized to improve data locality; the results of load balancing and data locality optimization are analyzed, and a predetermined fault tolerance mechanism is started; the energy consumption of the processor is predicted, and the processor frequency is adjusted using a PID controller to control energy consumption.

2. The method according to claim 1, in, The memory access cost of each processor is quantified by analyzing the specifications and capabilities of the multi-core or multi-processor in the system, including: The processor specifications and multi-core or multi-processor configuration information are obtained through the hardware abstraction layer; it is determined whether the obtained multi-core or multi-processor configuration information meets the system specification requirements; if so, the operating status and computing power data of each processor are collected using CPU-Z; based on the collected operating status and computing power data, the data dependency between the processors is calculated and analyzed, and if the data dependency is higher than the preset data dependency threshold, resource scheduling is performed through the stochastic gradient descent algorithm; based on resource scheduling, the memory access cost of each processor is quantified through the Pearson correlation coefficient.

3. The method according to claim 1, in, The method of obtaining a list of various computing tasks to be performed and quantifying the complexity of each task according to the time and resources required by the task includes: Obtain a list of pending computing tasks from the task management system or database, and use query statements to filter out pending tasks; For each task to be performed, obtain the complexity assessment information of the task, including the time and resources required for the task; Quantify the complexity of each task based on the amount of time and resources it requires.

4. The method according to claim 1, in, The preliminary mapping of tasks to processors is performed based on the memory access cost of the processor and the complexity of the task, including: Calculate the mapping score for each task based on the processor's memory access cost and the complexity of the task; Traverse all processors and select the processor with the highest mapping score as the mapping processor for the current task; determine whether the selected processor has sufficient computing resources and memory capacity to execute the current task. If not, select the next processor with the highest score; check whether the current task has dependencies. If so, check whether the dependent task has been mapped to the processor; If the dependent task has not been mapped, wait until the dependent task is completed before mapping; update the load of the selected processor, and repeat the mapping of tasks to processors until all tasks are mapped to processors.

5. The method according to claim 1, in, The analyzing the time constraints of the real-time tasks and modifying the preliminary mapping scheme according to the time constraints include: Evaluate the processor's processing speed, data transfer speed, and memory capacity parameters for real-time tasks. Modify the mapping scheme to optimize the processor's processing speed and data transmission speed to the highest level; test the mapping scheme to evaluate whether the time it takes to complete the task unit meets the time constraints of the real-time task; if the time it takes to complete the task unit does not meet the time constraints of the real-time task, readjust the mapping scheme.

6. The method according to claim 1, in, The optimization of the data transmission path between processors to reduce the estimated communication cost includes: The data transmission path between processors is analyzed by Bellman-Ford algorithm, and the shortest path estimation value of each node is updated step by step until it converges to the optimal solution, and the estimated communication cost on the path is obtained; according to the estimated communication cost, it is determined whether optimization is needed; If optimization is required, the path is reconstructed based on the estimated communication cost to obtain the data volume under the new path; the transmission rate of the path is calculated based on the data volume under the new path; based on the calculated transmission rate, it is determined whether the path meets the delay limit; if the delay limit is met, the network flow check of the data integrity of the new path is performed; through the network flow check, it is determined whether the new path has redundancy; if so, the redundant path is deleted.

7. The method according to claim 1, in, The greedy algorithm is used to adjust according to the communication cost and processing capacity between processors to optimize load balancing, including: The communication costs between processors are sorted by a greedy algorithm to form a basic processor usage order; the correlation between the communication cost between processors and the data flow is determined, and the basic processor usage order is adjusted according to the new data flow demand; according to the new processor usage order, the communication protocol is calculated and adjusted to adapt to the new data flow demand; based on the new communication protocol, the communication cost between processors is re-evaluated; the correlation between the computing density of the processor and the newly evaluated communication cost is determined, and the communication cost is reduced through adjustment; on the basis of reducing the communication cost, the processing capacity of the processor is sorted to obtain the optimized processor usage order.

8. The method according to claim 1, in, The method of optimizing the data storage location according to the load balancing result to improve data locality includes: Analyze the load balancing results to obtain hotspot data; use the K-means algorithm to analyze the data locality of the hotspot data to determine whether there is room for optimization; determine which hotspot data can be moved to the same or adjacent storage locations based on data dependency and data consistency; use the greedy algorithm to calculate the weight ratio of storage cost and access speed; Quantitatively evaluate the I / O bandwidth and system response time. If the I / O bandwidth is lower than the preset bandwidth threshold or the system response time is higher than the preset response time threshold, data migration is recommended based on the obtained weight ratio and storage location. After data migration, verify data availability and data integrity.

9. The method according to claim 1, in, The analysis of the results of load balancing and data locality optimization and starting a predetermined fault tolerance mechanism include: Use K-means algorithm to classify the load of the current system, use Euclidean distance as the similarity measure, and obtain the classification result through iterative calculation; adjust the load balancing parameters according to the classification results of the K-means algorithm; perform task allocation and generate task execution report according to the adjusted load balancing parameters; use greedy algorithm to analyze task execution report and data locality, perform resource scheduling, and generate performance report after resource scheduling; analyze the performance report to determine whether the system stability is lower than the predetermined stability threshold, and execute the predetermined fault tolerance mechanism; analyze the performance report after the fault tolerance mechanism is executed, and if there is still a problem, trigger the predetermined fault tolerance mechanism again.

10. The method according to claim 1, in, The method of predicting the energy consumption of the processor and adjusting the processor frequency by using a PID controller to control the energy consumption includes: The historical energy consumption data and corresponding input features of the processor are obtained, and the input features are normalized, wherein the input features include load, temperature and voltage; the energy consumption of the processor is used as the target variable, and the input features are used as the input of the model; the LSTM network structure is used for training, and the data is divided into a training set and a validation set, and mean square error and Adam model optimization are used at the same time; the real-time input features are used as input, and the trained LSTM model is used to predict the energy consumption of the processor; the prediction result is compared with the actual energy consumption data to calculate the prediction error; the prediction error is used as a feedback signal and input into the PID controller; a control signal, i.e., the frequency adjustment amount, is calculated according to the size and change rate of the prediction error; the control signal is applied to the processor to adjust the operating frequency accordingly to control the energy consumption; new actual energy consumption data is continuously collected, and prediction and control are performed.

Citation Information

Cited By

  • Server device control method and device and medium

    CN119576589A

  • A control method, device and medium for a server device

    CN119576589B

  • Cloud computing resource management system

    CN120011087A

  • Parallel processor power consumption dynamic optimization method and system

    CN120029436A

  • A method and system for dynamically optimizing power consumption of parallel processors

    CN120029436B