Scientific calculation application performance prediction method based on multilayer perceptron

Abstract

The invention discloses a scientific calculation application performance prediction method based on a multilayer perceptron. The method comprises the following steps: adopting diversified data sources; labeling the collected data source; the marked data is preprocessed; identifying features having significant influence on performance prediction through a feature selection technology, and performing feature extraction and construction to reduce feature dimensions; a multi-layer perceptron (MLP) is used as a performance prediction model to utilize the ability of learning a complex nonlinear relationship; the MLP model is trained by using the preprocessed data, the stability and generalization ability of the model are evaluated by using a cross validation method, and the over-fitting risk is reduced; the prediction error and the accuracy index of the model are calculated by verifying the data set, and the prediction performance of the model is evaluated; for different HPC systems, a small amount of representative performance data is collected, and the basic model is finely adjusted by using the performance data, so that the model can learn performance characteristics of a new system.

Description

Technical Field

[0001] The present invention relates to the field of AI prediction of scientific computing performance technology, and in particular to a scientific computing application performance prediction method based on a multi-layer perceptron. Background Art

[0002] In the field of high-performance computing (HPC), as the complexity of scientific computing applications continues to increase, the demand for efficient use of computing resources and performance optimization is becoming increasingly urgent. In recent years, deep learning-based technologies have gained widespread attention and rapid development in the field of scientific computing due to their powerful data processing capabilities and model generalization capabilities. Deep learning has not only achieved remarkable results in traditional fields such as image recognition and natural language processing, but also demonstrated great potential in the emerging field of scientific computing, especially in processing large-scale data and high-dimensional nonlinear problems. Its advantages are particularly obvious.

[0003] Scientific computing applications often involve complex data processing and model solving processes. Traditional methods often face challenges such as low computational efficiency, difficulty in processing high-dimensional data and nonlinear relationships when dealing with such problems. In contrast, deep learning provides a new solution for scientific computing applications with its excellent representation learning ability and efficient parallel computing capabilities. In the HPC environment, scientific computing applications based on deep learning usually adopt a distributed machine learning architecture, which significantly improves computing efficiency and performance by parallelizing the training and reasoning processes.

[0004] However, in the distributed training process, performance modeling and prediction becomes a key issue. The performance model is not only crucial for task management and scheduling, but can also guide the scheduler to reasonably allocate tasks to the optimal computing nodes, thereby improving the overall utilization of the HPC system; at the same time, it can also provide developers with in-depth insights into application behavior, helping to optimize applications and explore expansion potential. In addition, the performance model is also an important basis for users to estimate computing resource requirements and reasonably plan computing time and resource requests.

[0005] Traditional performance modeling methods rely on mathematical formulas to describe the performance of parallel programs. This method requires researchers to have a deep understanding and rich experience in specific HPC applications, and to invest a lot of time in complex time complexity analysis. However, due to the breadth and diversity of HPC application fields, it is extremely difficult to build a universal analysis model, and the promotion and application of the model in different fields also face many challenges.

[0006] More importantly, existing performance prediction models are often limited to specific HPC system environments for training and testing, which leads to significant limitations in predicting the performance of the models on different HPC systems and makes it difficult to meet the cross-platform and cross-system performance prediction needs in actual applications.

[0007] In summary, it is particularly important to develop a performance prediction method for scientific computing applications that can overcome the above limitations and have stronger generalization ability and adaptability. Summary of the invention

[0008] In view of this, the present invention proposes a scientific computing application performance prediction method based on a multi-layer perceptron to solve the problems of poor universality of HPC deep learning performance models and inaccurate cross-system predictions in the prior art.

[0009] The specific technical solutions of the present invention are as follows:

[0010] A method for predicting performance of scientific computing applications based on a multi-layer perceptron, comprising:

[0011] Use diverse data sources; label the collected data sources;

[0012] Preprocess the labeled data; identify features that have a significant impact on performance prediction through feature selection technology, and perform feature extraction and construction to reduce feature dimensions;

[0013] Use multi-layer perceptron (MLP) as the performance prediction model to take advantage of its ability to learn complex nonlinear relationships;

[0014] The MLP model is trained using preprocessed data, and the cross-validation method is used to evaluate the stability and generalization ability of the model and reduce the risk of overfitting;

[0015] The prediction error and accuracy of the model are calculated through the validation data set to evaluate the prediction performance of the model.

[0016] For different HPC systems, a small amount of representative performance data is collected and used to fine-tune the basic model so that the model can learn the performance characteristics of the new system.

[0017] Furthermore, a variety of data sources are used including data collected directly from the HPC system, simulation data, historical records, and third-party data sources.

[0018] Furthermore, using diverse data sources also includes: using monitoring tools and log systems to capture performance data in real time.

[0019] Furthermore, the collected data is annotated, including the time to complete an epoch in distributed training.

[0020] Furthermore, the hyperparameters of the MLP model are optimized using a grid search or random search strategy to find the best model configuration.

[0021] Furthermore, the hyperparameters of the MLP model include: learning rate, batch size, number of layers, and number of neurons.

[0022] Furthermore, the Adam optimization algorithm is applied to update the MLP model parameters to accelerate the training process and improve the convergence effect.

[0023] Furthermore, the method also includes: retraining the fine-tuned model on the new system data, and using a test data set to evaluate its prediction performance on the new system; comparing the performance indicators before and after fine-tuning to verify the fine-tuning effect.

[0024] Furthermore, feature extraction and construction include: building high-level features of system load, task parallelism, and memory access patterns based on basic system configuration and performance indicators to more comprehensively describe system status and application behavior.

[0025] Furthermore, feature extraction and construction include: using feature importance evaluation methods such as random forests or gradient boosting trees to determine key features, and increasing data set diversity and improving model generalization capabilities through data enhancement techniques such as sound addition and feature transformation.

[0026] The beneficial effects of the present invention are:

[0027] (1) Significantly improve prediction efficiency and accuracy: The multi-layer perceptron (MLP) deep learning model is used to predict performance based on the time it takes to complete an epoch in distributed training. Compared with traditional mathematical modeling methods, this method greatly reduces the reliance on researchers’ domain knowledge and significantly improves prediction efficiency and accuracy.

[0028] (2) Wide applicability: This method is highly versatile and is not limited to specific HPC application fields. It can be applied to a variety of different scientific computing tasks, which enhances the adaptability and application scope of the model.

[0029] (3) Flexibility and scalability: Through the fine-tuning mechanism, the model can quickly adapt to and optimize performance predictions on different HPC systems, making the model not only highly versatile but also highly flexible and scalable, making it easy to promote to new application scenarios.

[0030] (4) Resource conservation and performance optimization: The fine-tuning strategy reduces the amount of data collection on the new system, saving resources and time costs. At the same time, accurate performance prediction provides strong support for program optimization, helps improve parallel computing efficiency, and reduces the waste of computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0032] Figure 1 The figure is a flow chart of the method for predicting the performance of scientific computing applications based on a multi-layer perceptron according to the present invention. DETAILED DESCRIPTION

[0033] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0034] The present invention proposes a method for predicting the performance of scientific computing applications based on a multi-layer perceptron, that is, using a deep learning model called a multi-layer perceptron (MLP) to predict the time it takes for a scientific application based on deep learning to complete an epoch in distributed training. The core of this innovation is that the MLP model can automatically learn and capture the complex nonlinear relationship between input features (such as system configuration, data set size, network bandwidth, etc.) and performance indicators (i.e., training time). Through a carefully designed feature engineering and model training process, an MLP model that can predict the performance of HPC applications under different configurations and conditions was successfully trained. The advantage of this method is that it avoids the tedious process of researchers needing to deeply understand the time complexity of specific applications, and only requires collecting and organizing relevant data to perform efficient performance predictions.

[0035] However, we also realize a problem: the training results of the performance prediction model on a specific HPC system may not be directly applicable to other systems. To solve this problem, this application proposes to fine-tune the model on a specific system.

[0036] Fine-Tuning, as an effective transfer learning method, has been successfully applied to the performance prediction model of this application. First, this application trains a basic model on a general HPC system, which already has certain performance prediction capabilities. Then, a small amount of performance data is collected on different HPC systems and the basic model is fine-tuned. Through the fine-tuning process, the model can learn the performance characteristics unique to the new system and thus improve the prediction accuracy on different systems. This method not only improves the flexibility and versatility of the model, but also significantly saves the resources and time costs of data collection and model training. More importantly, accurate performance predictions provide researchers with valuable reference information, which helps to better optimize programs, improve parallel computing efficiency and reduce the waste of computing resources.

[0037] In brief, the application method includes:

[0038] First, ensure the comprehensiveness and diversity of data collection, covering not only a wide range of system configurations (such as CPU models, memory size, storage type, etc.), but also multiple application scenarios (such as scientific computing, big data analysis, AI model training, etc.) to ensure that the model can learn the performance characteristics in various environments.

[0039] In the feature engineering stage, key features were carefully selected and constructed, and the potential impact of high-dimensional data on model training speed and performance was effectively avoided through dimensionality reduction and feature conversion techniques. Statistical methods and machine learning algorithms were used to identify the most valuable features for performance prediction, and redundant and irrelevant features were eliminated, which improved the efficiency and accuracy of the model.

[0040] The model training uses a variety of regularization techniques (such as L1 / L2 regularization, Dropout, etc.) to limit the model complexity and prevent overfitting. At the same time, combined with the early stopping strategy, the training is stopped in time when the performance of the validation set begins to decline, ensuring that the model achieves a good balance between the training set and the validation set.

[0041] During the model fine-tuning phase, we customized a suitable fine-tuning strategy based on the characteristics and requirements of the new system, and carefully selected a representative fine-tuning dataset. By retraining on the new system data, we further optimized the model parameters to better adapt to the new environment and improve the model's generalization ability on different systems.

[0042] The specific implementation steps of this application method are as follows:

[0043] Step 1: Data collection and feature engineering.

[0044] The data collection phase is critical, requiring comprehensive and detailed collection of various performance data related to the HPC system and its running applications. This includes, but is not limited to, detailed information on the system configuration, such as the CPU model, number of cores, main frequency, memory size, type and speed, and storage device type (SSD, HDD, etc.), capacity and I / O performance. At the same time, the actual use of network bandwidth must also be recorded to reflect the efficiency and latency of data transmission. The size of the data set and the complexity of the task are also factors that must be considered, as they directly affect the execution time and resource consumption of the computing task. In particular, in order to evaluate the training efficiency, the collected performance indicators should explicitly include the time to complete an epoch in distributed training, which is an important indicator for measuring training speed and stability.

[0045] Next, the feature engineering stage is the process of in-depth processing and optimization of the collected data. First, data cleaning is performed to remove noise, outliers, and incomplete records to ensure the accuracy and reliability of the data. Subsequently, data normalization and standardization are performed, and feature values ​​of different dimensions and distribution ranges are converted to the same scale through mathematical transformation to facilitate subsequent data analysis and model training. In the feature selection and extraction stage, statistical methods and domain knowledge are used to comprehensively evaluate the initial feature set, screen out important features that are closely related to performance prediction, and construct new features to capture potential patterns and relationships in the data. Finally, feature selection methods such as filtering, wrapping, or embedding are used to further reduce feature dimensions, reduce model complexity, and improve training efficiency and prediction accuracy.

[0046] Step 2: Model building and optimization.

[0047] In the model building stage, this application selected the multi-layer perceptron (MLP) as the main model for performance prediction. With its multi-layer structure, nonlinear activation function and powerful feature learning ability, MLP has shown significant advantages in handling complex prediction tasks. In order to ensure that the model can achieve optimal performance, this application further uses efficient methods such as grid search or random search to optimize the model's hyperparameters.

[0048] Specifically, this application has carefully tuned key hyperparameters such as learning rate, batch size, number of hidden layers, and number of neurons in each layer. The learning rate is an important parameter that controls the learning speed of the model. If it is too large, the model may not converge stably, and if it is too small, the training process will become slow. The batch size determines the number of samples used for each gradient update, which affects the generalization ability and training stability of the model. The number of hidden layers and the number of neurons in each layer directly determine the complexity and learning ability of the model.

[0049] Through grid search or random search, this application conducts extensive exploration in the hyperparameter space to find the best hyperparameter combination that can maximize the performance of the model on the validation set. Although this process is computationally expensive, it ensures that the model has a high starting point before training.

[0050] Step 3: Model training and performance evaluation.

[0051] Entering the model training stage, this application uses a preprocessed training data set to train the MLP model. In order to improve the robustness of the model and reduce the risk of overfitting, this application adopts a cross-validation method. By dividing the training data set into multiple subsets, and using one part as a training set and the other part as a validation set to train the model in turn, the evaluation results of multiple models can be obtained, and the model with the best performance can be selected as the final model.

[0052] During the training process, this application uses the Adam optimization algorithm to update the parameters of the model. The Adam algorithm combines the advantages of the momentum method and the RMSprop algorithm, and can adaptively adjust the learning rate of each parameter, thereby accelerating the convergence of the model and improving performance. Through the iterative execution of the steps of forward propagation to calculate the predicted value, back propagation to calculate the gradient, and parameter update, the weights and biases of the model are gradually adjusted to minimize the loss function.

[0053] After completing the model training, this application uses an independent validation data set to conduct a comprehensive performance evaluation of the model. The method of this application calculates key indicators such as the model's prediction error (such as mean square error MSE) and accuracy to quantify the model's predictive ability on unseen data. At the same time, learning curves and validation curves are plotted to observe the performance change trend of the model during the training process. The learning curve shows the changes in the loss function value on the training set and the validation set, while the validation curve may reflect the changes in accuracy or other performance indicators. Through these curves, you can intuitively check whether the model has overfitting or underfitting, and further adjust and optimize the model accordingly.

[0054] Step 4: Model fine-tuning and testing.

[0055] In order to improve the generalization ability and practical application effect of the model on different high-performance computing (HPC) systems, this application adopts the strategy of model fine-tuning. First, small-scale performance data samples are collected on multiple different HPC systems. These data cover the unique hardware characteristics, resource configuration and potential performance bottlenecks of each system, providing a solid foundation for subsequent model fine-tuning.

[0056] During the fine-tuning phase, the original model is retrained using the newly collected data. During this process, the model architecture remains basically unchanged, but the model parameters, including weights and biases, are optimized to better adapt to the data distribution and characteristics in the new system environment. In order to speed up the fine-tuning process and prevent overfitting, this application usually uses a smaller learning rate and adjusts the batch size and training rounds when necessary.

[0057] After fine-tuning, this application uses an independent test dataset to conduct a comprehensive evaluation of the model. This test dataset is completely isolated from the data used for fine-tuning to ensure the objectivity and accuracy of the evaluation results. By calculating the prediction error (such as mean square error MSE) and key performance indicators such as accuracy of the fine-tuned model on the new system, we can intuitively understand the performance of the model in the new environment.

[0058] In order to further verify the effect of fine-tuning, this application compares the model performance indicators before and after fine-tuning. This comparison not only covers the improvement of prediction accuracy, but also may include improvements in other aspects such as model training time and resource consumption. Through comparative analysis, we can clearly see the positive role that fine-tuning plays in helping the model adapt to the new system environment, thereby enhancing confidence in the performance of the model in practical applications.

[0059] The innovations of this application are mainly reflected in the following aspects:

[0060] 1. Application of Multi-layer Perceptron (MLP): Abandoning the complexity and limitations of traditional mathematical modeling methods, a multi-layer perceptron, a deep learning model, is used for performance prediction. MLP can automatically learn and extract high-level features from data without complex physical assumptions and prior knowledge, significantly improving the accuracy of predictions.

[0061] 2. Model fine-tuning technology: The innovative use of model fine-tuning technology improves the adaptability of the model on different HPC systems by retraining on new system data. This technology not only enhances the versatility and flexibility of the model, but also enables the model to adapt to new environments more quickly, reducing deployment costs and time.

[0062] 3. Combination of deep learning and big data: It makes full use of the advantages of deep learning in feature extraction and learning ability, and combines a large amount of performance data for training. This data-driven approach enables the model to more accurately capture the changing laws of system performance, significantly improving the accuracy and efficiency of performance prediction.

[0063] 4. Systematic implementation plan: A systematic implementation plan is provided, which covers various links such as data collection, model building, training, fine-tuning and testing. This systematic process not only ensures the integrity and operability of the plan, but also improves the standardization and repeatability of project implementation. Through the close connection and mutual support of various links, the goal of high-performance computing system performance prediction is successfully achieved.

[0064] The beneficial effects of the present invention are:

[0065] (1) Broad applicability: The method of the present invention can be applied to a variety of high-performance computing (HPC) applications, breaking the limitations of traditional methods that are restricted to specific fields and demonstrating strong cross-domain applicability.

[0066] (2) Significantly improve efficiency: Using the multi-layer perceptron (MLP) model in deep learning to perform performance prediction greatly reduces the need for researchers to have an in-depth understanding of the time complexity of specific applications, significantly reduces labor costs, and improves the automation and intelligence level of the prediction process.

[0067] (3) High accuracy: By training on large amounts of data, deep learning models can capture and learn complex performance relationships, significantly improving the accuracy and reliability of performance predictions compared to traditional mathematical modeling methods.

[0068] (4) Flexible scalability: This method can be easily extended to new application areas. It only needs to collect relevant data in the new field for training, and can quickly adapt to new scenarios without redesigning complex analysis models, demonstrating strong adaptability and scalability.

[0069] (5) System fine-tuning and optimization: The introduction of fine-tuning technology enables the model to be further optimized on a specific HPC system. By retraining on a small amount of data on a new system, the model can learn the unique performance characteristics of the new system, thereby improving the prediction accuracy on different systems and enhancing the flexibility and practicality of the model.

[0070] (6) Resource saving: The fine-tuning method only requires collecting a small amount of data on the new system for retraining, which effectively reduces the resources and time costs required for data collection and model training, and achieves efficient use of resources.

[0071] (7) Performance optimization support: Accurate performance prediction provides researchers with a strong basis for decision-making, helps to better optimize programs, improve the efficiency of parallel computing, reduce the waste of computing resources, and promote the sustainable development of the field of scientific computing.

[0072] (8) Systematic implementation plan: The present invention provides a set of systematic implementation plans, from data collection, feature engineering, model building, training, fine-tuning to testing, each link is closely linked to ensure the integrity and operability of the plan, and provide detailed guidance for practical applications.

[0073] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for predicting the performance of scientific computing applications based on a multi-layer perceptron, characterized in that: include: Use diverse data sources; label the collected data sources; Preprocess the labeled data; Identify features that have a significant impact on performance prediction through feature selection technology, and perform feature extraction and construction to reduce feature dimensions; Use multi-layer perceptron (MLP) as the performance prediction model to take advantage of its ability to learn complex nonlinear relationships; The MLP model is trained using preprocessed data, and the cross-validation method is used to evaluate the stability and generalization ability of the model and reduce the risk of overfitting; The prediction error and accuracy of the model are calculated through the validation data set to evaluate the prediction performance of the model. For different HPC systems, a small amount of representative performance data is collected and used to fine-tune the basic model so that the model can learn the performance characteristics of the new system.

2. The method for predicting the performance of scientific computing applications based on a multi-layer perceptron as claimed in claim 1, characterized in that: The diverse data sources include: data collected directly from the HPC system, simulation data, historical records, and third-party data sources.

3. The method for predicting the performance of scientific computing applications based on a multi-layer perceptron as claimed in claim 1, characterized in that: The use of diverse data sources also includes: using monitoring tools and log systems to capture performance data in real time.

4. The method for predicting the performance of scientific computing applications based on a multi-layer perceptron as claimed in claim 1, characterized in that: The collected data is labeled, including: the time to complete an epoch in distributed training.

5. The method for predicting the performance of scientific computing applications based on a multi-layer perceptron as claimed in claim 1, characterized in that: The hyperparameters of the MLP model are optimized using a grid search or random search strategy to find the best model configuration.

6. The method for predicting the performance of scientific computing applications based on a multi-layer perceptron as claimed in claim 5, characterized in that: The hyperparameters of the MLP model include: learning rate, batch size, number of layers, and number of neurons.

7. The method for predicting the performance of scientific computing applications based on a multi-layer perceptron as claimed in claim 1, characterized in that: The Adam optimization algorithm is applied to update the MLP model parameters to accelerate the training process and improve the convergence effect.

8. The method for predicting the performance of scientific computing applications based on a multi-layer perceptron as claimed in claim 1, characterized in that: The method also includes: retraining the fine-tuned model on new system data and evaluating its prediction performance on the new system using a test data set; and comparing performance indicators before and after fine-tuning to verify the fine-tuning effect.

9. The method for predicting the performance of scientific computing applications based on a multi-layer perceptron as claimed in claim 1, characterized in that: The feature extraction and construction includes: building high-level features of system load, task parallelism, and memory access patterns based on basic system configuration and performance indicators to more comprehensively describe system status and application behavior.

10. The method for predicting the performance of scientific computing applications based on a multi-layer perceptron according to claim 1, characterized in that: The feature extraction and construction includes: using a feature importance evaluation method of a random forest or gradient boosting tree to determine key features, and increasing the diversity of the data set and improving the generalization ability of the model through data enhancement techniques such as sound addition and feature transformation.

Images