Efficiency tuning system, method and device for intelligent computing cluster

Abstract

The invention provides an efficiency tuning system, method and device oriented to an intelligent computing cluster, and belongs to the technical field of intelligent computing clusters, and the system comprises a data processing module which is used for obtaining task operation index data of an intelligent computing task in the intelligent computing cluster; the efficiency strategy configuration module is used for configuring an efficiency judgment strategy for the intelligent calculation task based on the task operation index data; the efficiency monitoring module is used for carrying out efficiency detection on the running task according to the efficiency judgment strategy and generating alarm information under the condition that the running task does not meet the efficiency judgment strategy; and the efficiency adjusting and optimizing module is used for receiving the alarm information and generating an efficiency adjusting and optimizing strategy according to the efficiency judgment strategy. According to the method, the task operation index data of the intelligent calculation tasks are analyzed, and the alarm judgment and tuning strategy is combined, so that the mismatch phenomenon of the intelligent calculation tasks and the calculation resources can be alarmed, a more reasonable resource use scheme is provided for the intelligent calculation tasks, and the overall use efficiency of the cluster is improved while the task execution efficiency is ensured.

Description

Technical Field

[0001] The present invention relates to the field of intelligent computing cluster technology, and in particular to a performance tuning system, method and device for an intelligent computing cluster. Background Art

[0002] With the rapid development of big data and artificial intelligence technologies, intelligent computing clusters have become key infrastructure for processing massive data and complex intelligent tasks, providing support for the implementation of various intelligent models and algorithms. However, due to the diversity of tasks and different resource requirements, problems such as unbalanced and unreasonable resource utilization and frequent occurrence of inefficient tasks in the cluster often occur. The mismatch between tasks and required resources has led to a huge waste of computing resources and reduced the overall efficiency of the cluster.

[0003] In intelligent computing clusters, different tasks have significantly different resource requirements. Some tasks may require a large amount of computing resources (such as CPU and GPU), while other tasks may require higher memory or storage resources. If resource allocation is unreasonable, some node resources will be overloaded, while other node resources will be idle, and the overall resource utilization rate will be low. The resource allocation method currently adopted by intelligent computing clusters is manually specified by users before use, and it is a fixed combination. Often, the CPU, GPU and memory cannot be combined more flexibly. More importantly, users have no professional reserves of the resources required for the task before executing the task, which causes a large number of task mismatches. After the mismatch, the cluster no longer intervenes, resulting in low cluster efficiency.

[0004] On the other hand, due to reasons such as irregular user code writing, unreasonable parameter settings or data set problems, the execution efficiency of intelligent computing tasks is low and still occupies a large amount of computing resources. For example, there are training tasks in the cluster whose losses are no longer converging. The invalid calculation of such tasks leads to a waste of resources. However, there are currently few effective methods and means for clusters to identify and discover such tasks.

[0005] At present, most monitoring and operation and maintenance systems can monitor various resource indicators in the cluster and set relevant thresholds to monitor abnormal resource usage. However, these tools are designed for stable and safe operation of clusters in the operation and maintenance field. Most of them lack intelligent analysis and tuning functions, and cannot effectively identify inefficient tasks and provide targeted optimization suggestions. At the same time, such systems are often unable to fully analyze unstructured log data in various formats, ignoring the huge amount of information contained in massive log information, making it difficult to locate and solve performance bottlenecks.

[0006] Therefore, it is particularly necessary and urgent to develop a smart computing cluster data processing method and system that can monitor, analyze and tune in real time. This method should be able to comprehensively monitor the resource usage of tasks, automatically identify inefficient tasks, automatically warn of task mismatches, and provide optimization suggestions, thereby improving the overall efficiency of the cluster, avoiding resource waste, and ensuring stable and efficient operation of the cluster. Summary of the invention

[0007] The present invention provides a performance tuning system, method and device for intelligent computing clusters, which are used to assist in solving the defects of high task failure rate and resource waste of intelligent computing cluster platforms in the prior art, and to improve the stability and reliability of the system.

[0008] In a first aspect, the present invention provides a performance tuning system, method and device for an intelligent computing cluster, including:

[0009] The data processing module is used to obtain the task running indicator data of the intelligent computing tasks in the intelligent computing cluster;

[0010] An efficiency strategy configuration module, used to configure an efficiency determination strategy for the intelligent computing task based on the task operation indicator data, and the efficiency determination strategy is output after matching through a strategy configuration library;

[0011] A performance monitoring module, configured to perform performance detection on the running task according to the performance determination strategy, and generate an alarm message when the running task does not meet the performance determination strategy;

[0012] The performance tuning module is used to receive the alarm information and generate a performance tuning strategy according to the performance determination strategy.

[0013] According to the performance tuning system for intelligent computing clusters, the data processing module includes:

[0014] The structured data aggregation unit is used to obtain structured task details and resource utilization information in intelligent computing tasks;

[0015] Unstructured data aggregation unit, used to obtain unstructured log information in intelligent computing tasks;

[0016] A structured data processing unit, used for preprocessing the structured task details information and resource utilization information to obtain structured task operation indicator data;

[0017] The unstructured data processing unit is used to extract information from the unstructured log information to obtain unstructured task operation indicator data.

[0018] According to the performance tuning system for intelligent computing clusters, the preprocessing includes data cleaning, standardization, interpolation of missing values, removal of stains and abnormal data, and conversion of formats and measurement units.

[0019] According to the performance tuning system for intelligent computing clusters, the performance strategy configuration module includes:

[0020] A first performance strategy configuration unit is used to obtain task operation indicator data, identify the mismatch pattern between tasks and resources, and configure the performance determination strategy for the intelligent computing task in combination with the rule logic in the expert knowledge base;

[0021] A second performance strategy configuration unit, used to access the AI ​​to configure a performance determination strategy for the intelligent computing task;

[0022] A cross-analysis unit is used to cross-judge multiple resources of task operation indicators and optimize the intelligent computing task configuration efficiency judgment strategy for the combination of multiple resources;

[0023] The performance strategy configuration library is used to store the performance determination strategy and obtain the performance determination strategy corresponding to the real-time task running indicator data.

[0024] According to the performance tuning system for intelligent computing clusters, the performance monitoring module includes:

[0025] The performance monitoring unit is used to perform performance detection on the running task according to the performance determination strategy, and generate alarm information when the running task does not meet the performance determination strategy.

[0026] According to the performance tuning system for intelligent computing clusters, the performance tuning module includes an AI model for receiving the alarm information and generating a performance tuning strategy according to the performance determination strategy.

[0027] According to the performance tuning system for intelligent computing clusters, the performance tuning module further includes:

[0028] The cross-analysis performance tuning unit is used to cross-judge multiple resources of task operation indicators, and generate performance tuning strategies for multiple resource intelligent computing tasks based on the performance judgment strategy based on a single resource intelligent computing task.

[0029] According to the performance tuning system for intelligent computing clusters, the performance tuning module also includes:

[0030] The historical status analysis unit is used to obtain the historical task operation indicator data of the preset time period of the intelligent computing task, and obtain the corresponding performance judgment strategy to generate a performance tuning strategy.

[0031] In a second aspect, a performance tuning method for an intelligent computing cluster is also provided, the method being implemented based on any one of the above systems, and comprising:

[0032] Obtain task running indicator data of intelligent computing tasks in the intelligent computing cluster;

[0033] Based on the task running index data, the efficiency determination strategy is configured for the intelligent computing task, and the efficiency determination strategy is output after matching through the strategy configuration library;

[0034] Performing performance detection on the running task according to the performance determination strategy, and generating alarm information when the running task does not meet the performance determination strategy;

[0035] The alarm information is received, and a performance tuning strategy is generated according to the performance determination strategy.

[0036] In a third aspect, a computer device is also provided, including a memory, a processor, and a computer program stored in the memory, and when the processor executes, the steps of the above-mentioned performance tuning method for intelligent computing clusters are implemented.

[0037] The performance tuning system for intelligent computing clusters provided by the present invention has the following technical advantages compared with the prior art:

[0038] The present invention uses a data processing module to obtain task operation index data of intelligent computing tasks in an intelligent computing cluster; an efficiency strategy configuration module is used to configure an efficiency judgment strategy for the intelligent computing task based on the task operation index data; an efficiency monitoring module is used to perform efficiency detection on the running task according to the efficiency judgment strategy, and generate an alarm message when the running task does not meet the efficiency judgment strategy; an efficiency tuning module is used to receive the alarm message and generate an efficiency tuning strategy according to the efficiency judgment strategy. The present invention analyzes and alarms process indicators such as task details information, resource utilization and log information of intelligent computing tasks, and the system can provide real-time monitoring and early warning of the intelligent computing cluster. These rules will help the system automatically identify potential problems in similar situations and issue warnings in time, thereby improving the stability and reliability of the system. By comprehensively analyzing the alarm information and then providing an efficiency tuning strategy, the efficiency of optimizing the execution of intelligent computing tasks can be achieved. Therefore, the present invention can identify potential root causes of problems and provide targeted optimization suggestions to help system administrators and related personnel better deal with and solve problems. This intelligent analysis and optimization strategy can effectively improve the operating efficiency and stability of the system, and provide effective guidance and support for the system optimization of the intelligent computing cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 It is a structural schematic diagram of the performance tuning system for intelligent computing clusters provided by the present invention;

[0041] Figure 2 It is a flow chart of the performance tuning method for intelligent computing clusters provided by the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] It should be noted that in the description of the embodiments of the present invention, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "include one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements. The orientation or position relationship indicated by the terms "upper", "lower" and the like is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the system or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. Unless otherwise clearly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0044] The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0045] Combine the following Figure 1 The present invention describes a performance tuning system for intelligent computing clusters provided in an embodiment of the present invention.

[0046] Figure 1 is a schematic diagram of the structure of the performance tuning system for intelligent computing clusters provided by the present invention, such as Figure 1 As shown, the system includes: a data processing module, an efficiency strategy configuration module, an efficiency monitoring module and an efficiency tuning module.

[0047] The data processing module is used to obtain the task running indicator data of the intelligent computing tasks in the intelligent computing cluster;

[0048] An efficiency strategy configuration module, used to configure an efficiency determination strategy for the intelligent computing task based on the task operation indicator data, and the efficiency determination strategy is output after matching through a strategy configuration library;

[0049] A performance monitoring module, configured to perform performance detection on the running task according to the performance determination strategy, and generate an alarm message when the running task does not meet the performance determination strategy;

[0050] The performance tuning module is used to receive the alarm information and generate a performance tuning strategy according to the performance determination strategy.

[0051] In this embodiment, the data processing module obtains task running index data of the intelligent computing tasks running in the intelligent computing cluster. The task running index data includes task details information, resource utilization and log information.

[0052] The task details include task id, user id, start and end time, status, cluster, resource type, image name, startup file name and other task details parameters. Resource utilization includes the usage of CPU, MLU, DCU, GCU, NPU, memory, storage and other indicators, such as CPU occupancy, memory occupancy, GPU occupancy, NPU occupancy, DCU occupancy, GCU occupancy, GPU memory occupancy, NPU memory utilization, GCU memory utilization, DCU memory utilization, etc. Log information is generated by user code, and is a task log with non-fixed format and content.

[0053] In one embodiment, the data processing module needs to access wide-area data, and uses an interface adapter tool to adapt the HTTP call interface with different rules for each cluster to obtain the task operation indicator data of the intelligent computing tasks in the intelligent computing resources managed by the intelligent computing cluster, including GET and POST methods. The interface adapter tool shields the heterogeneity of the data interfaces of each cluster. Then, a language conversion tool is used to convert the return values ​​of the different format fields of each computing center into a unified JSON standard, and by setting a scheduled task, task-related data is periodically obtained through the HTTP interface. Then, a unified access tool is used to complete the standard input of the above json interface to other system components.

[0054] For the technical implementation of wide-area data access, a data access method based on the peer-to-peer collaboration mechanism (PCM) can be used. This method does not change the data interface of any original cluster, and builds a proxy component on the original system interface. This component is responsible for completing the access of heterogeneous data of different clusters and different interfaces, shielding heterogeneity downward and providing unified access upward.

[0055] Based on the content of the above embodiment, the data processing module in the performance tuning system for intelligent computing clusters provided by the present invention includes:

[0056] The structured data aggregation unit is used to obtain structured task details and resource utilization information in intelligent computing tasks;

[0057] Unstructured data aggregation unit, used to obtain unstructured log information in intelligent computing tasks;

[0058] A structured data processing unit, used for preprocessing the structured task details information and resource utilization information to obtain structured task operation indicator data;

[0059] The unstructured data processing unit is used to extract information from the unstructured log information to obtain unstructured task operation indicator data.

[0060] Preprocessing includes but is not limited to data cleaning, standardization, interpolation of missing values, removal of tainted and abnormal data, and conversion of formats and measurement units.

[0061] The unstructured data processing unit is used to extract information from the unstructured log information to obtain unstructured task operation index data. The unit can use two methods to extract task operation index data of user task execution efficiency from unstructured log data. One is to obtain various types of loss and accuracy curves in the log through keyword matching through expert experience. The second is to extract the indicators for highly customized indicators that cannot be matched by keywords by calling the general large language model through the API interface of the standard openai specification. Among them, the use of system prompt words can be exemplified as "You are a log analysis robot. You will analyze the data of user logs in the intelligent computing cluster and extract indicators according to requirements. Please output all possible loss values ​​according to the following log: [Log file], please note that the extraction is performed according to the number of epochs, and the standard json format is returned. Please note that only the standard format is returned."

[0062] Based on the content of the above embodiment, the data processing module in the performance tuning system for intelligent computing clusters provided by the present invention further includes:

[0063] The data formatting storage unit is used to store the cleaned and integrated data in the Elasticsearch library, using its powerful search and analysis capabilities to provide support for subsequent data retrieval and in-depth analysis.

[0064] In this embodiment, the performance strategy configuration module is used to configure the performance determination strategy for the intelligent computing task based on the task operation indicator data, and the performance determination strategy is output after matching through the strategy configuration library.

[0065] Among them, the formulation of performance judgment strategies in the strategy configuration library can be divided into two categories as a whole. One is based on expert experience. This type of strategy is generated based on expert experience by accumulating the mismatch between tasks and resources that appear in massive historical tasks. The other is based on the generation of performance judgment strategies based on intelligent language models.

[0066] In one embodiment, the performance strategy configuration module includes: a first performance strategy configuration unit, which is used to obtain task operation indicator data, identify the mismatch pattern between tasks and resources, and configure the performance judgment strategy for the intelligent computing task in combination with the rule logic in the expert knowledge base.

[0067] A second performance strategy configuration unit, used to access the AI ​​to configure a performance determination strategy for the intelligent computing task;

[0068] The cross-analysis unit is used to cross-judge multiple resources of task operation indicators and adopt the efficiency judgment strategy for the combination optimization of multiple resources.

[0069] The performance strategy configuration library is used to store the performance determination strategy and obtain the performance determination strategy corresponding to the real-time task running indicator data.

[0070] Specifically, the first performance strategy configuration unit is responsible for obtaining the operation index data of the intelligent computing task, which includes but is not limited to key performance indicators such as task execution time, resource utilization, and task queue length. The configuration unit further analyzes these index data to identify whether there is a mismatch or mismatch pattern between the intelligent computing task and the computing resources. The mismatch pattern may include but is not limited to resource surplus, resource shortage, or uneven resource allocation.

[0071] In order to formulate the performance determination strategy, the first performance strategy configuration unit also combines the rule logic in the expert knowledge base. The expert knowledge base contains the experience and knowledge of domain experts on the performance optimization of intelligent computing tasks. These experiences are encoded into executable rule logic to derive the performance determination strategy for specific mismatch patterns.

[0072] The rule logic in the expert knowledge base can be divided into GPU performance rules, CPU performance rules, NPU performance rules, MLU performance rules, GCU performance rules, memory performance rules, DCU performance rules, storage performance rules, task validity rules and multi-resource cross-performance rules.

[0073] Specifically, use regular expressions to formulate the efficiency utilization of single resources and multiple resources. For example, the regular expression "CPU_util>1&CPU_usage<0.2" indicates that a multi-core CPU is applied for a single task, but the utilization is less than 20% during the entire task cycle. According to expert experience, the number of CPU applications by this user is obviously excessive, and the efficiency utilization is not high. It is recommended to replace the CPU with a lower number of cores. The regular expression "GPU_name='A100'&GPU_usage<0.2" indicates that the user has applied for a high-performance GPU, but the GPU utilization is less than 20% throughout the process. According to expert experience, the user's GPU efficiency utilization is not high, and a GPU with lower computing power should be replaced. Simple rules can be used to perform efficiency judgments through regular expression matching.

[0074] Through the AND or OR operation of simple rules, more complex performance calculation rule formulas can be extracted. For example, the rule expression "time>90%&CPU_usage<20%" means that for the entire computing task, excluding 10% of the maximum and minimum values, the CPU utilization is lower than 20% throughout the entire process. This rule can filter out the low CPU utilization in most task cycles except for the peak utilization of 10% of the task time in the task cycle. It is used to determine the imbalance of task utilization throughout the cycle and whether to apply for excessive resources for this extremely rare peak utilization. At this time, the rule expression needs to implement the rule logic through a specific program.

[0075] The cross-analysis unit is used to make cross-judgments on indicators that use multiple resources. For example, tasks with GPU utilization rates higher than 60%, 70%, 80%, and 90% are further screened, and the number of tasks with GPU memory utilization rates lower than 20%, 30%, 40%, and 50% is counted. This type of rule makes judgments on the combined optimization of multiple resources, such as different video memory versions of computing resources with the same acceleration capabilities.

[0076] Commonly used regular expression examples and recommended strategies are shown in the following table:

[0077]

[0078]

[0079] The second performance strategy configuration unit is connected to the artificial intelligence (AI) model to assist in the configuration of the performance determination strategy. The configuration unit uses the analysis capability of the AI ​​model to conduct deep learning and prediction of the performance of intelligent computing tasks to generate a performance determination strategy. The AI ​​model is able to process large amounts of data, learn complex patterns, and adaptively adjust the performance determination strategy to cope with dynamically changing task loads and resource states.

[0080] Through the collaborative work of the above configuration units, the performance strategy configuration module can combine the advantages of expert knowledge and AI models to generate performance determination strategies suitable for different intelligent computing tasks. These strategies can be applied to task scheduling, resource allocation, load balancing, etc. to improve the overall performance of intelligent computing tasks.

[0081] The performance strategy configuration module of the present invention combines the power of expert experience and AI models to provide a flexible, efficient and adaptive performance determination strategy configuration method for intelligent computing tasks. This method can not only identify and correct the mismatch between tasks and resources, but also continuously learn and optimize to adapt to the ever-changing intelligent computing environment.

[0082] The performance strategy configuration library is a database for storing and managing performance determination strategies. The library not only stores the performance determination strategies generated by the first performance strategy configuration unit and the second performance strategy configuration unit, but also saves the application history and effect evaluation data of these strategies.

[0083] The performance strategy configuration library is further connected to the real-time performance monitoring module to obtain real-time task operation indicator data. By matching these real-time data with the stored performance judgment strategy, the performance strategy configuration library can provide the most suitable performance judgment strategy for the current intelligent computing task. This dynamic matching mechanism ensures that the performance judgment strategy can adapt to changes in the intelligent computing environment in a timely manner and improve the operating efficiency of intelligent computing tasks.

[0084] The performance strategy configuration module of the present invention can dynamically apply these performance determination strategies to cope with the complexity and dynamics of intelligent computing tasks. This integrated and intelligent performance management method significantly improves the resource utilization and task processing efficiency of the intelligent computing system.

[0085] Furthermore, a manual policy configuration function is also provided for the first performance policy configuration unit.

[0086] In this embodiment, the first performance strategy configuration unit includes a CPU performance strategy configuration sub-unit, which is used to perform strategy configuration on the CPU computing resource utilization efficiency in the intelligent computing cluster, and manually configure the absolute upper limit, lower limit and relative change upper limit and lower limit of the CPU utilization. Common rule expressions can be used to set the performance judgment strategy of the CPU utilization at a single time point and in a continuous time period.

[0087] Memory efficiency strategy configuration subunit: used to configure strategies for memory resource utilization efficiency in intelligent computing clusters, manually configure the absolute upper and lower limits and relative change upper and lower limits of memory utilization, and use common rule expressions to set memory utilization efficiency judgment strategies at a single time point and in continuous time periods.

[0088] GPU performance strategy configuration subunit: used to configure the strategy for the efficiency of GPU computing resource utilization in the intelligent computing cluster, manually configure the absolute upper and lower limits and relative change upper and lower limits of GPU utilization, and use common regular expressions to set the performance judgment strategy of GPU utilization at a single time point and in continuous time periods.

[0089] NPU performance strategy configuration subunit: used to configure the strategy for the efficiency of NPU computing resource utilization in the intelligent computing cluster, manually configure the absolute upper limit, lower limit and relative change upper limit and lower limit of NPU utilization, and use common rule expressions to set the performance judgment strategy of NPU utilization at a single time point and in a continuous time period.

[0090] GCU efficiency strategy configuration subunit: used to configure the strategy for the efficiency of GCU computing resource utilization in the intelligent computing cluster, manually configure the absolute upper limit, lower limit and relative change upper limit and lower limit of GCU utilization, and use common rule expressions to set the efficiency judgment strategy of GCU utilization at a single time point and in a continuous time period.

[0091] MLU efficiency strategy configuration subunit: used to configure the strategy for the efficiency of MLU computing resource utilization in the intelligent computing cluster, manually configure the absolute upper limit, lower limit and relative change upper limit and lower limit of MLU utilization, and use common rule expressions to set the efficiency judgment strategy of MLU utilization at a single time point and in a continuous time period.

[0092] DCU performance strategy configuration subunit: used to configure the strategy for the efficiency of DCU computing resource utilization in the intelligent computing cluster, manually configure the absolute upper limit, lower limit and relative change upper limit and lower limit of DCU utilization, and use common rule expressions to set the efficiency judgment strategy of DCU utilization at a single time point and in a continuous time period.

[0093] Storage resource efficiency policy configuration subunit: used to configure policies for storage resource utilization efficiency in the intelligent computing cluster, manually configure the absolute upper and lower limits and relative change upper and lower limits of storage resource utilization, and use common rule expressions to set the efficiency judgment strategy of storage resource utilization at a single time point and in continuous time periods.

[0094] Training task validity configuration subunit: used to configure strategies for training task output indicators in intelligent computing tasks, manually configure the absolute upper and lower limits of various loss functions and accuracy function curves in training tasks, and use common rule expressions to set training task validity judgment rules.

[0095] In this embodiment, the performance monitoring module is used to monitor the running performance of the intelligent computing task in real time and provide timely warnings and strategy recommendations when performance degradation is detected. The module mainly includes two core units: a performance monitoring unit and a historical status analysis unit.

[0096] The performance monitoring unit is used to perform real-time performance monitoring on the running intelligent computing tasks by adopting real-time strategy recommendations according to the preset performance judgment strategy. The unit collects the running indicator data of the task, such as execution time, resource consumption, error rate, etc., and compares them with the standards defined in the performance judgment strategy. Among them, the real-time strategy recommendation can be achieved by starting the microservice in the background, and sending the performance judgment strategy to the microservice rule interface. The microservice performs real-time intelligent monitoring and analysis on the tasks with the status of "running", and recommends strategies for the tasks according to the established strategies in the rule library.

[0097] If the monitored task operation indicators do not meet the requirements of the performance judgment strategy, that is, the task operation status does not meet the predetermined performance level, the performance monitoring unit will automatically generate an alarm message. These alarm messages are intended to notify system administrators or automatically trigger the repair process so that timely measures can be taken to correct the problem of insufficient performance.

[0098] In particular, the task execution efficiency can be judged by analyzing the absolute value of indicators such as utilization rate. The effectiveness of task calculation can be judged by judging the rate of change of performance indicators output during task execution, such as judging the changes in various types of loss in training tasks. When the loss no longer decreases within 20 epochs, the task training task has converged or a gradient explosion has occurred. Continuing to execute the training task will no longer have any benefit on the task result. Therefore, if the task continues, the subsequent performance of the task will be at a very low level, resulting in a waste of resources, and then generating an alarm message.

[0099] The historical status analysis unit is used to regularly analyze the historical tasks of the cluster at certain stages, analyze the resource efficiency within a given time and range, point out the distribution of low efficiency, and give recommended disposal suggestions for different roles.

[0100] The given range includes a given time period, a given computing power type, a given cluster, etc.

[0101] The historical status analysis unit obtains the historical task operation index data of the intelligent computing task within the preset time period, and conducts in-depth analysis in combination with the performance judgment strategy applied during the same period. The unit performs statistical analysis of the results of the rule analysis of the specified task group within a period of time. It can use the method of scheduled tasks, such as analyzing all completed tasks of the cluster from 0:00 to 4:00 every day. The analysis first selects the analysis task group target through historical data management, such as a cluster training task in the past month or year, and then obtains the performance statistics of the task group at the specified time through multi-dimensional data analysis. The data can be used to guide the performance optimization construction at the cluster or resource management level. For example, if the tasks of a cluster last month are counted and it is found that the memory utilization rate of the cluster is 85% low, then a suggestion to reduce the memory quota in resource allocation is given.

[0102] By analyzing historical performance data, the unit can identify trends and patterns of insufficient performance and provide system administrators with performance tuning strategy recommendations. Performance tuning strategy recommendations include, for example, adjusting resource allocation, optimizing task scheduling logic, and updating performance determination strategies, aiming to improve the performance of future tasks.

[0103] Furthermore, the performance monitoring module can provide external services through the auxiliary decision-making gateway. The gateway is connected to Nginx and can provide data to the front-end page and operation control platform in a web manner. At the same time, it can access the API interface and enterprise WeChat to push user information. The interfaces used are all standard API interfaces in the Restful style, which is convenient for system integration and expansion.

[0104] In this embodiment, the performance tuning module is used to receive the alarm information and generate a performance tuning strategy according to the performance determination strategy.

[0105] This module is used to receive alarm information and automatically generate tuning strategies based on the preset performance judgment strategy to optimize the running performance of intelligent computing tasks. The core of this module is the integrated artificial intelligence (AI) model, which is responsible for processing alarm information and generating corresponding performance tuning strategies.

[0106] The main function of the performance tuning module is to receive alarm information from the performance monitoring module. These alarm messages indicate that the running performance of the intelligent computing task does not meet the standards defined by the preset performance judgment strategy. When receiving the alarm information, the performance tuning module will activate its AI model to analyze the cause of the alarm and formulate a corresponding tuning strategy.

[0107] The AI ​​model integrated in the performance tuning module uses a trained mature model that can understand the alarm information and the corresponding performance judgment strategy. Through deep learning and pattern recognition technology, the AI ​​model can identify the root cause of the performance degradation and generate specific performance tuning strategies based on these analysis results.

[0108] Among them, performance tuning strategies include but are not limited to reconfiguring resource allocation, adjusting task priorities, optimizing data processing processes, etc. These strategies are designed to correct problems that lead to performance degradation and restore or improve the operating performance of intelligent computing tasks to a predetermined level.

[0109] Once the performance tuning strategy is generated, it will be sent to the relevant management module or system administrator for approval and implementation. In some systems with a high degree of automation, the performance tuning module can also interact directly with the resource management system to automatically implement the tuning strategy to reduce manual intervention and speed up response time.

[0110] The performance tuning module of the present invention realizes automatic monitoring and tuning of the performance of intelligent computing tasks by integrating AI models. The module can quickly respond to alarm information, use AI models to analyze the root causes of problems, and generate practical tuning strategies, thereby improving the operating efficiency and stability of the intelligent computing system. This intelligent tuning mechanism significantly improves the management efficiency of intelligent computing tasks, reduces operation and maintenance costs, and provides continuous performance assurance for the system.

[0111] Based on the contents of the above embodiments, the performance tuning module in the performance tuning system for intelligent computing clusters of the present invention also includes: a cross-analysis performance tuning unit, which is used to cross-judge multiple resources of task operation indicators and configure performance tuning strategies for intelligent computing tasks that optimize the combination of multiple resources.

[0112] The cross-analysis performance tuning unit is used to make comprehensive cross-judgments on the multiple resources involved in the intelligent computing task, and to configure performance tuning strategies for the intelligent computing task configuration based on the combination of multiple resources. In intelligent computing tasks, the use of resources and the operating efficiency of tasks often affect each other, and the performance indicators of a single resource are not sufficient to fully reflect the operating status of the system. Therefore, the cross-analysis performance tuning unit analyzes the combination of multiple resources, identifies the dependencies and interactions between resources, and thus discovers potential performance bottlenecks and optimization opportunities. On this basis, the cross-analysis performance tuning unit uses the performance judgment strategy of a single resource to optimize the combination of multiple resources to obtain a performance tuning strategy.

[0113] Based on the same inventive concept, an embodiment of the present application provides a performance tuning method for an intelligent computing cluster. Figure 2 , Figure 2 FIG. 1 is a flow chart of a performance tuning method for an intelligent computing cluster proposed in an embodiment of the present application. Figure 2 As shown, the method includes:

[0114] S21: Obtain task running indicator data of intelligent computing tasks in the intelligent computing cluster;

[0115] S22: configuring a performance determination strategy for the intelligent computing task based on the task operation indicator data, and outputting the performance determination strategy after matching through a strategy configuration library;

[0116] S23: performing performance detection on the running task according to the performance determination strategy, and generating alarm information if the running task does not meet the performance determination strategy;

[0117] S24: receiving the alarm information, and generating a performance tuning strategy according to the performance determination strategy.

[0118] The task operation indicator data includes task details information, resource utilization status and log information.

[0119] The task details include task id, user id, start and end time, status, cluster, resource type, image name, startup file name and other task details parameters. Resource utilization includes the usage of CPU, MLU, DCU, GCU, NPU, memory, storage and other indicators, such as CPU occupancy, memory occupancy, GPU occupancy, NPU occupancy, DCU occupancy, GCU occupancy, GPU memory occupancy, NPU memory utilization, GCU memory utilization, DCU memory utilization, etc. Log information is generated by user code, and is a task log with non-fixed format and content.

[0120] Optionally, the step of obtaining task running indicator data of the intelligent computing task in the intelligent computing cluster includes:

[0121] Obtain structured task details and resource utilization information in intelligent computing tasks;

[0122] Obtain unstructured log information in intelligent computing tasks;

[0123] Preprocessing the structured task details information and resource utilization information to obtain structured task operation indicator data;

[0124] Information extraction is performed on the unstructured log information to obtain unstructured task operation indicator data.

[0125] Optionally, the preprocessing includes data cleaning, standardization, interpolation of missing values, removal of tainted and abnormal data, and conversion of formats and measurement units.

[0126] Optionally, the step of configuring a performance determination strategy for the intelligent computing task based on the task running indicator data includes:

[0127] Obtain task operation indicator data, identify the mismatch pattern between tasks and resources, and configure the efficiency judgment strategy for the intelligent computing task in combination with the rule logic in the expert knowledge base;

[0128] Access AI to configure performance determination strategies for the intelligent computing tasks;

[0129] Cross-judgment of multiple resources of task operation indicators is carried out, and the strategy for determining the efficiency of intelligent task configuration is optimized for the combination of multiple resources;

[0130] The performance determination strategy is stored in the performance strategy configuration library, and the real-time task running indicator data is obtained to match the corresponding performance determination strategy.

[0131] Optionally, an AI model is used to receive the alarm information, and a performance tuning strategy is generated according to the performance determination strategy.

[0132] Optionally, the step of accepting the alarm information and generating a performance tuning strategy according to the performance judgment strategy also includes: cross-judgment of multiple resources of task operation indicators, and generating performance tuning strategies for multiple resource intelligent computing tasks based on the performance judgment strategy based on a single resource intelligent computing task.

[0133] Optionally, the step of accepting the alarm information and generating a performance tuning strategy according to the performance judgment strategy also includes: obtaining historical task operation indicator data of a preset time period of the intelligent computing task, and obtaining the corresponding performance judgment strategy to generate a performance tuning strategy.

[0134] Based on the same inventive concept, an embodiment of the present application provides a computer device. The computer device includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by the computer, the computer can execute the performance tuning method for the intelligent computing cluster provided in the above embodiments, the method comprising: obtaining task operation index data of the intelligent computing task in the intelligent computing cluster; configuring a performance judgment strategy for the intelligent computing task based on the task operation index data, and the performance judgment strategy is output after matching through a policy configuration library; performing performance detection on the running task according to the performance judgment strategy, and generating an alarm message when the running task does not meet the performance judgment strategy; receiving the alarm message, and generating a performance tuning strategy according to the performance judgment strategy.

[0135] The system embodiment described above is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0136] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0137] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A performance tuning system for intelligent computing clusters, characterized in that: include: The data processing module is used to obtain the task running indicator data of the intelligent computing tasks in the intelligent computing cluster; An efficiency strategy configuration module, used to configure an efficiency determination strategy for the intelligent computing task based on the task operation indicator data, and the efficiency determination strategy is output after matching through a strategy configuration library; A performance monitoring module, configured to perform performance detection on the running task according to the performance determination strategy, and generate an alarm message when the running task does not meet the performance determination strategy; The performance tuning module is used to receive the alarm information and generate a performance tuning strategy according to the performance determination strategy.

2. The performance tuning system for intelligent computing clusters according to claim 1, characterized in that: The data processing module comprises: The structured data aggregation unit is used to obtain structured task details and resource utilization information in intelligent computing tasks; Unstructured data aggregation unit, used to obtain unstructured log information in intelligent computing tasks; A structured data processing unit, used for preprocessing the structured task details information and resource utilization information to obtain structured task operation indicator data; The unstructured data processing unit is used to extract information from the unstructured log information to obtain unstructured task operation indicator data.

3. The performance tuning system for intelligent computing clusters according to claim 2, characterized in that: The preprocessing includes data cleaning, standardization, interpolation of missing values, removal of tainted and abnormal data, and conversion of formats and measurement units.

4. The performance tuning system for intelligent computing clusters according to claim 1, characterized in that: The performance strategy configuration module includes: A first performance strategy configuration unit is used to obtain task operation indicator data, identify the mismatch pattern between tasks and resources, and configure the performance determination strategy for the intelligent computing task in combination with the rule logic in the expert knowledge base; A second performance strategy configuration unit, used to access the AI ​​to configure a performance determination strategy for the intelligent computing task; A cross-analysis unit is used to cross-judge multiple resources of task operation indicators and optimize the intelligent calculation task configuration efficiency judgment strategy for the combination of multiple resources; The performance strategy configuration library is used to store the performance determination strategy and obtain the performance determination strategy corresponding to the real-time task running indicator data.

5. The performance tuning system for intelligent computing clusters according to claim 1, characterized in that: The performance monitoring module includes: The performance monitoring unit is used to perform performance detection on the running task according to the performance determination strategy, and generate alarm information when the running task does not meet the performance determination strategy.

6. The performance tuning system for intelligent computing clusters according to claim 1, characterized in that: The performance tuning module includes an AI model for receiving the alarm information and generating a corresponding performance tuning strategy according to the performance determination strategy.

7. The performance tuning system for intelligent computing clusters according to claim 6, characterized in that: The performance tuning module also includes: The cross-analysis performance tuning unit is used to cross-judge multiple resources of task operation indicators, and generate performance tuning strategies for multiple resource intelligent computing tasks based on the performance judgment strategy based on a single resource intelligent computing task.

8. The performance tuning system for intelligent computing clusters according to claim 6, characterized in that: The performance tuning module also includes: The historical status analysis unit is used to obtain the historical task operation indicator data of the preset time period of the intelligent computing task, and obtain the corresponding performance judgment strategy to generate a performance tuning strategy.

9. A performance tuning method for intelligent computing clusters, the method being implemented based on the system described in any one of claims 1 to 8, characterized in that: include: Obtain task running indicator data of intelligent computing tasks in the intelligent computing cluster; Based on the task running index data, the efficiency determination strategy is configured for the intelligent computing task, and the efficiency determination strategy is output after matching through the strategy configuration library; Performing performance detection on the running task according to the performance determination strategy, and generating alarm information when the running task does not meet the performance determination strategy; The alarm information is received, and a performance tuning strategy is generated according to the performance determination strategy.

10. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: When executed, the processor implements the steps of the method of claim 9.

Images