A data collection method and system based on multi-dimensional intelligent evaluation and electronic equipment

Abstract

This application belongs to the field of artificial intelligence technology and discloses a data acquisition method, system, and electronic device based on multi-dimensional intelligent evaluation. The method includes quantifying the acquired multi-dimensional performance indicators to obtain their respective health scores; constructing feature vectors corresponding to each of the multi-dimensional performance indicators based on the health scores; performing standardized preprocessing on the feature vectors, and then weighting and fusing the preprocessed feature vectors using learnable attention weights to obtain a fused feature vector; determining the anomaly probability using an intelligent anomaly detection model and the fused feature vector; and triggering trace acquisition when the anomaly probability exceeds an adaptive trigger threshold. This enables intelligent decision-making for trace data acquisition.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of artificial intelligence technology, and in particular to a data acquisition method, system and electronic device based on multidimensional intelligent evaluation. Background Technology

[0002] With the construction of domestic computing centers and the rapid development of artificial intelligence (AI) and embodied intelligence, large language models undoubtedly play a fundamental role. Therefore, to meet the configuration requirements of large language models (thousands or even tens of thousands of CPUs), higher demands are placed on the computing power, network, energy consumption, and heat dissipation of computing centers. Improving the resource utilization of computing centers while ensuring energy efficiency is a complex systems engineering project. First, the hardware of the computing center must be compatible, meaning the computing hardware must match its supporting network topology and cooling devices to maximize computing performance. Second, the training tasks of large models must be standardized. For large model training tasks submitted by numerous users, the allocation of training resources must be accurately estimated in advance to ensure effective allocation, improve the success rate of multi-task execution, and reduce ineffective resource loss. Finally, for the training task of a single large model, the computing center must have a clear understanding of the training process to avoid wasting global resources due to local bottlenecks. Furthermore, based on the analysis of historical training tasks, it should understand the patterns of large model training to better serve the efficient completion of each large model training task.

[0003] The purpose of model training simulation is to improve simulation accuracy, achieving high-precision simulation of the actual training process of large models with various parameters and configurations. However, the actual goal of large model training is to complete training faster and better, that is, to complete the training of large models better within a controllable cost range. There is a fundamental difference between the two objectives, but large model training simulation is a full-process, fine-grained simulation. This means that the simulation process can effectively simulate the multi-dimensional behavior of multiple objects during the large model training process, thus playing an important role in identifying bottleneck processes and objects in the training process. This goal presupposes the effective mining and characterization of the behavior of various hardware and software components during the large model training process, through research on various technologies and products on the market. The Chakra technology framework, jointly developed by Meta and other institutions, can effectively mine the trace (execution trajectory) of the large model training process and effectively utilizes its simulation system. However, it only considers the acquisition of behavioral snapshots of a single loop process. Considering that the actual large model training process may be subject to external interference from various parties and performance fluctuations of its own components, using a single loop process to characterize the effectiveness of the entire large model training process is not representative. Summary of the Invention

[0004] The purpose of this application is to provide a data acquisition method, system, and electronic device based on multidimensional intelligent evaluation, so as to solve the problem that the existing technology only collects the execution trajectory of a single-cycle model training and cannot reflect the ever-changing state of the model training process.

[0005] The embodiments of this application adopt the following technical solutions: This application provides a data acquisition method based on multidimensional intelligent evaluation, the data acquisition method based on multidimensional intelligent evaluation includes: The collected multi-dimensional performance indicators were quantified to obtain their respective health scores; Based on the health score, construct the feature vector corresponding to each of the multi-dimensional performance indicators; After standardizing and preprocessing the feature vectors separately, the preprocessed feature vectors are weighted and fused using learnable attention weights to obtain a fused feature vector. The probability of an anomaly is determined using the intelligent anomaly detection model and the fused feature vector. When the abnormal probability exceeds the adaptive trigger threshold, the execution trajectory is collected.

[0006] This application also provides a data acquisition system based on multidimensional intelligent evaluation, the data acquisition system based on multidimensional intelligent evaluation includes: The quantification module quantifies the collected multi-dimensional performance indicators to obtain their respective health scores; The construction module constructs feature vectors corresponding to each of the multi-dimensional performance indicators based on the health score; The fusion module performs standardized preprocessing on the feature vectors separately, and then performs weighted fusion on the preprocessed feature vectors using learnable attention weights to obtain a fused feature vector. The determination module uses the intelligent anomaly detection model and the fused feature vector to determine the anomaly probability; The triggering module triggers the collection of execution trajectories when the abnormal probability exceeds the adaptive triggering threshold.

[0007] This application also provides an electronic device, including at least one processor and a memory, wherein the memory stores a program and is configured to have at least one processor perform the following steps: The collected multi-dimensional performance indicators were quantified to obtain their respective health scores; Based on the health score, construct the feature vector corresponding to each of the multi-dimensional performance indicators; After standardizing and preprocessing the feature vectors separately, the preprocessed feature vectors are weighted and fused using learnable attention weights to obtain a fused feature vector. The probability of an anomaly is determined using the intelligent anomaly detection model and the fused feature vector. When the abnormal probability exceeds the adaptive trigger threshold, the execution trajectory is collected.

[0008] Based on the data acquisition method, system, and electronic device of multi-dimensional intelligent evaluation in the embodiments of this specification, the multi-dimensional performance indicators are quantified to obtain their respective health scores. Feature vectors corresponding to each multi-dimensional performance indicator are constructed based on the health scores. After standardization preprocessing of the feature vectors, the preprocessed feature vectors are weighted and fused using learnable attention weights to obtain fused feature vectors. An intelligent anomaly detection model is used to determine the anomaly probability. When the anomaly power exceeds the adaptive trigger threshold, the execution trajectory acquisition is triggered.

[0009] In this way, by constructing a multi-dimensional comprehensive health assessment system, the blindness of traditional random collection is overcome, and the performance degradation or abnormal state of the computing power cluster can be accurately perceived, ensuring that the collected trace data corresponds to the critical or abnormal moment of system operation, which greatly enhances the intrinsic value of the data.

[0010] In addition, by introducing feature fusion and intelligent anomaly detection models based on attention mechanisms, it is possible to automatically learn and identify true anomaly patterns from complex, nonlinear, multi-dimensional performance fluctuations, avoiding the problems of low accuracy and poor adaptability caused by single thresholds or simple rule triggers, and realizing intelligent decision-making for trace collection. Attached Figure Description

[0011] The accompanying drawings, which are included to provide a further understanding of the embodiments of this specification and form part of the embodiments of this specification, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A flowchart illustrating a data acquisition method based on multidimensional intelligent evaluation provided in the embodiments of this specification; Figure 2 This specification provides an application structure diagram of a data acquisition method system module based on multidimensional intelligent evaluation, as shown in the embodiments of this specification. Figure 3 A schematic diagram of the structure of a data acquisition system based on multidimensional intelligent evaluation provided in the embodiments of this specification; Figure 4 This is a schematic diagram of the electronic device corresponding to a data acquisition method based on multidimensional intelligent evaluation provided in the embodiments of this specification. Detailed Implementation

[0012] In existing technologies, training large language models places extremely high demands on computing power, network, and storage resources, often resulting in resource bottlenecks and performance fluctuations during the training process. Traditional trace collection methods (such as Chakra) typically collect traces randomly under stable training conditions, failing to capture critical moments of system anomalies or performance degradation. This leads to insufficient representativeness of the collected traces, making them unsuitable for real-world performance analysis and system optimization.

[0013] On the one hand, tools such as Chakra only collect single-cycle traces, which cannot reflect the changing states of the entire training process; on the other hand, the timing of execution trace collection lacks intelligent judgment and cannot trigger collection when the system is abnormal or performance degrades; secondly, existing technologies also lack a unified health assessment and integration mechanism for multi-dimensional resources such as computing, network, and memory.

[0014] Therefore, this specification provides a data acquisition method, system, and electronic device based on multi-dimensional intelligent evaluation. By quantifying the acquired multi-dimensional performance indicators, a corresponding health score is obtained for each. Based on the health score, feature vectors corresponding to each multi-dimensional performance indicator are constructed. After standardizing and preprocessing the feature vectors, the preprocessed feature vectors are weighted and fused using learnable attention weights to obtain a fused feature vector. An intelligent anomaly detection model is used to determine the anomaly probability. When the anomaly power exceeds the adaptive trigger threshold, the execution trajectory acquisition is triggered.

[0015] In this way, by constructing a multi-dimensional comprehensive health assessment system, the blindness of traditional random collection is overcome, and the performance degradation or abnormal state of the computing power cluster can be accurately perceived, ensuring that the collected trace data corresponds to the critical or abnormal moment of system operation, which greatly enhances the intrinsic value of the data.

[0016] In addition, by introducing feature fusion and intelligent anomaly detection models based on attention mechanisms, it is possible to automatically learn and identify true anomaly patterns from complex, nonlinear, multi-dimensional performance fluctuations, avoiding the problems of low accuracy and poor adaptability caused by single thresholds or simple rule triggers, and realizing intelligent decision-making for trace collection.

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0019] Please see Figure 1 This is a flowchart illustrating a data acquisition method based on multidimensional intelligent evaluation, provided in an embodiment of this specification.

[0020] In the embodiments of this specification, the data acquisition method based on multidimensional intelligent evaluation may specifically include the following steps: S101: Quantify the collected multi-dimensional performance indicators to obtain their respective health scores; S103: Construct feature vectors corresponding to each of the multi-dimensional performance indicators based on the health scores; S105: After performing standardized preprocessing on the feature vectors respectively, the preprocessed feature vectors are weighted and fused using learnable attention weights to obtain a fused feature vector; S107: Determine the anomaly probability using the intelligent anomaly detection model and the fused feature vector; S109: When the abnormal probability exceeds the adaptive trigger threshold, the execution trajectory collection is triggered.

[0021] In the embodiments of this specification, a comprehensive health assessment system integrating multi-dimensional performance indicators such as computation, network, and memory storage is constructed to overcome the blindness of traditional methods that randomly collect traces in a stable training state. Furthermore, a multi-dimensional feature fusion and intelligent anomaly detection model based on an attention mechanism is introduced to solve the problems of low accuracy and poor adaptability of single threshold triggering in complex nonlinear systems. Through deep feature learning and dynamic threshold adjustment, true anomalies can be accurately identified from complex performance fluctuations, achieving intelligent and automated trace collection and significantly improving the representativeness and analytical value of the collected data.

[0022] As an application embodiment of this specification, step S101, which involves quantifying the collected multi-dimensional performance indicators to obtain their respective health scores, may specifically include: Collect multi-dimensional performance metrics, including computing power, network communication, and memory storage metrics; The performance indicators, including computing power, network communication, and memory storage, are quantified to obtain computing power health score, network health score, and memory health score.

[0023] In the embodiments of this specification, the timing of trace acquisition is selectively chosen based on performance monitoring of the computing cluster. The main performance monitored includes computing power, network bandwidth, and memory storage. Based on a comprehensive indexing of these three performance components, health scores for computing resources, network communication, and memory storage are obtained respectively. Then, thresholds of a certain magnitude are set, and traces exceeding these thresholds are collected.

[0024] In specific application scenarios, for computing power metrics, the computing power metrics are quantified to obtain the computing power health score, which may specifically include: Collect raw data of the computing power metrics, including GPU utilization, CPU utilization, GPU SM streaming multiprocessor utilization, and computing queue depth. After standardizing and decomposing the original index data, the first principal component score is extracted using principal component analysis. The computing power health score is obtained by linearly mapping the quantiles of historical data.

[0025] In the embodiments of this specification, during the training of a large model in a computing cluster, the characteristics of computing resources are that the indicators are highly correlated and the patterns are relatively stable. Principal component analysis can effectively extract the main variation directions in such stable patterns and is suitable for discovering situations that deviate from normal computing patterns.

[0026] In practical application examples, GPU utilization can be collected using the NVML library, and the sampling interval can be set to 100ms. CPU utilization can be calculated using / proc / stat, and the sampling interval can also be set to 100ms. The utilization of the GPU's streaming multiprocessor (SM) can be statistically analyzed using the CUDA PTX instruction to calculate the proportion of active thread bundles. The computation queue depth can be monitored through the CUDA streaming queue to count the number of kernels to be executed. For the convenience of subsequent computation, statistics and analysis, the value is taken as a positive integer.

[0027] It should be noted that the above-mentioned method for collecting raw indicator data can be used as an illustration of a specific application embodiment. Other methods can also be used for data collection, which are not specifically limited here.

[0028] Furthermore, a linear mapping is performed based on the quantiles of historical data to obtain the computing power health score, which may specifically include: Obtain the historical monitoring dataset of the first principal component score; The historical monitoring datasets are arranged in ascending order; A preset first quantile is selected as the minimum quantile value, and a preset second quantile is selected as the maximum quantile value, wherein the sum of the preset first quantile and the preset second quantile is 1; Based on the minimum score and the maximum score, a linear mapping is performed on the computing power health score to obtain the computing power health score.

[0029] In the embodiments of this specification, the linear mapping of the computing power health score can be performed according to the following formula (1). ; ; Formula (1) in, This represents the standardized score of the first principal component. The minimum score. The maximum score. The score for the first principal component. This indicates the calculation of the computing power health score.

[0030] In another application embodiment of this specification, for network communication indicators, quantifying the network communication indicators to obtain the network health score may specifically include: Collect network communication metrics including bandwidth utilization, communication latency, packet loss rate, and communication error rate; The network communication indicators are processed using fuzzy set theory. Using a pre-defined fuzzy rule base, the rule strength of the fuzzy set corresponding to each of the network communication indicators is calculated; Based on the strength of the rule, output a fuzzy set; The fuzzy set is defuzzified to obtain the network health score.

[0031] In the embodiments of this specification, bandwidth utilization BU It can be obtained through the ifconfig command and the dedicated network card counter equipped in the computing cluster. The sampling frequency can be once per second, and the average value within this second is taken. Communication delay It can be obtained through ICMP ping and RDMA latency tests, with sampling done by sending 10 test packets per second and taking their 95th percentile. Packet loss rate It can be obtained through the calculator of network devices and dedicated probes, and the sampling is based on a 30-second sliding window for statistical analysis; Communication error rate The network card ERROR can be obtained through a network card ERROR calculator and verified at the protocol layer, with statistics collected based on a 60-second sliding window.

[0032] It should be noted that the above-mentioned methods for collecting network communication indicators can be used as an illustration of specific application embodiments. Other methods can also be used for collection, and no specific limitations are made here.

[0033] Furthermore, the preset fuzzy rule base may specifically include ideal network state, acceptable heavy load state, latency-sensitive state, serious network problem, and light load abnormal state.

[0034] In another application embodiment of this specification, for memory storage metrics, the memory storage metrics are quantified to obtain the memory health score, which may specifically include: Collect memory storage metrics including memory utilization, I / O throughput, paging frequency, and storage latency; Each of the aforementioned memory storage metrics is normalized. The normalized memory storage metrics are weighted and averaged to obtain the memory health score.

[0035] In the embodiments of this specification, memory utilization (MU) can be obtained through the / proc / meminfo command. The MU can be obtained by calculating the ratio of used memory to total memory. I / O throughput (IOT) can be obtained using the iostat command, and the unit is usually MB / s; Page Fault Frequency (PFR) can be obtained via / proc / vmstat, and the unit is pages / sec; Storage latency (SL) can be obtained via a probe, measured in milliseconds.

[0036] It should be noted that the above method for collecting memory storage metrics can be used as an illustration of a specific application embodiment. Other methods can also be used for collection, and no specific limitations are made here.

[0037] In addition, each memory storage metric is normalized separately, and different normalization methods are used for each memory storage metric to better reflect the characteristics of each memory storage metric.

[0038] Furthermore, as an application embodiment of this specification, for step S105, after performing standardized preprocessing on the feature vectors respectively, the preprocessed feature vectors are weighted and fused using learnable attention weights to obtain a fused feature vector, which may specifically include: Each of the aforementioned feature vectors is subjected to data normalization and time series detrending processing respectively; For the preprocessed feature vectors corresponding to the computing power index, the network communication index, and the memory storage index, calculate the attention score for each preprocessed feature vector. Based on the attention score, determine the attention weight corresponding to each preprocessed feature vector; The preprocessed feature vectors are weighted and fused based on the attention weights to obtain the fused feature vector.

[0039] In the embodiments of this specification, due to the differences in the dimensions and inherent trends of the feature vectors corresponding to the computing power index, the network communication index, and the memory storage index, standardization processing is required.

[0040] Furthermore, after obtaining the fused feature vector, the method further includes: The fused feature vector is subjected to linear dimensionality reduction.

[0041] In the embodiments of this specification, since the fused feature vector obtained after weighted fusion of attention weights has a dimension of 13, there are a number of problems, such as the curse of dimensionality and feature redundancy. By performing linear dimensionality reduction, a low-dimensional fused feature vector with high discriminative power is obtained, which can eliminate multicollinearity between features, improve model stability and interpretability, reduce computational resource consumption, maintain model interpretability and computational efficiency, and retain the most important information of the model.

[0042] As an application embodiment of this specification, for step S107, determining the anomaly probability using the intelligent anomaly detection model and the fused feature vector includes: The intelligent anomaly detection model is used to calculate the fused feature vector to obtain a probability value representing the degree of anomaly in the current state, and the probability value is the anomaly probability.

[0043] In the embodiments of this specification, by leveraging the efficiency and high accuracy of the intelligent anomaly detection model in processing tabular data, complex anomaly patterns that are difficult to describe by manual rules are learned from fused features, thereby achieving accurate and automatic identification of real anomaly states during the training process.

[0044] Furthermore, after determining the anomaly probability, the method further includes: The baseline threshold for triggering execution trajectory collection is calculated based on the statistical value of the abnormal probability during historical normal operation periods; The baseline threshold is dynamically adjusted based on the real-time system load rate to obtain an adaptive trigger threshold.

[0045] In the embodiments of this specification, since the training of a large language model is a process that is greatly affected by external factors, the set trigger threshold needs to be dynamically changed.

[0046] The statistical values ​​for the probability of anomalies may include the mean and standard deviation, without specific limitations here.

[0047] In addition, the system load rate needs to take into account the CPU, GPU and storage. Therefore, the system load rate can be the average of the utilization rates of the CPU, GPU and storage respectively. In specific application scenarios, the system load rate can be calculated once according to a preset period, and then the average of multiple system load rate values ​​in the past preset time period can be obtained to obtain the historical load benchmark. Furthermore, the benchmark threshold can be dynamically adjusted based on the obtained historical load benchmark.

[0048] As an application embodiment of this specification, for step S109, when the anomaly probability exceeds the adaptive trigger threshold, triggering the execution trajectory acquisition may specifically include: The triggering conditions for collecting execution trajectories are multi-level, including at least one of the following methods: When the anomaly probability exceeds the adaptive trigger threshold, execution trajectory acquisition is triggered. When the probability of the anomaly continues to rise over multiple consecutive sampling periods, an early warning level collection is triggered.

[0049] In the embodiments of this specification, unlike the previous method of setting a single fixed threshold, since the computing power cluster itself is a system engineering project, it is difficult to use a certain fixed value of a single physical quantity as the only threshold. Considering the core requirements of this complex system, namely performance anomalies, and the continuous observation results over a period of time, namely the system showing a trend of performance decline, two modes are set: main triggering conditions and trend warning triggering.

[0050] The main triggering condition is that when the probability of an anomaly exceeds the adaptive triggering threshold, the execution trajectory collection is immediately triggered. The trend warning is triggered when the probability of the anomaly continues to rise over multiple consecutive sampling periods, triggering a warning-level collection.

[0051] It should be noted that the number of consecutive sampling periods can be dynamically adjusted according to actual application needs, and no specific limit is made here.

[0052] The embodiments in this specification are designed with a dynamic threshold adjustment mechanism that combines historical statistics and real-time load, as well as multi-level triggering conditions that include immediate triggering and trend warnings. This not only captures significant anomalies that have occurred, but also warns of potential, gradual performance degradation trends. It effectively avoids false triggering caused by instantaneous fluctuations, enhances the robustness of the system under different loads and operating environments, and provides forward-looking data for "proactive" operation and maintenance.

[0053] This specification provides a data acquisition method based on multi-dimensional intelligent evaluation. By quantifying the acquired multi-dimensional performance indicators, a corresponding health score is obtained for each. Based on the health score, feature vectors corresponding to each multi-dimensional performance indicator are constructed. After standardizing and preprocessing the feature vectors, the preprocessed feature vectors are weighted and fused using learnable attention weights to obtain a fused feature vector. An intelligent anomaly detection model is used to determine the anomaly probability. When the anomaly power exceeds an adaptive trigger threshold, the execution trajectory acquisition is triggered.

[0054] In this way, by constructing a multi-dimensional comprehensive health assessment system, the blindness of traditional random collection is overcome, and the performance degradation or abnormal state of the computing power cluster can be accurately perceived, ensuring that the collected trace data corresponds to the critical or abnormal moment of system operation, which greatly enhances the intrinsic value of the data.

[0055] In addition, by introducing feature fusion and intelligent anomaly detection models based on attention mechanisms, it is possible to automatically learn and identify true anomaly patterns from complex, nonlinear, multi-dimensional performance fluctuations, avoiding the problems of low accuracy and poor adaptability caused by single thresholds or simple rule triggers, and realizing intelligent decision-making for trace collection.

[0056] It should be noted that the above-described specific data acquisition method based on multidimensional intelligent evaluation is merely a specific application example and does not limit the scope of the embodiments in this specification. Other specific embodiments may also be included, which will not be elaborated here.

[0057] Based on the same inventive concept, this specification also provides specific application examples of the above-described data acquisition method based on multidimensional intelligent evaluation.

[0058] like Figure 2 The diagram shown is an application structure diagram of a system module corresponding to a data acquisition method based on multidimensional intelligent evaluation provided in an embodiment of this specification.

[0059] In the embodiments of this specification, the data acquisition method based on multidimensional intelligent assessment may specifically include the following functional modules: data acquisition layer, health assessment module, feature fusion module, anomaly detection module, threshold triggering module, and trace data acquisition module.

[0060] Specifically, the health assessment module may include a computing power health assessment module, a network health assessment module, and a memory health assessment module.

[0061] The data acquisition layer is used to collect the raw data for each indicator.

[0062] The computing health assessment module collects metrics such as GPU utilization, CPU utilization, SM (streaming multiprocessor) utilization, and computing queue depth. After standardization, it uses principal component analysis (PCA) to extract the first principal component score and performs a linear mapping based on the quantiles of historical data to obtain the computing health score.

[0063] The network health assessment module collects indicators such as bandwidth utilization, communication latency, packet loss rate, and communication error rate. It performs inference by constructing a fuzzy logic system (including input variable fuzzification, fuzzy rule base, and centroid defuzzification) and outputs a network health score.

[0064] The memory health assessment module collects metrics such as memory usage, I / O throughput, paging frequency, and storage latency. After processing these metrics using different normalization methods (such as inverse normalization and Sigmoid normalization), they are combined using a weighted geometric mean to obtain a storage health score.

[0065] The feature fusion module is used to construct a multi-dimensional feature vector from the above three types of health scores and their derived time-series features (such as gradient and volatility). After standardization and detrending, the feature vector is weighted and fused through a learnable attention mechanism, and then linear dimensionality reduction is performed using a fully connected layer to obtain a low-dimensional fused feature vector with high discriminative power.

[0066] The intelligent anomaly detection module receives the fused feature vector, calculates it using a pre-trained LightGBM gradient boosting decision tree model, and outputs a probability value representing the degree of anomaly in the current state of the system.

[0067] The dynamic threshold triggering module is used to calculate the baseline threshold based on the statistical values ​​(mean and standard deviation) of the abnormal probability during recent historical normal operation periods, and dynamically adjust it in combination with the real-time system load rate to generate the final adaptive triggering threshold.

[0068] The Trace data acquisition module monitors the anomaly probability. When this value exceeds the dynamic trigger threshold, it immediately sends a command to the Trace acquisition tool (such as Chakra) to trigger a Trace acquisition. Furthermore, this module can also support early warning triggering based on the worsening trend of the anomaly probability.

[0069] Traditional simulation tools such as Astra-Sim and SimAi often collect one epoch of the large model training process and use it as a representation of the entire large model training process. However, for the users of simulation tools—the construction and operation and maintenance units of computing power clusters—they have a more practical need for the actual trace generated in the current computing power cluster. That is, how to obtain the trace that truly represents the large model training process and the performance changes of the computing power cluster is the main research content of the embodiments in this specification.

[0070] Unlike the traditional method of randomly selecting traces at a certain moment during stable training, this embodiment of the specification will selectively choose the timing of trace acquisition based on performance monitoring of the computing cluster. The main performance metrics monitored include computing power, network bandwidth, and memory storage. Based on a comprehensive indexing of these three performance metrics, health scores for computing resources, network communication, and memory storage are obtained respectively. Then, threshold values ​​are set for specific parameters, and traces exceeding these threshold values ​​are collected. The specific process is as follows: First, regarding computing power metrics.

[0071] During the training of large models in computing clusters, the characteristics of computing resources are that the indicators are highly correlated and the patterns are relatively stable. The PCA method (principal component analysis) can effectively extract the main direction of variation in this stable pattern and is suitable for discovering situations that deviate from the normal computing pattern.

[0072] (1) First, the first step is to collect the original indicators.

[0073] The raw metrics collection period must be at least 7x24 hours of continuous normal model training, and the collected data scale should generally be no less than 10,000 complete samples at different time points. The training phases covered should include typical workloads such as normal training, data loading, and checkpoint saving. Specific performance metrics collected include GPU utilization (collected via the NVML library at 100ms intervals), CPU utilization (calculated via / proc / stat, also at 100ms intervals), GPU SM streaming multiprocessor utilization (calculated using CUDA PTX instructions to determine the proportion of active thread bundles), and finally, computation queue depth (monitored via CUDA streaming queues to determine the number of kernels to be executed; for ease of subsequent calculation, statistics, and analysis, the value is taken as a positive integer).

[0074] (2) Then, the second step is to perform data standardization.

[0075] For the four types of performance indicators monitored above The Z-score method is used for standardization, which can be performed according to the following formula (2): Formula (2) in, for The value after Z-score standardization for The mean of the sequence, for Standard deviation of the sequence.

[0076] (3) Secondly, the third step is to calculate the covariance matrix and perform eigenvalue decomposition.

[0077] Use the collected historical standardized data matrix , where n represents the number of time points for all collected data. The covariance matrix can then be calculated using the following formula (3): Formula (3) The eigenvalue decomposition process involves solving for the eigenvalues ​​of the covariance matrix. and the corresponding eigenvectors are .

[0078] (4) Then, the fourth step is the selection of principal components and the construction of the projection matrix.

[0079] According to the requirements of the PCA method, the contribution rate of the cumulative variance here needs to reach more than 95%, which can be calculated according to the following formula (4): Formula (4) Once a suitable value of k is found, the first k eigenvectors are selected to construct the projection matrix. It can be calculated using the following formula (5): Formula (5).

[0080] (5) Finally, the fifth step is to perform real-time scoring calculation.

[0081] For real-time standardized data The scores of the principal components are calculated according to the following formula (6): Formula (6) Where scores represent real-time scores, and the score of the first principal component is... .

[0082] Following the method described above, the scores of the first principal component are calculated on all historical monitoring datasets. Sort the samples in ascending order and take the 2.5th percentile as the minimum score. Take the 97.5th percentile as This covers 95% of normal operating conditions. Based on the above mapping using a linear mapping function to calculate the computing power health score, it can be specifically calculated according to the following formula (7): ; Formula (7) in, This represents the standardized score of the first principal component. This represents the health score of the computing power.

[0083] The computing power health status provided in the embodiments of this specification is based on PCA principal component analysis, which reduces the dimensionality of indicators such as GPU utilization, CPU utilization, SM utilization, and computing queue depth, and obtains the health status score by mapping the quantile of the first principal component score.

[0084] Secondly, regarding network bandwidth metrics.

[0085] The evaluation of network resources needs to consider a large number of factors that are difficult to quantify precisely. Fuzzy logic provides a natural framework for transforming qualitative experience into quantitative evaluation, which is suitable for dealing with the gray areas in network performance.

[0086] (1) First, the first step is to define and process the input variables.

[0087] The bandwidth utilization (BU) will be obtained through the ifconfig command and the dedicated network card counter equipped in the computing cluster. The sampling frequency is once per second, and the average value within this second is taken. Specifically, it can be calculated according to the following formula (8): Formula (8) in, This represents the bandwidth utilization at time t. This represents the actual throughput at time t. This represents the theoretical maximum bandwidth.

[0088] Communication delay Acquired through ICMP ping and RDMA latency tests, with sampling done by sending 10 test packets per second and taking the 95th percentile.

[0089] Packet loss rate The data is obtained through the calculator and dedicated probe of the network device. The sampling is based on a 30-second sliding window for statistical analysis, and can be calculated according to the following formula (9): Formula (9) in, and These represent the total number of packets sent and the number of packets lost at time t, respectively.

[0090] Communication error rate The network card ERROR is obtained through a network card ERROR calculator and verified at the protocol layer. The data is collected based on a 60-second sliding window and statistically analyzed. Specifically, it can be calculated according to the following formula (10): Formula (10) in, and These represent the total number of transmitted packets and the number of erroneous packets at time t, respectively.

[0091] (2) Then, the second step is the design of fuzzy sets and membership functions.

[0092] The fuzzy set of bandwidth utilization (BU) is divided into three categories: low, medium, and high. The low category uses a triangular membership function with the parameter set [0, 0, 0.5]. Specifically, it can be calculated according to the following formula (11): Formula (11) Among them, the medium value uses a trigonometric membership function with the parameter set [0.3, 0.5, 0.7], which can be calculated according to the following formula (12): Formula (12) The high value is calculated using the trigonometric membership function with the parameter set [0.5, 1.0, 1.0]. Specifically, it can be calculated using the following formula (13): Formula (13) Communication delay The fuzzy set is divided into three categories: low, medium, and high. The low category uses the triangular membership function with the parameter set [0,0,20]. Specifically, it can be calculated according to the following formula (14): Formula (14) The medium value uses trigonometric membership functions, and the parameter set used is... Specifically, it can be calculated according to the following formula (15): Formula (15) The high parameter uses the trigonometric membership function, and the parameter set used is... Specifically, it can be calculated according to the following formula (16): Formula (16) Packet loss rate The fuzzy set is divided into three categories: none, low, and high. The none category uses the triangular membership function with the parameter set [0,0,0.01]. Specifically, it can be calculated according to the following formula (17): Formula (17) The low-level function uses trigonometric membership functions, and the parameter set used is... Specifically, it can be calculated according to the following formula (18): Formula (18) The high parameter uses the trigonometric membership function, and the parameter set used is... Specifically, it can be calculated according to the following formula (19): Formula (19) Communication error rate The fuzzy set is divided into three categories: none, low, and high. The none category uses the triangular membership function with the parameter set [0, 0, 0.005]. Specifically, it can be calculated according to the following formula (20): Formula (20) The low-level function uses trigonometric membership functions, and the parameter set used is... Specifically, it can be calculated according to the following formula (21): . Formula (21) (3) The third step is to define the output variables.

[0093] In this specification, the embodiments define Network Health (NH) as the output variable, with its universe of discourse defined as [0, 1.0], where 0 represents complete failure and 1.0 represents an ideal network state. This output table uses three fuzzy sets for description: ① Difference: Use trigonometric membership functions with the parameter set [0,0,0.4]; ② In the middle: use the trigonometric membership function, with the parameter set [0.3, 0.5, 0.7]; ③ Good: Use the trigonometric membership function with the parameter set [0.6, 1.0].

[0094] (4) The fourth step is to design a fuzzy rule base.

[0095] Based on network performance engineering practices, the embodiments in this specification design at least five core fuzzy rules: ① Rule 1: Ideal Network State Description: If bandwidth utilization is high, communication latency is low, packet loss rate is none, and error rate is none, then network health is good. The idea is that high bandwidth utilization combined with low latency, no packet loss and errors indicates that the network maintains excellent performance under heavy load, which is the ideal network state for training large models.

[0096] ② Rule 2: Acceptable overload states Description: If bandwidth utilization is high, communication latency is low, packet loss rate is none or low, and error rate is none or low, then network health is good. The idea is that under heavy load, minor packet loss or errors can usually be handled by upper-layer protocols, and as long as the latency remains low, the network is still in an acceptable state.

[0097] ③ Rule 3: Delay-sensitive state Description: IF (bandwidth utilization is low OR bandwidth utilization is high) AND communication latency is high AND packet loss rate is none AND error rate is none THEN network health is fair Approach: High latency has a significant impact on distributed training performance, and should be monitored even if other metrics are normal.

[0098] ④ Rule 4: Serious Network Problems Description: IF (bandwidth utilization is low OR bandwidth utilization is high) AND communication latency is high AND (packet loss rate is high OR error rate is high) THEN network health is poor Approach: High latency combined with high packet loss or high error rate indicates a serious network problem that requires immediate intervention.

[0099] ⑤ Rule 5: Light Load Abnormal State Description: If bandwidth utilization is low, communication latency is low, and packet loss rate is high or error rate is high, then network health is fair. Reasoning: High packet loss or errors under light load conditions may indicate hardware failure or configuration problems.

[0100] (5) The fifth step is to implement the mechanism of fuzzy reasoning.

[0101] The embodiments in this specification use a Mamdani-type fuzzy inference system. The inference process first calculates the premise strength of each rule, which can be calculated according to the following formula (22): Formula (22) In the above formula, represents the membership degree of each input variable in the corresponding fuzzy set.

[0102] For rules involving OR operations, the maximum value is used for calculation, specifically according to the following formula (23): . Formula (23) (6) The sixth step is to output the fuzzy set and perform defuzzification calculation.

[0103] First, the output fuzzy set of each rule is truncated using the MIN operation, specifically calculated according to the following formula (24): Formula (24) The output fuzzy sets of all rules are aggregated using the MAX operation to form the final output fuzzy set, which can be calculated according to the following formula (25): Formula (25) in, Let represent the fuzzy set of the final output, and y represent the coordinates of a discrete point in the universe of discourse representing the network health score.

[0104] Then, deblurring calculation is performed. In the embodiments of this specification, the centroid method is used for deblurring to calculate the accurate network health score. Specifically, it can be calculated according to the following formula (26): . Formula (26) The network communication health status provided in the embodiments of this specification is based on a fuzzy logic system. It processes fuzzy inputs such as bandwidth utilization, communication latency, packet loss rate, and communication error rate, and defuzzifies them using a preset fuzzy rule base and centroid method to output an accurate health status score.

[0105] Secondly, regarding memory storage metrics.

[0106] The performance of storage resources is often determined by the weakest link. The "weakest link effect" of weighted geometric average perfectly matches this characteristic, ensuring that the deterioration of any single indicator is fully reflected in the overall score.

[0107] (1) First, the first step is to define the input metrics.

[0108] In the embodiments described in this specification, the memory storage health assessment is based on the following four core indicators: ① Memory utilization (MU), this metric can be obtained through the / proc / meminfo command. The MU can be obtained by calculating the ratio of used memory to total memory; ② I / O throughput (IOT), this metric is obtained through the iostat command, and the unit is usually MB / s; ③ Page Fault Frequency (PFR): Obtained via / proc / vmstat, in pages / sec ④ Storage latency (SL): obtained via probe, in milliseconds.

[0109] (2) The second step is to normalize the indicators. Different normalization methods are used for each indicator to reflect its characteristics. Specifically: ① The memory utilization rate MU is calculated using inverse normalization, specifically according to the following formula (27): Formula (27) ② I / O throughput (IOT) is normalized using Sigmoid, and can be calculated using the following formula (28): Formula (28) ③ The page refresh rate (PFR) is normalized by exponential decay and can be calculated using the following formula (29): Formula (29) ④ The storage latency SL is inversely normalized, and can be calculated according to the following formula (30): Formula (30) above, These are the normalized values ​​of MU, IOT, PFR, and SL, respectively.

[0110] (3) The third step is to perform weighted set average calculation.

[0111] Based on testing and expert experience, the weights are set as follows: memory utilization (MU) has a weight of 0.3, I / O throughput (IOT) has a weight of 0.25, page refresh rate (PFR) has a weight of 0.25, and storage latency (SL) has a weight of 0.2. The weighted average can then be calculated using the following formula (31): Formula (31) Here, SH represents the memory storage health index.

[0112] The memory storage health indicator provided in the embodiments of this specification is based on a weighted geometric average method, which integrates indicators such as memory utilization, I / O throughput, paging frequency, and storage latency. It utilizes the "weakest link effect" characteristic to ensure that the deterioration of any indicator is fully reflected in the overall score.

[0113] Furthermore, the embodiments in this specification also provide a comprehensive health assessment and Trace triggering mechanism.

[0114] The above embodiments have obtained health assessments of computing resources, network communication, and memory storage. Now, it is necessary to effectively integrate the health of these three aspects. This specification describes a deep feature fusion method for multiple health indicators, constructing multi-dimensional health features, fusing these features through an attention mechanism, and then building an anomaly detection model based on LightGBM to learn the threshold for Trace collection triggering. Multi-level triggering conditions are established to achieve intelligent and effective collection triggering. The specific steps are as follows: (1) Feature engineering and representation learning.

[0115] The first step is to construct the input feature vector, including: ① Calculate resource feature vectors Specifically, it can be calculated according to the following formula (32): , formula (32) in, This represents the time gradient of the calculated resource health score, used to reflect the changing trend of the calculated resource health over time. The volatility of the calculated resource health score is represented by the standard deviation of the calculated resource health score sequence, calculated using a time window with a width of 30 seconds.

[0116] ② Network communication feature vector Specifically, it can be calculated according to the following formula (33): , is a 5-dimensional vector. Formula (33) Here, NH represents the network communication health score, BU represents network bandwidth utilization, obtained through the ifconfig command and a dedicated network interface card (NIC) counter on the computing cluster, collected once per second, and the average value within that second is taken. TL represents communication latency, sampled by sending 10 test packets per second and taking the 95th percentile. PLR represents packet loss rate, obtained through the network device's calculator and a dedicated probe. ER represents communication error rate, obtained through the NIC's ERROR calculator and verified at the protocol layer.

[0117] ③ Memory storage of feature vectors Specifically, it can be calculated according to the following formula (34): , is a 5-dimensional vector. Formula (34) Wherein, SH represents the memory storage health score, MU represents the memory utilization rate, IOT represents the I / O throughput, PFR represents the paging frequency, and SL represents the storage latency.

[0118] It should be noted that the specific methods for obtaining and calculating each item have already been explained above, and will not be repeated here.

[0119] (2) Feature standardization processing.

[0120] The data obtained from the three aspects mentioned above need to be standardized due to their differences in dimensions and inherent trends.

[0121] ① First, data normalization is performed. Here, the Z-score standardization method is used, for example, for computing resource feature vectors. Specifically, it can be calculated according to the following formula (35): Formula (35) in, Represents the feature vector of computing resources The value after normalization These are the computational resource feature vectors. The sequence mean and standard deviation.

[0122] ② Time series data removal to eliminate trend effects. Due to the differences in the objects and environments from which the data is generated, some physical quantities exhibit inherent trends. To ensure the effectiveness of subsequent fusion, these trends need to be eliminated beforehand. Here, a first-order difference method is used, for example, for calculating resource feature vectors. Specifically, it can be calculated according to the following formula (36): Formula (36) in, This represents the computational resource feature vector. The value after trend elimination is complete.

[0123] (3) Calculation of attention weights.

[0124] As mentioned earlier, the fusion of multi-dimensional features based on the attention mechanism requires the calculation of attention weights.

[0125] First, initialize the learnable parameters, including: query vector. Weight matrix Bias term For each dimension of the feature vector Calculate intermediate representation Specifically, it can be calculated according to the following formula (37): . Formula (37) Attention score The specific calculation can be performed according to the following formula (38): . Formula (38) Furthermore, the attention weights can be obtained by normalization using the softmax function, specifically calculated according to the following formula (39): Formula (39) in, Indicates attention weights, And there are , These are attention scores for computing resources, network communication, and memory storage, respectively.

[0126] (4) Implementation of weighted feature fusion.

[0127] This stage is relatively simple and clear. It involves taking the features of each dimension and multiplying them by the corresponding attention weights. Specifically, it can be calculated according to the following formula (40): Formula (40) in, This indicates weighted features, and the concatenated feature dimensions are 3+5+3=13.

[0128] This specification proposes a multi-dimensional feature fusion framework based on an attention mechanism. After standardizing and detrending the feature vectors from computation, network, and memory storage, the framework performs weighted fusion using learnable attention weights to generate highly discriminative fused feature vectors. Furthermore, it utilizes fully connected layers for linear dimensionality reduction to improve the efficiency and stability of subsequent models.

[0129] (5) Dimension reduction processing.

[0130] After attention-weighted fusion, the feature vector reaches a dimension of 13, which presents several problems, such as: ① Curse of Dimensionality: High-dimensional features lead to an exponential increase in the demand for training samples for subsequent models, which is difficult to meet in real-world production environments. ② Feature redundancy: There is information overlap between features of different modalities, and there is correlation between features. ③ Computational efficiency: High-dimensional features increase model training and inference time, affecting real-time performance. ④ Risk of overfitting: An imbalance between dimensionality and sample size can easily lead to model overfitting and reduce generalization ability. Therefore, the example described in this specification performs dimensionality reduction on the feature vectors, compressing them from 13 dimensions to 8 dimensions. The purpose of dimensionality reduction is primarily to eliminate multicollinearity among features, improve model stability and interpretability, and reduce computational resource consumption. Eight dimensions are chosen because they can retain over 95% of the information. Here, a fully connected layer is used for linear dimensionality reduction, rather than non-linear dimensionality reduction, to maintain model interpretability and computational efficiency. Furthermore, linear dimensionality reduction can be viewed as a feature recombination process, retaining the most important information through linear combination of the original features. The specific steps for dimensionality reduction are as follows: Step 1: Set the weight matrix .

[0131] Since the input dimension is 13, and the required output dimension is 8, a weight matrix is ​​needed. .

[0132] Step 2: Set the bias term .

[0133] The bias term is initialized with a value of 0 to avoid introducing biases from the initial state. Specifically, it can be calculated according to the following formula (41): Formula (41) Step 3: Dimensionality reduction calculation process.

[0134] After setting the weight matrix and bias terms Then, dimensionality reduction calculation is performed, which can be calculated according to the following formula (42): Formula (42) Step 4: Reconstruction error assessment.

[0135] First, the specific operation of refactoring can be calculated according to the following formula (43): . Formula (43) The reconstruction error can be calculated using the following formula (44): . Formula (44) When the reconstruction error is less than 0.05, it indicates that the information loss is within an acceptable range; otherwise, the dimensionality reduction is considered to have failed.

[0136] (6) Construct an anomaly detection model based on LightGBM.

[0137] After completing the attention-weighted fusion and dimensionality reduction of multimodal features to obtain an 8-dimensional fusion feature vector with high discriminativeness, this specification introduces the LightGBM gradient boosting decision tree model to construct an intelligent triggering decision core to address the low accuracy and poor adaptability of traditional threshold triggering methods when dealing with performance fluctuations in complex and nonlinear systems. The core objective is to leverage LightGBM's efficiency and high accuracy in processing tabular data to learn complex anomaly patterns from the fusion features that are difficult to describe through manual rules, thereby achieving accurate and automatic identification of real anomaly states during training. Specifically, using the aforementioned 8-dimensional fusion features as input, multiple weak decision trees are integrated to minimize the binary classification cross-entropy loss, and their inherent feature importance ranking ability is utilized to enhance model interpretability. Early stopping and regularization strategies are employed during training to prevent overfitting, ultimately outputting an anomaly probability between 0 and 1, providing a reliable and robust decision basis for subsequent dynamic threshold triggering. The specific steps are as follows: Step 1: Model Information Description.

[0138] The input layer of the model is the aforementioned 8-dimensional fused feature vector. The hidden layer is an ensemble learning model consisting of multiple decision trees, with 100 trees, a tree depth of 8, a leaf node dimension of 63, and a learning rate of 0.1. It uses a decreasing strategy to converge gradually.

[0139] The output layer outputs the anomaly probability through the sigmoid activation function. The calculation method can be specifically calculated according to the following formula (45): , formula (45) Where z is the output value of the final decision tree set.

[0140] The feature importance learning mechanism automatically sums the split gains of all decision trees to derive feature importance and outputs a ranking of feature importance for model interpretation and feature selection. An importance threshold retains features with importance > 0.01 in the final decision.

[0141] Step 2: Training data construction process.

[0142] The first step is the collection of positive samples, which involves marking outlier moments in historical data, including anomalies in computing resources (…). <0.4 and lasting for more than 5 cycles), network communication abnormalities (NH<0.4 and BU>0.9), and memory storage abnormalities (SH<0.4 and PFR>100).

[0143] Next, negative samples are collected by randomly selecting a time when the system is operating normally, ensuring that the ratio of positive to negative samples is 1:3. The total number of samples is set according to the experimental effect testing settings. The model covers different load patterns, with data split into 70% training set, 20% validation set, and 10% test set. Gaussian noise with a standard deviation of 0.01 is added to augment the training data. The model training loss function uses binary cross-entropy loss. Training is typically stopped when the validation set does not decrease for 10 consecutive rounds.

[0144] Step 3: Dynamic threshold adjustment mechanism.

[0145] Because training a large language model is highly susceptible to external influences, the set threshold needs to be dynamically adjusted. The first step is to adjust the baseline threshold by collecting data from the last 100 successful runs. The value is the benchmark threshold. Specifically, it can be calculated using the following formula (46): Formula (46) in, These are the 100 normal operation times. The mean and standard deviation of the values. In practice, the baseline threshold needs to be updated weekly.

[0146] Next is load-aware adjustment, which defines the system load rate by comprehensively considering CPU, GPU, and storage. Specifically, it can be calculated according to the following formula (47): , formula (47) in, These are the utilization rates of CPU, GPU, and storage, calculated every 10 minutes. .

[0147] Then the 144 from the past 24 hours Calculate the average value Since the historical load is the baseline, the load adjustment can be calculated according to the following formula (48): , formula (48) in, This represents the baseline threshold after load adjustment. This represents the load adjustment factor.

[0148] To ensure that the threshold remains within a relatively stable and reasonable range, the final threshold is as follows: . Formula (49) Step 4: Multi-level triggering conditions.

[0149] Unlike previous methods that fixed a single threshold, this approach addresses the challenge of using a single physical quantity as the sole threshold due to the complex nature of a computing cluster as a system engineering project. It considers the core requirement of this complex system: performance anomalies, and the continuous observation over a period of time, indicating a performance degradation trend. Two triggering modes are implemented: a primary trigger condition and a trend warning trigger.

[0150] First is the main triggering condition, namely IF. Then, `Trigger_Trace()` is called, where `Trigger_Trace()` indicates the action that triggers trace data collection. To prevent false triggers due to momentary fluctuations, a 100ms buffer is set. The next step is trend warning triggering, which analyzes the probability of anomalies. The trend changes over time; set parameters for abnormal changes. Specifically, it can be calculated according to the following formula (50): Formula (50) Continuously sample for three cycles (the sampling period is generally 1 second), and when abnormal changes occur in the parameter within the three cycles... All less than the threshold The system triggers trend alerts and collects traces. The triggering levels and geographical locations for trend alerts are specified.

[0151] This specification's embodiments utilize a dynamic threshold adjustment and multi-level triggering condition mechanism: Dynamic threshold adjustment: Based on the recent abnormal probability benchmark and real-time system load rate, the trigger threshold is dynamically adjusted to adapt to different operating environments.

[0152] Multi-level triggering: Set the main triggering conditions and trend warning triggers to achieve accurate response to anomalies of different severity and types.

[0153] By constructing an intelligent anomaly detection model based on LightGBM as the core trigger decision, and using dimensionality-reduced fused features as input, the model learns complex anomaly patterns by integrating multiple decision trees, and outputs anomaly probabilities. The model training employs a positive-negative sample balancing strategy, early stopping, and regularization to ensure its accuracy and generalization ability.

[0154] Finally, a complete end-to-end intelligent trace acquisition process is defined, which includes multi-source performance data acquisition, health calculation of various dimensions, feature fusion and dimensionality reduction, intelligent anomaly detection and dynamic trigger acquisition. This forms a systematic, adaptive and high-precision trace acquisition solution, providing a high-quality data foundation for performance analysis of computing clusters and large model training optimization.

[0155] This specification's embodiments overcome the blindness of traditional methods that randomly collect traces in a stable training state by constructing a comprehensive health assessment system encompassing computing, network, and memory storage. By quantifying the health status of these three resources using methods such as PCA, fuzzy logic, and weighted geometric average, it achieves accurate perception of changes in computing cluster performance, ensuring that the collected traces truly reflect the critical state of the system.

[0156] Furthermore, by introducing multi-dimensional feature fusion based on an attention mechanism and the LightGBM intelligent anomaly detection model, the problems of low accuracy and poor adaptability of single threshold triggering in complex nonlinear systems are solved. Through deep feature learning and dynamic threshold adjustment, true anomalies can be accurately identified from complex performance fluctuations, realizing intelligent and automated trace collection and significantly improving the representativeness and analytical value of the collected data.

[0157] Secondly, the embodiments in this specification design multi-level triggering conditions and combine them with a dynamic threshold adjustment mechanism, which can effectively avoid false triggering caused by instantaneous fluctuations, while also capturing the degradation trend of system performance. This enables the method not only to capture anomalies that have already occurred, but also to provide early warnings of potential performance degradation, providing forward-looking data support for proactive operation and maintenance and performance optimization of computing clusters.

[0158] The specific implementation process of the embodiments in this specification can be referred to the various implementation steps corresponding to the above embodiments, and will not be repeated here.

[0159] Based on the same inventive concept, embodiments of this specification also provide a data acquisition system based on multi-dimensional intelligent evaluation. For example... Figure 3 The diagram shown is a structural schematic of a data acquisition system based on multidimensional intelligent evaluation provided in an embodiment of this specification.

[0160] Specifically, the data acquisition system based on multidimensional intelligent evaluation may include: The quantification module 301 quantifies the collected multi-dimensional performance indicators to obtain their respective health scores. Construction module 302 constructs feature vectors corresponding to each of the multi-dimensional performance indicators based on the health score; The fusion module 303, after performing standardized preprocessing on the feature vectors respectively, performs weighted fusion on the preprocessed feature vectors through learnable attention weights to obtain a fused feature vector; The determination module 304 uses the intelligent anomaly detection model and the fused feature vector to determine the anomaly probability; Trigger module 305 triggers execution trajectory acquisition when the abnormal probability exceeds the adaptive trigger threshold.

[0161] based on Figure 3 The system described in this specification also provides some specific implementation schemes of the system, which will be described below.

[0162] Furthermore, the collected multi-dimensional performance indicators are quantified to obtain their respective health scores, including: Collect multi-dimensional performance metrics, including computing power, network communication, and memory storage metrics; The performance indicators, including computing power, network communication, and memory storage, are quantified to obtain computing power health score, network health score, and memory health score.

[0163] Furthermore, for the computing power indicator, the computing power indicator is quantified to obtain the computing power health score, including: Collect raw data of the computing power metrics, including GPU utilization, CPU utilization, GPU SM streaming multiprocessor utilization, and computing queue depth. After standardizing and decomposing the original index data, the first principal component score is extracted using principal component analysis. The computing power health score is obtained by linearly mapping the quantiles of historical data.

[0164] Furthermore, a linear mapping is performed based on the quantiles of historical data to obtain the computing power health score, including: Obtain the historical monitoring dataset of the first principal component score; The historical monitoring datasets are arranged in ascending order; A preset first quantile is selected as the minimum quantile value, and a preset second quantile is selected as the maximum quantile value, wherein the sum of the preset first quantile and the preset second quantile is 1; Based on the minimum score and the maximum score, a linear mapping is performed on the computing power health score to obtain the computing power health score.

[0165] Furthermore, the computational power health score is linearly mapped according to the following formula: ; ; in, This represents the standardized score of the first principal component. The minimum score. The maximum score. The score for the first principal component. This indicates the calculation of the computing power health score.

[0166] Furthermore, for network communication indicators, the network communication indicators are quantified to obtain the network health score, including: Collect network communication metrics including bandwidth utilization, communication latency, packet loss rate, and communication error rate; The network communication indicators are processed using fuzzy set theory. Using a pre-defined fuzzy rule base, the rule strength of the fuzzy set corresponding to each of the network communication indicators is calculated; Based on the strength of the rule, output a fuzzy set; The fuzzy set is defuzzified to obtain the network health score.

[0167] Furthermore, the preset fuzzy rule base includes ideal network state, acceptable heavy load state, latency-sensitive state, serious network problem, and light load abnormal state.

[0168] Furthermore, for memory storage metrics, the memory storage metrics are quantified to obtain the memory health score, including: Collect memory storage metrics including memory utilization, I / O throughput, paging frequency, and storage latency; Each of the aforementioned memory storage metrics is normalized. The normalized memory storage metrics are weighted and averaged to obtain the memory health score.

[0169] Furthermore, after standardizing and preprocessing the feature vectors separately, the preprocessed feature vectors are weighted and fused using learnable attention weights to obtain a fused feature vector, including: Each of the aforementioned feature vectors is subjected to data normalization and time series detrending processing respectively; For the preprocessed feature vectors corresponding to the computing power index, the network communication index, and the memory storage index, calculate the attention score for each preprocessed feature vector. Based on the attention score, determine the attention weight corresponding to each preprocessed feature vector; The preprocessed feature vectors are weighted and fused based on the attention weights to obtain the fused feature vector.

[0170] Furthermore, after obtaining the fused feature vector, the system further includes: The fused feature vector is subjected to linear dimensionality reduction.

[0171] Furthermore, the anomaly probability is determined using the intelligent anomaly detection model and the fused feature vector, including: The intelligent anomaly detection model is used to calculate the fused feature vector to obtain a probability value representing the degree of anomaly in the current state, and the probability value is the anomaly probability.

[0172] Furthermore, after determining the anomaly probability, the system further includes: The baseline threshold for triggering execution trajectory collection is calculated based on the statistical value of the abnormal probability during historical normal operation periods; The baseline threshold is dynamically adjusted based on the real-time system load rate to obtain an adaptive trigger threshold.

[0173] Furthermore, when the anomaly probability exceeds the adaptive trigger threshold, execution trajectory acquisition is triggered, including: The triggering conditions for collecting execution trajectories are multi-level, including at least one of the following methods: When the anomaly probability exceeds the adaptive trigger threshold, execution trajectory acquisition is triggered. When the probability of the anomaly continues to rise over multiple consecutive sampling periods, an early warning level collection is triggered.

[0174] This specification provides a data acquisition system based on multi-dimensional intelligent evaluation. By quantifying the acquired multi-dimensional performance indicators, a corresponding health score is obtained for each. Based on the health score, feature vectors corresponding to each multi-dimensional performance indicator are constructed. After standardizing and preprocessing the feature vectors, the preprocessed feature vectors are weighted and fused using learnable attention weights to obtain a fused feature vector. An intelligent anomaly detection model is used to determine the anomaly probability. When the anomaly power exceeds the adaptive trigger threshold, the execution trajectory acquisition is triggered.

[0175] In this way, by constructing a multi-dimensional comprehensive health assessment system, the blindness of traditional random collection is overcome, and the performance degradation or abnormal state of the computing power cluster can be accurately perceived, ensuring that the collected trace data corresponds to the critical or abnormal moment of system operation, which greatly enhances the intrinsic value of the data.

[0176] In addition, by introducing feature fusion and intelligent anomaly detection models based on attention mechanisms, it is possible to automatically learn and identify true anomaly patterns from complex, nonlinear, multi-dimensional performance fluctuations, avoiding the problems of low accuracy and poor adaptability caused by single thresholds or simple rule triggers, and realizing intelligent decision-making for trace collection.

[0177] Based on the same inventive concept, embodiments of this specification also provide an electronic device corresponding to the above method, wherein... Figure 4 The embodiments provided in this specification correspond to Figure 1 A schematic diagram of the structure of a data acquisition device based on multidimensional intelligent evaluation.

[0178] like Figure 4 As shown, device 400 may include: At least one processor 410; and, Memory 430 communicatively connected to the at least one processor; wherein, The memory 430 stores instructions 420 that can be executed by the at least one processor 410, the instructions being executed by the at least one processor 410 to enable the at least one processor 410 to: The collected multi-dimensional performance indicators were quantified to obtain their respective health scores; Based on the health score, construct the feature vector corresponding to each of the multi-dimensional performance indicators; After standardizing and preprocessing the feature vectors separately, the preprocessed feature vectors are weighted and fused using learnable attention weights to obtain a fused feature vector. The probability of an anomaly is determined using the intelligent anomaly detection model and the fused feature vector. When the abnormal probability exceeds the adaptive trigger threshold, the execution trajectory is collected.

[0179] Other functions of the processor can be found in the above embodiments, and will not be repeated here.

[0180] Based on the same inventive concept, embodiments of this specification also provide a computer-readable storage medium, including a program for use in conjunction with an electronic device, the program being executable by a processor to perform the following steps: The collected multi-dimensional performance indicators were quantified to obtain their respective health scores; Based on the health score, construct the feature vector corresponding to each of the multi-dimensional performance indicators; After standardizing and preprocessing the feature vectors separately, the preprocessed feature vectors are weighted and fused using learnable attention weights to obtain a fused feature vector. The probability of an anomaly is determined using the intelligent anomaly detection model and the fused feature vector. When the abnormal probability exceeds the adaptive trigger threshold, the execution trajectory is collected.

[0181] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0182] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0183] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0184] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0185] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0186] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0187] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0188] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0189] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0190] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0191] This application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0192] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0193] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of this application should be included within the scope of protection of the claims of this application.

Claims

1. A data collection method based on multi-dimensional intelligent assessment, characterized in that, The data acquisition method based on multidimensional intelligent evaluation includes: Collect multi-dimensional performance metrics, including computing power, network communication, and memory storage metrics; The performance metrics, including computing power, network communication, and memory storage metrics, are quantified to obtain computing power health scores, network health scores, and memory health scores. Specifically, the quantification of computing power metrics includes: extracting the first principal component score from raw metric data, including GPU utilization, CPU utilization, GPU SM streaming multiprocessor utilization, and computing queue depth, and performing a linear mapping based on the quantiles of historical data to obtain the computing power health score. The quantification of network communication metrics includes: performing fuzzy set processing on collected network communication metrics, including bandwidth utilization, communication latency, packet loss rate, and communication error rate, and outputting a fuzzy set to obtain the network health score. The quantification of memory storage metrics includes: collecting memory storage metrics, including memory utilization, I / O throughput, paging frequency, and storage latency, normalizing each metric, and then performing a weighted average fusion to obtain the memory health score. Based on the computing power health score, the network health score, and the memory health score, construct feature vectors corresponding to each of the multi-dimensional performance indicators; After standardizing and preprocessing the feature vectors separately, the preprocessed feature vectors are weighted and fused using learnable attention weights to obtain a fused feature vector. The probability of an anomaly is determined using the intelligent anomaly detection model and the fused feature vector. When the anomaly probability exceeds the adaptive trigger threshold, multi-level trigger execution trajectory collection is initiated. The adaptive trigger threshold is a baseline threshold for triggering execution trajectory collection calculated based on the statistical value of the anomaly probability during historical normal operation periods, and dynamically adjusted by combining the baseline threshold with the real-time system load rate.

2. The method as described in claim 1, characterized in that, For computing power metrics, the computing power metrics are quantified to obtain the computing power health score, including: After standardizing and decomposing the original index data, the first principal component score is extracted using principal component analysis. The computing power health score is obtained by linearly mapping the quantiles of historical data.

3. The method as described in claim 2, characterized in that, The computing power health score is obtained by linearly mapping based on the quantiles of historical data, including: Obtain the historical monitoring dataset of the first principal component score; The historical monitoring datasets are arranged in ascending order; A preset first quantile is selected as the minimum quantile value, and a preset second quantile is selected as the maximum quantile value, wherein the sum of the preset first quantile and the preset second quantile is 1; Based on the minimum score and the maximum score, a linear mapping is performed on the computing power health score to obtain the computing power health score.

4. The method as described in claim 3, characterized in that, The computing power health score is linearly mapped according to the following formula. ； ； in, This represents the standardized score of the first principal component. The minimum score. The maximum score. The score for the first principal component. This indicates the calculation of the computing power health score.

5. The method as described in claim 1, characterized in that, For network communication metrics, the network communication metrics are quantified to obtain the network health score, including: The network communication indicators are processed using fuzzy set theory. Using a pre-defined fuzzy rule base, the rule strength of the fuzzy set corresponding to each of the network communication indicators is calculated; Based on the strength of the rule, output a fuzzy set; The fuzzy set is defuzzified to obtain the network health score.

6. The method as described in claim 5, characterized in that, The preset fuzzy rule base includes ideal network state, acceptable heavy load state, latency-sensitive state, serious network problem, and light load abnormal state.

7. The method as described in claim 1, characterized in that, For memory storage metrics, the memory storage metrics are quantified to obtain the memory health score, including: Each of the aforementioned memory storage metrics is normalized. The normalized memory storage metrics are weighted and averaged to obtain the memory health score.

8. The method as described in claim 1, characterized in that, After standardizing and preprocessing the feature vectors separately, the preprocessed feature vectors are weighted and fused using learnable attention weights to obtain a fused feature vector, including: Each of the aforementioned feature vectors is subjected to data normalization and time series detrending processing respectively; For the preprocessed feature vectors corresponding to the computing power index, the network communication index, and the memory storage index, calculate the attention score for each preprocessed feature vector. Based on the attention score, determine the attention weight corresponding to each preprocessed feature vector; The preprocessed feature vectors are weighted and fused based on the attention weights to obtain the fused feature vector.

9. The method as described in claim 8, characterized in that, After obtaining the fused feature vector, the method further includes: The fused feature vector is subjected to linear dimensionality reduction.

10. The method as described in claim 1, characterized in that, Using the intelligent anomaly detection model and the fused feature vector, the anomaly probability is determined, including: The intelligent anomaly detection model is used to calculate the fused feature vector to obtain a probability value representing the degree of anomaly in the current state, and the probability value is the anomaly probability.

11. The method as described in claim 1, characterized in that, When the anomaly probability exceeds the adaptive trigger threshold, execution trajectory acquisition is triggered, including: The triggering conditions for collecting execution trajectories are multi-level, including at least one of the following methods: When the anomaly probability exceeds the adaptive trigger threshold, execution trajectory acquisition is triggered. When the probability of the anomaly continues to rise over multiple consecutive sampling periods, an early warning level collection is triggered.

12. A data acquisition system based on multidimensional intelligent evaluation, characterized in that, The data acquisition system based on multidimensional intelligent evaluation includes: The quantization module collects multi-dimensional performance indicators, including computing power, network communication, and memory storage metrics. It then quantifies these indicators to obtain computing power health scores, network health scores, and memory health scores. Specifically, the quantification of computing power includes extracting the first principal component score from raw data such as GPU utilization, CPU utilization, GPU SM streaming multiprocessor utilization, and computing queue depth, and performing a linear mapping based on the quantiles of historical data to obtain the computing power health score. The quantification of network communication metrics includes performing fuzzy set processing on collected network communication metrics such as bandwidth utilization, communication latency, packet loss rate, and communication error rate, outputting a fuzzy set to obtain the network health score. The quantification of memory storage metrics includes collecting memory usage, I / O throughput, paging frequency, and storage latency, normalizing each metric, and then performing a weighted average fusion to obtain the memory health score. The construction module constructs feature vectors corresponding to each of the multi-dimensional performance indicators based on the computing power health score, the network health score, and the memory health score; The fusion module performs standardized preprocessing on the feature vectors separately, and then performs weighted fusion on the preprocessed feature vectors using learnable attention weights to obtain a fused feature vector. The determination module uses the intelligent anomaly detection model and the fused feature vector to determine the anomaly probability; The triggering module triggers multi-level execution trajectory collection when the anomaly probability exceeds the adaptive triggering threshold. The adaptive triggering threshold is a baseline threshold for triggering execution trajectory collection calculated based on the statistical value of the anomaly probability during historical normal operation periods, and dynamically adjusted by combining the baseline threshold with the real-time system load rate.

13. An electronic device comprising at least one processor and a memory, the memory storing a program and configured to have at least one processor perform the following steps: Collect multi-dimensional performance metrics, including computing power, network communication, and memory storage metrics; The performance metrics, including computing power, network communication, and memory storage, were quantified to obtain computing power health scores, network health scores, and memory health scores. The quantification of the computing power indicators includes: extracting the first principal component score from the collected raw indicator data, including GPU utilization, CPU utilization, GPU SM streaming multiprocessor utilization, and computing queue depth, and performing a linear mapping based on the quantiles of historical data to obtain the computing power health score; the quantification of the network communication indicators includes: performing fuzzy set processing on the collected network communication indicators, including bandwidth utilization, communication latency, packet loss rate, and communication error rate, outputting a fuzzy set to obtain the network health score; the quantification of the memory storage indicators includes: collecting memory storage indicators, including memory utilization, I / O throughput, paging frequency, and storage latency, performing normalization processing on each, and then performing a weighted average fusion to obtain the memory health score. Based on the computing power health score, the network health score, and the memory health score, construct feature vectors corresponding to each of the multi-dimensional performance indicators; After standardizing and preprocessing the feature vectors separately, the preprocessed feature vectors are weighted and fused using learnable attention weights to obtain a fused feature vector. The probability of an anomaly is determined using the intelligent anomaly detection model and the fused feature vector. When the anomaly probability exceeds the adaptive trigger threshold, multi-level trigger execution trajectory collection is initiated. The adaptive trigger threshold is a baseline threshold for triggering execution trajectory collection calculated based on the statistical value of the anomaly probability during historical normal operation periods, and dynamically adjusted by combining the baseline threshold with the real-time system load rate.

Images