Method and device for creating intelligent decision data set based on log data, electronic equipment, storage medium and program product

Abstract

The invention provides a method and device for creating an intelligent decision data set based on log data, and relates to the technical field of cloud computing, cluster management, big data processing and artificial intelligence, and the method comprises the steps: collecting original log data from a server and a cluster, and achieving the collection and preprocessing of the log data; key information is extracted from the preprocessed log data, and log event analysis and structured processing are realized based on a self-attention mechanism; high-quality features are extracted and constructed from structured log data, and feature engineering and data enhancement are achieved; and combining the enhanced feature set with a predefined decision target to generate an intelligent decision data set for model training and evaluation, thereby realizing construction and evaluation of the intelligent decision data set. According to the method, high-quality data support is provided for the intelligent decision model, the system resource demand, the task execution time and the failure risk can be effectively predicted, the resource allocation efficiency is greatly optimized, and a solid foundation is provided for large-scale computing system resource management and scheduling.

Description

Technical Field

[0001] This application relates to the fields of cloud computing, cluster management, big data processing and artificial intelligence, and in particular to a method and apparatus for creating intelligent decision datasets based on log data. Background Technology

[0002] In large-scale server and cluster environments, resource management and scheduling have always been crucial issues that cannot be ignored in computing systems. With the rapid development of cloud computing and virtualization technologies and the ever-expanding demand for big data processing, resource allocation and scheduling in clusters have become even more complex.

[0003] In traditional resource management methods, administrators typically rely on static rules and human experience to adjust resource allocation strategies. However, the limitations of this approach are becoming increasingly apparent. Especially when facing varying workloads and system states, traditional methods fail to provide flexible adjustments and efficient optimization, leading to resource waste, task delays, and system performance bottlenecks.

[0004] Most existing resource allocation methods rely on static configuration or experience-based manual scheduling, which cannot effectively adapt to complex workloads and dynamically changing cluster states. For example, certain jobs, such as time-skewed jobs and data-skewed jobs, may cause severe performance degradation or even cluster downtime. However, traditional methods cannot detect these potential risks in a timely manner, thus affecting system stability and efficient resource utilization. Furthermore, with the widespread adoption of cloud computing and big data applications, the scale of servers and clusters has increased dramatically, and traditional rule-based resource allocation methods cannot efficiently handle large-scale and diverse workloads.

[0005] Server and cluster log data are core data sources for recording critical information such as system resource usage, job execution status, and error messages. This log data contains a wealth of valuable information, but due to its massive volume, complex format, and high noise levels, traditional analysis methods struggle to extract meaningful features for further decision support. Specifically, cluster and server logs include information on system load, CPU, memory, disk, and network resource usage, job start and end times, error logs, and scheduling logs. How to extract key information from this complex data and utilize it effectively has become a pressing issue in resource management. To improve resource allocation efficiency and system stability, increasing research and practice are focusing on extracting valuable information from massive log data to guide intelligent resource allocation.

[0006] In recent years, with the advancement of machine learning and data mining technologies, an increasing number of studies have attempted to improve the intelligence and automation of resource allocation through in-depth analysis of log data. By analyzing log data using machine learning algorithms, system load changes, abnormal events, and dependencies between jobs can be identified, thereby providing more accurate and real-time decision support for resource scheduling.

[0007] However, existing log data analysis methods still have many problems: First, most methods rely on manually designed features or static rules, lacking the ability to dynamically learn and adaptively adjust to the data; second, existing technologies mostly focus on a single resource type, such as analyzing only CPU or memory usage, ignoring the interrelationships and comprehensive requirements between different types of resources, making it difficult to achieve efficient resource allocation in complex systems; finally, due to the complexity of log data itself, how to effectively perform data preprocessing, feature extraction, and noise reduction remains a challenge, and existing dataset construction methods often struggle to handle the diversity and scale of log data, failing to provide sufficient support for intelligent decision-making.

[0008] In summary, how to construct a high-quality intelligent decision dataset to effectively extract key information from massive and diverse server and cluster log data and generate a decision dataset for intelligent resource allocation has become a major challenge in current research and application. Summary of the Invention

[0009] This application discloses a method and apparatus for creating intelligent decision datasets based on log data. The purpose is to provide a solution for creating intelligent decision datasets based on server and cluster log data, aiming to solve the problems of low resource allocation efficiency and poor system stability in the prior art, and to promote the intelligence and automation of resource management.

[0010] In a first aspect, this application discloses a method for creating an intelligent decision-making dataset based on log data, the method comprising: Raw log data is collected from servers and clusters, and then cleaned, transformed, and normalized to achieve log data collection and preprocessing; the preprocessing includes at least removing duplicate data, filling in missing values, and removing redundant information. Extract key information from preprocessed log data and structure it into a data format that is easy to analyze and utilize based on a self-attention mechanism to achieve log event parsing and structured processing; High-quality features are extracted and constructed from structured log data to achieve feature engineering and data augmentation; the feature engineering includes feature selection, feature transformation, and feature construction, and the data augmentation includes synthesizing minority class samples, data balancing, and random perturbation; The enhanced feature set is combined with a predefined decision objective to generate an intelligent decision dataset for model training and evaluation, thereby realizing the construction and evaluation of the intelligent decision dataset. The construction of the intelligent decision dataset includes dataset partitioning, label annotation, data consistency checking, and format conversion.

[0011] Optionally, the steps of collecting raw log data from servers and clusters, and cleaning, transforming, and normalizing it to achieve log data collection and preprocessing include: Based on the set time window Logs within a timestamp range are read from each log source according to the following expression. The log sources for data collection include at least operating systems, applications, and distributed systems. (1) in, Indicates within the time window Log entries for internal log data, Indicates the timestamp of a log entry. Indicates the current time.

[0012] Optionally, the steps of collecting raw log data from servers and clusters, and cleaning, transforming, and normalizing it to achieve log data collection and preprocessing include: Duplicate data removal is achieved by comparing the timestamps and contents of log entries in the log data. For handling missing values ​​in numeric fields, the mean imputation method shown in expression (2) is used to impute missing values. Process it; (2) in, This represents the number of samples with non-missing values. Indicates samples with non-missing values. This represents the average value of the field. If the log entry contains resource usage Satisfies expression (3) Then determine the data. For noise and remove, where, This indicates the resource usage at the current point in time. This indicates the maximum allowed change threshold.

[0013] Optionally, the step of extracting key information from the preprocessed log data and structuring it into a data format that is easy to analyze and utilize based on a self-attention mechanism, thereby realizing log event parsing and structured processing, includes: The text of log entries in the log data is parsed using the self-attention mechanism shown in expression (4). Based on the input log text sequence, different weights are assigned to each word or field; the log entries are then parsed. The text portion is converted into a word embedding sequence. And input it into the self-attention model shown in expression (5): (4) (5) By calculating attention scores Assign weights to each word to obtain a weighted representation of each field in the log entry: (6) in, Represents log entries Embedded representation, Indicates the embedding dimension. This represents the output representation of the self-attention mechanism. , , These represent matrices corresponding to the query, key, and value fields, respectively. , , These represent the weight matrices obtained from the learning process and corresponding to the query, key, and value fields, respectively.

[0014] Optionally, the step of extracting key information from the preprocessed log data and structuring it into a data format that is easy to analyze and utilize based on a self-attention mechanism, thereby realizing log event parsing and structured processing, includes: The text features are fused with the numerical features extracted from the logs according to the following expression to construct a multimodal joint representation; the numerical features include at least resource usage, job execution status, and error codes; (7) in, This represents the log text representation obtained through a self-attention mechanism. This represents the feature representation of numeric fields extracted from log text. This represents a hyperparameter that controls the fusion ratio of textual features and numerical features.

[0015] Optionally, the step of extracting key information from the preprocessed log data and structuring it into a data format that is easy to analyze and utilize based on a self-attention mechanism, thereby realizing log event parsing and structured processing, includes: An event dependency graph is introduced, and a graph neural network (GNN) model is constructed to analyze the temporal and causal relationships between log events in the following manner; Hypothetical event In time It happened, and with the event There are dependencies between events; a graph structure is constructed where nodes represent log events and edges represent dependencies between events; the dependencies between events are modeled using a Graph Convolutional Network (GCN), outputting an updated representation for each event: (8) in, For the event The updated version indicates that For the event The previous update indicated that... Representation and event The set of adjacent nodes, Represents the adjacency matrix of a graph. This is the activation function.

[0016] Optionally, the step of extracting key information from the preprocessed log data and structuring it into a data format that is easy to analyze and utilize based on a self-attention mechanism, thereby realizing log event parsing and structured processing, includes: After completing text parsing, feature fusion, and event dependency modeling of the logs, a structured representation of each log entry is generated: (9) Each log entry should include at least the following information: timestamp This represents the time information of the collected log events; text feature representation. This represents the log text representation extracted through a self-attention mechanism; combined with the time representation. , representing the log synthesis representation obtained by feature fusion from expression (7); event-dependent representation , representing the association information between events extracted from the event dependency graph model obtained by expression (8).

[0017] Optionally, the step of extracting and constructing high-quality features from structured log data to achieve feature engineering and data augmentation includes: Structured log data Transform into a matrix ,in, Indicates the number of log samples. Indicates the feature dimension; The original data is standardized according to expression (10) so that the mean of each feature is 0 and the standard deviation is 1: (10) in, This represents the mean of each feature. This represents the standard deviation of each feature; The encoder processes the normalized data according to expression (11). Mapping to a low-dimensional feature representation space: (11) in, Represents the encoder weight matrix; Represents the encoder bias vector; Indicates the activation function; Represents the hidden layer feature dimension, satisfying ; An attention mechanism is introduced based on the low-dimensional feature representation space Z. The importance of different features is weighted according to expressions (12) and (13) to automatically focus on key information: (12) (13) in, This is the attention weight matrix; This is the attention bias vector; This represents element-wise multiplication; The decoder will convert the weighted low-dimensional representation according to expression (14). Reconstructed to the original data format: (14) The training objective is determined according to expression (15), which is to minimize the mean square error between the original data and the reconstructed data: (15) Combining automatically extracted features with expert knowledge, it transforms manually designed features With the output characteristics of the auto encoder The feature set is formed by fusion according to expression (16). : (16) in, For weight fusion.

[0018] Optionally, the step of extracting and constructing high-quality features from structured log data to achieve feature engineering and data augmentation includes: Let the minority class sample set be ,from Randomly select a sample ; The sample's position in the minority class set is calculated based on Euclidean distance. The nearest neighbor sample set ; Selected samples Nearest neighbor samples Synthetic samples are generated through random interpolation. : (17) The generated synthetic sample The enhanced feature set is formed by merging it with the original feature set F. (18) in, is a random number uniformly distributed in the interval [0, 1]. This represents the newly generated set of samples.

[0019] Optionally, the step of combining the enhanced feature set with a predefined decision objective to generate an intelligent decision dataset for model training and evaluation, thereby realizing the construction and evaluation of the intelligent decision dataset, includes: Define the set of decision target labels as follows: (19) in, Indicates the first The decision objectives corresponding to each sample; Enhanced feature set With tags Combined, a complete dataset is formed. : (20) The dataset is divided into training, validation, and test sets according to a set ratio; the dataset uses CSV and TFRecord formats.

[0020] Optionally, the step of combining the enhanced feature set with a predefined decision objective to generate an intelligent decision dataset for model training and evaluation, thereby realizing the construction and evaluation of the intelligent decision dataset, includes: For regression tasks: using linear regression or XGBoost regression models, the mean squared error (MSE) is evaluated according to expression (21) and the mean absolute error (MAE) is evaluated according to expression (22): (twenty one) (twenty two) in, The model's predicted values ​​are represented by MSE, which represents the mean squared deviation between the predicted and actual values; and MAE, which represents the mean absolute deviation of the predictions. For classification tasks: select logistic regression, random forest, or XGBoost classifier models and evaluate them according to the accuracy (23), precision (24), recall (25), and F1 score (26): (twenty three) (twenty four) (25) (26) Where TP represents the number of true positive samples, TN represents the number of true negative samples, FP represents the number of false positive samples, and FN represents the number of false negative samples.

[0021] Secondly, this application discloses an apparatus for creating an intelligent decision-making dataset based on log data, the apparatus comprising: The log data acquisition and preprocessing module is used to collect raw log data from servers and clusters, and to clean, transform, and normalize it to achieve log data acquisition and preprocessing; the preprocessing includes at least removing duplicate data, filling in missing values, and removing redundant information. The log time parsing and structuring module is used to extract key information from preprocessed log data and structure it into a data form that is easy to analyze and utilize based on a self-attention mechanism, thereby realizing log event parsing and structured processing. The feature engineering and data augmentation module is used to extract and construct high-quality features from structured log data, and to implement feature engineering and data augmentation. The feature engineering includes feature selection, feature transformation and feature construction, and the data augmentation includes synthesizing minority class samples, data balancing and random perturbation. The intelligent decision tree set construction and evaluation module is used to combine the enhanced feature set with the predefined decision objective to generate an intelligent decision dataset for model training and evaluation, thereby realizing the construction and evaluation of the intelligent decision dataset. The intelligent decision dataset construction includes dataset partitioning, label annotation, data consistency checking and format conversion.

[0022] Thirdly, this application discloses an electronic device comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to perform the method as described in any of the preceding aspects.

[0023] Fourthly, this application discloses a non-transitory computer-readable storage medium in which, when the instructions in the storage medium are executed by a processor of an electronic device, enable the electronic device to perform the methods described in any of the preceding aspects.

[0024] Fifthly, this application discloses a computer program product in which, when the instructions in the computer program product are executed by a processor of an electronic device, the electronic device is enabled to perform the method described in any of the preceding aspects.

[0025] The technical solution provided in this application may include the following beneficial effects: This application provides a scheme for constructing an intelligent decision dataset for optimizing the allocation of computing power and storage resources based on server and cluster log data. The scheme extracts key information from massive and diverse server and cluster logs, and generates a high-quality intelligent decision dataset for improving resource allocation efficiency and system stability through structured processing and feature engineering.

[0026] By extracting key information from complex and massive server and cluster log data and performing structured processing, this application can construct a high-quality intelligent decision-making dataset, thereby providing a more accurate and scientific basis for the optimized allocation of computing and storage resources. The solution provided by this application not only improves the efficiency of resource allocation, avoiding resource waste and shortages, but also enhances system stability, promptly detects potential anomalies and bottlenecks, and prevents system crashes or performance degradation. Furthermore, through intelligent analysis and feature engineering, this application can support various resource allocation decision objectives, such as predicting job execution time, assessing job failure risks, and recommending the optimal resource configuration scheme, providing flexible decision support for dynamically changing computing environments. Ultimately, the purpose of this application is to achieve an intelligent, automated, and efficient resource management method, optimize system performance, improve resource utilization, and ensure stable operation in large-scale cluster and server environments. Attached Figure Description

[0027] Figure 1 A flowchart illustrating the method for creating an intelligent decision dataset based on log data provided in this application; Figure 2 This is a flowchart of log data collection and preprocessing; Figure 3 This is a flowchart of log event parsing and structured model; Figure 4 This is a flowchart of feature engineering and data augmentation; Figure 5 A structural diagram of the apparatus for creating intelligent decision datasets based on log data provided in this application; Figure 6 A block diagram of an electronic device provided in this application; Figure 7 A block diagram of another electronic device provided in this application. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] To improve resource utilization efficiency and system stability, this application provides a method and apparatus for creating intelligent decision-making datasets based on log data. Through intelligent analysis and processing of log data, the allocation of computing and storage resources is optimized. The purpose of this application is to provide a solution for creating intelligent decision-making datasets based on server and cluster log data, aiming to solve the problems of low resource allocation efficiency and poor system stability in existing technologies, and to promote the intelligentization and automation of resource management.

[0030] The method for creating intelligent decision-making datasets based on log data, as provided in this application, is described below. Please refer to [link to relevant documentation]. Figures 1 to 4 .

[0031] Example 1 Reference Figure 1 This is a flowchart illustrating a method for creating an intelligent decision-making dataset based on log data, as provided in this application. This method can be applied to electronic devices, and specifically includes the following steps: Step S101: Collect raw log data from the server and cluster, and clean, transform and normalize it to realize log data collection and preprocessing; the preprocessing includes at least removing duplicate data, filling missing values ​​and removing redundant information to ensure data quality and reduce errors in data processing.

[0032] Step S102: Extract key information from the preprocessed log data and structure it into a data format that is easy to analyze and utilize based on the self-attention mechanism, thereby realizing log event parsing and structured processing.

[0033] It should be noted that this step involves classifying and parsing the log content, transforming different types of logs, such as resource usage, job execution status, and error messages, into structured data to provide clear input for subsequent analysis and decision-making.

[0034] Step S103: Extract and construct high-quality features from structured log data to provide accurate data support for the intelligent decision-making model, achieving feature engineering and data augmentation. Feature engineering includes feature selection, feature transformation, and feature construction to ensure data representativeness and discriminative power. Data augmentation includes synthesizing minority class samples, data balancing, and random perturbation. It should be noted that data augmentation techniques expand the scale and diversity of the dataset, improving the model's generalization ability and robustness.

[0035] Step S104: Combine the enhanced feature set with the predefined decision objective to generate an intelligent decision dataset for model training and evaluation, thus realizing the construction and evaluation of the intelligent decision dataset. The construction of the intelligent decision dataset includes dataset partitioning, label annotation, data consistency checks, and format conversion to ensure data integrity and labeling accuracy. It should be noted that the dataset quality is evaluated through statistical analysis and model validation to ensure that the dataset provides reliable support for the intelligent decision system.

[0036] Step S101 mainly includes two operations: log data collection and preprocessing. Log data collection includes multiple sources such as operating systems, applications, and distributed systems. When collecting log files, it is necessary to ensure the timeliness and completeness of the logs. Specifically, in step S101, data collection can be performed as follows: Based on the set time window Logs within a timestamp range are read from each log source according to the following expression. The log sources for data collection include at least operating systems, applications, and distributed systems. (1) in, Indicates within the time window Log entries for internal log data, Indicates the timestamp of a log entry. Indicates the current time.

[0037] Furthermore, in step S101, preprocessing can be performed as follows: Duplicate data removal is achieved by comparing the timestamps and contents of log entries in the log data. For handling missing values ​​in numeric fields, the mean imputation method shown in expression (2) is used to impute missing values. Process it; (2) in, This represents the number of samples with non-missing values. Indicates samples with non-missing values. This represents the average value of the field. After handling missing values, noise removal is performed. Specifically, if the log entry contains resource usage... Satisfies expression (3) Then determine the data. For noise and remove, where, This indicates the resource usage at the current point in time. This indicates the maximum allowed change threshold.

[0038] Furthermore, noise removal is achieved by identifying and removing outliers by calculating fluctuations in resource usage. If the resource usage in the log entries... Satisfies expression (3) Then determine the data. For noise and remove, where, This indicates the resource usage at the current point in time. This indicates the predetermined threshold, which is the allowed threshold for variation.

[0039] Through the above calculations, a weighted representation of each field in the log entry is obtained, providing contextual information for subsequent steps. The core innovation of this step lies in automatically capturing the dependencies between different fields through a self-attention mechanism, enabling log parsing to move beyond fixed rules and dynamically adjust the parsing strategy based on the context.

[0040] After collecting and preprocessing the data, it needs to be parsed and structured. Traditional log parsing and structuring often relies on static rules and patterns to extract key information. However, with the increasing diversity and complexity of log data, simple rule and pattern matching methods are no longer sufficient. To address this, we introduce a deep learning model based on a self-attention mechanism to parse and structure log data. This model can automatically learn and capture complex patterns in log events, providing more efficient parsing and structuring capabilities.

[0041] In step S102, log event parsing and structuring are performed as follows: The text of log entries in the log data is parsed using the self-attention mechanism shown in expression (4). Based on the input log text sequence, different weights are assigned to each word or field. The log entries obtained in step S101 are then parsed. The text portion is converted into a word embedding sequence. And input it into the self-attention model shown in expression (5): (4) (5) By calculating attention scores Assign weights to each word to obtain a weighted representation of each field in the log entry. : (6) in, Represents log entries Embedded representation, Indicates the embedding dimension. This represents the output representation of the self-attention mechanism. , , These represent matrices corresponding to the query, key, and value fields, respectively. , , These represent the learned weight matrices corresponding to the query, key, and value fields, respectively. It should be noted that... , , This indicates the importance of each field.

[0042] In one scenario, key information can be extracted from preprocessed log data in the following way, and then structured into a data format that is easy to analyze and utilize based on a self-attention mechanism, thereby achieving log event parsing and structured processing: After extracting the contextual information from the log text, we need to fuse the text features with the numerical features extracted from the log according to the following expression to construct a multimodal joint representation; the numerical features include at least resource usage, job execution status, and error codes; (7) in, This represents the log text representation obtained through a self-attention mechanism. This represents the feature representation of numeric fields extracted from log text. This represents a hyperparameter that controls the fusion ratio of textual features and numerical features.

[0043] It should be noted that this fusion step can effectively combine textual and numerical features to obtain a comprehensive log event representation. This representation contains rich contextual information and the dynamic characteristics of numerical data, giving it stronger expressive power. This representation will serve as the input feature for the next step of event-dependent modeling.

[0044] It's important to note that log data is often event-driven, and these events may have temporal or causal relationships. For example, the failure of one job may cause delays or failures in the execution of subsequent jobs. To capture these complex dependencies, we introduce an event dependency graph. By constructing a graph neural network (GNN) model, we analyze the temporal and causal relationships between log events in the following manner; Hypothetical event In time It happened, and with the event There are dependencies between events; a graph structure is constructed where nodes represent log events and edges represent dependencies between events; the dependencies between events are modeled using a Graph Convolutional Network (GCN), outputting an updated representation for each event: (8) in, For the event The updated version indicates that For the event The previous update indicated that... Representation and event The set of adjacent nodes, The adjacency matrix of a graph (representing events) and (dependencies between them) This is the activation function.

[0045] It should be noted that this step can effectively model the dependencies between events and identify the upstream and downstream impacts of events, thereby providing key information for subsequent decision-making and resource allocation.

[0046] Furthermore, the steps of extracting key information from the preprocessed log data and structuring it into a data format that is easy to analyze and utilize based on a self-attention mechanism, thus realizing log event parsing and structured processing, also include: After completing text parsing, feature fusion, and event dependency modeling of the logs, a structured representation of each log entry is generated: (9) Each log entry should include at least the following information: Timestamp That is, the time information of the log events collected in step S101. Text feature representation That is, the log text representation extracted through a self-attention mechanism; integrated time representation. That is, the log comprehensive representation obtained by feature fusion from expression (7); event-dependent representation This refers to the association information between events extracted from the event dependency graph model, obtained by expression (8). It should be noted that the final structured log data... It will be stored in a structured data table, providing a convenient way to access and process features and data augmentation in subsequent step S103.

[0047] Step S103 mainly includes two steps: feature engineering and data augmentation. Based on the structured log data obtained in step S102, high-quality intelligent decision features are automatically extracted and constructed. Data augmentation techniques are then used to expand the dataset's size and diversity, thereby improving the generalization ability and robustness of the subsequent decision model. Specifically, the feature engineering and data augmentation steps can be implemented as follows: Structured log data Transform into a matrix ,in, Indicates the number of log samples. These represent feature dimensions, including CPU utilization, memory usage, disk I / O, network latency, job execution time, and error codes.

[0048] To automatically extract deep features, an attention-enhanced autoencoder (AAE) is used to capture latent patterns from massive and diverse log data. First, the original data is standardized according to expression (10) to make the mean of each feature 0 and the standard deviation 1, eliminating differences in feature dimensions and improving the stability of model training. (10) in, This represents the mean of each feature. This represents the standard deviation of each feature; The encoder processes the normalized data according to expression (11). Mapping to a low-dimensional feature representation space: (11) in, Represents the encoder weight matrix; Represents the encoder bias vector; Indicates the activation function; Represents the hidden layer feature dimension, satisfying ; An attention mechanism is introduced based on the low-dimensional feature representation space Z. The importance of different features is weighted according to expressions (12) and (13) to automatically focus on key information: (12) (13) in, This is the attention weight matrix; This is the attention bias vector; This represents element-wise multiplication; To ensure that the extracted features fully preserve the information of the original data, the decoder calculates the weighted low-dimensional representation according to expression (14). Reconstructed to the original data format: (14) The training objective is determined according to expression (15), which is to minimize the mean square error between the original data and the reconstructed data: (15) Combining automatically extracted features with expert knowledge, it transforms manually designed features With the output characteristics of the auto encoder The feature set is formed by fusion according to expression (16). : (16) in, To determine the optimal weights, cross-validation can be used.

[0049] To address the potential class imbalance issue in log data and to further expand the dataset size and improve the robustness of model training, we employ data augmentation techniques to increase the number of training samples and balance the data distribution, thereby enhancing the model's robustness and generalization ability. We utilize Synthetic Minority Oversampling Technique (SMOTE) to resolve the class imbalance problem in log data and expand the sample size to include minority class samples such as abnormal events and fault records.

[0050] In step S103, the step of extracting and constructing high-quality features from structured log data to achieve feature engineering and data augmentation includes: Let the minority class sample set be ,from Randomly select a sample ; The sample's position in the minority class set is calculated based on Euclidean distance. The nearest neighbor sample set ; Selected samples Nearest neighbor samples Synthetic samples are generated through random interpolation. : (17) The generated synthetic sample The enhanced feature set is formed by merging it with the original feature set F. (18) in, is a random number uniformly distributed in the interval [0, 1]. This represents the newly generated set of samples.

[0051] Step S104 includes two steps: constructing and evaluating the intelligent decision-making dataset. The enhanced feature set is combined with the decision target label to construct a high-quality intelligent decision-making dataset. The quality and effectiveness of the dataset are evaluated to ensure that the dataset supports the training and deployment of the intelligent decision-making model.

[0052] In step S104, the intelligent decision dataset can be constructed in the following manner: Define the set of decision target labels as follows: (19) in, Indicates the first Each sample corresponds to a decision objective, such as job execution time, failure risk, and resource requirements. It should be noted that the specific type of label depends on the nature of the task: if the task is a regression problem, such as predicting execution time, the label... It's a continuous numerical value; if it's a classification problem, such as determining whether an assignment will fail, the label... It is a discrete category, such as "success" or "failure".

[0053] Enhanced feature set With tags Combined, a complete dataset is formed. : (20) The training set, validation set, and test set are divided according to a set ratio. In one case, the training set, validation set, and test set can be divided in a 7:2:1 ratio. This division method can ensure that the model can fully learn the data features during the training process, while the generalization ability and performance of the model can be evaluated through the validation set and test set.

[0054] The dataset is available in CSV and TFRecord formats. It's worth noting that CSV format is used for general machine learning libraries, while TFRecord format is used for TensorFlow and offers more efficient data reading and processing capabilities. The dataset can also be converted to HDF5, Parquet, and other formats as needed.

[0055] It should be noted that data quality is one of the key factors affecting model performance. After the dataset is built, a comprehensive evaluation of data quality should be conducted, including completeness, accuracy, consistency, and timeliness.

[0056] Completeness: Checking for missing values, duplicate data, or outliers in the dataset. After preprocessing in S101 and structuring in S102, the original log data has had some noise and redundant information removed, but some missing or outliers may still exist. After feature engineering and data augmentation in S103, the intelligent decision dataset will further fill in missing values ​​and expand the minority class data through synthetic samples, thereby improving overall completeness.

[0057] Accuracy: Assessing the true extent to which labels and features are reflected in the data. Raw log data may contain incomplete records or errors; after preprocessing and structuring, the information is presented more clearly. The intelligent decision-making dataset further integrates deep features and expert-designed features. Its accuracy can be verified through manual sampling or preliminary model predictions (such as MSE, MAE for regression tasks or precision, recall for classification tasks), and compared with the original data. Typically, the latter shows higher accuracy.

[0058] Consistency: Ensuring consistency in data format, units, and data types. The structured log data extracted through the self-attention mechanism in stage S102 has a unified format, while the standardization processing and autoencoder extraction in stage S103 further unify the representation of each feature. Compared to the original log data, the inconsistency between different data sources is significantly reduced, resulting in higher consistency in the intelligent decision-making dataset.

[0059] Timeliness: Check whether the data can reflect the latest status of the system in a timely manner. Raw log data may have collection delays, but after real-time or scheduled collection, preprocessing, and structuring, the timeliness of the data is improved. The intelligent decision-making dataset should be continuously updated during the construction process to reflect the latest system status in subsequent decisions, and its timeliness is higher than that of unprocessed data.

[0060] After verifying the data quality, the next step is to evaluate the dataset's effectiveness in supporting decision-making tasks. This step is typically achieved by training a machine learning model and testing its performance. Specifically, the model can be trained on a training set first, and then its predictive ability can be evaluated on a test set. The chosen model and evaluation metrics will vary depending on the task type. In step S104, the intelligent decision-making dataset for model training and evaluation can be generated as follows, thus realizing the construction and evaluation of the intelligent decision-making dataset: For regression tasks: using linear regression or XGBoost regression models, the mean squared error (MSE) is evaluated according to expression (21) and the mean absolute error (MAE) is evaluated according to expression (22): (twenty one) (twenty two) in, The model's predicted values ​​are represented by MSE, which represents the mean squared deviation between the predicted and actual values; and MAE, which represents the mean absolute deviation of the predictions. For classification tasks: Select a logistic regression, random forest, or XGBoost classifier model and evaluate it according to expression (23) accuracy, expression (24) precision, expression (25) recall, and expression (26) F1 score: (twenty three) (twenty four) (25) (26) Where TP represents the number of true positive samples, TN represents the number of true negative samples, FP represents the number of false positive samples, and FN represents the number of false negative samples.

[0061] It should be noted that accuracy represents the proportion of correctly predicted samples; precision represents the proportion of samples predicted as positive that are actually positive; recall represents the proportion of samples that are actually positive that are correctly predicted; and the F1 score is the harmonic mean of precision and recall, and is applicable to imbalanced datasets.

[0062] In summary, this application provides a scheme for constructing an intelligent decision dataset for optimizing the allocation of computing and storage resources based on server and cluster log data. This scheme extracts key information from massive and diverse server and cluster logs, and generates a high-quality intelligent decision dataset for improving resource allocation efficiency and system stability through structured processing and feature engineering.

[0063] By extracting key information from complex and massive server and cluster log data and performing structured processing, this application can construct a high-quality intelligent decision-making dataset, thereby providing a more accurate and scientific basis for the optimized allocation of computing and storage resources. The solution provided by this application not only improves the efficiency of resource allocation, avoiding resource waste and shortages, but also enhances system stability, promptly detects potential anomalies and bottlenecks, and prevents system crashes or performance degradation. Furthermore, through intelligent analysis and feature engineering, this application can support various resource allocation decision objectives, such as predicting job execution time, assessing job failure risks, and recommending the optimal resource configuration scheme, providing flexible decision support for dynamically changing computing environments. Ultimately, the purpose of this application is to achieve an intelligent, automated, and efficient resource management method, optimize system performance, improve resource utilization, and ensure stable operation in large-scale cluster and server environments.

[0064] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions involved are not necessarily required by this application.

[0065] The following section provides a detailed explanation of the method for creating intelligent decision datasets based on log data provided in this application, using a specific example.

[0066] I. The following describes the implementation environment of this application. (1) Hardware environment Server: Inspur NF5280M5, configured with Intel Xeon Gold 6240 CPU @ 2.60GHz, 18 cores, 2 x 36 physical cores, 16GB RAM, 960GB system disk, and 1.92TB*2 SCRADIUS disks.

[0067] GPU: Tesla V100 * 2.

[0068] Storage devices: Huawei OceanStor 9000 series, including HWdata9, HWdata10, HWdata11, etc., with total capacity ranging from 1.865 PB to 9.944 PB.

[0069] (2) Software environment The operating system is Linux; the programming language is Python 3.8+; the deep learning framework is PyTorch 1.10; the machine learning libraries include Scikit-learn, Pandas, and NumPy; and the data processing libraries are Pandas and NumPy.

[0070] II. The following is an explanation of data preparation. Log data is collected from servers and clusters in actual operation, including system resource usage (CPU, memory, disk I / O, network latency, etc.), job execution status (start time, end time, failure information, etc.), error logs, and scheduling logs. Data sources include f277-f281 clusters, f282 clusters, and f283-f286 clusters.

[0071] The collected log data was cleaned to remove outliers and noise, fill in missing values, and normalize to convert the data to the same scale for easier subsequent analysis. Furthermore, the data was divided into training, validation, and test sets in a ratio of 7:2:1.

[0072] III. Explanation of the specific steps in this application. Step 1: Log Data Collection and Preprocessing Log data is collected periodically from multiple cluster nodes, with a collection window of 5 minutes to ensure data timeliness and completeness. After collection, the Pandas library is used to clean the collected log data, removing duplicate logs. By comparing the timestamps and content of log entries, if two logs have the same timestamp and identical content, one of them is discarded. For missing values, methods such as mean imputation or forward imputation are used, for example, using the mean of the field for imputation. By calculating the fluctuation of resource usage, outliers are identified and removed. If the resource usage value in a log entry exceeds a predetermined threshold, such as CPU utilization exceeding 100%, the data is considered noise and deleted. Finally, the cleaned log data is standardized in format, and redundant information is removed to lay the foundation for subsequent feature extraction. Log Data Collection and Preprocessing Step 2: Log Event Parsing and Structuring After converting log entries into word embeddings, they are input into a self-attention model. The self-attention mechanism parses the log text, assigning different weights to each word or field to capture their correlations. Attention scores are calculated to obtain a weighted representation of each field in the log entry, providing contextual information for subsequent steps. Next, the contextual information extracted from the log text is fused with numerical features (such as resource usage and job execution status) to construct a multimodal joint representation. A specific formula combines textual and numerical features to obtain a comprehensive log event representation. Then, an event dependency graph is constructed. A graph neural network (GNN) model analyzes the temporal and causal relationships between log events. Assuming events occur at a certain time and have dependencies on other events, a graph convolutional network (GCN) is used to model these dependencies, obtaining an updated representation for each event. Finally, a structured representation of each log entry is generated, containing timestamps, textual feature representations, numerical feature representations, and event dependency representations. This structured log data is stored in a structured data table for subsequent analysis and modeling.

[0073] Step 3: Feature Engineering and Data Augmentation Structured log data is transformed into matrix form, including features such as CPU utilization, memory usage, disk I / O, network latency, job execution time, and error codes. An attention-based autoencoder (AAE) is then employed to capture latent patterns from the massive log data. The original data is standardized so that each feature has a mean of 0 and a standard deviation of 1, eliminating differences in feature dimensions and improving the stability of model training. Based on the low-dimensional representation, an attention mechanism is introduced to weight the importance of different features, automatically focusing on key information. Manually designed features are then fused with the autoencoder output features to form the final feature set. To address potential class imbalance in log data, Synthetic Minority Oversampling Technique (SMOTE) is used to augment minority class samples. For example, for minority class samples such as abnormal events and fault records, new synthetic samples are generated through random interpolation. These synthetic samples are then merged with the original feature set to form an enhanced feature set, improving the robustness and generalization ability of model training.

[0074] Step 4: Construction and Evaluation of Intelligent Decision-Making Dataset Define a set of decision target labels, such as job execution time, failure risk, and resource requirements. Depending on the task nature, define the labels as continuous numerical values ​​(for regression tasks) or discrete categories (for classification tasks). Combine the enhanced feature set with the labels to form a complete dataset, and divide it into training, validation, and test sets in a 7:2:1 ratio. Convert the dataset to a format suitable for training machine learning models. Conduct a comprehensive evaluation of data quality, including completeness, accuracy, consistency, and timeliness. Evaluate the dataset's effectiveness in supporting decision tasks by training machine learning models and testing their performance. For regression tasks (such as predicting job execution time), use linear regression or XGBoost regression models, with evaluation metrics including mean squared error (MSE) and mean absolute error (MAE). For classification tasks (such as determining job failure risk), select models such as logistic regression, random forest, or XGBoost classifiers, with evaluation metrics including accuracy, precision, recall, and F1 score.

[0075] IV. Explanation of Experimental Results This embodiment achieved high model performance on the test set, with the following specific metrics: (1) Regression task (predicting job execution time): Mean squared error (MSE): 0.05 Mean Absolute Error (MAE): 0.2 (2) Classify tasks (assess the risk of task failure): Accuracy: 95% or higher Accuracy: 93% or higher Recall rate: over 94% F1 score: 93% or higher As can be seen from the dataset quality assessment results, the intelligent decision-making dataset, after preprocessing, structuring, feature engineering, and data augmentation, is superior to the original log data in terms of completeness, accuracy, consistency, and timeliness, and can provide high-quality data support for intelligent decision-making models.

[0076] In summary, this embodiment details how to use the method proposed in this application to create an intelligent decision-making dataset based on server and cluster log data, and how to process and analyze log data in a real-world cluster environment. Experimental results show that this method can effectively extract key information from log data, generate a high-quality intelligent decision-making dataset, and provide reliable data support for resource allocation optimization and system stability improvement, demonstrating promising application prospects.

[0077] It should be noted that the specific parameters used in this embodiment, such as the hidden layer dimension of the autoencoder and the oversampling ratio of SMOTE, are merely examples. In actual applications, adjustments need to be made based on the specific dataset and application scenario. During actual deployment, factors such as data volume and hardware resources need to be considered, and optimizations should be made for specific situations to achieve the best performance. This embodiment only uses a portion of the cluster log data; using a larger dataset can further improve the model's generalization ability and robustness.

[0078] Example 2 Reference Figure 5 This is a structural diagram of an apparatus for creating an intelligent decision-making dataset based on log data, as provided in this application. The apparatus includes: The log data acquisition and preprocessing module 210 is used to collect raw log data from servers and clusters, and to clean, transform and normalize it to realize log data acquisition and preprocessing; the preprocessing includes at least removing duplicate data, filling in missing values ​​and removing redundant information. The log time parsing and structuring module 220 is used to extract key information from preprocessed log data and structure it into a data form that is easy to analyze and use based on a self-attention mechanism, thereby realizing log event parsing and structured processing. The feature engineering and data augmentation module 230 is used to extract and construct high-quality features from structured log data to achieve feature engineering and data augmentation; the feature engineering includes feature selection, feature transformation and feature construction, and the data augmentation includes synthesizing minority class samples, data balancing and random perturbation; The intelligent decision tree set construction and evaluation module 240 is used to combine the enhanced feature set with the predefined decision objective to generate an intelligent decision dataset for model training and evaluation, thereby realizing the construction and evaluation of the intelligent decision dataset; the intelligent decision dataset construction includes dataset partitioning, label annotation, data consistency checking and format conversion.

[0079] This application constructs an intelligent decision-making dataset based on server and cluster log data, achieving intelligent and automated resource management and significantly improving system performance and resource utilization efficiency. Employing self-attention mechanisms and event-dependent modeling, it extracts multimodal and deep-level features from complex, massive log data, combining an attention-based autoencoder (AAE) and expert knowledge features to ensure the comprehensiveness and accuracy of feature representation. To address the class imbalance problem, this method introduces Synthetic Minority Oversampling Technique (SMOTE) to expand minority class samples such as abnormal events and fault records, improving the robustness and generalization ability of model training. During the data construction and evaluation phases, strict data cleaning, structuring, feature fusion, enhancement, and dataset partitioning strategies are implemented to ensure data integrity, accuracy, consistency, and timeliness. Ultimately, this method provides high-quality data support for intelligent decision-making models, effectively predicting system resource requirements, task execution time, and failure risks, significantly optimizing resource allocation efficiency, enhancing system stability, and providing a solid foundation for resource management and intelligent scheduling in large-scale computing systems.

[0080] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0081] Example 3 Optionally, this application also provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the various processes of the above method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0082] This application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0083] Figure 6 This application provides a block diagram of an electronic device 800. For example, the electronic device 800 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0084] Reference Figure 6The electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0085] Processing component 802 typically controls the overall operation of electronic device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0086] Memory 804 is configured to store various types of data to support the operation of device 800. Examples of this data include instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, images, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0087] Power supply component 806 provides power to various components of electronic device 800. Power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800.

[0088] Multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0089] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when electronic device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0090] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0091] Sensor assembly 814 includes one or more sensors for providing state assessments of various aspects of electronic device 800. For example, sensor assembly 814 may detect the on / off state of device 800, the relative positioning of components such as the display and keypad of electronic device 800, changes in position of electronic device 800 or a component of electronic device 800, the presence or absence of user contact with electronic device 800, orientation or acceleration / deceleration of electronic device 800, and temperature changes of electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0092] Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. Electronic device 800 can access wireless networks based on communication standards, such as WiFi, carrier networks (such as 2G, 3G, 4G, or 5G), or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast operation information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0093] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0094] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, which can be executed by a processor 820 of an electronic device 800 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0095] Example 4 Figure 7 A block diagram of another electronic device 1900 provided for this application. For example, electronic device 1900 may be provided as a server.

[0096] Reference Figure 7 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0097] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output (I / O) interface 1958. Electronic device 1900 can operate on an operating system stored in memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or similar.

[0098] Example 5 Fifthly, this application discloses a computer program product in which, when the instructions in the computer program product are executed by a processor of an electronic device, the electronic device is enabled to perform the method described in any of the preceding aspects.

[0099] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0100] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0101] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0102] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0103] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0104] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0105] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0106] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0107] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0108] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for creating an intelligent decision-making dataset based on log data, characterized in that, The method includes: Raw log data is collected from servers and clusters, and then cleaned, transformed, and normalized to achieve log data collection and preprocessing; the preprocessing includes at least removing duplicate data, filling in missing values, and removing redundant information. Extract key information from preprocessed log data and structure it into a data format that is easy to analyze and utilize based on a self-attention mechanism to achieve log event parsing and structured processing; High-quality features are extracted and constructed from structured log data to achieve feature engineering and data augmentation; the feature engineering includes feature selection, feature transformation, and feature construction, and the data augmentation includes synthesizing minority class samples, data balancing, and random perturbation; The enhanced feature set is combined with a predefined decision objective to generate an intelligent decision dataset for model training and evaluation, thereby realizing the construction and evaluation of the intelligent decision dataset. The construction of the intelligent decision dataset includes dataset partitioning, label annotation, data consistency checking, and format conversion.

2. The method for creating an intelligent decision-making dataset based on log data according to claim 1, characterized in that, The steps of collecting raw log data from servers and clusters, and then cleaning, transforming, and normalizing it to achieve log data collection and preprocessing include: Based on the set time window Logs within a timestamp range are read from each log source according to the following expression. The log sources for data collection include at least operating systems, applications, and distributed systems. （1） in, Indicates within the time window Log entries for internal log data, Indicates the timestamp of a log entry. Indicates the current time.

3. The method for creating an intelligent decision-making dataset based on log data according to claim 1, characterized in that, The steps of collecting raw log data from servers and clusters, and then cleaning, transforming, and normalizing it to achieve log data collection and preprocessing include: Duplicate data removal is achieved by comparing the timestamps and contents of log entries in the log data. For handling missing values ​​in numeric fields, the mean imputation method shown in expression (2) is used to impute missing values. Process it; (2) in, This represents the number of samples with non-missing values. Indicates samples with non-missing values. This represents the average value of the field. If the log entry contains resource usage Satisfies expression (3) Then determine the data. For noise and remove, where, This indicates the resource usage at the current point in time. This indicates the maximum allowed change threshold.

4. The method for creating an intelligent decision-making dataset based on log data according to claim 1, characterized in that, The steps of extracting key information from preprocessed log data and structuring it into a data format that is easy to analyze and utilize based on a self-attention mechanism to achieve log event parsing and structured processing include: The text of log entries in the log data is parsed using the self-attention mechanism shown in expression (4). Based on the input log text sequence, different weights are assigned to each word or field; the log entries are then parsed. The text portion is converted into a word embedding sequence. And input it into the self-attention model shown in expression (5): (4) (5) By calculating attention scores Assign weights to each word to obtain a weighted representation of each field in the log entry: (6) in, Represents log entries Embedded representation, Indicates the embedding dimension. This represents the output representation of the self-attention mechanism. , , These represent matrices corresponding to the query, key, and value fields, respectively. , , These represent the weight matrices obtained from the learning process and corresponding to the query, key, and value fields, respectively.

5. The method for creating an intelligent decision-making dataset based on log data according to claim 4, characterized in that, The steps of extracting key information from preprocessed log data and structuring it into a data format that is easy to analyze and utilize based on a self-attention mechanism to achieve log event parsing and structured processing include: The text features are fused with the numerical features extracted from the logs according to the following expression to construct a multimodal joint representation; the numerical features include at least resource usage, job execution status, and error codes; (7) in, This represents the log text representation obtained through a self-attention mechanism. This represents the feature representation of numeric fields extracted from log text. This represents a hyperparameter that controls the fusion ratio of textual features and numerical features.

6. The method for creating an intelligent decision-making dataset based on log data according to claim 5, characterized in that, The steps of extracting key information from preprocessed log data and structuring it into a data format that is easy to analyze and utilize based on a self-attention mechanism to achieve log event parsing and structured processing include: An event dependency graph is introduced, and a graph neural network (GNN) model is constructed to analyze the temporal and causal relationships between log events in the following manner; Hypothetical event In time It happened, and with the event There are dependencies between events; a graph structure is constructed where nodes represent log events and edges represent dependencies between events; the dependencies between events are modeled using a Graph Convolutional Network (GCN), outputting an updated representation for each event: (8) in, For the event The updated version indicates that For the event The previous update indicated that... Representation and event The set of adjacent nodes, Represents the adjacency matrix of a graph. This is the activation function.

7. The method for creating an intelligent decision-making dataset based on log data according to claim 6, characterized in that, The steps of extracting key information from preprocessed log data and structuring it into a data format that is easy to analyze and utilize based on a self-attention mechanism to achieve log event parsing and structured processing include: After completing text parsing, feature fusion, and event dependency modeling of the logs, a structured representation of each log entry is generated: (9) Each log entry should include at least the following information: timestamp This represents the time information of the collected log events; text feature representation. This represents the log text representation extracted through a self-attention mechanism; combined with the time representation. , representing the log synthesis representation obtained by feature fusion from expression (7); event-dependent representation , representing the association information between events extracted from the event dependency graph model obtained by expression (8).

8. The method for creating an intelligent decision-making dataset based on log data according to claim 1, characterized in that, The steps for extracting and constructing high-quality features from structured log data to achieve feature engineering and data augmentation include: Structured log data Transform into a matrix ,in, Indicates the number of log samples. Indicates the feature dimension; The original data is standardized according to expression (10) so that the mean of each feature is 0 and the standard deviation is 1: (10) in, This represents the mean of each feature. This represents the standard deviation of each feature; The encoder processes the normalized data according to expression (11). Mapping to a low-dimensional feature representation space: (11) in, Represents the encoder weight matrix; Represents the encoder bias vector; Indicates the activation function; Represents the hidden layer feature dimension, satisfying ; An attention mechanism is introduced based on the low-dimensional feature representation space Z. The importance of different features is weighted according to expressions (12) and (13) to automatically focus on key information: (12) (13) in, This is the attention weight matrix; This is the attention bias vector; This represents element-wise multiplication; The decoder will convert the weighted low-dimensional representation according to expression (14). Reconstructed to the original data format: (14) The training objective is determined according to expression (15), which is to minimize the mean square error between the original data and the reconstructed data: (15) Combining automatically extracted features with expert knowledge, it transforms manually designed features With the output characteristics of the auto encoder The feature set is formed by fusion according to expression (16). : (16) in, For weight fusion.

9. The method for creating an intelligent decision-making dataset based on log data according to claim 8, characterized in that, The steps for extracting and constructing high-quality features from structured log data to achieve feature engineering and data augmentation include: Let the minority class sample set be ,from Randomly select a sample ; The sample's position in the minority class set is calculated based on Euclidean distance. The nearest neighbor sample set ; Selected samples Nearest neighbor samples Synthetic samples are generated through random interpolation. : (17) The generated synthetic sample The enhanced feature set is formed by merging it with the original feature set F. (18) in, is a random number uniformly distributed in the interval [0, 1]. This represents the newly generated set of samples.

10. The method for creating an intelligent decision-making dataset based on log data according to claim 1, characterized in that, The steps of combining the enhanced feature set with a predefined decision objective to generate an intelligent decision dataset for model training and evaluation, and realizing the construction and evaluation of the intelligent decision dataset, include: Define the set of decision target labels as follows: (19) in, Indicates the first The decision objectives corresponding to each sample; Enhanced feature set With tags Combined, a complete dataset is formed. : (20) The dataset is divided into training, validation, and test sets according to a set ratio; the dataset uses CSV and TFRecord formats.

11. The method for creating an intelligent decision-making dataset based on log data according to claim 10, characterized in that, The steps of combining the enhanced feature set with a predefined decision objective to generate an intelligent decision dataset for model training and evaluation, and realizing the construction and evaluation of the intelligent decision dataset, include: For regression tasks: using linear regression or XGBoost regression models, the mean squared error (MSE) is evaluated according to expression (21) and the mean absolute error (MAE) is evaluated according to expression (22): (21) (22) in, The model's predicted values ​​are represented by MSE, which represents the mean squared deviation between the predicted and actual values; and MAE, which represents the mean absolute deviation of the predictions. For classification tasks: select logistic regression, random forest, or XGBoost classifier models and evaluate them according to the accuracy (23), precision (24), recall (25), and F1 score (26): (23) (24) (25) (26) Where TP represents the number of true positive samples, TN represents the number of true negative samples, FP represents the number of false positive samples, and FN represents the number of false negative samples.

12. An apparatus for creating an intelligent decision-making dataset based on log data, characterized in that, The device includes: The log data acquisition and preprocessing module is used to collect raw log data from servers and clusters, and to clean, transform, and normalize it to achieve log data acquisition and preprocessing; the preprocessing includes at least removing duplicate data, filling in missing values, and removing redundant information. The log time parsing and structuring module is used to extract key information from preprocessed log data and structure it into a data form that is easy to analyze and utilize based on a self-attention mechanism, thereby realizing log event parsing and structured processing. The feature engineering and data augmentation module is used to extract and construct high-quality features from structured log data, and to implement feature engineering and data augmentation. The feature engineering includes feature selection, feature transformation and feature construction, and the data augmentation includes synthesizing minority class samples, data balancing and random perturbation. The intelligent decision tree set construction and evaluation module is used to combine the enhanced feature set with the predefined decision objective to generate an intelligent decision dataset for model training and evaluation, thereby realizing the construction and evaluation of the intelligent decision dataset. The intelligent decision dataset construction includes dataset partitioning, label annotation, data consistency checking and format conversion.

13. An electronic device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 11.

15. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device implements the method as described in any one of claims 1 to 11.

Images