Big model-based ide scheduling platform task monitoring method and system
By adopting a task monitoring method based on the Transformer large model, the problems of low monitoring efficiency and insufficient accuracy of IDE scheduling platforms are solved, achieving efficient and accurate fault diagnosis and resource optimization, and improving the task monitoring capabilities of enterprise-level scheduling platforms.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SUNING COM CO LTD
- Filing Date
- 2026-02-06
- Publication Date
- 2026-06-05
AI Technical Summary
Existing enterprise integrated development environment (IDE) scheduling platforms suffer from inefficient task monitoring, insufficient accuracy, and significant resource waste. Traditional monitoring methods are unable to adapt to dynamic environments and complex faults.
A task monitoring method based on the Transformer large model is adopted. By collecting and preprocessing data in real time, key feature matrices are extracted, and the large model is used for fault analysis and early warning. Combined with distributed training, lightweight and incremental training techniques, fault root cause reports are generated.
It improves the adaptability and accuracy of monitoring, reduces operation and maintenance costs, enhances the efficiency and accuracy of fault diagnosis, and achieves efficient resource utilization.
Smart Images
Figure CN122152570A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the intersection of computer technology and artificial intelligence, and in particular to a task monitoring method and system for an IDE scheduling platform based on a large model. Background Technology
[0002] Current enterprise-level integrated development environment (IDE) scheduling platforms manage tasks on the order of hundreds of thousands (such as Spark and Hive tasks), with over 200,000 task instances running daily. Traditional monitoring relies on manual inspections and static rule engines, which have the following drawbacks:
[0003] (1) Inefficient: Humans cannot process large-scale tasks and system data in real time, and it is difficult to capture complex faults.
[0004] (2) Insufficient accuracy: Static rules are difficult to adapt to dynamic environments, resulting in high false alarm and false negative rates.
[0005] (3) Waste of resources: Full data analysis consumes a lot of computing resources and has a significant response delay. Summary of the Invention
[0006] The technical problem to be solved by this invention is to address the aforementioned shortcomings of the existing technology by resolving the issues of low efficiency, insufficient accuracy, and resource waste in task monitoring of the IDE scheduling platform.
[0007] To address the aforementioned technical problems, the present invention proposes a technical solution, in its first aspect, a task monitoring method for an IDE scheduling platform based on a large model, comprising the following steps:
[0008] Real-time collection of task execution data, system performance data, and task execution log text data from the IDE scheduling platform;
[0009] Key features such as task dependencies, resource consumption trends, and data partition size and variance are extracted from task execution data and system performance data to form a task feature matrix; each task corresponds to a task feature matrix.
[0010] Input the task feature matrix into the Transformer large model and output the timeout probability of the corresponding task; when the timeout probability exceeds the preset timeout threshold, trigger an alarm and extract the system performance data and task operation log text data of the task.
[0011] The abnormal fluctuation data segments in the system performance data of this task are marked as abnormal performance data. Fault keywords are extracted from the task operation log text data of this task. The abnormal performance data features and fault keyword features are fused to obtain the abnormal task feature matrix. The abnormal task feature matrix is input into the Transformer large model, and the root cause of the fault and its weight ratio are output using the attention weight mechanism.
[0012] Generate a root cause analysis report and send it to the user for optimization.
[0013] In one implementation, a ParameterServer architecture is used to perform distributed training on a large Transformer model.
[0014] In one implementation, model pruning and quantization techniques are used to lightweight the large Transformer model.
[0015] In one implementation, the Transformer large model is incrementally trained periodically, with the latest task data generated by the IDE scheduling platform used as training samples each time to dynamically optimize the weights of the Transformer large model.
[0016] In one implementation, when the execution time of a task exceeds a preset execution time threshold, the corresponding task is marked as a delayed task; key features of the delayed task are extracted first and then processed.
[0017] In one implementation, the data is preprocessed before key features are extracted; the preprocessing includes cleaning noise and removing data that deviates from the normal range.
[0018] In one implementation, the root cause analysis report of the failure is visualized through task heatmaps, resource usage curves, and real-time early warning lists.
[0019] Secondly, a task monitoring system for an IDE scheduling platform based on a large model is proposed, including:
[0020] The data acquisition module is used to collect task execution data, system performance data, and task execution log text data from the IDE scheduling platform in real time.
[0021] The data preprocessing module is used to extract key features from task execution data and system performance data, such as task dependencies, resource consumption trends, and data partition size and variance, as a task feature matrix; each task corresponds to a task feature matrix.
[0022] The intelligent monitoring and analysis module is used to input the task feature matrix into the Transformer large model and output the timeout probability of the corresponding task. When the timeout probability exceeds the preset timeout threshold, an alarm is triggered, and the system performance data and task operation log text data of the task are extracted at the same time.
[0023] The abnormal fluctuation data segments in the system performance data of this task are marked as abnormal performance data. Fault keywords are extracted from the task operation log text data of this task. The abnormal performance data features and fault keyword features are fused to obtain the abnormal task feature matrix. The abnormal task feature matrix is input into the Transformer large model, and the root cause of the fault and its weight ratio are output using the attention weight mechanism.
[0024] Generate a root cause analysis report and send it to the user for optimization.
[0025] In one implementation, it also includes a large model building and training module, which is used to perform distributed training of the Transformer large model using a ParameterServer architecture; to perform lightweight processing of the Transformer large model using model pruning and quantization techniques; and to perform incremental training of the Transformer large model on a regular basis, using the latest task data generated by the IDE scheduling platform as training samples each time, and dynamically optimizing the weights of the Transformer large model.
[0026] In one implementation, a visualization module is also included, which is used to visualize the root cause analysis report of the failure through task heatmaps, resource usage curves, and real-time warning lists.
[0027] The beneficial effects of this invention are:
[0028] 1. Traditional rule engines rely on manually preset static rules, which cannot adapt to dynamically changing task scenarios and complex fault modes. Applying the Transformer large model to scheduling platform monitoring overcomes the limitations of traditional rule engines and improves monitoring accuracy. The large model used in this invention can automatically capture complex correlation features in task and system data through end-to-end learning, without the need for manual intervention in rule design, significantly improving the adaptability and generalization ability of monitoring.
[0029] 2. By replacing full data analysis with real-time inference, data with a high probability of timeout in the future can be filtered out from massive task data. Then, the root causes of failures in the filtered data can be analyzed in depth, computing resources can be allocated reasonably, monitoring efficiency and accuracy can be improved, the completion rate of core tasks can be guaranteed, and maintenance labor costs can be reduced.
[0030] 3. By fusing structured system performance data with unstructured log data in a multimodal manner, and using the attention mechanism of a large model to mine the deep correlation between different modal data, the root cause of the fault can be directly output. This avoids the information loss and delay caused by splicing multiple modules in traditional methods, and improves the accuracy and efficiency of fault diagnosis.
[0031] 4. Integrating distributed training and lightweight deployment technologies to achieve a balance between high accuracy and low latency. Distributed training via the ParameterServer architecture supports large-scale data processing and training models with hundreds of billions of parameters, ensuring high model accuracy. Simultaneously, combining model pruning, quantization, and inference acceleration technologies enables lightweight model deployment, meeting the real-time monitoring needs of the scheduling platform and significantly improving monitoring efficiency. Attached Figure Description
[0032] The invention will now be further described with reference to the accompanying drawings.
[0033] Figure 1 This is a flowchart of a task monitoring method for an IDE scheduling platform based on a large model, according to an embodiment of the present invention.
[0034] Figure 2 This is an architecture diagram of a task monitoring system for an IDE scheduling platform based on a large model, according to an embodiment of the present invention.
[0035] Figure 3 This is a data flow diagram of a task monitoring system for an IDE scheduling platform based on a large model, according to an embodiment of the present invention. Detailed Implementation
[0036] like Figure 1-3 As shown, embodiments of the present invention provide a task monitoring method and system for an IDE scheduling platform based on a large model, wherein the method includes the following steps:
[0037] S1 collects task execution data, system performance data, and task execution log text data in real time from the IDE scheduling platform;
[0038] Specifically, the system integrates with the scheduling platform via API to collect real-time task execution data, including status, execution duration, and resource consumption; system performance data, including CPU, memory, and network metrics; task execution log text data; and historical fault logs. It supports second-level data retrieval and event-driven push notifications. All collected data is in JSON format to ensure data transmission compatibility and consistency, and it also features breakpoint resume functionality to prevent data loss. Kafka is used as a data caching middleware to support high-concurrency writes and reads.
[0039] In one implementation, the data is preprocessed before key features are extracted; preprocessing includes noise removal and data removal that deviates from the normal range. Specifically, noise removal includes heartbeat packets and debugging logs to improve data quality. A statistical outlier detection algorithm is used; in this embodiment, the Z-score algorithm is used to remove data that deviates from the normal range.
[0040] S2 extracts key features from task execution data and system performance data, including task dependencies, resource consumption trends, and data partition size and variance, as a task feature matrix; each task corresponds to a task feature matrix.
[0041] In this embodiment, the extracted key features, such as task dependencies and resource consumption trends, are enhanced with a temporal difference algorithm to construct a temporal feature matrix. A 30-second sliding window is used to extract the feature statistics (mean, maximum, minimum, and variance) within each window, ensuring the timeliness of the features. Task dependencies specifically include features such as task dependency chain complexity; resource consumption trends specifically include features such as node CPU load fluctuations; and the variance of data partition sizes is obtained by calculating the ratio of the standard deviation to the mean of each data partition size. The preprocessed task feature matrix (dimension N×M, where N is the time step and M is the number of features) is then serialized in Parquet format and stored with high compression ratio using the Snappy algorithm.
[0042] In one implementation, when the execution time of a task exceeds a preset execution time threshold, the corresponding task is marked as a delayed task; key features of delayed tasks are extracted first and then processed. In this embodiment, when the execution time of a task exceeds 1.5 times the average execution time of that task type, it is marked as delayed data.
[0043] S3: Input the task feature matrix into the Transformer large model and output the timeout probability of the corresponding task; when the timeout probability exceeds the preset timeout threshold, trigger an alarm and extract the system performance data and task running log text data of the task.
[0044] Specifically, the input is the preprocessed task feature matrix (dimension N×M, where N is the time step and M is the number of features).
[0045] Step 1: Normalize the feature matrix (using Min-Max normalization, range [0,1]);
[0046] Step 2: Input the normalized feature matrix into the large model, and perform feature extraction and context modeling through the Transformer encoder;
[0047] Step 3: Map the output of the timeout probability for the next hour through a fully connected layer (output range [0,1]);
[0048] Step 4: If the probability exceeds the threshold (default 80%, supports user-defined configuration, configuration range 50%-90%), trigger an alert and recommend optimization strategies (such as adjusting parallelism, optimizing data partitioning, expanding resources, etc., the strategy is dynamically matched according to the task type and the current system resource status).
[0049] The abnormal fluctuation data segments in the system performance data of this task are marked as abnormal performance data. Fault keywords are extracted from the task operation log text data of this task. The abnormal performance data features and fault keyword features are fused to obtain the abnormal task feature matrix. The abnormal task feature matrix is input into the Transformer large model, and the root cause of the fault and its weight ratio are output using the attention weight mechanism.
[0050] Specifically, the inputs are: system performance data time series and task execution log text data.
[0051] Step 1: Perform anomaly detection on system performance data (using the Isolation Forest algorithm) to identify abnormal fluctuation data segments;
[0052] Step 2: Perform text segmentation and keyword extraction (using the TF-IDF algorithm) on the task execution log to extract fault-related keywords;
[0053] Step 3: Fuse the abnormal performance data features with the log keyword features and input them into the large model for correlation analysis;
[0054] Step 4: Identify potential bottlenecks (such as insufficient network bandwidth causing data exchange delays, CPU overload causing task scheduling blockages, etc.).
[0055] Step 5: Output a root cause report, indicating the scope of impact (number of affected tasks, number of nodes involved) and remediation priority (based on a comprehensive assessment of business importance, scope of impact, and remediation difficulty, with priorities divided into P0 / P1 / P2 / P3). This embodiment will use the top 5 features with the highest attention weight as core fault association features to generate a fault root cause analysis report. Other numbers of fault root cause features can be selected as needed.
[0056] The above reasoning process uses batch processing (batch size 64) to improve processing efficiency.
[0057] Specifically, the Transformer large model has 12 layers, 16 heads, and 1024 hidden layer dimensions. It combines a multi-head attention mechanism to capture long-distance dependencies. Supervised learning is performed based on historical labeled data (including 500,000+ task running samples and 100,000+ failure cases). The loss function is the cross-entropy loss function, and the optimizer is AdamW (with an initial learning rate of 1e-4, adjusted using a cosine annealing strategy).
[0058] In one implementation, a ParameterServer architecture is used for distributed training of large Transformer models. Specifically, the ParameterServer architecture consists of one parameter server (PS) node and eight worker nodes, supporting parallel training of models with hundreds of billions of parameters. Data is distributed to each worker node by sharding according to feature dimensions. The PS node is responsible for parameter aggregation and updating. Gradient compression (using Top-k gradient sparsity, where k is 10% of the total number of gradients) reduces network transmission overhead. The training process supports resuming training from breakpoints, and a training checkpoint is saved every 10 minutes.
[0059] In one implementation, model pruning and quantization techniques are used to lightweight the large Transformer model. Specifically, lightweighting is achieved through model pruning and quantization. Pruning employs structured pruning (removing neurons whose absolute weight value is less than a threshold of 0.01), and quantization uses INT8 quantization, reducing the number of parameters by 60% and compressing the model size to 35% of its original size. Inference latency is optimized to <50ms, and inference is accelerated using TensorRT. Combined with a CPU / GPU hybrid inference mode (simple tasks are handled by the CPU, and complex tasks are handled by the GPU), efficient operation is ensured under different hardware environments.
[0060] In one implementation, the Transformer large model is incrementally trained periodically, with the latest task data generated by the IDE scheduling platform used as training samples each time to dynamically optimize the Transformer large model weights. Specifically, incremental training is performed hourly, with the training data being the latest task execution data and system performance data. An incremental learning algorithm (Finetune method, freezing the parameters of the bottom 6 layers of the network and training only the upper layers) is used to avoid catastrophic forgetting. At the same time, a model performance evaluation mechanism is set up. After each incremental training, the accuracy, recall, and F1 score are evaluated using a test set (containing 100,000+ samples). If the evaluation metrics drop by more than 5%, the model is automatically rolled back to the previous version.
[0061] S4 generates a root cause analysis report and sends the report to the user for optimization.
[0062] In one implementation, the root cause analysis report of a failure is visualized through task heatmaps, resource usage curves, and real-time alert lists. A dashboard is provided displaying task heatmaps (divided by task type and execution node dimension), resource usage curves (supporting real-time plotting and historical backtracking of indicators such as CPU, memory, and network, with time granularity selectable by seconds / minutes / hours), and a real-time alert list (sorted by alert level: urgent / important / general). Drill-down analysis is supported, allowing users to drill down from alert events to corresponding task details, system performance data, and feature weight distribution. Handling suggestions (such as resource expansion, task restart, and optimization of data partitioning strategies) are provided, sorted based on the success rate of historical handling cases, and showcasing recommended priorities and implementation steps.
[0063] Taking e-commerce platforms as an example:
[0064] Data Acquisition: Through API integration with the e-commerce platform's scheduling platform, real-time data is collected from the execution data of 50,000 Spark tasks. This includes task status (running / successful / failed / paused), execution duration, resource consumption (CPU utilization, memory usage, disk I / O throughput), and system performance metrics such as average CPU load, memory utilization, network bandwidth utilization, and inter-node data transmission latency. Second-level fetching and event-driven push are supported, with fetching occurring every second. Event-driven push is triggered in real-time for critical events such as task status changes and exceeding system performance limits, ensuring data timeliness and accuracy. Data transmission uses HTTPS encryption to prevent data leakage.
[0065] Data preprocessing: Cleaning noisy data, such as heartbeat packets (filtering duplicate heartbeat data with a frequency higher than 10 times / second) and debug logs (removing log entries containing the "DEBUG" label), to improve data quality; extracting key features, such as "data partition size variance", "task dependency depth", and "node CPU load fluctuation coefficient", constructing a time-series feature matrix, and using a 30-second sliding window to extract feature statistics (mean, maximum, minimum, and variance) within each window, forming a 128-dimensional feature vector, providing a structured data foundation for the training and inference of large models.
[0066] Model Inference: Real-time inference is performed using a large model with a Transformer architecture and a GPU (NVIDIA A100) in the inference hardware environment. The input is a preprocessed task feature matrix, and the large model performs deep modeling of the features through an encoder, outputting a timeout probability of 92% for a certain Spark task. Since the timeout probability exceeds the preset threshold (80%), the system triggers an early warning mechanism and generates an early warning event, which includes information such as task ID, early warning time, timeout probability, and early warning level (urgent).
[0067] Root cause analysis: Utilizing the attention weight mechanism of the large model, we conducted an in-depth analysis of the reasons for task timeouts. By analyzing the attention weights output by the model, we found that the weight of the "data skew" feature accounted for as high as 75%, the weight of the "insufficient node storage IO throughput" feature accounted for 12%, and the weight of the "unreasonable task parallelism configuration" feature accounted for 8%. Based on this, we determined that data skew is the main cause of task timeouts. The system recommends optimizing the data partitioning strategy (partitioning by user ID hash and increasing the number of partitions from 10 to 20), and also suggests temporarily increasing the storage IO priority of the node where the task is located.
[0068] Effect Verification: After implementing the optimization strategy, the task execution was monitored again. The results showed that the execution time of the Spark task was reduced from 180 seconds to 108 seconds, a reduction of 40%; the average CPU utilization decreased from 85% to 60%, memory usage decreased by 30%, and disk I / O throughput increased by 25%. Subsequent tracking of this type of task showed that the timeout probability decreased from 35% before optimization to 8%, verifying the effectiveness of the optimization strategy and the practicality of the technical solution of this invention.
[0069] A task monitoring system for an IDE scheduling platform based on a large model, comprising:
[0070] The data acquisition module is used to collect task execution data, system performance data, and task execution log text data from the IDE scheduling platform in real time.
[0071] The data preprocessing module is used to extract key features from task execution data and system performance data, such as task dependencies, resource consumption trends, and data partition size and variance, as a task feature matrix; each task corresponds to a task feature matrix.
[0072] The intelligent monitoring and analysis module is used to input the task feature matrix into the Transformer large model and output the timeout probability of the corresponding task. When the timeout probability exceeds the preset timeout threshold, an alarm is triggered, and the system performance data and task operation log text data of the task are extracted at the same time.
[0073] The abnormal fluctuation data segments in the system performance data of this task are marked as abnormal performance data. Fault keywords are extracted from the task operation log text data of this task. The abnormal performance data features and fault keyword features are fused to obtain the abnormal task feature matrix. The abnormal task feature matrix is input into the Transformer large model, and the root cause of the fault and its weight ratio are output using the attention weight mechanism.
[0074] Generate a root cause analysis report and send it to the user for optimization.
[0075] In one implementation, it also includes a large model building and training module, which is used to perform distributed training of the Transformer large model using a ParameterServer architecture; to perform lightweight processing of the Transformer large model using model pruning and quantization techniques; and to perform incremental training of the Transformer large model on a regular basis, using the latest task data generated by the IDE scheduling platform as training samples each time, and dynamically optimizing the weights of the Transformer large model.
[0076] In one implementation, a visualization module is also included, which is used to visualize the root cause analysis report of the failure through task heatmaps, resource usage curves, and real-time warning lists.
Claims
1. A task monitoring method for an IDE scheduling platform based on a large model, characterized in that, Includes the following steps: Real-time collection of task execution data, system performance data, and task execution log text data from the IDE scheduling platform; Key features such as task dependencies, resource consumption trends, and data partition size and variance are extracted from task execution data and system performance data to form a task feature matrix; each task corresponds to a task feature matrix. Input the task feature matrix into the Transformer large model and output the timeout probability of the corresponding task; when the timeout probability exceeds the preset timeout threshold, trigger an alarm and extract the system performance data and task operation log text data of the task. The abnormal fluctuation data segments in the system performance data of this task are marked as abnormal performance data. Fault keywords are extracted from the task operation log text data of this task. The abnormal performance data features and fault keyword features are fused to obtain the abnormal task feature matrix. The abnormal task feature matrix is input into the Transformer large model, and the root cause of the fault and its weight ratio are output using the attention weight mechanism. Generate a root cause analysis report and send it to the user for optimization.
2. The task monitoring method for an IDE scheduling platform based on a large model according to claim 1, characterized in that: The ParameterServer architecture is used for distributed training of large Transformer models.
3. The task monitoring method for an IDE scheduling platform based on a large model according to claim 1, characterized in that: We employ model pruning and quantization techniques to lightweight the large Transformer model.
4. The task monitoring method for an IDE scheduling platform based on a large model according to claim 1, characterized in that: The Transformer large model is incrementally trained periodically, with the latest task data generated by the IDE scheduling platform used as the training sample each time, and the weights of the Transformer large model are dynamically optimized.
5. The task monitoring method for an IDE scheduling platform based on a large model according to claim 1, characterized in that: When the execution time of a task exceeds a preset execution time threshold, the corresponding task is marked as a delayed task; the key features of the delayed task are extracted first and then processed.
6. The task monitoring method for an IDE scheduling platform based on a large model according to claim 1, characterized in that: Before extracting key features, the data is preprocessed; preprocessing includes cleaning noise and removing data that deviates from the normal range.
7. The task monitoring method for an IDE scheduling platform based on a large model according to claim 1, characterized in that: The root cause analysis report of the failure is visualized through task heatmaps, resource usage curves, and real-time early warning lists.
8. A task monitoring system for an IDE scheduling platform based on a large model, characterized in that, include: The data acquisition module is used to collect task execution data, system performance data, and task execution log text data from the IDE scheduling platform in real time. The data preprocessing module is used to extract key features from task execution data and system performance data, such as task dependencies, resource consumption trends, and data partition size and variance, as a task feature matrix; each task corresponds to a task feature matrix. The intelligent monitoring and analysis module is used to input the task feature matrix into the Transformer large model and output the timeout probability of the corresponding task. When the timeout probability exceeds the preset timeout threshold, an alarm is triggered, and the system performance data and task operation log text data of the task are extracted at the same time. The abnormal fluctuation data segments in the system performance data of this task are marked as abnormal performance data. Fault keywords are extracted from the task operation log text data of this task. The abnormal performance data features and fault keyword features are fused to obtain the abnormal task feature matrix. The abnormal task feature matrix is input into the Transformer large model, and the root cause of the fault and its weight ratio are output using the attention weight mechanism. Generate a root cause analysis report and send it to the user for optimization.
9. The task monitoring system for an IDE scheduling platform based on a large model according to claim 8, characterized in that: It also includes a large model building and training module, which uses the ParameterServer architecture to perform distributed training of the Transformer large model; uses model pruning and quantization techniques to perform lightweight processing on the Transformer large model; and performs incremental training on the Transformer large model on a regular basis, using the latest task data generated by the IDE scheduling platform as training samples each time to dynamically optimize the weights of the Transformer large model.
10. The task monitoring system for an IDE scheduling platform based on a large model according to claim 8, characterized in that: It also includes a visualization module, which uses task heatmaps, resource usage curves, and real-time warning lists to visualize the root cause analysis report of the failure.