A deep learning-based regulatory task optimization and dynamic resource management method

Abstract

The application discloses a kind of based on deep learning's supervision task optimization and dynamic resource management method, it is related to computer technology field, it solves the task scheduling method of machine learning in prior art lack of flexibility and universality, it is difficult to meet the diversification demand under complex computing power network environment Problem;The method comprises: real-time acquisition of each operation index data of big data platform;Each operation index data is input into the supervision model trained to obtain the abnormal probability score corresponding to each operation index data in each time step;Wherein, the supervision model comprises: embedding layer, transformer encoder and anomaly detector connected in turn;Big data platform adjusts the current scheduling queue according to the abnormal probability score of each time step;Realize the dependent relationship and priority between reasonable analysis task, flexible allocation of resources, to improve the efficiency and stability of overall task scheduling.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method for optimizing supervisory tasks and managing dynamic resources based on deep learning. Background Technology

[0002] Addressing the complexities of big data platform security scenarios, the dynamic changes in regulatory objectives, the frequent occurrence of anomalies, and the need for closed-loop security control execution, this paper proposes a flexible scheduling architecture for big data platform security supervision. This architecture overcomes key technical challenges such as security supervision task dependency and functional system design, security supervision task registration, supervision task orchestration, and monitoring of the granularity of supervision task execution, achieving coordinated operation within the security supervision platform. Comprehensive management of big data platform security supervision requires issuing security supervision tasks, collecting regulatory data, analyzing anomalies, and classifying and responding to anomalies. These tasks may not be static due to the suddenness of security events, requiring the comprehensive supervision management module to dynamically configure and schedule service resources based on service resource status and regulatory processing needs to meet the comprehensive management requirements of big data platform security supervision. Algorithms applied to task scheduling typically fall into three categories: heuristics, metaheuristics, and reinforcement learning. Heuristic methods prioritize generality, ease of understanding and implementation, but usually use fixed parameters and rule strategies for scheduling. They can be used as part of the scheduling system to handle possible abnormal scheduling decisions, thereby increasing system stability. Metaheuristic algorithms are improvements on heuristic algorithms, derived by combining random algorithms and local search algorithms. A feasible solution to a problem is obtained within acceptable costs, such as time and space. This feasible solution is searched in the solution space, so it is not necessarily the optimal solution, and the deviation from the optimal solution cannot be estimated in advance.

[0003] In recent years, with the rapid development of Large Language Model (LLM) technology, intelligent processing solutions based on large models have been widely applied in many fields such as natural language processing, knowledge question answering, and content generation.

[0004] The current state of research shows that scheduling objectives vary from one to several. Common objectives include shortest completion time, resource utilization, load balancing, SLA, and cost. Most studies focus on completion time and load balancing, neglecting the inherent relationship between the two, which is based on user-level agreements. As the number of users increases, the problem scale also grows larger. This increased problem scale poses a challenge to heuristic algorithms, leading to convergence difficulties, poor convergence accuracy, or susceptibility to local optima. For reinforcement learning-based scheduling algorithms, the increased problem scale also leads to a larger state space and action space, resulting in the curse of dimensionality. A solution that can meet the optimization objectives of both users and service providers under large-scale tasks and adaptively schedule between them will undoubtedly become one of the urgent challenges to be addressed in cloud scheduling in the era of big data.

[0005] Artificial intelligence technology can significantly improve the performance and expand the functionality of traditional task scheduling systems in areas such as task anomaly detection, dynamic scheduling decision-making, task dependency analysis, environment migration, and distributed collaboration. These innovative application solutions are worthy of further exploration and research.

[0006] Existing machine learning-based task scheduling methods typically optimize only a single task scheduling objective, such as minimizing task execution time or improving resource utilization, while neglecting other important scheduling metrics, such as task priority and fairness. This results in scheduling outcomes that fail to meet the diverse needs of different application scenarios. For example, in a cloud computing environment, different users' tasks have different priorities and latency requirements; simply using resource utilization as the objective clearly cannot satisfy the needs of all users. While many existing monitoring tools can provide real-time data, they may experience latency or inaccuracies under high loads, failing to react promptly. In cloud computing or hybrid environments, resource monitoring becomes more complex, requiring the integration of monitoring data from multiple platforms. Automated allocation relies on carefully designed strategies and rules, but formulating these strategies requires a deep understanding of the system's workload and application performance requirements, and these strategies need continuous adjustment as business changes occur, increasing maintenance costs. During environment refactoring, due to complex dependencies and significant environmental differences, unexpected problems can easily arise, leading to project delays or unforeseen system failures. Furthermore, these methods rely on large amounts of historical data for training, but real-world task and resource conditions may dynamically change, and the trained models cannot adapt to these changes. When network topology is adjusted, hardware is upgraded, or user behavior patterns change, model performance degrades significantly. This hinders the provision of stable and reliable scheduling services in rapidly changing environments. Furthermore, model training and inference processes incur computational overhead, making it impossible to make efficient real-time scheduling decisions in rapidly changing environments. For some latency-sensitive applications, this can cause serious performance problems. In summary, existing machine learning-based task scheduling methods lack flexibility and versatility, making it difficult to meet the diverse needs of complex computing network environments. Summary of the Invention

[0007] This invention provides a deep learning-based method for optimizing supervisory tasks and managing dynamic resources. This method addresses the lack of flexibility and versatility in existing machine learning task scheduling methods, which struggle to meet the diverse needs of complex computing network environments. It enables the rational analysis of dependencies and priorities between tasks and the flexible allocation of resources, thereby improving the overall efficiency and stability of task scheduling.

[0008] This invention provides a deep learning-based method for optimizing supervisory tasks and managing dynamic resources, the method comprising:

[0009] Real-time collection of various operational metrics data from the big data platform;

[0010] The operational indicator data are input into the trained monitoring model to obtain the anomaly probability score corresponding to each operational indicator data in each time step. The monitoring model includes an embedding layer, a transformer encoder, and an anomaly detector connected in sequence. The embedding layer converts the operational indicator data into corresponding low-dimensional vectors. The transformer encoder uses a multi-head attention mechanism to calculate the correlation of the low-dimensional vectors to obtain an output feature matrix. The anomaly detector outputs the anomaly probability score corresponding to each operational indicator data in each time step based on the output feature matrix.

[0011] The current scheduling queue of the big data platform is adjusted based on the anomaly probability score at each time step.

[0012] In one possible implementation, the embedding layer converts the various operational metric data into corresponding low-dimensional vectors, including:

[0013] All the aforementioned operational indicator data are converted into corresponding low-dimensional vectors of fixed length.

[0014] In one possible implementation, the transformer encoder utilizes a multi-head attention mechanism to perform correlation calculations on the low-dimensional vector to obtain an output feature matrix, including:

[0015] The low-dimensional vector is divided into multiple subspaces, each of which includes a portion of the low-dimensional vector.

[0016] The first correlation between features in each subspace is calculated by dot product operation, and the first correlation is scaled using a scaling factor to obtain the correlation within the subspace.

[0017] The correlation between each subspace and the other subspaces is calculated by dot product operation to obtain the second correlation. The second correlation is then scaled using a scaling factor to obtain the correlation between subspaces.

[0018] The correlations within the subspace and the correlations between the subspaces are combined to obtain the attention matrix;

[0019] The low-dimensional vector is weighted using the attention matrix to obtain the output feature matrix.

[0020] In one possible implementation, the anomaly detector includes: multiple fully connected layers, an activation function, and an output layer;

[0021] The multiple fully connected layers are used to perform linear transformation on the output feature matrix and extract abnormal features from the linearly transformed output feature matrix to obtain an abnormal feature vector sequence.

[0022] The activation function is used to judge each element in the abnormal feature vector sequence. If the element is less than a preset value, the element in the abnormal feature vector sequence is modified to 0. If the element is greater than or equal to the preset value, the element in the abnormal feature vector sequence is not modified until the first feature vector is obtained.

[0023] The output layer is used to calculate the anomaly probability score corresponding to each of the operational indicator data in each time step based on the first feature vector.

[0024] In one possible implementation, the activation function is the ReLU function.

[0025] In one possible implementation, the output layer calculates the anomaly probability score corresponding to each operational indicator data within each time step based on the first feature vector, including:

[0026] Based on the first feature vector, the abnormal probability score corresponding to each of the operational indicator data in each time step is calculated using the abnormal probability score formula.

[0027] The abnormality probability score formula is expressed as follows:

[0028] p=σ(W0·ReLU(W f ·z+b f )+b0);

[0029] Where W0 represents the weight matrix of the output layer; b0 represents the bias vector of the output layer; W f b represents the weight matrix of the fully connected layer. f Let represent the bias vector of the fully connected layer; σ represents the sigmoid activation function; p represents the output anomaly probability score vector, where each element p i represents the anomaly probability at the corresponding time step; ReLU(·) represents the ReLU function; z represents the first eigenvector.

[0030] In one possible implementation, the training process of the regulatory model includes:

[0031] Cross-validation is used to split the validation set and training set multiple times to obtain multiple different training and validation sets;

[0032] The regulatory model is trained using multiple different training and validation sets to obtain a fully trained regulatory model.

[0033] In one possible implementation, the training process of the regulatory model further includes optimizing the regulatory model using a sequence modeling-assisted losser.

[0034] In one possible implementation, adjusting the current scheduling queue of the big data platform based on the anomaly probability score at each time step includes:

[0035] The relationship between the anomaly probability and the preset anomaly threshold is determined. If the anomaly probability is greater than or equal to the preset anomaly threshold, the big data platform will postpone the scheduling task corresponding to the anomaly probability.

[0036] If the anomaly probability is less than the preset anomaly threshold, the big data platform will not adjust the scheduling queue.

[0037] One or more technical solutions provided in this invention have at least the following technical effects or advantages:

[0038] This invention uses a monitoring model to adaptively predict task execution progress. It combines big data model technology to analyze the characteristics of monitored task execution, dynamically checks for abnormal information, and effectively improves the anomaly detection and handling capabilities of the task scheduling system. At the same time, it can also reasonably analyze the dependencies and priorities between tasks, flexibly allocate resources, thereby improving the overall efficiency and stability of task scheduling. Furthermore, it combines reasonable scheduling strategies and triggering conditions to achieve automated resource allocation and environment reconstruction. Attached Figure Description

[0039] Figure 1 A flowchart illustrating the steps of a deep learning-based method for optimizing supervisory tasks and managing dynamic resources, as provided in an embodiment of the present invention. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0041] This invention provides a deep learning-based method for optimizing supervisory tasks and managing dynamic resources, such as... Figure 1 As shown, the method includes the following steps S101 to S103.

[0042] S101 collects various operational metrics data from the big data platform in real time.

[0043] Here, the various operational metrics refer to metrics such as service type, analysis object, analysis scope, CPU frequency and utilization, memory usage, database interaction, and external interface calls.

[0044] S102, input the various operational indicator data into the trained monitoring model to obtain the anomaly probability score corresponding to each operational indicator data in each time step. The monitoring model includes an embedding layer, a transformer encoder, and an anomaly detector connected in sequence.

[0045] The embedding layer converts various operational metrics data into corresponding low-dimensional vectors.

[0046] Here, all operational indicator data are converted into corresponding low-dimensional vectors of fixed length.

[0047] For example, the raw data is converted into a low-dimensional vector representation to prepare for subsequent model processing. A low-dimensional vector is a representation that compresses the raw data into a smaller dimensional space. In this approach, the raw big data platform operating metric data is converted into a fixed-length low-dimensional vector to facilitate subsequent machine learning model processing.

[0048] The transformer encoder uses a multi-head attention mechanism to calculate the correlation of low-dimensional vectors and obtain the output feature matrix.

[0049] Here, the transformer encoder is used to perform correlation calculations on the low-dimensional vector using a multi-head attention mechanism to obtain the output feature matrix, including:

[0050] (1) Divide the low-dimensional vector into multiple subspaces, each subspace including a part of the low-dimensional vector;

[0051] (2) Calculate the first correlation between features in each subspace by dot product operation, and scale the first correlation using a scaling factor to obtain the correlation in the subspace.

[0052] (3) Calculate the correlation between each subspace and the other subspaces by dot product operation to obtain the second correlation, and scale the second correlation using a scaling factor to obtain the correlation between subspaces.

[0053] (4) Combine the correlations within the subspace with the correlations between subspaces to obtain the attention matrix;

[0054] (5) Use the attention matrix to weight the low-dimensional vector to obtain the output feature matrix.

[0055] For example, an architecture based on a multi-head attention mechanism is adopted to capture the complex dependencies between features in the input sequence, enhancing the supervisory model's ability to process input data. Multi-head attention is a key component of the Transformer model; it captures the complex dependencies between features in the input sequence by dividing the input features into multiple subspaces and calculating the correlations between each subspace. The subspace correlation calculation process is as follows:

[0056] ① Subspace partitioning: The input feature matrix is ​​divided into multiple subspaces, each containing a portion of the input features.

[0057] ② Correlation calculation within subspaces: Within each subspace, the correlation between features is calculated. This is achieved by calculating the dot product between features, which measures the linear correlation between features. The result of the dot product calculation is scaled using a scaling factor to improve computational stability.

[0058] ③ Correlation calculation between subspaces: After calculating the correlation within a subspace, the correlation between different subspaces is calculated. This is achieved by performing a dot product between the feature matrix of one subspace and the feature matrices of all other subspaces. The results of the correlation calculation between subspaces are also scaled using a scaling factor.

[0059] ④ Merging Subspace Correlation: Merge the correlations within all subspaces and the correlations between subspaces to obtain the final attention matrix. This attention matrix can be viewed as a weighted sum of the original feature matrix, where each feature is assigned a different weight based on its correlation in different subspaces.

[0060] ⑤ Feature Weighting: The original features are weighted using an attention matrix to obtain a weighted feature sequence. This weighting process can be implemented using matrix multiplication, where the attention matrix serves as the weight matrix and the original feature matrix serves as the input matrix.

[0061] The above mechanism can enhance the regulatory model's ability to process input data because it allows the regulatory model to focus on multiple aspects of the input data simultaneously, thereby better understanding and modeling the complexity of the data.

[0062] The output of the converter encoder is an output feature matrix with higher semantic representation capabilities. The feature sequence output by the converter encoder refers to the sequence processed by the multi-head attention mechanism. These features represent higher-level semantic information of the input data. The feature sequence at each time step contains a weighted representation of each feature in the input data. These representations can capture key information in the data and provide a basis for anomaly detection.

[0063] The anomaly detector is used to output the anomaly probability score corresponding to each operational indicator data in each time step based on the output feature matrix.

[0064] Here, the anomaly detector includes: multiple fully connected layers, activation functions, and an output layer.

[0065] Multiple fully connected layers are used to perform linear transformation on the output feature matrix. The linearly transformed output feature matrix is ​​then used to extract abnormal features, resulting in an abnormal feature vector sequence.

[0066] The activation function is used to evaluate each element in the abnormal feature vector sequence. If the element is less than a preset value, the element in the abnormal feature vector sequence is modified to 0. If the element is greater than or equal to the preset value, the element in the abnormal feature vector sequence is not modified, until the first feature vector is obtained. Here, the activation function is the ReLU function.

[0067] The output layer calculates the anomaly probability score corresponding to each operational indicator data within each time step using multiple neurons based on the first feature vector. Specifically, the output layer calculates the anomaly probability score corresponding to each operational indicator data within each time step using multiple neurons based on the first feature vector, including:

[0068] Based on the first feature vector, the abnormal probability score corresponding to each operational indicator data in each time step is calculated using the abnormal probability score formula.

[0069] The abnormality probability score formula is expressed as follows:

[0070] p=σ(W0·ReLU(W f ·z+b f )+b0);

[0071] Where W0 represents the weight matrix of the output layer; b0 represents the bias vector of the output layer; W f b represents the weight matrix of the fully connected layer. f Let represent the bias vector of the fully connected layer; σ represents the sigmoid activation function; p represents the output anomaly probability score vector, where each element p i represents the anomaly probability at the corresponding time step; ReLU(·) represents the ReLU function; z represents the first eigenvector.

[0072] For example, an anomaly detector consists of multiple fully connected layers, activation functions, and an output layer, outputting an anomaly probability score for each time step. The fully connected layers further compress and transform the output feature matrix of the transformer encoder to extract key anomaly features. The activation function is ReLU, used to increase the non-linear expressive power of the supervisory model, enabling it to better learn complex patterns in the data. The output layer outputs the anomaly probability score for each time step, which can be used to determine whether the indicator data at the current time step constitutes an anomaly. The relationship between the three is as follows:

[0073] ① Fully connected layer:

[0074] Input: Output feature matrix from the transformer encoder.

[0075] Processing: The fully connected layer is responsible for compressing and transforming the feature sequence output by the transformer encoder. It extracts key anomalous features by performing a linear transformation on the feature sequence. The fully connected layer can be viewed as a feature extraction and compression module; its purpose is to map high-dimensional features to a new low-dimensional space to facilitate subsequent processing.

[0076] ② Activation function (ReLU):

[0077] Input: Output from the fully connected layer.

[0078] Processing: Activation functions are used to enhance the non-linear expressiveness of the monitoring model. In anomaly detection heads, ReLU is commonly used as the activation function because it introduces non-linearity, allowing the monitoring model to better capture complex patterns in the data. The ReLU function outputs the input value when the input is greater than 0 and outputs 0 when the input is less than or equal to 0. This avoids the gradient vanishing problem during training and helps the monitoring model converge.

[0079] ③ Output layer:

[0080] Input: The output from the activation function.

[0081] Processing: The output layer is responsible for outputting the anomaly probability score for each time step. In this scheme, the goal of the anomaly detection head is to determine whether the indicator data at the current time step belongs to an anomaly. Therefore, the output layer needs to be able to output an anomaly probability score, which can be used as a judgment criterion. The output of the output layer is usually a feature sequence processed by an activation function, and the anomaly probability score is calculated through one or more neurons (nodes).

[0082] S103, adjust the current scheduling queue of the big data platform based on the anomaly probability score at each time step.

[0083] Specifically, in step S103, the current scheduling queue of the big data platform is adjusted based on the anomaly probability score at each time step, including:

[0084] S1031, determine the relationship between the anomaly probability and the preset anomaly threshold. If the anomaly probability is greater than or equal to the preset anomaly threshold, the big data platform will postpone the scheduling task corresponding to the anomaly probability.

[0085] S1032, If the probability of an anomaly is less than the preset anomaly threshold, the big data platform will not adjust the scheduling queue.

[0086] For example, in a specific embodiment provided by this invention, during online anomaly detection, real-time operational indicator data from a big data platform is collected, processed through feature engineering, and then input into a trained monitoring model. The monitoring model, having learned the characteristics of normal operation during training, can calculate anomaly probability scores for each time step based on the input data. These scores reflect the degree of anomaly in the data at that time step; a higher score indicates a greater likelihood of an anomaly. To determine which scores are considered anomaly, the system needs to set a preset anomaly threshold. This preset anomaly threshold is set by analyzing historical data and expert experience. Through statistical analysis, the distribution of data under normal conditions can be understood, thereby determining a reasonable preset anomaly threshold. This preset anomaly threshold should effectively detect anomalies without causing excessive false alarms. Once the anomaly probability score calculated by the monitoring model exceeds the set preset anomaly threshold, the system triggers an anomaly alarm, notifying relevant personnel. The alarm information includes the specific time of the anomaly, detailed information about the anomaly data, and the anomaly probability score. Furthermore, the system also feeds back the anomaly detection results to the task scheduling system for corresponding adjustments and optimizations to the tasks.

[0087] Finally, we will continue to monitor the performance of the regulatory model in the production environment, update the training samples in a timely manner, analyze and optimize anomalies such as missed or false alarms, and ensure that the regulatory model can always adapt to the latest operating conditions of the big data platform.

[0088] In a specific embodiment provided by this invention, the training process of the regulatory model includes:

[0089] (1) The validation set and training set are divided multiple times by cross-validation to obtain multiple different training sets and validation sets;

[0090] (2) The regulatory model is trained using multiple different training and validation sets to obtain the trained regulatory model.

[0091] The training process of the regulatory model also includes optimizing the regulatory model using a sequence modeling-assisted losser.

[0092] For example, during training, hyperparameters, regularization techniques, and data augmentation strategies are tuned to improve the generalization ability of the oversight model. Simultaneously, attention is paid to the inference efficiency of the oversight model, employing techniques such as oversight model compression and quantization for optimization, so as to deploy it in a real-world big data platform. Hyperparameters, regularization techniques, and data augmentation strategies all have a significant impact on the performance and generalization ability of the oversight model, thus requiring tuning. Hyperparameters include learning rate, batch size, number of layers, number of neurons, and activation function type. Regularization techniques such as L1 regularization (Lasso), L2 regularization (Ridge), and dropout in deep learning are used to prevent overfitting of the oversight model. Data augmentation strategies transform and modify the original data through rotation, scaling, pruning, color transformation, and noise addition to generate more training data. Hyperparameter tuning typically involves cross-validation, repeatedly splitting the training data into validation and training sets to evaluate the performance of the oversight model under different configurations. The selection and strength of regularization techniques are determined by observing the training and validation errors of the oversight model. Optimizing data augmentation strategies involves trying different combinations and intensities of augmentation techniques to find the best augmentation solution.

[0093] Based on anomaly probability scores, the regulatory model predicts various indicator data for the next time step and calculates an auxiliary loss based on the difference between the predicted and actual data, which is then used to optimize the regulatory model. This approach allows the regulatory model to learn the temporal characteristics of the data while detecting anomalies, improving the accuracy and robustness of anomaly detection.

[0094] In a specific embodiment provided by this invention, the various operational indicator data are represented as follows:

[0095] Task type: A, B, C; Analysis object: X, Y, Z; Analysis scope: small, medium, large; CPU frequency: 2.5GHz, 3.0GHz, 3.5GHz; CPU utilization: 20%, 50%, 80%; Memory usage: 4GB, 8GB, 16GB; Database interaction count: 100, 200, 300; External interface call count: 50, 100, 150.

[0096] Dimensionality reduction techniques are used to transform the above indicators into low-dimensional vectors. For example, each indicator might be mapped to a 3-dimensional space. Here is a simplified example:

[0097] Task type A is represented as: [0.1, 0.2, 0.3]; Analysis object X is represented as: [0.4, 0.5, 0.6]; Small analysis scope is represented as: [0.7, 0.8, 0.9]; CPU frequency of 2.5GHz is represented as: [0.2, 0.3, 0.4]; CPU utilization of 20% is represented as: [0.5, 0.6, 0.7]; Memory usage of 4GB is represented as: [0.8, 0.9, 1.0]; Number of database interactions of 100 is represented as: [0.3, 0.4, 0.5]; Number of external interface calls of 50 is represented as: [0.6, 0.7, 0.8];

[0098] Now, suppose these low-dimensional vectors are part of time-series data, collected at several consecutive time points. Concatenate these vectors to form a sequence:

[0099] Time point 1: [0.1, 0.2, 0.3, 0.4, 0.5, 0.8, 0.3, 0.6];

[0100] Time point 2: [0.2, 0.3, 0.4, 0.5, 0.6, 0.9, 0.4, 0.7];

[0101] Time point 3: [0.7, 0.8, 0.9, 0.6, 0.7, 1.0, 0.5, 0.8];

[0102] These feature sequences can be combined into a feature matrix, where each row represents a feature vector at a given time point:

[0103] During the computation of the multi-head attention mechanism, the feature matrix is ​​processed to capture the complex dependencies between features. The processing steps are as follows:

[0104] Subspace partitioning: Dividing each row of the feature matrix into multiple subspaces. For example, 8 features can be divided into 4 subspaces, with each subspace containing 2 features.

[0105] Correlation calculation within subspaces: Within each subspace, the correlation between features is calculated. For example, for the first subspace, the dot product between the first and second features is calculated.

[0106] Correlation calculation between subspaces: Calculate the correlation between different subspaces. For example, calculate the dot product of the features of the first subspace with the features of all other subspaces.

[0107] Merge subspace correlations: Merge all correlations within and between subspaces to form the final attention matrix.

[0108] Feature weighting: The original features are weighted using an attention matrix to obtain the output feature matrix. The specific output feature matrix is ​​represented as follows:

[0109]

[0110] The output feature matrix contains a sequence of features processed by a multi-head attention mechanism, with each feature assigned a different weight based on its relevance in different subspaces. This weighted representation captures key information from the data, providing a basis for subsequent anomaly detection.

[0111] The default value in the anomaly detector is set to 0. Assuming there is a feature vector [0.12, -0.05, 0.34, 0.02], after applying the ReLU function: ReLU(0.12) = 0.12 (because 0.12 > 0); ReLU(-0.05) = 0 (because -0.05 <= 0); ReLU(0.34) = 0.34 (because 0.34 > 0); ReLU(0.02) = 0.02 (because 0.02 > 0); therefore, the first feature vector after ReLU processing will be [0.12, 0, 0.34, 0.02].

[0112] Then, the anomaly probability score corresponding to each operational indicator data is calculated according to the anomaly probability score formula. The big data platform adjusts the current scheduling queue based on the anomaly probability score at each time step.

[0113] Each embodiment in this specification is described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. All or part of this invention can be used in numerous general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, mobile communication terminals, multiprocessor systems, microprocessor-based systems, programmable electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0114] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the present invention.

Claims

1. A method for optimizing supervisory tasks and managing dynamic resources based on deep learning, characterized in that, include: Real-time collection of various operational metrics data from the big data platform; wherein, the operational metrics data include multiple items such as task type, analysis object, CPU utilization, memory usage, number of database interactions, and number of external interface calls; The operational indicator data are input into the trained monitoring model to obtain the anomaly probability score corresponding to each operational indicator data at each time step. The monitoring model includes an embedding layer, a transformer encoder, and an anomaly detector connected in sequence. The embedding layer converts the operational indicator data into corresponding low-dimensional vectors. The transformer encoder uses a multi-head attention mechanism to calculate the correlation of the low-dimensional vectors, obtaining an output feature matrix. The multi-head attention mechanism calculation includes: dividing the low-dimensional vector into multiple subspaces, calculating the first correlation of features within each subspace and the second correlation of features between subspaces, merging them to obtain an attention matrix, and then weighting the low-dimensional vector using the attention matrix. The anomaly detector outputs the anomaly probability score corresponding to each operational indicator data at each time step based on the output feature matrix. The anomaly detector includes multiple fully connected layers, an activation function, and an output layer. The activation function is the ReLU function. The current scheduling queue of the big data platform is adjusted based on the anomaly probability score at each time step.

2. The method for optimizing supervisory tasks and managing dynamic resources based on deep learning according to claim 1, characterized in that, The embedding layer converts the various operational metric data into corresponding low-dimensional vectors, including: All the aforementioned operational indicator data are converted into corresponding low-dimensional vectors of fixed length.

3. The method for optimizing supervisory tasks and managing dynamic resources based on deep learning according to claim 1, characterized in that, The converter encoder uses a multi-head attention mechanism to calculate the correlation of the low-dimensional vector to obtain an output feature matrix, including: The low-dimensional vector is divided into multiple subspaces, each of which includes a portion of the low-dimensional vector. The first correlation between features in each subspace is calculated by dot product operation, and the first correlation is scaled using a scaling factor to obtain the correlation within the subspace. The second correlation between each subspace and the other subspaces is calculated by dot product operation, and the second correlation is scaled using a scaling factor to obtain the correlation between subspaces. The correlations within the subspace and the correlations between the subspaces are combined to obtain the attention matrix; The low-dimensional vector is weighted using the attention matrix to obtain the output feature matrix.

4. The method for optimizing supervisory tasks and managing dynamic resources based on deep learning according to claim 1, characterized in that, The anomaly detector includes: multiple fully connected layers, an activation function, and an output layer; The multiple fully connected layers are used to perform linear transformation on the output feature matrix and extract abnormal features from the linearly transformed output feature matrix to obtain an abnormal feature vector sequence. The activation function is used to judge each element in the abnormal feature vector sequence. If the element is less than a preset value, the element in the abnormal feature vector sequence is modified to 0. If the element is greater than or equal to the preset value, the element in the abnormal feature vector sequence is not modified until the first feature vector is obtained. The output layer is used to calculate the anomaly probability score corresponding to each of the operational indicator data in each time step based on the first feature vector.

5. The deep learning-based method for optimizing supervisory tasks and managing dynamic resources according to claim 4, characterized in that, The activation function is the ReLU function.

6. The deep learning-based method for optimizing supervisory tasks and managing dynamic resources according to claim 4, characterized in that, The output layer calculates the anomaly probability score corresponding to each operational indicator data within each time step based on the first feature vector, including: Based on the first feature vector, the anomaly probability score corresponding to each of the operational indicator data in each time step is calculated using the anomaly probability score formula. The abnormality probability score formula is expressed as follows: ； in, This represents the weight matrix of the output layer; This represents the bias vector of the output layer; This represents the weight matrix of the fully connected layer; This represents the bias vector of the fully connected layer; This represents the sigmoid activation function; This represents the output anomaly probability score vector, where each element... This represents the probability of an anomaly at the corresponding time step. Represents the ReLU function; This represents the first eigenvector.

7. The method for optimizing supervisory tasks and managing dynamic resources based on deep learning according to claim 1, characterized in that, The training process of the regulatory model includes: Cross-validation is used to split the validation set and training set multiple times to obtain multiple different training and validation sets; The regulatory model is trained using multiple different training and validation sets to obtain a fully trained regulatory model.

8. The method for optimizing supervisory tasks and managing dynamic resources based on deep learning according to claim 7, characterized in that, The training process of the regulatory model also includes optimizing the regulatory model using a sequence modeling-assisted losser.

9. The method for optimizing supervisory tasks and managing dynamic resources based on deep learning according to claim 1, characterized in that, The current scheduling queue of the big data platform is adjusted based on the anomaly probability score at each time step, including: The relationship between the anomaly probability and the preset anomaly threshold is determined. If the anomaly probability is greater than or equal to the preset anomaly threshold, the big data platform will postpone the scheduling task corresponding to the anomaly probability. If the anomaly probability is less than the preset anomaly threshold, the big data platform will not adjust the scheduling queue.

Images