Supervision task optimization and dynamic resource management method based on deep learning

Abstract

The invention discloses a supervision task optimization and dynamic resource management method based on deep learning, relates to the technical field of computers, and solves the problem that a task scheduling method of machine learning in the prior art is lack of flexibility and universality and is difficult to meet diversified requirements in a complex computing power network environment. The method comprises the following steps: collecting various operation index data of the big data platform in real time; inputting each operation index data into the trained supervision model to obtain an abnormal probability score corresponding to each operation index data in each time step; wherein the supervision model comprises an embedding layer, a converter encoder and an anomaly detector which are connected in sequence; the big data platform adjusts the current scheduling queue according to the abnormal probability score of each time step; according to the method, the dependency relationship and priority among the tasks are reasonably analyzed, and resources are flexibly allocated, so that the overall task scheduling efficiency and stability are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a supervision task optimization and dynamic resource management method based on deep learning. Background Art

[0002] In view of the characteristics or requirements of the big data platform's complex security business scenarios, dynamic changes in regulatory targets, frequent abnormal events, and closed-loop security management and control, a flexible scheduling architecture for big data platform security supervision is proposed, breaking through key technologies such as security supervision task dependence and functional system design, security supervision task registration, supervision task orchestration, and supervision task execution granularity monitoring, to achieve internal coordination and linkage of the security supervision platform. The comprehensive management of big data platform security supervision requires issuing security supervision tasks, collecting regulatory collection data, analyzing abnormal events, and completing classification and response to abnormal situations. These tasks may not be static due to the suddenness of security events. The comprehensive management module of supervision needs to dynamically configure and schedule service resources according to the status of service resources and regulatory processing requirements to meet the needs of comprehensive management of big data platform security supervision. The algorithms used for task scheduling are usually divided into three categories: heuristic, meta-heuristic, and reinforcement learning. Heuristic methods give priority to generality and are easy to understand and implement, but usually use fixed parameters and rule strategies for scheduling. They can be used as part of the scheduling system to deal with possible abnormal scheduling decisions and increase system stability. Meta-heuristic algorithms are an improvement on heuristic algorithms, which are obtained by combining random algorithms and local search algorithms. A feasible solution to the problem is obtained at an acceptable cost such as time and space. The feasible solution is obtained by searching in the solution space, so the feasible solution is not necessarily the optimal solution, and the deviation from the optimal solution cannot necessarily be estimated in advance.

[0003] In recent years, with the rapid development of large language model (LLM) technology, intelligent processing solutions based on large models have been widely used in many fields such as natural language processing, knowledge question and answer, and content generation.

[0004] From the above research status, it can be seen that the scheduling objectives vary from one to multiple. Common scheduling objectives include the shortest completion time, resource utilization, load balancing, SLA and cost, etc. Most of them consider the two objectives of completion time and load balancing, ignoring the inherent relationship between the two based on the premise of user-level agreement. With the increase in the number of users, the scale of the problem is also getting larger and larger. The increased scale of the problem poses a challenge to the heuristic algorithm, making it difficult to converge, with poor convergence accuracy or easy to fall into the local optimum. For the scheduling algorithm using reinforcement learning, the increase in the scale of the problem will also lead to an increase in the state space and action space, which will lead to the dimensionality disaster. A solution that can meet the optimization goals of users and service providers under large-scale tasks and perform adaptive scheduling between the two will surely become one of the challenges that need to be urgently solved in cloud scheduling in the era of big data.

[0005] Artificial intelligence technology can bring significant performance improvements and functional expansion to traditional task scheduling systems in terms of task anomaly detection, dynamic scheduling decisions, task dependency analysis, environment migration, and distributed collaboration. These innovative application solutions are worthy of further exploration and research.

[0006] Existing machine learning-based task scheduling methods usually only optimize a single task scheduling goal, such as minimizing task execution time or improving resource utilization, while ignoring other important scheduling indicators, such as task priority and fairness. This results in scheduling results that cannot meet the diverse needs of different application scenarios. For example, in a cloud computing environment, tasks of different users have different priorities and latency requirements. Simply taking resource utilization as a goal obviously cannot take into account the needs of all users. Although many existing monitoring tools can provide real-time data, they may be delayed or inaccurate under high load and cannot respond in time. In cloud computing or hybrid environments, resource monitoring becomes more complicated and requires the integration of monitoring data from multiple platforms. Automated allocation relies on carefully designed strategies and rules, but the formulation of these strategies requires an in-depth understanding of the system's workload, application performance requirements, etc., and needs to be constantly adjusted as the business changes, increasing maintenance costs. In the process of environmental reconstruction, due to complex dependencies and large environmental differences, unexpected problems are prone to occur, resulting in project delays or unforeseen system failures. At the same time, these methods rely on a large amount of historical data for training, but the actual task and resource conditions may change dynamically, and the trained models cannot adapt to these changes. When the network topology is adjusted, the hardware is upgraded, or the user behavior pattern changes, the performance of the model will drop significantly. This is not conducive to providing stable and reliable scheduling services in a rapidly changing environment. In addition, the model training and inference process also has a certain computational overhead, and it is impossible to make efficient scheduling decisions in real time in a rapidly changing environment. For some latency-sensitive applications, this may cause serious performance problems. In general, the existing task scheduling methods based on machine learning lack flexibility and versatility, and it is difficult to meet the diverse needs in a complex computing network environment. Summary of the invention

[0007] The present invention provides a method for optimizing supervision tasks and dynamic resource management based on deep learning, which solves the problem that the task scheduling method of machine learning in the prior art lacks flexibility and versatility and is difficult to meet the diversified needs in a complex computing network environment. It realizes the reasonable analysis of the dependencies and priorities between tasks and the flexible allocation of resources, thereby improving the efficiency and stability of the overall task scheduling.

[0008] The present invention provides a supervision task optimization and dynamic resource management method based on deep learning, the method comprising:

[0009] Collect various operating indicator data of the big data platform in real time;

[0010] Input the various operating indicator data into the trained supervision model to obtain the abnormal probability score corresponding to the various operating indicator data in each time step; wherein the supervision model includes an embedding layer, a transformer encoder and an anomaly detector connected in sequence; the embedding layer is used to convert the various operating indicator data into corresponding low-dimensional vectors; the transformer encoder is used to calculate the correlation of the low-dimensional vectors using a multi-head attention mechanism to obtain an output feature matrix; the anomaly detector is used to output the abnormal probability score corresponding to the various operating indicator data in each time step according to the output feature matrix;

[0011] The current scheduling queue of the big data platform is adjusted according to the abnormal probability score of each time step.

[0012] In a possible implementation, the embedding layer converts the various operating indicator data into corresponding low-dimensional vectors, including:

[0013] Each of the operating indicator data is converted into a corresponding low-dimensional vector of fixed length.

[0014] In a possible implementation, the transformer encoder uses a multi-head attention mechanism to perform correlation calculation on the low-dimensional vector to obtain an output feature matrix, including:

[0015] Dividing the low-dimensional vector into a plurality of subspaces, each of the subspaces including a portion of the low-dimensional vector;

[0016] Calculating a first correlation between features in each subspace by a dot product operation, and scaling the first correlation by a scaling factor to obtain a correlation in the subspace;

[0017] Calculating the correlation between each subspace and the remaining subspaces by a dot product operation to obtain a second correlation, and scaling the second correlation by a scaling factor to obtain a correlation between the subspaces;

[0018] Combining the correlation within the subspace with the correlation between the subspaces to obtain an attention matrix;

[0019] The low-dimensional vector is weighted using the attention matrix to obtain an output feature matrix.

[0020] In a possible implementation, the anomaly detector includes: a plurality of fully connected layers, an activation function, and an output layer;

[0021] The multiple fully connected layers are used to perform linear transformation on the output feature matrix, and extract abnormal features from the output feature matrix after the linear transformation to obtain an abnormal feature vector sequence;

[0022] The activation function is used to judge each element in the abnormal feature vector sequence, if the element is less than a preset value, then modify the element in the abnormal feature vector sequence to 0, if the element is greater than or equal to the preset value, then do not modify the element in the abnormal feature vector sequence, until the first feature vector is obtained;

[0023] The output layer is used to calculate the abnormal probability score corresponding to each item of operating indicator data in each time step according to the first eigenvector.

[0024] In a possible implementation, the activation function is a ReLU function.

[0025] In a possible implementation, the output layer calculates the abnormal probability score corresponding to each item of operating indicator data in each time step according to the first eigenvector, including:

[0026] According to the first eigenvector, the abnormal probability score corresponding to each operation indicator data in each time step is calculated using the abnormal probability score formula;

[0027] The abnormal probability score formula is expressed as:

[0028] p=σ(W 0 ReLU(W f ·z+b f )+b 0 );

[0029] Among them, W 0 represents the weight matrix of the output layer; b 0 represents the bias vector of the output layer; W f represents the weight matrix of the fully connected layer; b f represents the bias vector of the fully connected layer; σ represents the sigmoid activation function; p represents the output abnormal probability score vector, and each element p i represents the abnormal probability of the corresponding time step; ReLU(·) represents the ReLU function; z represents the first eigenvector.

[0030] In a possible implementation, the training process of the supervision model includes:

[0031] The validation set and training set are divided multiple times through cross-validation to obtain multiple different training sets and validation sets;

[0032] The supervision model is trained according to a plurality of different training sets and validation sets to obtain a trained supervision model.

[0033] In a possible implementation, the training process of the supervision model further includes: optimizing the supervision model using a sequence modeling auxiliary loss device.

[0034] In a possible implementation, adjusting the current scheduling queue of the big data platform according to the abnormal probability score of each time step includes:

[0035] Determine the magnitude relationship between the abnormal probability and the preset abnormal threshold. If the abnormal probability is greater than or equal to the preset abnormal threshold, the big data platform postpones the scheduling task corresponding to the abnormal probability.

[0036] If the abnormal probability is less than the preset abnormal threshold, the big data platform does not adjust the scheduling queue.

[0037] One or more technical solutions provided in the present invention have at least the following technical effects or advantages:

[0038] The present invention realizes adaptive prediction of task execution progress through the supervision model, combines large model technology to analyze the characteristics of supervision task execution, dynamically checks abnormal information, and effectively improves the abnormality detection and processing capabilities of the task scheduling system; at the same time, it can also reasonably analyze the dependencies and priorities between tasks, flexibly allocate resources, thereby improving the efficiency and stability of the overall task scheduling, and realizes automatic resource allocation and environment reconstruction in combination with reasonable scheduling strategies and trigger conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A flowchart of the steps of the deep learning-based supervision task optimization and dynamic resource management method provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] The present invention provides a supervision task optimization and dynamic resource management method based on deep learning, such as Figure 1 As shown, the method includes the following steps S101 to S103.

[0042] S101, collect various operating indicator data of the big data platform in real time.

[0043] Here, the various operating indicator data refer to indicator data such as service type, analysis object, analysis scope, CPU frequency and utilization, memory usage, database interaction, and external interface call.

[0044] S102, inputting each operation indicator data into the trained supervision model to obtain the abnormal probability score corresponding to each operation indicator data in each time step. The supervision model includes an embedding layer, a transformer encoder and an anomaly detector connected in sequence.

[0045] The embedding layer converts various operating indicator data into corresponding low-dimensional vectors.

[0046] Here, each operating indicator data is converted into a corresponding low-dimensional vector of fixed length.

[0047] Exemplarily, the original data is converted into a low-dimensional vector representation to prepare for subsequent model processing. A low-dimensional vector is a representation form that compresses the original data into a smaller dimensional space. In this scheme, the original big data platform operation indicator data is converted into a low-dimensional vector of fixed length to facilitate subsequent machine learning model processing.

[0048] The transformer encoder uses a multi-head attention mechanism to calculate the correlation of low-dimensional vectors and obtain the output feature matrix.

[0049] Here, the transformer encoder is used to calculate the relevance of the low-dimensional vector using the multi-head attention mechanism to obtain the output feature matrix, including:

[0050] (1) Divide the low-dimensional vector into multiple subspaces, each subspace includes a part of the low-dimensional vector;

[0051] (2) calculating the first correlation between the features in each subspace by a dot product operation, and scaling the first correlation by a scaling factor to obtain the correlation within the subspace;

[0052] (3) calculating the correlation between each subspace and the remaining subspaces by a dot product operation to obtain a second correlation, and scaling the second correlation by a scaling factor to obtain a correlation between the subspaces;

[0053] (4) Merge the correlation within the subspace with the correlation between subspaces to obtain the attention matrix;

[0054] (5) Use the attention matrix to weight the low-dimensional vector and obtain the output feature matrix.

[0055] For example, an architecture based on a multi-head attention mechanism is used to capture the complex dependencies between features in the input sequence and enhance the supervision model's ability to process input data. The multi-head attention mechanism is a key component of the Transformer model. It captures the complex dependencies between features in the input sequence by dividing the input features into multiple subspaces and calculating the correlation between each subspace. The subspace correlation calculation process is as follows:

[0056] ① Subspace partitioning: Divide the input feature matrix into multiple subspaces, each of which contains a part of the input features.

[0057] ② Correlation calculation within subspace: In each subspace, the correlation between features is calculated by calculating the dot product between features. The dot product can measure the linear correlation between features. The result of the dot product calculation is scaled by a scaling factor to improve the stability of the calculation.

[0058] ③ Calculation of correlation between subspaces: After calculating the correlation within a subspace, the correlation between different subspaces is calculated by calculating the dot product of the feature matrix within the subspace with the feature matrices in all other subspaces. The calculation results of the correlation between subspaces are also scaled by the scaling factor.

[0059] ④ Merge subspace correlations: Merge all the correlations within subspaces and between subspaces to get the final attention matrix. This attention matrix can be regarded as a weighting of the original feature matrix, where each feature is given a different weight according to its correlation in different subspaces.

[0060] ⑤ Feature weighting: Use the attention matrix to weight the original features to obtain a weighted feature sequence. This weighting process can be implemented through matrix multiplication, where the attention matrix is ​​used as the weight matrix and the original feature matrix is ​​used as the input matrix.

[0061] The above mechanism can enhance the supervision model's ability to process input data because it allows the supervision model to focus on multiple aspects of the input data at the same time, thereby better understanding and modeling the complexity of the data.

[0062] The output of the transformer encoder has an output feature matrix with higher semantic representation ability. The feature sequence output by the transformer encoder refers to the sequence processed by the multi-head attention mechanism. These features represent the higher-level semantic information of the input data. The feature sequence of each time step contains a weighted representation of each feature in the input data. These representations can capture the key information in the data and provide a basis for anomaly detection.

[0063] The anomaly detector is used to output the anomaly probability score corresponding to each operating indicator data in each time step according to the output feature matrix.

[0064] Here, the anomaly detector includes: multiple fully connected layers, activation functions and output layers.

[0065] Multiple fully connected layers are used to perform linear transformation on the output feature matrix, extract abnormal features from the output feature matrix after linear transformation, and obtain an abnormal feature vector sequence.

[0066] The activation function is used to judge each element in the abnormal feature vector sequence. If the element is less than a preset value, the element in the abnormal feature vector sequence is modified to 0. If the element is greater than or equal to the preset value, the element in the abnormal feature vector sequence is not modified until the first feature vector is obtained. Here, the activation function is the ReLU function.

[0067] The output layer uses multiple neurons to calculate the abnormal probability scores corresponding to the various operating indicator data in each time step according to the first eigenvector. Specifically, the output layer uses multiple neurons to calculate the abnormal probability scores corresponding to the various operating indicator data in each time step according to the first eigenvector, including:

[0068] According to the first eigenvector, the abnormal probability score corresponding to each operating indicator data in each time step is calculated using the abnormal probability score formula;

[0069] The abnormal probability score formula is expressed as:

[0070] p=σ(W 0 ReLU(W f ·z+b f )+b 0 );

[0071] Among them, W 0 represents the weight matrix of the output layer; b 0 Represents the bias vector of the output layer; W f represents the weight matrix of the fully connected layer; b f represents the bias vector of the fully connected layer; σ represents the sigmoid activation function; p represents the output abnormal probability score vector, and each element p i represents the abnormal probability of the corresponding time step; ReLU(·) represents the ReLU function; z represents the first eigenvector.

[0072] Exemplarily, the anomaly detector has multiple fully connected layers, activation functions, and output layers, which output anomaly probability scores for each time step. The fully connected layer is used to further compress and transform the output feature matrix of the transformer encoder to extract key anomaly features. The activation function is ReLU, which is used to increase the nonlinear expression ability of the supervision model so that the supervision model can better learn the complex patterns of the data. The output layer is used to output the anomaly probability scores for each time step, which can be used to determine whether the indicator data of the current time step is an abnormal situation. The relationship between the three is:

[0073] ①Fully connected layer:

[0074] Input: Output feature matrix from the transformer encoder.

[0075] Processing: The fully connected layer is responsible for compressing and transforming the feature sequence output by the transformer encoder. It extracts key abnormal features by performing a linear transformation on the feature sequence. The fully connected layer can be regarded as a feature extraction and compression module. Its purpose is to map high-dimensional features to a new low-dimensional space to facilitate subsequent processing.

[0076] ②Activation function (ReLU):

[0077] Input: Output from the fully connected layer.

[0078] Processing: The activation function is used to increase the nonlinear expression ability of the supervision model. In the anomaly detection head, ReLU is usually used as the activation function because it can introduce nonlinearity, allowing the supervision model to better capture complex patterns in the data. The ReLU function outputs the input value when the input is greater than 0, and outputs 0 when the input is less than or equal to 0. This can avoid the problem of gradient disappearance during training and help the convergence of the supervision model.

[0079] ③Output layer:

[0080] Input: Output from the activation function.

[0081] Processing: The output layer is responsible for outputting the anomaly probability score for each time step. In this scheme, the goal of the anomaly detection head is to determine whether the indicator data at the current time step is an abnormal situation, so the output layer needs to be able to output an anomaly probability score, which can be used as a basis for judgment. The output of the output layer is usually a feature sequence processed by an activation function, and the anomaly probability score is calculated through one or more neurons (nodes).

[0082] S103, adjusting the current scheduling queue of the big data platform according to the abnormal probability score of each time step.

[0083] Specifically, in step S103, the current scheduling queue of the big data platform is adjusted according to the abnormal probability score of each time step, including:

[0084] S1031, determine the relationship between the abnormal probability and the preset abnormal threshold. If the abnormal probability is greater than or equal to the preset abnormal threshold, the big data platform will postpone the scheduling task corresponding to the abnormal probability.

[0085] S1032: If the abnormal probability is less than the preset abnormal threshold, the big data platform does not adjust the scheduling queue.

[0086] Exemplarily, in a specific embodiment provided by the present invention, during online anomaly detection, the operation index data of the big data platform is collected in real time, and after feature engineering processing, it is input into the trained supervision model. The supervision model has learned the characteristics of the normal operation state during the training process, and can calculate the abnormal probability score of each time step according to the input data. These scores reflect the degree of abnormality of the data at the time step. The higher the score, the more likely the data is to be abnormal. In order to determine which scores are considered abnormal, the system needs to set a preset abnormal threshold. This preset abnormal threshold is set by analyzing historical data and expert experience. Through statistical analysis, the distribution of data under normal circumstances can be understood, so as to determine a reasonable preset abnormal threshold. This preset abnormal threshold should be able to effectively detect abnormal situations without causing too many false alarms. Once the abnormal probability score calculated by the supervision model exceeds the preset abnormal threshold, the system will trigger an abnormal alarm and notify relevant personnel. The alarm information will include the specific time when the abnormality occurs, detailed information of the abnormal data, and the abnormal probability score. In addition, the system will also feed back the abnormal detection results to the task scheduling system so that the tasks can be adjusted and optimized accordingly.

[0087] Finally, continue to pay attention to the performance of the supervision model in the production environment, update training samples in a timely manner, analyze and optimize missed or false positive anomalies, and ensure that the supervision model can always adapt to the latest operating status of the big data platform.

[0088] In a specific embodiment provided by the present invention, the training process of the supervision model includes:

[0089] (1) Divide the validation set and training set multiple times through cross-validation to obtain multiple different training sets and validation sets;

[0090] (2) The supervision model is trained according to multiple sets of different training sets and validation sets to obtain a trained supervision model.

[0091] The training process of the supervision model also includes: optimizing the supervision model using the sequence modeling auxiliary loss.

[0092] Exemplarily, during the training process, hyperparameters, regularization techniques, data enhancement strategies, etc. are tuned to improve the generalization ability of the supervision model. At the same time, attention is paid to the reasoning efficiency of the supervision model, and the supervision model compression, quantization and other technologies are used to optimize it so that it can be deployed in the actual big data platform. Hyperparameters, regularization techniques and data enhancement strategies all have an important impact on the performance and generalization ability of the supervision model, so they need to be tuned. Hyperparameters include learning rate, batch size, number of layers, number of neurons, activation function type, etc. Regularization techniques such as L1 regularization (Lasso), L2 regularization (Ridge) and dropout in deep learning are used to prevent the supervision model from overfitting. Data enhancement strategies transform and modify the original data by rotating, scaling, cropping, color conversion, adding noise, etc. to generate more training data. Hyperparameter tuning usually evaluates the performance of the supervision model under different configurations by dividing the validation set and training set multiple times on the training data through cross-validation. The selection and strength of the regularization technique are determined by observing the training error and validation error of the supervision model. The data augmentation strategy is tuned to find the best augmentation solution by trying different combinations and strengths of augmentation techniques.

[0093] Based on the anomaly probability score, the supervision model predicts the indicator data of the next time step, and calculates the auxiliary loss based on the difference between the actual data for supervision model optimization. This method enables the supervision model to learn the temporal characteristics of the data while detecting anomalies, thus improving the accuracy and robustness of anomaly detection.

[0094] In a specific embodiment provided by the present invention, various operating index data are expressed as:

[0095] Task type: A, B, C; Analysis object: X, Y, Z; Analysis scope: small, medium, large; CPU frequency: 2.5GHz, 3.0GHz, 3.5GHz; CPU utilization: 20%, 50%, 80%; Memory usage: 4GB, 8GB, 16GB; Number of database interactions: 100 times, 200 times, 300 times; Number of external interface calls: 50 times, 100 times, 150 times.

[0096] Through dimensionality reduction techniques, the above indicators are converted into low-dimensional vectors. For example, it is possible to map each indicator into a 3-dimensional space. Here is a simplified example:

[0097] Task type A is represented by: [0.1, 0.2, 0.3]; analysis object X is represented by: 0.4, 0.5, 0.6]; small analysis range is represented by: [0.7, 0.8, 0.9]; CPU frequency 2.5GHz is represented by: [0.2, 0.3, 0.4]; CPU utilization 20% is represented by: [0.5, 0.6, 0.7]; memory usage 4GB is represented by: [0.8, 0.9, 1.0]; database interaction times 100 times is represented by: [0.3, 0.4, 0.5]; external interface call times 50 times is represented by: [0.6, 0.7, 0.8];

[0098] Now, suppose these low-dimensional vectors are part of time series data, for example, collected at several consecutive time points. Concatenate these vectors to form a sequence:

[0099] Time point 1: [0.1, 0.2, 0.3, 0.4, 0.5, 0.8, 0.3, 0.6];

[0100] Time point 2: [0.2, 0.3, 0.4, 0.5, 0.6, 0.9, 0.4, 0.7];

[0101] Time point 3: [0.7, 0.8, 0.9, 0.6, 0.7, 1.0, 0.5, 0.8];

[0102] These feature sequences can be combined into a feature matrix, where each row represents a feature vector at a time point:

[0103] During the calculation of the multi-head attention mechanism, the feature matrix will be processed to capture the complex dependencies between features. The following are the processing steps:

[0104] Subspace partitioning: Each row of the feature matrix is ​​divided into multiple subspaces. For example, 8 features can be divided into 4 subspaces, each containing 2 features.

[0105] Correlation calculation within subspace: Within each subspace, calculate the correlation between features. For example, for the first subspace, calculate the dot product between the first and second features.

[0106] Correlation calculation between subspaces: Calculate the correlation between different subspaces. For example, calculate the dot product of the features of the first subspace with the features of all other subspaces.

[0107] Merge subspace correlations: Merge all correlations within and between subspaces to form the final attention matrix.

[0108] Feature weighting: Use the attention matrix to weight the original features to obtain the output feature matrix. The specific output feature matrix is ​​expressed as:

[0109]

[0110] The output feature matrix contains the feature sequence processed by the multi-head attention mechanism, and each feature is assigned different weights according to its relevance in different subspaces. This weighted representation can capture the key information in the data and provide a basis for subsequent anomaly detection.

[0111] The preset value in the anomaly detector is set to 0. Suppose there is a feature vector [0.12, -0.05, 0.34, 0.02. After applying the ReLU function: ReLU(0.12) = 0.12 (because 0.12>0); ReLU(-0.05) = 0 (because -0.05<=0); ReLU(0.34) = 0.34 (because 0.34>0); ReLU(0.02) = 0.02 (because 0.02>0); Therefore, the first feature vector after ReLU processing will be [0.12, 0, 0.34, 0.02].

[0112] Then, the abnormal probability score corresponding to each operation indicator data is calculated according to the abnormal probability score formula. The big data platform adjusts the current scheduling queue according to the abnormal probability score of each time step.

[0113] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between each embodiment can be referred to each other, and each embodiment focuses on the differences from other embodiments. All or part of the present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, mobile communication terminals, multi-processor systems, microprocessor-based systems, programmable electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0114] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the present invention.

Claims

1. A supervision task optimization and dynamic resource management method based on deep learning, characterized in that: include: Collect various operating indicator data of the big data platform in real time; Input the various operating indicator data into the trained supervision model to obtain the abnormal probability score corresponding to the various operating indicator data in each time step; wherein the supervision model includes an embedding layer, a transformer encoder and an anomaly detector connected in sequence; the embedding layer is used to convert the various operating indicator data into corresponding low-dimensional vectors; the transformer encoder is used to calculate the correlation of the low-dimensional vectors using a multi-head attention mechanism to obtain an output feature matrix; the anomaly detector is used to output the abnormal probability score corresponding to the various operating indicator data in each time step according to the output feature matrix; The current scheduling queue of the big data platform is adjusted according to the abnormal probability score of each time step.

2. The method for optimizing supervision tasks and managing dynamic resources based on deep learning according to claim 1, characterized in that: The embedding layer converts the various operating indicator data into corresponding low-dimensional vectors, including: Each of the operating indicator data is converted into a corresponding low-dimensional vector of fixed length.

3. The method for optimizing supervision tasks and managing dynamic resources based on deep learning according to claim 1 is characterized in that: The transformer encoder uses a multi-head attention mechanism to perform correlation calculation on the low-dimensional vector to obtain an output feature matrix, including: Dividing the low-dimensional vector into a plurality of subspaces, each of the subspaces including a portion of the low-dimensional vector; Calculating a first correlation between features in each subspace by a dot product operation, and scaling the first correlation by a scaling factor to obtain a correlation in the subspace; Calculating the second correlation between each subspace and the remaining subspaces by a dot product operation, and scaling the second correlation by a scaling factor to obtain the correlation between the subspaces; Combining the correlation within the subspace with the correlation between the subspaces to obtain an attention matrix; The low-dimensional vector is weighted using the attention matrix to obtain an output feature matrix.

4. The method for optimizing supervision tasks and managing dynamic resources based on deep learning according to claim 1, characterized in that: The anomaly detector comprises: a plurality of fully connected layers, an activation function and an output layer; The multiple fully connected layers are used to perform linear transformation on the output feature matrix, and extract abnormal features from the output feature matrix after the linear transformation to obtain an abnormal feature vector sequence; The activation function is used to judge each element in the abnormal feature vector sequence, if the element is less than a preset value, then modify the element in the abnormal feature vector sequence to 0, if the element is greater than or equal to the preset value, then do not modify the element in the abnormal feature vector sequence, until the first feature vector is obtained; The output layer is used to calculate the abnormal probability score corresponding to each item of operating indicator data in each time step according to the first eigenvector.

5. The method for optimizing supervision tasks and managing dynamic resources based on deep learning according to claim 4 is characterized in that: The activation function is the ReLU function.

6. The method for optimizing supervision tasks and managing dynamic resources based on deep learning according to claim 4 is characterized in that: The output layer calculates the abnormal probability score corresponding to each operation indicator data in each time step according to the first eigenvector, including: According to the first eigenvector, the abnormal probability score corresponding to each operation indicator data in each time step is calculated using the abnormal probability score formula; The abnormal probability score formula is expressed as: p=σ(W0 ReLU(W f ·z+b f )+b0); Among them, W0 represents the weight matrix of the output layer; b0 represents the bias vector of the output layer; W f represents the weight matrix of the fully connected layer; b f represents the bias vector of the fully connected layer; σ represents the sigmoid activation function; p represents the output abnormal probability score vector, and each element p i represents the abnormal probability of the corresponding time step; ReLU(·) represents the ReLU function; z represents the first eigenvector.

7. The method for optimizing supervision tasks and managing dynamic resources based on deep learning according to claim 1, characterized in that: The training process of the supervision model includes: The validation set and training set are divided multiple times through cross-validation to obtain multiple different training sets and validation sets; The supervision model is trained according to a plurality of different training sets and validation sets to obtain a trained supervision model.

8. The method for optimizing supervision tasks and managing dynamic resources based on deep learning according to claim 7 is characterized in that: The training process of the supervision model also includes: optimizing the supervision model using a sequence modeling auxiliary loss device.

9. The method for optimizing supervision tasks and managing dynamic resources based on deep learning according to claim 1, characterized in that: The current scheduling queue of the big data platform is adjusted according to the abnormal probability score of each time step, including: Determine the magnitude relationship between the abnormal probability and the preset abnormal threshold. If the abnormal probability is greater than or equal to the preset abnormal threshold, the big data platform postpones the scheduling task corresponding to the abnormal probability. If the abnormal probability is less than the preset abnormal threshold, the big data platform does not adjust the scheduling queue.

Images