An abnormal resource occupation real-time early warning method and system based on deep learning

Abstract

The application relates to the technical field of data processing, and particularly discloses an abnormal resource occupation real-time early warning method and system fusing deep learning, which comprises the following steps: collecting a calling event stream of a system, constructing a layered dynamic resource directed graph based on the calling event stream; based on the layered dynamic resource directed graph, optimizing and extracting topological features by using a layered graph attention network, introducing a time perception mechanism based on the topological features to generate a time sequence topological graph embedding sequence describing the evolution of a system resource interaction mode; based on the time sequence topological graph embedding sequence, using a hierarchical variational autoencoder model combined with an abnormal reconstruction mechanism to guide optimization training, calculating the deviation degree of a current interaction mode from a normal and healthy mode to obtain a mode abnormal score sequence; and analyzing the continuous deviation of the mode abnormal score to give a real-time early warning for abnormal resource occupation before abnormality occurs in macroscopic resource indexes.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and system for real-time early warning of abnormal resource usage that integrates deep learning. Background Technology

[0002] With the continuous expansion of cloud computing and distributed systems, abnormal system resource usage has become a critical issue affecting service stability. Early anomaly detection methods mainly relied on threshold rules and statistical indicators (such as CPU utilization and memory usage), achieving anomaly detection by setting static thresholds for resource indicators. While these methods are simple and easy to use, they struggle to capture the dynamic dependencies and interaction patterns between multiple resources in complex systems, resulting in high false positive and false negative rates.

[0003] Currently, most existing methods for early warning of abnormal resource usage analyze resources at a single level, such as process call chains or resource indicator sequences, lacking the ability to uniformly model cross-level resource interactions within the system. Furthermore, traditional methods often rely solely on macroscopic indicator anomalies as a basis for judgment, making it difficult to provide effective early warnings before actual resource anomalies occur. They also lack sensitivity to the differences between normal and abnormal patterns, failing to accurately identify potential anomalies in complex and ever-changing environments, leading to delayed warnings or frequent false alarms. Summary of the Invention

[0004] To address the problems in existing technologies, this invention provides a real-time early warning method and system for abnormal resource occupancy that integrates deep learning. This method can accurately model cross-level resource interaction relationships, capture the evolution trend of system resource interaction patterns, and achieve early warning of anomalies.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] In one aspect, this invention proposes a real-time early warning method for abnormal resource occupancy that integrates deep learning, the method comprising:

[0007] The system collects the call event flow and constructs a hierarchical dynamic resource directed graph based on the call event flow;

[0008] Based on the hierarchical dynamic resource directed graph, the hierarchical graph attention network is used for optimization and its topological features are extracted. Based on the topological features, a time-aware mechanism is introduced to generate a time-series topological graph embedding sequence that describes the evolution of the system's resource interaction mode.

[0009] Based on the temporal topological graph embedding sequence, a hierarchical variational autoencoder model combined with an anomaly reconstruction mechanism is used to guide optimization training, calculate the degree of deviation between the current interaction mode and the normal healthy mode, and obtain the pattern anomaly score sequence.

[0010] The system analyzes continuous deviations in abnormal scores and provides real-time warnings of abnormal resource usage before macro-resource indicators show abnormalities.

[0011] Optionally, the construction of the hierarchical dynamic resource directed graph includes:

[0012] A hierarchical dynamic resource directed graph includes a set of nodes and a set of directed superedges;

[0013] The node set includes process layer nodes and resource layer nodes, and the directed superedge set includes directed superedges and directed superedges;

[0014] Assign weights to different directed hyperedges based on the event call flow to construct a hierarchical directed hypergraph for dynamic resource operations.

[0015] Optionally, the optimization and extraction of topological features using a hierarchical graph attention network includes:

[0016] For each node in the directed hypergraph of hierarchical dynamic resource operations, a high-dimensional feature vector is randomly obtained initially. The high-dimensional feature vectors of all nodes in different levels are collected, as well as the weights of the directed edges between nodes in the same level. For each node in each level, all its neighboring nodes are traversed to obtain the attention weights between the node and its neighboring nodes.

[0017] Update the node characteristics of each level in the directed hypergraph for hierarchical dynamic resource operations;

[0018] After updating the feature representation, for the node pairs that need to interact between layers, the inter-layer attention weights between the node pairs are obtained again. The inter-layer attention weights are then used to perform weighted fusion of the features of cross-layer nodes to obtain node features containing inter-layer interaction information.

[0019] The hierarchical dynamic resource directed graph is optimized by iteratively using the l-layer network of the hierarchical graph attention network. The optimized hierarchical dynamic resource directed graph is then input back into the hierarchical graph attention network to obtain topological features.

[0020] Optionally, the generation of the temporal topology graph embedding sequence describing the evolution of system resource interaction patterns includes:

[0021] Pooling operations are performed based on topological features to generate initial graph embedding information for the current time window;

[0022] By introducing a Long Short-Term Memory (LSTM) structure and fusing the initial graph embedding information of the historical window with the LSTM structure, the hidden state of the LSTM in time can be obtained.

[0023] The initial graph embedding information of the current window is fused with the LSTM hidden state to obtain the final time-aware graph embedding vector.

[0024] Arrange the final graph embedding vectors of all time windows in chronological order to obtain a temporal topology graph embedding sequence describing the evolution of system resource interaction patterns.

[0025] Optionally, the hierarchical variational autoencoder model includes a global encoder, a global decoder, and local encoders and local decoders;

[0026] The global encoder maps the temporal topology graph embedding sequence to a global latent distribution, reconstructs the global latent representation, and obtains a global trend representation of the temporal topology graph embedding sequence;

[0027] The local encoder maps the temporal topology graph embedding sequence of each time window to a local latent distribution;

[0028] The local decoder reconstructs the graph embedding sequence corresponding to the local latent representation for the time window, and obtains a local trend representation that measures the short-term pattern fit of a single time window.

[0029] Optionally, the optimization training guided by using a hierarchical variational autoencoder model combined with an anomaly reconstruction mechanism includes:

[0030] The time-series topology graph embedding sequences under normal and abnormal conditions are collected, and a global loss function is designed that simultaneously considers the fitting of normal samples and the deviation of abnormal samples. The global loss function includes variational lower bound loss based on normal samples and abnormal loss based on abnormal guiding coefficients and abnormal samples.

[0031] During training, the components of variational lower bound loss and outlier loss are simultaneously optimized using the backpropagation algorithm.

[0032] Optionally, obtaining the pattern anomaly score sequence includes:

[0033] For the graph embedding sequence of the current time window, the degree of deviation from the normal healthy pattern is comprehensively measured to finally obtain the pattern abnormality score;

[0034] For each time step, the corresponding pattern anomaly score is calculated for the graph temporal topology embedding sequence, resulting in a pattern anomaly score sequence.

[0035] In another aspect, the present invention proposes a real-time early warning system for abnormal resource occupancy that integrates deep learning, implemented according to any one of claims 1-7, comprising:

[0036] The data acquisition module is used to collect the system's call event stream and construct a hierarchical dynamic resource directed graph based on the call event stream;

[0037] The topology analysis module is used to optimize and extract topological features of a hierarchical dynamic resource directed graph using a hierarchical graph attention network. Based on the topological features, a time-aware mechanism is introduced to generate a time-series topology graph embedding sequence that describes the evolution of the system's resource interaction patterns.

[0038] The anomaly quantification module is used to optimize training based on the time-series topology graph embedded sequence, using a hierarchical variational autoencoder model combined with an anomaly reconstruction mechanism to calculate the degree of deviation between the current interaction mode and the normal healthy mode, and obtain the pattern anomaly score sequence.

[0039] The early warning output module is used to analyze the continuous deviation of the abnormal score of the mode and to provide real-time early warning of abnormal resource usage before the macro resource indicators become abnormal.

[0040] In a third aspect, the present invention provides an electronic device including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the real-time early warning method for abnormal resource occupancy fused with deep learning as described in any one aspect.

[0041] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the real-time early warning method for abnormal resource occupancy fused with deep learning as described in the first aspect.

[0042] The present invention has the following beneficial effects:

[0043] 1. By constructing a hierarchical dynamic directed hypergraph that includes process and resource layers, it is possible to accurately characterize the complex relationships of cross-level resource operations in the system, such as process-mapped memory and multi-process resource sharing. Furthermore, by utilizing the hierarchical graph attention mechanism, it achieves dual aggregation of intra-layer and inter-layer node features, significantly enhancing the representation capability of topological features.

[0044] 2. By introducing an LSTM structure and a time-aware fusion mechanism based on the extracted graph structure features, the static graph embedding is extended into a temporal embedding sequence, thereby realizing the modeling of the long-term evolution trend of system resource interaction patterns and providing more discriminative temporal features for subsequent anomaly detection.

[0045] 3. By designing a hierarchical variational autoencoder that includes global and local encoders and decoders, and combining it with a loss function trained on both normal and abnormal samples, the model can explicitly increase the reconstruction error of abnormal patterns while fitting healthy patterns, thereby effectively identifying subtle abnormal deviations and improving the robustness and generalization ability of the model.

[0046] 4. By continuously analyzing the abnormal score sequence of the pattern, potential anomalies in resource interaction patterns can be identified in advance before macro resource indicators show obvious abnormalities, thereby achieving true real-time early warning, providing valuable response time for system operation and maintenance, and reducing the failure rate and business impact. Attached Figure Description

[0047] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0048] Figure 1 This is a flowchart illustrating the steps of a real-time early warning method for abnormal resource usage that integrates deep learning, as proposed in this invention.

[0049] Figure 2 This is a block diagram of a real-time early warning system for abnormal resource usage that integrates deep learning, as proposed in this invention.

[0050] Figure 3 This is a schematic diagram of the structure of an electronic device used to implement the methods and system embodiments of this application. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] Example 1: As Figure 1 As shown, the first embodiment of the present invention proposes a real-time early warning method for abnormal resource occupancy that integrates deep learning. The method specifically includes the following steps:

[0053] Step S101: Collect the system's call event stream and construct a hierarchical dynamic resource directed graph based on the call event stream;

[0054] Step S102: Based on the hierarchical dynamic resource directed graph, optimize it using a hierarchical graph attention network and extract its topological features. Based on the topological features, introduce a time-aware mechanism to generate a time-series topological graph embedding sequence that describes the evolution of the system's resource interaction mode.

[0055] Step S103: Based on the temporal topology graph embedding sequence, use a hierarchical variational autoencoder model combined with an anomaly reconstruction mechanism to guide optimization training, calculate the degree of deviation between the current interaction mode and the normal healthy mode, and obtain the mode anomaly score sequence.

[0056] Step S104: Analyze the continuous deviation of the abnormal score of the analysis mode, and provide real-time early warning of abnormal resource usage before the macro resource indicators become abnormal.

[0057] In step S101 above, the invocation event stream of the acquisition system, and the construction of a hierarchical dynamic resource directed graph based on the invocation event stream, specifically include:

[0058] System call event data, including the call initiator (PID), call type, and timestamp, is collected through kernel tracing tools (such as ftrace and perf in Linux, and ETW in Windows) and user-space monitoring frameworks (such as eBPF programs and SystemTap scripts) to form the system call event stream D.

[0059] The call type includes the process user-mode stack, process resource quota, and system resource load data at the time of the call.

[0060] In the above embodiments, the process of constructing a hierarchical dynamic resource directed graph based on the call event stream is as follows:

[0061] A hierarchical dynamic resource directed graph includes a node set V and a directed superedge set E;

[0062] Based on the event flow D, we first perform node layering and refinement to obtain the node set V of the layered dynamic resource directed graph, including process layer nodes. and resource layer nodes ;

[0063] Process layer nodes are represented as Corresponding to the process PID in the event stream D, the process is subcategorized according to its business type, such as Web service process, database process, and background task process, and child nodes are generated. At the same time, the user-mode stack information, used memory amount, and number of file descriptors of the process are extracted from the stack (user-mode stack) and quota (process resource quota) fields of the event stream as attributes of the process layer node.

[0064] Resource layer nodes are represented as The child nodes are divided into different sub-layers based on the resource type (memory, file, network socket, CPU time slice) in the event stream D, including memory sub-layer and file sub-node. The resource layer node extracts the load information of the system to which the resource belongs from the load (system resource load) field of the event stream as the node's attribute.

[0065] Define a set of directed hyperedges E, which includes directed hyperedges. and directed superedge ;

[0066] Traditional directed edges are one-to-one node connections, while directed hyperedges can represent operations from one source node to multiple target nodes, or operations from multiple source nodes to one target node.

[0067] When a process maps multiple memory pages simultaneously using the mmap system call, it constructs hyper-edges from the process node to the multiple memory page nodes. ;

[0068] When multiple processes access the same memory segment through shared memory, a directed hyperedge is constructed from the multiple process nodes to the shared memory segment nodes. ;

[0069] Then, weights are assigned to different directed superedges. The weights are assigned to the edges based on factors such as the criticality of the calls in the event flow D and the scope of the impact of resource operations. For example, the weight of frequent and critical memory allocation calls such as malloc is higher than that of ordinary getpid calls. When the system resource load is high, the weight of the edges of resource operations is increased because resource interactions are more likely to cause exceptions at this time.

[0070] The final constructed hierarchical directed hypergraph of dynamic resource operations The node set V contains process layer nodes. and resource layer nodes The edge set E contains weighted directed superedges. and directed superedge It can more accurately depict the complex interactions of system resources.

[0071] Step S102 above, based on the hierarchical dynamic resource directed graph, optimizes and extracts its topological features using a hierarchical graph attention network, and introduces a time-aware mechanism based on the topological features to generate a time-series topological graph describing the evolution of the system's resource interaction patterns and embeds it into the sequence. The process of optimizing and extracting its topological features using a hierarchical graph attention network is as follows:

[0072] First, the directed hypergraph G of hierarchical dynamic resource operations is initialized. For all nodes, they are classified according to their level (process layer, resource layer), and a high-dimensional feature vector is randomly initialized for each node. Based on the above, a high-dimensional feature vector is randomly initialized for each node of the directed hypergraph G that performs hierarchical dynamic resource operations. .

[0073] Suppose the hierarchical graph attention network has l layers. For each layer, collect the high-dimensional feature vectors of all nodes within that layer. and the weights of directed edges between nodes within the layer. For each node in each of its respective levels traverse all its neighbor nodes Get nodes with neighboring nodes Attention weights between .

[0074] Then, the directed hypergraph for hierarchical dynamic resource operations. Update the node features of each level, using the obtained attention weights. The features of neighboring nodes are weighted and aggregated to update the node. The features are represented as follows:

[0075]

[0076] in, For activation function, express The characteristics of neighboring nodes;

[0077] Specifically, through this step, each node aggregates information from its neighboring nodes within the layer, making the node characteristics better reflect the resource interaction relationships within the layer.

[0078] After updating the feature representation, the nodes that need to perform inter-layer interactions are identified. and , Then, obtain the new attention weights between node pairs again, i.e., the inter-layer attention weights. Utilizing inter-layer attention weights The features of cross-layer nodes are weighted and fused to obtain node features that include inter-layer interaction information. ;

[0079] The hierarchical attention network iterates through the l-layer network. The computation of each layer updates the node features based on the node features of the previous layer and obtains the inter-layer attention weights. This further aggregates a wider range of neighbor and cross-layer node information, so that the node features gradually include the topology and resource interaction features of the entire hierarchical dynamic resource operation directed hypergraph, thus completing the optimization of the hierarchical dynamic resource directed graph.

[0080] Based on the optimized hierarchical dynamic resource directed graph Divide time into equal-length time windows For each time window Optimized hierarchical dynamic resource directed graph within The input is fed into a hierarchical graph attention network, which then optimizes the hierarchical dynamic resource directed graph. The structure (nodes, edges and edge weights, node enhancement features) is analyzed, and its topological features are extracted to obtain the topological feature H.

[0081] Specifically, the topological feature H is a high-dimensional feature vector of node v after being updated by the attention mechanism at time window t and level l. It contains key information about the interaction between the node and nodes at the same and different levels within the current time window.

[0082] The process of generating the temporal topology graph embedding sequence describing the evolution of system resource interaction patterns is as follows:

[0083] The temporal topology graph embedding sequence is the entire time window. The inner-layer dynamic resource operation is a global representation of the directed hypergraph, rather than the features of a single node. Therefore, it is necessary to perform a pooling operation on the topological features H. The purpose is to aggregate the features of a large number of nodes into a low-dimensional vector that can represent the entire graph (or the hypergraph within the current time window).

[0084] Based on the topological feature H, a pooling operation is first performed to generate the initial graph embedding information for the current time window, expressed by the formula:

[0085] ;

[0086] in, This indicates a pooling operation. This indicates the embedded information in the initial graph;

[0087] Specifically, the temporal topology graph embedding sequence should reflect the overall characteristics of the hierarchical dynamic resource operation directed graph within the entire time window, rather than the characteristics of a single node. Therefore, it is necessary to perform a pooling operation on the topology features H, aggregating the features of a large number of nodes into a low-dimensional vector that can represent the entire graph, and generating the initial graph embedding for the current time window. ;

[0088] Then, a Long Short-Term Memory (LSTM) structure is introduced. By fusing the initial graph embedding information of the historical window with the LSTM structure, the hidden state of the LSTM at time t is obtained. Then embed information of the initial image of the current window. With LSTM hidden state The fusion process yields the final time-aware graph embedding vector:

[0089]

[0090] in, The fusion coefficient;

[0091] Specifically, by introducing a Long Short-Term Memory (LSTM) structure, which excels at capturing long-term dependencies in time-series data, we can use it to fuse the initial graph embedding information from the historical window to obtain the hidden state of the LSTM at time t. This hidden state contains graph embedding evolution information from the history to the current time window;

[0092] Finally, by arranging the final graph embedding vectors of all time windows in chronological order, we obtain the temporal topology graph embedding sequence describing the evolution of system resource interaction patterns. This time-series topology graph embedding sequence can clearly show the dynamic changes in the system resource interaction pattern over time, represented as: = ;

[0093] Specifically, the temporal topology graph embedding sequence is a time window covering the entire resource occupancy period. The inner-layered dynamic resource operation provides a holistic representation of the directed hypergraph, transforming complex graph structures (containing numerous nodes, edges, and their topological relationships) into low-dimensional, easily processed vector forms. In subsequent processes, various key operations can be performed based on the temporal topological graph embedding sequence, including:

[0094] Anomaly detection: By analyzing the changing patterns of graph embedding vector sequences, deviations between system resource interaction patterns and normal healthy patterns can be identified, thus providing a basis for proactive real-time early warning of abnormal resource usage;

[0095] Pattern analysis can uncover the evolutionary patterns of system resource interaction patterns under different time windows, which helps to deeply understand the system resource usage mechanism and provides a reference for system optimization and resource scheduling.

[0096] Subsequent model input: As input to models such as variational autoencoders, it is used to calculate the degree of deviation between the current interaction mode and the normal mode, and thus obtain the mode anomalous score.

[0097] In step S103 above, based on the temporal topological graph embedding sequence, a hierarchical variational autoencoder model combined with an anomaly reconstruction mechanism is used to guide optimization training, calculate the degree of deviation between the current interaction mode and the normal healthy mode, and obtain the pattern anomaly score sequence, specifically including:

[0098] Embedded sequences of time-aware temporal topology graphs A hierarchical variational autoencoder model is adopted;

[0099] The structure of a hierarchical variational autoencoder model includes: a global encoder and a global decoder, as well as a local encoder and a local decoder;

[0100] The global encoder and global decoder are used to capture the global distribution features of the entire graph embedding vector sequence and learn the long-term trend of system resource interaction under normal and healthy conditions.

[0101] Specifically, the global encoder embeds the temporal topology graph into the sequence. Mapping to global latent distribution ; Representing the global latent representation, the global decoder... Reconstruction is performed to obtain the temporal topology graph embedding sequence. Global trend representation ;

[0102] Local encoders and global decoders embed sequences of temporal topology graphs for each time window. To capture local features of short-term resource interactions;

[0103] Specifically, the local encoder embeds the temporal topology map of each time window into the sequence. Mapping to local latent distribution , This represents the local latent representation, and the local decoder performs processing on the local latent representation. Graph embedding sequence corresponding to time window Reconstruction is performed to obtain a local trend representation that measures the short-term pattern fit for a single time window. .

[0104] In the above embodiments, the process of guiding optimization training in conjunction with the anomaly reconstruction mechanism is as follows:

[0105] Select the time-series topology graph embedding sequence generated when the system is in normal operation. These sequences come from historical periods when the system was stable and resource interactions were normal, and contain the global distribution and local characteristics of resource interactions under normal conditions.

[0106] Simultaneously, by extracting the time-series topology graph embedding sequence corresponding to the abnormal operating state through historical failure cases (such as system crashes and resource exhaustion scenarios), The goal is to enable the model to learn the differences between abnormal and normal patterns;

[0107] Based on the time-series topology graph embedding sequences under normal and abnormal states, a global loss function is designed that simultaneously considers the fitting of normal samples and the deviation of abnormal samples, expressed by the formula:

[0108] ;

[0109] in, This represents the variational lower bound loss for normal samples. This indicates the abnormal loss of outlier samples. Indicates the abnormal guidance coefficient. This represents the embedded sequence of the time-series topology graph under normal operating conditions. This represents the embedded sequence of the time-series topology graph under abnormal operating conditions.

[0110] According to the above formula, the global loss function includes: variational lower bound loss based on normal samples and abnormal loss based on abnormal guiding coefficients and abnormal samples.

[0111] During training, the backpropagation algorithm is used to simultaneously optimize the components of variational lower bound loss and anomaly loss. Based on the product of variational lower bound loss for normal samples and normal sequences, the distribution characteristics of normal resource interaction patterns are learned; based on the product of anomaly guidance coefficient, anomaly loss for anomaly samples, and anomaly sequences, the reconstruction error of anomaly patterns is increased.

[0112] The variational lower bound loss for normal samples ensures the model's ability to extract and reconstruct features from healthy patterns, while the anomalous loss for abnormal samples enhances the model's ability to distinguish abnormal patterns. The anomaly guide coefficient is used to adjust the weight ratio of abnormal samples in the overall loss, balancing the relationship between normal pattern fitting and anomalous pattern recognition.

[0113] Through this bidirectional optimization training mechanism, the model not only learns the characteristics of normal resource interaction patterns, but also improves its sensitivity to abnormal patterns, thereby enhancing the overall accuracy of anomaly detection.

[0114] Furthermore, the variational lower bound loss (ELBO) for normal samples is the core training objective of the variational autoencoder (VAE). It consists of two parts: reconstruction loss and KL divergence. The reconstruction loss measures the ability of the hierarchical VAE model to reconstruct normal samples, i.e., the difference between the graph embedding reconstructed by the decoder from the latent distribution and the original graph embedding. The KL divergence, on the other hand, constrains the latent distribution generated by the hierarchical VAE model to approximate the prior distribution, ensuring the normality of the latent representation and minimizing... It enables hierarchical variational autoencoder models to learn to accurately fit normal patterns; The anomalous loss represents the reconstruction error of the anomalous sample and the KL divergence between the latent distribution and the prior distribution of the anomalous sample. It is the difference between the reconstruction result and the original anomalous sample after the anomalous sample is encoded and decoded by the hierarchical variational autoencoder model. The greater the difference, the stronger the model's maladaptation to anomalous patterns.

[0115] maximize It can make hierarchical variational autoencoder models more sensitive to anomalous patterns; This represents the abnormal guidance coefficient.

[0116] Furthermore, the process of obtaining the pattern anomaly score sequence is as follows:

[0117] Then, for the graph embedding sequence of the current time window, the degree of deviation from the normal healthy pattern is comprehensively measured from local, global, short-term, and long-term dimensions to finally obtain the pattern abnormality score. Represented as:

[0118] ;

[0119] in, This represents the local reconstruction error, and this represents the local KL divergence. This represents the global reconstruction error. Indicates the global KL divergence;

[0120] Specifically, local reconstruction error Based on the local latent representation of the current time window, the difference between the local latent representation and the local trend representation of the original graph embedding sequence is obtained, i.e.: ;

[0121] Local KL divergence By measuring the current time window within a short-term time window Local potential distribution Local trend representation compared to normal mode The deviation is obtained as follows: ; This indicates a local trend under normal mode. The prior distribution;

[0122] Global reconstruction error Based on the global latent representation of the current time window, the difference between the global latent representation and the global trend representation of the original graph embedding sequence is obtained: ;

[0123] Global KL divergence By measuring the current time window within a long-term time window global latent distribution Representing global trends The deviation is obtained, and the formula is expressed as: ; This represents the prior distribution of the global trend representation under normal conditions;

[0124] Embedding sequence of graph temporal topology at each time step Calculate the corresponding pattern anomaly score st for each, and obtain the pattern anomaly score sequence. This sequence is the core basis for subsequent analysis of continuous deviations and for achieving forward-looking early warning.

[0125] In step S104 above, analyzing the continuous deviation of the abnormal score of the analysis mode and providing real-time early warning of abnormal resource usage before the macro resource indicators become abnormal specifically includes:

[0126] Preset a normal threshold Early warning rules are set by using thresholds and abnormal scores:

[0127] A real-time alert is triggered when any of the following conditions are met:

[0128] outlier score at a single time step Exceeding the threshold This indicates that the resource interaction pattern at the current time step deviates significantly from the normal pattern, and there is a sudden abnormal resource occupation.

[0129] Exceeding the threshold within the sliding window If the percentage of abnormal scores exceeds 50%, meaning that more than half of the time steps within the window have abnormal scores exceeding the limit, it indicates that the resource interaction mode has been deviating from normal for a period of time, and the trend of abnormal resource occupation is quite obvious.

[0130] After the alert is triggered, the system issues a real-time alarm for abnormal resource usage. Based on the relevant hierarchical dynamic resource directed graph and topological feature information, root cause analysis is performed, such as:

[0131] By examining the hierarchical dynamic resource directed graph corresponding to the time step when the abnormal score exceeds the threshold, we can analyze the abnormal interactions of the nodes (processes or resources) and determine whether the problem lies in the interaction within or between layers, thereby quickly locating the source of the abnormal resource usage.

[0132] Although the steps in the above embodiments are described in the above order, those skilled in the art will understand that in order to achieve the effect of this embodiment, different steps do not need to be executed in such an order. They can be executed simultaneously (in parallel) or in a reverse order. These simple variations are all within the protection scope of this invention.

[0133] Example 2: Based on the above technical concept, the second embodiment of the present invention proposes a virtual system corresponding to the method of the first embodiment, namely, a real-time early warning system for abnormal resource occupancy that integrates deep learning. The following is a detailed explanation... Figure 2 The various parts of the second embodiment of the present invention will be described in detail below.

[0134] like Figure 2 The system describes a real-time early warning system for abnormal resource occupancy that integrates deep learning. The various modules of this system work collaboratively according to the method described in Embodiment 1 above to achieve real-time early warning of abnormal resource occupancy. It includes:

[0135] Data acquisition module 210 is used to collect the system's call event stream and construct a hierarchical dynamic resource directed graph based on the call event stream;

[0136] Topology analysis module 220 is used to optimize and extract topological features based on hierarchical dynamic resource directed graph using hierarchical graph attention network, and generate a time-aware mechanism based on topological features to describe the evolution of system resource interaction patterns.

[0137] The anomaly quantification module 230 is used to optimize training based on the time-series topology graph embedded sequence, using a hierarchical variational autoencoder model combined with an anomaly reconstruction mechanism to calculate the degree of deviation between the current interaction mode and the normal healthy mode, and obtain the pattern anomaly score sequence.

[0138] The early warning output module 240 is used to analyze the continuous deviation of the abnormal score of the mode and to provide real-time early warning of abnormal resource usage before the macro resource indicators become abnormal.

[0139] In this application, several formulas are calculated by removing dimensions and taking their numerical values. The formulas are established by collecting a large amount of data and simulating the most recent real situation. Some coefficients or weights in the formulas are set by those skilled in the art according to the actual situation, so they will not be elaborated here.

[0140] Example 4: This embodiment of the invention also provides an electronic device and a computer-readable storage medium corresponding to Examples 1 and 2.

[0141] One electronic device includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor performs the steps of any one of S101-S105 of the real-time early warning method for abnormal resource occupancy that integrates deep learning.

[0142] like Figure 3 As shown, the electronic device may include: at least one processor 31, at least one network interface 35, user interface 34, memory 36, and at least one communication bus 32.

[0143] The communication bus 32 is used to enable communication between these components.

[0144] The user interface 34 may include a display screen and a camera. Optionally, the user interface 34 may also include a standard wired interface and a wireless interface.

[0145] The network interface 35 may optionally include a standard wired interface or a wireless interface (such as a WIFI interface).

[0146] The processor 31 may include one or more processing cores. The processor 31 connects to various parts of the server using various interfaces and lines, and performs various server functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 36, and by calling data stored in the memory 36. Optionally, the processor 31 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 31 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 31 and may be implemented as a separate chip.

[0147] The memory 36 may include random access memory (RAM) or read-only memory. Optionally, the memory 36 may include a non-transitory computer-readable storage medium. The memory 36 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 36 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 36 may also be at least one storage device located remotely from the aforementioned processor 31. Figure 3 As shown, the memory 36, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an application program for a real-time early warning method, system, electronic device, and storage medium that integrates deep learning for abnormal resource usage.

[0148] exist Figure 3In the electronic device shown, the user interface 34 is mainly used to provide an input interface for the user and obtain the user input data; while the processor 31 can be used to call the application stored in the memory 36, which is a real-time warning method for abnormal resource usage that integrates deep learning. When executed by one or more processors 31, the electronic device performs one or more of the methods described in steps S101-S104 of the above embodiment.

[0149] Those skilled in the art will clearly understand that the technical solutions of this application can be implemented using software and / or hardware. In this specification, "unit" and "module" refer to software and / or hardware capable of independently or in conjunction with other components to perform a specific function. Hardware may include, for example, a Field-Programmable Gate Array (FPGA), an Integrated Circuit (IC), etc.

[0150] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any one of S101-S104 of a real-time early warning method for abnormal resource occupancy incorporating deep learning.

[0151] Specifically, the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0152] The code of the computer program can be in the form of source code, object code, executable file, or some intermediate form.

[0153] Computer-readable storage media may include cache, high-speed random access memory (RAM), such as the common double data rate synchronous dynamic random access memory (DDR SDRAM), and may also include non-volatile memory (NVRAM), such as one or more read-only memory (ROM), disk storage devices, flash memory devices, or other non-volatile solid-state storage devices such as optical discs (CD-ROM, DVD-ROM), floppy disks, or data tapes.

[0154] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.

Claims

1. A real-time early warning method for abnormal resource occupancy integrating deep learning, characterized in that, The method includes: The system collects the call event flow and constructs a hierarchical dynamic resource directed graph based on the call event flow, specifically including the process layer and the resource layer; Based on the hierarchical dynamic resource directed graph, the hierarchical graph attention network is used for optimization and its topological features are extracted. Based on the topological features, a time-aware mechanism is introduced to generate a time-series topological graph embedding sequence that describes the evolution of the system's resource interaction mode. Based on the temporal topological graph embedding sequence, a hierarchical variational autoencoder model combined with an anomaly reconstruction mechanism is used to guide optimization training, calculate the degree of deviation between the current interaction mode and the normal healthy mode, and obtain the pattern anomaly score sequence. The system analyzes the continuous deviation of abnormal scores in the analysis mode and provides real-time early warning of abnormal resource usage before macro resource indicators become abnormal. The optimization and extraction of topological features using a hierarchical graph attention network includes: For each node in the directed hypergraph of hierarchical dynamic resource operations, a high-dimensional feature vector is randomly obtained initially. The high-dimensional feature vectors of all nodes in different levels are collected, as well as the weights of the directed edges between nodes in the same level. For each node in each level, all its neighboring nodes are traversed to obtain the attention weights between the node and its neighboring nodes. Update the node characteristics of each level in the directed hypergraph for hierarchical dynamic resource operations; After updating the feature representation, for the node pairs that need to interact between layers, the inter-layer attention weights between the node pairs are obtained again. The inter-layer attention weights are then used to perform weighted fusion of the features of cross-layer nodes to obtain node features containing inter-layer interaction information. The hierarchical dynamic resource directed graph is optimized by iteratively using the l-layer network of the hierarchical graph attention network. The optimized hierarchical dynamic resource directed graph is then input back into the hierarchical graph attention network to obtain topological features. The optimization training guided by the use of a hierarchical variational autoencoder model combined with an anomaly reconstruction mechanism includes: Collect time-series topology graph embedding sequences under normal and abnormal conditions, the time-series topology graph embedding sequences including normal sequences and abnormal sequences; Design a global loss function that simultaneously considers the normal sample fitting of normal sequences and the abnormal sample deviation of abnormal sequences; the global loss function includes the product of the variational lower bound loss based on normal samples and the normal sequence, and the product of the abnormal loss based on the abnormal guiding coefficient and the abnormal sequence. During training, the variational lower bound loss of normal samples is optimized by using the backpropagation algorithm, which is based on the product of the variational lower bound loss of normal samples and the normal sequence; the abnormal loss of abnormal samples is optimized by using the product of the abnormal guidance coefficient, the abnormal loss of abnormal samples, and the abnormal sequence.

2. The method according to claim 1, characterized in that, The construction of the hierarchical dynamic resource directed graph includes: A hierarchical dynamic resource directed graph includes a set of nodes and a set of directed superedges; The node set includes process layer nodes and resource layer nodes, and the directed superedge set includes directed superedges and directed superedges. Assign weights to different directed hyperedges based on the event call flow to construct a hierarchical directed hypergraph for dynamic resource operations.

3. The method according to claim 1, characterized in that, The temporal topology graph embedding sequence that generates the description of the evolution of system resource interaction patterns includes: Pooling operations are performed based on topological features to generate initial graph embedding information for the current time window; By introducing a Long Short-Term Memory (LSTM) structure and fusing the initial graph embedding information of the historical window with the LSTM structure, the hidden state of the LSTM in time can be obtained. The initial graph embedding information of the current window is fused with the LSTM hidden state to obtain the final time-aware graph embedding vector. Arrange the final graph embedding vectors of all time windows in chronological order to obtain a temporal topology graph embedding sequence describing the evolution of system resource interaction patterns.

4. The method according to claim 3, characterized in that, The hierarchical variational autoencoder model includes a global encoder, a global decoder, a local encoder, and a local decoder. The global encoder maps the temporal topology graph embedding sequence to a global latent distribution, reconstructs the global latent representation, and obtains a global trend representation of the temporal topology graph embedding sequence; The local encoder maps the temporal topology graph embedding sequence of each time window to a local latent distribution; The local decoder reconstructs the graph embedding sequence corresponding to the local latent representation for the time window, and obtains a local trend representation that measures the short-term pattern fit of a single time window.

5. The method according to claim 4, characterized in that, The acquisition of the pattern anomaly score sequence includes: For the graph embedding sequence of the current time window, the degree of deviation from the normal healthy pattern is comprehensively measured to finally obtain the pattern abnormality score; For each time step, the corresponding pattern anomaly score is calculated for the graph temporal topology embedding sequence, resulting in a pattern anomaly score sequence.

6. A real-time early warning system for abnormal resource usage integrating deep learning, implemented according to the method of any one of claims 1-5, characterized in that, include: The data acquisition module is used to collect the system's call event stream and construct a hierarchical dynamic resource directed graph based on the call event stream, specifically including the process layer and the resource layer; The topology analysis module is used to optimize and extract topological features of a hierarchical dynamic resource directed graph using a hierarchical graph attention network. Based on the topological features, a time-aware mechanism is introduced to generate a time-series topology graph embedding sequence that describes the evolution of the system's resource interaction patterns. The optimization and extraction of topological features using a hierarchical graph attention network includes: For each node in the directed hypergraph of hierarchical dynamic resource operations, a high-dimensional feature vector is randomly obtained initially. The high-dimensional feature vectors of all nodes in different levels are collected, as well as the weights of the directed edges between nodes in the same level. For each node in each level, all its neighboring nodes are traversed to obtain the attention weights between the node and its neighboring nodes. Update the node characteristics of each level in the directed hypergraph for hierarchical dynamic resource operations; After updating the feature representation, for the node pairs that need to interact between layers, the inter-layer attention weights between the node pairs are obtained again. The inter-layer attention weights are then used to perform weighted fusion of the features of cross-layer nodes to obtain node features containing inter-layer interaction information. The hierarchical dynamic resource directed graph is optimized by iteratively using the l-layer network of the hierarchical graph attention network. The optimized hierarchical dynamic resource directed graph is then input back into the hierarchical graph attention network to obtain topological features. The anomaly quantification module is used to optimize training based on the time-series topology graph embedded sequence, using a hierarchical variational autoencoder model combined with an anomaly reconstruction mechanism to calculate the degree of deviation between the current interaction mode and the normal healthy mode, and obtain the pattern anomaly score sequence. The optimization training guided by the use of a hierarchical variational autoencoder model combined with an anomaly reconstruction mechanism includes: Collect time-series topology graph embedding sequences under normal and abnormal conditions, the time-series topology graph embedding sequences including normal sequences and abnormal sequences; Design a global loss function that simultaneously considers the normal sample fitting of normal sequences and the abnormal sample deviation of abnormal sequences; the global loss function includes the product of the variational lower bound loss based on normal samples and the normal sequence, and the product of the abnormal loss based on the abnormal guiding coefficient and the abnormal sequence. During training, the variational lower bound loss of normal samples is optimized by using the backpropagation algorithm based on the product of the variational lower bound loss of normal samples and the normal sequence; the abnormal loss of abnormal samples is optimized by using the product of the abnormal guidance coefficient, the abnormal loss of abnormal samples, and the abnormal sequence. The early warning output module is used to analyze the continuous deviation of the abnormal score of the mode and to provide real-time early warning of abnormal resource usage before the macro resource indicators become abnormal.

7. An electronic device, comprising a memory and a processor, characterized in that, The memory stores a computer program, which, when executed by the processor, causes the processor to perform the steps of the real-time early warning method for abnormal resource consumption fusion based on any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the real-time early warning method for abnormal resource consumption fusion deep learning as described in any one of claims 1 to 5.

Images