A system multi-fault detection method based on big data

Abstract

The present invention belongs to the field of deep learning and relates to a system multi-fault detection method based on big data. The training process of the detection model includes: obtaining historical data to obtain an initial data set; obtaining the fault source, and deriving features of the initial data set according to the fault source to obtain a structured data set; constructing a dynamic label relationship graph according to the structured data set, and inputting the dynamic label relationship graph into a dynamic graph neural network to obtain label semantic embedding; obtaining a feature representation of each fault label according to the label semantic embedding and a feature vector; obtaining a classification result according to the feature representation of the fault label; calculating a loss function value according to the classification result and the label semantic embedding, and updating the model parameters according to the loss function value until the model training is completed; the present invention updates the node features of the dynamic label relationship graph according to the dynamic graph neural network to obtain the label semantic embedding, thereby capturing the topological relationship in the fault label space while capturing the time dependency between the fault labels.

Description

Technical Field

[0001] The present invention belongs to the field of machine learning and deep learning, and specifically relates to a system multi-fault detection method based on big data. Background Art

[0002] With the rapid development of information technology, modern systems have become increasingly complex, including a large number of components and modules, with complex interactions between these components and modules. This complexity makes failures in the system no longer isolated events, but may cause chain reactions, and one failure point may become the fuse that triggers other failures. This phenomenon is called the multi-failure phenomenon, which not only increases the difficulty of system operation and maintenance, but may also have a serious impact on the stability and reliability of the system.

[0003] Existing multi-fault detection methods introduce big data technology and machine learning algorithms. Big data technology can process and analyze massive amounts of data, extract useful features and patterns from them, and provide strong support for fault detection. At the same time, machine learning algorithms can automatically detect possible fault phenomena in the system through training and learning. The introduction of big data technology and machine learning algorithms makes it possible for fault detection to no longer rely on manually set rules, but to automatically discover potential fault modes through algorithms.

[0004] In complex systems, faults do not just occur statically, but often have temporal dependencies. For example, a fault may trigger other related faults at a certain point in time, or multiple faults may have a higher probability of occurring within a certain time interval. Therefore, when constructing a label relationship graph, only considering the static co-occurrence relationship of labels may not be sufficient to fully capture the complex interactions between faults.

[0005] Therefore, the existing methods not only ignore the correlation between faults and the specific characteristics corresponding to the faults, making it difficult to effectively detect multiple faults in the system at the same time, but also ignore the dynamic relationship between faults. Summary of the invention

[0006] In order to solve the above problems of the prior art, the present invention proposes a system multi-fault detection method based on big data, including: obtaining historical data of the system, inputting the historical data into a trained system multi-fault detection model, and obtaining a detection result; the system multi-fault detection model includes: a dynamic graph neural network and a fully connected layer;

[0007] The training process of the system multi-fault detection model includes:

[0008] S1. Obtain the historical data of each server in the system to obtain the initial data set;

[0009] S2, obtaining a fault source, and deriving features of the initial data in the initial data set according to the fault source to obtain a structured data set; the structured data set includes: a feature vector matrix and a label vector matrix;

[0010] S3. Construct a dynamic label relationship graph according to the structured data set, wherein the dynamic label relationship graph includes label relationship graphs of multiple time steps; input the dynamic label relationship graph into the dynamic graph neural network to obtain the label semantic embedding H of each time step. t ,t∈[1,T]; T is the last time step;

[0011] S4. Embedding label semantics into H T The and feature vector matrices are mapped to the latent space and Hadamard products are performed to obtain specific feature representations for each fault label;

[0012] S5. Input the specific feature representation of the fault label into the fully connected layer to obtain the classification result;

[0013] S6. Embed H based on the classification results and the label semantics of each time step t Calculate the loss function value and update the model parameters according to the loss function value. When the loss function value is minimized, the model training is completed.

[0014] Derivation of features from the initial data in the initial data set according to the fault source includes:

[0015] S21. Extract the statistical features, periodic features, and text features of the initial data to obtain the feature vector x of each server. k ∈R 1×d , the feature vector x of each server k Combine and get the eigenvector matrix X∈R m×d ;

[0016] S22, numbering each fault source to obtain a label of each fault source;

[0017] S23, extract the relationship between the server and the fault source according to the initial data, and obtain the label vector y of each server according to the relationship between the server and the fault source k ∈R 1×q , the label vector y of each server k Combine and get the label vector matrix Y∈R m×q ;

[0018] S24, combining the feature vector matrix X and the label vector matrix Y to obtain a structured data set;

[0019] Where d is the number of features of the initial data, k is the index of the server, q is the number of fault sources, and m is the number of servers.

[0020] Building a dynamic label relationship graph based on a structured data set includes:

[0021] S31. Consider the label of the fault source at each time step t in the structured data set as a node, and obtain the node set V t ;

[0022] S32. Consider the co-occurrence relationship between the labels of the fault sources at each time step t in the structured data set as an edge, and obtain the edge set E t ;

[0023] S33, according to the node set V t and edge set E t Construct dynamic label relationship graph G t =(V t ,E t ).

[0024] The dynamic graph neural network processes the dynamic label relationship graph including:

[0025] Calculate the adjacency matrix of the dynamic label relationship graph, initialize the node feature matrix of the dynamic label relationship graph, obtain the initialized node feature matrix, update the initialized node feature matrix through the dynamic graph neural network combined with the adjacency matrix, and obtain the node feature matrix of each time step, that is, the label semantic embedding of each time step.

[0026] Calculating the adjacency matrix A(t) of the dynamic label relationship graph includes:

[0027] Calculate the symmetric temporal co-occurrence probability between the labels of the fault source, expressed as:

[0028]

[0029] Taking the symmetric temporal co-occurrence probability as the value of the adjacency matrix, we get the adjacency matrix A(t)∈R q×q ;

[0030] Among them, l i , l j denote the labels of the i-th and j-th fault sources respectively, P(l j ∣l i ,t) indicates label l i In the case of the label l at time step t j The probability of occurrence within the time interval Δt, P(l i ∣l j ,t)] indicates label l j In the case of the label l at time step t i The probability of occurrence within the time interval Δt, q is the number of fault sources.

[0031] Initializing the node feature matrix of the dynamic label relationship graph includes:

[0032] The features of the fault nodes of the dynamic label relationship graph are extracted from the historical data to obtain an initial node feature matrix; the features include: the type of fault, the severity of the fault and the frequency of the fault.

[0033] The updated node feature matrix after initialization through the dynamic graph neural network combined with the adjacency matrix includes:

[0034]

[0035] Among them, A ij (t) is the adjacency matrix at time step t, is the feature vector of node i at time step t, is the attention weight between node i and node j at time step t, N(i,t) is the set of neighbor nodes of node i at time step t, W is the weight of the dynamic graph neural network, and σ is the activation function.

[0036] Embed the label semantics into H T The and eigenvector matrices are mapped to the latent space and the Hadamard product is performed, including:

[0037] The label semantics are embedded into H T The latent space representation of the feature vector matrix and the latent space representation of the label semantic embedding are mapped to the latent space representation, and the Hadamard product is performed on the latent space representation of the feature vector matrix and the latent space representation of the label semantic embedding to obtain the specific feature representation of each label.

[0038] The loss function L includes:

[0039] L=L ce +λL LAL

[0040] Among them, L ce Represents the loss of classification results, L LAL represents the label alignment loss, and λ represents the balancing weight.

[0041] Loss function L LAL include:

[0042]

[0043] Among them, E t is the edge set at time step t, Embed H for label semantics t The feature vector of node i, Embed H for label semantics t The feature vector of node j, Embed H for label semantics t The attention weight between node i and node j, y i ,y j are the i-th and j-th columns of the label vector matrix respectively.

[0044] Beneficial effects:

[0045] 1. The present invention considers the time interval and order of label appearance, constructs a dynamic label relationship graph through a structured data set, and can more accurately reflect the dynamic relationship between labels, which helps to capture time dependency and improve the accuracy and robustness of the fault detection model; 2. The present invention considers the frequency and correlation of labels at different time points to calculate the time series co-occurrence probability, and can identify fault modes in time series; 3. The present invention updates the node features of the dynamic label relationship graph according to the dynamic graph neural network to obtain label semantic embedding, thereby capturing the topological relationship in the fault label space while comprehensively capturing the time dependency and dynamic relationship between fault labels, which can more comprehensively The present invention designs a label alignment loss function based on the node feature distribution optimization task, which can better capture the correlation between labels and enhance the consistency of node features by minimizing the differences between the features and label vectors of adjacent nodes; 5. The present invention also designs a label specific selection method, which maps the label semantic embedding and feature vector to another latent space, and guides the model to select specific features corresponding to the label through the Hadamard product to improve the fault detection performance; 6. The loss function of the present invention combines the multi-label classification task and the node feature distribution optimization task to enhance the robustness and generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A flowchart of a system multi-fault detection method based on big data provided by an embodiment of the present invention;

[0047] Figure 2 A schematic diagram of a dynamic label relationship graph constructed based on a structured data set provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] like Figure 1As shown, the present invention proposes a system multi-fault detection method based on big data, including: obtaining historical data of the system, inputting the historical data into a trained system multi-fault detection model, and obtaining a detection result; the system multi-fault detection model includes: a dynamic graph neural network and a fully connected layer;

[0050] The training process of the system multi-fault detection model includes:

[0051] S1. Obtain the historical data of each server in the system to obtain the initial data set;

[0052] Specifically, obtaining the historical data of the server of the system specifically includes:

[0053] S11. Configure Docker Swarm API access permissions: Access the API of the Docker Swarm cluster;

[0054] S12. After obtaining access to the Docker Swarm API, create a monitoring class for each running service of the system, which can detect the availability, performance, and other key indicators of the service; create a monitoring class for each node of the system, which can monitor the resource utilization, load, and health status of the node; create a monitoring class for each container of the system, which can analyze log data to detect potential problems or anomalies;

[0055] S13. Use the Docker Python client or other supported client libraries to interact with the Docker Swarm API to obtain information about all nodes in the cluster (including node names and IP addresses), information about all running services (including service names and associated containers), and log data for specific containers.

[0056] S2, obtaining a fault source, and deriving features of the initial data in the initial data set according to the fault source to obtain a structured data set; the structured data set includes: a feature vector and a label vector;

[0057] Derivation of features from the initial data in the initial data set according to the fault source includes:

[0058] S21. Extract the statistical features, periodic features, and text features of the initial data to obtain the feature vector x of each server. k ∈R 1×d , the feature vector x of each server k Combine them to get the eigenvector matrix X = [x1, x2, ..., x m ]∈R m×d ;

[0059] Statistical feature extraction includes: using sliding window technology to calculate the statistical features of service monitoring, node monitoring and log monitoring in each window, such as mean, maximum, minimum, standard deviation, variance, etc.; by adjusting the window size, features of different granularities can be captured;

[0060] Periodic feature extraction includes: extracting periodic features of service monitoring, node monitoring, and log monitoring by calculating the first-order and second-order differences of the time series of the initial data to reveal trends and seasonal changes, extracting frequency domain features by Fourier transform to analyze periodic signals, and extracting more fine-grained features by grouping services;

[0061] Text feature extraction includes: For data from log monitoring sources, the bag-of-words model is used to convert log data into a bag-of-words representation. The text is converted into a multidimensional vector by counting the frequency of each word in the text. At the same time, a pre-trained word embedding model is used to map the words into a low-dimensional vector space, where the vector representation of the words can capture the semantic and contextual information of the words. In addition, the inverse document frequency (IDF) technology is introduced to adjust the weight of the words. By combining IDF with the bag-of-words model, words with higher discrimination can be obtained.

[0062] S22, number each fault source to obtain a label of each fault source: [1, 2, ..., q];

[0063] S23, extract the relationship between the server and the fault source according to the initial data, and obtain the label vector y of each server according to the relationship between the server and the fault source k ∈R 1×q , the label vector y of each server k Combine them to get the label vector matrix Y=[y1,y2,...,y m ]∈R m×q ; For example, if the server numbered 5 experiences faults 1, 2, and 5 at the same time, its server label vector y5 = [1, 1, 0, 0, 1, ..., 0];

[0064] S24, combining the feature vector matrix X and the label vector matrix Y to obtain a structured data set;

[0065] Among them, d represents the number of features of the initial data, that is, the dimension of the feature vector, k represents the index of the server, q is the number of fault sources, and m is the number of servers.

[0066] S3. Construct a dynamic label relationship graph according to the structured data set, wherein the dynamic label relationship graph includes label relationship graphs of multiple time steps; input the dynamic label relationship graph into the dynamic graph neural network to obtain the label semantic embedding H of each time step t.t ,t∈[1,T]; T is the last time step;

[0067] like Figure 2 As shown in FIG. 1 , constructing a dynamic label relationship graph based on a structured data set includes:

[0068] S31. Consider the label of the fault source at each time step t in the structured data set as a node, and obtain the node set V t = {l i,t};

[0069] S32. Consider the co-occurrence relationship between the labels of the fault sources at each time step t in the structured data set as an edge, and obtain the edge set

[0070] S33, according to the node set V t , edge set E t Construct dynamic label relationship graph G t =(V t ,E t ).

[0071] The dynamic graph neural network processes the dynamic label relationship graph including:

[0072] Calculate the adjacency matrix of the dynamic label relationship graph, initialize the node feature matrix of the dynamic label relationship graph, obtain the initialized node feature matrix, update the initialized node feature matrix through the dynamic graph neural network combined with the adjacency matrix, and obtain the node feature matrix of each time step, that is, the label semantic embedding of each time step.

[0073] Calculating the adjacency matrix A(t) of the dynamic label relationship graph includes:

[0074] Calculate the symmetric temporal co-occurrence probability between the labels of the fault source, expressed as:

[0075]

[0076]

[0077] Taking the symmetric co-occurrence probability as the value of the adjacency matrix, we get the adjacency matrix A∈R q×q ;

[0078] Among them, l i , l j denote the labels of the i-th and j-th fault sources respectively, P(l j ∣l i ,t) indicates that the label l at time step t i In case of label l j The probability of occurrence within the time interval Δt, P(l i ∣lj ,t) indicates label l j In the case of the label l at time step t i The probability of occurrence within the time interval Δt is, is the label in server k at time step t In case of label l j,k The number of occurrences at time step t′, is the label in server k at time step t In case of label l i,k The number of occurrences at time step t′, represents the label in server k at time step t The number of occurrences, represents the label in server k at time step t The number of occurrences, q is the number of fault sources, m is the number of servers, and k is the index of the server.

[0079] Initializing the node feature matrix of the dynamic label relationship graph includes:

[0080] Extract the features of the fault nodes of the dynamic label relationship graph from the historical data and obtain the initial node feature matrix H 0 ; The characteristics include but are not limited to: the type of fault, the severity of the fault and the frequency of the fault.

[0081] The updated node feature matrix after initialization through the dynamic graph neural network combined with the adjacency matrix includes:

[0082]

[0083] Among them, A ij (t) is the adjacency matrix, is the feature vector of node i at time step t, is the attention weight between node i and node j at time step t, N(i,t) is the adjacency matrix of node i at time step t, W is the weight of the dynamic graph neural network, and σ is the ReLU activation function.

[0084] S4. Embedding label semantics into H T The and feature vector matrices are mapped to the latent space and Hadamard products are performed to obtain specific feature representations for each fault label;

[0085] S41, map the label semantic embedding to the latent space, the calculation formula is:

[0086]

[0087] Among them, h′ i The latent space representation representing the semantic embedding of labels, Represents the label semantic embedding H T The feature of label node i in W e and b e represents a learnable parameter, and σ represents the sigmoid activation function.

[0088] S42, map the feature vector to the latent space, and the calculation formula is:

[0089] x′ k =ζ(W x x k +b x )

[0090] Among them, x′ k represents the latent space representation of the feature vector, x k represents the feature vector, W x and b x represents the learnable parameter between labels, and ζ represents the ELU activation function.

[0091] S43. Performing Hadamard product on the latent space representation of the feature vector and the latent space representation of the label semantic embedding to obtain a specific feature representation of each label, including:

[0092] z i =ζ[W o (x′ k ⊙h′ i +b o )

[0093] Among them, z i represents the specific feature representation of label i, W o and b o represents a shareable learnable parameter and ⊙ represents the Hadamard product.

[0094] S5. Input the specific feature representation of the fault label into the fully connected layer to obtain the classification result of the structured data;

[0095] For each category of fault label l i ∈Y, attaching a fully connected layer to the corresponding label-specific features z i To predict the fault label l i The probability of existence is calculated as:

[0096]

[0097] Among them, g i represents the fully connected layer, s i Indicates the label l i The probability of existence, w i and b irepresents the learnable parameters, and σ represents the sigmoid activation function.

[0098] S6. Calculate the loss function value based on the classification results and label semantic embedding, update the model parameters based on the loss function value, and complete the model training when the loss function value is minimized.

[0099] Calculate the overall objective loss function of the model:

[0100] L=L ce +λL LAL

[0101] Among them, L ce Represents the loss of classification results, L LAL represents the label alignment loss, and λ represents the balance weight, which is used to control the importance of both in the loss function. Specifically, L ce The calculation formula is:

[0102]

[0103] Where q is the number of labels, y i represents the true label, and s i Represents the label probability predicted by the model;

[0104] Loss function L LAL include:

[0105]

[0106] Among them, E t is the edge set at time step t, Embed H for label semantics t The feature vector of node i, Embed H for label semantics t The feature vector of node j, Embed H for label semantics t The attention weight between node i and node j, y i ,y j are the i-th and j-th columns of the label vector matrix respectively.

[0107] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A system multi-fault detection method based on big data, characterized in that: include: Obtain the historical data of the system, input the historical data into the trained system multi-fault detection model, and obtain the detection results; The system multi-fault detection model includes: a dynamic graph neural network and a fully connected layer; The training process of the system multi-fault detection model includes: S1. Obtain the historical data of each server in the system to obtain the initial data set; S2, obtaining a fault source, and deriving features of the initial data in the initial data set according to the fault source to obtain a structured data set; the structured data set includes: a feature vector matrix and a label vector matrix; S3. Construct a dynamic label relationship graph according to the structured data set, wherein the dynamic label relationship graph includes label relationship graphs of multiple time steps; input the dynamic label relationship graph into the dynamic graph neural network to obtain the label semantic embedding H of each time step. t ,t∈[1,T]; T is the last time step; S4. Embedding label semantics into H T The and feature vector matrices are mapped to the latent space and Hadamard products are performed to obtain specific feature representations for each fault label; S5. Input the specific feature representation of the fault label into the fully connected layer to obtain the classification result; S6. Embed H based on the classification results and the label semantics of each time step t Calculate the loss function value and update the model parameters according to the loss function value. When the loss function value is minimized, the model training is completed.

2. The method for detecting multiple faults in a system based on big data according to claim 1, characterized in that: Derivation of features from the initial data in the initial data set according to the fault source includes: S21. Extract the statistical features, periodic features, and text features of the initial data to obtain the feature vector x of each server. k ∈R 1×d , the feature vector x of each server k Combine and get the eigenvector matrix X∈R m×d ; S22, numbering each fault source to obtain a label of each fault source; S23, extract the relationship between the server and the fault source according to the initial data, and obtain the label vector y of each server according to the relationship between the server and the fault source k ∈R 1×q , the label vector y of each server k Combine and get the label vector matrix Y∈R m ×q ; S24, combining the feature vector matrix X and the label vector matrix Y to obtain a structured data set; Where d is the number of features of the initial data, k is the index of the server, q is the number of fault sources, and m is the number of servers.

3. The system multi-fault detection method based on big data according to claim 1 is characterized in that: Building a dynamic label relationship graph based on a structured data set includes: S31. Consider the label of the fault source at each time step t in the structured data set as a node, and obtain the node set V t ; S32. Consider the co-occurrence relationship between the labels of the fault sources at each time step t in the structured data set as an edge, and obtain the edge set E t ; S33, according to the node set V t and edge set E t Construct dynamic label relationship graph G t =(V t ,E t ).

4. The system multi-fault detection method based on big data according to claim 1 is characterized in that: The dynamic graph neural network processes the dynamic label relationship graph including: Calculate the adjacency matrix of the dynamic label relationship graph, initialize the node feature matrix of the dynamic label relationship graph, obtain the initialized node feature matrix, update the initialized node feature matrix through the dynamic graph neural network combined with the adjacency matrix, and obtain the node feature matrix of each time step, that is, the label semantic embedding of each time step.

5. The method for detecting multiple faults in a system based on big data according to claim 4, characterized in that: Calculating the adjacency matrix A(t) of the dynamic label relationship graph includes: Calculate the symmetric temporal co-occurrence probability between the labels of the fault source, expressed as: Taking the symmetric temporal co-occurrence probability as the value of the adjacency matrix, we get the adjacency matrix A(t)∈R q×q ; Among them, l i , l j denote the labels of the i-th and j-th fault sources respectively, P(l j ∣l i ,t) indicates label l i In the case of the label l at time step t j The probability of occurrence within the time interval Δt, P(l i ∣l j ,t)] indicates label l j In the case of the label l at time step t i The probability of occurrence within the time interval Δt, q is the number of fault sources.

6. The method for detecting multiple faults in a system based on big data according to claim 4, characterized in that: Initializing the node feature matrix of the dynamic label relationship graph includes: The features of the fault nodes of the dynamic label relationship graph are extracted from the historical data to obtain an initial node feature matrix; the features include: the type of fault, the severity of the fault and the frequency of the fault.

7. The method for detecting multiple system faults based on big data according to claim 4, characterized in that: The updated node feature matrix after initialization through the dynamic graph neural network combined with the adjacency matrix includes: Among them, A ij (t) is the adjacency matrix at time step t, is the feature vector of node i at time step t, is the attention weight between node i and node j at time step t, N(i,t) is the set of neighbor nodes of node i at time step t, W is the weight of the dynamic graph neural network, and σ is the activation function.

8. The method for detecting multiple faults in a system based on big data according to claim 1, characterized in that: Embed the label semantics into H T The and eigenvector matrices are mapped to the latent space and the Hadamard product is performed, including: The label semantics are embedded into H T The latent space representation of the feature vector matrix and the latent space representation of the label semantic embedding are mapped to the latent space representation, and the Hadamard product is performed on the latent space representation of the feature vector matrix and the latent space representation of the label semantic embedding to obtain the specific feature representation of each label.

9. The system multi-fault detection method based on big data according to claim 1 is characterized in that: Loss function L include: L=L ce +λL LAL Among them, L ce Represents the loss of classification results, L LAL represents the label alignment loss, and λ represents the balancing weight.

10. The method for detecting multiple faults in a system based on big data according to claim 9, characterized in that: Loss function L LAL include: Among them, E t is the edge set at time step t, Embed H for label semantics t The feature vector of node i, Embed H for label semantics t The feature vector of node j, Embed H for label semantics t The attention weight between node i and node j, y i ,y j are the i-th and j-th columns of the label vector matrix respectively.

Images