Abnormal log detection method and system based on graph convolutional network and collaborative learning

Abstract

The invention belongs to the technical field of artificial intelligence, and discloses an abnormal log detection method and system based on a graph convolutional network and collaborative learning, and the method combines the powerful graph structure modeling capability of the graph convolutional network with the information sharing mechanism of collaborative learning, effectively improves the recognition capability of abnormal logs, and improves the detection efficiency. And meanwhile, the dependence on large-scale annotation data is reduced, and the generalization and adaptability of the system are improved. Firstly, original log data are preprocessed, a directed graph between log events is constructed, nodes represent the log events, and edges represent the time sequence or incidence relation between the events. Thirdly, performing feature extraction on the log graph by using a graph convolutional network, capturing complex association between log events through an adaptive attention mechanism, and improving the recognition capability of an abnormal mode; the method can effectively make up for the deficiency of complex log relation modeling ability of a traditional method, enhances the adaptability to an unknown abnormal mode through collaborative learning, and reduces the dependence on large-scale annotation data.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence, and particularly relates to an abnormal log detection method and system based on a graph convolutional network and collaborative learning. Background Art

[0002] With the rapid development of network technology, log data has become an important basis for ensuring the safe operation of systems, fault diagnosis, and anomaly detection. However, in the face of an increasingly complex and changing network environment, traditional abnormal log detection methods face great challenges. Especially when dealing with large-scale and diverse log data, the accuracy and adaptability of existing methods are often insufficient. Although many traditional methods can work effectively in certain specific situations, their limitations are gradually emerging, especially when dealing with complex network attacks and abnormal patterns, the effect gradually weakens.

[0003] Currently, log anomaly detection methods are mainly divided into signature-based methods and anomaly-based methods. Signature-based methods rely on predefined rules or feature matching to identify known abnormal patterns. Typical methods include rule matching, regular expression detection, and intrusion detection systems (such as Snort, Suricata), etc. Although these methods have good detection effects on known attacks, they are difficult to deal with unknown attacks, and the rule maintenance cost is relatively high, and they need to be continuously updated to adapt to new attack means. Anomaly-based methods use statistical analysis, machine learning, or deep learning techniques to model normal log behavior and detect abnormal logs that deviate from the normal pattern. Common methods include anomaly detection methods based on Isolation Forest, Autoencoder, and Graph Neural Network (GNN). These methods can identify unknown abnormal patterns, but they often rely on a large amount of high-quality training data, have limited generalization ability, and have a large computational overhead, and there are certain limitations in actual deployment.

[0004] In early research, traditional abnormal log detection methods usually based on rule matching and machine learning techniques. However, these methods have the following problems: (1) The cost of abnormal data annotation is high. Traditional abnormal log detection relies on a large number of abnormal annotation samples to train the model, but the annotation of high-quality abnormal log data is both time-consuming and expensive. Especially in the face of the continuous emergence of new abnormal patterns, it is extremely challenging to construct a comprehensive annotation data set. (2) The adaptability of the model is insufficient. With the evolution of the system structure and the change of log format, traditional detection models are often difficult to quickly adapt to the new environment, resulting in a decline in detection performance. For example, in the cloud computing and distributed system environment, the detection accuracy of traditional methods will be greatly affected.

[0005] In view of the limitations of existing methods, the present invention proposes an abnormal log detection method and system based on graph convolutional network and collaborative learning. Through its powerful graph structure modeling ability, the graph convolutional network can effectively capture the complex relationships and potential dependencies in log data, thereby improving the ability to identify abnormal logs. By performing graph modeling on log data, the graph convolutional network can deeply explore the connections between different log events and identify abnormal patterns that cannot be captured by traditional methods.

[0006] On the other hand, as a new learning strategy, collaborative learning can achieve information sharing and enhance the detection effect through the collaborative work of multiple models. Different from traditional single-model methods, collaborative learning can utilize the complementary advantages of multiple models, reduce the over-reliance on labeled data, and improve the robustness and adaptability of the system. In the task of abnormal log detection, collaborative learning can not only improve the ability to identify diverse abnormal behaviors, but also enhance the real-time response ability of the system, thus meeting the requirements of efficient detection.

[0007] Through the above analysis, the problems and defects existing in the prior art are as follows:

[0008] Existing methods often fail to adapt to environmental changes in a timely manner and usually rely on a large number of labeled samples. The high dependence of traditional abnormal log detection methods on labeled data brings significant limitations and challenges. Traditional methods require a large number of accurately labeled log data sets to train models in order to obtain sufficient knowledge and features to cope with constantly changing abnormal log patterns. However, collecting and labeling these data sets requires a large amount of time, labor, and financial resources. Especially in the context of the continuous emergence of new attack patterns and log formats, traditional methods often have difficulty quickly responding to these changes. Summary of the Invention

[0009] In view of the problems existing in the prior art, the present invention provides an abnormal log detection method and system based on graph convolutional network and collaborative learning.

[0010] The present invention is implemented as follows. An abnormal log detection method based on graph convolutional network and collaborative learning includes:

[0011] S1, converting the data set into a log transformation directed graph and dividing it into three parts: a normal sample training set, a mixed sample training set, and a mixed sample test set;

[0012] S2, training a graph convolutional model on the normal sample training set to learn the normal patterns in the log data;

[0013] S3, on the mixed sample training set, the graph attention network and the graph convolutional network perform unsupervised learning simultaneously; through graph structure modeling, the two models jointly learn the abnormal patterns and normal patterns in the mixed samples;

[0014] S4. In the initial stage of the mixed sample training set, the graph convolutional network guides the graph attention network for training, and the learning samples of the graph attention network rely on the output of the graph convolutional network as the true labels.

[0015] S5. When the prediction accuracy of the graph attention network reaches 70% of that of the graph convolutional network, the two models start to enter the collaborative learning stage. In this stage, the graph attention network and the graph convolutional network will guide each other, swap the roles of teacher and student, and share and fuse their respective feature information.

[0016] S6. During the collaborative learning process, the graph attention network and the graph convolutional network transfer the learned knowledge and share gradients with each other by sharing and fusing feature information.

[0017] S7. Through multiple rounds of collaborative learning, the two models gradually converge and identify abnormal patterns in a wider range of log data.

[0018] S8. Finally, the trained models are used to make predictions on the data of the mixed sample test set.

[0019] Furthermore, the specific content of S2 includes:

[0020] Pre-train the graph convolutional network on the normal sample training set to learn the structural patterns and feature relationships between normal logs.

[0021] Assume H (l) represents the node representation matrix of the l-th layer, A is the adjacency matrix of the directed graph, W (l) is the weight matrix of the l-th layer, and σ is the activation function. The graph convolutional network represents the nodes through the following formula:

[0022] H (l+h) = σ(AH (l) W (l) #(h).

[0023] Furthermore, the specific content of S3 includes:

[0024] On the mixed sample training set, the graph attention network and the graph convolutional network perform unsupervised learning simultaneously. The two models jointly learn the normal patterns and abnormal patterns in the mixed samples through the graph structure. In this stage, the graph attention network calculates the attention weights between adjacent nodes and performs weighted aggregation on the neighbors of each node to learn the interdependent relationships between nodes. Assume there is a graph containing N nodes, and the feature vector of each node is represented as h = {h h , h2,..., h N}, where represents the feature vector of node i, and F is the feature dimension. The graph attention network calculates the attention coefficient e between adjacent nodes i and j ijThe formula is as follows:

[0025] e ij = LeakyReLU(a T [Wh i ||Wh j #(2).

[0026] Furthermore, the specific steps of S4 are as follows:

[0027] In the initial stage of the mixed sample training set, the graph convolutional network serves as the teacher network to guide the training of the graph attention network; at this stage, the learning samples of the graph attention network depend on the output of the graph convolutional network, and the prediction result of the graph convolutional network is used as the true label of the graph attention network; during the training process, the loss functions of the graph attention network and the graph convolutional network are combined, and the training of the graph attention network is accelerated through collaborative learning, enabling it to better learn abnormal patterns; let y be the predicted value of the graph convolutional network, be the predicted value of the graph attention network, and σ be the sigmod function. The training loss function is as follows:

[0028]

[0029] Furthermore, the specific steps of S5 are as follows:

[0030] When the prediction accuracy of the graph attention network reaches 70% of the prediction accuracy of the graph convolutional network, the two models start to enter the collaborative learning stage; at this stage, the graph attention network and the graph convolutional network guide each other and alternately serve as the teacher and student roles; the two models share their respective feature information and are optimized through gradient sharing; at this time, the graph attention network can be optimized based on the output of the graph convolutional network, and at the same time, the graph convolutional network will also refer to the prediction result of the graph attention network for adjustment to improve the overall detection performance.

[0031] Furthermore, the specific steps of S6 are as follows: During the collaborative learning process, the graph attention network and the graph convolutional network share and fuse feature information, and mutually transfer the learned knowledge and shared gradients; through this knowledge sharing mechanism, the two models can continuously improve each other, enhance the generalization ability and adaptability of the model, and thus better adapt to various complex and dynamic log data;

[0032] The specific steps of S7 are as follows:

[0033] Through multiple rounds of collaborative learning, the two models gradually converge and can identify abnormal patterns in a wider range of log data; at this time, the two models have, to a certain extent, eliminated the dependence on large-scale labeled data and can continuously and effectively identify abnormal patterns under different log formats and changes, improving the robustness of the model; through the iteration of the loss value and gradient after each iteration, the accuracy of the entire model is improved, and finally, a good prediction can be made for each sample.

[0034] Specifically, S8 includes: predicting the trained model on the mixed sample test set; in the test phase, the model infers based on the sample features in the test set and outputs the prediction results of whether each sample is abnormal; the test results will be evaluated according to indicators such as accuracy, recall rate, and F1 score to verify the actual performance and effectiveness of the model.

[0035] Another object of the present invention is to provide an abnormal log detection system based on graph convolutional network and collaborative learning, including:

[0036] A conversion module for converting the data set into a log conversion directed graph and dividing it into three parts: a normal sample training set, a mixed sample training set, and a mixed sample test set;

[0037] A model training module for training a graph convolutional model on the normal sample training set to learn the normal patterns in the log data;

[0038] A learning module for simultaneously performing unsupervised learning of the graph attention network and the graph convolutional network on the mixed sample training set; through graph structure modeling, the two models jointly learn the abnormal patterns and normal patterns in the mixed samples;

[0039] A network training module for guiding the graph attention network to train in the initial stage of the mixed sample training set, and the learning samples of the graph attention network depend on the output of the graph convolutional network as the true label;

[0040] A collaboration module for when the prediction accuracy of the graph attention network reaches 70% of the graph convolutional network, the two models start to enter the collaborative learning stage; in this stage, the graph attention network and the graph convolutional network will guide each other, exchange the roles of teacher and student, and share and fuse their respective feature information;

[0041] A shared fusion module for, in the collaborative learning process, the graph attention network and the graph convolutional network mutually transfer the learned knowledge and share gradients by sharing and fusing feature information;

[0042] An identification module for, through multiple rounds of collaborative learning, the two models gradually converge to identify abnormal patterns in a wider range of log data;

[0043] A prediction module for finally predicting the trained model on the data of the mixed sample test set.

[0044] Another object of the present invention is to provide a computer device, the computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the abnormal log detection method based on the graph convolutional network and collaborative learning.

[0045] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which when executed by a processor causes the processor to perform the steps of the abnormal log detection method based on a graph convolutional network and collaborative learning.

[0046] Another object of the present invention is to provide an information data processing terminal for implementing the abnormal log detection system based on a graph convolutional network and collaborative learning.

[0047] Combined with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:

[0048] First, in view of the technical problems and their difficulty analysis existing in the above-mentioned prior art, it is difficult for the existing abnormal log detection methods to balance the efficient detection of known anomalies and the accurate identification of unknown anomalies, and they face problems such as high rule maintenance costs, insufficient generalization ability, and large computational overheads. The present invention proposes an abnormal log detection method based on a graph convolutional network and collaborative learning, which can effectively solve the above technical problems and bring innovative technical effects.

[0049] The present invention constructs a graph structure of log data and uses a graph convolutional network (GCN) to extract the topological relationship and context information of log sequences, thereby enhancing the ability to identify complex abnormal patterns. Compared with traditional anomaly-based methods, the present invention can learn the potential associations between logs without relying on a large amount of labeled data, improving the accuracy of unknown anomaly detection while reducing the false alarm rate.

[0050] The present invention combines a signature-based method and an anomaly-based method, adopts a two-branch architecture, while ensuring the efficient detection of known anomalies, uses the anomaly detection branch to identify unknown anomalies, avoiding the limitations of a single method. This method can adaptively optimize in a highly dynamic environment, reduce the manual maintenance cost, and significantly improve the generalization ability of the model.

[0051] The present invention introduces a collaborative learning mechanism, and improves the robustness and adaptability of anomaly detection through information sharing and joint optimization among multiple sub-models. This method can adapt to different log environments, enhance the real-time performance and stability of the detection system, thereby improving the overall performance of abnormal log detection.

[0052] Through experimental verification, the method of the present invention has achieved better detection effects than traditional methods on multiple public log data sets, especially having significant advantages in the ability to identify unknown anomalies, false alarm rate control, and computational efficiency, and can effectively meet the anomaly detection requirements in a large-scale log environment.

[0053] The present invention proposes an innovative method and system for detecting abnormal logs, based on graph convolutional networks and collaborative learning techniques, aiming to solve the dependence of traditional methods on a large number of labeled samples. Through its graph structure modeling ability, the graph convolutional network can effectively capture the complex relationships and potential abnormal patterns among various events in the log data, thereby improving the detection accuracy and reducing the need for labeled data. In abnormal log detection, the graph convolutional network can focus on key log patterns through an attention mechanism and adapt to the diversity and complexity of the data.

[0054] In addition, as another key technology, collaborative learning improves the overall detection performance by the collaborative work among multiple models, sharing knowledge and features. Different from traditional single models, collaborative learning can optimize the detection effect through the complementary advantages of multiple models and enhance the adaptability of the system in a changing environment. Through collaborative learning, different models can guide each other, share valuable information, thereby accelerating the recognition of abnormal patterns, reducing the need for large-scale labeled data, and enhancing the real-time performance and robustness of the system.

[0055] Generally speaking, the present invention combines the advantages of graph convolutional networks and collaborative learning, providing a more flexible, efficient and cost-effective method and system for detecting abnormal logs. This method can not only reduce the dependence on a large number of abnormal labeled data, but also effectively cope with the diversity of abnormal log patterns and the dynamic changes of the network environment, having broad application prospects and important commercial value.

[0056] The present invention proposes collaborative learning, and conducts model training based on graph convolutional networks and graph attention networks. During the training process, first, the graph convolutional network is trained on the normal sample training set to learn the normal patterns in the log data; subsequently, in the mixed sample training set, the two models perform unsupervised learning simultaneously, learning the normal patterns and abnormal patterns from each other. At the initial stage of training, the graph convolutional network guides the graph attention network for training and uses its own output as the true label of the graph attention network. When the prediction accuracy of the graph attention network reaches 70% of that of the graph convolutional network, the two models enter the collaborative learning stage, guiding each other and sharing feature information. In this way, the present invention can effectively improve the accuracy of log anomaly detection, achieving a high F1 value on the mixed sample test set and being able to effectively identify malicious logs.

[0057] The technical effects and advantages of the technical solution of the present invention: First, our method reduces the dependence on abnormal labeled data. In the log anomaly detection task, traditional methods usually require a large amount of labeled data, while the present invention adopts unsupervised learning combined with collaborative training, reducing the need for manually labeled data. Second, compared with traditional methods, the present invention has obvious advantages in detection accuracy and generalization ability, can adapt to different log data distributions, and maintain good detection performance in different environments.

[0058] Second, the technical problems solved by the present invention:

[0059] 1. Limitations of traditional methods: Existing log anomaly detection methods often rely on a large number of labeled samples, which leads to difficulty for the model to quickly adapt to new log data when the detection environment changes. Through collaborative learning, the present invention enables two models to quickly adjust their learning strategies in the new data environment, improving the adaptability of detection.

[0060] 2. Poor generalization ability: Traditional methods usually target a specific type of single dataset and lack wide applicability. By introducing collaborative training in the unsupervised learning process, the present invention enables two models to share and integrate different feature information, thereby improving the generalization ability of the models on different log data.

[0061] 3. Low accuracy: Traditional log anomaly detection methods may have low detection accuracy due to unbalanced data distribution or insufficient model generalization ability. By introducing collaborative learning, the present invention enables two models to continuously optimize their anomaly detection capabilities during the process of learning from each other, thereby improving the overall detection accuracy and F1 value.

[0062] Significant technological progress achieved:

[0063] Reduced the required number of anomaly label samples: The present invention adopts a collaborative learning method, enabling two models to learn and guide each other. When only using some normal samples for pre-training, it can effectively detect abnormal logs, reducing the dependence on large-scale abnormal labeled data.

[0064] Improved generalization ability: The method of the present invention is not only applicable to specific log datasets, but also applicable to different types of log data and can adapt to changes in log formats, improving the applicability of the model in different environments.

[0065] Improved accuracy and efficiency: Through collaborative learning, the graph convolutional network and the graph attention network guide each other, gradually optimizing the model performance, thereby improving the detection accuracy and reducing the computational overhead brought by single model training.

[0066] The method provided by the present invention has achieved significant progress technically, effectively solved multiple defects of the existing technology, and provided an innovative solution in the field of log anomaly detection. These technological advancements not only improve the detection accuracy but also reduce the labeled samples required for pre-training, which is of great significance for the automation and intelligent development of log anomaly detection.

[0067] The model and technical design adopted by the log anomaly detection method based on collaborative learning of the present invention address several key technical problems in the existing technology and have achieved significant technological progress.

[0068] Collaborative learning mechanism: The present invention adopts a collaborative learning strategy of a graph convolutional network and a graph attention network. In the initial stage, the graph convolutional network guides the training of the graph attention network. When the accuracy of the graph attention network reaches a certain threshold, the two models enter a collaborative learning stage of mutual guidance to improve the recognition ability of abnormal patterns.

[0069] Mutual model supervision: During the collaborative learning process, the two models alternately act as teachers and students for training. By sharing and fusing feature information, the models can learn more fully about abnormal patterns and improve the detection performance.

[0070] Reduce the dependence on large-scale labeled data: Through unsupervised learning and collaborative training, the present invention can effectively learn abnormal log patterns with only partial normal log samples for pre-training, reducing the dependence on large-scale labeled data.

[0071] Third, the technical problems solved by the present invention:

[0072] 1. Limitations of a single model: Traditional log anomaly detection methods usually rely on a single model, which easily leads to limited detection ability. Through collaborative learning, the present invention enables the two models to complement each other and improve the detection ability.

[0073] 2. Insufficient generalization ability: Previous methods usually target specific types of log data and are difficult to adapt to different log formats and anomaly types. Through collaborative learning and feature sharing, the present invention improves the adaptability to different log data.

[0074] 3. Accuracy of anomaly detection: Existing methods may have problems of high false alarm rate or missed detection when detecting abnormal logs. Through the collaborative learning strategy, the two models are continuously optimized during the learning process, improving the accuracy and stability of detection.

[0075] Obtained technological progress:

[0076] Improve the accuracy of log anomaly detection: Through the collaborative learning of the graph convolutional network and the graph attention network, the present invention effectively improves the accuracy of anomaly detection.

[0077] Reduce the labeled samples required for model training: The present invention mainly relies on unsupervised learning for anomaly pattern recognition, greatly reducing the demand for large-scale labeled data.

[0078] Enhance the adaptability and generalization ability of the system: The collaborative learning mechanism enables the present invention to adapt to different log formats and improves the detection ability for unknown abnormal patterns.

[0079] The present invention not only solves the existing problems technically, but also demonstrates significant technical advantages in practical applications, providing an efficient and flexible new method for log anomaly detection.

[0080] Fourthly, as the creative auxiliary evidence of the claims of the present invention, it is also reflected in the following important aspects:

[0081] Firstly, the present invention converts log data into a graph structure and processes it using a graph convolutional network (GCN). This innovative idea effectively solves the problem that traditional methods cannot capture the complex relationships and potential dependencies between logs. Traditional anomaly detection methods usually focus on the independence of logs or the sequentiality based on time series, but these methods often ignore the global and local correlations between logs. Through the graph-structured representation, the present invention can regard log events as nodes in the graph and capture the potential connections between different logs through the graph convolutional network, forming deep dependency relationships. This method can handle multi-level correlation features and has obvious advantages when facing complex anomaly patterns, especially new attacks. Therefore, when detecting unknown anomalies, the present invention has higher accuracy and robustness than traditional methods and can achieve efficient identification of abnormal logs in a complex and changeable environment.

[0082] Secondly, the present invention combines signature-based methods and anomaly-based methods and adopts a dual-branch architecture design, enabling the system to handle both known and unknown anomaly patterns simultaneously. Traditional signature-based methods perform well in known attack scenarios but are ineffective when facing unknown attacks. While anomaly-based methods can handle unknown attacks, they have a high false positive rate and are easily interfered by noise. The dual-branch architecture of the present invention organically combines these two methods. On the one hand, it uses the signature-based branch to efficiently identify known attacks, and on the other hand, it uses the anomaly-based branch to detect unknown attacks. In this way, the present invention can simultaneously improve the detection capabilities for both known and unknown anomalies, reduce the false positive rate, and improve the overall detection accuracy. This method shows significant advantages in real-time detection and accurate identification of complex attack patterns.

[0083] Furthermore, the introduction of the collaborative learning mechanism is one of the important innovations of this invention. Traditional anomaly detection methods usually rely on a single model for prediction, lacking sufficient flexibility and adaptability. In contrast, collaborative learning can effectively improve the robustness and generalization ability of the model through the mutual cooperation among multiple models. In this invention, different sub-models share information and complement each other under the collaborative learning framework, enabling the model to adaptively adjust and optimize when facing different log environments, thereby enhancing the detection system's ability to identify various abnormal behaviors. Through collaborative learning, this invention can reduce the dependence on a large amount of labeled data while improving the adaptability of the detection system, enabling the model to better cope with changes in log formats, network environments, and the emergence of new attacks.

[0084] In addition, this invention significantly reduces the cost of manual maintenance through automated feature extraction and optimization. Traditional anomaly detection methods rely on the continuous update and maintenance of rules. Especially when facing continuously changing network attack methods and log formats, the rule bases of traditional methods need to be continuously expanded and adjusted, which not only increases the workload but also brings relatively high costs. Different from this, this invention automatically learns the deep features in log data through a graph convolutional network and combines it with the collaborative learning mechanism, enabling the system to maintain high detection accuracy and adaptability without a large amount of manual intervention. This method also has significant advantages in the scalability and real-time performance of the system. Especially when dealing with large-scale log data, it can perform anomaly detection efficiently and accurately.

[0085] In summary, this invention demonstrates remarkable innovation in solving multiple technical problems in the prior art. By combining a graph convolutional network, a two-branch architecture, and a collaborative learning mechanism, this invention not only improves the accuracy and adaptability of anomaly log detection but also effectively reduces the dependence on labeled data, lowers the maintenance cost of the system, and enhances the real-time performance and robustness of the model. The combined effect of these innovative technical means enables this invention to efficiently perform anomaly log detection in a large-scale and dynamically changing network environment, with broad application prospects and market value.

[0086] The expected benefits are to improve the overall efficiency and accuracy of log anomaly detection, provide an efficient detection model for related log analysis systems, and support cross-platform log data processing.

[0087] It solves the problem of poor detection effect in the field of log anomaly detection caused by the scarcity of labeled data. Through collaborative learning, the present invention enables two models to act as teachers and students for each other, guiding each other to learn normal and abnormal patterns in log data, reducing the dependence on large-scale labeled data to a certain extent, and improving the detection accuracy. In addition, the mutual learning and feature sharing mechanism of the two models enables the present method to adapt to different log formats and data changes, improving the robustness and adaptability of detection.

[0088] Through the proposed collaborative learning, the present invention enables two models to interactively learn under unsupervised conditions, and optimizes the detection effect by sharing knowledge and gradients, thereby improving the accuracy and generalization ability of log anomaly detection. Brief Description of the Drawings

[0089] Figure 1 It is a flowchart of an anomaly log detection method based on a graph convolutional network and collaborative learning provided by an embodiment of the present invention.

[0090] Figure 2 It is the result of comparing the anomaly log detection system based on a graph convolutional network and collaborative learning provided by an embodiment of the present invention with other methods.

[0091] Figure 3 It is a block diagram of the structure of an anomaly log detection system based on a graph convolutional network and collaborative learning provided by an embodiment of the present invention.

[0092] Figure 4 It is a flowchart of converting a log into a directed graph provided by an embodiment of the present invention.

[0093] Figure 5 It is a structural diagram of an anomaly log detection model based on collaborative learning provided by an embodiment of the present invention. Detailed Description of the Embodiments

[0094] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the following further describes the present invention in detail with reference to embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0095] As Figure 1 shown, an anomaly log detection method based on a graph convolutional network and collaborative learning provided by an embodiment of the present invention includes the following steps:

[0096] S1. Convert the data set into a log directed graph and divide it into three parts: a normal sample training set, a mixed sample training set, and a mixed sample test set;

[0097] S2. Train a graph convolutional model on the normal sample training set to learn the normal patterns in the log data;

[0098] S3. On the mixed sample training set, the graph attention network and the graph convolutional network perform unsupervised learning simultaneously; through graph structure modeling, the two models jointly learn the abnormal patterns and normal patterns in the mixed samples;

[0099] S4. In the initial stage of the mixed sample training set, the graph convolutional network guides the training of the graph attention network, and the learning samples of the graph attention network depend on the output of the graph convolutional network as the true label;

[0100] S5. When the prediction accuracy of the graph attention network reaches 70% of that of the graph convolutional network, the two models start to enter the collaborative learning stage; in this stage, the graph attention network and the graph convolutional network will guide each other, swap the roles of teacher and student, and share and fuse their respective feature information;

[0101] S6. During the collaborative learning process, the graph attention network and the graph convolutional network transfer the learned knowledge and share gradients with each other by sharing and fusing feature information;

[0102] S7. Through multiple rounds of collaborative learning, the two models gradually converge and identify abnormal patterns in a wider range of log data;

[0103] S8. Finally, the trained model is used to make predictions on the data of the mixed sample test set.

[0104] Specifically, S2 provided by the embodiment of the present invention includes:

[0105] Using the graph convolutional network for pre-training on the normal sample training set to learn the structural patterns and feature relationships between normal logs;

[0106] Assume H (l) represents the node representation matrix of the l-th layer, A is the adjacency matrix of the directed graph, W (l) is the weight matrix of the l-th layer, and σ is the activation function; the graph convolutional network represents the nodes through the following formula:

[0107] H (l+h) = σ(AH (l) W (l) ) #(h).

[0108] Specifically, S3 provided by the embodiment of the present invention includes:

[0109] On the mixed sample training set, the graph attention network and the graph convolutional network perform unsupervised learning simultaneously; the two models jointly learn the normal patterns and abnormal patterns in the mixed samples through the graph structure; in this stage, the graph attention network calculates the attention weights between adjacent nodes, performs weighted aggregation on the neighbors of each node, and learns the interdependent relationships between nodes; assume there is a graph containing N nodes, and the feature vector of each node is represented as h = {hh , h2, ..., h N}, where LeakReLU is the activation function, represents the feature vector of node i, and F is the feature dimension; Note that the graph attention network calculates the attention coefficient e between adjacent nodes i and j ij The formula is as follows:

[0110] e ij = LeakyReLU(a T [Wh i ||Wh j )#(2).

[0111] Specifically, S4 provided by the embodiments of the present invention includes:

[0112] In the initial stage of the mixed sample training set, the graph convolutional network is used as the teacher network to guide the training of the graph attention network; at this stage, the learning samples of the graph attention network depend on the output of the graph convolutional network, and the prediction result of the graph convolutional network is used as the true label of the graph attention network; during the training process, the loss functions of the graph attention network and the graph convolutional network are combined, and the training of the graph attention network is accelerated through collaborative learning, so that it can better learn abnormal patterns; let L BCE be the collaborative loss during the training process, y is the predicted value of the graph convolutional network, is the predicted value of the graph attention network, σ is the sigmod function, and the training loss function is as follows:

[0113]

[0114] Specifically, S5 provided by the embodiments of the present invention includes:

[0115] When the prediction accuracy of the graph attention network reaches 70% of the prediction accuracy of the graph convolutional network, the two models start to enter the collaborative learning stage; at this stage, the graph attention network and the graph convolutional network guide each other and alternately act as the teacher and the student; the two models share their respective feature information and are optimized through gradient sharing; at this time, the graph attention network can be optimized based on the output of the graph convolutional network, and at the same time, the graph convolutional network will also refer to the prediction result of the graph attention network for adjustment to improve the overall detection performance.

[0116] Specifically, S6 provided by the embodiments of the present invention includes: During the collaborative learning process, the graph attention network and the graph convolutional network share and fuse feature information, and mutually transfer the learned knowledge and shared gradients; through this knowledge sharing mechanism, the two models can continuously improve each other, enhance the generalization ability and adaptability of the models, so as to better adapt to various complex and dynamic log data;

[0117] Specifically, S7 includes:

[0118] Through multiple rounds of collaborative learning, the two models gradually converge and can identify abnormal patterns in a wider range of log data. At this time, the two models have, to a certain extent, eliminated the dependence on large-scale labeled data and can continuously and effectively identify abnormal patterns under different log formats and changes, improving the robustness of the models. Through the iteration of the loss value and gradient after each iteration, the accuracy of the entire model is improved, and finally, a good prediction can be made for each sample.

[0119] S8 specifically includes: making predictions on the mixed sample test set using the trained model. In the test phase, the model infers based on the sample features in the test set and outputs the prediction results indicating whether each sample is abnormal. The test results will be evaluated according to metrics such as accuracy, recall rate, and F1-score to verify the actual performance and effectiveness of the model.

[0120] As Figure 3 shown, an abnormal log detection system based on graph convolutional network and collaborative learning provided by an embodiment of the present invention includes:

[0121] A conversion module for converting the data set into a log conversion directed graph and dividing it into three parts: a normal sample training set, a mixed sample training set, and a mixed sample test set.

[0122] A model training module for training a graph convolutional model on the normal sample training set to learn the normal patterns in the log data.

[0123] A learning module for simultaneously performing unsupervised learning on the mixed sample training set by the graph attention network and the graph convolutional network. Through graph structure modeling, the two models jointly learn the abnormal patterns and normal patterns in the mixed samples.

[0124] A network training module for, in the initial stage of the mixed sample training set, guiding the graph attention network to train by the graph convolutional network. The learning samples of the graph attention network depend on the output of the graph convolutional network as the true label.

[0125] A collaboration module for when the prediction accuracy of the graph attention network reaches 70% of that of the graph convolutional network, the two models start to enter the collaborative learning stage. In this stage, the graph attention network and the graph convolutional network will guide each other, swap the roles of teacher and student, and share and fuse their respective feature information.

[0126] A shared fusion module for, during the collaborative learning process, the graph attention network and the graph convolutional network mutually transfer the learned knowledge and share gradients by sharing and fusing feature information.

[0127] An identification module for, through multiple rounds of collaborative learning, the two models gradually converge and identify abnormal patterns in a wider range of log data.

[0128] A prediction module, which is used to finally predict the trained model on the data of the mixed sample test set.

[0129] Another object of the present invention is to provide a computer device, which includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor executes the steps of the abnormal log detection method based on graph convolutional network and collaborative learning.

[0130] Another object of the present invention is to provide a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the abnormal log detection method based on graph convolutional network and collaborative learning.

[0131] Another object of the present invention is to provide an information data processing terminal, which is used to implement the abnormal log detection system based on graph convolutional network and collaborative learning.

[0132] Specific implementation of the present invention:

[0133] Embodiment 1: Abnormal log detection in the Internet of Things environment

[0134] 1. Data preprocessing: First, remove the matching tokens of common variables such as IP addresses and block IDs from the original log messages in the logs; then start searching from the root node of the parsing tree based on the log message length, and select the path to the first-layer node according to the number of tokens in the preprocessed message; then traverse from the first-layer node selected in the first step to the leaf nodes based on the previous tokens, and select the next-layer node according to the tokens at the starting position of the message; then calculate the similarity between the log message and each log event in the log group contained in the reached leaf node based on token similarity and compare it with the threshold to determine the most suitable log group; finally, if a suitable group is found, update the log ID and event in the group, and if not, create a new group and update the parsing tree.

[0135] 2. Training the graph convolutional network on normal samples: Train the graph convolutional network model on the normal sample data set to learn the features of normal logs. Extract the temporal features in the logs through the graph convolutional layer, and use the fully connected layer to classify the log events to ensure that the model can recognize the patterns of normal logs.

[0136] 3. Training on mixed samples: On the mixed sample data set, the graph convolutional network and the graph attention network perform collaborative learning. The two models model through the graph structure and learn the normal and abnormal patterns in the mixed samples respectively. The graph convolutional network guides the graph attention network to train in the initial stage, and the graph attention network uses the output of the graph convolutional network as the true label, so as to jointly optimize the learning process.

[0137] 4. Collaborative learning stage: As the training progresses, the accuracies of the graph convolutional network and the graph attention network gradually approach. Especially when the accuracy of the graph attention network reaches more than 70% of that of the graph convolutional network, the two models enter the collaborative learning stage, guiding each other, swapping the roles of teacher and student, and sharing and integrating their respective feature information, thereby further improving the generalization ability of the model.

[0138] 4. Classification and detection: After multiple rounds of collaborative learning, the final model can classify new log samples to determine whether they are abnormal logs. The model generates classification labels through the prediction output of the samples and evaluates the performance according to indicators such as accuracy and F1 value.

[0139] This embodiment shows how to perform efficient abnormal log detection in the Internet of Things environment by combining a graph convolutional network and a graph attention network with collaborative learning technology. By introducing collaborative learning and graph structure modeling, the model can adapt to log data in different formats and variations, improving the accuracy and robustness of detection.

[0140] As Figure 3 shown, the method for abnormal log detection based on collaborative learning provided by the embodiment of the present invention includes:

[0141] Step 1, convert the data set into a log conversion directed graph and divide it into three parts: a normal sample training set, a mixed sample training set, and a mixed sample test set;

[0142] Step 2, train a graph convolutional model on the normal sample training set to learn the normal patterns in the log data;

[0143] Step 3, on the mixed sample training set, the graph attention network and the graph convolutional network perform unsupervised learning simultaneously. Through graph structure modeling, the two models jointly learn the abnormal patterns and normal patterns in the mixed samples;

[0144] Step 4, at the initial stage of the mixed sample training set, the graph convolutional network guides the graph attention network to train, and the learning samples of the graph attention network depend on the output of the graph convolutional network as the true label;

[0145] Step 5, when the prediction accuracy of the graph attention network reaches 70% of that of the graph convolutional network, the two models start to enter the collaborative learning stage. In this stage, the graph attention network and the graph convolutional network will guide each other, swap the roles of teacher and student, and share and integrate their respective feature information;

[0146] Step 6, during the collaborative learning process, the graph attention network and the graph convolutional network enhance the generalization ability and adaptability of the model by sharing and integrating feature information, transmitting the learned knowledge and sharing gradients with each other;

[0147] Step 7, through multiple rounds of collaborative learning, the two models gradually converge and can identify abnormal patterns in a wider range of log data. At this time, the two models have, to a certain extent, eliminated the dependence on large-scale labeled data and can effectively adapt to different log formats and changes;

[0148] Step 8, finally, the trained model is used to make predictions on the data of the mixed sample test set.

[0149] The anomaly log detection method based on collaborative learning provided by the embodiments of the present invention involves converting logs into directed graphs, training graph convolutional models on normal samples, collaborating and learning with each other on mixed samples, and then testing on the test set. The signal and data processing processes in each step will be explained in detail below:

[0150] Step 1, convert the dataset into a directed graph of logs and divide it into three parts: a normal sample training set, a mixed sample training set, and a mixed sample test set;

[0151] Data conversion: First, convert the log data into a directed graph, where each log event or entry is a node in the graph, and the relationship between nodes is modeled through the chronological order and relevance of events in the log. Then, divide the dataset into three parts: a normal sample training set, a mixed sample training set, and a mixed sample test set to ensure that the performance of the model can be effectively evaluated during both the training and testing phases.

[0152] Step 2, train a graph convolutional network on the normal sample training set to learn the normal patterns in the log data;

[0153] Training normal patterns: Use a graph convolutional network (GCN) to train on the normal sample training set. The model learns the normal patterns in the log data through graph convolutional operations, extracts time series and structural features, so as to accurately distinguish normal traffic from potential abnormal patterns.

[0154] Step 3, on the mixed sample training set, the graph attention network and the graph convolutional network perform unsupervised learning simultaneously. Through graph structure modeling, the two models jointly learn the abnormal and normal patterns in the mixed samples;

[0155] Unsupervised learning: On the mixed sample training set, the graph attention network and the graph convolutional network jointly model the graph structure of the log data through unsupervised learning. The two models capture the normal and abnormal patterns in the log data through parallel learning respectively, in order to better understand the potential complex relationships in the data.

[0156] Step 4, in the initial stage of the mixed sample training set, the graph convolutional network guides the training of the graph attention network, and the learning samples of the graph attention network depend on the output of the graph convolutional network as the true label;

[0157] Initial one-way guidance: In the initial stage of the mixed sample training set, the graph convolutional network plays a guiding role and provides the labels of the training samples for the graph attention network. At this stage, the graph convolutional network acts as the teacher network to guide the learning of the graph attention network, ensuring that the graph attention network can accurately learn normal and abnormal traffic patterns.

[0158] Step 5, when the prediction accuracy of the graph attention network reaches 70% of that of the graph convolutional network, the two models start to enter the collaborative learning stage. In this stage, the graph attention network and the graph convolutional network will guide each other, exchange the roles of teacher and student, and share and fuse their respective feature information;

[0159] Collaborative learning: As the training progresses, when the accuracy of the graph attention network gradually approaches 70% of that of the graph convolutional network, the two models enter the collaborative learning stage. In this stage, the two models guide each other, exchange the roles of teacher and student, share and fuse each other's feature information, and further improve the learning efficiency and accuracy of the models.

[0160] Step 6, during the collaborative learning process, the graph attention network and the graph convolutional network enhance the generalization ability and adaptability of the models by sharing and fusing feature information, transmitting the learned knowledge and sharing gradients with each other;

[0161] Knowledge sharing: In the collaborative learning stage, the graph convolutional network and the graph attention network not only share the features they have learned, but also further adjust the parameters by sharing gradients to strengthen each other's learning effects. This process can enhance the generalization ability of the models and make them adapt to more complex log data.

[0162] Step 7, through multiple rounds of collaborative learning, the two models gradually converge and can identify abnormal patterns in a wider range of log data. At this time, the two models have, to a certain extent, eliminated the dependence on large-scale labeled data and can effectively adapt to different log formats and changes;

[0163] Model consistency: After multiple rounds of collaborative learning, the prediction results of the graph convolutional network and the graph attention network gradually converge and can identify abnormal patterns in a wider and changing range of log data. At this time, the two models have basically eliminated the dependence on a large amount of labeled data and can effectively process different formats and types of log data, improving the adaptability of the models.

[0164] Step 8, finally, the trained models are used to make predictions on the data in the mixed sample test set.

[0165] Final prediction: After completing multiple rounds of collaborative learning, the trained models will be applied to the data in the mixed sample test set. Through this test set, the models can make predictions on new log samples to determine whether they belong to the abnormal category, thus providing accurate results for subsequent anomaly detection.

[0166] Throughout the process, each step involves the processing and transformation of data to meet the requirements of subsequent steps. Meanwhile, by combining graph convolutional networks, graph attention networks, and collaborative learning, this method can more comprehensively extract abnormal patterns in log data and improve the accuracy of abnormal traffic detection.

[0167] Step 1 specifically includes: converting the original log data set into a directed graph and dividing it into a normal sample training set, a mixed sample training set, and a mixed sample test set. Among them, the normal sample training set only contains normal log data, the mixed sample training set contains a certain proportion of abnormal logs, and the mixed sample test set serves as the final evaluation data set. In data preprocessing, duplicate removal, format normalization, timestamp conversion, and feature vectorization are performed on the log data to ensure that the input data is suitable for graph neural network modeling. The directed graph after log data conversion can be represented as a heterogeneous graph G=(V, E), where V is the node set and E is the edge set.

[0168] Step 2 specifically includes: using a graph convolutional network for pre-training on the normal sample training set to learn the structural patterns and feature relationships among normal logs. Assume that H (l) represents the node representation matrix of the l-th layer, A is the adjacency matrix of the directed graph, W (l) is the weight matrix of the l-th layer, and σ is the activation function. The graph convolutional network represents the nodes through the following formula:

[0169] H (l+h) =σ(AH (l) W (l) )#(h)

[0170] Step 3 specifically includes: on the mixed sample training set, the graph attention network and the graph convolutional network perform unsupervised learning simultaneously. The two models jointly learn the normal and abnormal patterns in the mixed samples through the graph structure. At this stage, the graph attention network calculates the attention weights between adjacent nodes, performs weighted aggregation on the neighbors of each node, and learns the interdependent relationships between nodes. Suppose there is a graph containing N nodes, and the feature vector of each node is represented as h={h h , h2,..., h N}, where represents the feature vector of node i, and F is the feature dimension. The graph attention network calculates the attention coefficient e ij between adjacent nodes i and j as follows:

[0171] e ij =LeakyReLU(a T [Wh i ||Wh j )#(2)

[0172] Step 4 specifically includes: In the initial stage of the mixed sample training set, the graph convolutional network acts as the teacher network to guide the training of the graph attention network. At this stage, the learning samples of the graph attention network depend on the output of the graph convolutional network, and the prediction results of the graph convolutional network are used as the true labels of the graph attention network. During the training process, the loss functions of the graph attention network and the graph convolutional network are combined, and the training of the graph attention network is accelerated through collaborative learning, enabling it to better learn abnormal patterns. Let y be the predicted value of the graph convolutional network, be the predicted value of the graph attention network, σ be the sigmod function, and the training loss function is as follows:

[0173]

[0174] Step 5 specifically includes: When the prediction accuracy of the graph attention network reaches 70% of the prediction accuracy of the graph convolutional network, the two models start to enter the collaborative learning stage. At this stage, the graph attention network and the graph convolutional network guide each other and alternately act as the teacher and the student. The two models share their respective feature information and are optimized through gradient sharing. At this time, the graph attention network can be optimized based on the output of the graph convolutional network, and at the same time, the graph convolutional network will also refer to the prediction results of the graph attention network for adjustment to improve the overall detection performance.

[0175] Step 6 specifically includes: During the collaborative learning process, the graph attention network and the graph convolutional network share and fuse feature information, and mutually transfer the learned knowledge and shared gradients. Through this knowledge sharing mechanism, the two models can continuously improve each other, enhance the generalization ability and adaptability of the models, and thus better adapt to various complex and dynamic log data.

[0176] Step 7 specifically includes: Through multiple rounds of collaborative learning, the two models gradually converge and can identify abnormal patterns in a wider range of log data. At this time, the two models have, to a certain extent, eliminated the dependence on large-scale labeled data and can continuously and effectively identify abnormal patterns under different log formats and changes, improving the robustness of the models. Through the iteration of the loss value and the gradient after each iteration, the accuracy of the entire model is improved, and finally, a good prediction can be made for each sample.

[0177] Step 8 specifically includes: Finally, the trained model is used to make predictions on the mixed sample test set. In the test stage, the model infers based on the sample features in the test set and outputs the prediction results of whether each sample is abnormal. The test results will be evaluated according to indicators such as accuracy, recall rate, and F1 score to verify the actual performance and effectiveness of the model.

[0178] As Figure 4 shown, the anomaly log detection model based on collaborative learning provided by the embodiments of the present invention includes:

[0179] Data preprocessing module: used to convert log data into a directed graph.

[0180] Graph Convolutional Network module: used for early normal sample learning and later joint learning with other modules.

[0181] Graph Attention Network module: this module is used for later joint learning with the Graph Convolutional Network module to share gradients, so as to achieve the result of joint improvement of the two modules.

[0182] Collaborative learning module: Collaborative learning is achieved by passing knowledge between the two modules and sharing gradients to achieve the effect of collaborative learning of the two modules.

[0183] An application embodiment of the present invention provides a log anomaly detection device, which includes a memory and a processor. A computer program is stored in the memory. When the computer program is executed by the processor, the processor can execute the steps of the log anomaly detection method based on the graph neural network.

[0184] An application embodiment of the present invention provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by the processor, the processor executes the steps of the log anomaly detection method based on the graph neural network.

[0185] An application embodiment of the present invention provides an information data processing terminal, which is used to implement a log anomaly detection system based on the graph neural network, so as to improve the ability to identify abnormal logs.

[0186] Embodiment:

[0187] A log anomaly detection method based on the graph neural network, characterized in that a collaborative learning mechanism is introduced during the model training process to improve the detection performance. The method includes the following steps:

[0188] Step 1: Load the publicly available HDFS / BGL dataset and parse the original log data;

[0189] Step 2: Convert the log data into a directed graph, with log events or entries as nodes in the graph, and the relationships between nodes are modeled through the log time sequence and relevance;

[0190] Step 3: Divide the converted graph dataset into three parts: normal sample training set, mixed sample training set, and mixed sample test set to ensure good generalization ability for model training and testing;

[0191] Step 4: Design a detection model based on the graph neural network, using GAT and DiGCN as the core models, and introducing an attention mechanism into the model structure to enhance the feature expression ability;

[0192] Step 5: Pretrain the GTN and DiGCN models, and conduct preliminary training only using the normal sample training set to ensure that the models can learn normal log patterns;

[0193] Step 6: Conduct model training on the mixed sample training set and introduce the knowledge distillation mechanism, enabling the two models to guide each other during the training process to optimize the detection performance.

[0194] Step 7: During the training process, dynamically adjust the knowledge distillation loss weight to enable the models to perform better collaborative optimization at different training stages;

[0195] Step 8: After training is completed, conduct model evaluation on the mixed sample test set, calculate performance metrics such as accuracy and F1 score, and compare with other log anomaly detection methods;

[0196] Step 9: Use the adversarial sample generation method to test the robustness of the model and optimize the model structure to enhance its detection ability for unknown anomalies;

[0197] Step 10: In the final experimental results, the proposed log anomaly detection model based on collaborative learning achieves a detection accuracy of 99.87% on the BGL dataset, showing better detection performance compared with other methods;

[0198] Some positive effects have been achieved during the research and development or use of the embodiments of the present invention, and it indeed has great advantages compared with the prior art. The following content will be described in combination with the data, charts, etc. of the experimental process.

[0199]

[0200] As shown in the above table, the data marked with "Our" are the experimental data of our model. By comparing the experimental data of other models, it can be seen that generally our model has an advantage.

[0201] The present invention relates to the technology of anomaly log detection, which is applicable to various computer systems and network environments, and has wide application value especially in the fields of large-scale distributed systems, cloud computing environments, enterprise security management, industrial control systems, and Internet of Things security. With the continuous improvement of the complexity of information systems, log data has become the key basis for network security monitoring, fault diagnosis, and system optimization. However, due to problems such as accuracy, real-time performance, and adaptability, traditional log analysis methods are difficult to meet the requirements of modern high-complexity systems. The present invention provides an efficient and intelligent log analysis and anomaly detection solution for multiple industries through the anomaly log detection method based on graph convolutional networks and collaborative learning.

[0202] In cloud computing and distributed systems, the present invention can be applied to scenarios such as data centers, microservice architectures, container orchestration (such as Kubernetes), virtual machine monitoring systems, etc. Due to the wide variety of log types and inconsistent formats in the cloud environment, and the problem of log correlation across platforms and services, it is difficult for traditional methods to effectively identify complex anomalies. The graph convolutional network of the present invention can model the topological relationships between different logs, and combined with collaborative learning, improve the adaptability to new environments, and enhance the security and operational stability of cloud computing platforms.

[0203] In enterprise security management, the present invention can be used in products such as security information and event management (SIEM), intrusion detection systems, and log management platforms. The security operation and maintenance within an enterprise rely on a large number of logs for anomaly detection and risk assessment, while the existing signature-based and traditional machine learning methods have limited detection capabilities for unknown attacks. The present invention combines signature-based methods with anomaly-based methods to provide more accurate anomaly detection capabilities, and can effectively identify security threats such as network attacks, abnormal logins, and data leaks.

[0204] In the field of industrial control system (ICS) and Internet of Things (IoT) security, the present invention can be applied to scenarios such as smart grids, intelligent manufacturing, and automated control systems. Due to the wide variety of devices in industrial control networks and different log formats, it is difficult for traditional methods to adapt to complex log environments. The graph convolutional network of the present invention can model the log interaction relationships between devices, and combined with collaborative learning to optimize the model, enabling the detection system to provide more accurate early warning and response capabilities when facing device anomalies, network attacks, and system failures in industrial control systems.

[0205] In addition, the present invention can also be used in fields such as the financial industry, e-commerce, and intelligent operations (AIOps). For example, in the analysis of bank transaction logs, payment system logs, and user behavior logs, the method of the present invention is applied for fraud detection, business anomaly analysis, and system failure prediction, effectively improving business security and stability. Therefore, the technical solution of the present invention is not only applicable to traditional log anomaly detection, but also can meet the higher requirements of emerging application scenarios for intelligence, security, and efficiency.

[0206] The present invention has been tested in multiple actual application scenarios and experimental environments, and compared with existing methods to verify its technical advantages. Through experimental data analysis, the present invention shows obvious advantages over traditional methods in terms of anomaly detection accuracy, false alarm rate control, computing efficiency, model adaptability, etc., as follows:

[0207] First, in terms of detection accuracy and false alarm rate, the present invention uses multiple publicly available log datasets for testing, including the HDFS log dataset, the BGL log dataset, the Cyber Threat Detection Dataset, etc. The experimental results show that compared with traditional signature-based methods, the present invention can effectively detect unknown abnormal patterns, increasing the detection accuracy by 15% to 25%, while reducing the false alarm rate by 30%, significantly reducing the interference to security operation and maintenance personnel.

[0208] Second, in terms of model adaptability, the collaborative learning mechanism of the present invention enables the model to maintain high detection performance under different system environments and log format changes. The test results show that in the transfer learning experiments with different log formats, the method of the present invention, compared with traditional deep learning methods (such as LSTM, Autoencoder), can still maintain stable detection effects after migrating to a new log environment, while the detection accuracy of traditional methods drops by more than 40% without retraining. This shows that the present invention has stronger generalization ability and environmental adaptability in the face of log format changes.

[0209] Third, in terms of computational efficiency, the present invention uses the efficient feature extraction ability of graph convolutional networks to reduce the computational complexity of the model, enabling it to be applicable to large-scale log data processing. The experimental data shows that in an environment of 1 billion log data, the method of the present invention is 2.5 times faster than traditional RNN and Transformer models in training speed and reduces the inference time by 50%, meeting the requirements of real-time log anomaly detection while ensuring high detection accuracy.

[0210] In addition, the actual deployment effects in different application scenarios further verify the technical value of the present invention. In the actual application of a certain enterprise's security operation and maintenance center, the method of the present invention, compared with the existing security information and event management system, successfully detected multiple abnormal attack behaviors that could not be identified by traditional rule libraries and completed the response in a short time, avoiding potential security risks. In the deployment experiment of industrial control systems, the present invention detected abnormal device behaviors in the log monitoring of a certain intelligent manufacturing production line, enabling timely early warnings before system failures, thus reducing the losses of enterprises.

[0211] In summary, the present invention shows superior technical effects in multiple actual environments and experimental tests. It not only improves the accuracy and adaptability of abnormal log detection, but also reduces the false alarm rate and computational overhead, meeting the requirements of efficient log anomaly detection in different application fields and providing an innovative, intelligent, and low-maintenance-cost anomaly detection solution for related industries.

[0212] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated designed hardware. Those of ordinary skill in the art can understand that the above devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code is provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and their modules of the present invention can be implemented by hardware circuits of programmable hardware devices such as very large scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, or can be implemented by a combination of the above hardware circuits and software such as firmware.

[0213] As described above, it is only the specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention by those skilled in the art within the technical scope disclosed by the present invention shall be covered by the protection scope of the present invention.

Claims

1. An anomaly log detection method based on graph convolutional network and collaborative learning, characterized in that It includes the following steps: S1. Convert the data set into a log conversion directed graph and divide it into three parts: a normal sample training set, a mixed sample training set, and a mixed sample test set; S2. Train a graph convolutional model on the normal sample training set to learn the normal patterns in the log data; S3. On the mixed sample training set, the graph attention network and the graph convolutional network perform unsupervised learning simultaneously; through graph structure modeling, the two models jointly learn the abnormal patterns and normal patterns in the mixed samples; S4. At the initial stage of the mixed sample training set, the graph convolutional network guides the training of the graph attention network, and the learning samples of the graph attention network depend on the output of the graph convolutional network as the true label; S5. When the prediction accuracy of the graph attention network reaches 70% of that of the graph convolutional network, the two models start to enter the collaborative learning stage; in this stage, the graph attention network and the graph convolutional network will guide each other, swap the roles of teacher and student, and share and fuse their respective feature information; S6. During the collaborative learning process, the graph attention network and the graph convolutional network transfer the learned knowledge and share gradients with each other by sharing and fusing feature information; S7. Through multiple rounds of collaborative learning, the two models gradually converge and identify abnormal patterns in a wider range of log data; S8. Finally, the trained model is used to make predictions on the data of the mixed sample test set.

2. The anomaly log detection method based on graph convolutional network and collaborative learning according to claim 1, wherein The specific content of S2 includes: Use the graph convolutional network to perform pre-training on the normal sample training set to learn the structural patterns and feature relationships between normal logs; Hypothesis H (l) represents the node representation matrix of the $l$-th layer, $A$ is the adjacency matrix of the directed graph, and $W$ (l) is the weight matrix of the l $l$-th layer, and $\sigma$ is the activation function; the graph convolutional network represents the nodes through the following formula: H (l+h) = σ(AH (l) W (l) )#(h).

3. The anomaly log detection method based on graph convolutional network and collaborative learning according to claim 1, characterized in that The specific content of S3 includes: On the mixed sample training set, the graph attention network and the graph convolutional network perform unsupervised learning simultaneously; the two models jointly learn the normal and abnormal patterns in the mixed samples through the graph structure; at this stage, the graph attention network calculates the attention weights between adjacent nodes, performs weighted aggregation on the neighbors of each node, and learns the interdependent relationships between nodes; assume there is a graph containing N nodes, and the feature vector of each node is represented as h = {h1, h2,..., h N}, where represents the feature vector of node i, and F is the feature dimension; the graph attention network calculates the attention coefficient e ij The formula is as follows: e ij = LeakyReLU(a T [Wh i ||Wh j )#(2).

4. The anomaly log detection method based on graph convolutional network and collaborative learning according to claim 1, characterized in that, The specific content of S4 includes: In the initial stage of the mixed sample training set, the graph convolutional network acts as a teacher network to guide the training of the graph attention network; at this stage, the learning samples of the graph attention network depend on the output of the graph convolutional network, and the prediction result of the graph convolutional network is used as the true label of the graph attention network; during the training process, the loss functions of the graph attention network and the graph convolutional network are combined, and the training of the graph attention network is accelerated through collaborative learning, enabling it to better learn abnormal patterns; let y be the predicted value of the graph convolutional network, be the predicted value of the graph attention network, σ be the sigmod function, and the training loss function is as follows:

5. The anomaly log detection method based on graph convolutional network and collaborative learning according to claim 1, wherein The specific content of S5 includes: When the prediction accuracy of the graph attention network reaches 70% of the prediction accuracy of the graph convolutional network, the two models start to enter the collaborative learning stage; in this stage, the graph attention network and the graph convolutional network guide each other and alternately act as teachers and students; the two models share their respective feature information and are optimized through gradient sharing.

6. The anomaly log detection method based on graph convolutional network and collaborative learning according to claim 1, characterized in that The specific content of S6 includes: During the collaborative learning process, the graph attention network and the graph convolutional network transfer the learned knowledge and share gradients with each other by sharing and fusing feature information; through this knowledge sharing mechanism; The specific content of S7 includes: Through multiple rounds of collaborative learning, the two models gradually converge and can identify abnormal patterns in a wider range of log data; The specific content of S8 includes: Use the trained model to make predictions on the mixed sample test set; in the test stage, the model infers based on the sample features in the test set and outputs the prediction results of whether each sample is abnormal; the test results will be evaluated according to indicators such as accuracy, recall rate, and F1 score to verify the actual performance and effectiveness of the model.

7. An anomaly log detection system based on graph convolutional network and collaborative learning for implementing the anomaly log detection method based on graph convolutional network and collaborative learning according to any one of claims 1-6, characterized in that, The abnormal log detection system based on the graph convolutional network and collaborative learning includes: A conversion module for converting the data set into a log conversion directed graph and dividing it into three parts: a normal sample training set, a mixed sample training set, and a mixed sample test set; A model training module for training a graph convolutional model on the normal sample training set to learn the normal patterns in the log data; A learning module for performing unsupervised learning on a mixed sample training set simultaneously by a graph attention network and a graph convolutional network; through graph structure modeling, the two models jointly learn the abnormal patterns and normal patterns in the mixed samples; A network training module for guiding the graph attention network to train in the initial stage of the mixed sample training set, and the learning samples of the graph attention network depend on the output of the graph convolutional network as the true labels; A collaboration module for when the prediction accuracy of the graph attention network reaches 70% of that of the graph convolutional network, the two models start to enter the collaborative learning stage; in this stage, the graph attention network and the graph convolutional network will guide each other, swap the roles of teacher and student, and share and fuse their respective feature information; A shared fusion module for in the collaborative learning process, the graph attention network and the graph convolutional network transfer the learned knowledge and shared gradients to each other by sharing and fusing feature information; An identification module for through multiple rounds of collaborative learning, the two models gradually converge and identify abnormal patterns in a wider range of log data; A prediction module for finally predicting the data of the mixed sample test set by the trained model.

8. A computer device, characterized in that, The computer device includes a memory and a processor. When the computer program stored in the memory is executed by the processor, the processor executes the steps of the abnormal log detection method based on the graph convolutional network and collaborative learning as described in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, which when executed by a processor, causes the processor to execute the steps of the abnormal log detection method based on the graph convolutional network and collaborative learning as described in any one of claims 1-6.

10. An information data processing terminal, characterized in that, The information data processing terminal is used to implement the abnormal log detection system based on the graph convolutional network and collaborative learning as described in claim 7.

Images