Core network log anomaly detection method and system

Abstract

The invention discloses a core network log anomaly detection method and system, and the method comprises the steps: collecting each piece of historical log data, constructing a log sequence of each time period, obtaining a corresponding log word embedding vector, carrying out the iterative clustering optimization analysis of all log word embedding vectors, and carrying out the further dimension reduction processing, obtaining each log clustering main template cluster, and obtaining a matched log clustering main template cluster and each target log template of each piece of to-be-tested log data based on similarity characteristics between each piece of to-be-tested log data and all log clustering main template clusters; and according to difference characteristics between all the target log templates of each piece of to-be-detected log data and the cluster center of the corresponding matched log clustering main template cluster, calculating a correction prediction error anomaly index of each piece of to-be-detected log data so as to perform anomaly detection on the core network logs. According to the core network log anomaly detection method and system provided by the embodiment of the invention, the accuracy of core network log anomaly detection is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of network fault detection, and in particular to a core network log anomaly detection method and system. Background Art

[0002] The core network is an important part of the communication network and is called the "management center". The essence of the core network is the processing and distribution of data. It is responsible for managing, sorting and directing data to ensure that data is transmitted to the correct destination. Therefore, when the core network fails or behaves abnormally, it will lead to a decline in network service quality and even cause network interruption. Through log anomaly detection, these problems can be discovered and solved in a timely manner to ensure the stable operation of the network.

[0003] However, since the actual operation of the core network involves a large amount of complex log data, it will interfere with the anomaly detection of the core network logs, resulting in low accuracy of the existing technology for anomaly detection of the core network logs.

[0004] It can be seen that how to improve the accuracy of core network log anomaly detection has become a technical problem that technical personnel in this field need to solve urgently. Summary of the invention

[0005] The present invention provides a core network log anomaly detection method and system to solve the technical problem that the accuracy of core network log anomaly detection in the prior art is low.

[0006] In order to solve the above technical problem, an embodiment of the present invention provides a core network log anomaly detection method.

[0007] Collecting various historical log data, and constructing a log sequence for each time period based on the sorting and analysis results of all the historical log data; using a pre-trained long short-term memory network to perform word segmentation and tagging processing on each log sequence to obtain a corresponding log tag sequence;

[0008] The GloVe model is used to process each of the log tag sequences to obtain the corresponding log word embedding vector; all the log word embedding vectors are subjected to iterative clustering optimization analysis to obtain the optimal template clusters for each log clustering; the principal component analysis algorithm is used to perform dimensionality reduction processing on each of the optimal template clusters for log clustering to obtain the main template clusters for each log clustering;

[0009] Collect each log data to be tested in real time, and obtain the matching log clustering main template cluster and each target log template for each log data to be tested based on the similarity characteristics between each log data to be tested and all the log clustering main template clusters;

[0010] The corrected prediction error anomaly index of each of the log data to be tested is calculated according to the difference characteristics between all the target log templates of each of the log data to be tested and the corresponding cluster center of the matching log clustering main template cluster, so as to perform anomaly detection on the core network log.

[0011] As one of the preferred solutions, the iterative clustering optimization analysis is performed on all the log word embedding vectors to obtain the optimal template clusters for each log clustering, including:

[0012] The parameters of the Gaussian mixture model are estimated by using an expectation maximization algorithm, and the Gaussian mixture model is used to perform iterative clustering processing on all the log word embedding vectors to obtain an optimized clustering template cluster for each iterative clustering processing, and all the optimized clustering template clusters obtained by each iterative clustering processing are used as inputs for the next iterative clustering processing;

[0013] The number of clusters in each iterative clustering process is equal to the corresponding number of iterations;

[0014] The elbow method is used to process all the optimized clustering template clusters obtained by the iterative clustering process to obtain the best clustering number, and all the optimized clustering template clusters corresponding to the best clustering number are used as the optimal template clusters for log clustering.

[0015] As one of the preferred solutions, each of the log clustering main template clusters includes several log clustering main templates, and the matching log clustering main template cluster and each target log template of each of the log data to be tested are obtained based on the similarity features between each of the log data to be tested and all of the log clustering main template clusters, including:

[0016] Calculating a first similarity between each of the log data to be tested and a cluster center of each of the log clustering main template clusters, and determining a matching log clustering main template cluster for each of the log data to be tested based on first sorting results of all the first similarities;

[0017] The second similarity between each of the log data to be tested and each of the log clustering main templates in the corresponding matching log clustering main template cluster is calculated, and each target log template of each of the log data to be tested is obtained based on the second sorting results of all the second similarities.

[0018] As one of the preferred solutions, the determining of the matching log clustering main template cluster of each of the log data to be tested based on the first sorting results of all the first similarities includes:

[0019] Arrange the first similarities between each of the log data to be tested and the cluster centers of all the log clustering main template clusters in a first descending order, and use the log clustering main template cluster corresponding to the largest first similarity as the matching log clustering main template cluster for each of the log data to be tested;

[0020] The obtaining of each target log template of each log data to be tested based on the second sorting results of all the second similarities includes:

[0021] The second similarities between each of the log data to be tested and all the log clustering main templates in the corresponding matching log clustering main template cluster are arranged in a second descending order, and the first three log clustering main templates corresponding to the second similarities are used as the target log templates for each of the log data to be tested.

[0022] As one of the preferred solutions, the method of calculating the modified prediction error anomaly index of each log data to be tested according to the difference characteristics between all the target log templates of each log data to be tested and the corresponding cluster center of the matching log clustering main template cluster includes:

[0023] Taking the square prediction error between each of the target log templates of each of the log data to be tested and the cluster center of the corresponding matching log clustering main template cluster as the prediction error abnormality index of each of the target log templates of each of the log data to be tested;

[0024] The average value of the prediction error abnormality indexes of all the target log templates of each of the log data to be tested is used as the corrected prediction error abnormality index of each of the log data to be tested.

[0025] Another embodiment of the present invention provides a core network log anomaly detection system, the system comprising:

[0026] The log tagging module is used to collect various historical log data, and construct a log sequence for each time period based on the sorting and analysis results of all the historical log data; use a pre-trained long short-term memory network to perform word segmentation and tagging processing on each log sequence to obtain a corresponding log tag sequence;

[0027] The log template extraction module is used to process each of the log tag sequences using the GloVe model to obtain the corresponding log word embedding vector; perform iterative clustering optimization analysis on all the log word embedding vectors to obtain the optimal template clusters for each log clustering; and perform dimensionality reduction processing on each of the optimal template clusters for log clustering using the principal component analysis algorithm to obtain the main template clusters for each log clustering;

[0028] A nearest neighbor template matching module is used to collect each log data to be tested in real time, and obtain the matching log clustering main template cluster and each target log template for each log data to be tested based on the similarity characteristics between each log data to be tested and all the log clustering main template clusters;

[0029] The log anomaly correction module is used to calculate the corrected prediction error anomaly index of each log data to be tested based on the difference characteristics between all the target log templates of each log data to be tested and the corresponding cluster center of the matching log clustering main template cluster, so as to perform anomaly detection on the core network log.

[0030] As one of the preferred solutions, the iterative clustering optimization analysis is performed on all the log word embedding vectors to obtain the optimal template clusters for each log clustering, including:

[0031] The parameters of the Gaussian mixture model are estimated by using an expectation maximization algorithm, and the Gaussian mixture model is used to perform iterative clustering processing on all the log word embedding vectors to obtain an optimized clustering template cluster for each iterative clustering processing, and all the optimized clustering template clusters obtained by each iterative clustering processing are used as inputs for the next iterative clustering processing;

[0032] The number of clusters in each iterative clustering process is equal to the corresponding number of iterations;

[0033] The elbow method is used to process all the optimized clustering template clusters obtained by the iterative clustering process to obtain the best clustering number, and all the optimized clustering template clusters corresponding to the best clustering number are used as the optimal template clusters for log clustering.

[0034] As one of the preferred solutions, each of the log clustering main template clusters includes several log clustering main templates, and the matching log clustering main template cluster and each target log template of each of the log data to be tested are obtained based on the similarity features between each of the log data to be tested and all of the log clustering main template clusters, including:

[0035] Calculating a first similarity between each of the log data to be tested and a cluster center of each of the log clustering main template clusters, and determining a matching log clustering main template cluster for each of the log data to be tested based on first sorting results of all the first similarities;

[0036] The second similarity between each of the log data to be tested and each of the log clustering main templates in the corresponding matching log clustering main template cluster is calculated, and each target log template of each of the log data to be tested is obtained based on the second sorting results of all the second similarities.

[0037] As one of the preferred solutions, the determining of the matching log clustering main template cluster of each of the log data to be tested based on the first sorting results of all the first similarities includes:

[0038] Arrange the first similarities between each of the log data to be tested and the cluster centers of all the log clustering main template clusters in a first descending order, and use the log clustering main template cluster corresponding to the largest first similarity as the matching log clustering main template cluster for each of the log data to be tested;

[0039] The obtaining of each target log template of each log data to be tested based on the second sorting results of all the second similarities includes:

[0040] The second similarities between each of the log data to be tested and all the log clustering main templates in the corresponding matching log clustering main template cluster are arranged in a second descending order, and the first three log clustering main templates corresponding to the second similarities are used as the target log templates for each of the log data to be tested.

[0041] As one of the preferred solutions, the method of calculating the modified prediction error anomaly index of each log data to be tested according to the difference characteristics between all the target log templates of each log data to be tested and the corresponding cluster center of the matching log clustering main template cluster includes:

[0042] Taking the square prediction error between each of the target log templates of each of the log data to be tested and the cluster center of the corresponding matching log clustering main template cluster as the prediction error abnormality index of each of the target log templates of each of the log data to be tested;

[0043] The average value of the prediction error abnormality indexes of all the target log templates of each of the log data to be tested is used as the corrected prediction error abnormality index of each of the log data to be tested.

[0044] Compared with the prior art, the embodiments of the present invention have the following advantages:

[0045] (1) Collect each historical log data, construct a log sequence for each time period based on the sorting and analysis results of all historical log data, use a pre-trained long short-term memory network to perform word segmentation and tagging on each log sequence to obtain a corresponding log tag sequence, and convert the log message into a tag sequence through word segmentation and tagging for subsequent feature extraction, analysis or other natural language processing tasks, thereby capturing the complex patterns and features in the historical log data;

[0046] (2) Considering the complexity, large size and difficulty of processing of core network logs, the logs are first templated before anomaly detection is performed on the core network logs. The GloVe model is used to process each log tag sequence to obtain the corresponding log word embedding vector. All log word embedding vectors are iteratively clustered and optimized to obtain the optimal template clusters for each log clustering. By clustering similar log messages into templates, the redundancy of information can be reduced, making the log information more interpretable. Then, the principal component analysis algorithm is used to reduce the dimensionality of each log clustering optimal template cluster to obtain the main template clusters for each log clustering, thereby avoiding the interference of noise and redundant information on the anomaly detection of the core network logs and improving the accuracy of the anomaly detection of the core network logs.

[0047] (3) Collect each log data to be tested in real time, obtain the matching log clustering main template cluster and each target log template for each log data to be tested based on the similarity features between each log data to be tested and all log clustering main template clusters, calculate the corrected prediction error anomaly index of each log data to be tested according to the difference features between all target log templates of each log data to be tested and the corresponding matching log clustering main template cluster, so as to perform anomaly detection on the core network log. Through the correction of the nearest neighbor method, the false alarm and missed alarm caused by data inconsistency or error are reduced, and the accuracy of anomaly detection on the core network log is further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a flow chart of a core network log anomaly detection method in one embodiment of the present invention;

[0049] Figure 2 A schematic diagram of obtaining a modified prediction error anomaly index in one embodiment of the present invention;

[0050] Figure 3 The present invention is a structural block diagram of a core network log anomaly detection system in one embodiment of the present invention. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] In the description of this application, the terms "first", "second", "third", etc. are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first", "second", "third", etc. may explicitly or implicitly include one or more of the feature. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0053] In the description of the present application, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be a connection between the two elements. The terms "vertical", "horizontal", "left", "right", "upper", "lower" and similar expressions used herein are only for illustrative purposes, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0054] In the description of this application, it should be noted that, unless otherwise defined, all technical and scientific terms used in the present invention have the same meaning as those commonly understood by those skilled in the art. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood by specific circumstances.

[0055] An embodiment of the present invention provides a core network log anomaly detection method. For details, see Figure 1 ˉ Figure 2 , Figure 1 FIG. 1 is a flow chart of a core network log anomaly detection method in one embodiment of the present invention. Figure 2 It is a schematic diagram showing the acquisition of the modified prediction error anomaly index in one embodiment of the present invention.

[0056] A flowchart of a core network log anomaly detection method in one embodiment of the present invention includes the following steps S1-S4, which are as follows:

[0057] Step S1: Collect various historical log data, build a log sequence for each time period based on the sorting analysis results of all historical log data, and use a pre-trained long short-term memory network to perform word segmentation and tagging on each log sequence to obtain a corresponding log tag sequence.

[0058] It should be noted that due to the complexity and size of logs, it is difficult to process them. In order to process text data more conveniently for subsequent feature extraction, analysis or other natural language processing tasks, the log data needs to be converted into a unified digital format. The long short-term memory network pre-trained with a large-scale corpus can learn the context information, part of speech, dependency relationship and other features in the logs. The prediction results of the model can be directly used to segment text and generate tag sequences.

[0059] Specifically, the collected historical log data is first divided into preset time periods, and all historical log data of each time period is arranged in ascending chronological order to obtain a log sequence for each time period. Then, a pre-trained long short-term memory network is used to perform word segmentation and tagging on each log sequence to obtain a corresponding log tag sequence. Special cases such as abbreviations, dates, and numbers in the generated log tag sequence are processed to output the final log tag sequence.

[0060] It should be noted that the long short-term memory network LSTM is usually used to process and learn long-term dependency problems. Each LSTM unit in the long short-term memory network LSTM contains a memory unit for storing and processing information, and three important gates: input gate, forget gate and output gate. The long short-term memory network LSTM is a well-known technology and will not be elaborated in this application.

[0061] It should be further explained that after word segmentation and tagging, the text log messages in each time period will be represented as a numerical sequence, thereby reducing the dimension of the data and helping to identify and extract key information in the text, such as important keywords, phrases or patterns, so that clustering can focus more on meaningful information.

[0062] Step S2: Use the GloVe model to process each log tag sequence to obtain the corresponding log word embedding vector; perform iterative clustering optimization analysis on all log word embedding vectors to obtain the optimal template clusters for each log clustering; use the principal component analysis algorithm to perform dimensionality reduction processing on each log clustering optimal template cluster to obtain each log clustering main template cluster.

[0063] Specifically, the GloVe model is used to process each log tag sequence to obtain the corresponding log word embedding vector. It should be noted that the GloVe model is an unsupervised learning algorithm whose design goal is to capture the semantic relationship between words by learning global vocabulary statistics.

[0064] It should be further explained that when a system generates a large number of logs, these logs may contain various information, but it is difficult to understand intuitively. By clustering similar log messages into templates, the redundancy of information can be reduced, making the log information more interpretable. The Gaussian mixture model is a clustering method using statistical mixture models. It can handle complex data distributions, including clusters of different shapes and sizes. The expectation maximization algorithm is usually used to estimate the parameters of the Gaussian mixture model. In the expectation maximization algorithm, each iteration is divided into two steps: the expectation step and the maximum step. The elbow method is a technique commonly used to determine the optimal number of clusters in cluster analysis.

[0065] Specifically, the expectation maximization algorithm is used to estimate the parameters of the Gaussian mixture model, and the Gaussian mixture model is used to iteratively cluster all log word embedding vectors to obtain optimized clustering template clusters for each iterative clustering process, and all optimized clustering template clusters obtained from each iterative clustering process are used as input for the next iterative clustering process.

[0066] Furthermore, the elbow method is used to process all optimized clustering template clusters obtained by all iterative clustering processes to obtain the optimal number of clusters, and all optimized clustering template clusters corresponding to the optimal number of clusters are used as the optimal template clusters for log clustering.

[0067] It should be noted that the number of clusters in each iterative clustering process is equal to the corresponding number of iterations. The expectation maximization algorithm, Gaussian mixture model and elbow method are all well-known technologies, and this application will not go into details here. The principal component analysis algorithm is mainly used for dimensionality reduction and feature extraction. In order to further reduce the interference of noise and redundant data, the principal component analysis algorithm is used to perform dimensionality reduction processing on each log clustering optimal template cluster to obtain each log clustering main template cluster.

[0068] Step S3: collect each log data to be tested in real time, and obtain the matching log clustering main template cluster and each target log template for each log data to be tested based on the similarity features between each log data to be tested and all log clustering main template clusters.

[0069] It should be noted that each log clustering master template cluster contains several log clustering master templates. If the log data to be tested is directly compared with the log clustering master template cluster to achieve anomaly detection of the core network log, it may cause false positives and false negatives due to data inconsistency or errors.

[0070] It should be noted that cosine similarity evaluates the similarity of two vectors by calculating the cosine value of the angle between them. In order to further reduce false positives and false negatives, the similarity features between each log data to be tested and all log clustering master template clusters are analyzed, and the cluster center represents the center point of each cluster in the clustering result. By analyzing the similarity features between the log data to be tested and the cluster centers of the log clustering master template clusters, the log clustering master template clusters with similar features to the log data to be tested can be obtained.

[0071] Specifically, the cosine similarity is used to calculate the first similarity between each log data to be tested and the cluster center of each log clustering main template cluster, and the matching log clustering main template cluster of each log data to be tested is determined based on the first sorting results of all the first similarities.

[0072] The cosine similarity is used to calculate the second similarity between each log data to be tested and each log clustering main template in the corresponding matching log clustering main template cluster, and each target log template of each log data to be tested is obtained based on the second sorting results of all the second similarities.

[0073] Specifically, the specific steps for determining the matching log clustering main template cluster for each log data to be tested based on the first sorting results of all first similarities are: arranging the first similarities between each log data to be tested and the cluster centers of all log clustering main template clusters in a first descending order, and taking the log clustering main template cluster corresponding to the largest first similarity as the matching log clustering main template cluster for each log data to be tested.

[0074] The specific steps for obtaining the target log templates for each log data to be tested based on the second sorting results of all second similarities are: arranging the second similarities between each log data to be tested and all log clustering main templates in the corresponding matching log clustering main template cluster in a second descending order, and taking the log clustering main templates corresponding to the first three second similarities as the target log templates for each log data to be tested.

[0075] Step S4: Calculate the corrected prediction error anomaly index of each log data to be tested based on the difference characteristics between all target log templates of each log data to be tested and the cluster center of the corresponding matching log clustering main template cluster to perform anomaly detection on the core network log.

[0076] It should be noted that the squared prediction error is an indicator used to measure the difference between the model prediction and the actual observation. In log anomaly detection, the squared prediction error can be used to measure the difference between the log data to be tested and the cluster center of the matching log clustering main template cluster, so as to determine whether the log data to be tested has anomalies.

[0077] Specifically, the square prediction error between each target log template of each log data to be tested and the cluster center of the corresponding matching log clustering main template cluster is used as the prediction error anomaly index of each target log template of each log data to be tested. The average value of the prediction error anomaly index of all target log templates of each log data to be tested is used as the corrected prediction error anomaly index of each log data to be tested.

[0078] Another embodiment of the present invention provides a core network log anomaly detection system. For details, see Figure 3 , Figure 3 The following is a structural block diagram of a core network log anomaly detection system in one embodiment of the present invention. The system includes:

[0079] The log tagging module 11 is used to collect various historical log data, and construct a log sequence for each time period based on the sorting and analysis results of all historical log data; use a pre-trained long short-term memory network to perform word segmentation and tagging processing on each log sequence to obtain a corresponding log tag sequence;

[0080] The log template extraction module 12 is used to process each log tag sequence using the GloVe model to obtain the corresponding log word embedding vector; perform iterative clustering optimization analysis on all log word embedding vectors to obtain the optimal template clusters for each log clustering; use the principal component analysis algorithm to perform dimensionality reduction processing on each log clustering optimal template cluster to obtain each log clustering main template cluster;

[0081] The nearest neighbor template matching module 13 is used to collect each log data to be tested in real time, and obtain the matching log clustering main template cluster and each target log template for each log data to be tested based on the similarity characteristics between each log data to be tested and all log clustering main template clusters;

[0082] The log anomaly correction module 14 is used to calculate the corrected prediction error anomaly index of each log data to be tested based on the difference characteristics between all target log templates of each log data to be tested and the cluster center of the corresponding matching log clustering main template cluster, so as to perform anomaly detection on the core network log.

[0083] Compared with the prior art, the core network log anomaly detection method and system provided by the embodiment of the present invention have the following beneficial effects:

[0084] (1) Collect each historical log data, construct a log sequence for each time period based on the sorting and analysis results of all historical log data, use a pre-trained long short-term memory network to perform word segmentation and tagging on each log sequence to obtain a corresponding log tag sequence, and convert the log message into a tag sequence through word segmentation and tagging for subsequent feature extraction, analysis or other natural language processing tasks, thereby capturing the complex patterns and features in the historical log data;

[0085] (2) Considering the complexity, large size and difficulty of processing of core network logs, the logs are first templated before anomaly detection is performed on the core network logs. The GloVe model is used to process each log tag sequence to obtain the corresponding log word embedding vector. All log word embedding vectors are iteratively clustered and optimized to obtain the optimal template clusters for each log clustering. By clustering similar log messages into templates, the redundancy of information can be reduced, making the log information more interpretable. Then, the principal component analysis algorithm is used to reduce the dimensionality of each log clustering optimal template cluster to obtain the main template clusters for each log clustering, thereby avoiding the interference of noise and redundant information on the anomaly detection of the core network logs and improving the accuracy of the anomaly detection of the core network logs.

[0086] (3) Collect each log data to be tested in real time, obtain the matching log clustering main template cluster and each target log template for each log data to be tested based on the similarity features between each log data to be tested and all log clustering main template clusters, calculate the corrected prediction error anomaly index of each log data to be tested according to the difference features between all target log templates of each log data to be tested and the corresponding matching log clustering main template cluster, so as to perform anomaly detection on the core network log. Through the correction of the nearest neighbor method, the false alarm and missed alarm caused by data inconsistency or error are reduced, and the accuracy of anomaly detection on the core network log is further improved.

[0087] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A core network log anomaly detection method, characterized in that: The method comprises: Collecting various historical log data, and constructing a log sequence for each time period based on the sorting and analysis results of all the historical log data; using a pre-trained long short-term memory network to perform word segmentation and tagging processing on each log sequence to obtain a corresponding log tag sequence; The GloVe model is used to process each of the log tag sequences to obtain the corresponding log word embedding vector; all the log word embedding vectors are subjected to iterative clustering optimization analysis to obtain the optimal template clusters for each log clustering; the principal component analysis algorithm is used to perform dimensionality reduction processing on each of the optimal template clusters for log clustering to obtain the main template clusters for each log clustering; Collect each log data to be tested in real time, and obtain the matching log clustering main template cluster and each target log template for each log data to be tested based on the similarity characteristics between each log data to be tested and all the log clustering main template clusters; The corrected prediction error anomaly index of each of the log data to be tested is calculated according to the difference characteristics between all the target log templates of each of the log data to be tested and the corresponding cluster center of the matching log clustering main template cluster, so as to perform anomaly detection on the core network log.

2. A core network log anomaly detection method according to claim 1, characterized in that: The iterative clustering optimization analysis is performed on all the log word embedding vectors to obtain the optimal template clusters for each log clustering, including: The parameters of the Gaussian mixture model are estimated by using an expectation maximization algorithm, and the Gaussian mixture model is used to perform iterative clustering processing on all the log word embedding vectors to obtain an optimized clustering template cluster for each iterative clustering processing, and all the optimized clustering template clusters obtained by each iterative clustering processing are used as inputs for the next iterative clustering processing; Wherein, the number of clusters in each iterative clustering process is equal to the corresponding number of iterations; The elbow method is used to process all the optimized clustering template clusters obtained by the iterative clustering process to obtain the best clustering number, and all the optimized clustering template clusters corresponding to the best clustering number are used as the optimal template clusters for log clustering.

3. A core network log anomaly detection method according to claim 1, characterized in that: Each of the log clustering main template clusters includes a plurality of log clustering main templates, and the process of obtaining a matching log clustering main template cluster and each target log template for each of the log data to be tested based on similarity features between each of the log data to be tested and all of the log clustering main template clusters includes: Calculating a first similarity between each of the log data to be tested and a cluster center of each of the log clustering main template clusters, and determining a matching log clustering main template cluster for each of the log data to be tested based on first sorting results of all the first similarities; The second similarity between each of the log data to be tested and each of the log clustering main templates in the corresponding matching log clustering main template cluster is calculated, and each target log template of each of the log data to be tested is obtained based on the second sorting results of all the second similarities.

4. A core network log anomaly detection method according to claim 3, characterized in that: The determining of a matching log clustering main template cluster for each of the log data to be tested based on the first sorting results of all the first similarities includes: Arrange the first similarities between each of the log data to be tested and the cluster centers of all the log clustering main template clusters in a first descending order, and use the log clustering main template cluster corresponding to the largest first similarity as the matching log clustering main template cluster for each of the log data to be tested; The obtaining of each target log template of each log data to be tested based on the second sorting results of all the second similarities includes: The second similarities between each of the log data to be tested and all the log clustering main templates in the corresponding matching log clustering main template cluster are arranged in a second descending order, and the first three log clustering main templates corresponding to the second similarities are used as the target log templates for each of the log data to be tested.

5. A core network log anomaly detection method according to claim 1, characterized in that: The calculating of the corrected prediction error anomaly index of each of the log data to be tested according to the difference characteristics between all the target log templates of each of the log data to be tested and the corresponding cluster center of the matching log clustering main template cluster includes: Taking the square prediction error between each of the target log templates of each of the log data to be tested and the cluster center of the corresponding matching log clustering main template cluster as the prediction error abnormality index of each of the target log templates of each of the log data to be tested; The average value of the prediction error abnormality indexes of all the target log templates of each of the log data to be tested is used as the corrected prediction error abnormality index of each of the log data to be tested.

6. A core network log anomaly detection system, characterized in that: The system comprises: The log tagging module is used to collect various historical log data, and construct a log sequence for each time period based on the sorting and analysis results of all the historical log data; use a pre-trained long short-term memory network to perform word segmentation and tagging processing on each log sequence to obtain a corresponding log tag sequence; The log template extraction module is used to process each of the log tag sequences using the GloVe model to obtain the corresponding log word embedding vector; perform iterative clustering optimization analysis on all the log word embedding vectors to obtain the optimal template clusters for each log clustering; and perform dimensionality reduction processing on each of the optimal template clusters for log clustering using the principal component analysis algorithm to obtain the main template clusters for each log clustering; A nearest neighbor template matching module is used to collect each log data to be tested in real time, and obtain the matching log clustering main template cluster and each target log template for each log data to be tested based on the similarity characteristics between each log data to be tested and all the log clustering main template clusters; The log anomaly correction module is used to calculate the corrected prediction error anomaly index of each log data to be tested based on the difference characteristics between all the target log templates of each log data to be tested and the corresponding cluster center of the matching log clustering main template cluster, so as to perform anomaly detection on the core network log.

7. A core network log anomaly detection system according to claim 6, characterized in that: The iterative clustering optimization analysis is performed on all the log word embedding vectors to obtain the optimal template clusters for each log clustering, including: The parameters of the Gaussian mixture model are estimated by using an expectation maximization algorithm, and the Gaussian mixture model is used to perform iterative clustering processing on all the log word embedding vectors to obtain an optimized clustering template cluster for each iterative clustering processing, and all the optimized clustering template clusters obtained by each iterative clustering processing are used as inputs for the next iterative clustering processing; Wherein, the number of clusters in each iterative clustering process is equal to the corresponding number of iterations; The elbow method is used to process all the optimized clustering template clusters obtained by the iterative clustering process to obtain the best clustering number, and all the optimized clustering template clusters corresponding to the best clustering number are used as the optimal template clusters for log clustering.

8. A core network log anomaly detection system according to claim 6, characterized in that: Each of the log clustering main template clusters includes a plurality of log clustering main templates, and the process of obtaining a matching log clustering main template cluster and each target log template for each of the log data to be tested based on similarity features between each of the log data to be tested and all of the log clustering main template clusters includes: Calculating a first similarity between each of the log data to be tested and a cluster center of each of the log clustering main template clusters, and determining a matching log clustering main template cluster for each of the log data to be tested based on first sorting results of all the first similarities; The second similarity between each of the log data to be tested and each of the log clustering main templates in the corresponding matching log clustering main template cluster is calculated, and each target log template of each of the log data to be tested is obtained based on the second sorting results of all the second similarities.

9. A core network log anomaly detection system according to claim 8, characterized in that: The determining of a matching log clustering main template cluster for each of the log data to be tested based on the first sorting results of all the first similarities includes: Arrange the first similarities between each of the log data to be tested and the cluster centers of all the log clustering main template clusters in a first descending order, and use the log clustering main template cluster corresponding to the largest first similarity as the matching log clustering main template cluster for each of the log data to be tested; The obtaining of each target log template of each log data to be tested based on the second sorting results of all the second similarities includes: The second similarities between each of the log data to be tested and all the log clustering main templates in the corresponding matching log clustering main template cluster are arranged in a second descending order, and the first three log clustering main templates corresponding to the second similarities are used as the target log templates for each of the log data to be tested.

10. A core network log anomaly detection system according to claim 6, characterized in that: The calculating of the corrected prediction error anomaly index of each of the log data to be tested according to the difference characteristics between all the target log templates of each of the log data to be tested and the corresponding cluster center of the matching log clustering main template cluster includes: Taking the square prediction error between each of the target log templates of each of the log data to be tested and the cluster center of the corresponding matching log clustering main template cluster as the prediction error abnormality index of each of the target log templates of each of the log data to be tested; The average value of the prediction error abnormality indexes of all the target log templates of each of the log data to be tested is used as the corrected prediction error abnormality index of each of the log data to be tested.

Images