A method and apparatus for anomaly detection in network log data

Abstract

This application relates to the field of log processing technology and discloses a method and apparatus for anomaly detection of network log data. The method includes: acquiring first network log data; determining, based on the string characteristics of the first network log data and the string characteristics of existing classes in a first clustering model, whether the first network log data belongs to an existing class in the first clustering model or to a first class; determining, based on the unique character characteristics of the first network log data and the unique character characteristics of existing classes in a second clustering model, whether the first network log data belongs to an existing class in a second clustering model or to a second class; if the first network log data belongs to the first class and / or the first network log data belongs to the second class, then the first network log data is determined to be anomaly data. Thus, by performing anomaly detection on the first network log data based on its string characteristics and unique character characteristics, dynamically changing log data can be flexibly handled, improving detection accuracy.

Description

Technical Field

[0001] This application relates to the field of log processing technology, and in particular to a method and apparatus for detecting anomalies in network log data. Background Technology

[0002] The rapid development of network technology and the widespread application of the Internet have led to an explosive growth in network log data. Enterprises are placing various application services on remote server platforms, and the activities of enterprises, organizations, and individuals on the network are recorded in the form of log data within the network system. Network log data is a crucial resource for understanding network activity, system health, and user behavior.

[0003] However, while network log data brings many conveniences to people, cyberattacks targeting big data platform servers have become more prevalent. As cyber threats and attack methods continue to evolve, the characteristics and patterns of network log data change over time. Therefore, how to detect anomalies in dynamically changing network log data still requires further research. Summary of the Invention

[0004] This application provides a method and apparatus for detecting anomalies in network log data, which detects network log data based on the string characteristics and unique character characteristics of the network log data, thereby improving the detection accuracy of network log data in the event of dynamic changes in the network log data.

[0005] In a first aspect, embodiments of this application provide an anomaly detection method for network log data. The method includes: acquiring first network log data; determining, based on the string characteristics of the first network log data and the string characteristics of existing classes in a first clustering model, that the first network log data belongs to an existing class in the first clustering model or belongs to a first class, wherein the first class is a new class created by the first clustering model; determining, based on the unique character characteristics of the first network log data and the unique character characteristics of existing classes in a second clustering model, that the first network log data belongs to an existing class in a second clustering model or belongs to a second class, wherein the second class is a new class created by the second clustering model, wherein the unique character of the first network log data is a character that is not repeated in the first network log data; if the first network log data belongs to the first class and / or the first network log data belongs to the second class, then determining that the first network log data is abnormal data.

[0006] Using the above method, the type of the first network log data is determined based on the string characteristics of the first network log data and the string characteristics of the existing classes in the first clustering model; and the type of the first network log data is determined based on the unique character characteristics of the first network log data and the unique character characteristics of the existing classes in the second clustering model. That is, the type of the first network log data is characterized from different features, which facilitates flexible adaptation to new patterns of log data, enabling anomaly detection of dynamically changing network log data, reducing the model's dependence on data staticity, and improving the accuracy of anomaly detection.

[0007] In an optional embodiment, the method further includes: acquiring network log data to be processed; removing spaces, punctuation marks, and timestamps from the network log data to be processed, and processing the format of the network log data to be processed into a preset format to obtain the first network log data.

[0008] Using the above method, since the clustering model focuses more on the text similarity between log data than on specific times, spaces, and punctuation marks, removing spaces, punctuation marks, and timestamps from the network log data to be processed can reduce the amount of computation. Processing the network log data to be processed into a preset format can ensure the uniformity of the data format, so as to perform similarity calculations more accurately.

[0009] In an optional embodiment, determining whether the first network log data belongs to an existing class of the first clustering model or to a first category based on the string features of the first network log data and the string features of existing classes in the first clustering model includes: selecting a first cluster with a similarity greater than a first threshold from the existing classes of the first clustering model based on the similarity between the string features of the first network log data and the string features of representative log data of the existing classes in the first clustering model; filtering the first cluster to obtain a second cluster; selecting the class with the smallest distance to the first network log data from the second cluster as the class to which the first network log data belongs; if the second cluster is empty, then determining that the first network log data belongs to the first category.

[0010] In one optional embodiment, filtering the first cluster to obtain a second cluster includes: filtering the first cluster based on length filtering and short word filtering to obtain a second cluster; wherein, the long word filtering includes selecting classes in the first cluster, wherein the length difference between the representative log data of the selected class and the first network log data is within a preset first range; the short word filtering includes selecting classes in the first cluster, wherein the number of matching characters between the representative log data of the selected class and the first network log data is greater than a second threshold.

[0011] By using the above method, clusters that do not meet the predefined conditions in the existing classes of the first clustering model can be removed based on length filtering and short word filtering, which can reduce the complexity of subsequent calculations and improve computational efficiency.

[0012] In an optional embodiment, determining whether the first network log data belongs to an existing class of the second clustering model or a second class based on the unique character features of the first network log data and the unique character features of existing classes in the second clustering model includes: selecting a third cluster with a similarity greater than a second threshold from the existing classes of the second clustering model based on the similarity between the unique character features of the first network log data and the unique character features of representative log data of the existing classes in the second clustering model; selecting the class with the smallest distance to the first network log data from the third cluster as the class to which the first network log data belongs; if the second cluster is empty, then determining that the first network log data belongs to the second class.

[0013] In one optional embodiment, the number of log data in the existing clusters of the first clustering model is greater than or equal to a third threshold; the number of log data in the existing clusters of the second clustering model is greater than or equal to the third threshold.

[0014] Using the above method, since there may be abnormal data during the clustering process of the first clustering model and the second clustering model, by setting a third threshold to remove abnormal clusters, the existing classes of the first clustering model and the existing classes of the second clustering model can be obtained. This can reduce the impact of abnormal data on the final detection accuracy and facilitate the improvement of the accuracy of anomaly detection.

[0015] Secondly, embodiments of this application provide an anomaly detection device for network log data, the device comprising:

[0016] The acquisition module is used to acquire the first network log data;

[0017] The processing module is configured to determine, based on the string characteristics of the first network log data and the string characteristics of the existing classes in the first clustering model, whether the first network log data belongs to an existing class in the first clustering model or to a first class, where the first class is a new class created by the first clustering model; and based on the unique character characteristics of the first network log data and the unique character characteristics of the existing classes in the second clustering model, determine whether the first network log data belongs to an existing class in the second clustering model or to a second class, where the second class is a new class created by the second clustering model, wherein the unique character of the first network log data is a character that is not repeated in the first network log data;

[0018] The determination module is used to determine that the first network log data is abnormal data if the first network log data belongs to the first category and / or the first network log data belongs to the second category.

[0019] In an optional embodiment, the acquisition module is further configured to acquire network log data to be processed; remove spaces, punctuation marks and timestamps from the network log data to be processed, and process the format of the network log data to be processed into a preset format to obtain the first network log data.

[0020] In an optional embodiment, the processing module is specifically configured to: based on the similarity between the string features of the first network log data and the string features of representative log data of existing classes in the first clustering model, select a first cluster with a similarity greater than a first threshold from the existing classes of the first clustering model; filter the first cluster to obtain a second cluster; select the class with the smallest distance to the first network log data from the second cluster as the class to which the first network log data belongs; if the second cluster is empty, determine that the first network log data belongs to the first class.

[0021] In an optional embodiment, the processing module is further configured to filter the first cluster based on length filtering and short word filtering to obtain a second cluster; wherein, the long word filtering includes selecting classes in the first cluster, wherein the length difference between the representative log data of the selected class and the first network log data is within a preset first range; the short word filtering includes selecting classes in the first cluster, wherein the number of matching characters between the representative log data of the selected class and the first network log data is greater than a second threshold.

[0022] In an optional embodiment, the processing module is further configured to: based on the unique character features of the first network log data and the similarity of the unique character features of representative log data of the existing classes of the second clustering model, select a third cluster whose similarity is greater than a second threshold from the existing classes of the second clustering model; select the class with the smallest distance to the first network log data from the third cluster as the class to which the first network log data belongs; if the second cluster is empty, determine that the first network log data belongs to the second class.

[0023] In one optional embodiment, the number of log data in the existing clusters of the first clustering model is greater than or equal to a third threshold; the number of log data in the existing clusters of the second clustering model is greater than or equal to the third threshold.

[0024] Thirdly, embodiments of the present invention provide an anomaly detection device for network log data, comprising: a memory for storing a computer program; and a processor for executing the method described in the first aspect according to the obtained program when executing the computer program stored in the memory.

[0025] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, wherein when a computer reads and executes the computer program, the method described in the first aspect is performed.

[0026] Fifthly, embodiments of the present invention provide a computer program product that, when read and executed by a computer, causes the method described in the first aspect to be executed. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 A schematic diagram illustrating a scenario for the anomaly detection method for network log data provided in this application embodiment;

[0029] Figure 2 A flowchart corresponding to a method for detecting anomalies in network log data provided in an embodiment of this application;

[0030] Figure 3 A flowchart corresponding to a method for detecting anomalies in network log data provided in an embodiment of this application;

[0031] Figure 4 A flowchart corresponding to a method for detecting anomalies in network log data provided in an embodiment of this application;

[0032] Figure 5 A schematic diagram of the structure of an anomaly detection device for network log data provided in an embodiment of this application;

[0033] Figure 6 This is a schematic diagram of the structure of an anomaly detection device for network log data provided in an embodiment of this application. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0035] Based on the exemplary embodiments shown in this application, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this application. Furthermore, although the disclosures in this application are presented by way of one or more exemplary examples, it should be understood that each aspect of these disclosures can constitute a complete technical solution on its own.

[0036] It should be understood that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate, for example, to allow implementation in orders other than those given in the embodiments illustrated or described in this application.

[0037] With the widespread adoption of the internet, the scale of network log data has grown exponentially. Modern network systems generate millions of log data points every day, resulting in a massive volume of log data. The configuration of network systems, user behavior, and network traffic are highly dynamic, meaning that the characteristics and patterns of the generated network log data change over time. Traditional static analysis methods cannot cope with the dynamic changes in network log data. Therefore, an anomaly detection method is needed to capture abnormal data in the network environment in a timely manner.

[0038] Based on this, this application provides an anomaly detection method for network log data. This method processes different features of the first network log data using a first clustering model and a second clustering model, which can flexibly adapt to the dynamic changes of network log data. Based on the fusion result of the two models, it determines whether the first network log data is abnormal data, which can reduce the false alarm rate and improve the detection accuracy.

[0039] Figure 1This illustration shows a scenario diagram of an anomaly detection method for network log data provided in an embodiment of this application. The application scenario includes at least one terminal device 101 and at least one server 102. The terminal device 101 can be a mobile phone, tablet computer, laptop computer, desktop computer, etc., but is not limited to these. Users can access various services and generate corresponding network log data through a client on the terminal device 101. The client can be software (e.g., a browser, instant messaging software, etc.), or a webpage, mini-program, etc. The server 102 is used to store the network log data generated by users accessing services. It can be an independent physical server, a server cluster composed of multiple physical servers, or a distributed system. It can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal device 101 and the server 102 can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0040] It should be noted that, Figure 1 The examples shown are merely illustrative; in reality, the number of terminal devices 101 and servers 102 is not limited and is not specifically limited in this embodiment.

[0041] Figure 2 This is a flowchart illustrating a method for detecting anomalies in network log data provided in this application. This method can be executed by a network log data anomaly detection device (hereinafter referred to as the detection device), which can be... Figure 1 It can be any one of at least one servers, or it can be a standalone deployed device; there is no specific limitation. For example... Figure 2 As shown, the method includes:

[0042] Step 201: The detection device acquires the first network log data.

[0043] For example, the detection device acquires the network log to be processed, preprocesses the network log data to be processed, and obtains the first network log data. There are multiple ways to acquire the network log data to be processed, such as collecting it from network devices, servers, or terminals. No specific limitation is made in this embodiment.

[0044] For example, the detection device preprocesses the network log data to be processed, including data cleaning and data normalization. Data cleaning includes removing spaces, punctuation marks, and timestamps from the network log data to be processed. Data normalization is to process the format of the network log data to be processed into a preset format so that similarity can be calculated under a unified format later. For example, if the characters in the network log data to be processed contain both uppercase and lowercase, and the characters in the preset format are lowercase, then the format of the network log data to be processed needs to be unified to lowercase.

[0045] Step 202: The detection device determines whether the first network log data belongs to an existing class of the first clustering model or to the first class based on the string characteristics of the first network log data and the string characteristics of the existing classes of the first clustering model.

[0046] The first class is a new class created by the first clustering model. When the first network log data does not belong to an existing class of the first clustering model, the first clustering model will create a new class (i.e., the first class) and classify the first network log data into the first class. The existing classes of the first clustering model include at least one cluster, and each cluster includes at least one network log data. Each cluster selects a representative log data as the characteristic feature of the cluster. For example, the existing classes of the first clustering model are a cluster set C, which includes 10 clusters. Each cluster in the cluster set C selects a representative log data as the characteristic feature of the cluster. Each representative log data is determined based on the distance between the character features of the log data in the cluster and the character features of the log data in the remaining 9 clusters. The log data with the smallest distance to the character features of the log data in the remaining 9 clusters is selected as the representative log data.

[0047] For example, the specific process of determining whether the first network log data belongs to an existing class in the first clustering model or belongs to the first class is described in [reference needed]. Figure 3 It mainly includes the following steps:

[0048] Step 2021: Based on the similarity between the string features of the first network log data and the string features of representative log data of the existing classes in the first clustering model, select the first cluster cluster with a similarity greater than the first threshold from the existing classes in the first clustering model.

[0049] For example, the similarity between the string features of the first network log data and the string features of representative log data of the existing classes in the first clustering model is calculated. A first cluster with a similarity greater than a first threshold is selected from the existing classes in the first clustering model, and clusters with low similarity to the first network log data are excluded. Taking the above cluster set C as an example, cluster set C includes 10 clusters. The similarity between the string features of the first network log data and the string features of representative log data of the 10 clusters in cluster set C is calculated. Assuming that 6 clusters have a similarity greater than the first threshold to the string features of the first network log data, these 6 clusters constitute a candidate cluster set C1 (i.e., the first cluster). The similarity calculation is usually based on the distance between strings, which can be edit distance, and is not limited here.

[0050] Step 2022: Filter the first cluster to obtain the second cluster.

[0051] For example, in order to further narrow down the first cluster (i.e., the candidate cluster set C1), the first cluster is filtered based on length filtering and short word filtering to obtain the second cluster.

[0052] The long word filtering includes selecting classes from the first cluster, wherein the length difference between the representative log data of the selected class and the first network log data is within a preset first range; the short word filtering includes selecting classes from the first cluster, wherein the number of matching characters between the representative log data of the selected class and the first network log data is greater than a second threshold.

[0053] For example, the first preset range is ±10%, and the second threshold is 10. When the length difference between the representative log data of the cluster in candidate set C1 (i.e., the first cluster) and the first network log data is within ±10%, and when the number of matching characters between the representative log data of the cluster in candidate set C1 and the first network log data is greater than 10, the cluster is retained. Each cluster in candidate set C1 is traversed to obtain candidate set C2 (i.e., the second cluster).

[0054] Step 2023: Select the class with the smallest distance to the first network log data in the second cluster as the class to which the first network log data belongs. If the second cluster is empty, then the first network log data is determined to be the first class.

[0055] Specifically, in the second cluster, the class with the smallest distance to the first network log data is selected as the target class of the first network log data, and the first network log data is added to the target class in the second cluster. If the second cluster obtained after filtering in steps 2022 and 2023 is empty, a new class is created and the first network log data is added to the new class.

[0056] Optionally, if the second cluster obtained after filtering in steps 2022 and 2023 is not empty, the first network log data is added to the class with the smallest distance from the first network log data in the second cluster, and the first network log data replaces the original representative log data of the class to which it belongs, which can ensure the dynamic nature of the model.

[0057] Step 203: Based on the unique character features of the first network log data and the unique character features of the existing classes in the second clustering model, determine whether the first network log data belongs to an existing class in the second clustering model or belongs to the second class.

[0058] The second category is a new category created by the second clustering model. When the first network log data does not belong to an existing category in the second clustering model, the second clustering model will create a new category (i.e., the second category) and assign the first network log data to the second category. The existing categories of the second clustering model include at least one cluster, and each cluster includes at least one network log data. Each cluster selects a representative log data as its characteristic feature. For example, the existing categories of the second clustering model are cluster set C. ′ Cluster set C ′ It includes n clusters, and the cluster set C ′ Each cluster selects a representative log data point as its defining feature. This representative log data point is determined by the distance between the character features of the log data in that cluster and the character features of the log data in the remaining n-1 clusters. The log data point with the smallest distance to the character features of the log data in the remaining n-1 clusters is selected as the representative log data. The unique characters in the first network log data are those that do not contain repeated characters. For example, if a log data line is "error", then the unique characters in that line are "e", "r", and "o".

[0059] For example, the specific process of determining whether the first network log data belongs to an existing class in the second clustering model or belongs to the second class is described in [reference needed]. Figure 4 It mainly includes the following steps:

[0060] Step 2031: Based on the unique character features of the first network log data and the similarity of the unique character features of the representative log data of the existing classes in the second clustering model, select a third cluster with a similarity greater than the second threshold from the existing classes in the second clustering model.

[0061] For example, by replacing the first network log data in text format and the representative log data of the existing classes of the second clustering model with a numerical space, the unique character features of the first network log data and the unique character features of the representative log data of the existing classes of the second clustering model are obtained. The text format data is converted into a numerical space representation for subsequent numerical calculation. The specific implementation is as follows: count the number of unique characters and unique characters of the first network log data and the representative log data of the existing classes of the second clustering model; take the union of the unique characters of the first network log data and the representative log data of the existing classes of the second clustering model to determine the dimension of the numerical space; set the position corresponding to the unique character in each log data in the first network log data and the representative log data of the existing classes of the second clustering model to 1, and the rest to 0.

[0062] For example, suppose there are n log records. The first log record is the first network log record, and the second to nth log records are representative log records of existing classes in the second clustering model. The unique characters of the first log record are 'a' and 'b', the unique characters of the second log record are 'e' and 'f', ..., the unique characters of the nth log record are 'a', 'e', ​​and 'h'. The union of the unique characters of the n log records is a, b, e, f, h. In this case, the data space dimension of the n log records is 5. Table 1 shows an example of the unique character features of the n log records.

[0063] Table 1: Examples of unique character characteristics of n log records

[0064] a b e f g Log entry 1 1 1 0 0 0 2nd log entry 0 0 1 1 0 ... ... ... ... ... ... The nth log entry 1 0 1 0 1

[0065] For example, in order to overcome the computational complexity caused by high-dimensional data, the unique character features of the first network log data and the unique character features of the representative log data of the existing classes in the second clustering model are reduced in dimensionality based on principal component analysis (PCA). The relationship between the dimensions of the log data is calculated using the covariance matrix. By calculating the eigenvalues ​​and eigenvectors of the covariance matrix, the principal components of the log data are determined. The principal components with larger eigenvalues ​​are selected, which can retain the effective information of the features while reducing dimensionality.

[0066] For the dimensionality-reduced data, calculate the similarity between the unique character features of the first network log data and the unique character features of representative log data from the existing classes of the second clustering model. Select third clusters from the existing classes of the second clustering model whose similarity is greater than a second threshold, and exclude clusters from the existing classes of the second clustering model that have low similarity to the first network log data. The resulting cluster set C is then used. ′ For example, cluster set C ′ It includes n clusters, calculates the unique character features of the first network log data and the cluster set C. ′ The similarity of unique character features of representative log data from n clusters is calculated. Specifically, the similarity between the first log entry in Table 1 and log entries 2 through n is calculated. Assuming that 4 log entries from 2 to n have a similarity greater than the second threshold with the first log entry (i.e., the first network log entry), then these 4 log entries are clustered to form a candidate cluster set C. ′ 1 (i.e., the third cluster), where similarity calculation is usually based on numerical distance, which can be Euclidean distance or Manhattan distance, and is not limited here.

[0067] Step 2032: Select the class with the smallest distance to the first network log data in the third cluster as the class to which the first network log data belongs. If the second cluster is empty, then determine that the first network log data belongs to the second class.

[0068] Specifically, in the third cluster, the class with the smallest distance to the first network log data is selected as the target class of the first network log data, and the first network log data is added to the target class in the third cluster. If the third cluster obtained after filtering in step 2032 is empty, a new class (i.e., the second class) is created, and the first network log data is added to the new class.

[0069] Optionally, if the second cluster obtained after filtering in step 2032 is not empty, the first network log data is added to the class with the smallest distance from the first network log data in the third cluster, and the first network log data replaces the original representative log data of the class to which it belongs, which can ensure the dynamic nature of the model.

[0070] Step 204: If the first network log data belongs to the first category and / or the first network log data belongs to the second category, then the first network log data is determined to be abnormal data.

[0071] Specifically, if the first network log data belongs to the first category and the first network log data belongs to the second category, then the first network log data is determined to be abnormal data; if the first network log data belongs to the first category and the first network log data belongs to an existing category of the second clustering model, then the first network log data is determined to be abnormal data; if the first network log data belongs to an existing category of the first clustering model and the first network log data belongs to the second category, then the first network log data is determined to be abnormal data; if the first network log data belongs to an existing category of the first clustering model and the first network log data belongs to an existing category of the second clustering model, then the first network log data is determined to be normal data.

[0072] Optionally, the first and second clustering models mentioned above are semi-supervised models. Semi-supervised models may contain outlier data during training. To reduce the impact of outlier data on the final detection accuracy, a third threshold is set during the clustering process of both the first and second clustering models to remove outlier clusters, resulting in existing clusters for both models. Specifically, the number of log data entries in both existing clusters of the first and second clustering models is greater than or equal to the third threshold. Removing outlier clusters during the clustering process of both models can reduce the false positive rate of anomaly detection.

[0073] Using the above method, the type of the first network log data is determined based on the string characteristics of the first network log data and the string characteristics of the existing classes in the first clustering model. The type is further determined based on the unique character characteristics of the first network log data and the unique character characteristics of the existing classes in the second clustering model. By characterizing the type of the first network log data from different features, it is easier to flexibly adapt to new data patterns and perform anomaly detection in the context of dynamically changing network logs. The fusion of the first and second clustering models can improve the accuracy of anomaly detection. Furthermore, before determining the category of the first network log data, the existing classes in both the first and second clustering models are filtered to remove clusters that do not meet predefined conditions. This effectively reduces the complexity of subsequent calculations, improves computational efficiency, and consequently reduces memory usage.

[0074] Based on the same technical concept, this application also provides an anomaly detection device 5000 for network log data. Figure 5 A schematic diagram of the structure of the network log data anomaly detection device provided in the embodiments of this application is shown below. Figure 5 As shown, the device 5000 includes:

[0075] Module 501 is used to acquire the first network log data;

[0076] Processing module 501 is configured to determine, based on the string characteristics of the first network log data and the string characteristics of the existing classes of the first clustering model, whether the first network log data belongs to an existing class of the first clustering model or to a first class, wherein the first class is a new class created by the first clustering model; and based on the unique character characteristics of the first network log data and the unique character characteristics of the existing classes of the second clustering model, determine whether the first network log data belongs to an existing class of the second clustering model or to a second class, wherein the second class is a new class created by the second clustering model, wherein the unique character of the first network log data is a character that is not repeated in the first network log data;

[0077] The determination module 503 is used to determine that the first network log data is abnormal data if the first network log data belongs to the first category and / or the first network log data belongs to the second category.

[0078] Optionally, the acquisition module 501 is further configured to acquire network log data to be processed; remove spaces, punctuation marks and timestamps from the network log data to be processed, and process the format of the network log data to be processed into a preset format to obtain the first network log data.

[0079] Optionally, the processing module 502 is specifically configured to: based on the similarity between the string features of the first network log data and the string features of representative log data of the existing classes of the first clustering model, select a first cluster with a similarity greater than a first threshold from the existing classes of the first clustering model; filter the first cluster to obtain a second cluster; select the class with the smallest distance to the first network log data from the second cluster as the class to which the first network log data belongs; if the second cluster is empty, determine that the first network log data is the first class.

[0080] Optionally, the processing module 502 is further configured to filter the first cluster based on length filtering and short word filtering to obtain a second cluster; wherein, the long word filtering includes selecting classes in the first cluster, wherein the length difference between the representative log data of the selected class and the first network log data is within a preset first range; the short word filtering includes selecting classes in the first cluster, wherein the number of matching characters between the representative log data of the selected class and the first network log data is greater than a second threshold.

[0081] Optionally, the processing module 502 is further configured to: based on the unique character features of the first network log data and the similarity of the unique character features of representative log data of the existing classes of the second clustering model, select a third cluster whose similarity is greater than a second threshold from the existing classes of the second clustering model; select the class with the smallest distance to the first network log data from the third cluster as the class to which the first network log data belongs; if the second cluster is empty, determine that the first network log data belongs to the second class.

[0082] Optionally, the number of log data in the existing clusters of the first clustering model is greater than or equal to the third threshold; the number of log data in the existing clusters of the second clustering model is greater than or equal to the third threshold.

[0083] Based on the same technological concept Figure 6 This is a schematic diagram of the structure of a device 6000 provided in an embodiment of this application, as shown below. Figure 6 As shown, the device 6000 includes at least one processor 601 and a memory 602 connected to the at least one processor 601. In this embodiment, the specific connection medium between the processor 601 and the memory 602 is not limited. Figure 6 Taking the connection between processor 601 and memory 602 via a bus as an example, the bus can be divided into address bus, data bus, control bus, etc. In this embodiment of the invention, memory 602 stores instructions that can be executed by at least one processor 601. By executing the instructions stored in memory 602, at least one processor 601 can implement the steps of the above-mentioned anomaly detection method for network log data.

[0084] The processor 601 is the control center of the computer device, capable of connecting various parts of the computer device via various interfaces and lines. It performs resource configuration by running or executing instructions stored in the memory 602 and accessing data stored in the memory 602. Optionally, the processor 601 may include one or more processing units. The processor 601 may integrate an application processor and a modem processor. The application processor primarily handles the operating system, user interface, and applications, while the modem processor primarily handles wireless communication. It is understood that the modem processor may not be integrated into the processor 601. In some embodiments, the processor 601 and the memory 602 may be implemented on the same chip; in other embodiments, they may be implemented on separate chips.

[0085] Processor 601 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0086] Memory 602, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 602 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. Memory 602 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. In the embodiments of this application, memory 602 can also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0087] Based on the same technical concept, embodiments of the present invention provide a computer-readable storage medium storing a computer program, wherein the computer program is executed by a processor using the above-described method for detecting anomalies in network log data.

[0088] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0089] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0090] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0091] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0092] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for detecting anomalies in network log data, characterized in that, The method includes: Obtain the first network log data; Based on the similarity between the string features of the first network log data and the string features of representative log data of the existing classes of the first clustering model, a first cluster with a similarity greater than a first threshold is selected from the existing classes of the first clustering model. The first cluster is filtered based on length filtering and short word filtering to obtain a second cluster; wherein, the length filtering includes selecting classes in the first cluster, and the length difference between the representative log data of the selected class and the first network log data is within a preset first range; the short word filtering includes selecting classes in the first cluster, and the number of matching characters between the representative log data of the selected class and the first network log data is greater than a second threshold. In the second cluster, the class with the smallest distance to the first network log data is selected as the class to which the first network log data belongs. If the second cluster is empty, the first network log data is determined to be the first class, which is a new class created by the first clustering model. Based on the unique character features of the first network log data and the similarity of the unique character features of the representative log data of the existing classes of the second clustering model, a third cluster with a similarity greater than a second threshold is selected from the existing classes of the second clustering model. In the third cluster, the class with the smallest distance to the first network log data is selected as the class to which the first network log data belongs. If the second cluster is empty, the first network log data is determined to be the second class. The second class is a new class created by the second clustering model. The unique character of the first network log data is that there are no duplicate characters in the first network log data. If the first network log data belongs to the first category and / or the first network log data belongs to the second category, then the first network log data is determined to be abnormal data.

2. The method according to claim 1, characterized in that, The method further includes: Obtain network log data to be processed; Remove spaces, punctuation marks, and timestamps from the network log data to be processed, and process the network log data to be processed into a preset format to obtain the first network log data.

3. The method according to claim 1, characterized in that, The number of log data in the existing clusters of the first clustering model is greater than or equal to the third threshold; The number of log data in the existing clusters of the second clustering model is greater than or equal to the third threshold.

4. An anomaly detection device for network log data, characterized in that, The device includes: The acquisition module is used to acquire the first network log data; The processing module is configured to: ... The class to which the first network log data belongs is determined. If the second cluster is empty, the first network log data is determined to be of the first class, which is a new class created by the first clustering model. Based on the unique character features of the first network log data and the similarity of the unique character features of representative log data of the existing classes of the second clustering model, a third cluster with a similarity greater than a second threshold is selected from the existing classes of the second clustering model. The class with the smallest distance to the first network log data is selected from the third cluster as the class to which the first network log data belongs. If the second cluster is empty, the first network log data is determined to be of the second class, which is a new class created by the second clustering model. The unique character of the first network log data is a character that is not repeated in the first network log data. The determination module is used to determine that the first network log data is abnormal data if the first network log data belongs to the first category and / or the first network log data belongs to the second category.

5. An anomaly detection device for network log data, characterized in that, The device includes: Memory, used to store program instructions; A processor is configured to invoke program instructions stored in the memory and execute the steps of the method according to any one of claims 1-3 in accordance with the obtained program instructions.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a computer, cause the method as described in any one of claims 1-3 to be performed.

7. A computer program product, characterized in that, The computer program product includes computer program code that, when run on a computer, causes any one of claims 1-3 to be executed.

Images