Abnormality detection method and device for weblog data

Abstract

The invention relates to the technical field of log processing, and discloses a weblog data anomaly detection method and device. The method comprises the following steps: acquiring first weblog data; according to the character string feature of the first weblog data and the character string feature of the existing class of the first clustering model, determining that the first weblog data belongs to the existing class of the first clustering model or belongs to the first class; according to the unique character feature of the first weblog data and the unique character feature of the existing class of the second clustering model, determining that the first weblog data belongs to the existing class of the second clustering model or belongs to a second class; and if the first weblog data belongs to the first class and / or the first weblog data belongs to the second class, determining that the first weblog data is abnormal data. Thus, anomaly detection is performed on the first weblog data based on the character string feature and the unique character feature of the first weblog data, dynamically changing log data can be flexibly processed, and the detection precision is improved.

Description

Technical Field

[0001] The present application relates to the technical field of log processing, and in particular to a method and device for detecting anomalies in network log data. Background Art

[0002] With the rapid development of network technology and the widespread application of the Internet, network log data has exploded. Enterprises put various application services on remote server platforms, and the traces of enterprise, organization and individual activities on the network are recorded in the network system in the form of log data. Network log data is an important resource for understanding network activities, system health and user behavior.

[0003] However, while network log data brings many conveniences to people, network attacks against big data platform servers have become more common. As network threats and attack methods continue to evolve, the characteristics and patterns of network log data will change over time. Therefore, how to detect anomalies in network log data when network log data changes dynamically still needs further research. Summary of the invention

[0004] The embodiments of the present application provide a method and device for detecting anomalies in network log data, which are used to detect network log data based on string features and unique character features of the network log data, thereby improving the detection accuracy of the network log data when the network log data changes dynamically.

[0005] In a first aspect, an embodiment of the present application provides a method for detecting anomalies in network log data, the method comprising: acquiring first network log data; determining, based on a string feature of the first network log data and a string feature of an existing class of a first clustering model, whether the first network log data belongs to an existing class of the first clustering model or to a first class, wherein the first class is a new class created by the first clustering model; determining, based on a unique character feature of the first network log data and a unique character feature of an existing class of a second clustering model, whether the first network log data belongs to an existing class of the second clustering model or to a second class, wherein the second class is a new class created by the second clustering model, wherein the unique characters of the first network log data are characters that are not repeated in the first network log data; if the first network log data belongs to the first class and / or the first network log data belongs to the second class, determining that the first network log data is abnormal data.

[0006] By adopting the above method, the type of the first network log data is determined according to the string features of the first network log data and the string features of the existing classes of the first clustering model; and the type of the first network log data is determined according to the unique character features of the first network log data and the unique character features of the existing classes of the second clustering model; that is, the type of the first network log data is characterized from different features respectively, so as to facilitate flexible adaptation to new patterns of log data, so as to perform anomaly detection on dynamically changing network log data, reduce the model's dependence on data staticity, and improve the accuracy of anomaly detection.

[0007] In an optional embodiment, the method further includes: acquiring network log data to be processed; removing spaces, punctuation marks and timestamps in the network log data to be processed, and processing the format of the network log data to be processed into a preset format to obtain the first network log data.

[0008] Using the above method, since the clustering model pays more attention to the text similarity between log data rather than the specific time, space and punctuation, removing the spaces, punctuation and timestamps in the network log data to be processed can reduce the amount of calculation, and processing the format of the network log data to be processed into a preset format can ensure the uniformity of the data format, so as to perform similarity calculation more accurately.

[0009] In an optional embodiment, determining whether the first network log data belongs to the existing class of the first clustering model or the first class based on the string features of the first network log data and the string features of the existing class of the first clustering model includes: screening out a first clustering cluster having a similarity greater than a first threshold from the existing classes of the first clustering model based on the similarity between the string features of the first network log data and the string features of representative log data of the existing class of the first clustering model; screening the first clustering cluster to obtain a second clustering cluster; selecting a class with the smallest distance from the first network log data in the second clustering cluster as the class to which the first network log data belongs, and if the second clustering cluster is empty, determining that the first network log data belongs to the first class.

[0010] In an optional embodiment, the first clustering cluster is screened to obtain a second clustering cluster, including: screening the first clustering cluster based on length filtering and short word filtering to obtain a second clustering cluster; wherein the long word filtering includes selecting a class in the first clustering cluster, and the length difference between the representative log data of the selected class and the first network log data is within a preset first range; the short word filtering includes selecting a class in the first clustering cluster, and the number of matching characters between the representative log data of the selected class and the first network log data is greater than a second threshold.

[0011] By adopting the above method, clusters that do not meet the predefined conditions in the existing classes of the first clustering model are removed based on length filtering and short word filtering, which can reduce the complexity of subsequent calculations and improve calculation efficiency.

[0012] In an optional embodiment, determining whether the first network log data belongs to the existing class of the second clustering model or the second class based on the unique character features of the first network log data and the unique character features of the existing class of the second clustering model includes: screening out a third clustering cluster having a similarity greater than a second threshold from the existing classes of the second clustering model based on the similarity between the unique character features of the first network log data and the representative log data of the existing class of the second clustering model; selecting a class with the smallest distance from the first network log data in the third clustering cluster as the class to which the first network log data belongs, and if the second clustering cluster is empty, determining that the first network log data belongs to the second class.

[0013] In an optional embodiment, the number of log data in the existing classes of the first clustering model is greater than or equal to a third threshold; the number of log data in the existing classes of the second clustering model is greater than or equal to the third threshold.

[0014] Using the above method, since there may be abnormal data in the clustering process of the first clustering model and the second clustering model, by setting the third threshold and eliminating abnormal clusters, the existing classes of the first clustering model and the existing classes of the second clustering model are obtained, which can reduce the impact of abnormal data on the final detection accuracy and facilitate improving the accuracy of anomaly detection.

[0015] In a second aspect, an embodiment of the present application provides a device for detecting anomalies in network log data, the device comprising:

[0016] An acquisition module, used for acquiring first network log data;

[0017] a processing module, configured to determine, based on the character string features of the first network log data and the character string features of the existing classes of the first clustering model, whether the first network log data belongs to the existing classes of the first clustering model or to the first class, wherein the first class is a new class created by the first clustering model; and to determine, based on the unique character features of the first network log data and the unique character features of the existing classes of the second clustering model, whether the first network log data belongs to the existing classes of the second clustering model or to the second class, wherein the second class is a new class created by the second clustering model, wherein the unique characters of the first network log data are characters that are not repeated in the first network log data;

[0018] The determination module is configured to determine that the first network log data is abnormal data if the first network log data belongs to the first category and / or the first network log data belongs to the second category.

[0019] In an optional embodiment, the acquisition module is further used to acquire the network log data to be processed; remove spaces, punctuation marks and timestamps in the network log data to be processed, and process the format of the network log data to be processed into a preset format to obtain the first network log data.

[0020] In an optional embodiment, the processing module is specifically used to, based on the similarity between the string features of the first network log data and the string features of representative log data of the existing classes of the first clustering model, screen out a first clustering cluster whose similarity is greater than a first threshold from the existing classes of the first clustering model; screen the first clustering cluster to obtain a second clustering cluster; select the class with the shortest distance from the first network log data in the second clustering cluster as the class to which the first network log data belongs, and if the second clustering cluster is empty, determine that the first network log data is of the first class.

[0021] In an optional embodiment, the processing module is also used to screen the first clustering cluster based on length filtering and short word filtering to obtain a second clustering cluster; wherein the long word filtering includes selecting a class in the first clustering cluster, and the length difference between the representative log data of the selected class and the first network log data is within a preset first range; the short word filtering includes selecting a class in the first clustering cluster, and the number of matching characters between the representative log data of the selected class and the first network log data is greater than a second threshold.

[0022] In an optional embodiment, the processing module is also used to screen out a third clustering cluster with a similarity greater than a second threshold from the existing classes of the second clustering model based on the similarity between the unique character features of the first network log data and the representative log data of the existing classes of the second clustering model; select the class with the smallest distance to the first network log data from the third clustering cluster as the class to which the first network log data belongs, and if the second clustering cluster is empty, determine that the first network log data belongs to the second class.

[0023] In an optional embodiment, the number of log data in the existing classes of the first clustering model is greater than or equal to a third threshold; the number of log data in the existing classes of the second clustering model is greater than or equal to the third threshold.

[0024] In a third aspect, an embodiment of the present invention provides an anomaly detection device for network log data, comprising: a memory for storing a computer program; and a processor for executing the method described in the first aspect according to the obtained program when executing the computer program stored in the memory.

[0025] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when a computer reads and executes the computer program, the method described in the first aspect above is executed.

[0026] In a fifth aspect, an embodiment of the present invention provides a computer program product. When a computer reads and executes the computer program product, the method described in the first aspect is executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0028] Figure 1 A schematic diagram of a scenario of an anomaly detection method for network log data provided in an embodiment of the present application;

[0029] Figure 2 A flowchart corresponding to a method for detecting anomalies in network log data provided in an embodiment of the present application;

[0030] Figure 3 A flowchart corresponding to a method for detecting anomalies in network log data provided in an embodiment of the present application;

[0031] Figure 4 A flowchart corresponding to a method for detecting anomalies in network log data provided in an embodiment of the present application;

[0032] Figure 5 A schematic diagram of the structure of a network log data anomaly detection device provided in an embodiment of the present application;

[0033] Figure 6 A schematic diagram of the structure of a network log data anomaly detection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solutions and advantages of this application clearer, this application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0035] Based on the exemplary embodiments shown in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application. In addition, although the disclosure in this application is introduced according to one or several exemplary examples, it should be understood that each aspect of these disclosures can also constitute a complete technical solution separately.

[0036] It should be understood that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, for example, they can be implemented according to an order other than those given in the diagrams or descriptions of the embodiments of this application.

[0037] With the popularization of the Internet, the scale of network log data has grown exponentially. Modern network systems generate millions of log data every day, which makes the scale of log data huge. The configuration of network systems, user behavior, and network traffic are highly dynamic, which means that the characteristics and patterns of the generated network log data will change over time. Traditional static analysis methods cannot cope with the dynamic changes of network log data. Therefore, an anomaly detection method is needed to capture abnormal data in a network environment in a timely manner.

[0038] Based on this, an embodiment of the present application provides a method for detecting anomalies in network log data. The method processes different features of the first network log data respectively through a first clustering model and a second clustering model, can flexibly adapt to the dynamic changes of the network log data, and judges whether the first network log data is abnormal data based on the fusion result of the two models, which can reduce the false alarm rate and improve the detection accuracy.

[0039] Figure 1The schematic diagram of the scenario of the abnormal detection method of network log data provided by the embodiment of the present application is exemplarily shown, and the application scenario includes at least one terminal device 101 and at least one server 102. Among them, the terminal device 101 can be a mobile phone, a tablet computer, a laptop computer, a desktop computer and other devices, but is not limited to this. The user can access various services through the client on the terminal device 101 and generate corresponding network log data. The client can be software (such as a browser, instant messaging software, etc.), or a web page, a small program, etc. The server 102 is used to store the network log data generated by the user accessing the service. It can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms. The terminal device 101 and the server 102 can be directly or indirectly connected by wired or wireless communication, and this application is not limited here.

[0040] It should be noted that Figure 1 The figure is only an example. In fact, the number of terminal devices 101 and servers 102 is not limited and is not specifically limited in the embodiments of the present application.

[0041] Figure 2 This is a flowchart corresponding to a method for detecting anomalies in network log data provided in an embodiment of the present application. The method can be executed by a device for detecting anomalies in network log data (hereinafter referred to as a detection device), which can be Figure 1 Any server in at least one server, or an independently deployed device, is not specifically limited. Figure 2 As shown, the method includes:

[0042] Step 201: The detection device obtains first network log data.

[0043] Exemplarily, the detection device obtains the network log to be processed, preprocesses the network log data to be processed, and obtains the first network log data. There are many ways to obtain the network log data to be processed here, and the network log data to be processed can be collected from network devices, servers, terminals and other devices. No specific limitation is made in this embodiment.

[0044] Exemplarily, the detection device performs preprocessing on the network log data to be processed, including data cleaning and data normalization, wherein data cleaning includes removing spaces, punctuation marks and timestamps in the network log data to be processed, and data normalization is to process the format of the network log data to be processed into a preset format so that the similarity can be calculated in a unified format later. For example, the characters in the network log data to be processed contain uppercase and lowercase, and the characters in the preset format are lowercase. At this time, the format of the network log data to be processed needs to be unified into lowercase.

[0045] Step 202: The detection device determines whether the first network log data belongs to the existing class of the first clustering model or to the first class according to the character string features of the first network log data and the character string features of the existing class of the first clustering model.

[0046] Among them, the first category is a new category created by the first clustering model. When the first network log data does not belong to the existing category of the first clustering model, the first clustering model will create a new category (i.e., the first category) and classify the first network log data into the first category. The existing categories of the first clustering model include at least one cluster, each cluster includes at least one network log data, and each cluster selects a representative log data as the iconic feature of the cluster. For example, the existing category of the first clustering model is cluster set C, and cluster set C includes 10 clusters. Each cluster in cluster set C selects a representative log data as the iconic feature of the cluster. Each representative log data is determined based on the distance between the character features of the log data in the cluster and the character features of the log data in the remaining 9 clusters. The log data with the smallest distance to the character features of the log data in the remaining 9 clusters is selected as the representative log data.

[0047] For example, the specific process of determining that the first network log data belongs to an existing class of the first clustering model or belongs to the first class is as follows: Figure 3 , mainly including the following steps:

[0048] Step 2021 , based on the similarity between the string feature of the first network log data and the string feature of the representative log data of the existing class of the first clustering model, select a first cluster whose similarity is greater than a first threshold from the existing classes of the first clustering model.

[0049] Exemplarily, the similarity between the string features of the first network log data and the string features of the representative log data of the existing classes of the first clustering model is calculated, and the first cluster cluster whose similarity is greater than the first threshold is screened out from the existing classes of the first clustering model, and the clusters in the existing classes of the first clustering model with low similarity to the first network log data are excluded. Taking the above-mentioned cluster set C as an example, the cluster set C includes 10 clusters, and the similarity between the string features of the first network log data and the string features of the representative log data of the 10 clusters in the cluster set C is calculated. Assuming that at this time there are 6 clusters whose similarity with the string features of the first network log data is greater than the first threshold, these 6 clusters constitute the candidate cluster set C1 (i.e., the first cluster cluster), wherein the similarity calculation is usually obtained based on the distance of the string, which can be the edit distance, which is not limited here.

[0050] Step 2022: Screen the first cluster to obtain a second cluster.

[0051] Exemplarily, in order to further narrow down the first cluster (ie, the candidate cluster set C1), the first cluster is screened based on length filtering and short word filtering to obtain a second cluster.

[0052] Among them, the long word filtering includes selecting a class in the first clustering cluster, and the length difference between the representative log data of the selected class and the first network log data is within a preset first range; the short word filtering includes selecting a class in the first clustering cluster, and the number of matching characters between the representative log data of the selected class and the first network log data is greater than a second threshold.

[0053] For example, the preset first range is ±10%, and the second threshold is 10. When the length difference between the representative log data of the cluster in the candidate set C1 (i.e., the first cluster cluster) and the first network log data is within ±10%, and when the number of matching characters between the representative log data of the cluster in the candidate set C1 and the first network log data is greater than 10, the cluster is retained, and each cluster in the candidate set C1 is traversed to obtain the candidate set C2 (i.e., the second cluster cluster).

[0054] Step 2023 , selecting a class with the shortest distance from the first network log data in the second cluster as the class to which the first network log data belongs. If the second cluster is empty, determining that the first network log data belongs to the first class.

[0055] Specifically, a class with the shortest distance to the first network log data is selected in the second cluster as the target class of the first network log data, and the first network log data is added to the target class in the second cluster. If the second cluster obtained by screening in steps 2022 and 2023 is empty, a new class is created and the first network log data is added to the new class.

[0056] Optionally, if the second cluster obtained by screening in steps 2022 and 2023 is not empty, the first network log data is added to the class in the second cluster that has the smallest distance from the first network log data, and the first network log data replaces the original representative log data of the class to which it belongs, so as to ensure the dynamic nature of the model.

[0057] Step 203 : determining whether the first web log data belongs to the existing class of the second clustering model or to the second class based on the unique character feature of the first web log data and the unique character feature of the existing class of the second clustering model.

[0058] The second category is a new category created by the second clustering model. When the first network log data does not belong to the existing category of the second clustering model, the second clustering model will create a new category (i.e., the second category) and classify the first network log data into the second category. The existing categories of the second clustering model include at least one cluster, each cluster includes at least one network log data, and each cluster selects a representative log data as the iconic feature of the cluster. For example, the existing categories of the second clustering model are cluster set C ′ , cluster set C ′ There are n clusters in the cluster set C ′ Each cluster in selects a representative log data as the characteristic feature of the cluster. Each representative log data is determined based on the distance between the character feature of the log data in the cluster and the character feature of the log data in the remaining n-1 clusters. The log data with the smallest distance to the character feature of the log data in the remaining n-1 clusters is selected as the representative log data. The unique characters of the first network log data are the characters that do not have duplication in the first network log data. For example, assuming that a line of log data is error, the unique characters of the line of log data are e, r, and o.

[0059] For example, the specific process of determining that the first network log data belongs to the existing class of the second clustering model or belongs to the second class is as follows. Figure 4 , mainly including the following steps:

[0060] Step 2031, based on the similarity between the unique character feature of the first network log data and the unique character feature of the representative log data of the existing class of the second clustering model, a third cluster whose similarity is greater than a second threshold is selected from the existing classes of the second clustering model;

[0061] Exemplarily, by converting the first network log data in text format and the representative log data of the existing classes of the second clustering model into a numerical space, the unique character features of the first network log data and the representative log data of the existing classes of the second clustering model are obtained, and the data in text format are converted into a numerical space representation for subsequent numerical calculations. The specific implementation is as follows: the number of unique characters and the unique characters of the representative log data of the existing classes of the first network log data and the second clustering model are counted, the dimension of the numerical space is determined by taking the union of the unique characters of the representative log data of the existing classes of the first network log data and the second clustering model, and the position corresponding to the unique character involved in each log data in the representative log data of the existing classes of the first network log data and the second clustering model is set to 1, and the rest are set to 0.

[0062] For example, suppose there are n log data, the first log data is the first network log data, the second to n log data are representative log data of the existing class of the second clustering model, the unique characters of the first log data are a and b, the unique characters of the second log data are e and f, ... the unique characters of the nth log data are a, e and h, and the union of the unique characters of the n log data is a, b, e, f, h. At this time, the data space dimension of the n log data is 5. Table 1 is an example of the unique character features of the n log data, as shown in Table 1.

[0063] Table 1: Examples of unique character features of n log data

[0064] a b e f g Article 1 Log Data 1 1 0 0 0 Article 2 Log Data 0 0 1 1 0 ... ... ... ... ... ... The nth log data 1 0 1 0 1

[0065] Exemplarily, in order to overcome the computational complexity caused by high-dimensional data, the unique character features of the first network log data and the unique character features of the representative log data of the existing classes of the second clustering model are reduced in dimensionality based on principal component analysis (PCA). The covariance matrix is ​​used to calculate the relationship between the dimensions of the log data. The principal components of the log data are determined by calculating the eigenvalues ​​and eigenvectors of the covariance matrix. The principal components with larger eigenvalues ​​are selected, which can reduce the dimensionality while retaining the effective information of the features.

[0066] For the data after dimensionality reduction, the similarity between the unique character features of the first network log data and the unique character features of the representative log data of the existing classes of the second clustering model is calculated, and the third cluster cluster with a similarity greater than the second threshold is selected from the existing classes of the second clustering model, and the clusters with low similarity to the first network log data in the existing classes of the second clustering model are excluded. ′ For example, the cluster set C ′ There are n clusters in the network, and the unique character features and cluster set C of the first network log data are calculated. ′ The similarity of the unique character features of the representative log data of the n clusters in Table 1 is calculated. That is, the similarity between the first log data and the 2 to n log data in Table 1 is calculated. Assume that at this time, the similarity between 4 log data in the 2 to n log data and the 1st log data (i.e., the first network log data) is greater than the second threshold, then the clusters of these 4 log data constitute the candidate cluster set C ′ 1 (i.e., the third cluster), wherein the similarity calculation is usually obtained based on the numerical distance, which can be the Euclidean distance or the Manhattan distance, which is not limited here.

[0067] Step 2032: Select the class with the shortest distance from the first network log data in the third cluster as the class to which the first network log data belongs. If the second cluster is empty, determine that the first network log data belongs to the second class.

[0068] Specifically, in the third cluster, the class with the shortest distance to the first network log data is selected as the target class of the first network log data, and the first network log data is added to the target class in the third cluster. If the third cluster obtained by screening in step 2032 is empty, a new class (i.e., the second class) is created and the first network log data is added to the new class.

[0069] Optionally, if the second cluster obtained by screening in step 2032 is not empty, the first network log data is added to the class with the smallest distance from the first network log data in the third cluster, and the first network log data replaces the original representative log data of the class to which it belongs, so as to ensure the dynamic nature of the model.

[0070] Step 204: If the first network log data belongs to the first category and / or the first network log data belongs to the second category, then determine that the first network log data is abnormal data.

[0071] Specifically, if the first network log data belongs to the first category and the first network log data belongs to the second category, then the first network log data is determined to be abnormal data; if the first network log data belongs to the first category and the first network log data belongs to an existing class of the second clustering model, then the first network log data is determined to be abnormal data; if the first network log data belongs to the existing class of the first clustering model and the first network log data belongs to the second category, then the first network log data is determined to be abnormal data; if the first network log data belongs to the existing class of the first clustering model and the first network log data belongs to the existing class of the second clustering model, then the first network log data is determined to be normal data.

[0072] Optionally, the first clustering model and the second clustering model adopt a semi-supervised model. There may be abnormal data in the training process of the semi-supervised model. In order to reduce the impact of abnormal data on the final detection accuracy, during the clustering process of the first clustering model and the second clustering model, a third threshold is set to eliminate abnormal clusters in the clustering process of the first clustering model and the second clustering model, and the existing classes of the first clustering model and the existing classes of the second clustering model are obtained, wherein the number of log data in the existing classes of the first clustering model is greater than or equal to the third threshold; the number of log data in the existing classes of the second clustering model is greater than or equal to the third threshold. Eliminating abnormal clusters during the clustering process of the first clustering model and the second clustering model can reduce the false alarm rate of abnormality detection.

[0073] By adopting the above method, the type of the first network log data is determined according to the string features of the first network log data and the string features of the existing classes of the first clustering model, and the type of the first network log data is determined according to the unique character features of the first network log data and the unique character features of the existing classes of the second clustering model. The type of the first network log data is characterized from different features, so as to flexibly adapt to the new mode of data, and perform anomaly detection in the case of dynamic changes in network logs. Based on the fusion of the first clustering model and the second clustering model, the accuracy of anomaly detection can be improved. At the same time, before determining the category to which the first network log data belongs, the existing classes of the first clustering model and the existing classes of the second clustering model are screened to remove clusters that do not meet the predefined conditions, which can effectively reduce the complexity of subsequent calculations, improve calculation efficiency, and thus reduce memory usage.

[0074] Based on the same technical concept, the embodiment of the present application also provides an abnormality detection device 5000 for network log data. Figure 5 A schematic diagram of the structure of the network log data anomaly detection device provided in the embodiment of the present application, as shown in FIG. Figure 5 As shown, the device 5000 includes:

[0075] An acquisition module 501 is used to acquire first network log data;

[0076] Processing module 501, for determining, based on the string features of the first network log data and the string features of the existing classes of the first clustering model, whether the first network log data belongs to the existing classes of the first clustering model or to the first class, wherein the first class is a new class created by the first clustering model; determining, based on the unique character features of the first network log data and the unique character features of the existing classes of the second clustering model, whether the first network log data belongs to the existing classes of the second clustering model or to the second class, wherein the second class is a new class created by the second clustering model, wherein the unique characters of the first network log data are characters that are not repeated in the first network log data;

[0077] The determination module 503 is configured to determine that the first network log data is abnormal data if the first network log data belongs to the first category and / or the first network log data belongs to the second category.

[0078] Optionally, the acquisition module 501 is further used to acquire the network log data to be processed; remove spaces, punctuation marks and timestamps in the network log data to be processed, and process the format of the network log data to be processed into a preset format to obtain the first network log data.

[0079] Optionally, the processing module 502 is specifically used to, based on the similarity between the string features of the first network log data and the string features of representative log data of the existing classes of the first clustering model, screen out a first clustering cluster whose similarity is greater than a first threshold from the existing classes of the first clustering model; screen the first clustering cluster to obtain a second clustering cluster; and select a class with the shortest distance to the first network log data from the second clustering cluster as the class to which the first network log data belongs. If the second clustering cluster is empty, the first network log data is determined to be of the first class.

[0080] Optionally, the processing module 502 is also specifically used to screen the first clustering cluster based on length filtering and short word filtering to obtain a second clustering cluster; wherein the long word filtering includes selecting a class in the first clustering cluster, and the length difference between the representative log data of the selected class and the first network log data is within a preset first range; the short word filtering includes selecting a class in the first clustering cluster, and the number of matching characters between the representative log data of the selected class and the first network log data is greater than a second threshold.

[0081] Optionally, the processing module 502 is also specifically used to screen out a third clustering cluster with a similarity greater than a second threshold from the existing classes of the second clustering model based on the similarity between the unique character features of the first network log data and the representative log data of the existing classes of the second clustering model; select the class with the smallest distance to the first network log data from the third clustering cluster as the class to which the first network log data belongs; if the second clustering cluster is empty, determine that the first network log data belongs to the second class.

[0082] Optionally, the number of log data in the existing classes of the first clustering model is greater than or equal to a third threshold; the number of log data in the existing classes of the second clustering model is greater than or equal to the third threshold.

[0083] Based on the same technical concept, Figure 6 A schematic diagram of the structure of a device 6000 provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the device 6000 includes at least one processor 601 and a memory 602 connected to the at least one processor 601. The specific connection medium between the processor 601 and the memory 602 is not limited in the embodiment of the present application. Figure 6 For example, the processor 601 and the memory 602 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, etc. In an embodiment of the present invention, the memory 602 stores instructions that can be executed by at least one processor 601. The at least one processor 601 can implement the steps of the above-mentioned network log data anomaly detection method by executing the instructions stored in the memory 602.

[0084] The processor 601 is the control center of the computer device, and can use various interfaces and lines to connect various parts of the computer device, and perform resource settings by running or executing instructions stored in the memory 602 and calling data stored in the memory 602. Optionally, the processor 601 may include one or more processing units, and the processor 601 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 601. In some embodiments, the processor 601 and the memory 602 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on independent chips.

[0085] Processor 601 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware processor for execution, or can be executed by a combination of hardware and software modules in the processor.

[0086] The memory 602 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 602 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (RAM), a static random access memory (SRAM), a programmable read-only memory (PROM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic memory, a disk, an optical disk, and the like. The memory 602 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 602 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0087] Based on the same technical concept, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the above-mentioned method for detecting abnormalities in network log data.

[0088] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0089] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0090] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0091] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0092] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for detecting anomalies in network log data, characterized in that: The method comprises: Obtaining first network log data; Determining, based on the string features of the first network log data and the string features of the existing classes of the first clustering model, whether the first network log data belongs to the existing classes of the first clustering model or to the first class, where the first class is a new class created by the first clustering model; Determining, based on the unique character features of the first web log data and the unique character features of the existing classes of the second clustering model, that the first web log data belongs to the existing classes of the second clustering model or to the second class, where the second class is a new class created by the second clustering model, and the unique characters of the first web log data are characters that are not repeated in the first web log data; If the first network log data belongs to the first category and / or the first network log data belongs to the second category, it is determined that the first network log data is abnormal data.

2. The method according to claim 1, characterized in that The method further comprises: Get the network log data to be processed; The spaces, punctuation marks and timestamps in the network log data to be processed are removed, and the format of the network log data to be processed is processed into a preset format to obtain the first network log data.

3. The method according to claim 1, characterized in that Determining, according to the character string features of the first network log data and the character string features of the existing classes of the first clustering model, whether the first network log data belongs to the existing classes of the first clustering model or to the first class includes: According to the similarity between the string feature of the first network log data and the string feature of the representative log data of the existing class of the first clustering model, a first clustering cluster having a similarity greater than a first threshold is selected from the existing classes of the first clustering model; Screening the first cluster to obtain a second cluster; A class having the smallest distance to the first network log data is selected from the second cluster as the class to which the first network log data belongs. If the second cluster is empty, the first network log data is determined to be of the first class.

4. The method according to claim 3, characterized in that The first cluster is screened to obtain a second cluster, comprising: Screening the first cluster based on length filtering and short word filtering to obtain a second cluster; The long word filtering includes selecting a class in the first cluster, and the length difference between the representative log data of the selected class and the first network log data is within a preset first range; The short word filtering includes selecting a class in the first clustering cluster, and the number of matching characters between the representative log data of the selected class and the first network log data is greater than a second threshold.

5. The method according to claim 1, characterized in that Determining, according to the unique character feature of the first web log data and the unique character feature of the existing class of the second clustering model, that the first web log data belongs to the existing class of the second clustering model or belongs to the second class includes: According to the similarity between the unique character feature of the first network log data and the unique character feature of the representative log data of the existing class of the second clustering model, a third cluster whose similarity is greater than a second threshold is selected from the existing classes of the second clustering model; A class having the smallest distance to the first network log data is selected from the third cluster as the class to which the first network log data belongs; if the second cluster is empty, the first network log data is determined to be of the second class.

6. The method according to claim 1, characterized in that The number of log data in the existing classes of the first clustering model is greater than or equal to a third threshold; The number of log data in the existing classes of the second clustering model is greater than or equal to the third threshold.

7. A network log data anomaly detection device, characterized in that: The device comprises: An acquisition module, used for acquiring first network log data; a processing module, configured to determine, based on the character string features of the first network log data and the character string features of the existing classes of the first clustering model, whether the first network log data belongs to the existing classes of the first clustering model or to the first class, wherein the first class is a new class created by the first clustering model; and to determine, based on the unique character features of the first network log data and the unique character features of the existing classes of the second clustering model, whether the first network log data belongs to the existing classes of the second clustering model or to the second class, wherein the second class is a new class created by the second clustering model, wherein the unique characters of the first network log data are characters that are not repeated in the first network log data; The determination module is configured to determine that the first network log data is abnormal data if the first network log data belongs to the first category and / or the first network log data belongs to the second category.

8. A network log data anomaly detection device, characterized in that: The device comprises: A memory for storing program instructions; A processor is used to call the program instructions stored in the memory, and execute the steps included in any one of claims 1-6 according to the obtained program instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a computer, the method according to any one of claims 1 to 6 is executed.

10. A computer program product, characterized in that The computer program product comprises a computer program code, which causes any one of claims 1 to 6 to be performed when the computer program code is run on a computer.

Images