Log anomaly detection method and device, electronic equipment and storage medium

By dividing the log set into subsets and performing density value filtering and clustering algorithms, the problem of high computational complexity in log anomaly detection in cloud platforms is solved, and efficient log anomaly detection is achieved.

CN118940033APending Publication Date: 2024-11-12CHINA MOBILE (SUZHOU) SOFTWARE TECH CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410987862.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

Because cloud platforms contain a large number of logs, existing technologies for anomaly detection of each log result in high computational load and complexity.

Method used

The log set to be detected is divided into at least two subsets, a preset density algorithm is called to calculate the density value of each subset, subsets with density values ​​greater than the threshold are selected, and abnormal logs are detected by clustering algorithm.

Benefits of technology

This reduces the computational complexity of log anomaly detection and improves detection efficiency and accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118940033A_ABST
    Figure CN118940033A_ABST
Patent Text Reader

Abstract

The invention discloses a log exception detection method and device, electronic equipment and a storage medium, and relates to the technical field of data processing, and the main technical scheme comprises the following steps: obtaining a to-be-detected log set, and dividing the to-be-detected log set into at least two to-be-detected log subsets; calling a preset density algorithm, respectively calculating the at least two to-be-detected log subsets, and respectively obtaining a density value of each to-be-detected log subset; determining the to-be-detected log subset corresponding to the density value of which the numerical value is greater than a preset first threshold value in the density values as a target to-be-detected log subset; and scheduling a preset clustering algorithm, and calculating to-be-detected logs in the target to-be-detected log subset to obtain an abnormal log set. Compared with the prior art, the embodiment of the invention realizes that only the to-be-detected log subset with a relatively high density value is subjected to anomaly detection, and the calculation complexity of log anomaly detection is reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a method and device for detecting log anomalies, an electronic device, and a storage medium. Background Art

[0002] When users access the cloud platform, logs are generated. The logs contain not only the users' normal business access, but also abnormal access that attacks the cloud platform. In order to protect the security of the cloud platform, it is necessary to detect log anomalies.

[0003] In the related technologies of log anomaly detection, all logs are usually traversed to detect whether there is an anomaly in each log. Since there are a large number of logs in the cloud platform, anomaly detection for each log will result in a large amount of computational complexity for log anomaly detection, which in turn leads to a high computational complexity for log anomaly detection. Summary of the invention

[0004] The present invention provides a method and device for detecting log anomalies, an electronic device and a storage medium. The main purpose is to solve the problem that due to the presence of a large number of logs in the cloud platform, performing anomaly detection on each log will result in a large amount of calculation for detecting log anomalies, thereby resulting in a high computational complexity for detecting log anomalies.

[0005] According to a first aspect of the present disclosure, a method for detecting log anomalies is provided, which includes:

[0006] Obtain a log set to be detected, and divide the log set to be detected into at least two log subsets to be detected;

[0007] Calling a preset density algorithm to calculate the at least two log subsets to be detected respectively, and obtaining a density value of each log subset to be detected respectively;

[0008] Determine the to-be-detected log subset corresponding to the density value whose value among the density values ​​is greater than the preset first threshold as the target to-be-detected log subset;

[0009] A preset clustering algorithm is scheduled to calculate the logs to be detected in the target log subset to obtain an abnormal log set.

[0010] Optionally, obtaining the log set to be detected includes:

[0011] Get the local density of the log to be detected;

[0012] Determine the log to be detected corresponding to the local density whose value in the local density is greater than the preset second threshold as the target log to be detected;

[0013] The target logs to be detected are combined into the log set to be detected.

[0014] Optionally, dividing the to-be-detected log set into at least two to-be-detected log subsets includes:

[0015] Obtaining the number of types of logs to be detected in the log set to be detected, and obtaining the first total number of logs to be detected;

[0016] Calculating the number of types and the sum of the numbers to obtain the division length of the log to be detected;

[0017] The log to be detected is divided according to the division length to obtain the at least two log subsets to be detected.

[0018] Optionally, calling a preset density algorithm to calculate the at least two log subsets to be detected respectively, and obtaining a density value of each log subset to be detected respectively includes:

[0019] Traversing each of the to-be-detected log subsets respectively to obtain a second total number of to-be-detected logs in each of the to-be-detected log subsets;

[0020] The second sum of quantities is determined as a density value of the log subset to be detected.

[0021] Optionally, the scheduling preset clustering algorithm calculates the logs to be detected in the target log subset to obtain the abnormal log set including:

[0022] Obtain the cluster center weight of the log to be detected in the target log subset to be detected;

[0023] Determine the log to be detected corresponding to the cluster center weight whose value among the cluster center weights is greater than a preset third threshold as the cluster center;

[0024] Perform clustering calculation on the logs to be detected in the target log subset except the cluster center and the cluster center respectively to obtain at least two log sets;

[0025] The log set with the least number of logs to be detected among the at least two log sets is determined as the abnormal log set.

[0026] Optionally, obtaining the cluster center weight of the log to be detected in the target log subset to be detected includes:

[0027] Obtaining the local density of the logs to be detected in the target log subset to be detected;

[0028] Performing norm calculation on two different logs to be detected in the target log subset to be detected respectively to obtain a relative distance between the two different logs to be detected;

[0029] The local terminal density and the corresponding relative distance are calculated to obtain the cluster center weight.

[0030] According to a second aspect of the present disclosure, a device for detecting log anomalies is provided, comprising:

[0031] An acquisition unit, used to acquire a log set to be detected;

[0032] A division unit, used for dividing the log set to be detected into at least two log subsets to be detected;

[0033] A first calculation unit is used to call a preset density algorithm to calculate the at least two log subsets to be detected respectively, and obtain a density value of each log subset to be detected respectively;

[0034] A determination unit, configured to determine the to-be-detected log subset corresponding to the density values ​​whose values ​​among the density values ​​are greater than a preset first threshold as a target to-be-detected log subset;

[0035] The second calculation unit is used to schedule a preset clustering algorithm to calculate the logs to be detected in the target log subset to be detected to obtain an abnormal log set.

[0036] Optionally, the acquiring unit includes:

[0037] The first acquisition module is used to obtain the local density of the log to be detected;

[0038] A first determination module, configured to determine the log to be detected corresponding to the local density whose value in the local density is greater than a preset second threshold as a target log to be detected;

[0039] The combining module is used to combine the target logs to be detected into the log set to be detected.

[0040] Optionally, the division unit includes:

[0041] A second acquisition module is used to acquire the number of types of logs to be detected in the log set to be detected, and to acquire the first total number of logs to be detected;

[0042] A first calculation module, used for calculating the number of types and the sum of the numbers to obtain the division length of the log to be detected;

[0043] The division module is used to divide the log to be detected according to the division length to obtain the at least two log subsets to be detected.

[0044] Optionally, the first computing unit includes:

[0045] A traversal module, used for traversing each of the log subsets to be detected respectively, to obtain a second total number of logs to be detected in each of the log subsets to be detected;

[0046] The second determining module is used to determine the sum of the second quantities as the density value of the log subset to be detected.

[0047] Optionally, the second computing unit includes:

[0048] The third acquisition module is used to obtain the cluster center weight of the log to be detected in the target log subset to be detected;

[0049] A third determination module is used to determine the log to be detected corresponding to the cluster center weight whose value among the cluster center weights is greater than a preset third threshold as the cluster center;

[0050] A second calculation module is used to perform cluster calculation on the logs to be detected in the target log subset except the cluster center and the cluster center to obtain at least two log sets;

[0051] The fourth determination module is used to determine the log set with the least number of logs to be detected among the at least two log sets as the abnormal log set.

[0052] Optionally, the third acquisition module is further used to:

[0053] Obtaining the local density of the logs to be detected in the target log subset to be detected;

[0054] Performing norm calculation on two different logs to be detected in the target log subset to be detected respectively to obtain a relative distance between the two different logs to be detected;

[0055] The local terminal density and the corresponding relative distance are calculated to obtain the cluster center weight.

[0056] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0057] at least one processor; and

[0058] a memory communicatively connected to the at least one processor; wherein,

[0059] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect.

[0060] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in the first aspect.

[0061] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method as described in the first aspect above.

[0062] The log anomaly detection method and device, electronic device and storage medium provided by the present disclosure obtain a log set to be detected, and divide the log set to be detected into at least two log subsets to be detected; call a preset density algorithm, respectively calculate the at least two log subsets to be detected, and respectively obtain the density value of each log subset to be detected; determine the log subset to be detected corresponding to the density value greater than the preset first threshold value in the density value as the target log subset to be detected; schedule a preset clustering algorithm, calculate the logs to be detected in the target log subset to be detected, and obtain the abnormal log set. Compared with the related art, the embodiment of the present disclosure divides the log set to be detected into at least two log subsets to be detected, and calculates the density value of each log subset to be detected, and screens out the target log subset to be detected whose density value is greater than the preset first threshold value, and finally clusters the target log subset to be detected to obtain the daily log set, so as to achieve abnormal detection only for the log subset to be detected with a higher density value, and reduce the computational complexity of log anomaly detection.

[0063] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0065] Figure 1 A flow chart of a method for detecting log anomalies provided by an embodiment of the present disclosure;

[0066] Figure 2 A flowchart of a method for obtaining a log set to be detected provided by an embodiment of the present disclosure;

[0067] Figure 3 A schematic diagram of a log anomaly detection process provided by an embodiment of the present disclosure;

[0068] Figure 4 A schematic diagram of the structure of a log anomaly detection device provided by an embodiment of the present disclosure;

[0069] Figure 5 A schematic diagram of the structure of another device for detecting log anomalies provided by an embodiment of the present disclosure;

[0070] Figure 6 A schematic block diagram of an exemplary electronic device provided for an embodiment of the present disclosure. DETAILED DESCRIPTION

[0071] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0072] The following describes a method and apparatus for detecting anomalies in logs, an electronic device, and a storage medium according to an embodiment of the present disclosure with reference to the accompanying drawings.

[0073] Figure 1 A flow chart of a method for detecting log anomalies provided by an embodiment of the present disclosure.

[0074] like Figure 1 As shown, the method is applied in a server, and the method comprises the following steps:

[0075] Step 101: obtain a log set to be detected, and divide the log set to be detected into at least two log subsets to be detected.

[0076] The log set to be detected is a set of logs that need to be analyzed for anomaly detection. Logs usually record the activity information of the system, application or device for monitoring and problem diagnosis. The log set to be detected contains at least two logs to be detected. The log subset to be detected is a component of the log set to be detected.

[0077] For ease of understanding, an example is provided. Assume that the log set to be detected contains four logs to be detected, a, b, c, and d. The log set to be detected is divided into two log subsets to be detected, namely A and B, where A contains a and b, and B contains c and d.

[0078] The data source of the log to be detected usually provides an application programming interface (API). The log to be detected can be collected by writing scripts or using tools to call the API. However, it should be clear that this statement is not intended to limit the log to be detected to be obtained only through the API. The log to be detected can also be obtained through other methods.

[0079] According to the number of types of logs to be detected and the total number of logs to be detected in the log set to be detected, the division length of the logs to be detected is calculated, and the log set to be detected is divided by the division length of the logs to be detected to obtain at least two subsets of logs to be detected.

[0080] The calculation formula of the partition length can be realized by formula (1):

[0081]

[0082] Among them, d is the number of types of logs to be detected in the log set to be detected; N is the total number of logs to be detected in the log set to be detected; μ is the preset parameter, h i is the left boundary of the i-th type of log to be detected in the log set to be detected, l i It is the right boundary of the i-th type of log to be detected in the log set to be detected.

[0083] For ease of understanding, an example is provided. Assume that the log set to be detected contains 12 logs to be detected, including 2 types of logs to be detected, namely a and b. The number of type a logs to be detected is 6, the number of type b logs to be detected is 6, the preset parameter is 2, and the division length is 1.

[0084] By dividing the data into smaller subsets, the computational complexity on each subset can be simplified, making the overall processing more efficient.

[0085] Step 102: Call a preset density algorithm to perform calculations on the at least two log subsets to be detected, respectively, to obtain density values ​​of each log subset to be detected.

[0086] The preset density algorithm is an algorithm for calculating the distribution density of the log subset to be detected in the log set to be detected. The density value is the distribution density of the log subset to be detected in the log set to be detected. The higher the density value, the higher the density of the logs to be detected in the log subset to be detected.

[0087] The calculation formula of the preset density algorithm can be implemented by formula (2):

[0088] w i = count(g i )i=1,2,…p(2)

[0089] Where count(·) represents the total number of logs to be detected in the log subset to be detected, g i is the i-th log subset to be detected.

[0090] For ease of understanding, an example is provided. Assume that there are two subsets of logs to be detected in the log set to be detected, namely C and D. The total number of logs to be detected in C is 5, and the total number of logs to be detected in D is 6. Then the density value of C is 5, and the density value of D is 6.

[0091] The main purpose of calling the preset density algorithm to calculate the density value of each log subset to be detected is to identify abnormal or important log areas, reduce the amount of data for subsequent processing, improve the accuracy of analysis, and support adaptive analysis.

[0092] Step 103: determine the to-be-detected log subset corresponding to the density value whose value among the density values ​​is greater than the preset first threshold as the target to-be-detected log subset.

[0093] The preset first threshold value can be any value, for example, 4, but it should be clear that this statement is not intended to limit the preset first threshold value to 4, it can also be other values, and the embodiment of the present disclosure does not limit the specific value of the preset first threshold value.

[0094] For ease of understanding, an example is provided. It is assumed that there are two log subsets to be detected in the log set to be detected, namely E and F, where the density value of E is 3, the total number of logs to be detected in F is 6, and the preset first threshold is 4, then the target log subset to be detected is F.

[0095] Processing and analyzing large amounts of log data requires a lot of computing resources and time. By limiting the analysis scope to a subset of target logs with higher density values, resource utilization can be optimized, unnecessary computing and storage overhead can be reduced, and overall processing efficiency can be improved.

[0096] Step 104 , scheduling a preset clustering algorithm to calculate the logs to be detected in the target log subset to be detected to obtain an abnormal log set.

[0097] The preset clustering algorithm is used to divide the logs to be detected into several groups (clusters), that is, several different log sets, so that the similarity of the logs to be detected in the same log set is high, while the similarity of the logs to be detected between different log sets is low.

[0098] By obtaining the cluster center weights of the logs to be detected in the target log subset to be detected, the logs to be detected corresponding to the cluster center weights whose values ​​in the cluster center weights are greater than the preset third threshold are determined as cluster centers, and similarities are calculated between the logs to be detected in the target log subset except the cluster center and the cluster center respectively to obtain the similarities between the logs to be detected in the log subset except the cluster center and the cluster center, the logs to be detected whose similarities are greater than the preset fourth threshold and the cluster center are put into the same log set, and the log set with the smallest total number of logs to be detected in the log set is the abnormal log set.

[0099] Since the target log subset to be detected has been screened out by the preset density algorithm, the log entries it contains are more likely to contain valuable information or anomalies. Therefore, applying the preset clustering algorithm to the target log subset to perform anomaly detection can further improve the accuracy of detection. The preset clustering algorithm can group the logs to be detected based on the similarities or differences between them, thereby more accurately identifying abnormal logs.

[0100] The log anomaly detection method provided by the present disclosure obtains a log set to be detected, and divides the log set to be detected into at least two log subsets to be detected; calls a preset density algorithm, calculates the at least two log subsets to be detected respectively, and obtains the density value of each log subset to be detected respectively; determines the log subset to be detected corresponding to the density value greater than the preset first threshold value in the density value as the target log subset to be detected; schedules a preset clustering algorithm, calculates the logs to be detected in the target log subset to be detected, and obtains the abnormal log set. Compared with the related art, the embodiment of the present disclosure divides the log set to be detected into at least two log subsets to be detected, and calculates the density value of each log subset to be detected, screens out the target log subset to be detected whose density value is greater than the preset first threshold value, and finally clusters the target log subset to be detected to obtain the daily log set, thereby realizing abnormal detection only for the log subset to be detected with a higher density value, and reducing the computational complexity of log anomaly detection.

[0101] As a refinement of step 101, when executing the acquisition of the log set to be detected, the following methods may be adopted but are not limited to: Figure 2 As shown, Figure 2 A flowchart of a method for obtaining a log set to be detected provided by an embodiment of the present disclosure includes:

[0102] Step 201: Obtain the local density of the log to be detected.

[0103] Local density refers to the density of similarity or correlation between a target log to be detected and other logs to be detected within a preset range, with a target log to be detected as the center in the set of logs to be detected.

[0104] The calculation formula of local density can be realized by formula (3):

[0105]

[0106] Among them, d ij is the distance between two different logs to be detected, N is the total number of logs to be detected, d c are preset parameters.

[0107] d ij The calculation formula can be realized by formula (4):

[0108] d ij =|||u i |-|u j |||2 (4)

[0109] Among them, u i 、u j are different logs to be detected, ||·|| is the norm operation, and |·| is the modulo operation.

[0110] The logs to be detected usually include valid information and invalid information of users accessing the cloud platform. Among them, valid information mainly includes the resource request type of the user's current access to the cloud platform and whether there is dangerous attack information of abnormal IP access, etc., while invalid information does not contain user access resource information, which is of little help to log anomaly detection and may even interfere with the success rate of log data anomaly detection to a certain extent. The local density of limited information is high, and the local density of invalid information is low. By obtaining the local density of the logs to be detected, it is possible to effectively distinguish whether the logs to be detected are valid information or invalid information.

[0111] Step 202: determine the log to be detected corresponding to the local density whose value in the local density is greater than a preset second threshold as the target log to be detected.

[0112] The preset second threshold value can be any value, for example, 5, but it should be clear that this statement is not intended to limit the preset second threshold value to 5, it can also be other values, and the embodiment of the present disclosure does not limit the specific value of the preset second threshold value.

[0113] For ease of understanding, an example is provided. Suppose there are three logs to be detected in the cloud platform, namely c, d, and e, where the local density of c is 6, the local density of d is 7, and the local density of d is 4. The preset second threshold is 5, then the target logs to be detected are c and d.

[0114] Anomaly detection for a large amount of log data requires a lot of computing resources and time. By limiting the scope of anomaly detection to the target logs with high local density, the efficiency of anomaly detection can be significantly improved.

[0115] Step 203: combine the target logs to be detected into the log set to be detected.

[0116] Merge the target log sets to be detected into a single set for unified management and processing. This allows for more efficient data processing during analysis or response, avoiding omissions or confusion.

[0117] As a refinement of step 101, when executing the division of the log set to be detected into at least two log subsets to be detected, it can be implemented in the following manner but not limited to: obtaining the number of types of logs to be detected in the log set to be detected, and obtaining the first total number of logs to be detected; calculating the number of types and the total number to obtain the division length of the log to be detected; dividing the log to be detected according to the division length to obtain the at least two log subsets to be detected; wherein the division length can be implemented by formula (1). Dividing the log set to be detected into multiple log subsets to be detected can enable the log subsets to be detected to be processed in parallel on different computing resources, thereby improving the overall processing speed and efficiency.

[0118] As a refinement of step 102, when executing the calling of the preset density algorithm, calculating the at least two log subsets to be detected respectively, and obtaining the density value of each log subset to be detected respectively, it can be implemented in but not limited to the following way: traversing each log subset to be detected respectively, and obtaining the second sum of the number of logs to be detected in each log subset to be detected; and determining the second sum as the density value of the log subset to be detected.

[0119] Specifically, the implementation process of this embodiment is a textual description of formula (2).

[0120] As a refinement of step 104, when executing the scheduled preset clustering algorithm, calculating the logs to be detected in the target log subset to be detected to obtain the abnormal log set, it can be implemented in but not limited to the following manner: obtaining the cluster center weights of the logs to be detected in the target log subset to be detected; determining the logs to be detected corresponding to the cluster center weights whose values ​​in the cluster center weights are greater than a preset third threshold as the cluster center; clustering the logs to be detected in the target log subset to be detected except the cluster center with the cluster center respectively to obtain at least two log sets; and determining the log set with the least number of logs to be detected in the at least two log sets as the abnormal log set.

[0121] Exemplarily, the cluster center weight is an indicator used to measure the importance of each log to be detected in the target log subset as a potential cluster center (or cluster center). The higher the value of the cluster center weight, the higher the probability that the log to be detected corresponding to the cluster center weight is selected as the cluster center.

[0122] The calculation formula of cluster center weight can be realized by formula (5):

[0123]

[0124] Among them, ρ i is the local density of the log to be detected, δ i It is the maximum value of the distance between the log to be detected and other logs to be detected.

[0125] The calculation formula of the preset third threshold can be implemented by formula (6):

[0126]

[0127] Among them, N nor is the total number of logs to be detected in the target log subset, γ i is the cluster center weight of the log to be detected, and ζ is the preset parameter.

[0128] The cluster center is selected by using the cluster center weight and the preset third threshold, which can adaptively select normal and abnormal cluster center points in the data set and improve the accuracy of anomaly detection for the log to be detected.

[0129] As a refinement of the above embodiment, when executing the method of obtaining the cluster center weight of the log to be detected in the target log subset to be detected, it can be implemented by but not limited to the following methods: obtaining the local density of the log to be detected in the target log subset to be detected; performing norm calculation on two different logs to be detected in the target log subset to obtain the relative distance between the two different logs to be detected; calculating the local end density and the corresponding relative distance to obtain the cluster center weight.

[0130] Specifically, the implementation process of this embodiment is a textual description of formula (4) and formula (5).

[0131] In one possible implementation of the embodiment of the present disclosure, in order to better understand the entire process of detecting log anomalies, as follows: Figure 3 As shown, Figure 3 A schematic diagram of a processing process for detecting anomalies of logs provided by an embodiment of the present disclosure is provided. The logs to be detected in the cloud platform are obtained. By presetting a second threshold, the logs to be detected corresponding to the local density whose value is greater than the preset second threshold in the local density are screened out to be determined as target logs to be detected, and the target logs to be detected are combined into the log set to be detected; the log set to be detected is divided into at least two log subsets to be detected; a preset density algorithm is called to calculate the at least two log subsets to be detected respectively, and the density value of each log subset to be detected is obtained respectively, and the log subset to be detected corresponding to the density value whose value is greater than the preset first threshold in the density value is determined as the target log subset to be detected; the cluster center weights of the logs to be detected in the target log subset to be detected are calculated, and the logs to be detected corresponding to the cluster center weight whose value is greater than the preset third threshold in the cluster center weight are determined as cluster centers; the logs to be detected in the target log subset to be detected except the cluster center are clustered with the cluster center to obtain at least two log sets; the log set with the least number of logs to be detected in the at least two log sets is determined as the abnormal log set.

[0132] In summary, the embodiments of the present disclosure can achieve the following effects:

[0133] The present invention divides the log set to be detected into at least two log subsets to be detected, calculates the density value of each log subset to be detected, screens out the target log subset to be detected whose density value is greater than a preset first threshold, and finally clusters the target log subset to be detected to obtain the daily log set, thereby realizing anomaly detection only for the log subset to be detected with a higher density value, thereby reducing the computational complexity of log anomaly detection.

[0134] Corresponding to the above-mentioned log anomaly detection method, the present invention also provides a log anomaly detection device. Since the device embodiment of the present invention corresponds to the above-mentioned method embodiment, the details not disclosed in the device embodiment can be referred to the above-mentioned method embodiment, and will not be repeated in the present invention.

[0135] Figure 4 A schematic diagram of a log anomaly detection device provided by an embodiment of the present disclosure is shown in FIG. Figure 4 As shown, the device is applied to a server and includes:

[0136] An acquisition unit 31 is used to acquire a log set to be detected;

[0137] A division unit 32, configured to divide the log set to be detected into at least two log subsets to be detected;

[0138] The first calculation unit 33 is used to call a preset density algorithm to calculate the at least two log subsets to be detected respectively, and obtain a density value of each log subset to be detected;

[0139] A determination unit 34, configured to determine the to-be-detected log subset corresponding to the density value whose value among the density values ​​is greater than a preset first threshold as a target to-be-detected log subset;

[0140] The second calculation unit 35 is used to schedule a preset clustering algorithm to calculate the logs to be detected in the target log subset to be detected to obtain an abnormal log set.

[0141] The log anomaly detection device provided by the present disclosure obtains a log set to be detected, and divides the log set to be detected into at least two log subsets to be detected; calls a preset density algorithm, calculates the at least two log subsets to be detected respectively, and obtains the density value of each log subset to be detected respectively; determines the log subset to be detected corresponding to the density value greater than the preset first threshold value in the density value as the target log subset to be detected; schedules a preset clustering algorithm, calculates the logs to be detected in the target log subset to be detected, and obtains the abnormal log set. Compared with the related art, the embodiment of the present disclosure divides the log set to be detected into at least two log subsets to be detected, and calculates the density value of each log subset to be detected, screens out the target log subset to be detected whose density value is greater than the preset first threshold value, and finally clusters the target log subset to be detected to obtain the daily log set, thereby realizing abnormal detection only for the log subset to be detected with a higher density value, and reducing the computational complexity of log anomaly detection.

[0142] Furthermore, in a possible implementation of this embodiment, as Figure 5 As shown, the acquisition unit 31 includes:

[0143] A first acquisition module 311 is used to acquire the local density of the log to be detected;

[0144] A first determination module 312 is used to determine the log to be detected corresponding to the local density whose value in the local density is greater than a preset second threshold as a target log to be detected;

[0145] The combining module 313 is used to combine the target logs to be detected into the log set to be detected.

[0146] Furthermore, in a possible implementation of this embodiment, as Figure 5 As shown, the division unit 32 includes:

[0147] The second acquisition module 321 is used to acquire the number of types of logs to be detected in the log set to be detected, and acquire the first total number of logs to be detected;

[0148] A first calculation module 322 is used to calculate the number of types and the total number to obtain the division length of the log to be detected;

[0149] The division module 323 is used to divide the log to be detected according to the division length to obtain the at least two log subsets to be detected.

[0150] Furthermore, in a possible implementation of this embodiment, as Figure 5 As shown, the first calculation unit 33 includes:

[0151] A traversal module 331 is used to traverse each of the log subsets to be detected respectively to obtain a second total number of logs to be detected in each of the log subsets to be detected;

[0152] The second determination module 332 is configured to determine the second quantity sum as the density value of the log subset to be detected.

[0153] Furthermore, in a possible implementation of this embodiment, as Figure 5 As shown, the second calculation unit 35 includes:

[0154] The third acquisition module 351 is used to obtain the cluster center weight of the log to be detected in the target log subset to be detected;

[0155] A third determination module 352 is used to determine the log to be detected corresponding to the cluster center weight whose value among the cluster center weights is greater than a preset third threshold as the cluster center;

[0156] A second calculation module 353 is used to perform cluster calculation on the logs to be detected in the target log subset except the cluster center and the cluster center to obtain at least two log sets;

[0157] The fourth determination module 354 is configured to determine the log set with the least number of logs to be detected among the at least two log sets as the abnormal log set.

[0158] Furthermore, in a possible implementation of this embodiment, as Figure 5 As shown, the third acquisition module 351 is also used for:

[0159] Obtaining the local density of the logs to be detected in the target log subset to be detected;

[0160] Performing norm calculation on two different logs to be detected in the target log subset to be detected respectively to obtain a relative distance between the two different logs to be detected;

[0161] The local terminal density and the corresponding relative distance are calculated to obtain the cluster center weight.

[0162] It should be noted that the above explanation of the method embodiment is also applicable to the device of the embodiment of the present disclosure, and the principle is the same, which is no longer limited in the embodiment of the present disclosure.

[0163] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0164] Figure 6 A schematic block diagram of an example electronic device 400 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0165] like Figure 6As shown, the device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a ROM (Read-Only Memory) 402 or a computer program loaded from a storage unit 408 to a RAM (Random Access Memory) 403. In the RAM 403, various programs and data required for the operation of the device 400 can also be stored. The computing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An I / O (Input / Output) interface 405 is also connected to the bus 404.

[0166] A number of components in the device 400 are connected to the I / O interface 405, including: an input unit 406, such as a keyboard, a mouse, etc.; an output unit 407, such as various types of displays, speakers, etc.; a storage unit 408, such as a disk, an optical disk, etc.; and a communication unit 409, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 409 allows the device 400 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0167] The computing unit 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Units), various dedicated AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, a DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, etc. The computing unit 401 performs the various methods and processes described above, such as a method for detecting log anomalies. For example, in some embodiments, the method for detecting log anomalies may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 400 via the ROM 402 and / or the communication unit 409. When the computer program is loaded into the RAM 403 and executed by the computing unit 401, one or more steps of the method described above may be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to execute the aforementioned log anomaly detection method in any other appropriate manner (for example, by means of firmware).

[0168] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application Specific Standard Products), SOCs (System On Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that may be executed and / or interpreted on a programmable system including at least one programmable processor that may be a dedicated or general-purpose programmable processor that may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0169] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0170] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a RAM, a ROM, an EPROM (Electrically Programmable Read-Only-Memory) or a flash memory, an optical fiber, a CD-ROM (Compact Dis sc Read-Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0171] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0172] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: LAN (Local Area Network), WAN (Wide Area Network), the Internet, and blockchain networks.

[0173] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0174] It should be noted that artificial intelligence is a discipline that studies how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.), and includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0175] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0176] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for detecting log anomalies, characterized in that: include: Obtain a log set to be detected, and divide the log set to be detected into at least two log subsets to be detected; Calling a preset density algorithm to calculate the at least two log subsets to be detected respectively, and obtaining a density value of each log subset to be detected respectively; Determine the to-be-detected log subset corresponding to the density value whose value among the density values ​​is greater than the preset first threshold as the target to-be-detected log subset; A preset clustering algorithm is scheduled to calculate the logs to be detected in the target log subset to obtain an abnormal log set.

2. The method according to claim 1, characterized in that: The obtaining of the log set to be detected includes: Get the local density of the log to be detected; Determine the log to be detected corresponding to the local density whose value in the local density is greater than the preset second threshold as the target log to be detected; The target logs to be detected are combined into the log set to be detected.

3. The method according to claim 1, characterized in that: The dividing the log set to be detected into at least two log subsets to be detected comprises: Obtaining the number of types of logs to be detected in the log set to be detected, and obtaining the first total number of logs to be detected; Calculating the number of types and the sum of the numbers to obtain the division length of the log to be detected; The log to be detected is divided according to the division length to obtain the at least two log subsets to be detected.

4. The method according to claim 1, characterized in that: The calling of the preset density algorithm to calculate the at least two log subsets to be detected respectively, and obtaining the density value of each log subset to be detected respectively includes: Traversing each of the to-be-detected log subsets respectively to obtain a second total number of to-be-detected logs in each of the to-be-detected log subsets; The second sum of quantities is determined as a density value of the log subset to be detected.

5. The method according to claim 1, characterized in that: The scheduling preset clustering algorithm calculates the logs to be detected in the target log subset to obtain the abnormal log set including: Obtain the cluster center weight of the log to be detected in the target log subset to be detected; Determine the log to be detected corresponding to the cluster center weight whose value among the cluster center weights is greater than a preset third threshold as the cluster center; Perform clustering calculation on the logs to be detected in the target log subset except the cluster center and the cluster center respectively to obtain at least two log sets; The log set with the least number of logs to be detected among the at least two log sets is determined as the abnormal log set.

6. The method according to claim 5, characterized in that The step of obtaining the cluster center weight of the logs to be detected in the target log subset to be detected includes: Obtaining the local density of the logs to be detected in the target log subset to be detected; Performing norm calculation on two different logs to be detected in the target log subset to be detected respectively to obtain a relative distance between the two different logs to be detected; The local terminal density and the corresponding relative distance are calculated to obtain the cluster center weight.

7. A device for detecting log anomalies, characterized in that: include: An acquisition unit, used to acquire a log set to be detected; A division unit, used for dividing the log set to be detected into at least two log subsets to be detected; A first calculation unit is used to call a preset density algorithm to calculate the at least two log subsets to be detected respectively, and obtain a density value of each log subset to be detected respectively; A determination unit, configured to determine the to-be-detected log subset corresponding to the density values ​​whose values ​​among the density values ​​are greater than a preset first threshold as a target to-be-detected log subset; The second calculation unit is used to schedule a preset clustering algorithm to calculate the logs to be detected in the target log subset to be detected to obtain an abnormal log set.

8. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.

10. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 6.