Log detection method, device, and computer-readable storage medium

Abstract

The present application provides a log detection method, device, and computer-readable storage medium, relating to the field of data detection, capable of determining an abnormal time window including abnormal logs from multiple time windows. The method comprises: obtaining multiple logs to be detected within a preset time period including multiple time windows; determining a matrix to be detected based on multiple preset log templates and multiple logs to be detected; an element in the matrix to be detected represents the number of logs to be detected corresponding to a preset log template within a time window; determining multiple degrees of difference between frequency vectors of multiple time windows in the matrix to be detected and the preset matrix; a frequency vector represents the number of logs to be detected corresponding to multiple preset log templates within a time window, and an element in the preset matrix represents the number of normal logs corresponding to a preset log template within a time window; and determining an abnormal time window including abnormal logs from the multiple time windows based on the multiple degrees of difference.

Description

Technical Field

[0001] The present application relates to the field of data detection, and in particular to a log detection method, device, and computer-readable storage medium. Background Art

[0002] During operation, network systems regularly generate logs to record detailed information about the system's operation. Logs generated during normal network operation are called normal logs, while logs generated during abnormal network operation are called abnormal logs. By analyzing abnormal logs, network system failures can be diagnosed.

[0003] Before analyzing the abnormal logs, it is necessary to determine an abnormal time window including the abnormal logs from multiple time windows, so as to analyze the abnormal logs in the window. Summary of the Invention

[0004] The present application provides a log detection method, device, and computer-readable storage medium, which can determine an abnormal time window including abnormal logs from multiple time windows.

[0005] To achieve the above objectives, this application adopts the following technical solutions:

[0006] In a first aspect, a log detection method is provided, which can be executed by a log detection device, and includes: obtaining multiple logs to be detected within a preset time period; the preset time period includes multiple time windows; determining a matrix to be detected based on multiple preset log templates and multiple logs to be detected; an element in the matrix to be detected represents the number of logs to be detected corresponding to a preset log template in a time window; determining the difference between a frequency vector of each time window in the matrix to be detected and the preset matrix respectively, to obtain multiple difference degrees; a frequency vector is used to represent the number of logs to be detected corresponding to multiple preset log templates in a time window, and an element in the preset matrix represents the number of normal logs corresponding to a preset log template in a time window; and determining an abnormal time window including abnormal logs from the multiple time windows based on the multiple difference degrees.

[0007] Based on this scheme, the matrix to be detected is determined according to multiple preset log templates and multiple logs to be detected. Since the log corresponding to each element in the preset matrix is ​​a normal log, by comparing the difference between the frequency vector of each time window in the matrix to be detected and the preset matrix including normal logs, the abnormal time window including abnormal logs in the matrix to be detected can be determined.

[0008] In combination with the first aspect, in certain embodiments of the first aspect, based on multiple preset log templates and multiple logs to be detected, a matrix to be detected is determined, including: determining the number of logs to be detected corresponding to each preset log template in each time window to obtain an initial matrix to be detected; multiplying the target element in the initial matrix to be detected by a preset coefficient to obtain the matrix to be detected, where the target element corresponds to an abnormal log template in the multiple preset log templates.

[0009] Based on this solution, multiplying the target element corresponding to the abnormal log template in the initial matrix to be detected by a preset coefficient can increase the difference between the abnormal time window including the abnormal log and the preset matrix, making it easier to determine the abnormal time window from multiple time windows.

[0010] In combination with the first aspect, in certain embodiments of the first aspect, the log detection method also includes: obtaining multiple normal logs within a preset time period; based on multiple preset log templates and multiple normal logs, determining the number of normal logs corresponding to each preset log template in each time window to obtain a preset matrix.

[0011] Based on this solution, the preset matrix can be determined by determining the number of normal logs corresponding to each preset log template in each time window.

[0012] In combination with the first aspect, in certain embodiments of the first aspect, the degree of difference is the Mahalanobis distance, and determining the abnormal time window including the abnormal log from multiple time windows based on multiple degrees of difference includes: determining a normal distribution graph of multiple Mahalanobis distances; determining the standard deviation interval of the normal distribution graph; and determining the time window corresponding to the target Mahalanobis distance that is not in the standard deviation interval as including the abnormal time window.

[0013] Based on this scheme, when the degree of difference is the Mahalanobis distance, by determining the standard deviation interval of the normal distribution graph of multiple Mahalanobis distances, the target Mahalanobis distance that is not in the standard deviation interval is determined from the multiple Mahalanobis distances, and the time window corresponding to the target Mahalanobis distance can be determined to include the abnormal time window. At the same time, since the Mahalanobis distance can eliminate mutual interference between elements, the accuracy of log detection can be improved.

[0014] In a second aspect, a log detection device is provided for implementing the log detection method of the first aspect. The log detection device includes modules, units, or means corresponding to the aforementioned method. The modules, units, or means may be implemented in hardware, software, or by hardware executing corresponding software implementations. The hardware or software includes one or more modules or units corresponding to the aforementioned functions.

[0015] In combination with the second aspect, in certain embodiments of the second aspect, the log detection device includes: an acquisition module and a processing module; the acquisition module is used to acquire multiple logs to be detected within a preset time period; the preset time period includes multiple time windows; the processing module is used to determine the matrix to be detected based on multiple preset log templates and multiple logs to be detected; an element in the matrix to be detected represents the number of logs to be detected corresponding to a preset log template in a time window; the processing module is also used to respectively determine the difference between the frequency vector of each time window in the matrix to be detected and the preset matrix to obtain multiple difference degrees; a frequency vector is used to represent the number of logs to be detected corresponding to multiple preset log templates in a time window, and an element in the preset matrix represents the number of normal logs corresponding to a preset log template in a time window; the processing module is also used to determine the abnormal time window including abnormal logs from multiple time windows based on multiple difference degrees.

[0016] In combination with the second aspect, in certain embodiments of the second aspect, the processing module is used to determine the matrix to be detected based on multiple preset log templates and multiple logs to be detected, including: determining the number of logs to be detected corresponding to each preset log template in each time window to obtain an initial matrix to be detected; multiplying the target element in the initial matrix to be detected by a preset coefficient to obtain the matrix to be detected, where the target element corresponds to the abnormal log template in the multiple preset log templates.

[0017] In combination with the second aspect, in certain embodiments of the second aspect, the processing module is also used to: obtain multiple normal logs within a preset time period; based on multiple preset log templates and multiple normal logs, determine the number of normal logs corresponding to each preset log template in each time window to obtain a preset matrix.

[0018] In combination with the second aspect, in certain embodiments of the second aspect, the degree of difference is the Mahalanobis distance, and the processing module is further used to determine the abnormal time window including the abnormal log from multiple time windows based on multiple degrees of difference, including: determining a normal distribution graph of multiple Mahalanobis distances; determining the standard deviation interval of the normal distribution graph; and determining the time window corresponding to the target Mahalanobis distance that is not in the standard deviation interval as including the abnormal time window.

[0019] In a third aspect, a log detection device is provided, comprising: at least one processor and a memory for storing instructions executable by the processor; wherein the processor is configured to execute instructions to implement the log detection method provided in the first aspect and any possible design thereof.

[0020] In a fourth aspect, a computer-readable storage medium is provided. When the instructions in the computer-readable storage medium are executed by the processor of the log detection device, the log detection device is enabled to perform the log detection method provided in the first aspect and any possible design thereof.

[0021] In a fifth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute the method of the first aspect.

[0022] In the sixth aspect, a chip system is provided, comprising: a processor and an interface circuit; the interface circuit is used to receive a computer program or instruction and transmit it to the processor; the processor is used to execute the computer program or instruction so that the chip system executes the method described in the first aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A schematic diagram of the architecture of a log detection system provided in this application;

[0024] Figure 2 A flow chart of a log detection method provided in this application;

[0025] Figure 3 A schematic diagram of a process for determining a matrix to be detected provided by this application;

[0026] Figure 4a A schematic diagram of a process for determining an abnormal time window provided by this application;

[0027] Figure 4b An example of a normal distribution graph of multiple Mahalanobis distances provided for this application;

[0028] Figure 4c An example diagram of the standard deviation interval in a normal distribution diagram provided in this application;

[0029] Figure 5 This is an example diagram of a process for determining a preset matrix provided by this application;

[0030] Figure 6 A schematic diagram of the structure of a log detection device provided by this application;

[0031] Figure 7 This is a structural diagram of another log detection device provided by this application. DETAILED DESCRIPTION

[0032] In order to facilitate understanding of the technical solutions of the embodiments of the present application, a brief introduction to the relevant terms of the present application is first given as follows.

[0033] 1. Mahalanobis distance. The Mahalanobis distance, proposed by Indian statistician Mahalanobis, represents the distance between a point and a distribution. It is an effective method for calculating the difference between two unknown sample sets. The Mahalanobis distance between two points is independent of the measurement unit of the original data; the Mahalanobis distance calculated using standardized data and centered data (i.e., the difference between the original data and the mean) is the same. The Mahalanobis distance also eliminates interference from correlations between elements.

[0034] 2. Laida Criterion. The Laida Criterion assumes that a set of test data contains only random errors. The standard deviation range is calculated and processed. Any error exceeding the standard deviation range is considered not random but gross error. This criterion can be used for sample data with a normal or nearly normal distribution. In a normal distribution, σ represents the standard deviation, and μ represents the mean. The 3σ principle states that the probability of a value being distributed in the range (μ - σ, μ + σ) is 0.6826, the probability of a value being distributed in the range (μ - 2σ, μ + 2σ) is 0.9544, and the probability of a value being distributed in the range (μ - 3σ, μ + 3σ) is 0.9974. This suggests that the data values ​​are almost entirely concentrated within the range (μ - 3σ, μ + 3σ), with less than 0.3% of the data falling outside this range.

[0035] In the description of this application, unless otherwise specified, "plurality" means two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0036] In addition, to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. Those skilled in the art will understand that the words "first" and "second" do not limit the quantity or execution order, and the words "first" and "second" do not necessarily mean different.

[0037] At the same time, in the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner to facilitate understanding.

[0038] It will be understood that the “embodiment” mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, the various embodiments in the entire specification do not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It will be understood that in the various embodiments of the present application, the size of the sequence number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0039] It can be understood that in this application, "when", "if" and "if" all mean that corresponding processing will be taken under certain objective circumstances, and do not limit the time, nor do they require judgment actions when implementing them, nor do they mean that there are other limitations.

[0040] It is understood that some optional features in the embodiments of the present application may, in certain scenarios, be implemented independently of other features, such as the solution on which they are currently based, to solve corresponding technical problems and achieve corresponding effects. They may also be combined with other features in certain scenarios as needed. Accordingly, the devices provided in the embodiments of the present application may also implement these features or functions accordingly, which will not be described in detail here.

[0041] In this application, unless otherwise specified, the same or similar parts between the various embodiments can refer to each other. In the various embodiments of this application, and the various implementation methods in each embodiment, if there is no special explanation and logical conflict, the terms and / or descriptions between different embodiments and the various implementation methods in each embodiment are consistent and can be referenced to each other. The technical features in different embodiments and the various implementation methods in each embodiment can be combined to form new embodiments, implementation methods, implementation methods, or implementation methods according to their inherent logical relationships. The following implementation methods of this application do not constitute a limitation on the scope of protection of this application.

[0042] Figure 1 This is a schematic diagram of the architecture of a log detection system provided by this application. The technical solution of the embodiment of this application can be applied to Figure 1 The log detection system shown in Figure 1 As shown, the log detection system 10 includes a log detection device 11 and a log sending device 12 .

[0043] The log detection device 11 is directly or indirectly connected to the log sending device 12 . In this connection relationship, a wired connection or a wireless connection may be adopted, which is not limited in the embodiment of the present application.

[0044] The log detection device 11 can be used to receive and detect the log to be detected from the log sending device 12 .

[0045] The log sending device 12 can be used to send the log to be detected or the normal log to the log detecting device 11 .

[0046] It should be noted that the log detection device 11 and the log sending device 12 can be independent devices or integrated into the same device, and this disclosure does not make any specific limitation on this.

[0047] When the log detection device 11 and the log sending device 12 are integrated into the same device, the communication between the log detection device 11 and the log sending device 12 is carried out between the modules within the device. In this case, the communication process between the two is the same as the communication process between the log detection device 11 and the log sending device 12 when they are independent of each other.

[0048] In the following embodiments provided by the present disclosure, the present disclosure is described by taking an example in which the log detection device 11 and the log sending device 12 are independently configured.

[0049] In practical applications, the log detection method provided in the embodiment of the present application can be applied to the log detection device 11, and can also be applied to the devices included in the log detection device 11.

[0050] The log detection method provided in the embodiment of the present application is described below with reference to the accompanying drawings, taking the application of the log detection method to the log detection device 11 as an example.

[0051] Figure 2 A flow chart of a log detection method provided in this application is as follows: Figure 2 As shown, the method includes the following steps:

[0052] S201: The log detection device obtains a plurality of logs to be detected within a preset time period.

[0053] The preset time period includes multiple time windows.

[0054] As a possible implementation method, the log detection device can be Figure 1 The log sending device shown obtains multiple logs to be detected within a preset time period.

[0055] It should be noted that the preset time period can be one day, or the preset time period can be half a day. Of course, the preset time period can also have other dates or other time lengths, and this application does not limit this. The time window can be 1 hour. In this case, if the preset time period is one day, the preset time period includes 24 time windows, and if the preset time period is half a day, the preset time period includes 12 time windows. Of course, the time window can also have other time lengths, and this application does not limit this.

[0056] The log to be detected includes the time when the log to be detected was generated, which is within a time window. Each time window can contain multiple logs to be detected, and the number of logs to be detected in different time windows is different.

[0057] S202: The log detection device determines a matrix to be detected based on multiple preset log templates and multiple logs to be detected.

[0058] Among them, one element in the matrix to be detected represents the number of logs to be detected corresponding to a preset log template in a time window.

[0059] As a possible implementation method, the log detection device determines the number a1 of logs to be detected corresponding to the first preset log template in the first time window, and uses a1 as the element of the first row and first column of the matrix to be detected, and so on to obtain the matrix to be detected.

[0060] As another possible implementation method, the log detection device determines the number a1 of logs to be detected corresponding to the first preset log template in the first time window, and uses a1 as the element of the first row and first column of the initial matrix to be detected, and so on to obtain the initial matrix to be detected.

[0061] Then, the log detection device multiplies the target element in the initial matrix to be detected by a preset coefficient to obtain the matrix to be detected.

[0062] It should be noted that the specific description of this possible implementation method can be found in the subsequent Figure 3 The relevant description of the flowchart of determining the matrix to be detected is not repeated here.

[0063] It should be noted that the preset log template can be a log template based on log keywords. For example, a log is sysmonitor

[39896] |systemd[1](parent:swapper / 0[0])send SIGTERM to irqbalance

[72173] , and the keyword of the log is send...to..., then the corresponding preset log template includes send*to*. The preset log template can also be a log template based on the network element identifier of the log. For example, a log is sysmonitor

[39896] |systemd[1](parent:swapper / 0[0])send SIGTERM to irqbalance

[72173] , and the network element identifier of the log is sysmonitor..., then the corresponding preset log template includes sysmonitor*. Of course, the preset log template can also be a log template based on other log features, and this application does not limit this.

[0064] It should be noted that the preset log template may also include an identifier of the preset log template. For example, if the preset log template is a log template based on a log keyword, and a log is sysmonitor

[39896] |systemd[1](parent:swapper / 0[0])send SIGTERM to irqbalance

[72173] , then the corresponding preset log template includes T1:send*to*.

[0065] It is understandable that, in the case where the preset log template includes an identifier of the preset log template, the preset log template can be marked with the identifier of the preset log template, which can simplify the description of the preset log template.

[0066] As an example, an example of a representation of a matrix to be detected provided in this application is as follows:

[0067]

[0068] In this example, an element in the pending detection matrix represents the number of pending logs corresponding to a preset log template within a time window. For example, the element in the first row and first column indicates that the number of pending logs corresponding to the first preset log template in the first time window is a1. The element in the second row and second column indicates that the number of pending logs corresponding to the second preset log template in the second time window is e1. And so on.

[0069] It should be noted that each row in the matrix to be detected shown in this example can represent the number of logs to be detected corresponding to a preset log template in multiple time windows. In this case, each column in the matrix to be detected represents the number of logs to be detected corresponding to multiple preset log templates in a time window.

[0070] Each row in the matrix to be detected shown in this example can also represent the number of logs to be detected corresponding to multiple preset log templates in a time window. In this case, each column in the matrix to be detected represents the number of logs to be detected corresponding to a preset log template in multiple time windows.

[0071] S203 : The log detection apparatus determines the difference between the frequency vector of each time window in the matrix to be detected and the preset matrix respectively, and obtains a plurality of difference degrees.

[0072] Among them, a frequency vector is used to represent the number of logs to be detected corresponding to multiple preset log templates in a time window, and an element in the preset matrix represents the number of normal logs corresponding to a preset log template in a time window.

[0073] In one possible design, elements in a preset matrix correspond to elements in a matrix to be detected. The correspondence between the elements in the preset matrix and the elements in the matrix to be detected can be understood as follows: the preset log template corresponding to an element in the preset matrix is ​​the same as the preset log template corresponding to the element in the matrix to be detected, and the time window corresponding to an element in the preset matrix is ​​the same as the time window corresponding to the element in the matrix to be detected.

[0074] For example, if element 1 in the preset matrix represents the number of normal logs corresponding to the preset log template 1 in the time window 1, then element 2 corresponding to element 1 in the matrix to be detected represents the number of logs to be detected corresponding to the preset log template 1 in the time window 1. The matrix to be tested is For example, a2 corresponds to a1, b2 corresponds to b1, c2 corresponds to c1, and so on. Where a2 represents the number of normal logs corresponding to preset log template 1 in time window 1, and a1 represents the number of logs to be detected corresponding to preset log template 1 in time window 1; b2 represents the number of normal logs corresponding to preset log template 2 in time window 2, and b1 represents the number of logs to be detected corresponding to preset log template 2 in time window 2; c2 represents the number of normal logs corresponding to preset log template 3 in time window 3, and c1 represents the number of logs to be detected corresponding to preset log template 3 in time window 3.

[0075] As an example, taking the representation of the matrix to be detected shown in S202 as an example, if each row in the matrix to be detected represents the number of logs to be detected corresponding to a preset log template in multiple time windows, in this case, the frequency vector of the first time window in the matrix to be detected is as follows:

[0076]

[0077] As another example, taking the representation of the matrix to be detected shown in S202 as an example, if each row in the matrix to be detected represents the number of logs to be detected corresponding to multiple preset log templates in a time window, in this case, the frequency vector of the first time window in the matrix to be detected is as follows:

[0078] [a1 b1 c1]

[0079] It should be noted that the difference can be the Mahalanobis distance, or the difference can also be the chi-square value. Of course, the difference can also be other data that can represent the difference, and this application does not limit this.

[0080] As one possible implementation, when the difference is the Mahalanobis distance, the log detection device first determines the average number of normal logs corresponding to each preset log template in the preset matrix over multiple time windows to obtain an average matrix. It then determines the covariance between the frequency vectors of any two preset log templates in the preset matrix to obtain a covariance matrix. For the frequency vector of a time window in the matrix to be detected, the log detection device calculates the Mahalanobis distance corresponding to the frequency vector of that time window based on the frequency vector of that time window, the average matrix, and the covariance matrix.

[0081] As an example, when the difference is the Mahalanobis distance, the preset matrix is The matrix to be tested is For example, each row in the preset matrix represents the number of normal logs corresponding to a preset log template in multiple time windows, and each row in the to-be-detected matrix represents the number of to-be-detected logs corresponding to a preset log template in multiple time windows.

[0082] First, the log detection device determines the average value of the number of normal logs corresponding to each preset log template in a preset matrix in multiple time windows, and obtains the average value m matrix. The m matrix is

[0083] Secondly, the log detection device determines the covariance between the frequency vectors of two preset log templates in the preset matrix, and obtains the covariance ∑ matrix, which is Where A is the covariance between [a2 b2 c2] and [a2 b2 c2], B is the covariance between [a2 b2 c2] and [d2 e2 f2], F is the covariance between [d2 e2 f2] and [a2 b2 c2], G is the covariance between [d2 e2 f2] and [d2 e2 f2], and so on.

[0084] Finally, for the frequency vector y of the i-th time window in the matrix to be detected i , the log detection device determines y according to the following formula 1 i The Mahalanobis distance Di from the preset matrix.

[0085]

[0086] Among them, (y i -m) T The matrix is ​​(y i -m) matrix transpose, Σ -1 The matrix is ​​the inverse of the ∑ matrix.

[0087] It should be noted that, since i The Mahalanobis distance Di of the preset matrix involves the ∑ matrix of the preset matrix -1 The necessary and sufficient condition for a matrix to be reversible is that the matrix is ​​a full-rank matrix. Therefore, if the ∑ matrix is ​​not a full-rank matrix, the log detection device can process the preset matrix based on the principal component analysis (PCA) algorithm. The following is a detailed explanation:

[0088] First, the log detection device determines the average number of normal logs corresponding to each preset log template in a preset matrix in multiple time windows, and subtracts the corresponding average value from each element in the preset matrix to obtain a standardized preset matrix.

[0089] Secondly, the log detection device determines the covariance matrix of the standardized preset matrix, performs eigendecomposition on the covariance matrix, and obtains the eigenvalues ​​of the covariance matrix and the eigenvectors corresponding to the eigenvalues.

[0090] Secondly, the log detection device sorts the eigenvalues ​​from large to small, selects the first k eigenvalues ​​that are not 0, and records λ1≥λ2≥…≥λ k , and the corresponding standard orthogonal eigenvectors are denoted as p1, p2…p k .

[0091] Secondly, the log detection device projects the standardized preset matrix onto the matrix [p1 p2… p k ], and obtain the preset matrix after dimensionality reduction.

[0092] Finally, the log detection device also performs the above PCA dimensionality reduction step on the matrix to be detected, and the number of eigenvalues ​​k is the same as the number of eigenvalues ​​k when the preset matrix is ​​subjected to PCA.

[0093] As another possible implementation method, when the difference is a chi-square value, for the frequency vector of a time window in the matrix to be detected, the log detection device first determines the chi-square value of each element in the frequency vector of the time window, and then accumulates the chi-square values ​​of multiple elements in the frequency vector of the time window to obtain the chi-square value of the frequency vector of the time window.

[0094] S204: The log detection apparatus determines an abnormal time window including abnormal logs from multiple time windows according to multiple difference degrees.

[0095] It should be noted that if a time window is determined to be an abnormal time window, it means that there are abnormal logs in the time window.

[0096] As a possible implementation, the log detection device may determine the standard deviation interval of a normal distribution graph of multiple differences, and determine the time window corresponding to the target difference that is not within the standard deviation interval as an abnormal time window. For example, the difference may be a Mahalanobis distance, and the log detection device may determine the standard deviation interval of a normal distribution graph of multiple Mahalanobis distances based on the Laida criterion, and determine the time window corresponding to the target Mahalanobis distance that is not within the standard deviation interval as an abnormal time window. For another example, the difference may be a chi-square value, and the log detection device may determine the standard deviation interval of a normal distribution graph of multiple chi-square values, and determine the time window corresponding to the target chi-square value that is not within the standard deviation interval as an abnormal time window.

[0097] It should be noted that for the case where the difference is the Mahalanobis distance, reference can be made to the relevant description of the flowchart for determining the abnormal time window shown in the subsequent FIG4, which will not be repeated here. For the case where the difference is the square chi-square value, reference can be made to the explanation of the method shown in FIG4 when the difference is the Mahalanobis distance, which will not be repeated in this embodiment of the application.

[0098] Based on this scheme, the matrix to be detected is determined according to multiple preset log templates and multiple logs to be detected. Since the log corresponding to each element in the preset matrix is ​​a normal log, by determining the difference between the frequency vector of each time window in the matrix to be detected and the preset matrix including normal logs, the abnormal time window including abnormal logs in the matrix to be detected can be determined.

[0099] The above is an overall description of the solution of this application. The solution provided by this application will be further described below.

[0100] In one design, Figure 3 A schematic diagram of a process for determining a matrix to be detected provided by this application is as follows: Figure 3 As shown, S202 provided in the embodiment of the present application specifically includes:

[0101] S301 : The log detection apparatus determines the number of logs to be detected corresponding to each preset log template in each time window, and obtains an initial matrix to be detected.

[0102] As an example, an example of an initial matrix to be detected provided in this application is shown below:

[0103]

[0104] In this example, each element in the initial pending detection matrix represents the number of pending logs corresponding to a preset log template within a time window. For example, the element in the first row and first column indicates that the number of pending logs corresponding to the first preset log template in the first time window is a1. The element in the second row and second column indicates that the number of pending logs corresponding to the second preset log template in the second time window is e1. And so on.

[0105] It should be noted that each row in the initial matrix to be detected shown in this example can represent the number of logs to be detected corresponding to a preset log template in multiple time windows. In this case, each column in the initial matrix to be detected represents the number of logs to be detected corresponding to multiple preset log templates in a time window. Each row in the initial matrix to be detected shown in this example can also represent the number of logs to be detected corresponding to multiple preset log templates in a time window. In this case, each column in the initial matrix to be detected represents the number of logs to be detected corresponding to a preset log template in multiple time windows.

[0106] S302: The log detection device multiplies the target element in the initial matrix to be detected by a preset coefficient to obtain the matrix to be detected.

[0107] The target element corresponds to an abnormal log template among multiple preset log templates.

[0108] It should be noted that the preset coefficient can be any real number greater than or equal to 1. For example, the preset coefficient can be 2, or the preset coefficient can be 3. Of course, the preset coefficient can also be other values, and this is not a limitation. It is understandable that in the embodiment of the present application, the log detection device can also not execute S302, but directly determine the initial matrix to be detected as the matrix to be detected.

[0109] As an example, taking the preset coefficient as 2, the initial matrix to be detected as the initial matrix to be detected in S301 above, and each row in the initial matrix to be detected representing the number of logs to be detected corresponding to a preset log template in multiple time windows, the elements in the last row of the initial matrix to be detected are defined as target elements, and the log detection device multiplies the elements in the last row of the initial matrix to be detected by 2. The resulting matrix to be detected can be shown as follows:

[0110]

[0111] It should be noted that, when a preset log template includes an identifier of the preset log template, the representation of the abnormal log template in multiple preset log templates is different from the identifier of the non-abnormal log template. For example, if the identifier of the non-abnormal log template is T*, the identifier of the abnormal log template may be E*. Of course, the identifier of the abnormal log template may also have other representation forms, and this application does not impose any restrictions on this.

[0112] Based on this solution, multiplying the target element corresponding to the abnormal log template in the initial matrix to be detected by a preset coefficient can increase the difference between the abnormal time window including the abnormal log and the preset matrix, making it easier to determine the abnormal time window from multiple time windows.

[0113] In one design, when the difference is the Mahalanobis distance, in order to determine the abnormal time window including the abnormal log from multiple time windows, Figure 4a A flow chart of determining an abnormal time window provided by this application is as follows: Figure 4a As shown, S204 provided in the embodiment of the present application specifically includes:

[0114] S401: The log detection device determines a normal distribution graph of multiple Mahalanobis distances.

[0115] As an example, Figure 4b An example of a normal distribution diagram of multiple Mahalanobis distances provided in this application is as follows: Figure 4b As shown, multiple Mahalanobis distances conform to the normal distribution.

[0116] S402: The log detection device determines the standard deviation interval of the normal distribution graph based on the Laida criterion.

[0117] As one possible implementation, the log detection device determines the mean μ and standard deviation σ of multiple Mahalanobis distances, and then determines the standard deviation interval as [μ-3σ, μ+3σ] based on the Laida criterion. It is understood that in addition to directly determining the interval determined based on the Laida criterion as the standard deviation interval, it is also possible to adjust an interval determined based on the Laida criterion and determine the adjusted interval as the standard deviation interval, without limitation.

[0118] As an example, taking the normal distribution diagram shown in S401 above as an example, Figure 4c This application provides an example diagram of the standard deviation interval in a normal distribution diagram, such as Figure 4c As shown, the standard deviation interval is [μ-3σ, μ+3σ].

[0119] S403: The log detection apparatus determines the time window corresponding to the target Mahalanobis distance that is not within the standard deviation interval as including the abnormal time window.

[0120] When determining that the standard deviation interval is [μ-3σ, μ+3σ], the log detection device may determine that the time window corresponding to the target Mahalanobis distance that is not in the standard deviation interval includes the abnormal time window.

[0121] Based on this scheme, when the degree of difference is the Mahalanobis distance, by determining the standard deviation interval of the normal distribution graph of multiple Mahalanobis distances, the target Mahalanobis distance that is not in the standard deviation interval is determined from the multiple Mahalanobis distances, and the time window corresponding to the target Mahalanobis distance can be determined to include the abnormal time window. At the same time, since the Mahalanobis distance can eliminate mutual interference between elements, the accuracy of log detection can be improved.

[0122] In one design, in order to determine the preset matrix, Figure 5 This is an example diagram of a process for determining a preset matrix provided by this application, such as Figure 5 As shown, the method provided by this application also includes the following steps:

[0123] S501: The log detection device obtains multiple normal logs within a preset time period.

[0124] It should be noted that the length of the preset time period in S501 is the same as the length of the preset time in S201 above. For example, if the preset time period in S201 above is one day, the preset time period in S501 is also one day.

[0125] However, the start time and end time of the preset time period in the above S201 may be different from the preset time period in S501. For example, taking the preset time period as one day as an example, the start time of the preset time period in the above S201 may be 00:00 on January 1, 2021, and the end time may be 24:00 on January 1, 2021. The start time of the preset time period in S501 may be 00:00 on January 2, 2021, and the end time may be 24:00 on January 2, 2021.

[0126] As a possible implementation method, the log detection device can be Figure 1The log sending device shown obtains multiple normal logs within a preset time period.

[0127] S502 : The log detection apparatus determines the number of normal logs corresponding to each preset log template in each time window based on multiple preset log templates and multiple normal logs, and obtains a preset matrix.

[0128] It should be noted that the multiple preset log templates include a normal log template and an abnormal log template, wherein the normal log template is a log template obtained based on a normal log, and the abnormal log template is a log template obtained based on an abnormal log.

[0129] It can be understood that in the case where multiple preset log templates include normal log templates and abnormal log templates, because one element represents the number of normal logs corresponding to the preset log template within the time window, the number of normal logs corresponding to the abnormal log template in the preset matrix within the time window is 0, so when there is an abnormal log in the log to be detected, that is, the number of normal logs corresponding to the abnormal log template in the matrix to be detected within the time window is not 0, the difference between the frequency vector of the time window including the abnormal log template in the matrix to be detected and the preset matrix can be increased.

[0130] As a possible implementation method, the log detection device determines the number a1 of normal logs corresponding to the first preset log template in the first time window, and uses a1 as the element of the first row and first column of the preset matrix, and so on to obtain the preset matrix.

[0131] As an example, an example of a representation of a preset matrix provided in this application is as follows:

[0132]

[0133] In this example, an element in the preset matrix represents the number of normal logs corresponding to a preset log template within a time window. For example, the element in the first row and first column indicates that the number of normal logs corresponding to the first preset log template in the first time window is a2. The element in the second row and second column indicates that the number of normal logs corresponding to the second preset log template in the second time window is e2. And so on.

[0134] It should be noted that each row in the preset matrix shown in this example can represent the number of normal logs corresponding to a preset log template in multiple time windows. In this case, each column in the preset matrix represents the number of normal logs corresponding to multiple preset log templates in a time window. Each row in the preset matrix shown in this example can also represent the number of normal logs corresponding to multiple preset log templates in a time window. In this case, each column in the preset matrix represents the number of normal logs corresponding to a preset log template in multiple time windows.

[0135] Based on this solution, the preset matrix can be determined by determining the number of normal logs corresponding to each preset log template in each time window.

[0136] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the log detection device executing the log detection method. In order to realize the above functions, the log detection device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the embodiments of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0137] The embodiment of the present application can divide the log detection device into functional modules according to the above method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. Optionally, the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, the "module" here can refer to a specific application-specific integrated circuit (ASIC), a circuit, a processor and memory that executes one or more software or firmware programs, an integrated logic circuit, and / or other devices that can provide the above functions.

[0138] In the case of functional module division, Figure 6 FIG. 1 shows a structural diagram of a log detection device. Figure 6 As shown, the log detection device 60 includes an acquisition module 601 and a processing module 602 .

[0139] In some embodiments, the log detection device 60 may further include a storage module ( Figure 6 ), for storing program instructions and data.

[0140] Among them, the acquisition module 601 is used to obtain multiple logs to be detected within a preset time period; the preset time period includes multiple time windows; the processing module 602 is used to determine the matrix to be detected based on multiple preset log templates and multiple logs to be detected; an element in the matrix to be detected represents the number of logs to be detected corresponding to a preset log template in a time window; the processing module 602 is also used to respectively determine the difference between the frequency vector of each time window in the matrix to be detected and the preset matrix to obtain multiple difference degrees; a frequency vector is used to represent the number of logs to be detected corresponding to multiple preset log templates in a time window, and an element in the preset matrix represents the number of normal logs corresponding to a preset log template in a time window; the processing module 602 is also used to determine the abnormal time window including abnormal logs from multiple time windows based on multiple difference degrees.

[0141] Optional, such as Figure 6 As shown, the processing module 602 provided in the embodiment of the present application is used to determine the matrix to be detected based on multiple preset log templates and multiple logs to be detected, including: determining the number of logs to be detected corresponding to each preset log template in each time window to obtain an initial matrix to be detected; multiplying the target element in the initial matrix to be detected by a preset coefficient to obtain the matrix to be detected, where the target element corresponds to the abnormal log template in the multiple preset log templates.

[0142] Optional, such as Figure 6 As shown, the processing module 602 provided in the embodiment of the present application is also used to: obtain multiple normal logs within a preset time period; based on multiple preset log templates and multiple normal logs, determine the number of normal logs corresponding to each preset log template in each time window to obtain a preset matrix.

[0143] Optionally, the difference involved in the embodiment of the present application is the Mahalanobis distance. In this case, if Figure 6 As shown, the processing module 602 provided in the embodiment of the present application is also used to determine the abnormal time window including the abnormal log from multiple time windows based on multiple difference degrees, including: determining the normal distribution graph of multiple Mahalanobis distances; determining the standard deviation interval of the normal distribution graph; and determining the time window corresponding to the target Mahalanobis distance that is not in the standard deviation interval as including the abnormal time window.

[0144] All relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0145] When the functions of the above functional modules are implemented in the form of hardware, Figure 7 FIG. 1 shows a structural diagram of a log detection device. Figure 7 As shown, the log detection device 70 includes a processor 701, a memory 702 and a bus 703. The processor 701 and the memory 702 may be connected via the bus 703.

[0146] Processor 701 is the control center of log detection device 70 and can be a single processor or a collective term for multiple processing elements. For example, processor 701 can be a general-purpose central processing unit (CPU) or other general-purpose processor. The general-purpose processor can be a microprocessor or any conventional processor.

[0147] As an embodiment, the processor 701 may include one or more CPUs, such as Figure 7 CPU 0 and CPU 1 are shown in Figure 1.

[0148] The memory 702 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0149] As a possible implementation, memory 702 can exist independently of processor 701 and can be connected to processor 701 via bus 703 to store instructions or program codes. When processor 701 calls and executes the instructions or program codes stored in memory 702, the method for using a one-time identity provided in the embodiment of the present application can be implemented.

[0150] In another possible implementation, the memory 702 may also be integrated with the processor 701 .

[0151] Bus 703 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. This bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0152] It should be pointed out that Figure 7 The structure shown does not constitute a limitation on the log detection device 70. Figure 7 In addition to the components shown, the log detection device 70 may include more or fewer components than shown, or combine certain components, or arrange the components differently.

[0153] As an example, combining Figure 6 The functions implemented by the acquisition module 601 and the processing module 602 in the log detection device 60 are the same as those implemented by Figure 7 The functions of the processor 701 in are the same.

[0154] Optional, such as Figure 7 As shown, the log detection device 70 provided in the embodiment of the present application may further include a communication interface 704 .

[0155] The communication interface 704 is used to connect to other devices via a communication network. The communication network can be Ethernet, wireless access network, wireless local area network (WLAN), etc. The communication interface 704 may include a receiving unit for receiving data and a sending unit for sending data.

[0156] In a possible implementation, in the log detection device 70 provided in the embodiment of the present application, the communication interface 704 may also be integrated into the processor 701 , which is not specifically limited in the embodiment of the present application.

[0157] As a possible product form, the log detection device of the embodiment of the present application can also be implemented using the following: one or more field programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gate logic, discrete hardware components, any other suitable circuits, or any combination of circuits capable of performing the various functions described throughout this application.

[0158] Through the description of the above embodiments, those skilled in the art will clearly understand that for the sake of convenience and brevity, only the division of the above-mentioned functional units is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units as needed, that is, the internal structure of the device can be divided into different functional units to complete all or part of the functions described above. The specific working processes of the above-mentioned systems, devices, and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0159] An embodiment of the present application also provides a computer-readable storage medium having a computer program or instruction stored thereon, wherein when the computer program or instruction is executed, the computer is caused to execute each step in the method flow shown in the above method embodiment.

[0160] An embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute each step of the method flow shown in the above method embodiment.

[0161] An embodiment of the present application provides a chip system, including: a processor and an interface circuit; the interface circuit is used to receive a computer program or instruction and transmit it to the processor; the processor is used to execute the computer program or instruction so that the chip system executes the method described in the first aspect above.

[0162] The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more conductors, a portable computer disk, and a hard disk. Random Access Memory (RAM), Read-Only Memory (ROM), Erasable Programmable Read-Only Memory (EPROM), registers, hard disks, optical fibers, portable Compact Disc Read-Only Memory (CD-ROM), optical storage devices, magnetic storage devices, or any other form of computer-readable storage medium known in the art, or any combination thereof. An exemplary storage medium is coupled to a processor so that the processor can read information from and write information to the storage medium. Of course, the storage medium may also be an integral part of the processor. The processor and storage medium may be located in a specific-purpose ASIC. In the embodiments of the present application, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0163] Since the log detection device, computer-readable storage medium, and computer program product provided in this embodiment can be applied to the log detection method provided in this embodiment, the technical effects that can be obtained can also refer to the above-mentioned method embodiments, and the embodiments of this application will not be repeated here.

[0164] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the claimed application, those skilled in the art may understand and implement other variations of the disclosed embodiments by reviewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple situations. A single processor or other unit may implement several functions listed in the claims. Certain measures are recorded in different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

[0165] Although the present application has been described with reference to specific features and embodiments thereof, it is apparent that various modifications and combinations may be made thereto without departing from the spirit and scope of the present application. Accordingly, this specification and the drawings are merely illustrative of the present application as defined by the appended claims and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, those skilled in the art may make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, the present application is intended to include such modifications and variations as fall within the scope of the claims of the present application and their equivalents.

Claims

1. A log detection method, characterized in that: The method comprises: Acquire multiple logs to be detected within a preset time period; the preset time period includes multiple time windows; Determine a matrix to be detected based on multiple preset log templates and the multiple logs to be detected; an element in the matrix to be detected represents the number of logs to be detected corresponding to a preset log template in a time window; Determining the difference between the frequency vector of each time window in the matrix to be detected and the preset matrix respectively, to obtain multiple difference degrees; one of the frequency vectors is used to represent the number of logs to be detected corresponding to the multiple preset log templates in the time window, and one element in the preset matrix represents the number of normal logs corresponding to the preset log template in the time window; determining an abnormal time window including an abnormal log from the multiple time windows according to the multiple difference degrees; The determining of a matrix to be detected based on the plurality of preset log templates and the plurality of logs to be detected includes: Determine the number of logs to be detected corresponding to each preset log template in each time window to obtain an initial matrix to be detected; The target element in the initial matrix to be detected is multiplied by a preset coefficient to obtain a matrix to be detected, wherein the target element corresponds to an abnormal log template among the multiple preset log templates; the preset log template corresponding to the element in the preset matrix is ​​the same as the preset log template corresponding to the element in the matrix to be detected corresponding to the element in the preset matrix; the time window corresponding to the element in the preset matrix is ​​the same as the time window corresponding to the element in the preset matrix corresponding to the element in the matrix to be detected.

2. The method according to claim 1, characterized in that The method further comprises: Get multiple normal logs within a preset time period; Based on the multiple preset log templates and the multiple normal logs, the number of normal logs corresponding to each preset log template in each time window is determined respectively to obtain the preset matrix.

3. The method according to any one of claims 1-2, characterized in that The difference degree is a Mahalanobis distance, and determining the abnormal time window including the abnormal log from the multiple time windows according to the multiple difference degrees includes: Determine the normal distribution plots of multiple Mahalanobis distances; determining a standard deviation interval of the normal distribution graph; A time window corresponding to the target Mahalanobis distance that is not within the standard deviation interval is determined as the abnormal time window.

4. A log detection device, characterized in that: The device comprises: an acquisition module and a processing module; The acquisition module is used to acquire multiple logs to be detected within a preset time period; the preset time period includes multiple time windows; The processing module is configured to determine a matrix to be detected based on a plurality of preset log templates and the plurality of logs to be detected; an element in the matrix to be detected represents the number of logs to be detected corresponding to a preset log template within a time window; The processing module is further configured to determine the difference between the frequency vector of each time window in the matrix to be detected and the preset matrix, respectively, to obtain multiple difference degrees; one of the frequency vectors is used to represent the number of logs to be detected corresponding to the multiple preset log templates in the time window, and one element in the preset matrix represents the number of normal logs corresponding to the preset log template in the time window; The processing module is further configured to determine an abnormal time window including an abnormal log from the multiple time windows according to the multiple difference degrees; The processing module is configured to determine a matrix to be detected based on a plurality of preset log templates and the plurality of logs to be detected, including: Determine the number of logs to be detected corresponding to each preset log template in each time window to obtain an initial matrix to be detected; The target element in the initial matrix to be detected is multiplied by a preset coefficient to obtain a matrix to be detected, wherein the target element corresponds to an abnormal log template among the multiple preset log templates; the preset log template corresponding to the element in the preset matrix is ​​the same as the preset log template corresponding to the element in the matrix to be detected corresponding to the element in the preset matrix; the time window corresponding to the element in the preset matrix is ​​the same as the time window corresponding to the element in the preset matrix corresponding to the element in the matrix to be detected.

5. The device according to claim 4, characterized in that The processing module is further configured to: Obtaining multiple normal logs within the preset time period; Based on the multiple preset log templates and the multiple normal logs, the number of normal logs corresponding to each preset log template in each time window is determined respectively to obtain the preset matrix.

6. The device according to any one of claims 4-5, characterized in that The difference degree is a Mahalanobis distance, and the processing module is further configured to determine an abnormal time window including an abnormal log from the multiple time windows according to the multiple difference degrees, including: Determine the normal distribution plots of multiple Mahalanobis distances; determining a standard deviation interval of the normal distribution graph; A time window corresponding to the target Mahalanobis distance that is not within the standard deviation interval is determined as the abnormal time window.

7. A log detection device, characterized in that: The log detection device includes: a processor, the processor is coupled to a memory, the memory is used to store programs or instructions, and when the program or instructions are executed by the processor, the device executes the method according to any one of claims 1 to 3.

8. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instructions are executed, the computer is caused to perform the method according to any one of claims 1 to 3.

Images