Anomaly detection method and device of computer cluster system, and storage medium

Abstract

The application provides an abnormality detection method and device of a computer cluster system, and a storage medium, wherein the method comprises: collecting a first log generated by the computer cluster system; analyzing error category logs in the first log through an error log model to determine whether the computer cluster system is abnormal, obtaining a first determination result, and obtaining a frequency component of the first log, determining whether the computer cluster system is abnormal according to whether the frequency component is within a preset range, and obtaining a second determination result, wherein the error log model is trained through machine learning using multiple sets of data, and each set of data in the multiple sets of data comprises error level logs; and determining whether the computer cluster system is abnormal according to the first determination result and the second determination result. By using the technical solution, the problems of a single detection result and the like in the prior art, in which an abnormal type is labeled in advance for detection, are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computers, in particular to an anomaly detection method and device of a computer cluster system and a storage medium. BACKGROUND

[0002] In a cloud computing environment, since cloud services are based on virtual resources, a large number of hardware devices and software systems work together, and in addition to the diversity and flexibility of user demand under multi-tenancy, the cloud computing system far exceeds a single system in scale and complexity, even exceeding traditional distributed systems. In addition, the complex component structure and software hierarchy in the cloud computing environment bring many system failure problems. In order to ensure the availability and reliability of the upper layer service, how to quickly and accurately identify and detect system failures becomes a key problem worth exploring.

[0003] In the process of system running, a large amount of performance data and logs will be generated, wherein the logs record the execution track of the system and exist in all components of the system, and are the most direct data representing the running state of the system and the user behavior, and contain a large amount of important and valuable information. System failure detection based on log analysis has become an important means to ensure the availability of software systems, and is also one of the important applications of log data. In related technologies, features representing system running anomalies are generally extracted, and a failure detection model is established based on these features, and finally the system failure is detected. In this way, there needs to be enough labeled abnormal data for training, and when detecting, only the labeled abnormal types can be detected, and the abnormalities not in the training data cannot be detected.

[0004] At present, there is no effective solution to the problems of the detection by the pre-labeled abnormal type in related technologies, and the single detection result. SUMMARY

[0005] The embodiments of the present application provide an anomaly detection method and device of a computer cluster system and a storage medium, to solve the problems of the detection by the pre-labeled abnormal type in related technologies, and the single detection result.

[0006] According to one embodiment of the present application, an anomaly detection method of a computer cluster system is provided, comprising: collecting a first log generated by the computer cluster system; analyzing, by means of an error log model, logs of an error category in the first log to determine whether the computer cluster system is abnormal, obtaining a first determination result, and obtaining a frequency component of the first log, determining whether the computer cluster system is abnormal according to whether the frequency component is within a preset range, and obtaining a second determination result, wherein the error log model is trained by machine learning using a plurality of sets of data, each set of data in the plurality of sets of data comprising: error level logs; and determining whether the computer cluster system is abnormal according to the first determination result and the second determination result.

[0007] According to another embodiment of the present application, an anomaly detection device of a computer cluster system is also provided, comprising: a collection module configured to collect a first log generated by the computer cluster system; a first determination module configured to analyze, by means of an error log model, logs of an error category in the first log to determine whether the computer cluster system is abnormal, obtain a first determination result, and obtain a frequency component of the first log, determine whether the computer cluster system is abnormal according to whether the frequency component is within a preset range, and obtain a second determination result, wherein the error log model is trained by machine learning using a plurality of sets of data, each set of data in the plurality of sets of data comprising: error level logs; and a second determination module configured to determine whether the computer cluster system is abnormal according to the first determination result and the second determination result.

[0008] According to still another embodiment of the present application, a storage medium is also provided, the storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the method embodiments described above when running.

[0009] According to still another embodiment of the present application, an electronic device is also provided, comprising a memory and a processor, the memory storing a computer program, and the processor being configured to run the computer program to execute the steps in any of the method embodiments described above.

[0010] By the present application, the error category logs in the collected first logs are analyzed by an error log model, whether the computer cluster system is abnormal is determined to obtain a first determination result, and the frequency component of the first logs is obtained, whether the computer cluster system is abnormal is determined according to whether the frequency component is in a preset range to obtain a second determination result, wherein the error log model is trained by machine learning using multiple sets of data, each set of data in the multiple sets of data includes error level logs, and the technical solution of determining whether the computer cluster system is abnormal according to the first determination result and the second determination result solves the problems in the related art, such as the single detection result of the pre-labeled detection method for the abnormal type, and can comprehensively determine whether the computer cluster system is abnormal. BRIEF DESCRIPTION OF DRAWINGS

[0011] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and serve to explain the principles of the application, and do not limit the application. In the drawings:

[0012] Figure 1 is a hardware structure block diagram of a terminal of a computer cluster system abnormality detection method according to an embodiment of the present application;

[0013] Figure 2 is a flowchart of a computer cluster system abnormality detection method according to an embodiment of the present application;

[0014] Figure 3 is a structure block diagram of a computer cluster system abnormality detection device according to an embodiment of the present application;

[0015] Figure 4 is another structure block diagram of a computer cluster system abnormality detection device according to an embodiment of the present application;

[0016] Figure 5 is still another flowchart of a computer cluster system abnormality detection method according to a preferred embodiment of the present application;

[0017] Figure 6 is a structure schematic diagram of a computer cluster system abnormality detection device according to a preferred embodiment of the present application;

[0018] Figure 7 is a flowchart of an optional error level log detection method according to a preferred embodiment of the present application;

[0019] Figure 8 is a result schematic diagram of an error log detection part according to a preferred embodiment of the present application;

[0020] Figure 9is a further result diagram of the error log detection part according to the preferred embodiment of the present application;

[0021] Figure 10 is a result diagram of the log frequency detection part according to the preferred embodiment of the present application;

[0022] Figure 11 is a further result diagram of the log frequency detection part according to the preferred embodiment of the present application. DETAILED DESCRIPTION

[0023] Hereinafter, the present application will be described in detail with reference to the accompanying drawings and embodiments. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0024] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence.

[0025] Embodiment 1

[0026] The method embodiment provided by Embodiment 1 of the present application can be executed in a terminal or similar computing device. Taking the case of running on a terminal, Figure 1 is a hardware structure block diagram of a terminal of an abnormality detection method of a computer cluster system according to an embodiment of the present application. As shown in Figure 1 , the terminal 10 can include one or more (only one is shown in Figure 1 ) processors 102 (the processor 102 can include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, and optionally, the above-mentioned terminal can also include a transmission device 106 for communication function and an input and output device 108. Those skilled in the art can understand that Figure 1 The structure shown is only schematic, which does not limit the structure of the above-mentioned terminal. For example, the terminal 10 can also include more or fewer components than those shown in Figure 1 , or have a different configuration with the same function or more function than that shown in Figure 1 . Figure 1 .

[0027] The memory 104 can be used to store computer programs, such as software programs of application software and modules, such as a computer program corresponding to the navigation method of the online car-hailing in the embodiments of the present application. The processor 102 can execute various functional applications and data processing, i.e., implement the above method, by running the computer programs stored in the memory 104. The memory 104 can include a high-speed random access memory, and can further include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 104 can further include memories remotely arranged with respect to the processor 102, which can be connected to the terminal 10 through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0028] The transmission device 106 is used to receive or send data via a network. Specific examples of the above network can include a wireless network provided by a communication provider of the terminal 10. In one example, the transmission device 106 includes a network adapter (NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet in a wireless manner.

[0029] In the present embodiment, an anomaly detection method of a computer cluster system running on a terminal is provided, Figure 2 which is a flowchart of the anomaly detection method of the computer cluster system according to the embodiments of the present application, as shown in Figure 2 The flowchart includes the following steps:

[0030] Step S202, collecting a first log generated by a computer cluster system;

[0031] Step S204, analyzing logs of error categories in the first log through an error log model to determine whether the computer cluster system is abnormal, obtaining a first determination result, and obtaining a frequency component of the first log, determining whether the computer cluster system is abnormal according to whether the frequency component is within a preset range, and obtaining a second determination result, wherein the error log model is trained by machine learning using a plurality of sets of data, and each set of data in the plurality of sets of data includes logs of error levels;

[0032] Step S206, determining whether the computer cluster system is abnormal according to the first determination result and the second determination result.

[0033] This invention analyzes error category logs in a first set of collected logs using an error log model to determine whether the computer cluster system is abnormal, obtaining a first determination result. It also obtains the frequency component of the first logs and determines whether the computer cluster system is abnormal based on whether the frequency component is within a preset range, obtaining a second determination result. The error log model is trained using multiple sets of data through machine learning, and each set of data includes logs at the error level. This technical solution for determining whether the computer cluster system is abnormal based on the first and second determination results solves the problems of pre-labeling anomaly types and resulting in limited detection results in related technologies, thus enabling a comprehensive determination of whether the computer cluster system is abnormal.

[0034] In an optional embodiment of the present invention, after collecting the first logs generated by the computer cluster system, the first logs are classified to obtain multiple log categories; the probability of occurrence of each log category within a specified statistical period is determined; and log categories with a probability of occurrence greater than a first threshold are added to the error log model.

[0035] Optionally, obtaining the frequency component of the first log includes: determining the frequency component of each reporting point of the first log at predetermined time intervals, wherein there are n reporting points, where n is a positive integer; and constructing the frequency component x of the first log using the frequency components of each reporting point. t =[x1,x2,…,x n ], where x1, x2, ..., x n This corresponds to the frequency components of different reporting points.

[0036] Specifically, determining whether the computer cluster system is abnormal based on whether the frequency component is within a preset range, and obtaining a second determination result, includes: based on the frequency component x t =[x1,x2,…,x n The frequency component x of the previous time step compared to the current time step t-1 Obtaining the cycle distance D ring According to the frequency component x t =[x1,x2,…,x n ] and the frequency component x t =[x1,x2,…,x n The center frequency component x center To obtain the center-to-distance D center Among them, the center frequency component x center Determined in the following ways: in, respectively, x1, x2, …, x n a mean vector of the ring distance D ring greater than a second threshold value, and the center distance D center greater than a third threshold value, determining that the computer cluster system is abnormal.

[0037] Step S206K can have various implementations. In one optional embodiment, determining whether the system is abnormal according to the first determination result and the second determination result comprises:

[0038] when the first determination result indicates that the computer cluster system is abnormal and the second determination result indicates that the computer cluster system is abnormal, determining that the computer cluster system is abnormal;

[0039] when the first determination result indicates that the computer cluster system is not abnormal and the second determination result indicates that the computer cluster system is abnormal, determining that the computer cluster system is abnormal;

[0040] when the first determination result indicates that the computer cluster system is abnormal and the second determination result indicates that the computer cluster system is not abnormal, determining that the computer cluster system is abnormal;

[0041] when the first determination result indicates that the computer cluster system is not abnormal and the second determination result indicates that the computer cluster system is not abnormal, determining that the computer cluster system is not abnormal.

[0042] Further, when the first determination result indicates that the computer cluster system is abnormal, setting the detection result of whether the computer cluster system is abnormal according to the first determination result and the second determination result as an important level;

[0043] when the first determination result indicates that the computer cluster system is not abnormal and the second determination result indicates that the computer cluster system is abnormal, setting the detection result of whether the computer cluster system is abnormal according to the first determination result and the second determination result as an alarm level.

[0044] In summary, the embodiment of the present application proposes a method for detecting abnormality according to system logs, which combines the detection algorithm based on error level logs and the detection algorithm based on log frequency components, considers the importance of error level logs and the regularity of logs generated when the system is normally running, and does not need manual labeling of training data.

[0045] Those skilled in the art can clearly understand that the method according to the above-mentioned embodiments can be realized by means of software and necessary general hardware platforms, and of course, can also be realized by hardware, but in many cases, the former is a better implementation. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, or an optical disc) and includes a plurality of instructions for causing a terminal device (which can be a mobile phone, a computer, a server, or a network device) to execute the method described in each embodiment of the present application.

[0046] In this embodiment, an anomaly detection device of a computer cluster system is also provided, which is used to implement the above-mentioned embodiments and preferred embodiments, and will not be described again. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, implementation of hardware, or a combination of software and hardware, is also possible and is contemplated.

[0047] Figure 3 is a structural block diagram of an anomaly detection device of a computer cluster system according to an embodiment of the present application, as shown in Figure 3 The device comprises:

[0048] The acquisition module 30 is configured to acquire a first log generated by the computer cluster system.

[0049] The first determination module 32 is configured to analyze error category logs in the first log by using an error log model, determine whether the computer cluster system is abnormal, obtain a first determination result, and obtain a frequency component of the first log, determine whether the computer cluster system is abnormal according to whether the frequency component is within a preset range, and obtain a second determination result. The error log model is trained by machine learning using a plurality of sets of data, and each set of data in the plurality of sets of data includes error level logs.

[0050] The second determination module 34 is configured to determine whether the computer cluster system is abnormal according to the first determination result and the second determination result.

[0051] The application determines whether the computer cluster system is abnormal by analyzing the error category logs in the collected first logs through an error log model, obtaining a first determination result, and obtaining a frequency component of the first logs, and determines whether the computer cluster system is abnormal according to whether the frequency component is within a preset range, to obtain a second determination result, wherein the error log model is trained by machine learning using multiple sets of data, and each set of data in the multiple sets of data includes error level logs, and the technical solution for determining whether the computer cluster system is abnormal according to the first determination result and the second determination result solves the problems in the related art, such as the single detection result of the pre-labeled abnormal type detection mode, and can comprehensively determine whether the computer cluster system is abnormal.

[0052] Figure 4 is another structural block diagram of an abnormality detection device of a computer cluster system according to an embodiment of the application, as shown in the figure, the device includes: Figure 4

[0053] The classification module 36 is configured to classify the first logs to obtain multiple log categories.

[0054] The third determination module 38 is configured to determine the occurrence probability of each log category in a specified statistical period.

[0055] The adding module 40 is configured to add the log category with the occurrence probability greater than the first threshold to the error log model.

[0056] Optionally, the first determination module 32 is further configured to determine the frequency component of each reporting point of the first logs at a predetermined time interval, wherein the reporting points exist n, and n is a positive integer; the frequency component x of the first logs is composed of the frequency component of each reporting point. t = [x1, x2, …, xn], wherein x1, x2, …, xn correspond to the frequency components of different reporting points. n n

[0057] Specifically, the first determination module 32 is further configured to: obtain the ring distance D between the frequency component x = [x1, x2, …, xn] and the frequency component x of the last moment of the current moment; t n t-1 ring t n t n center ​​​​​​​​​​​to obtain a center ratio distance D center wherein the center frequency component x center is determined by the following way: wherein, respectively corresponding to x1, x2, …, x n a mean vector of the ring ratio distance D ring is greater than a second threshold value, and the center ratio distance D center is greater than a third threshold value, it is determined that the computer cluster system is abnormal.

[0058] The second determination module 34 is further configured to determine that the computer cluster system is abnormal when the first determination result indicates that the computer cluster system is abnormal and the second determination result indicates that the computer cluster system is abnormal.

[0059] The second determination module 34 is further configured to determine that the computer cluster system is abnormal when the first determination result indicates that the computer cluster system is abnormal and the second determination result indicates that the computer cluster system is abnormal.

[0060] The second determination module 34 is further configured to determine that the computer cluster system is abnormal when the first determination result indicates that the computer cluster system is abnormal and the second determination result indicates that the computer cluster system is abnormal.

[0061] The second determination module 34 is further configured to determine that the computer cluster system is abnormal when the first determination result indicates that the computer cluster system is abnormal and the second determination result indicates that the computer cluster system is abnormal.

[0062] Further, when the first determination result indicates that the computer cluster system is abnormal, the detection result of whether the computer cluster system is abnormal determined according to the first determination result and the second determination result is set as an important level.

[0063] Further, when the first determination result indicates that the computer cluster system is abnormal, the detection result of whether the computer cluster system is abnormal determined according to the first determination result and the second determination result is set as an important level.

[0064] It should be noted that the above various modules can be implemented by software or hardware, and for the latter, the following implementation manners can be used, but are not limited thereto: the above modules are located in the same processor; or the above various modules are located in different processors in any combination.

[0065] The technical scheme of the following preferred embodiment of the present application mainly has the idea of detecting abnormalities in system generated logs. Generally, system generated logs are classified into multiple levels, such as ERROR / INFO / DEBUG, indicating the importance of the logs. Considering that logs of different levels have different importance, the technical scheme of the preferred embodiment of the present application combines the error level based log detection algorithm and the log frequency based detection algorithm to detect system abnormalities. The specific combination process is described in the following embodiment.

[0066] The error level based log detection algorithm removes unimportant logs through an algorithm and retains important logs for detecting system abnormalities. The log frequency based detection algorithm learns the log frequency rules during normal system operation, and judges the deviation of system logs and historical statistical values in real time. If the deviation exceeds the set threshold, it is considered that the system has an abnormality. The two detection methods complement each other and can detect most system abnormalities. The following preferred embodiment describes the process of the two detection methods complementing each other, but does not limit the technical scheme of the embodiment of the present application.

[0067] As shown in Figure 5 The technical scheme adopted by the preferred embodiment of the present application includes the following steps:

[0068] Step 1: Collect historical logs and perform structured processing to obtain the timestamp, log level, reporting service, reporting instance, and reporting host of each log.

[0069] Step 2: Extract log entries with log level higher than or equal to "ERROR" from the historical logs, and process them according to the following process A; process the entire set of logs according to process B.

[0070] The specific steps of process A are as follows.

[0071] It should be noted that the following steps only represent the operation steps required for process A, and do not represent the operation sequence of the process executed by process A.

[0072] Step A1: Extract templates for log classification from the original text of the logs and obtain multiple log classifications (C1, C2, ……Cn).

[0073] Step A2: Divide the historical logs into multiple log sets according to the hourly time granularity. For error level logs in each log set, use the log classification template for classification. Under each service, perform onehot encoding for each classification, for example (C1:1, C2:0, ……Cn:1)

[0074] Step A3: Calculate the occurrence probability of each log category for the entire statistical period. Assuming that the statistical period is N hours, the category Ci is coded as 1 in the first hour, coded as 1 in the second hour, and coded as 0 in other time periods, then the occurrence probability of the category Ci is

[0075] If Pi is higher than the occurrence probability threshold, it is considered as a long-term occurrence, which has no representation effect on system anomalies, and the log category is recorded in the error log model and marked as normal; other categories are marked as abnormal.

[0076] Step A4: The content in the model can be modified by manual annotation. If a log category is annotated as abnormal, the category is recorded in the error log model and marked as abnormal. Similarly, the log category can also be modified as normal.

[0077] Step A5: Collect the system real-time generated logs regularly; extract structured information from the logs in the current 5 minutes (or other time), and eliminate non-error level logs. Under each service, use the log category template for classification, and perform onehot encoding for each category. For example, (C1:1, C2:0, … Cn:1).

[0078] Step A6: Query the error log model of the corresponding service in the database, eliminate the log categories marked as normal, and obtain the effective log categories. According to the effective log categories, find the log original text for fault analysis, and the (service, instance, host) corresponding to the log original text is the fault point.

[0079] The specific steps of process B are as follows.

[0080] Step B1: For each service, count the log frequency x reported by each reporting point (host, instance) at a certain time interval. Assuming that there are n reporting points belonging to service 1, then the frequency vector dimension is equal to n, and the frequency vector x = [x1, x2, …, xn]. n ]。

[0081] Calculate the mean vector of all time interval frequency vectors to obtain the center frequency vector.

[0082]

[0083] Step B2: Calculate the distance between each frequency vector X t and the previous time frequency vector X t-1 to obtain the ring distance statistics. The greater the distance, the greater the change in the current point frequency distribution compared to the previous time.

[0084] Step B3: Calculate the distance between each frequency vector X t and the center frequency vector X center , and get the center ratio distance statistics. The greater the distance, the greater the deviation of the current point frequency distribution from the historical mean.

[0085] Step B4: For all time, use statistical methods to get the normal distance range, i.e. the ring ratio distance threshold T ring , which is considered to be too large if it exceeds the threshold.

[0086] Step B5: For all time, use statistical methods to get the normal distance range, i.e. the center ratio distance threshold T center , which is considered to be too large if it exceeds the threshold.

[0087] Step B6: Save the center frequency vector X center , threshold T ring and T center of each service to the statistical model.

[0088] Step B7: Collect the system real-time generated logs regularly, and extract the structured information of the logs. Calculate the log frequency vector X t under each service service.

[0089] Step B8: Calculate the distance between the frequency vector X t and the frequency vector X t-1 of the previous time, and get the ring ratio distance Dring.

[0090] Calculate the distance between the frequency vector X t and the center frequency vector X center , and get the center ratio distance D center .

[0091] Step B9: If the ring ratio distance D ring is greater than the threshold T ring , and the center ratio distance D center is greater than the threshold T center , it means that the current point not only deviates greatly from the historical center frequency, but also has a mutation compared with the previous time, and the abnormal start time is recorded; if the anomaly has started, only the deviation of the current time and the historical center frequency needs to be judged, if the current center ratio distance D center is greater than the threshold T center , the anomaly continues, otherwise the anomaly ends.

[0092] Step B10: If an anomaly is detected, score each dimension of the anomaly based on the distance calculation formula. Sort each contribution in descending order, and the top x contributions are added together, which is not less than the contribution threshold. The dimension is considered as an abnormal dimension, i.e., an abnormal reporting point (host, instance).

[0093] Step 3: Merge the detection results of process A and process B. If process A detects an anomaly, process B will determine the detection result as abnormal regardless of whether it is abnormal or not, set the detection result as an important level, and show the effective log original text to the user; if process A detects normally and process B detects an anomaly, set the detection result as a warning level, and show the log frequency curve to the user; if no anomaly is detected, the detection result is normal.

[0094] The technical scheme of the preferred embodiment of the present application utilizes the different use scenarios and characteristics of error logs and log frequencies, classifies data sources, and adopts different detection methods, so that the detection result is more accurate. The method proposed in the present application has high calculation efficiency and does not require too many computing resources. The training data requirement is not high, and manual labeling is not required, so the practical application range is wide.

[0095] In summary, the error log detection algorithm provided by the above preferred embodiment uses onehot encoding to eliminate the noise, i.e., the irrelevant error level logs output by the system. Specifically, see step A3: If Pi is higher than the threshold (the threshold can be set to 0.8 in this embodiment), it is considered as a long-term appearing log classification that has no representation effect on system anomalies, and is recorded to the error log model and marked as normal; other classifications are marked as abnormal; during ordinary log detection, multiple related frequencies are combined into a vector, and the center ratio distance and ring ratio distance of the vector are used to detect system anomalies, i.e., the deviation of the current point from the history is measured, and the deviation of the current point from the previous moment is also measured.

[0096] Figure 6 is a structural schematic diagram of an anomaly detection device of a computer cluster system according to the preferred embodiment of the present application, as Figure 6 shown, the entire device is divided into an acquisition unit 60, a log storage unit 62, a model training unit 64, a real-time detection unit 66, and a display unit 68. The functions of each unit are as follows:

[0097] The collection unit continuously collects original logs from the system to be detected, extracts structured information of each log, and classifies each log according to a log classification template to obtain a classification number. The log original text, classification number, and structured information are written into the log storage unit. The structured information includes the timestamp, log level, reporting service, reporting instance, and reporting host of each log.

[0098] The log storage unit saves the output information of the collection unit for subsequent log reading. According to actual needs, logs can be backed up in proportion, or expired logs can be periodically cleared to ensure availability.

[0099] The model training unit is divided into historical error level log modeling and historical log frequency modeling. The historical error level log modeling part reads logs of a certain historical period from the log storage unit, identifies whether each log classification represents system abnormalities, and records the results to the model storage unit. The historical log frequency modeling part reads logs of a certain historical period from the log storage unit, converts log text information into frequency, models the frequency, and records the model to the storage unit.

[0100] The model storage unit saves the modeling results of the model training unit for use during detection.

[0101] The real-time detection unit is divided into real-time error log detection and real-time log frequency detection. The real-time error log detection part reads logs within the current time interval from the log storage unit, judges whether there is an abnormal log classification according to the error log model; the real-time log frequency detection reads logs within the current time interval from the log storage unit, converts them into frequency, and calculates whether the log frequency deviates from the normal model according to the log frequency model.

[0102] The display unit presents the detection results to the user. The user can view the detection results and retrieve the corresponding log information. The user can mark whether the detection results are correct, and the display module adjusts the corresponding model accordingly.

[0103] The following will be further described in combination with the accompanying drawings Figures 1-2 The technical solutions are further described in detail, wherein, Figures 8-9 is Figure 6 In the above, Figures 8-9 The horizontal axis is time, and the vertical axis is the onehot encoding result corresponding to each log classification. Figure 8 is the result after step A5 encoding. Figure 9 is the onehot encoding result after step A6 elimination.

[0104] Figures 10-11 is Figure 6The result of the Chinese-Japanese log frequency detection part is shown in the schematic diagram, wherein the JSD distance is used to measure the distance between the frequency vectors, wherein the 10 reporting points constitute a 10-dimensional frequency vector, and the relative JSD statistics and the center JSD statistics de are obtained according to the frequency vector.

[0105] The historical log frequency modeling part needs to calculate the distance between two frequency vectors, and here a suitable measurement method needs to be used. The (Kullback-Leibler divergence, KL for short) divergence is a commonly used distance measurement method, which starts from the perspective of information entropy and is also called relative entropy, and measures the difference between two probability distributions in the same event space. However, due to the asymmetry of KL divergence and other reasons, the embodiment of the present application adopts a variant of KL divergence: (jensen-shannon) JSD distance.

[0106] Suppose two probability distributions P and Q, the definition of JSD distance is:

[0107]

[0108] wherein, D(P||M) is the KL distance. For a discrete probability distribution, let P=[p1,p2,…,p n ] and Q=[q1,q2,…,q n ];

[0109] Then

[0110]

[0111] Further, based on the JSD distance, the specific steps of the historical log frequency modeling part are as follows:

[0112] Step C1: According to a certain time interval (which can be 5 minutes in this embodiment, or other suitable time), the log frequency x reported by each reporting point (host, instance) is counted. Assuming that there are n reporting points belonging to service1, then the frequency vector dimension is equal to n, and the frequency vector x=[x1,x2,…,x n ].

[0113] Calculate the mean vector of all time interval frequency vectors to obtain the center frequency vector.

[0114]

[0115] Step C2: Convert the frequency vector to a frequency ratio vector P (conversion operation can also not be performed), the i-th element of P is:

[0116]

[0117] Similarly, convert the center frequency vector to the center frequency proportion vector P center .

[0118] Step C3: Calculate the JSD distance between each frequency proportion vector P t and the previous time frequency proportion vector P t-1 , to get the ring JSD distance statistics.

[0119] Step C4: Calculate the JSD distance between each frequency proportion vector P t and the center frequency proportion vector P center , to get the center JSD distance statistics.

[0120] Step C5: For all time ring distance, use the KDE (kernel density estimation) algorithm to calculate the ring distance threshold T ring . In this embodiment, the probability threshold is set to 95%.

[0121] Step C6: For all time center distance, use the KDE (kernel density estimation) algorithm to calculate the center distance threshold T center .

[0122] Step C7: Save the center frequency proportion vector P center , threshold T ring and T center of each service to the statistical model.

[0123] Further, based on the JSD distance, the specific steps of the real-time log frequency detection part are as follows:

[0124] Step D1: Collect the system real-time generated log regularly, which is collected once every 5 minutes in this embodiment. Extract the structured information of the log within the current 5 minutes. Statistically obtain the log frequency vector X t under each service service, and convert it to the frequency proportion vector P t .

[0125] Step D2: Calculate the JSD distance between the frequency proportion vector P t and the previous time frequency proportion vector P t-1 , to get the ring JSD distance D ring .

[0126] Calculate the JSD distance between the frequency proportion vector P t and the center frequency proportion vector P center , to get the center JSD distance D center .

[0127] Step D3: If the ring JSD distance D ringgreater than a threshold T ring , and the center JSD distance D center is greater than a threshold T center , then this is the start of an anomaly; if an anomaly has started, and the current center JSD distance D center is greater than a threshold T center , then the anomaly continues, otherwise the anomaly terminates.

[0128] Step D4: If an anomaly is detected, calculate the anomaly score of each dimension based on the center JSD distance.

[0129] According to the calculation formula of the above JSD distance, the contribution degree of the ith component to JSD(P||Q) is:

[0130]

[0131] Where P=P t , Q=P center .

[0132] Step D5: Sort each contribution degree in descending order, and the dimensions whose top x contribution degrees add up to no less than the contribution degree threshold (set to 80% in this embodiment) are considered as abnormal dimensions, i.e. abnormal reporting points (host, instance).

[0133] The specific steps of the historical error log modeling part are as follows:

[0134] Step E1: Divide the historical logs into multiple log sets according to the time granularity of hours, and classify the error level logs in each log set using the log classification template. Under each service, perform onehot encoding on each classification. Example (C1:1, C2:0, ……Cn:1);

[0135] Step E2: Calculate the occurrence probability of each log classification in the entire statistical period. Assuming that the statistical period is N hours, and the classification Ci is encoded as 1 in the first hour and 1 in the second hour, and 0 in other time periods, then the occurrence probability of the classification Ci is

[0136] If Pi is higher than the threshold (the threshold is set to 0.8 in this embodiment), it is considered as a long-term occurrence and has no representation effect on system anomalies, and the log classification is recorded in the error log model and marked as normal; other classifications are marked as abnormal.

[0137] Step E3: Artificial annotation can modify the content in the model. If a log classification is annotated as abnormal, the classification is recorded in the error log model and marked as abnormal. Similarly, the log classification can also be modified as normal.

[0138] The real-time error log detection part has the following specific steps:

[0139] Step F1: Collect the system real-time generated logs regularly, which is 5 minutes in this embodiment. Extract structured information from the logs in the current 5 minutes, and eliminate non-error level logs. Under each service, use the log classification template for classification, and onehot encode each classification. Example (C1:1, C2:0, ……Cn:1).

[0140] Step F2: Query the error log model corresponding to the service in the database, and eliminate the log classification marked as normal to obtain the effective log classification. According to the effective log classification, find the log original text for fault analysis, and the (service, instance, host) corresponding to the log original text is the fault point.

[0141] In view of the fact that many commercial products output only error level logs due to efficiency and resource shortage considerations, since the magnitude of error level logs is relatively small, the frequency data does not have statistical significance, and false negatives are easy to occur during anomaly detection. At this time, only the detection method based on error level logs can be used. Figure 7 The flowchart of this embodiment is shown, and the specific steps are as follows:

[0142] Step G1: Collect historical error level logs and perform structured processing to obtain the timestamp, log level, reporting service, reporting instance, and reporting host of each log.

[0143] Step G2: Extract the template for log classification from the log original text, and obtain multiple log classifications (C1, C2, ……Cn).

[0144] Step G3: Divide the historical logs into multiple log sets according to the hourly time granularity, and classify the error level logs in each log set using the log classification template. Under each service, onehot encode each classification. Example (C1:1, C2:0, ……Cn:1)

[0145] Step G4: Calculate the occurrence probability of each log classification for the entire statistical period. Assuming that the statistical period is N hours, and the classification Ci is encoded as 1 in the first hour, 1 in the second hour, and 0 in other time periods, then the occurrence probability of the classification Ci is

[0146] If Pi is higher than the threshold (the threshold is set to 0.8 in this embodiment), it is considered that the log classification is long-term, has no representation effect on system anomalies, and is recorded to the error log model and marked as normal; other classifications are marked as abnormal.

[0147] Step G5: Artificial marking can modify the content in the model. If a log classification is marked as abnormal, the classification is recorded to the error log model and marked as abnormal. Similarly, the log classification can also be modified as normal.

[0148] Step G6: Collect the logs generated in real time by the system regularly, and collect once every 5 minutes in this embodiment. Extract the structured information of the logs in the current 5 minutes, and eliminate the non-error level logs. Under each service, classify using the log classification template, and perform onehot coding on each classification. For example, (C1:1, C2:0, …, Cn:1).

[0149] Step G7: Query the error log model of the corresponding service in the database, eliminate the log classifications marked as normal, and obtain the effective log categories. According to the effective log categories, find the log original text for fault analysis, and the (service, instance, host) corresponding to the log original text is the fault point.

[0150] As can be seen from the above steps, the preferred embodiment steps of the present application do not require a large amount of computing resources, do not require artificial marking of data, can identify system anomalies, discover system hidden dangers in advance, and notify system maintenance personnel.

[0151] The embodiment of the present application also provides a storage medium, which stores a computer program, wherein the computer program is set to execute the steps in any one of the method embodiments when running.

[0152] Optionally, in the present embodiment, the above storage medium can be set to store a computer program for executing the following steps:

[0153] S1, collecting first logs generated by a computer cluster system;

[0154] S2, analyzing error category logs in the first logs through an error log model to determine whether the computer cluster system is abnormal, obtaining a first determination result, and obtaining a frequency component of the first logs, determining whether the computer cluster system is abnormal according to whether the frequency component is within a preset range, and obtaining a second determination result, wherein the error log model is trained by machine learning using a plurality of sets of data, and each set of data in the plurality of sets of data includes error level logs;

[0155] S3, determine whether the computer cluster system is abnormal according to the first determination result and the second determination result.

[0156] Optionally, in the embodiment, the storage medium can include, but is not limited to, a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various storage media that can store computer programs.

[0157] The embodiment of the application further provides a storage medium, which stores a computer program, and the computer program is configured to execute the steps in any one of the method embodiments when running.

[0158] The embodiment of the application further provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor is configured to execute the steps in any one of the method embodiments by running the computer program.

[0159] Optionally, the electronic device can further comprise a transmission device and an input / output device, wherein the transmission device is connected with the processor, and the input / output device is connected with the processor.

[0160] Optionally, in the embodiment, the processor can be configured to execute the following steps by the computer program:

[0161] S1, collect a first log generated by a computer cluster system;

[0162] S2, analyze the error category log in the first log by an error log model to determine whether the computer cluster system is abnormal, obtain a first determination result, and obtain a frequency component of the first log, determine whether the computer cluster system is abnormal according to whether the frequency component is in a preset range, and obtain a second determination result, wherein the error log model is trained by machine learning using a plurality of sets of data, and each set of data in the plurality of sets of data comprises an error level log;

[0163] S3, determine whether the computer cluster system is abnormal according to the first determination result and the second determination result.

[0164] Optionally, the specific examples in the embodiment can refer to the examples described in the above embodiments and optional implementation manners, and the embodiment will not be described here.

[0165] The embodiment of the present application further provides an electronic device, comprising a memory and a processor, the memory stores a computer program, and the processor is arranged to execute the computer program to perform the steps in any of the above method embodiments.

[0166] Optionally, the electronic device further comprises a transmission device and an input / output device, wherein the transmission device is connected with the processor, and the input / output device is connected with the processor.

[0167] Optionally, the specific examples in the embodiment can refer to the examples described in the above embodiments and optional implementation manners, and the embodiment will not be described here.

[0168] Obviously, those skilled in the art should understand that the modules or steps of the present application described above can be realized by general computing devices, which can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be realized by program codes executable by computing devices, so that they can be stored in storage devices and executed by computing devices, and in some cases, the steps shown or described can be executed in different order, or they can be manufactured into individual integrated circuit modules, or multiple modules or steps among them can be manufactured into a single integrated circuit module. Thus, the present application is not limited to any particular combination of hardware and software.

[0169] The above only describes the preferred embodiments of the present application and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. An anomaly detection method of a computer cluster system, characterized by, The method comprises the following steps: collecting first logs generated by a computer cluster system; finding log texts of logs of an error category in the first logs through an error log model, performing fault analysis based on the log texts, determining whether the computer cluster system is abnormal, obtaining a first determination result, and obtaining a frequency component of the first logs, and determining whether the computer cluster system is abnormal according to whether the frequency component is within a preset range, and obtaining a second determination result, wherein the error log model is trained by machine learning using a plurality of sets of data, and each set of data in the plurality of sets of data comprises logs of an error level; determining whether the computer cluster system is abnormal according to the first determination result and the second determination result.

2. The method of claim 1, wherein, After collecting the first logs generated by the computer cluster system, the method further comprises: classifying the first logs to obtain a plurality of log categories; determining the occurrence probability of each log category within a specified statistical period; adding log categories with an occurrence probability greater than a first threshold to the error log model.

3. The method of claim 1, wherein, Obtaining the frequency component of the first logs comprises: determining the frequency component of each reporting point of the first logs at a predetermined time interval, wherein there are n reporting points, and n is a positive integer; The frequency component of each reporting point is used to form the frequency component of the first log wherein, Corresponding to the frequency component of different reporting points.

4. The method of claim 3, wherein, determining whether the computer cluster system is abnormal according to whether the frequency component is within a preset range, and obtaining a second determination result, comprising: According to the frequency component The frequency component of the previous time from the current time The acquisition of the ring distance ; from the frequency components with the frequency components a central frequency component to obtain a central ratio distance where the central frequency component is determined by [ , , ], wherein , , corresponds to the mean vector of , respectively. In the ring distance Greater than the second threshold, and the center ratio distance If the value exceeds the third threshold, the computer cluster system is determined to be abnormal.

5. The method according to any one of claims 1 to 4, characterized in that, determining whether the system is abnormal according to the first determination result and the second determination result, comprising: determining that the computer cluster system is abnormal when the first determination result indicates that the computer cluster system is abnormal and the second determination result indicates that the computer cluster system is abnormal; determining that the computer cluster system is abnormal when the first determination result indicates that the computer cluster system is not abnormal and the second determination result indicates that the computer cluster system is abnormal; determining that the computer cluster system is abnormal when the first determination result indicates that the computer cluster system is abnormal and the second determination result indicates that the computer cluster system is not abnormal; determining that the computer cluster system is not abnormal when the first determination result indicates that the computer cluster system is not abnormal and the second determination result indicates that the computer cluster system is not abnormal.

6. The method of claim 5, wherein, The method further comprises: when the first determination result indicates that the computer cluster system is abnormal, setting the detection result of whether the computer cluster system is abnormal according to the first determination result and the second determination result to an important level; when the first determination result indicates that the computer cluster system is not abnormal and the second determination result indicates that the computer cluster system is abnormal, setting the detection result of whether the computer cluster system is abnormal according to the first determination result and the second determination result to an alarm level.

7. An anomaly detection device for a computer cluster system, characterized in that, The method comprises the following steps: a collection module for collecting first logs generated by a computer cluster system; The first determining module is configured to: find log original text of logs of an error category in the first log through an error log model, perform fault analysis based on the log original text, determine whether the computer cluster system is abnormal, obtain a first determination result, and obtain a frequency component of the first log, determine whether the computer cluster system is abnormal according to whether the frequency component is in a preset range, and obtain a second determination result. The error log model is trained by machine learning using a plurality of sets of data, and each set of data in the plurality of sets of data includes logs of an error level. The second determining module is configured to determine whether the computer cluster system is abnormal according to the first determination result and the second determination result.

8. The apparatus of claim 7, wherein, The apparatus further includes: The classification module is configured to classify the first log to obtain a plurality of log categories. The third determining module is configured to determine a probability of occurrence of each log category in a specified statistical period. The adding module is configured to add a log category with a probability of occurrence greater than a first threshold to the error log model.

9. A storage medium, characterized by The storage medium stores a computer program, and the computer program is configured to execute the method in any one of claims 1 to 6 when running. 10.An electronic device comprising a memory and a processor, the electronic device characterized by, The memory stores a computer program, and the processor is configured to execute the computer program to execute the method in any one of claims 1 to 6.

Images