Log-based alarm method and device

Abstract

The present invention discloses a method and device based on log alarm, and relates to the field of computer technology. A specific implementation of the method includes: establishing a log data set related to the system; extracting an abnormal indication information group from the log data set; clustering the log data associated with the abnormal indication information group according to a specific clustering depth to obtain a clustering result; and when the clustering result triggers a predetermined alarm threshold, sending an alarm message to a terminal. This implementation avoids errors introduced by embedding points and improves the timeliness of alarms.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device based on log alarm. Background Art

[0002] With the development of the Internet, various systems are becoming more and more complex, so system monitoring is particularly important. Currently, open source monitoring systems on the market include smokeping, cacti, nagios, zabbix, and ntop. Existing monitoring systems usually use a point-of-sale method to collect system information at regular intervals, and then analyze the collected information to decide whether to alarm. With this type of monitoring system in the prior art, we can only know that a problem has occurred through an alarm, but we don’t know where the problem is. If we want to know the problem, we have to look at the log.

[0003] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art:

[0004] 1. It is necessary to place points in the system. This method of periodic reporting requires placing points in advance. If there are no points during the development process, relevant information cannot be collected and alarms cannot be issued.

[0005] 2. Tracking is invasive to the code. In addition, if the tracking is improper, not only will the data not be collected, but bugs may also be introduced.

[0006] 3. The point-to-point monitoring method can only alarm the system operation status, and cannot directly reflect the points where the system problems occur.

[0007] 4. Because the data collection frequency cannot be too high, otherwise it will affect system performance, there is a delay in the alarm under the buried point monitoring method. Summary of the invention

[0008] In view of this, an embodiment of the present invention provides a method and apparatus based on log alarm, which can realize real-time monitoring of the operating status of a production system and abnormal alarm without pre-embedding points in the system.

[0009] To achieve the above objective, according to one aspect of an embodiment of the present invention, a log-based alarm method is provided.

[0010] The log alarm method according to an embodiment of the present invention includes:

[0011] Establish a log data set related to the system;

[0012] Extracting an abnormal indication information group from the log data set;

[0013] Clustering the log data associated with the abnormal indication information group according to a specific clustering depth to obtain a clustering result; and

[0014] When the clustering result triggers a predetermined alarm threshold, an alarm message is sent to the terminal.

[0015] Optionally, the abnormality indication information group includes one or more of the following associated with the abnormality:

[0016] Time information, machine information, abnormal source information, text prompt information, location information and special ID information.

[0017] Optionally, the specific clustering depth is associated with an optimal generalization representation, and the optimal generalization representation is calculated by the following formula group, and wherein the optimal generalization representation is g corresponding to the minimum H(C) value:

[0018]

[0019] in,

[0020] C represents the alarm log set; a represents the alarm log in C; g is a generalized representation of C; D represents the degree of dissimilarity between two elements; A i Indicates the properties of the alarm log; and Dom is A i The value range of .

[0021] Optionally, the log data set related to the system is a real-time log stream during the operation of the system.

[0022] According to yet another aspect of an embodiment of the present invention, a log-based alarm device is provided.

[0023] The device based on log alarm according to an embodiment of the present invention includes:

[0024] A data set establishment module is used to establish a log data set related to the system;

[0025] An information extraction module, used to extract an abnormal indication information group from the log data set;

[0026] A clustering module, configured to cluster the log data associated with the abnormal indication information group according to a specific clustering depth to obtain a clustering result; and

[0027] The alarm module is used to send an alarm message to the terminal when the clustering result triggers a predetermined alarm threshold.

[0028] Optionally, the abnormality indication information group includes one or more of the following associated with the abnormality:

[0029] Time information, machine information, abnormal source information, text prompt information, location information and special ID information.

[0030] Optionally, the specific clustering depth is associated with an optimal generalization representation, and the optimal generalization representation is calculated by the following formula group, and wherein the optimal generalization representation is g corresponding to the minimum H(C) value:

[0031]

[0032] in,

[0033] C represents the alarm log set; a represents the alarm log in C; g is a generalized representation of C; D represents the degree of dissimilarity between two elements; A i Indicates the properties of the alarm log; and Dom is A i The value range of .

[0034] Optionally, the log data set related to the system is a real-time log stream during the operation of the system.

[0035] According to still another aspect of an embodiment of the present invention, an electronic device based on log alarm is provided.

[0036] The electronic device based on log alarm according to an embodiment of the present invention is characterized by comprising:

[0037] one or more processors;

[0038] a storage device for storing one or more programs,

[0039] When the one or more programs are executed by the one or more processors, the one or more processors implement the log alarm-based method provided in the first aspect of the embodiment of the present invention.

[0040] According to still another aspect of an embodiment of the present invention, a computer readable medium is provided.

[0041] According to the computer-readable medium of the embodiment of the present invention, a computer program is stored thereon, and it is characterized in that when the program is executed by a processor, the method based on log alarm provided by the first aspect of the embodiment of the present invention is implemented.

[0042] One embodiment of the above invention has the following advantages or beneficial effects: because the monitoring and alarm technology based on the system log is adopted, the technical problem of the intrusion of the buried point into the system is overcome, thereby achieving the technical effect of avoiding the introduction of errors due to the buried point. And the system can be monitored based on real-time log data, so that the problem can be handled promptly and quickly. In addition, the problem of the system is further intuitively displayed when the alarm is triggered by clustering, which is convenient for operation and maintenance personnel to quickly locate the problem.

[0043] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0045] Figure 1 is a schematic diagram of the main process of the method based on log alarm according to an embodiment of the present invention;

[0046] Figure 2 is an example diagram of an error log according to an embodiment of the present invention;

[0047] Figure 3 is a schematic diagram of a clustering effect according to an embodiment of the present invention;

[0048] Figure 4 is a schematic diagram of a clustering process according to an embodiment of the present invention;

[0049] Figure 5 is a schematic diagram of part of data of an actual test according to an embodiment of the present invention;

[0050] Figure 6 is a flow chart of another method based on log alarm according to an embodiment of the present invention;

[0051] Figure 7 is a schematic diagram of main modules of a device based on log alarm according to an embodiment of the present invention;

[0052] Figure 8 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0053] Fig. 9 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0055] Figure 1 is a schematic diagram of the main process of the method based on log alarm according to an embodiment of the present invention, such as Figure 1As shown, the log-based alarm method according to an embodiment of the present invention includes steps S101, S102, S103 and S104.

[0056] Step S101: Establishing a log data set related to the system.

[0057] Optionally, the log data set related to the system is a real-time log stream during the operation of the system.

[0058] In one embodiment, the step of collecting log stream information is mainly responsible for collecting log streams during system operation. Here we use Kafka (by redirecting log output, the system logs are sent to Kafka). Kafka is chosen mainly because Kafka has its own stream. In addition, Kafka performs well in the field of log collection, so Kafka is the only choice for log collection.

[0059] Usually, logs are an important tool for us to track system operation, because logs record the key execution points of business processes and stack information when exceptions occur. For online systems, in most cases we can only locate system problems through logs. In simple application systems (such systems usually have simple business logic and relatively few external dependencies), the probability of system problems is relatively small, and operation and maintenance personnel can quickly locate problems through logs. However, in complex distributed systems, because the magnitude of error logs will increase dramatically, this will cause the contents of error logs to bury each other and affect each other, making it impossible for operation and maintenance personnel to quickly locate the problem, and thus cannot quickly solve the problem. If a problem that damages the business is not handled in a timely manner, it will cause irreparable losses to the company.

[0060] Step S102: extracting an abnormal indication information group from the log data set.

[0061] Optionally, the abnormality indication information group includes one or more of the following associated with the abnormality: time information, machine information, abnormality source information, text prompt information, location information and special ID information.

[0062] In one embodiment, after establishing a system-related log data set, it is necessary to filter the log to extract key information. Because what we need to do is to analyze the log and then determine whether an alarm is needed, we are concerned about the error information in the log. The online system should be running correctly in most cases, so the log information collected in the first step is mostly not what we need. It is necessary to filter when extracting, and only filter out the error log (if only the error log is needed, the filtering should be done during the collection process. We did not filter during the collection process, mainly to archive and store the log for future retrieval). Because what is more helpful for us to troubleshoot problems is the abnormal information.

[0063] Usually, when troubleshooting online problems, we focus on the following key information: the time when the exception occurred, the machine where the exception occurred (computer room, environment), the source of the exception, the exception text prompt, the location of the fault (number of lines of code, interface, class), case-related special IDs (order number, product number, user ID, etc.), etc. Therefore, we will cluster these important information.

[0064] Key information extraction algorithm implementation:

[0065] Fault location: locate the fault by searching the exception stack. Usually, the first exception in the exception stack is the real cause of the system fault, so you can search for the first local code in the exception stack and use it as the fault location. If it does not exist, use the log print location.

[0066] Exception source: After obtaining the exception stack, parse the exception type and determine the exception source (for example, NullPointException is definitely caused by a parameter being null and no judgment was made when the parameter value was taken. RpcException is definitely caused by a remote procedure call failure).

[0067] Key content of alarm log text: First look for an exception stack. If so, look for the last exception (usually the real cause of the fault). If it cannot be obtained, look for an error message in the log in the form of "code=..., message=...". If it cannot be obtained, take the first exception description in the log exception stack.

[0068] Step S103: clustering the log data associated with the abnormal indication information group according to a specific clustering depth to obtain a clustering result.

[0069] Optionally, the specific clustering depth is associated with an optimal generalization representation, and the optimal generalization representation is calculated by the following formula group, and wherein the optimal generalization representation is g corresponding to the minimum H(C) value:

[0070]

[0071] in,

[0072] C represents the alarm log set; a represents the alarm log in C; g is a generalized representation of C; D represents the degree of dissimilarity between two elements; A i Indicates the properties of the alarm log; and Dom is A i The value range of .

[0073] In one embodiment, this paper adopts the root cause analysis algorithm proposed in the paper "Clustering Intrusion Detection Alarms to Support Root Cause Analysis [Klaus Julisch, 2002]". The main reason for selecting this algorithm is that the generalized alarm obtained by generalizing the error log through this algorithm can represent the main characteristics of the alarm cluster. Figure 2-Figure 4 Let's explain the clustering steps in detail. Figure 2 FIG. 1 is an example diagram of an error log according to an embodiment of the present invention. Figure 2 As shown in the figure, there are several alarm logs. From the figure, we can see various exceptions and errors. It is difficult for us to know what exactly affects the availability or performance of our system over a period of time. If we can Figure 1 By summarizing and abstracting complex data, complex information will be simplified. Figure 3 is a schematic diagram of a clustering effect according to an embodiment of the present invention. The effect we want to achieve is similar to Figure 3 shown.

[0074] We know that the higher the level of abstraction, the fewer details, but the wider the scope it can cover; conversely, the lower the level of abstraction, the more useless information there may be, and the smaller the scope it can cover. Therefore, we need to determine a reasonable abstract depth (we must start from the bottom of the details and abstract upwards) so that the alarm can contain the details we care about while avoiding being bombarded.

[0075] To determine the depth of alarm clustering, we define:

[0076] Attribute: A type of information that constitutes the alarm log, such as machine, environment, time, etc. i express.

[0077] Domain: Attribute A i The range of values ​​of, for any given A i , there must be a Dom(A i ).

[0078] Generalization Hierarchy: For any attribute A we are interested in i Each corresponds to an abstract height H i express.

[0079] Dissimilarity: defined as D(a1,a2). It takes two alarms a1 and a2 as input and returns a numerical value indicating how dissimilar the two alarms are. In contrast to similarity, when D(a1,a2) is small, it means that alarms a1 and a2 are similar. In order to calculate D(a1,a2), the height of abstraction needs to be specified.

[0080] To calculate D(a1,a2), we make the following assumptions: Let x1, x2 be some attribute A i If x1 and x2 have two different values, then the dissimilarity between them is: i The shortest path length connecting x1 and x2 through a common parent node p.

[0081] Right now For two alarms a1 and a2, the calculation method is as follows:

[0082]

[0083] Assume that C represents the alarm set and g is a generalized representation of C, then Take the alarm set {"Server 1 failed to call the product interface through JSF to obtain product information", "Server 1 failed to call the price interface through JSF to obtain price"} as an example, "Server 1 failed to call JSF to obtain product information" is a generalized expression, and "Server JSF call failed" is also a generalized expression. For a certain alarm cluster, we hope to obtain a generalized expression that can cover its set and has the most concrete expression. In order to solve this problem, the following two evaluation indicators (Formula Group 2) are defined:

[0084]

[0085] The g corresponding to the minimum value is the most suitable generalized representation we are looking for. Based on the above concepts, the alarm log clustering problem is defined as: for a given log set L, min_size is a preset constant, G i (i=1,2,3……n) is attribute A i The generalization height of L is to find a subset C of L that satisfies |C|>=min_size, and The value is the smallest. min_size is used to control the height of the abstraction. In the extreme case, if min_size is the same as the size of the L set, then we can only use the ultimate abstraction. If min_size = 1, each alarm log is its own abstraction. After finding a cluster, we can remove these elements and then find other clusters in the remaining set of L. Unfortunately, this is an NP-complete problem, so the paper proposes a heuristic algorithm that satisfies |C|>=min_size, so that The value should be as small as possible.

[0086] Exemplarily, the algorithm description may be as follows:

[0087] 1. The algorithm assumes that all generalization hierarchies G i They are all trees, so that each alarm cluster has a unique, top-level generalization result.

[0088] 2. Define L as a set of original alarm logs, and the algorithm selects an attribute A i ,

[0089] Set all the alarm A in L i The value is replaced by G i Middle A i The parent value of the alarm is continuously generalized through this operation.

[0090] 3. Continue step 2 until a generalized alarm is found that covers more alarms than min_size.

[0091] 4. Output the alarm found in step 3.

[0092] Exemplarily, the algorithm pseudo code is as follows:

[0093] Input: error log set L, min_size, generalization hierarchy of each attribute G1,......,Gn

[0094] Output: All generalized alarms that meet the conditions

[0095]

[0096] The logic here is: if there is an alarm a that satisfies a[count]>=min_size, then for all attributes Ai, Fi>=fi(a[Ai])>=min_size can be satisfied. Conversely, if there is an attribute Ai whose Fi value is less than min_size, then a[count] cannot be greater than min_size. Therefore, selecting the attribute Ai with the smallest Fi value for generalization helps to achieve the clustering condition as soon as possible.

[0097] Note: If a min_size that is too large is chosen, the algorithm will be forced to merge alarms with different root causes. On the other hand, if it is too small, clustering may end prematurely, and alarms with the same root cause may appear in different clusters. Therefore, set an initial value, which can be denoted as ms0. Define a smaller value ε (0<ε<1), and when the clustering results are the same when min_size is ms0, ms0*(1-ε), and ms0*(1+ε), we say that the clustering is ε-robust. If they are not the same, set ms1=ms0*(1-ε), and repeat this test until a robust minimum value is found.

[0098] Combine the following Figure 4 The clustering process provided by the embodiment of the present invention is briefly described. Figure 4 is a schematic diagram of a clustering process according to an embodiment of the present invention. Figure 4 As shown, a clustering process according to an embodiment of the present invention includes steps S401, S402, S403, S404, S405, S406, S407, S408, S409 and S410.

[0099] Step S401: Start.

[0100] Step S402: Filter out error logs from the log stream and extract key information.

[0101] Step S403: Load the generalization level set.

[0102] Step S404: Merge identical alarms.

[0103] Step S405: Determine whether clustering needs to be continued.

[0104] Step S406: When the result of step S405 is "yes", find out the attribute with the smallest generalization degree.

[0105] Step S407: Perform replacement and abstraction.

[0106] Step S408: Merge alarms.

[0107] Step S409: When the result of step S405 is "No", output the clustering result.

[0108] Step S410: End.

[0109] Step S104: When the clustering result triggers a predetermined alarm threshold, an alarm message is sent to the terminal.

[0110] In one embodiment, the operation of this step is to determine whether the output triggers the alarm threshold based on the output of the third step in combination with the alarm configuration, and if the alarm threshold is triggered, an alarm is triggered.

[0111] Combine the following Figure 5 The following is a brief description of a test result of this method. This result is only partial data from a single test and does not limit the effectiveness of this method. Figure 5 FIG. 1 is a schematic diagram of some data of an actual test according to an embodiment of the present invention. Figure 5 As shown in the figure, the inventor collected data for testing the system in a pre-release environment. Overall, the algorithm works well, but there is room for optimization. With the development of data mining and machine learning, there will definitely be more efficient clustering algorithms.

[0112] The clustering alarm algorithm based on real-time log stream proposed in this paper has many advantages, but there are still many areas for improvement, mainly reflected in: 1. In-depth analysis of log content. We can improve the accuracy of log content analysis and improve the generalization effect by using text classification, text feature vector similarity, etc. 2. This paper only discusses the clustering algorithm that performs well when processing system error logs. For various types of alarms in the system, other clustering algorithms (such as K-Means) can also be used in the future to jointly process alarms and optimize the aggregation effect. 3. Adaptive alarm threshold, dynamically manage alarm thresholds, improve the quality and timeliness of alarms, and reduce the number of false alarms and missed alarms.

[0113] The present invention aims to realize real-time monitoring and abnormal alarm of the operation status of the production system by performing aggregation analysis on the log stream of the production system. The monitoring and alarm method based on the real-time log stream has the following advantages compared with the existing monitoring and alarm system: 1. The monitoring and alarm system based on the real-time log stream does not need to pre-bury points in the system, because we monitor the system operation status through the log of the production system, so we do not need to bury points in advance (the online system will definitely print key logs), so it is not invasive. 2. Because there is no intrusion, the possibility of introducing bugs due to burying points can be avoided. 3. It intuitively reflects the problem areas of the system, which is convenient for operation and maintenance personnel to quickly locate problems (the alarm naturally contains the exception stack). 4. Because we are based on quasi-real-time data, our alarms are also quasi-real-time.

[0114] One embodiment of the above invention has the following advantages or beneficial effects: because the monitoring and alarm technology based on system logs is adopted, the technical problem of the intrusion of the buried points into the system is overcome, thereby achieving the technical effect of avoiding the introduction of errors due to the buried points. And the system can be monitored based on real-time log data, so that problems can be handled promptly and quickly. In addition, the problem of the system is further displayed intuitively through clustering when an alarm is triggered, which is convenient for operation and maintenance personnel to quickly locate the problem

[0115] Figure 6is a flow chart of another method based on log alarm according to an embodiment of the present invention, such as Figure 6 As shown, another method based on log alarm according to an embodiment of the present invention includes steps S601, S602, S603 and S604.

[0116] Step S601: establishing a log data set related to the system, wherein the log data set includes a real-time log stream during the operation of the system.

[0117] Step S602: extracting an abnormal indication information group from the log data set.

[0118] Optionally, the abnormality indication information group includes one or more of the following associated with the abnormality:

[0119] Time information, machine information, abnormal source information, text prompt information, location information and special ID information.

[0120] Step S603: clustering the log data associated with the abnormal indication information group according to a specific clustering depth to obtain a clustering result.

[0121] Optionally, the specific clustering depth is associated with an optimal generalization representation, and the optimal generalization representation is calculated by the following formula group, and wherein the optimal generalization representation is g corresponding to the minimum H(C) value:

[0122]

[0123] in,

[0124] C represents the alarm log set; a represents the alarm log in C; g is a generalized representation of C; D represents the degree of dissimilarity between two elements; A i Indicates the properties of the alarm log; and Dom is A i The value range of .

[0125] Step S604: When the clustering result triggers a predetermined alarm threshold, an alarm message is sent to the terminal.

[0126] One embodiment of the above invention has the following advantages or beneficial effects: because the monitoring and alarm technology based on the system log is adopted, the technical problem of the intrusion of the buried point into the system is overcome, thereby achieving the technical effect of avoiding the introduction of errors due to the buried point. And the system can be monitored based on real-time log data, so that the problem can be handled promptly and quickly. In addition, the problem of the system is further intuitively displayed when the alarm is triggered by clustering, which is convenient for operation and maintenance personnel to quickly locate the problem.

[0127] Figure 7is a schematic diagram of the main modules of the device based on log alarm according to an embodiment of the present invention, such as Figure 7 As shown, the device 700 based on log alarm according to an embodiment of the present invention includes:

[0128] The data set establishment module 701 is used to establish a log data set related to the system.

[0129] The information extraction module 702 is used to extract an abnormality indication information group from the log data set.

[0130] The clustering module 703 is used to cluster the log data associated with the abnormal indication information group according to a specific clustering depth to obtain a clustering result.

[0131] The alarm module 704 is used to send an alarm message to the terminal when the clustering result triggers a predetermined alarm threshold.

[0132] Optionally, the log data set related to the system is a real-time log stream during the operation of the system.

[0133] Optionally, the abnormality indication information group includes one or more of the following associated with the abnormality:

[0134] Time information, machine information, abnormal source information, text prompt information, location information and special ID information.

[0135] Optionally, the specific clustering depth is associated with an optimal generalization representation, and the optimal generalization representation is calculated by the following formula group, and wherein the optimal generalization representation is g corresponding to the minimum H(C) value:

[0136]

[0137] in,

[0138] C represents the alarm log set; a represents the alarm log in C; g is a generalized representation of C; D represents the degree of dissimilarity between two elements; A i Indicates the properties of the alarm log; and Dom is A i The value range of .

[0139] One embodiment of the above invention has the following advantages or beneficial effects: because the monitoring and alarm technology based on the system log is adopted, the technical problem of the intrusion of the buried point into the system is overcome, thereby achieving the technical effect of avoiding the introduction of errors due to the buried point. And the system can be monitored based on real-time log data, so that the problem can be handled promptly and quickly. In addition, the problem of the system is further intuitively displayed when the alarm is triggered by clustering, which is convenient for operation and maintenance personnel to quickly locate the problem.

[0140] Figure 8 An exemplary system architecture 800 is shown to which the log-based alarm method or log-based alarm device according to the embodiment of the present invention can be applied.

[0141] like Figure 8 As shown, system architecture 800 may include terminal devices 801, 802, 803, a network 804, and a server 805. Network 804 is used to provide a medium for communication links between terminal devices 801, 802, 803 and server 805. Network 804 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0142] Users can use terminal devices 801, 802, and 803 to interact with server 805 through network 804 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 801, 802, and 803, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

[0143] The terminal devices 801 , 802 , and 803 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0144] The server 805 may be a server that provides various services, such as a backend management server (only an example) that provides support for shopping websites browsed by users using the terminal devices 801, 802, and 803. The backend management server may analyze and process the received data such as product information query requests, and feed back the processing results (such as target push information, product information - only an example) to the terminal device.

[0145] It should be noted that the log-based alarm method provided in the embodiment of the present invention is generally executed by the server 805 , and accordingly, the log-based alarm device is generally set in the server 805 .

[0146] It should be understood that Figure 8 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0147] Reference below Fig. 9 , which shows a schematic diagram of the structure of a computer system 900 of a terminal device suitable for implementing an embodiment of the present invention. Fig. 9 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0148] like Fig. 9As shown, the computer system 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage part 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the system 900 are also stored. The CPU 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0149] The following components are connected to the I / O interface 905: an input section 906 including a keyboard, a mouse, etc.; an output section 907 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, a modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 910 as needed, so that a computer program read therefrom is installed into the storage section 908 as needed.

[0150] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 909, and / or installed from the removable medium 911. When the computer program is executed by the central processing unit (CPU) 901, the above-mentioned functions defined in the system of the present invention are executed.

[0151] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0152] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0153] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor, for example, it may be described as: a processor includes a data set establishment module, an information extraction module, a clustering module and an alarm module. The names of these modules do not constitute a limitation on the modules themselves in some cases, for example, the data set establishment module may also be described as a "module for establishing a log data set related to the system".

[0154] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: establishing a log data set related to the system; extracting an abnormal indication information group from the log data set; clustering the log data associated with the abnormal indication information group according to a specific clustering depth to obtain a clustering result; and when the clustering result triggers a predetermined alarm threshold, sending an alarm message to the terminal.

[0155] One embodiment of the above invention has the following advantages or beneficial effects: because the monitoring and alarm technology based on the system log is adopted, the technical problem of the intrusion of the buried point into the system is overcome, thereby achieving the technical effect of avoiding the introduction of errors due to the buried point. And the system can be monitored based on real-time log data, so that the problem can be handled promptly and quickly. In addition, the problem of the system is further intuitively displayed when the alarm is triggered by clustering, which is convenient for operation and maintenance personnel to quickly locate the problem.

[0156] Finally, a brief explanation of the terms involved in this application:

[0157] Microservices: Microservice Architecture is an architectural concept that aims to decouple solutions by breaking down functionality into discrete services. You can think of it as applying many SOLID principles at the architectural level rather than at the class that gets the service. Microservice architecture is an interesting concept, and its main purpose is to decompose functionality into discrete services, thereby reducing the coupling of the system and providing more flexible service support.

[0158] SOA: Service-Oriented Architecture (SOA) is a component model that splits an application into different functional units (called services) and connects them through well-defined interfaces and contracts between these services.

[0159] Buried data: It is a common data collection method for website analysis

[0160] Monitoring and alarm system: monitors the running status of servers and services, and notifies relevant managers if any abnormality occurs in the status indicators of concern.

[0161] Configuration Center: An online repository for storing and accessing project configuration files. It usually has the ability to actively push configurations, and will push the latest configurations to related systems when the configuration changes.

[0162] Chain of Responsibility Pattern: A behavioral design pattern in which each receiver typically contains a reference to another receiver. If an object cannot handle the request, it passes the same request to the next receiver, and so on. Often used to decouple the sender and receiver of a request.

[0163] Design Patterns: Design patterns are solutions to common problems faced by software developers during software development. These solutions are developed by many software developers through trial and error over a long period of time.

[0164] RPC: RPC (Remote Procedure Call) - Remote Procedure Call, it is a protocol for requesting services from a remote computer program over the network without having to understand the underlying network technology. The RPC protocol assumes the existence of certain transport protocols, such as TCP or UDP, to carry information data between communicating programs. In the OSI network communication model, RPC spans the transport layer and the application layer. RPC makes it easier to develop applications including network distributed multi-programs. Common rpc frameworks include dubbo, Motan, gRpc, jsf, hsf, etc.

[0165] Interface: An interface generally refers to an abstract object (which can be another entity) that an entity provides to the outside world, which is used to separate external communication methods from internal operations so that it can be modified internally without affecting the way other external entities interact with it.

[0166] Code intrusion: If tool A is intrusive and tool A is used in the project, then if you do not want to use tool A anymore (replace it with another tool), you must modify the project code. On the contrary, if tool A is non-intrusive, and the project does not want to use tool A anymore but tool C, the project code does not need to be modified, only the configuration file or something.

[0167] Jar package: Java compressed file format, similar to the zip format.

[0168] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method based on log alarm, characterized in that: include: Establish system-related log data sets through log output redirection; Extracting the exception indication information group from the log data set; the specific method of extracting the exception indication information group is: by finding the first local code of the exception stack, taking the first local code as the fault location; after obtaining the exception stack, parsing the exception type and determining the exception source; preferentially finding whether there is an exception stack, and if so, finding the last exception; If it cannot be obtained, check whether there is an error message in the log; If it cannot be obtained, the first exception description in the log exception stack is obtained; Clustering the log data associated with the abnormal indication information group according to a specific clustering depth to obtain a clustering result; as well as When the clustering result triggers a predetermined alarm threshold, sending an alarm message to the terminal; The abnormality indication information group includes one or more of the following associated with the abnormality: Time information, machine information, abnormal source information, text prompt information, location information and special ID information.

2. The method according to claim 1, characterized in that in, The specific clustering depth is associated with an optimal generalization representation, which is calculated by the following set of formulas, and wherein the optimal generalization representation is g corresponding to the minimum H(C) value: in, C represents the alarm log set; a represents the alarm log in C; g is a generalized representation of C; D represents the degree of dissimilarity between two elements; A i Indicates the properties of the alarm log; and Dom is A i The value range of .

3. The method according to claim 1, characterized in that in, The log data set related to the system is a real-time log stream during the operation of the system.

4. A device based on log alarm, characterized in that: include: The data set establishment module is used to establish the log data set related to the system through log output redirection; The information extraction module is used to extract the exception indication information group from the log data set; the specific method of extracting the exception indication information group is: by finding the first local code of the exception stack, the first local code is used as the fault location; after obtaining the exception stack, the exception type is analyzed to determine the exception source; first find out whether there is an exception stack, if there is, find the last exception; If it cannot be obtained, check whether there is an error message in the log; If it cannot be obtained, the first exception description in the log exception stack is obtained; A clustering module, used for clustering the log data associated with the abnormal indication information group according to a specific clustering depth to obtain a clustering result; as well as An alarm module, used for sending an alarm message to a terminal when the clustering result triggers a predetermined alarm threshold; The abnormality indication information group includes one or more of the following associated with the abnormality: Time information, machine information, abnormal source information, text prompt information, location information and special ID information.

5. The device according to claim 4, characterized in that in, The specific clustering depth is associated with an optimal generalization representation, which is calculated by the following set of formulas, and wherein the optimal generalization representation is g corresponding to the minimum H(C) value: in, C represents the alarm log set; a represents the alarm log in C; g is a generalized representation of C; D represents the degree of dissimilarity between two elements; A i Indicates the properties of the alarm log; and Dom is A i The value range of .

6. The device according to claim 4, characterized in that in, The log data set related to the system is a real-time log stream during the operation of the system.

7. An electronic device based on log alarm, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 3.

8. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images