Log detection method, device, computer storage medium and equipment

Abstract

The application provides a log detection method, device, computer storage medium and equipment, the method comprises the following steps: receiving the original log stream of the first service sent by a terminal device; the original log stream is reordered according to the historical word frequency, and an ordered log stream is generated; the logs in the ordered log stream are matched with log templates respectively, and a novel log stream is determined, the log template is generated after a data mining model is updated using a historical ordered log stream; the logs in the novel log stream within a first preset time window are aggregated and converged to generate a novel log event. Through the log detection method, novel logs can be found in time and the log checking range can be narrowed when a system fault occurs.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a log detection method, apparatus, computer storage medium, and device. Background Technology

[0002] In recent years, the operations and maintenance (O&M) field has begun to evolve from Automated Operations (DevOps) to Intelligent Intelligence for IT Operations (AIOps). AIOps leverages Artificial Intelligence (AI) and big data technologies to predict and discover potential system failures in a timely manner and intelligently analyze the causes of system failures. Since logs provide the most direct and detailed reflection of system operation, they can be used as a basis for analyzing the causes of system failures.

[0003] In existing technologies, when analyzing system failures based on logs, error logs are mainly detected from aspects such as abnormal log content and abnormal log order. Regarding abnormal content, there are two methods for detecting error logs. The first method can detect error logs based on user-configured error rules. The second method can pre-label an error log dataset, then use the error log dataset to train a machine learning algorithm model, and finally use the trained machine learning algorithm model for error log detection.

[0004] However, both of the above detection methods, whether using user-configured error rules or pre-labeled error log datasets, are based on specific, previously occurring historical error logs. Therefore, existing log detection methods cannot discover novel error logs that have not yet appeared. Summary of the Invention

[0005] This application provides a log detection method, apparatus, computer storage medium, and device to solve the technical problem that existing technologies cannot detect novel error logs that have never appeared before.

[0006] Firstly, this application provides a log detection method, the method comprising:

[0007] Receive the raw log stream of the first service sent by the terminal device;

[0008] The logs in the ordered log stream are matched with the log template to determine the novel log stream. The original log stream is then reordered based on historical word frequency to generate an ordered log stream.

[0009] The log template is generated after updating the data mining model using a historical ordered log stream;

[0010] The logs in the novel log stream within the first preset time window are aggregated and converged to generate novel log events.

[0011] In one optional implementation, the step of matching each log in the ordered log stream with a log template to determine the novel log stream includes:

[0012] The logs in the ordered log stream are matched against the log template to identify the logs that fail to match.

[0013] The logs that failed to match were deduplicated using a Bloom filter to generate the novel log stream.

[0014] In one optional implementation, matching each log entry in the ordered log stream with the log template includes:

[0015] Match the log sequence in the ordered log stream with the path of the log template.

[0016] In one alternative implementation, after determining the novel log stream, the method further includes:

[0017] The historical ordered log stream is updated into the data mining model to generate an updated data mining model;

[0018] The updated data mining model is trimmed to generate the log template.

[0019] In one alternative implementation, the data mining model includes a frequent pattern tree model.

[0020] In one optional implementation, updating the data mining model with the historical ordered log stream to generate the updated data mining model includes:

[0021] Based on the order of the log word sequence in the historical ordered log stream, log words from the root node of the frequent pattern tree model are inserted one by one until the termination node of the frequent pattern tree model.

[0022] The termination node of the frequent pattern tree model is the node whose number of child nodes after the target layer is greater than the first threshold.

[0023] In one optional implementation, the step of reordering the original log stream based on historical word frequencies to generate an ordered log stream includes:

[0024] Obtain the historical word frequency of each word in the log word sequence of the original log stream;

[0025] The original log stream is reordered based on the historical word frequency of each word.

[0026] In one optional implementation, reordering the original log stream based on the historical word frequency of each word includes:

[0027] If the frequency difference between adjacent words is less than the second threshold, then the historical frequency of the word that comes later is replaced with the historical frequency of the word that comes earlier.

[0028] In one optional implementation, reordering the original log stream based on the historical word frequency of each word includes:

[0029] If two words have the same historical frequency, the logs corresponding to the two words are sorted according to the dictionary order.

[0030] In one optional implementation, before generating the ordered log stream, the method further includes:

[0031] Filter the target characters of the logs in the original log stream to form at least one word segmentation stream;

[0032] The historical word frequencies are updated based on the word frequencies in at least one word segmentation stream within the second preset time window.

[0033] In one alternative implementation, the target characters include at least one of the following: punctuation marks, numbers, and predefined words.

[0034] In one alternative implementation, after generating the novel log event, the method further includes:

[0035] Send a warning message to the terminal device to be warned, the warning message being used to warn of the novel log event.

[0036] Secondly, this application provides a log detection device, the device comprising:

[0037] The receiving module is used to receive the raw log stream of the first service sent by the terminal device;

[0038] The reordering module is used to reorder the original log stream according to historical word frequencies to generate an ordered log stream;

[0039] The matching module is used to match the logs in the ordered log stream with the log template to determine the novel log stream. The log template is generated after updating the data mining model using the historical ordered log stream.

[0040] The generation module is used to aggregate and converge the logs in the novel log stream within the first preset time window to generate novel log events.

[0041] In one optional implementation, the matching module is specifically used to match the logs in the ordered log stream with the log template respectively, identify the logs that fail to match, and use a Bloom filter to deduplicate the logs that fail to match, thereby generating the novel log stream.

[0042] In one optional implementation, the matching module is specifically used to match the log sequence in the ordered log stream with the path of the log template.

[0043] In one optional implementation, the matching module is further configured to update the data mining model with the historical ordered log stream to generate an updated data mining model; and to trim the updated data mining model to generate the log template.

[0044] In one alternative implementation, the data mining model includes a frequent pattern tree model.

[0045] In one optional implementation, the matching module is further configured to insert log words from the root node of the frequent pattern tree model one by one according to the order of the log word sequence of the historical ordered log stream, until the termination node of the frequent pattern tree model; wherein, the termination node of the frequent pattern tree model is a node whose number of child nodes after the target layer is greater than a first threshold.

[0046] In one optional implementation, the reordering module is specifically used to obtain the historical word frequency of each word in the log word sequence of the original log stream; and to reorder the original log stream according to the magnitude of the historical word frequency of each word.

[0047] In one optional implementation, the reordering module is specifically used to replace the historical frequency of the word that is arranged later with the historical frequency of the word that is arranged earlier if the word frequency difference between adjacent words is less than a second threshold.

[0048] In one optional implementation, the reordering module is specifically used to sort the logs corresponding to the two words according to the dictionary order if the two words have the same historical frequency.

[0049] In one optional implementation, the reordering module is further configured to filter target characters in the logs of the original log stream to form at least one segmented stream; and update the historical word frequency according to the word frequency in the at least one segmented stream within a second preset time window.

[0050] In one alternative implementation, the target characters include at least one of the following: punctuation marks, numbers, and predefined words.

[0051] In one optional implementation, the generation module is further configured to send warning information to the terminal device to be warned, the warning information being used to warn of the novel log event.

[0052] Thirdly, the present invention also provides a computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the method steps of any of the first aspects.

[0053] Fourthly, this application also provides an electronic device, comprising: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and executed as described in any of the first aspects.

[0054] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the method described in any one of the first aspects.

[0055] The log detection method, apparatus, computer storage medium, and device provided in this application first receive the raw log stream of a first service sent by a terminal device. Then, the raw log stream is reordered based on historical word frequencies to generate an ordered log stream. Logs in the ordered log stream are matched against a log template to identify novel log streams. The log template is generated after updating a data mining model using historical ordered log streams. Finally, the logs in the novel log stream within a first preset time window are aggregated and converged to generate novel log events. Through this method, by matching and aggregating the logs in the ordered log stream against the log template in real time after receiving the raw log stream generated by the system, novel logs are promptly detected, narrowing the log scope for troubleshooting. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0057] Figure 1 This is a schematic diagram illustrating an application scenario of a log detection method provided in an embodiment of this application;

[0058] Figure 2 A flowchart illustrating a log detection method provided in an embodiment of this application;

[0059] Figure 3 A flowchart illustrating a log template generation method provided in an embodiment of this application;

[0060] Figure 4 A flowchart illustrating an ordered log stream generation method provided in this application embodiment;

[0061] Figure 5 This is a schematic diagram of the structure of a log detection device provided in an embodiment of this application;

[0062] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0064] In recent years, the operations and maintenance (O&M) field has begun to evolve from DevOps to AIOps. AIOps applies artificial intelligence to the O&M field, and can use machine learning to transform real-world problems into algorithmic problems based on existing O&M data (such as logs, monitoring information, application information, etc.), thereby further solving problems that cannot be solved in automated O&M.

[0065] Logs are the most direct and detailed data reflecting system operation. Typically, after a failure occurs, operations and maintenance personnel check the logs in the system using keywords to analyze the cause of the system failure. In existing technologies, system failure analysis based on logs mainly detects errors in logs by examining abnormal log content and log order.

[0066] Currently, there are two methods for error log detection based on log content anomalies. The first method detects error logs based on user-configured error rules. The second method pre-labels the error log dataset, trains a machine learning algorithm model using this dataset, and then uses the trained model for error log detection. Machine learning algorithms include classification algorithms in supervised learning and deep learning algorithms, as well as clustering algorithms in unsupervised learning.

[0067] However, both of the above detection methods, whether using user-configured error rules or pre-labeled error log datasets, are based on specific, previously occurring historical error logs. Therefore, existing log detection methods cannot discover novel error logs that have not yet appeared.

[0068] In view of the above problems, this application provides a log detection method to comprehensively, accurately, and promptly discover novel error logs that have not appeared in the system before. In this application embodiment, the server receives the raw log stream sent by the terminal device and reorders it to generate an ordered log stream. The logs in the ordered log stream are matched with log templates to determine novel log streams. The logs in the novel log streams are then aggregated and converged to generate novel log events. This allows for the timely detection of novel logs in the event of a system failure, narrowing down the log scope for troubleshooting.

[0069] The application scenarios of the log detection method involved in this application are described below.

[0070] Figure 1 This is a schematic diagram illustrating an application scenario of a log detection method provided in an embodiment of this application. For example... Figure 1 As shown, the system includes a terminal device 101, a server 102, and a terminal device 103 awaiting alerts. When the terminal device 101 is running, it can generate a raw log stream. After receiving the raw log stream generated by the terminal device 101, the server 102 can process the raw log stream to obtain novel log events. The server 102 can then send these novel log events to the terminal device 103 awaiting alerts.

[0071] The aforementioned terminal devices can be mobile phones, tablets, computers with wireless transceiver capabilities, virtual reality (VR) terminal devices, augmented reality (AR) terminal devices, wireless terminals in self-driving vehicles, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in smart homes, etc. In this application embodiment, the device used to implement the terminal's functions can be the terminal itself, or a device capable of supporting the terminal in implementing those functions, such as a chip system, which can be installed in the terminal. In this application embodiment, the chip system can consist of chips, or it can include chips and other discrete components.

[0072] A server can be, but is not limited to, a single web server, a group of multiple web servers, or a cloud based on cloud computing consisting of a large number of computers or web servers.

[0073] In this application, the terminal device to be warned is not limited. In some embodiments, the terminal device to be warned may be the terminal device that sends the original log stream, and in other embodiments, it may be any other terminal device.

[0074] It should be understood that the application scenarios of the technical solution in this application can be... Figure 1 This can be applied to log detection scenarios, but not limited to them; it can also be applied to other scenarios that require log detection.

[0075] It is understood that the above log detection method can be implemented by the log detection device provided in the embodiments of this application. The log detection device can be part or all of a device, such as a server.

[0076] The following uses a processor with integrated or installed relevant execution code as an example to provide a detailed description of the technical solutions of the embodiments of this application. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0077] Figure 2 This is a flowchart illustrating a log detection method provided in an embodiment of this application. This embodiment relates to the process of detecting novel logs in a raw log stream. Figure 2 As shown, the method includes:

[0078] S201, Receive the raw log stream of the first service sent by the terminal device.

[0079] In this embodiment of the application, after the terminal device generates the original log stream, it can send the original log stream of the first service to the server to request the detection of novel logs in the original log stream.

[0080] In this embodiment, the raw log stream is a collection of logs from various service units generated by the terminal device during operation. The first service can be any service unit. The terminal device can send the raw log stream of one service unit to the server, or it can send the raw log streams of multiple service units. This embodiment does not impose any restrictions on this.

[0081] It should be noted that the embodiments of this application do not limit the source of the original log stream. In some embodiments, the server can obtain the original log stream from any terminal device. In other embodiments, the server can also obtain the original log stream from any device, system, service program, etc. that can generate logs.

[0082] S202. Reorder the original log stream according to historical word frequency to generate an ordered log stream.

[0083] In this step, after the server obtains the raw log stream of the first service generated by the terminal device, it can reorder the raw log stream according to historical word frequency to generate an ordered log stream.

[0084] It should be understood that the embodiments of this application do not limit how the original log stream is reordered. In some embodiments, the server may first obtain the historical word frequency of each word in the original log stream, and then reorder the words in each log stream from largest to smallest according to the historical word frequency of each word.

[0085] In this embodiment, after the server reorders the original log stream based on the historical word frequencies, the method further includes: if the frequency difference between adjacent words is less than a second threshold, then the historical frequency of the later-ranked word is replaced with the historical frequency of the earlier-ranked word. The second threshold is a manually set value, which the server can use to determine whether the frequency difference between adjacent words meets the replacement condition. This embodiment does not limit the specific value of the second threshold; for example, the second threshold can be P% of the highest frequency among the adjacent words.

[0086] In other embodiments, the method further includes: if two words have the same historical frequency, the server sorts the logs corresponding to the two words according to the order of the dictionary.

[0087] This application does not limit the computational model for reordering the original log stream to generate an ordered log stream in the embodiments. In some embodiments, a real-time streaming computing framework (Flink streaming computing framework) can be used to convert the original log stream into an ordered log stream.

[0088] In this application, by reordering the words in the original log stream, the server can avoid misjudging novel logs due to changes in word frequency during log detection, ensuring that the log template format is entirely determined by word frequency order. For example, every log entry "RealTimeCdrForMessage Begin" is followed by a log entry "RealTimeCdrForMessage End," with the words "Begin" and "End" having the same overall word frequency. However, if only one log entry is included within a certain time window, the statistically obtained word frequencies of "Begin" and "End" will differ, and the word reordering order in the original log stream containing either "Begin" or "End" will change, potentially leading to misjudgment as a novel log. The reordering of words in the original log stream avoids this problem during log detection.

[0089] S203. Match the logs in the ordered log stream with the log template to determine the novel log stream. The log template is generated after updating the data mining model using the historical ordered log stream.

[0090] In this step, after the server reorders the original log stream to generate an ordered log stream, the logs in the ordered log stream can be matched with the log template to determine the novel log stream.

[0091] It should be understood that the embodiments of this application do not limit the type of log template. In some embodiments, a log entry includes two parts: fixed content and variable content. The fixed content part serves as the unique identifier of the log entry and is called the log template. For example, in the log entry "2020-10-15 10:05:44.049 INFO client request 4 times", "2020-10-15 10:05:44.049 INFO" and the number "4" are the variable content part; "client request * times" is the fixed content part, and this is referred to as the log template of the above log entry.

[0092] In this embodiment, the data mining model includes a frequent pattern tree model. The frequent pattern tree model is a tree data structure in the art. Each node in the frequent pattern tree model stores a word, and all words on the path from the root node to the leaf node form a log template. The path structure of the frequent pattern tree model is used as the unique identifier of the log template. This embodiment does not limit the structure of the frequent pattern tree model used to store the log template; in some embodiments, an extended prefix tree structure (FT-Tree) can be used to store the log template.

[0093] It should be understood that the embodiments of this application do not limit how the logs in the ordered log stream are matched with the log template to determine the novel log stream. In some embodiments, the server may first identify the logs that fail to match by matching the logs in the ordered log stream with the log template, and then use a Bloom filter to deduplicate the logs that fail to match, thereby generating the novel log stream. The novel log stream is a collection of logs that have never appeared in history.

[0094] In some embodiments, the server matches logs in the ordered log stream with a log template, including: matching the word sequence of the logs in the ordered log stream with the path of the log template. For example, if the word sequence of a log completely matches the path of a frequent pattern tree, it is not a novel log; if the word sequence of a log does not completely match the path of a frequent pattern tree, and the matching length is less than L, it is a novel log; if the word sequence of a log does not completely match the path of a frequent pattern tree, and the matching length is greater than or equal to L, it is not a novel log.

[0095] In this embodiment, a Bloom filter is a long binary vector and a series of random mapping functions used to retrieve whether an element is in a set. Through the deduplication process of the Bloom filter, the server can avoid misclassification of novel logs due to changes in word order during log detection. For example, when a large number of logs containing a certain word X occur within a certain period, the frequency of word X will change, and the rearrangement order of words in the original log stream containing word X will also change, which may lead to misclassification as novel logs. The deduplication process of the Bloom filter can avoid such problems during log detection.

[0096] S204. Aggregate and converge the logs in the novel log stream within the first preset time window to generate novel log events.

[0097] In this step, after the server matches the logs in the ordered log stream with the log template to determine the novel log stream, it can aggregate and converge the logs in the novel log stream within the first preset time window to generate novel log events.

[0098] The first preset time window is a predetermined time period during which the server can aggregate and converge logs from the novel log stream. This embodiment does not limit the length of this time period and can be set according to actual conditions. In some embodiments, the length of the first preset time window can be 10 minutes, while in other embodiments, it can be 30 minutes.

[0099] It should be understood that the embodiments of this application do not limit how to aggregate and converge the logs in the novel log stream within the first preset time window. In some embodiments, the server can perform one-hot encoding on the word sequences of two logs in the novel log stream, calculate the distance between the two logs using cosine distance, and use a density-based spatial clustering of applications with noise (DBSCAN) algorithm to aggregate and converge the two logs.

[0100] One-hot encoding primarily uses 0s and 1s to represent parameters, employing an N-bit state register to encode N states. Each state has an independent register bit, and only one bit is active at any given time. The cosine distance, also known as cosine similarity, uses the cosine of the angle between two vectors in vector space as a measure of the difference between two individuals.

[0101] For example, the following demonstrates how to calculate the distance between two logs. Table 1 shows the word sequence of two logs in a novel log stream provided in an embodiment of this application.

[0102] Table 1

[0103] Log sequence number Word sequence 1 INFO,RealTimeCdrForMessage,Begin,to,File 2 INFO, RealTimeCdrForMessage, End, write, File, Success

[0104] First, the server obtains the word set of the two logs and performs one-hot encoding. For example, {0=INFO, 1=RealTimeCdrForMessage, 2=Begin, 3=to, 4=File, 5=End, 6=write, 7=Success}. The numbers 0 and 1 are used to indicate whether there is a word at the encoding position in each log. The two logs are encoded as 1111100 and 1100111 respectively. Then, the server converts the one-hot encoding into vectors (1, 1, 1, 1, 1, 0, 0) and (1, 1, 0, 0, 1, 1, 1), and uses the cosine formula to calculate the distance between the two logs.

[0105] It should be understood that in the embodiments of this application, DBSCAN is a density-based clustering algorithm. Such density-based clustering algorithms generally assume that categories can be determined by the density of sample distribution, assuming that samples of the same category are closely connected, and that any sample in that category must have samples of the same category around it. In some embodiments, using the DBSCAN clustering algorithm, the server can group closely connected samples into one category, obtaining a cluster category. By dividing all closely connected samples into different categories, all clustering category results can be obtained.

[0106] In some embodiments, the server aggregates and converges the logs in the novel log stream within a first preset time window to generate novel log events, and also sends warning information to the terminal devices to be warned.

[0107] In practice, some error logs, such as user-input error logs, a small number of database connection timeouts, and distributed cluster communication timeouts, are not considered faults. In this application, by sending alert information to the terminal devices to be alerted, developers or operations personnel can manually configure error log matching rules to filter out faulty error logs from the novel log events. This method effectively solves the problem of numerous false positives in error log detection by machine learning algorithms, reducing the workload of developers and operations personnel.

[0108] The log detection method provided in this application involves the server first receiving the raw log stream of a first service sent by a terminal device. Then, the server reorders the raw log stream based on historical word frequencies to generate an ordered log stream. The logs in the ordered log stream are matched against a log template to determine novel log streams. The log template is generated after updating the data mining model using historical ordered log streams. Finally, the logs in the novel log stream within a first preset time window are aggregated and converged to generate novel log events. This method can promptly detect novel logs and narrow down the log scope for system fault investigation, effectively solving the problems of late system fault detection and long troubleshooting cycles in scenarios such as missing manual configuration monitoring, abnormal manual operation, and new version releases.

[0109] Based on the above embodiments, the following explains how to generate log templates after updating the data mining model using historical ordered log streams. Figure 3 This is a flowchart illustrating a log template generation method provided in an embodiment of this application, as shown below. Figure 3 As shown, the method includes:

[0110] S301. Update the historical ordered log stream into the data mining model to generate the updated data mining model.

[0111] In this step, once the server identifies the novel log stream, it can update the data mining model with the historical ordered log stream, generating an updated data mining model. The historical ordered log stream refers to the ordered log stream portion after the novel log stream has been identified and removed.

[0112] It should be understood that the embodiments of this application do not limit the method of updating the data mining model. In some embodiments, the server can sequentially insert log words from the root node of the frequent pattern tree model one by one according to the word sequence of the logs in the historical ordered log stream, until the termination node of the frequent pattern tree model.

[0113] In this embodiment, the termination node of the frequent pattern tree model is a node whose number of child nodes after reaching the target layer exceeds a first threshold. The first threshold is the maximum number of child nodes a node in the frequent pattern tree model can have; child nodes with a number exceeding the first threshold are not considered templates but variables. It should be understood that this embodiment does not impose a limit on the value of the first threshold and can be set according to actual circumstances.

[0114] S302. Trim the updated data mining model to generate a log template.

[0115] In this step, after the server generates the updated data mining model, the updated data mining model can be trimmed to generate a log template.

[0116] It should be understood that the embodiments of this application do not limit how the updated data mining model is pruned. In some embodiments, when the number of matches between the log word sequence of the historical ordered log stream and the nodes of the frequent pattern tree model is greater than the third threshold, the server will prune and delete all child nodes after the third threshold in the data mining model.

[0117] The third threshold is the number of words that appear most frequently in the log word sequence of the historical ordered log stream, and these words can uniquely identify a log template. It should be understood that this application embodiment does not impose a limit on the value of the third threshold, and it can be set according to the actual situation.

[0118] It should be understood that the embodiments of this application do not limit the storage method of the log template. In some embodiments, the server can store the obtained log template in memory in real time and refresh the log template to external memory periodically.

[0119] The technical terms, technical effects, technical features, and optional implementation methods of S301-S302 can be found in [reference]. Figure 2 The explanations of S201-S204 are shown below. For content that is repeated, we will not repeat it here.

[0120] Based on the above embodiments, the following explains how to reorder the original log stream according to historical word frequencies to generate an ordered log stream. Figure 4 This is a flowchart illustrating an ordered log stream generation method provided in an embodiment of this application, as shown below. Figure 4As shown, the method includes:

[0121] S401. Filter the target characters of the logs in the original log stream to form at least one word segmentation stream.

[0122] In this step, after the server obtains the raw log stream generated by the terminal device, it can filter the target characters in the log stream to form at least one word segmentation stream.

[0123] It should be understood that the embodiments of this application do not limit the filtering method for target characters in the original log stream, and can be specifically set according to the actual situation. In some embodiments, the server can use an English word segmentation algorithm to segment the log, and then filter out the target characters in the log. The English word segmentation algorithm includes at least one of the following: not segmenting Chinese characters, English periods, English underscores, English hyphens, English slashes, and other characters with connecting meanings. Target characters include at least one of the following: punctuation marks, numbers, and words predefined in a stop word dictionary in the log.

[0124] S402. Update the historical word frequency based on the word frequency in at least one word segmentation stream within the second preset time window.

[0125] In this step, after the server obtains the target characters of the logs in the filtered original log stream and forms at least one segmented stream, it can update the historical word frequency based on the word frequency in at least one segmented stream within the second preset time window.

[0126] The second preset time window is a preset time period during which the server can count the word frequencies in the segmented word stream and update the historical word frequencies. This embodiment does not limit the length of this time period and can be set according to actual conditions. In some embodiments, the length of the second preset time window can be 10 minutes, while in other embodiments, the length can be 30 minutes.

[0127] Here, historical word frequency refers to the frequency of words that have appeared in the normal log stored in the memory. This application does not limit the method for updating historical word frequency. In some embodiments, the method for updating historical word frequency may be that the server counts the word frequency of each word in the segmented stream within a certain time period, and then updates the historical word frequency table in the memory storage in real time, and periodically updates the historical word frequency table in the external storage.

[0128] Furthermore, this application does not limit the structure of the historical word frequency table. In some embodiments, the historical word frequency table can be a key-value structure, where the key represents a word that has appeared in the log, and the value represents the frequency of that word.

[0129] S403. Reorder the word segmentation stream according to historical word frequency to generate an ordered log stream.

[0130] The technical terms, effects, features, and optional implementation methods of S401-S403 can be found in [reference]. Figure 2 The explanations of S201-S204 are shown below. For content that is repeated, we will not repeat it here.

[0131] The log detection method provided in this application involves the server first receiving the raw log stream of a first service sent by a terminal device. Then, the server reorders the raw log stream based on historical word frequencies to generate an ordered log stream. The logs in the ordered log stream are then matched against a log template to identify novel log streams. The log template is generated after updating the data mining model using historical ordered log streams. Finally, the logs in the novel log stream within a first preset time window are aggregated and converged to generate novel log events. This method, by matching and aggregating the logs in the ordered log stream against the log template in real time after receiving the raw log stream generated by the system, promptly identifies novel logs and narrows down the log scope for troubleshooting.

[0132] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as ROM, RAM, magnetic disk, or optical disk.

[0133] Figure 5 This is a schematic diagram of a log detection device provided in an embodiment of this application. The log detection device can be implemented through software, hardware, or a combination of both, and can be, for example, the server in the above embodiments, to execute the log detection method in the above embodiments. Figure 5 As shown, the log detection device 500 includes:

[0134] The receiving module 501 is used to receive the raw log stream of the first service sent by the terminal device;

[0135] The reordering module 502 is used to reorder the original log stream according to historical word frequencies to generate an ordered log stream.

[0136] The matching module 503 is used to match the logs in the ordered log stream with the log template to determine the novel log stream. The log template is generated after updating the data mining model using the historical ordered log stream.

[0137] The generation module 504 is used to aggregate and converge the logs in the novel log stream within the first preset time window to generate novel log events.

[0138] In one optional implementation, the matching module 503 is specifically used to match the logs in the ordered log stream with the log template respectively, determine the logs that fail to match, and use a Bloom filter to deduplicate the logs that fail to match, thereby generating a novel log stream.

[0139] In one optional implementation, the matching module 503 is specifically used to match the log sequence in the ordered log stream with the path of the log template.

[0140] In one optional implementation, the matching module 503 is further configured to update the data mining model with the historical ordered log stream to generate an updated data mining model; and to trim the updated data mining model to generate a log template.

[0141] In one optional implementation, the matching module 503 is further configured to insert log words from the root node of the frequent pattern tree model one by one according to the order of the log word sequence of the historical ordered log stream, until the termination node of the frequent pattern tree model; wherein, the termination node of the frequent pattern tree model is a node whose number of child nodes after the target layer is greater than a first threshold.

[0142] In one optional implementation, the reordering module 502 is specifically used to obtain the historical word frequency of each word in the log word sequence of the original log stream; and reorder the original log stream according to the magnitude of the historical word frequency of each word.

[0143] In one optional implementation, the reordering module 502 is specifically used to replace the historical frequency of the word that is arranged later with the historical frequency of the word that is arranged earlier if the word frequency difference between adjacent words is less than a second threshold.

[0144] In one optional implementation, the reordering module 502 is specifically used to sort the logs corresponding to the two words according to the dictionary order if the two words have the same historical frequency.

[0145] In one optional implementation, the reordering module 502 is further configured to filter target characters in the logs of the original log stream to form at least one word segmentation stream; and update the historical word frequency according to the word frequency in at least one word segmentation stream within a second preset time window.

[0146] In one optional implementation, the generation module 504 is further configured to send warning information to the terminal device to be warned, the warning information being used to warn of novel log events.

[0147] It needs to be explained that, Figure 5The log detection device provided in the illustrated embodiment can be used to execute the methods provided in any of the above embodiments. The specific implementation methods and technical effects are similar, and will not be described in detail here.

[0148] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 6 As shown, the electronic device may include at least one computing node 601 and a database node 602. Figure 6 The example shown is an electronic device consisting of a compute node and a database node.

[0149] Compute node 601 is used to store programs and execute instructions to implement the above-mentioned log detection method.

[0150] The computing node 601 may include at least one processor 6011 and memory 6012. Figure 6 The example shown is an electronic device using a processor.

[0151] The memory 6012 is used to store programs. Specifically, the program may include program code, which includes computer operation instructions.

[0152] The memory 6012 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0153] The processor 6011 is used to execute computer execution instructions stored in the memory 6012 to implement the above-mentioned log detection method;

[0154] The processor 6011 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0155] Optionally, in specific implementations, if the communication interface, memory 6012, and processor 6011 are implemented independently, they can be interconnected via a bus to complete communication. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc., but this does not imply that there is only one bus or one type of bus.

[0156] Optionally, in a specific implementation, if the communication interface, memory 6012, and processor 6011 are integrated on a single chip, then the communication interface, memory 6012, and processor 6011 can communicate through an internal interface.

[0157] Database node 602 is used to store programs. Specifically, the programs may include program code, which includes computer operation instructions.

[0158] Among them, compute node 601 and database node 602 can correspond to one or more process instances. Figure 6 The provided structural diagram only illustrates the relationship between the core components and does not depict devices such as buses, networks, input / output (I / O) devices, or other distributed environments on which they depend.

[0159] This application also provides a program that, when executed by a processor, performs the log detection method provided in the above method embodiments.

[0160] This application also provides a program product, such as a computer-readable storage medium, which stores instructions that, when run on a computer, cause the computer to execute the log detection method provided in the above-described method embodiments.

[0161] The present invention also provides a computer-readable storage medium, which may include various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a disk, or an optical disk. Specifically, the computer-readable storage medium stores program information, which is used in the above-mentioned log detection method.

[0162] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of the present invention is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0163] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A log detection method characterized by, The method comprises: receiving a first service original log stream sent by a terminal device; reordering the original log stream according to historical word frequencies to generate an ordered log stream; matching logs in the ordered log stream with log templates respectively to determine a novel log stream; the novel log stream is used to determine novel error logs that have not occurred; the log templates are generated after a data mining model is updated using a historical ordered log stream; the data mining model comprises a frequent pattern tree model; aggregating and converging logs in the novel log stream within a first preset time window to generate novel log events; matching logs in the ordered log stream with log templates respectively to determine a novel log stream, comprising: matching word sequences of logs in the ordered log stream with paths of the log templates respectively; if the word sequences of the logs completely match a path of the frequent pattern tree, or if the word sequences of the logs do not completely match a path of the frequent pattern tree but the matching length is greater than or equal to L, then the logs are non-novel logs of matching success; if the word sequences of the logs do not completely match any path of the frequent pattern tree and the matching length is less than L, then the logs are logs of matching failure; using a Bloom filter to remove duplicate logs of matching failure to generate the novel log stream.

2. The method of claim 1, wherein, After the novel log stream is determined, the method further comprises: updating the historical ordered log stream into the data mining model to generate an updated data mining model; trimming the updated data mining model to generate the log templates.

3. The method of claim 2, wherein, The updating of the historical ordered log stream into the data mining model to generate the updated data mining model comprises: inserting log words of the historical ordered log stream from a root node of the frequent pattern tree model according to the order of the log word sequences of the historical ordered log stream, until a terminal node of the frequent pattern tree model is reached; wherein the terminal node of the frequent pattern tree model is a node whose number of child nodes after a target layer is greater than a first threshold value.

4. The method of claim 1, wherein, The reordering of the original log stream according to historical word frequencies to generate an ordered log stream comprises: obtaining historical word frequencies of each word in log word sequences of the original log stream; reordering the original log stream according to the size of the historical word frequencies of each word.

5. The method of claim 4, wherein, The reordering of the original log stream according to the size of the historical word frequencies of each word comprises: if the frequency difference between adjacent words is less than a second threshold value, then replacing the historical frequency of a word arranged at the back with the historical frequency of a word arranged at the front.

6. The method of claim 4, wherein, The reordering of the original log stream according to the size of the historical word frequencies of each word comprises: if the historical frequencies of two words are the same, then sorting the logs corresponding to the two words according to the order of a dictionary.

7. The method of claim 1, wherein, Before the ordered log stream is generated, the method further comprises: filtering target characters of logs in the original log stream to form at least one token stream; updating the historical word frequencies according to word frequencies in the at least one token stream within a second preset time window.

8. The method of claim 7, wherein, The target character includes at least one of the following: a punctuation, a number, and a predefined word.

9. The method of claim 1, wherein, After the generation of the novelty log event, the method further includes: sending an early warning information to the terminal device to be warned, the early warning information being used for early warning the novelty log event.

10. A log detection apparatus characterized by comprising: The device includes: a receiving module configured to receive an original log stream of a first service sent by a terminal device; a reordering module configured to reorder the original log stream according to a historical word frequency to generate an ordered log stream; a matching module configured to match logs in the ordered log stream with log templates respectively to determine a novelty log stream; the novelty log stream is used for determining a novelty error log that has not occurred; the log templates are generated after a data mining model is updated using a historical ordered log stream; and the data mining model includes a frequent pattern tree model; The matching module is specifically configured to: match word sequences of logs in the ordered log stream with paths of the log templates respectively; if the word sequences of the logs are completely matched with a path of the frequent pattern tree, or if the word sequences of the logs are not completely matched with a path of the frequent pattern tree but the matching length is greater than or equal to L, then the logs are matched successfully and are non-novelty logs; if the word sequences of the logs are not completely matched with any path of the frequent pattern tree and the matching length is less than L, then the logs are matched unsuccessfully; perform deduplication processing on the logs matched unsuccessfully using a Bloom filter to generate the novelty log stream; a generating module configured to aggregate and converge logs in the novelty log stream within a first preset time window to generate a novelty log event.

11. A computer storage medium, characterized in that The computer storage medium stores a plurality of instructions, the instructions being adapted to be loaded and executed by the processor to perform the method steps of any one of claims 1-9.

12. An electronic device, comprising: The device includes: a processor and a memory; wherein the memory stores a computer program, the computer program being adapted to be loaded and executed by the processor to perform the method of any one of claims 1-9.

Images