Abnormal log detection method and device, electronic equipment and storage medium

Abstract

The invention provides an abnormal log detection method and device, electronic equipment and a storage medium, and relates to the technical field of log detection.The method comprises the steps that text steering processing is conducted on multiple logs generated in a target time period, and multiple semantic vectors corresponding to the multiple logs are obtained; clustering the plurality of logs based on the plurality of semantic vectors corresponding to the plurality of logs to obtain at least one log group; and determining whether abnormal logs exist in the log groups or not based on the target log number of the logs contained in each log group. According to the method, anomalies of natural language texts such as logs can be recognized from the semantic perspective, the problem that new anomalies cannot be detected due to the fact that special keywords are set is avoided, and the accuracy and comprehensiveness of abnormal log detection are improved.

Description

Technical Field

[0001] This application relates to the technical field of log detection, and particularly to an abnormal log detection method, apparatus, electronic device, and storage medium. Background Art

[0002] During the operation of an application, an event record called log is generated, and each line of the log records descriptions such as date, time, and related operations. By specifying the logs that the application needs to output, when the application is abnormal subsequently, the relevant operation information during the application abnormality can be traced back through the logs, providing relevant data information for subsequent application abnormality optimization.

[0003] In order to ensure the accuracy of subsequent log analysis, it is necessary to ensure that the application inputs logs normally. In the related art, when detecting abnormal logs, generally, special keywords of abnormal logs are manually marked, and the log content is detected based on the special keywords. However, for unforeseen abnormal logs, they cannot be discovered in time, resulting in poor detection accuracy of abnormal logs. Summary of the Invention

[0004] This application provides an abnormal log detection method, apparatus, electronic device, and storage medium, which can improve the detection accuracy of abnormal logs. The technical solutions are as follows:

[0005] According to one aspect of this application, an abnormal log detection method is provided, and the method includes:

[0006] Performing text-to-vector processing on multiple logs generated in a target time period to obtain multiple semantic vectors corresponding to the multiple logs;

[0007] Based on the multiple semantic vectors corresponding to the multiple logs, clustering the multiple logs to obtain at least one log group;

[0008] Based on the target log count of the logs included in each log group, determining whether there are abnormal logs in the log group.

[0009] According to another aspect of this application, an abnormal log detection apparatus is provided, and the apparatus includes:

[0010] A semantic extraction module, configured to perform text-to-vector processing on multiple logs generated in a target time period to obtain multiple semantic vectors corresponding to the multiple logs;

[0011] A clustering and grouping module, configured to cluster the multiple logs based on the multiple semantic vectors corresponding to the multiple logs to obtain at least one log group;

[0012] An exception determination module, configured to determine whether there is an abnormal log in the log group based on the number of target log entries included in each log group.

[0013] According to one aspect of the present application, there is provided an electronic device, including: a processor and a memory storing a program, the program including instructions that, when executed by the processor, cause the processor to execute the abnormal log detection method as described above.

[0014] According to another aspect of the present application, there is provided a non-transitory computer-readable storage medium storing computer instructions, the computer instructions being used to cause the computer to execute the abnormal log detection method as described above.

[0015] According to another aspect of the present application, there is provided a computer program product, the computer program product including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the abnormal log detection method as described above.

[0016] The beneficial effects brought by the technical solution provided by the embodiments of the present application at least include:

[0017] By converting the log into a text vector, extracting the semantic information of the log, and clustering and grouping the logs according to the similarity of the semantic information, a plurality of log groups are obtained; and then by quantitatively comparing the number of target log entries included in each log group, it is determined whether there is an abnormality in the log group. Compared with the related art solution of detecting abnormal logs by setting special keywords, the embodiments of the present application detect abnormal logs by semantic grouping and quantification of log groups, can identify the abnormalities of natural language text such as logs from a semantic perspective, avoid the problem that new abnormalities cannot be detected due to setting special keywords, and improve the accuracy and comprehensiveness of abnormal log detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In the following description of exemplary embodiments with reference to the drawings, more details, features, and advantages of the present application are disclosed. In the drawings:

[0019] Figure 1 A flowchart of an abnormal log detection method according to an exemplary embodiment of the present application is shown;

[0020] Figure 2 A flowchart of another abnormal log detection method according to an exemplary embodiment of the present application is shown;

[0021] Figure 3 A flowchart of another abnormal log detection method according to an exemplary embodiment of the present application is shown;

[0022] Figure 4 is a flowchart of a complete abnormal log detection method provided by an exemplary embodiment of the present application;

[0023] Figure 5 is a schematic structural diagram of an abnormal log detection device provided by an embodiment of the present application;

[0024] Figure 6 shows a structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present application. Detailed Embodiments

[0025] Embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Instead, these embodiments are provided to more thoroughly and completely understand the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not used to limit the protection scope of the present application.

[0026] It should be understood that the steps described in the method embodiments of the present application can be executed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present application is not limited in this regard.

[0027] As used herein, the term "including" and its variants are open-ended, that is, "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description. It should be noted that the concepts such as "first" and "second" mentioned in the present application are only used to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependence. It should be noted that the modifications of "one" and "multiple" mentioned in the present application are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly specified in the context, it should be understood as "one or more". The names of the messages or information exchanged between multiple devices in the embodiments of the present application are only for illustrative purposes and are not used to limit the scope of these messages or information.

[0028] The solutions of the present invention are described below with reference to the accompanying drawings. The technical solutions provided by the embodiments of the present application are described in detail through specific embodiments and their application scenarios.

[0029] In view of the problem of low detection accuracy in detecting abnormal logs by setting special keywords in the related art, the embodiments of the present application propose a new method for detecting abnormal logs. By performing a text-to-vector operation on the logs and grouping the logs according to semantics, subsequent calculation of quantization metrics is carried out to discover abnormal logs. It can discover abnormalities in the natural language text content of unformatted data logs, improving the detection accuracy and comprehensiveness of abnormal logs.

[0030] Please refer to Figure 1 , which shows a flowchart of a method for detecting abnormal logs according to an exemplary embodiment of the present application. This method is described by taking its application to an electronic device as an example. As Figure 1 shown, the method includes:

[0031] Step 101, perform a text-to-vector process on multiple logs generated during a target time period to obtain multiple semantic vectors corresponding to the multiple logs.

[0032] In order to improve the detection accuracy and comprehensiveness of abnormal logs, in a possible implementation, abnormal logs are mined by means of semantic extraction - clustering - quantization calculation. First, obtain multiple logs generated during the target time period, and then perform a text-to-vector process on the multiple logs through a pre-trained model, so as to obtain the semantic vector corresponding to each log in the multiple logs.

[0033] Exemplarily, the pre-trained model can be a model such as a BERT model or a Word2Vec model that can extract the semantic vector corresponding to the text. Optionally, before extracting the semantic vector of the log, data cleaning can also be performed on the log, and this data cleaning is mainly used to intercept logs of a fixed length and remove special symbols in the logs.

[0034] Among them, the target time period can be a manually set time period. Exemplarily, the target time period can be 1 day, or 5 days.

[0035] Step 102, based on the multiple semantic vectors corresponding to the multiple logs, cluster the multiple logs to obtain at least one log group.

[0036] After converting the text logs into semantic vectors, clustering processing is performed on the semantic vectors by using a clustering algorithm to obtain at least one log group. Among them, the logs included in the same log group have similar semantics, while the logs in different log groups have different semantics. Specifically, the similarity of the semantic vectors of the logs included in the same log group is greater than a preset threshold (or the distance between the semantic vectors in the multi-dimensional space is less than the distance threshold), while the similarity of the semantic vectors of the logs included in different log groups is less than the preset threshold (or the distance between the semantic vectors in the multi-dimensional space is greater than the distance threshold).

[0037] Exemplarily, for each log group, a unique group ID can be assigned to it to facilitate subsequent differentiation of different log groups.

[0038] Optionally, as logs are continuously generated, the log groups will also be continuously updated. Exemplarily, if there are existing log groups: log group A, log group B, and log group C; when new logs are obtained, first determine whether the new logs belong to the existing log groups. Specifically, determine the distance between the semantic vector of the new logs and the cluster centers in the existing log groups. If the distance is less than the distance threshold, it is determined that the new logs belong to the existing log groups; otherwise, if the distances are all greater than the distance threshold, the new logs are divided into a new log group.

[0039] Step 103, based on the target number of log entries included in each log group, determine whether there are abnormal logs in the log group.

[0040] After being divided into multiple log groups, for each log group, the number of logs generated for this type of log group should be uniform and not vary much within the target time period, which is a normal event; if the number of logs in a certain log group suddenly decreases, it indicates that there are abnormal logs. Therefore, in one possible implementation, it is possible to determine whether there are abnormal logs in the log group based on the target number of log entries included in each log group.

[0041] In summary, the embodiments of the present application provide an abnormal log detection method: by converting the text of the logs into vectors, extracting the semantic information of the logs, and clustering and grouping the logs according to the similarity of the semantic information to obtain multiple log groups; and then by quantitatively comparing the target number of log entries included in each log group, thereby determining whether there are abnormalities in the log group. Compared with the solution of detecting abnormal logs by setting special keywords in the related art, the embodiments of the present application detect abnormal logs by semantic grouping and quantification of log groups, can identify abnormalities in natural language text such as logs from a semantic perspective, avoid the problem that new abnormalities cannot be detected due to setting special keywords, and improve the accuracy and comprehensiveness of abnormal log detection.

[0042] When quantifying the log groups, it is mainly determined whether there are abnormalities by statistically calculating the probability of occurrence in the number-of-entries interval. If the probability of occurrence of a normal event log group in a certain number-of-entries interval is high, conversely, the probability of occurrence of an abnormal event in this log group in this number-of-entries interval is low.

[0043] Please refer to Figure 2 , which shows a flowchart of another abnormal log detection method according to an exemplary embodiment of the present application. This method is described by taking its application to an electronic device as an example. As Figure 2 shown, this method includes:

[0044] Step 201: Perform text-to-vector processing on multiple logs generated during the target time period to obtain multiple semantic vectors corresponding to the multiple logs.

[0045] Step 202: Cluster the multiple logs based on the multiple semantic vectors corresponding to the multiple logs to obtain at least one log group.

[0046] The implementation manners of Step 201 and Step 202 can refer to the above embodiments, and are not elaborated herein.

[0047] Step 203: For each log group, count the number of occurrences of logs in each preset time window within the target time period. The target time period is divided into multiple preset time windows.

[0048] Considering that if the log output is normal, the number of occurrences of the same group of logs within each unit time period should be roughly constant. To capture this quantitative feature, for each log group, first count the number of occurrences of logs in each preset time window within the target time period, that is, divide the target time period into multiple preset time windows and count the number of occurrences of logs in this log group within each preset time window. Then, based on the number of occurrences of logs, determine whether there are abnormal logs in this log group.

[0049] Exemplarily, the preset time window can be 1 minute. Taking the target time period as 1 day and the preset time window as 1 minute as an example, it is necessary to count the number of occurrences of logs in this log group within 1440 preset time windows. For example, the number of occurrences of logs in this log group within the first preset time window is 6 times, the number of occurrences of logs in this log group within the second preset time window is 7 times, the number of occurrences of logs in this log group within the third preset time window is 2 times, the number of occurrences of logs in this log group within the fourth preset time window is 6 times... and so on.

[0050] It should be noted that each log records the log output time (or log generation time). Therefore, the number of occurrences of logs in each preset time window can be determined according to the preset time window into which the log output time falls.

[0051] Step 204: Based on the number of occurrences of logs in each preset time window within the target time period, determine whether there are abnormal logs in the log group.

[0052] For the same group of logs (semantically identical logs), generally, the number of occurrences of logs in multiple preset time windows is the same or similar. If it suddenly decreases, it indicates that there are abnormal logs. For example, the required logs are not correctly output, or the log generation is abnormal due to abnormal application operation. Therefore, for each log group, by comparing the number of occurrences of logs in each preset time window within the target time period, it can be determined whether there are abnormal logs in this log group.

[0053] To more accurately quantify the feature of the number of occurrences of logs and avoid misidentification caused by accidental errors, by calculating the probability of the number of occurrences of logs corresponding to the target time period, it is determined whether there are abnormal logs. Corresponding to an exemplary example, step 204 may include step 204A and step 204B.

[0054] Step 204A, based on the number of occurrences of logs in each preset time window in the target time period, determines the probability of the number of occurrences of logs in each preset number range in the target time period.

[0055] When performing anomaly recognition, a small difference in the number of occurrences of logs in different preset time windows is allowed. Corresponding to calculating the probability of occurrence, the number of occurrences of logs is divided into multiple preset number ranges, and the histogram statistics method is used. Based on the number of occurrences of logs in each preset time window in the target time period, the probability of the number of occurrences of logs in each preset number range in the target time period is statistically calculated, and then based on the probability of occurrence, it is determined whether there are abnormal logs.

[0056] Exemplarily, when dividing the preset number range, the continuous number of occurrences can be divided into several preset number ranges in powers of 2, and then the probability of the number of occurrences of logs in each preset number range is statistically calculated.

[0057] Optionally, step 204A may further include step 204A1 and step 204A2.

[0058] Step 204A1, for each preset number range, determines the number of windows of the preset time window in which the number of occurrences of logs in the target time period is located in the preset number range.

[0059] Step 204A2, based on the number of windows and the total number of preset time windows included in the target time period, determines the probability of the number of occurrences of logs in the preset number range in the target time period.

[0060] Among them, the calculation formula for the probability of occurrence of a certain preset number range can be: probability of occurrence = the number of windows of the preset time window in which the number of occurrences of logs is located in the preset number range / the total number of preset time windows included in the target time period. Corresponding to a possible implementation manner, for the same log group, first, the number of occurrences of logs in each preset time window in the target time period is statistically calculated, and then according to the preset number range where the number of occurrences of logs is located, the number of windows of the preset time window in which the number of occurrences of logs in the target time period is located in the preset number range is statistically calculated. Then, according to the ratio of the number of windows and the total number of preset time windows included in the target time period, it is determined as the probability of the number of occurrences of logs in the preset number range in the target time period.

[0061] Exemplarily, if the preset time window can be 1 min. Taking the target time period as 1 day and the preset time window as 1 min as an example, the number of occurrences of the logs in the log group within 1440 preset time windows is counted. And four preset number-of-occurrences intervals are respectively divided: [0, 2], [2, 4], [4, 8], [8, 16]. For each log group, the number of windows of the preset time window where the number of occurrences of the logs is in the preset number-of-occurrences interval [0, 2] is 30, the number of windows of the preset time window where the number of occurrences of the logs is in the preset number-of-occurrences interval [2, 4] is 40, the number of windows of the preset time window where the number of occurrences of the logs is in the preset number-of-occurrences interval [4, 8] is 650, and the number of windows of the preset time window where the number of occurrences of the logs is in the preset number-of-occurrences interval [8, 16] is 720. Furthermore, the occurrence probability of the preset number-of-occurrences interval [0, 2] is calculated to be 2.08%, the occurrence probability of the preset number-of-occurrences interval [2, 4] is 2.78%, the occurrence probability of the preset number-of-occurrences interval [4, 8] is 45.14%, and the occurrence probability of the preset number-of-occurrences interval [8, 16] is 50%.

[0062] Step 204B, based on the occurrence probability, determine whether there are abnormal logs in the log group.

[0063] If the probability of occurrence of the normal time log group in this preset number-of-occurrences interval is high, and conversely the probability of occurrence of the abnormal time log group in this preset number-of-occurrences interval is low, then a preset probability threshold can be configured, and then by comparing the size relationship between the occurrence probability of each preset number-of-occurrences interval and this preset probability threshold, determine whether there are abnormal logs in this log group.

[0064] Specifically, if there is at least one occurrence probability less than the preset probability threshold, determine that there are abnormal logs in the log group; that is, if there is a situation where the occurrence probability is less than the preset probability threshold, determine that there are abnormal logs in this log group.

[0065] Exemplarily, the preset probability threshold can be 3%. Since the occurrence probabilities of the two preset number-of-occurrences intervals [0, 2] and [2, 4] are less than the preset probability threshold, it is determined that there are abnormal logs in this log group.

[0066] In this embodiment, by counting the occurrence probability of the number of occurrences of the logs within the target time period in each preset number-of-occurrences interval for each log group, and comparing the occurrence probability with the preset probability threshold, the identification and detection of abnormal logs are carried out. The purpose of detecting abnormal logs by quantifying the target log number of each log group is achieved.

[0067] After determining that there are abnormal logs in the log group, it is also necessary to locate the abnormal logs in the log group so as to issue an alarm based on the located abnormal logs.

[0068] Please refer to Figure 3 , which shows a flowchart of another abnormal log detection method according to an exemplary embodiment of the present application. This method is described by taking its application to an electronic device as an example. As Figure 3 shown, this method includes:

[0069] Step 301: Perform text-to-vector processing on multiple logs generated during a target time period to obtain multiple semantic vectors corresponding to the multiple logs.

[0070] Step 302: Cluster the multiple logs based on the multiple semantic vectors corresponding to the multiple logs to obtain at least one log group.

[0071] Step 303: For each log group, count the number of occurrences of logs within each preset time window in the target time period, and the target time period is divided into multiple preset time windows.

[0072] Step 304: For each preset number-of-occurrences interval, determine the number of windows of the preset time windows in which the number of occurrences of logs in the target time period is within the preset number-of-occurrences interval.

[0073] Step 305: Based on the number of windows and the total number of preset time windows included in the target time period, determine the occurrence probability that the number of occurrences of logs in the target time period is within the preset number-of-occurrences interval.

[0074] Step 306: If there is at least one occurrence probability less than a preset probability threshold, determine that there are abnormal logs in the log group.

[0075] The implementation manners of Steps 301 to 306 can refer to the above embodiments, and are not elaborated herein.

[0076] Step 307: If there are abnormal logs in the log group, determine the target number-of-occurrences interval in which the occurrence probability is less than the preset probability threshold.

[0077] Since the abnormal event (or abnormal log) occurs in the number-of-occurrences interval where the occurrence probability threshold is less than the preset probability threshold, when locating the abnormal log in the log group, first count the occurrence probability of each preset number-of-occurrences interval, and determine the target number-of-occurrences interval in which the occurrence probability is less than the preset probability threshold from them.

[0078] Exemplarily, taking the division of four preset pen number intervals: [0, 2], [2, 4], [4, 8], [8, 16] as an example, if the occurrence probability of the preset pen number interval [0, 2] is statistically obtained as 2.08%, the occurrence probability of the preset pen number interval [2, 4] is 2.78%, the occurrence probability of the preset pen number interval [4, 8] is 45.14%, and the occurrence probability of the preset pen number interval [8, 16] is 50%, and the preset probability threshold is 3%, then the two preset pen number intervals [0, 2] and [2, 4] are determined as the target pen number intervals.

[0079] Step 308, determine the target time window where the number of log occurrences is within the target pen number interval.

[0080] After determining the target pen number interval, further determine the target time window where the number of log occurrences is within the target pen number interval, indicating that the number of log occurrences generated within this target time window is relatively small and is an abnormal event.

[0081] Step 309, determine the logs generated within the target time window in the log group as abnormal logs.

[0082] Furthermore, determine the logs generated within the target time window in this log group as abnormal logs, and output an alarm message containing this abnormal log for subsequent abnormal analysis and follow-up.

[0083] Exemplarily, if the two preset pen number intervals [0, 2] and [2, 4] are the target pen number intervals, and from the previous statistics, the number of window quantities of the preset time window where the number of log occurrences is within the preset pen number interval [0, 2] is 30, and the number of window quantities of the preset time window where the number of log occurrences is within the preset pen number interval [2, 4] is 40, that is, these 70 preset time windows are determined as the target time windows, and the original logs generated within these 70 preset time windows are located as abnormal logs.

[0084] In this embodiment, by analyzing the target pen number interval and the target time window corresponding to the abnormal event, the time period where the abnormal log is located is located, and then the abnormal log is located to achieve the purpose of abnormal detection and abnormal location.

[0085] Please refer to Figure 4 , which is a flowchart of a complete abnormal log detection method provided by an exemplary embodiment of the present application. The process includes the following steps:

[0086] (1) Perform data cleaning on the consumed logs: intercept the log length and remove special symbols;

[0087] (2) Use a pre-trained model to convert the text log into a vector;

[0088] (3) Use the k-mean algorithm to cluster and group the vectors: For the pre-trained model, after converting the text into vectors, texts with the same semantics will generate vectors with close distances in the multi-dimensional space. Taking advantage of this, use a clustering algorithm (such as k-mean) to group the vectors. Achieve grouping of the logs, so that the same logs are assigned to the same group, and assign a globally unique group ID to the group and a group ID to each log entry;

[0089] (4) Select a time window (such as 1 minute) for data aggregation and count the number of occurrences of each log group;

[0090] (5) Use histogram statistics: Divide the number of occurrences of the log group into intervals, and divide the continuous number of occurrences into a finite number of intervals according to the nth power of 2. Count the occurrence frequency of different numbers of occurrences of each log group in the number-of-occurrences intervals;

[0091] (6) Verify each log group and count the occurrence probability of different numbers of occurrences of each log group in the number-of-occurrences intervals: Count the probability of each log group in each number-of-occurrences interval in the history (such as the history of the past 5 days). If it is a normal event log group, the probability of occurrence in this number-of-occurrences interval is high, and conversely, if it is an abnormal event, the probability of occurrence of this log group in this number-of-occurrences interval is low. Configure a probability threshold (such as 3%), and raise an alarm for the minority with an occurrence probability less than this threshold. Attach a latest specific original log to the alarm message for follow-up of the anomaly.

[0092] Please refer to Figure 5 , which is a schematic structural diagram of an abnormal log detection device provided by an embodiment of the present application. Exemplarily, as Figure 5 shown, the device 500 includes:

[0093] A semantic extraction module 501, configured to perform text-to-vector processing on multiple logs generated in a target time period to obtain multiple semantic vectors corresponding to the multiple logs;

[0094] A clustering and grouping module 502, configured to cluster the multiple logs based on the multiple semantic vectors corresponding to the multiple logs to obtain at least one log group;

[0095] An anomaly determination module 503, configured to determine whether there are abnormal logs in the log group based on the target log count of the logs included in each log group.

[0096] Optionally, the anomaly determination module 503 is further configured to:

[0097] For each of the log groups, count the number of occurrences of the logs in each preset time window within the target time period, and the target time period is divided into multiple preset time windows;

[0098] Determine whether there is any abnormal log in the log group based on the number of occurrences of the logs within each of the preset time windows in the target time period.

[0099] Optionally, the abnormal determination module 503 is further configured to:

[0100] Based on the number of occurrences of the logs within each of the preset time windows in the target time period, determine the occurrence probability of the number of occurrences of the logs within each preset number-of-occurrences interval in the target time period;

[0101] Based on the occurrence probability, determine whether there is any abnormal log in the log group.

[0102] Optionally, the abnormal determination module 503 is further configured to:

[0103] If there is at least one occurrence probability less than a preset probability threshold, determine that there is an abnormal log in the log group.

[0104] Optionally, the abnormal determination module 503 is further configured to:

[0105] For each of the preset number-of-occurrences intervals, determine the number of windows of the preset time windows within which the number of occurrences of the logs in the target time period is located in the preset number-of-occurrences interval;

[0106] Based on the number of windows and the total number of the preset time windows included in the target time period, determine the occurrence probability of the number of occurrences of the logs in the target time period within the preset number-of-occurrences interval.

[0107] Optionally, the apparatus further includes:

[0108] A first determination module, configured to determine, if there is an abnormal log in the log group, the target number-of-occurrences interval in which the occurrence probability is less than the preset probability threshold;

[0109] A second determination module, configured to determine the target time window within which the number of occurrences of the logs is located in the target number-of-occurrences interval;

[0110] A third determination module, configured to determine the logs generated within the target time window in the log group as the abnormal logs.

[0111] In summary, the embodiments of the present application provide an abnormal log detection method: by converting the log into a text vector, extracting the semantic information of the log, and clustering and grouping the logs according to the similarity of the semantic information to obtain multiple log groups; and then by quantitatively comparing the number of target log entries included in each log group, so as to determine whether there is an abnormality in the log group. Compared with the solution of detecting abnormal logs by setting special keywords in the related art, the embodiments of the present application detect abnormal logs by semantic grouping and quantification of log groups, can identify the abnormalities of natural language text such as logs from the semantic perspective, avoid the problem that new abnormalities cannot be detected due to setting special keywords, and improve the accuracy and comprehensiveness of abnormal log detection.

[0112] An exemplary embodiment of the present application further provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program that can be executed by the at least one processor, and when the computer program is executed by the at least one processor, it is used to cause the electronic device to execute the abnormal log detection method according to the embodiments of the present application.

[0113] An exemplary embodiment of the present application further provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program is used to cause a computer to execute the abnormal log detection method according to the embodiments of the present application when executed by a processor of the computer.

[0114] An exemplary embodiment of the present application further provides a computer program product, including a computer program, wherein the computer program is used to cause a computer to execute the abnormal log detection method according to the embodiments of the present application when executed by a processor of the computer.

[0115] Reference Figure 6 will now describe a block diagram of an electronic device 600 that can be a server or a client of the present application, which is an example of a hardware device that can be applied to various aspects of the present application. The electronic device is intended to represent various forms of digital electronic computer devices, such as, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, a personal digital processor, a cellular phone, a smart phone, a wearable device, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0116] As Figure 6As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to the computer program stored in the ROM 602 or the computer program loaded from the storage unit 608 into the RAM 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. The I / O interface 605 is also connected to the bus 604.

[0117] Multiple components in the electronic device 600 are connected to the I / O interface 605, including: an input unit 606, an output unit 607, a storage unit 608, and a communication unit 609. The input unit 606 can be any type of device that can input information into the electronic device 600. The input unit 606 can receive input digital or character information, and generate key signal inputs related to the user settings and / or function controls of the electronic device. The output unit 607 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 608 can include but is not limited to a magnetic disk, an optical disk. The communication unit 609 allows the electronic device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0118] Optionally, a single-channel electroencephalogram (EEG) signal acquisition module (not shown in the figure) is also provided in the electronic device 600. It is used to acquire EEG signals and transmit the EEG signals to the signal processor of the electronic device 600 for EEG signal processing.

[0119] The computing unit 601 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include but are not limited to a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 601 executes the various methods and processes described above. For example, in some embodiments, Figure 1 、 Figure 2 、 Figure 3The method shown can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 600 via the ROM 602 and / or the communication unit 609. In some embodiments, the computing unit 601 can be configured to execute Figure 1 , Figure 2 , Figure 3 the method shown.

[0120] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the program codes are executed by the processor or controller, the functions / operations specified in the flowchart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as an independent software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0121] In the context of the present application, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0122] As used in this application, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, device, and / or apparatus (e.g., a disk, an optical disc, a memory, a programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term “machine-readable signal” refers to any signal for providing machine instructions and / or data to a programmable processor.

[0123] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can also be used to provide for interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0124] The systems and techniques described here can be implemented in a computing system including a back-end component (e.g., as a data server), or a computing system including a middleware component (e.g., an application server), or a computing system including a front-end component (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described here), or a computing system including any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0125] A computer system can include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of the client and the server is generated by computer programs running on the respective computers and having a client-server relationship to each other.

Claims

1. A method for detecting abnormal logs, characterized in that: The method comprises: Performing text transformation processing on multiple logs generated in a target time period to obtain multiple semantic vectors corresponding to the multiple logs; Clustering the multiple logs based on the multiple semantic vectors corresponding to the multiple logs to obtain at least one log group; Based on the target number of log records contained in each log group, determine whether there is an abnormal log in the log group.

2. The method according to claim 1, characterized in that The step of determining whether there is an abnormal log in each log group based on the target number of logs included in the log group includes: For each of the log groups, counting the number of log occurrences in each preset time window in the target time period, wherein the target time period is divided into a plurality of the preset time windows; Based on the number of occurrences of the log in each of the preset time windows in the target time period, it is determined whether the abnormal log exists in the log group.

3. The method according to claim 2, characterized in that The determining whether the abnormal log exists in the log group based on the number of occurrences of the log in each of the preset time windows in the target time period includes: Based on the number of occurrences of the log records in each of the preset time windows in the target time period, determining the probability of occurrence of the number of occurrences of the log records in the target time period in each preset number interval; Based on the occurrence probability, determine whether the abnormal log exists in the log group.

4. The method according to claim 3, characterized in that The determining, based on the occurrence probability, whether the abnormal log exists in the log group includes: If there is at least one occurrence probability that is less than a preset probability threshold, it is determined that the abnormal log exists in the log group.

5. The method according to claim 3, characterized in that: The determining, based on the number of occurrences of the log records in each of the preset time windows in the target time period, the probability of occurrence of the number of occurrences of the log records in the target time period in each preset number interval includes: For each of the preset number of entries intervals, determine the number of windows of the preset time windows in which the number of entries in the logs within the target time period is within the preset number of entries interval; Based on the number of windows and the total number of the preset time windows included in the target time period, the occurrence probability of the number of log entries in the target time period within the preset number interval is determined.

6. The method according to claim 4, characterized in that The method further comprises: If the abnormal log exists in the log group, determining a target number interval in which the occurrence probability is less than the preset probability threshold; Determine a target time window in which the number of occurrences of the log is within the target number interval; The logs generated within the target time window in the log group are determined as the abnormal logs.

7. An abnormal log detection device, characterized in that: The device comprises: A semantic extraction module, used for performing text transformation processing on a plurality of logs generated in a target time period to obtain a plurality of semantic vectors corresponding to the plurality of logs; A clustering grouping module, configured to cluster the multiple logs based on the multiple semantic vectors corresponding to the multiple logs to obtain at least one log group; The abnormality determination module is used to determine whether there is an abnormal log in each log group based on the target number of log records contained in the log group.

8. The device according to claim 7, characterized in that The abnormality determination module is further used for: For each of the log groups, counting the number of log occurrences in each preset time window in the target time period, wherein the target time period is divided into a plurality of the preset time windows; Based on the number of occurrences of the log in each of the preset time windows in the target time period, it is determined whether the abnormal log exists in the log group.

9. An electronic device, comprising: processor; as well as Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to execute the abnormal log detection method according to any one of claims 1 to 6.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the abnormal log detection method according to any one of claims 1 to 6.

Images