Construction method, system and equipment of error model library of log and storage medium
Through the combination of sliding window technology and context information, a more comprehensive and accurate log error model library was built, which solved the shortcomings in the accuracy and coverage of log analysis in the existing technology, and achieved more efficient log analysis and fault handling.
Patent Information
- Application Number
- CN202410867537.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-30
- Publication Date
- 2024-11-01
AI Technical Summary
Existing log analysis methods use one-dimensional keywords or regular expression collections to build a model library, which cannot effectively combine context information, resulting in limited log error information, slow expansion, narrow coverage, low accuracy, and difficult to achieve more accurate log analysis.
By collecting error logs when a server runs an error, using two-dimensional arrays with multiple rows and columns, and using sliding window technology to extract suspicious information, put it into the error model library, and classifying and denoising the process with context information, building more A library of comprehensive and accurate error models.
It improves the accuracy and coverage of log analysis, enhances the amount of information in the error model library, can more effectively match and process logs to be analyzed, and improves the accuracy and efficiency of fault processing.
Smart Images

Figure CN118885451A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of servers, and in particular to a method, system, device and storage medium for constructing an error model library of a log. Background Art
[0002] Logs are information generated during the operation of the server. They record the operation of the operating system, application software and other hardware and software, and reflect the status of the server. It is particularly important to understand the log information during server maintenance and upgrades.
[0003] Most of the existing log analysis methods use regular expressions to perform line-by-line matching analysis. For example, in one current solution, regular expressions are used for pattern matching during log preprocessing, and then the tree frequency set algorithm is used to analyze the log data. In another current solution, the log analysis request includes an association rule template, which is parsed to extract key information, and then the task type is determined based on this information, so that different analysis engines are used for log analysis. In another current solution, after obtaining the associated message set of the request message, feature parameters are extracted for each message contained in the request message and the associated message set, and the feature parameter set is obtained and encapsulated, and finally log analysis is performed on the encapsulated log.
[0004] In general, the common disadvantage of the current solutions is that the model libraries used are mostly composed of one-dimensional keyword (rule) / regular expression sets, which contain limited log error information. In addition, since it is impossible to filter error information in context, the expansion is slow, the coverage is narrow, and the accuracy is low, which is not conducive to the subsequent implementation of more accurate log analysis.
[0005] In summary, how to effectively construct an error model library for logs so as to facilitate subsequent log analysis is a technical problem that those skilled in the art urgently need to solve. Summary of the invention
[0006] The purpose of the present invention is to provide a method, system, device and storage medium for constructing a log error model library, so as to effectively construct the log error model library, making it more conducive to subsequent log analysis.
[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:
[0008] In a first aspect, the present invention provides a method for constructing an error model library of a log, comprising:
[0009] When an error occurs during server operation, error logs are collected to obtain M groups of log information;
[0010] Wherein, for any group of log information, the group of log information is a two-dimensional array of multiple rows and columns; M is a positive integer;
[0011] For each group of log information, a plurality of window arrays are intercepted from the group of log information by sliding a preset first sliding window on the group of log information, and suspicious information is extracted based on each of the window arrays intercepted from the group of log information by the first sliding window;
[0012] Each of the extracted suspicious information is placed in an error model library to construct the error model library;
[0013] Wherein, at least one of the length and the width of the first sliding window is greater than 1.
[0014] On the other hand, it also includes:
[0015] For any one of the extracted suspicious information, denoising is performed on the suspicious information according to a preset denoising rule to remove meaningless data in the suspicious information.
[0016] On the other hand, for any one of the extracted suspicious information, denoising is performed on the suspicious information according to a preset denoising rule to remove meaningless data in the suspicious information, including:
[0017] For any one of the extracted suspicious information, punctuation marks, spaces and device codes in the suspicious information are removed to remove meaningless data in the suspicious information.
[0018] On the other hand, for each group of the log information, a plurality of window arrays are intercepted from the group of log information by sliding a preset first sliding window on the group of log information, and suspicious information is extracted based on each of the window arrays intercepted from the group of log information by the first sliding window, including:
[0019] For each group of log information, a plurality of window arrays are intercepted from the group of log information by sliding a preset first sliding window on the group of log information;
[0020] Based on each of the window arrays intercepted from the group of log information through the first sliding window, a data structure with an appearance frequency higher than a preset frequency threshold is taken as the extracted suspicious information.
[0021] On the other hand, it also includes:
[0022] For any one of the extracted suspicious information, when the suspicious information only contains a single data, the suspicious information is discarded.
[0023] On the other hand, it also includes:
[0024] Selecting the second sliding window to the Nth sliding window in sequence from the preset sliding window set;
[0025] When the i-th sliding window is selected, for each group of log information, a plurality of window arrays are intercepted from the group of log information by sliding the i-th sliding window on the group of log information, and suspicious information is extracted based on each of the window arrays intercepted from the group of log information by the i-th sliding window;
[0026] Wherein, i is a positive integer and ranges from 2 to N in sequence, and N is a positive integer not less than 2; at least one of the length and width of the i-th sliding window is greater than 1;
[0027] The sizes of any two sliding windows in the sliding window set are different from each other, and the size of any sliding window in the sliding window set is different from the size of the first sliding window.
[0028] On the other hand, it also includes:
[0029] Classifying the suspicious information in the error model library according to the classification principle that suspicious information with high similarity is classified into the same category;
[0030] The classification principle includes: when the number of data included in two suspicious information is consistent and the data content included is consistent, the two suspicious information are classified into the same category;
[0031] For any one category, determine the number of occurrences of each suspicious information in the category in the error model library, and sum them up, and use the sum result as a weight value for reflecting the importance of the category;
[0032] Collecting logs of the server during the operation of the server as logs to be analyzed;
[0033] Matching the log to be analyzed with the error model library, and when it is determined that the log to be analyzed matches any classification in the error model library, determining that the log to be analyzed is an abnormal log, and obtaining a weight value of the classification;
[0034] According to the preset corresponding relationship, determining the fault handling level corresponding to the weight value;
[0035] A corresponding fault handling operation is performed based on the fault handling level, and prompt information carrying the abnormal log and the obtained weight value is output.
[0036] In a second aspect, the present invention provides a system for constructing a log error model library, comprising:
[0037] The log information determination module is used to collect error logs when an error occurs during server operation to obtain M groups of log information;
[0038] Wherein, for any group of log information, the group of log information is a two-dimensional array of multiple rows and columns; M is a positive integer;
[0039] A suspicious information extraction module, configured to extract multiple window arrays from each group of log information by sliding a preset first sliding window on the group of log information, and extract suspicious information based on each window array intercepted from the group of log information by the first sliding window;
[0040] An error model library construction module, used for placing each of the extracted suspicious information into the error model library to construct the error model library;
[0041] Wherein, at least one of the length and the width of the first sliding window is greater than 1.
[0042] In a third aspect, the present invention provides a device for constructing a log error model library, comprising:
[0043] Memory for storing computer programs;
[0044] The processor is used to execute the computer program to implement the steps of the method for constructing the error model library of the log as described above.
[0045] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method for constructing an error model library of logs as described above are implemented.
[0046] By applying the technical solution provided by the embodiment of the present invention, when the server runs in error, the error log is collected, and after obtaining M groups of log information, the extraction of suspicious information is realized by means of a sliding window. Specifically, the first sliding window is used, and at least one of its length and width is greater than 1, which means that the suspicious information obtained by the present application scheme is the suspicious information obtained by combining the context information of the error log, that is, the traditional one-dimensional keyword is expanded into two-dimensional suspicious information, with a wider coverage and higher accuracy. The amount of information contained in the constructed error model library is greatly increased, which is also conducive to improving the accuracy of subsequent log analysis. Specifically, since it is necessary to obtain suspicious information in combination with context information, for any group of log information in the M groups of log information, the group of log information needs to be a two-dimensional array of multiple rows and columns. Afterwards, for each group of log information, multiple window arrays can be intercepted from the group of log information by sliding the preset first sliding window on the group of log information, and then based on each window array intercepted from the group of log information by the first sliding window, the suspicious information is extracted by regularity analysis, and then each extracted suspicious information is placed in the error model library to realize the construction of the error model library.
[0047] To sum up, the suspicious information obtained by the present application scheme is the suspicious information obtained by combining the contextual information of the error log, that is, expanding the traditional one-dimensional keywords into two-dimensional suspicious information, with a wider coverage and higher accuracy. The amount of information contained in the constructed error model library is greatly increased, which is conducive to improving the accuracy of subsequent log analysis. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0049] Figure 1 A flowchart of a method for constructing a log error model library provided by a specific implementation mode of the present invention;
[0050] Figure 2 This is a schematic diagram of a group of log information obtained in a specific implementation manner of the present invention;
[0051] Figure 3a A schematic diagram of a first sliding window in a specific implementation manner of the present invention;
[0052] Figure 3bThis is a schematic diagram of a first sliding window sliding on log information in a specific implementation manner of the present invention;
[0053] Figure 4 A schematic diagram of the principle of extracting suspicious information and removing meaningless data in a specific implementation of the present invention;
[0054] Figure 5 A schematic diagram of classifying suspicious information in a specific implementation manner of the present invention;
[0055] Figure 6 A schematic diagram of the structure of a system for constructing a log error model library provided by a specific implementation mode of the present invention;
[0056] Figure 7 A schematic diagram of the structure of a device for constructing a log error model library provided in a specific embodiment of the present invention;
[0057] Figure 8 The present invention is a schematic diagram of the structure of a computer-readable storage medium. DETAILED DESCRIPTION
[0058] The core of the present invention is to provide a method, system, device and storage medium for constructing a log error model library. The obtained suspicious information is the suspicious information obtained by combining the context information of the error log, that is, the traditional one-dimensional keyword is expanded into two-dimensional suspicious information, which has a wider coverage and higher accuracy. The amount of information contained in the constructed error model library is greatly increased, which is conducive to improving the accuracy of subsequent log analysis.
[0059] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0060] Please refer to Figure 1 , Figure 1 The present invention is a flowchart of a method for constructing a log error model library provided in a specific embodiment of the present invention. The method for constructing a log error model library may include the following steps:
[0061] Step S101: When an error occurs during server operation, error logs are collected to obtain M groups of log information.
[0062] Specifically, since the error model library constructed by the present application solution reflects the log status when the server runs an error, when constructing the error model library, it is necessary to collect error logs when the server runs an error to obtain M groups of log information, that is, it is necessary to collect logs when the server runs an error, and the collected logs are called error logs. In addition, it should be noted that since there can be many specific error types when the server runs an error, in actual applications, error logs can be collected under different error states to obtain more groups of log information, which is conducive to the subsequent construction of a more comprehensive and complete error model library.
[0063] Based on the collected error logs, M groups of log information can be obtained. For any group of log information, the group of log information is a two-dimensional array with multiple rows and columns; M is a positive integer.
[0064] For example, in one case, when a server fails to run, each time a log is collected, the content of the log is taken as a line, and after collecting K lines of logs, a set of log information is obtained. In another case, when a server fails to run, a log is taken as a line, and each log collected within a unit time is taken as a set of log information.
[0065] See also Figure 2 , is a schematic diagram of a set of log information obtained in a specific implementation mode, Figure 2 In the example, a set of log information is a two-dimensional array with 9 rows and 19 columns. Figure 2 This is just an example. In other specific implementations, a set of log information can be a two-dimensional array of other sizes, and the size is usually much higher than Figure 2 In addition, it should be noted that, in actual applications, the size of each set of log information can be the same or different. Of course, in order to facilitate operation and program design, each set of log information is usually set to a fixed size.
[0066] Step S102: For each group of log information, a plurality of window arrays are intercepted from the group of log information by sliding a preset first sliding window on the group of log information, and suspicious information is extracted based on each window array intercepted from the group of log information by the first sliding window.
[0067] For each set of log information, the first sliding window is used to extract suspicious information. Figure 2 This group of log information in is used as an example to illustrate. Figure 3a , which is a schematic diagram of the first sliding window in a specific implementation of the present invention.
[0068] Since the present application solution needs to extract the context information of the log, at least one of the length and width of the first sliding window must be greater than 1. Figure 3a In the example, it is a 4×4 sliding window, that is, the length of the first sliding window is 4 and the width is also 4, satisfying the requirement that at least one of the length and width is greater than 1.
[0069] The first sliding window will slide on the group of log information. Usually, it can start from the upper left corner of the group of log information, slide rightward and downward, slide one grid in the length direction (rightward) or one grid in the width direction (downward) each time, and finally reach the lower right corner of the group of log information. In this process, each sliding can obtain a window array. When it finally arrives, the window arrays intercepted from the group of log information by the first sliding window are obtained.
[0070] exist Figure 3a In the example, the first sliding window is located in the upper left corner of the group of log information. At this time, a window array is intercepted for the first time, and then you can refer to Figure 3b , the first sliding window slides one grid to the right, and the second window array is intercepted, and so on. Figure 3a For example, a total of 17×7=119 window arrays can be extracted from the group of log information through the first sliding window.
[0071] Afterwards, it is necessary to extract suspicious information based on each window array intercepted from the group of log information through the first sliding window. In the above example, one suspicious information is extracted based on 119 window arrays.
[0072] The specific operation of extracting suspicious information based on each window array intercepted from the group of log information through the first sliding window can be set and adjusted according to actual needs. Usually, the data structure that appears frequently in these window arrays is used as the extracted suspicious information.
[0073] For example, in a specific implementation of the present invention, step S102 may specifically include:
[0074] For each group of log information, a plurality of window arrays are intercepted from the group of log information by sliding a preset first sliding window on the group of log information;
[0075] Based on each window array intercepted from the group of log information through the first sliding window, a data structure with an appearance frequency higher than a preset frequency threshold is taken as one piece of suspicious information extracted.
[0076] This implementation is a very convenient implementation for extracting suspicious information. Specifically, after cutting out multiple window arrays from the log information by sliding the first sliding window, this implementation is to use the data structure with a frequency higher than a preset frequency threshold in these window arrays as the extracted suspicious information. Figure 4 ,For example Figure 4 In the example, by sliding the first sliding window, 100 window arrays are intercepted from the log information, for example, the frequency threshold is 50%, and for these 100 3×3 window arrays, for example, the first value of the first row in 60 window arrays is a, the second value of the first row in 60 window arrays is a space, the third value of the first row in 70 window arrays is d, and the second value of the second row in 55 window arrays is f. In addition, for the remaining 5 positions, no more than 50% of the window arrays have the same value. Figure 4 In the example, a "T-type" data structure is obtained as the extracted suspicious information.
[0077] Step S103: each extracted suspicious information is placed into the error model library to construct the error model library.
[0078] Whenever a suspicious information is obtained, it can be placed in the error model library, thereby constructing the error model library.
[0079] For example, in one implementation above, since M groups of log information are obtained, for each group of log information, the first sliding window can be slid on the group of log information, so that one suspicious information can be extracted by using the intercepted window arrays. In this example, a total of M suspicious information can be obtained and then placed in the error model library. In addition, it can be understood that these M suspicious information placed in the error model library may be repeated.
[0080] As another example, in the implementation method described later, more sliding windows of different sizes are provided. For each set of log information, sliding windows of different sizes can be slid in turn to extract suspicious information, so that more suspicious information can be put into the error model library.
[0081] After obtaining the error model library, log analysis can be implemented based on it. For example, during the operation of the server, the server logs can be collected as the logs to be analyzed, and then the logs to be analyzed can be matched based on the error model library. For example, the specific matching rule is that when there is a data structure in the log to be analyzed that is the same as at least one suspicious information in the error model library, the log to be analyzed is regarded as a fault log, and an alarm is issued.
[0082] Of course, in other implementations, there may be more complex log analysis methods. For example, based on an error model library, log analysis may be implemented in combination with graph theory, breadth retrieval, depth retrieval and other multi-dimensional methods, and may be set according to actual needs.
[0083] In a specific embodiment of the present invention, it may also include:
[0084] For any extracted suspicious information, the suspicious information is denoised according to the preset denoising rules to remove meaningless data in the suspicious information.
[0085] This implementation method takes into account that the obtained suspicious information may contain some meaningless data content. Therefore, the suspicious information can be denoised according to the preset denoising rules, so that the suspicious information after denoising can more accurately and effectively reflect the rules in the log information in the case of server failure, avoiding the interference of meaningless data content.
[0086] The specific content of the preset noise reduction rules can be set and adjusted according to actual needs. For example, in a specific embodiment of the present invention, for any extracted suspicious information, according to the preset noise reduction rules, the suspicious information is denoised to remove meaningless data in the suspicious information, which can specifically include:
[0087] For any extracted suspicious information, punctuation marks, spaces and device codes in the suspicious information are removed to remove meaningless data in the suspicious information.
[0088] This implementation method takes into account that punctuation marks, spaces, and device codes in the log are usually irrelevant to the server failure situation and have low analysis value. Therefore, the punctuation marks, spaces, and device codes in the suspicious information can be regarded as meaningless data and removed. It should be noted here that the device code is usually composed of numbers, and the numbers in the log are usually irrelevant to the server failure situation. Therefore, a more convenient implementation method in actual application is to directly regard the numbers as device codes.
[0089] See also Figure 4 ,exist Figure 4 In the example, Figure 4 The spaces in the suspicious information are removed.
[0090] In a specific embodiment of the present invention, it may also include:
[0091] For any extracted suspicious information, if the suspicious information only contains a single data, the suspicious information will be discarded.
[0092] This implementation method takes into account that the suspicious information in the present application scheme is the suspicious information obtained by combining the context information of the error log, that is, expanding the traditional one-dimensional keyword into two-dimensional suspicious information. Therefore, if the extracted suspicious information only contains a single data, it means that the suspicious information does not have much analytical value and is prone to misjudgment in the future. Therefore, such suspicious information can be directly discarded and not placed in the error model library.
[0093] In a specific embodiment of the present invention, it may also include:
[0094] Selecting the second sliding window to the Nth sliding window in sequence from the preset sliding window set;
[0095] When the i-th sliding window is selected, for each group of log information, a plurality of window arrays are intercepted from the group of log information by sliding the i-th sliding window on the group of log information, and suspicious information is extracted based on each window array intercepted from the group of log information by the i-th sliding window;
[0096] Wherein, i is a positive integer and ranges from 2 to N, and N is a positive integer not less than 2; at least one of the length and width of the i-th sliding window is greater than 1;
[0097] The sizes of any two sliding windows in the sliding window set are different from each other, and the size of any sliding window in the sliding window set is different from the size of the first sliding window.
[0098] In the above implementation, for each group of log information, the first sliding window is slid on the group of log information to extract suspicious information, so it can be understood that the size of the first sliding window will affect the extracted suspicious information. This implementation takes into account that more sliding windows of different sizes can be set to extract suspicious information, and more suspicious information can be placed in the error model library, which conveniently realizes the expansion of the error model library, and is conducive to improving the accuracy of subsequent log analysis.
[0099] Since different sliding windows differ only in size, when the i-th sliding window is selected, the principle of implementing suspicious information extraction can refer to the above description of the first sliding window, and will not be repeated here.
[0100] It should also be noted that, in this implementation, for any two sliding windows in the sliding window set, if the two have different lengths or different widths, they can be regarded as having different sizes.
[0101] For example, in one case, the sliding window set includes sliding windows of sizes 10×1, 10×2, and up to 10×x1, and also includes sliding windows of sizes 11×1, 11×2, and up to 11×x1. By analogy, it also includes sliding windows of sizes x2×1, x2×2, and up to x2×x1. That is, in this example, the sliding window set is set within the full size range of x2×x1 (the minimum value of x2 is 10). The x1 described here is the total number of rows of a single group of log information. For example, Figure 2 The total number of rows in the log information in the example is 9, and the x2 described here is the total number of columns in a single group of log information. For example, Figure 2 The total number of columns in the log information in the example is 19.
[0102] In a specific embodiment of the present invention, it may also include:
[0103] According to the classification principle that suspicious information with high similarity is classified into the same category, the suspicious information in the error model library is classified;
[0104] The classification principle includes: when the number of data included in two suspicious information is the same and the content of the included data is the same, the two suspicious information are classified into the same category;
[0105] For any one category, determine the number of occurrences of each suspicious information in the category in the error model library, and sum them up, and the sum result is used as the weight value used to reflect the importance of the category;
[0106] Collect server logs during server operation to obtain logs to be analyzed;
[0107] Matching the log to be analyzed with the error model library, when it is determined that the log to be analyzed matches any classification in the error model library, determining that the log to be analyzed is an abnormal log, and obtaining a weight value of the classification;
[0108] According to the preset corresponding relationship, determine the fault handling level corresponding to the weight value;
[0109] The corresponding fault handling operation is performed based on the fault handling level, and a prompt message carrying an abnormal log and the obtained weight value is output.
[0110] This implementation method takes into account that in the present application scheme, each suspicious information is placed in the error model library, thereby realizing the construction of the error model library. However, in the constructed error model library, there may be repeated suspicious information, and the more repeated, the more frequently the suspicious information appears in the error log. If the log to be analyzed matches such suspicious information, it means that the probability of the server currently running abnormally is very high, and even a more serious and obvious failure may have occurred, which requires more attention.
[0111] In this regard, in this implementation, suspicious information in the error model library will be classified according to the classification principle that suspicious information with high similarity is classified into the same category. It can be understood that there are many specific implementation methods for classifying suspicious information with high similarity into the same category, and this implementation provides a more convenient judgment standard for determining whether two suspicious information are highly similar, that is, if the number of data included in the two suspicious information is the same, and the data content included is the same, then the two suspicious information are suspicious information with high similarity and can be classified into the same category. For example Figure 5 In the example, the number of data included in suspicious information 1, suspicious information 2 and suspicious information 3 is 3, and the data contents included are a, d and f, so they are classified into the same category.
[0112] For example, in some cases, not only the quantity and content of the data included in two suspicious information are required to be consistent, but also the quantity, content and sequence of the data included in each line of the two are required to be consistent.
[0113] In other specific situations, more similarity determination methods can be added to accurately and effectively classify suspicious information with high similarity into the same category.
[0114] For any one category, in this implementation, the number of occurrences of each suspicious information in the category in the error model library will be determined and summed up, still using Figure 5 For example, Figure 5 The suspicious information 1 appears 50 times in the error model library. Figure 5 Suspicious information 2 appears 20 times in the error model library. Figure 5 The suspicious information 3 appears 35 times in the error model library, then 50+20+35=55, and 55 can be used as the weight value to reflect the importance of the classification.
[0115] Later, during the operation of the server, after the server logs are collected, they can be used as the logs to be analyzed. Then, the logs to be analyzed can be matched with the error model library. For example, in one case, it is determined that the log to be analyzed has Figure 5 The suspicious information 1 of the data structure can determine that the log to be analyzed matches the error model library, thereby determining that the log to be analyzed is an abnormal log. Figure 5 Suspicious information 1 in this classification matches, and since the weight value of this classification is 55, it is necessary to perform the corresponding fault handling operation according to the fault handling level corresponding to the weight value 55, and output a prompt message carrying the abnormal log and the obtained weight value so that the staff can pay attention to the situation. And from the above analysis, it can be seen that the larger the weight value, the higher the fault handling level, that is, the more clear it is that a fault has occurred, and it may be a more serious fault, and the staff needs to pay more attention to the current situation.
[0116] It should also be noted that there are many specific implementation methods for determining whether the log to be analyzed matches any category in the error model library. For example, a simple matching method is that if a data structure of a suspicious information appears in the log to be analyzed, it can be determined that the log to be analyzed matches the category to which the suspicious information in the error model library belongs. In addition, in some implementation methods, considering that the log to be analyzed is similar to the suspicious information in a category, but not completely consistent with any suspicious information in the category, it can also be considered that the log to be analyzed matches the category.
[0117] For example, in one case, for each suspicious information in any one category, the position of each data in the suspicious information will be determined, and then after obtaining the position of each data in each suspicious information, the position range of each data will be determined. In addition, when this implementation method is adopted, for any two suspicious information in the same category, it is usually specifically required that the number of data, data content and sequence included in each row of the two are consistent.
[0118] For easier understanding, please refer to Figure 5 For example, for each suspicious information in the classification, when determining the position of each data in suspicious information 1, for example, the position of the first data in the first row of suspicious information 1 can be recorded as (0, 0), so the position of each data in suspicious information 1 can be expressed as: a(0, 0), d(2, 0), f(1, -1). Similarly, for suspicious information 2, the position of the first data in the first row of suspicious information 2 can be recorded as (0, 0), so the position of each data in suspicious information 2 can be expressed as: a(0, 0), d(2, 0), f(-1, -1). Similarly, the position of each data in suspicious information 3 can be expressed as: a(0, 0), d(1, 0), f(1, -1)
[0119] After obtaining the position of each data in each suspicious information, the position range of each data in the suspicious information in the category can be obtained. For example, in the above example, after obtaining the position of each data in each suspicious information, the position range of each data content in the suspicious information in the category is expressed as: a(0, 0), d(1~2, 0), f(-1~1, -1), where 1~2 means that the horizontal coordinate of data d can be 1 to 2, and -1~1 means that the horizontal coordinate of data f can be -1 to 1.
[0120] After obtaining the respective position ranges of each data content in the suspicious information in the classification, for the log to be analyzed, if a certain structure appears, the structure matches the classification, and the positions of each numerical value in the structure are consistent with the respective position ranges of each data content in the suspicious information in the classification, then it can be considered that the structure appearing in the log to be analyzed matches the classification, that is, the log to be analyzed matches the classification in the error model library.
[0121] For example, the log to be analyzed appears Figure 5 The structure shown in the figure, the positions of the values in the structure can be expressed as: a(0,0), d(1,0), f(-1,-1). It can be seen that it meets the range of "a(0,0), d(1~2,0), f(-1~1,-1)" mentioned above. Therefore, although Figure 5 The structure in the log to be analyzed does not match Figure 5 The suspicious information 1, suspicious information 2 or suspicious information 3 are completely consistent, but it can be considered that the structure of the log to be analyzed is the same as Figure 5 This classification is matched. It can be seen that although this matching method is more complicated, it is conducive to more accurate log analysis and reduces the probability of missed judgment.
[0122] By applying the technical solution provided by the embodiment of the present invention, when the server runs in error, the error log is collected, and after obtaining M groups of log information, the extraction of suspicious information is realized by means of a sliding window. Specifically, the first sliding window is used, and at least one of its length and width is greater than 1, which means that the suspicious information obtained by the present application scheme is the suspicious information obtained by combining the context information of the error log, that is, the traditional one-dimensional keyword is expanded into two-dimensional suspicious information, with a wider coverage and higher accuracy. The amount of information contained in the constructed error model library is greatly increased, which is also conducive to improving the accuracy of subsequent log analysis. Specifically, since it is necessary to obtain suspicious information in combination with context information, for any group of log information in the M groups of log information, the group of log information needs to be a two-dimensional array of multiple rows and columns. Afterwards, for each group of log information, multiple window arrays can be intercepted from the group of log information by sliding the preset first sliding window on the group of log information, and then based on each window array intercepted from the group of log information by the first sliding window, the suspicious information is extracted by regularity analysis, and then each extracted suspicious information is placed in the error model library to realize the construction of the error model library.
[0123] To sum up, the suspicious information obtained by the present application scheme is the suspicious information obtained by combining the contextual information of the error log, that is, expanding the traditional one-dimensional keywords into two-dimensional suspicious information, with a wider coverage and higher accuracy. The amount of information contained in the constructed error model library is greatly increased, which is conducive to improving the accuracy of subsequent log analysis.
[0124] Corresponding to the above method embodiment, the embodiment of the present invention further provides a system for constructing a log error model library, which can be referred to in correspondence with the above.
[0125] See also Figure 6 FIG. 1 is a schematic diagram of a system for constructing a log error model library in the present invention, comprising:
[0126] The log information determination module 601 is used to collect error logs when an error occurs during server operation to obtain M groups of log information;
[0127] Wherein, for any set of log information, the set of log information is a two-dimensional array with multiple rows and columns; M is a positive integer;
[0128] The suspicious information extraction module 602 is used to extract multiple window arrays from each group of log information by sliding a preset first sliding window on the group of log information, and extract suspicious information based on each window array intercepted from the group of log information by the first sliding window;
[0129] The error model library construction module 603 is used to place each extracted suspicious information into the error model library to construct the error model library;
[0130] Among them, at least one of the length and the width of the first sliding window is greater than 1.
[0131] In a specific embodiment of the present invention, a noise reduction module is also included, which is used to:
[0132] For any extracted suspicious information, the suspicious information is denoised according to the preset denoising rules to remove meaningless data in the suspicious information.
[0133] In a specific embodiment of the present invention, the noise reduction module is specifically used for:
[0134] For any extracted suspicious information, punctuation marks, spaces and device codes in the suspicious information are removed to remove meaningless data in the suspicious information.
[0135] In a specific implementation of the present invention, the suspicious information extraction module 602 is specifically used to:
[0136] For each group of log information, a plurality of window arrays are intercepted from the group of log information by sliding a preset first sliding window on the group of log information;
[0137] Based on each window array intercepted from the group of log information through the first sliding window, a data structure with an appearance frequency higher than a preset frequency threshold is taken as one piece of suspicious information extracted.
[0138] In a specific embodiment of the present invention, a suspicious information discarding module is further included, which is used to:
[0139] For any extracted suspicious information, if the suspicious information only contains a single data, the suspicious information will be discarded.
[0140] In a specific embodiment of the present invention, the suspicious information extraction module 602 is further used to:
[0141] Selecting the second sliding window to the Nth sliding window in sequence from the preset sliding window set;
[0142] When the i-th sliding window is selected, for each group of log information, a plurality of window arrays are intercepted from the group of log information by sliding the i-th sliding window on the group of log information, and suspicious information is extracted based on each window array intercepted from the group of log information by the i-th sliding window;
[0143] Wherein, i is a positive integer and ranges from 2 to N, and N is a positive integer not less than 2; at least one of the length and width of the i-th sliding window is greater than 1;
[0144] The sizes of any two sliding windows in the sliding window set are different from each other, and the size of any sliding window in the sliding window set is different from the size of the first sliding window.
[0145] In a specific embodiment of the present invention, it also includes:
[0146] The suspicious information classification module is used to: classify the suspicious information in the error model library according to the classification principle that the suspicious information with high similarity is classified into the same category;
[0147] The classification principle includes: when the number of data included in two suspicious information is the same and the data content included is the same, the two suspicious information are classified into the same category;
[0148] The weight value calculation module is used to determine the number of occurrences of each suspicious information in any category in the error model library, and sum them up, and the sum result is used as the weight value used to reflect the importance of the category;
[0149] Analysis modules for:
[0150] Collect server logs during server operation to obtain logs to be analyzed;
[0151] Matching the log to be analyzed with the error model library, when it is determined that the log to be analyzed matches any classification in the error model library, determining that the log to be analyzed is an abnormal log, and obtaining a weight value of the classification;
[0152] According to the preset corresponding relationship, determine the fault handling level corresponding to the weight value;
[0153] The corresponding fault handling operation is performed based on the fault handling level, and a prompt message carrying an abnormal log and the obtained weight value is output.
[0154] Corresponding to the above method and system embodiments, an embodiment of the present invention further provides a device for constructing a log error model library, a computer-readable storage medium, and a computer program product, which can be referenced in correspondence with the above.
[0155] See also Figure 7 As shown, the device may include:
[0156] Memory 701, used for storing computer programs;
[0157] The processor 702 is used to execute a computer program to implement the steps of the method for constructing an error model library of a log in any of the above embodiments.
[0158] The computer program product includes a computer program / instruction, and when the computer program / instruction is executed by a processor, the steps of the method for constructing a log error model library in any of the above embodiments are implemented.
[0159] See also Figure 8 The computer readable storage medium 80 stores a computer program 81, and when the computer program 81 is executed by the processor, the steps of the method for constructing the error model library of the log in any of the above embodiments are implemented. The computer readable storage medium 80 mentioned here includes RAM (Random Access Memory), memory, ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), register, hard disk, removable disk, or any other form of storage medium known in the technical field.
[0160] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0161] Professionals may further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention, and the description of the above embodiments is only used to help understand the technical solution and its core idea of the present invention. It should be pointed out that for ordinary technicians in the technical field, without departing from the principle of the present invention, the present invention can also be improved and modified in a number of ways, and these improvements and modifications also fall within the scope of protection of the present invention.
Claims
1. A method for constructing a log error model library, characterized in that: include: When an error occurs during server operation, error logs are collected to obtain M groups of log information; Wherein, for any group of log information, the group of log information is a two-dimensional array of multiple rows and columns; M is a positive integer; For each group of log information, a plurality of window arrays are intercepted from the group of log information by sliding a preset first sliding window on the group of log information, and suspicious information is extracted based on each of the window arrays intercepted from the group of log information by the first sliding window; Each of the extracted suspicious information is placed in an error model library to construct the error model library; Wherein, at least one of the length and the width of the first sliding window is greater than 1.
2. The method for constructing an error model library of a log according to claim 1, characterized in that: Also includes: For any one of the extracted suspicious information, denoising is performed on the suspicious information according to a preset denoising rule to remove meaningless data in the suspicious information.
3. The method for constructing an error model library of a log according to claim 2, characterized in that: For any one of the extracted suspicious information, denoising the suspicious information according to a preset denoising rule to remove meaningless data in the suspicious information, including: For any one of the extracted suspicious information, punctuation marks, spaces and device codes in the suspicious information are removed to remove meaningless data in the suspicious information.
4. The method for constructing an error model library of a log according to claim 1, characterized in that: For each group of the log information, a plurality of window arrays are intercepted from the group of log information by sliding a preset first sliding window on the group of log information, and suspicious information is extracted based on each of the window arrays intercepted from the group of log information by the first sliding window, including: For each group of log information, a plurality of window arrays are intercepted from the group of log information by sliding a preset first sliding window on the group of log information; Based on each of the window arrays intercepted from the group of log information through the first sliding window, a data structure with an appearance frequency higher than a preset frequency threshold is taken as the extracted suspicious information.
5. The method for constructing an error model library of a log according to claim 1, characterized in that: Also includes: For any one of the extracted suspicious information, when the suspicious information only contains a single data, the suspicious information is discarded.
6. The method for constructing an error model library of a log according to claim 1, characterized in that: Also includes: Selecting the second sliding window to the Nth sliding window in sequence from the preset sliding window set; When the i-th sliding window is selected, for each group of log information, a plurality of window arrays are intercepted from the group of log information by sliding the i-th sliding window on the group of log information, and suspicious information is extracted based on each of the window arrays intercepted from the group of log information by the i-th sliding window; Wherein, i is a positive integer and ranges from 2 to N in sequence, and N is a positive integer not less than 2; at least one of the length and width of the i-th sliding window is greater than 1; The sizes of any two sliding windows in the sliding window set are different from each other, and the size of any sliding window in the sliding window set is different from the size of the first sliding window.
7. The method for constructing a log error model library according to any one of claims 1 to 6, characterized in that: Also includes: Classifying the suspicious information in the error model library according to the classification principle that suspicious information with high similarity is classified into the same category; The classification principle includes: when the number of data included in two suspicious information is consistent and the data content included is consistent, the two suspicious information are classified into the same category; For any one category, determine the number of occurrences of each suspicious information in the category in the error model library, and sum them up, and use the sum result as a weight value for reflecting the importance of the category; Collecting logs of the server during the operation of the server as logs to be analyzed; Matching the log to be analyzed with the error model library, and when it is determined that the log to be analyzed matches any classification in the error model library, determining that the log to be analyzed is an abnormal log, and obtaining a weight value of the classification; According to the preset corresponding relationship, determining the fault handling level corresponding to the weight value; A corresponding fault handling operation is performed based on the fault handling level, and prompt information carrying the abnormal log and the obtained weight value is output.
8. A system for constructing a log error model library, characterized in that: include: The log information determination module is used to collect error logs when an error occurs during server operation to obtain M groups of log information; Wherein, for any group of log information, the group of log information is a two-dimensional array of multiple rows and columns; M is a positive integer; A suspicious information extraction module, configured to extract multiple window arrays from each group of log information by sliding a preset first sliding window on the group of log information, and extract suspicious information based on each window array intercepted from the group of log information by the first sliding window; An error model library construction module, used for placing each of the extracted suspicious information into the error model library to construct the error model library; Wherein, at least one of the length and the width of the first sliding window is greater than 1.
9. A device for constructing a log error model library, characterized in that: include: Memory for storing computer programs; A processor is used to execute the computer program to implement the steps of the method for constructing an error model library of a log as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for constructing an error model library of a log as described in any one of claims 1 to 7 are implemented.
