A log collection method, device and storage medium based on time series

Abstract

The present invention relates to a log collection method, device and storage medium based on time series. Time series log data is analyzed to obtain abnormal values ​​of each module of the server, and the abnormal values ​​are arranged in time series to form an abnormal value sequence; the number of previous log data related to the current log data of the module is obtained through the Bayesian information criterion and the variance of the abnormal value sequence, and this number is the autocorrelation order; for any module, relevant log data is collected according to its autocorrelation order; a corresponding total weight is configured for each module through a preset weight configuration strategy, and relevant log data of different orders and the total weight are combined to form target log data. The target log data supplements the correlation of module log data over time, provides more complete and accurate logs for the subsequent diagnostic process, avoids inaccurate diagnostic results caused by diagnosis based on single real-time log data, improves the accuracy of diagnosis, and the total weight can provide the focus of relevant log data.

Description

Technical Field

[0001] The present invention relates to the field of log collection, and in particular to a time series-based log collection method, device, and storage medium. Background Art

[0002] Fault diagnosis based on application server log data provides an important reference for server operation and maintenance to find server anomalies. Accurate and timely server diagnosis results can greatly improve operation and maintenance efficiency and save manpower and time costs.

[0003] The accuracy of server fault diagnosis results depends on whether the log files uploaded to the diagnostic system are sufficient and effective. In current technology, one diagnostic method is to upload the currently collected single log data to the diagnostic system and use the single log data for diagnosis. The logs collected in this way only focus on the real-time data of the current server, such as temperature. This diagnostic method ignores the correlation between log data, and the analysis of log data that lacks correlation has poor diagnostic accuracy. Another diagnostic method extracts log data according to a fixed number of rows or a fixed number of days. The log data volume is large, and there may be redundant or insufficient information, which may lead to inaccurate log data analysis results. In addition, the large amount of log data makes it difficult to accurately find the focus of attention, and the targetedness is poor. Summary of the Invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present invention provides a time series-based log collection method, device and storage medium.

[0005] In a first aspect, the present invention provides a time series-based log collection method, comprising:

[0006] Analyze the time series log data to obtain abnormal values ​​of each module of the server, and arrange the abnormal values ​​in time series to form an abnormal value sequence;

[0007] The number of previous log data related to the current log data of the module is obtained by using the Bayesian information criterion and the variance of the outlier sequence of each module. This number is the autocorrelation order.

[0008] For any module, relevant log data are collected according to its autocorrelation order;

[0009] The corresponding total weight is configured for each module through the preset weight configuration strategy, and the relevant log data of different orders and the total weight are combined to form the target log data.

[0010] Furthermore, t log data covering the logs of each module in the server are collected sequentially at unit time intervals; each log data is diagnosed by the diagnostic system to obtain t diagnostic results, and each diagnostic result records the abnormal value of each module.

[0011] Furthermore, the number of preceding log data related to the current log data of the module is obtained through the Bayesian information criterion and the variance of the outlier sequence of each module, including: substituting the variance of the module outlier sequence into the Bayesian information criterion formula to calculate several BIC values, and taking the time series of the diagnosis result corresponding to the first minimum BIC value as the number of preceding log data related to the current log data of the module, that is, the autocorrelation order of the module.

[0012] Furthermore, the corresponding total weight is configured for each module through the preset weight configuration strategy, including:

[0013] Based on the autocorrelation order of any module, a corresponding autoregressive model is established for the module, and the error in the autoregressive model of the time series is used to form an error sequence;

[0014] The preset weight configuration strategy assigns a first weight and a second weight to each module according to the significance test statistic value of the autoregressive model and the white noise test statistic value of the error sequence;

[0015] The total weight is obtained by multiplying the first weight, the second weight and the current outlier value.

[0016] Furthermore, based on the autocorrelation order of any module, the corresponding autoregressive model is established for the module:

[0017] E ti =β1E t-1,i +β2E t-2,i +…+β p E t-p,i +ε t , where i represents the module, p is the autocorrelation order, ε t The error when building the autoregressive model for the tth outlier is the error sequence {ε1, ε2…ε t}, β={β1,β2,…,β p} is the coefficient set of the autoregressive model, and its elements are the coefficients in the autoregressive model. The value of each element in the coefficient set satisfies the minimum sum of squares of the residuals between the true outliers and the outliers estimated by the autoregressive model.

[0018] Furthermore, the preset weight configuration strategy assigns a corresponding first weight to each module according to the significance test statistic value of the autoregressive model, including:

[0019] Preset significance test statistic threshold;

[0020] Perform significance test on the autoregressive model of each module and obtain k significance test statistics;

[0021] Compare whether the significance test statistic value of the i-th module is greater than the preset significance test statistic threshold;

[0022] If so, according to Configuring a corresponding first weight for the module;

[0023] Otherwise, follow Configure the corresponding first weight for the module, where Q is the significance test statistic value.

[0024] Furthermore, the preset weight configuration strategy assigns a corresponding second weight to each module according to the white noise test statistic value of the error sequence, including:

[0025] Preset white noise test statistic threshold;

[0026] Perform white noise test on the error sequence of each module to obtain k white noise test statistic values;

[0027] Compare whether the white noise test statistic value of the i-th module is greater than the preset white noise test statistic threshold;

[0028] If so, according to Configuring a corresponding second weight for the module;

[0029] Otherwise, follow A corresponding second weight is configured for the module, where A is the white noise test statistic value.

[0030] Furthermore, for any module, when collecting relevant log data according to its autocorrelation order, previous log data is collected according to its autocorrelation order, and the previous log data and current log data are combined as relevant log data.

[0031] In a second aspect, the present invention provides a time series-based log collection device, comprising: a processing unit, a bus unit, and a storage unit, wherein the bus unit connects the storage unit and the processing unit, and the storage unit stores a computer program, which implements the time series-based log collection method when executed by the processing unit.

[0032] In a third aspect, the present invention provides a storage medium for implementing a time series-based log collection method, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the time series-based log collection method is implemented.

[0033] The above technical solution provided by the embodiment of the present invention has the following advantages compared with the prior art:

[0034] The present invention studies log data and diagnostic results, starting from the abnormal values ​​of the diagnostic results of each module of the server, analyzes the correlation between the abnormal values ​​of each module and its historical data, establishes an autoregressive model, and uses a preset weight configuration strategy to assign a first weight and a second weight to the historical log data of each module according to the significance test statistic value and the white noise test statistic value of the autoregressive model. The first weight and the second weight are combined with the current abnormal value to obtain a total weight of the corresponding autocorrelation order. The target log data is formed by integrating the relevant log data of different orders and the total weight. The target log data supplements the correlation of the module log data over time, provides a more complete and accurate log for the subsequent diagnostic process, avoids the inaccurate diagnostic results caused by diagnosis based on a single real-time log data, and improves the accuracy of the diagnosis. The total weight can also provide the focus of the relevant log data. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0037] Figure 1 A flow chart of a time series-based log collection method provided by an embodiment of the present invention;

[0038] Figure 2 A flowchart of obtaining the number of preceding log data related to the current log data of a module by using the Bayesian Information Criterion and the variance of the abnormal value sequence of each module provided by an embodiment of the present invention;

[0039] Figure 3 A flowchart of configuring the corresponding total weight for each module through a preset weight configuration strategy provided in an embodiment of the present invention;

[0040] Figure 4 A flow chart of a weight configuration strategy provided in an embodiment of the present invention for configuring a first weight for each module based on a comparison result of each significance test statistic value with a preset significance test statistic threshold;

[0041] Figure 5 A flow chart of configuring a second weight for each module according to a comparison result of each white noise test statistic value and a preset white noise test statistic threshold value according to a weight configuration strategy provided in an embodiment of the present invention;

[0042] Figure 6 A schematic diagram of a time series-based log collection device is provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0043] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0044] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0045] Example 1

[0046] See Figure 1 As shown, the present invention provides a log collection method based on time series, comprising:

[0047] S100, analyzing the time series log data to obtain abnormal values ​​of each module of the server, and arranging the abnormal values ​​of each module in time series to form an abnormal value sequence.

[0048] In the specific implementation process, S100 includes: using a log collection tool to collect t log data covering logs of various modules in the server at unit time intervals, forming a log data set {L1, L2...L t}, each log data in the log data set is arranged in time sequence, and each log data L in the log data set i The logs of each module of the server are covered. Assume that there are k modules in the server. The diagnostic system diagnoses the log data set and obtains t diagnostic results. Each log data is parsed according to the module. The diagnostic system analyzes the parsed log data and extracts the abnormal values ​​of each module in the diagnostic results corresponding to each log data, resulting in a t×k abnormal value matrix:

[0049]

[0050] The columns of the outlier matrix are the outlier sequences of each module, such as {E 11, E 21 ...E t1}.

[0051] S200, using the Bayesian information criterion and the variance of each module's outlier sequence to obtain the number of previous log data related to the module's current log data, which is the autocorrelation order. The outliers of each module obtained by diagnosing the current log data are affected by the previous log data. According to the theory of the time series autoregressive model, when the partial autocorrelation coefficient is truncated at the pth order, it is considered that the current log data has a strong correlation with the p previous log data. In the specific implementation process, please refer to Figure 2 As shown, S200 includes:

[0052] S201, calculate the variance of the outlier sequence of each module;

[0053] S202, the variance of the module outlier sequence is substituted into the Bayesian Information Criterion formula to obtain several BIC values. Specifically, k modules have corresponding k outlier sequences, and the variance of k outlier sequences is obtained. The variance of each outlier sequence is substituted into Where i represents the module, i=1,2...k, q represents the time sequence of the diagnosis result, q=1,2...t, is the variance of the outlier sequence of the ith module. This formula can be used to obtain t BIC values.

[0054] S203: Take the time series of the diagnosis result corresponding to the first minimum BIC value as the autocorrelation order of the module, that is, take the q corresponding to the first minimum BIC value as the autocorrelation order p value.

[0055] S300 , for any module, collect relevant log data according to its autocorrelation order; specifically, for any module, collect previous log data according to its autocorrelation order, and combine the previous log data and current log data as relevant log data.

[0056] S400: configuring a corresponding total weight for each module through a preset weight configuration strategy.

[0057] In the specific implementation process, please refer to Figure 3 As shown, S400 includes:

[0058] Based on the autocorrelation order of any module, a corresponding autoregressive model is established for the module, and the error in the time series autoregressive model is used to form an error sequence. Specifically, based on the autocorrelation order of any module, a corresponding autoregressive model is established for the module as follows:

[0059] E ti =β1E t-1,i +β2E t-2,i +…+β pE t-p,i +ε t , where i represents the module, p is the autocorrelation order, ε t The error when building the autoregressive model for the tth outlier is the error sequence {ε1, ε2…ε t}, β={β1,β2,…,β p} is the coefficient set of the autoregressive model, and its elements are the coefficients in the autoregressive model. The value of each element in the coefficient set satisfies the minimum sum of squares of the residuals between the true outliers and the outliers estimated by the autoregressive model.

[0060] The preset weight allocation strategy assigns a first weight and a second weight to each module based on the significance test statistic value of the autoregressive model and the white noise test statistic value of the error sequence. The weight allocation strategy allocates a first weight to each module based on the comparison result of each significance test statistic value with a preset significance test statistic threshold, and allocates a second weight to each module based on the comparison result of each white noise test statistic value with a preset white noise test statistic threshold.

[0061] In the specific implementation process, please refer to Figure 4 As shown, the weight configuration strategy configures a first weight for each module according to the comparison result of each significance test statistic value with a preset significance test statistic threshold, including:

[0062] S410: Preset a significance test statistic threshold.

[0063] S420: Perform a significance test on the autoregressive model of each module to obtain k significance test statistic values.

[0064] S430, traverse each module and compare whether the significance test statistic value of the traversed i-th module is greater than a preset significance test statistic threshold; if yes, execute S440, otherwise execute S450.

[0065] S440, add 1 to the normalized significance test statistic value: As the first weight of this module.

[0066] S450, subtract the normalized significance test statistic from 1: As the first weight corresponding to the module configuration.

[0067] Where Q is the significance test statistic value. In the specific implementation process, a feasible significance test statistic threshold value is 2. If the significance test statistic value of the i-th module autoregressive model is greater than 2, then according to Configure the corresponding first weight W for this module 1i, otherwise, according to Configure the corresponding first weight W for this module 1i , where Q is the significance test statistic.

[0068] See Figure 5 As shown, the weight configuration strategy configures the second weight for each module according to the comparison result of each white noise test statistic value and the preset white noise test statistic threshold value, including:

[0069] S401, preset a white noise test statistic threshold;

[0070] S402, performing a white noise test on the error sequence of each module to obtain k white noise test statistics;

[0071] S403, traverse each module and compare whether the white noise test statistic value of the traversed i-th module is greater than a preset white noise test statistic threshold; if yes, execute S404, otherwise execute S405.

[0072] S404, add 1 to the normalized white noise test statistic value, that is: As the second weight of this module.

[0073] S405, subtract the normalized white noise test statistic from 1, that is: As the second weight of this module.

[0074] Where A is the white noise test statistic value. In the specific implementation process, a feasible threshold value of the white noise test statistic is 0.05. If the white noise test statistic value of the i-th module autoregressive model is greater than 0.05, then Configure the corresponding second weight W for this module 2i , otherwise, according to Configure the corresponding second weight W for this module 2i , where A is the white noise test statistic value.

[0075] The total weight is obtained by multiplying the first weight, the second weight and the current outlier value. Specifically, the total weight of each module is calculated: W i =W 1i ×W 2i ×E i ,i=1,2,…,k, where W i is the total weight of the i-th module, E i is the abnormal value of the i-th module in the current log data.

[0076] S500: Target log data is formed using the total weight of the modules and related log data.

[0077] The present invention studies log data and diagnostic results, starting from the abnormal values ​​of the diagnostic results of each module of the server, analyzes the correlation between the abnormal values ​​of each module and its historical data, establishes an autoregressive model, and uses a preset weight configuration strategy to assign a first weight and a second weight to the historical log data of each module according to the significance test statistic value and the white noise test statistic value of the autoregressive model. The first weight and the second weight are combined with the current abnormal value to obtain a total weight of the corresponding autocorrelation order. The target log data is formed by integrating the relevant log data of different orders and the total weight. The target log data supplements the correlation of the module log data over time, provides a more complete and accurate log for the subsequent diagnostic process, avoids the inaccurate diagnostic results caused by diagnosis based on a single real-time log data, and improves the accuracy of the diagnosis. The total weight can also provide the focus of the relevant log data.

[0078] Example 2

[0079] See Figure 6 As shown, an embodiment of the present invention provides a time series-based log collection device, including: a processing unit, a bus unit, and a storage unit. The bus unit connects the storage unit and the processing unit. The storage unit stores a computer program. When the computer program is executed by the processing unit, the time series-based log collection method is implemented.

[0080] Example 3

[0081] An embodiment of the present invention provides a storage medium for implementing a time series-based log collection method. The storage medium stores a computer program. When the computer program is executed by a processor, the time series-based log collection method is implemented.

[0082] In the embodiments provided by the present invention, it should be understood that the disclosed structures and methods can be implemented in other ways. For example, the structural embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, structure or unit, which can be electrical, mechanical or other forms.

[0083] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0084] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0085] The foregoing description is intended only to provide specific embodiments of the present invention, which will enable those skilled in the art to understand and implement the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not intended to be limited to the embodiments shown herein, but is intended to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A log collection method based on time series, characterized in that: include: Analyze the time series log data to obtain abnormal values ​​of each module of the server, and arrange the abnormal values ​​in time series to form an abnormal value sequence; The number of previous log data related to the current log data of the module is obtained by using the Bayesian Information Criterion and the variance of the outlier sequence of each module, including: substituting the variance of the outlier sequence of the module into the Bayesian Information Criterion formula to calculate several BIC values, and taking the time series of the diagnosis result corresponding to the first minimum BIC value as the number of previous log data related to the current log data of the module, that is, the autocorrelation order of the module; For any module, relevant log data are collected according to its autocorrelation order; A corresponding total weight is configured for each module through a preset weight configuration strategy, including: establishing a corresponding autoregressive model for any module based on the autocorrelation order of the module, and using the errors in the time series autoregressive model to form an error sequence; the preset weight configuration strategy assigns a first weight and a second weight to each module based on the significance test statistic value of the autoregressive model and the white noise test statistic value of the error sequence; wherein the preset weight configuration strategy assigns a corresponding first weight to each module based on the significance test statistic value of the autoregressive model, including: presetting a significance test statistic threshold; performing a significance check on the autoregressive model of each module to obtain k significance test statistic values; traversing each module and comparing whether the significance test statistic value of the traversed i-th module is greater than the preset significance test statistic threshold; If so, according to Configuring a corresponding first weight for the module; Otherwise, follow Configure the corresponding first weight for the module, where is the significance test statistic value; The preset weight configuration strategy assigns a corresponding second weight to each module according to the white noise test statistic value of the error sequence, including: Preset a white noise test statistic threshold; perform a white noise test on the error sequence of each module to obtain k white noise test statistic values; traverse each module and compare whether the white noise test statistic value of the traversed i-th module is greater than the preset white noise test statistic threshold; If so, according to Configuring a corresponding second weight for the module; Otherwise, follow Configure the corresponding second weight for the module, where is the white noise test statistic value; The total weight is obtained by multiplying the first weight, the second weight and the current outlier value; The relevant log data of different orders and the total weight are combined to form the target log data.

2. The time series-based log collection method according to claim 1, characterized in that: T log data covering the logs of each module in the server are collected in a time series with unit time intervals; each log data is diagnosed by the diagnostic system to obtain t diagnostic results, and each diagnostic result records the abnormal value of each module.

3. The log collection method based on time series according to claim 1, characterized in that: Based on the autocorrelation order of any module, the corresponding autoregressive model is established for the module: , where i represents the module and p is the autocorrelation order. The error when building an autoregressive model for the tth outlier is { , … }, The coefficient set of the autoregressive model, whose elements are the coefficients in the autoregressive model, and the values ​​of each element in the coefficient set satisfy the minimum sum of squares of the residuals of the true outliers and the outliers estimated by the autoregressive model .

4. The log collection method based on time series according to claim 1, characterized in that: For any module, when collecting relevant log data according to its autocorrelation order, the previous log data is collected according to its autocorrelation order, and the previous log data and the current log data are combined as the relevant log data.

5. A log collection device based on time series, characterized in that: include: A processing unit, a bus unit, and a storage unit, wherein the bus unit connects the storage unit and the processing unit, and the storage unit stores a computer program. When the computer program is executed by the processing unit, the time series-based log collection method according to any one of claims 1 to 4 is implemented.

6. A storage medium for implementing a time series-based log collection method, wherein the storage medium stores a computer program, characterized in that: When the computer program is executed by a processor, the time series-based log collection method according to any one of claims 1 to 4 is implemented.

Images