A log data anomaly detection method, device, equipment and medium

Abstract

A log data anomaly detection method, device, equipment and medium are disclosed. The method comprises: collecting each log data of a target system in a target time interval; dividing the target time interval into multiple sub-intervals according to corresponding time windows; for each sub-interval, determining whether the parameter value of a key parameter in the log data in the sub-interval is abnormal according to the interquartile range of the parameter value of the key parameter, and the upper and lower abnormal limit values corresponding to the interquartile range; determining an abnormal mode label for the sub-interval with an abnormality detection result; and providing the abnormality detection result and detection associated information to a user through an abnormality detection page. The embodiments of the present application can automatically collect log data of a system in a specified time interval, reasonably divide the time interval into multiple sub-intervals, and quickly and accurately determine whether the parameter value of a key parameter in the log data in each sub-interval is abnormal.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method, apparatus, device, and medium for detecting log data anomalies. Background Technology

[0002] Enterprises typically have multiple systems of different types. These systems can include computer systems, network systems, and industrial control systems that provide specific functions. During operation, these systems periodically monitor various performance parameters, obtaining their values ​​and generating log data to record the operational process. Each log entry contains time and parameter information. The parameter information includes the name and value of each performance parameter. For any system requiring stability evaluation, the enterprise's technical personnel typically assess system stability over a specified time period by checking for anomalies in the values ​​of key parameters in the log data. Key parameters are those performance parameters that can be used to evaluate system stability.

[0003] In related technologies, a common log data anomaly detection scheme involves technicians manually collecting log data from the system within a specified time interval, dividing the time interval into multiple sub-intervals, and determining whether the parameter values ​​of key parameters in the log data within each sub-interval are abnormal. This log data anomaly detection scheme relies on manual operation, resulting in high time and labor costs, and its accuracy is difficult to guarantee. Summary of the Invention

[0004] This invention provides a method, apparatus, device, and medium for detecting log data anomalies, in order to solve the problems that log data anomaly detection schemes in related technologies rely on manual operation, which has high time and labor costs and makes it difficult to guarantee accuracy.

[0005] According to one aspect of the present invention, a method for detecting log data anomalies is provided, comprising:

[0006] After receiving the log anomaly detection request sent by the user corresponding to the target system, the target time interval of the log anomaly detection request is determined, and the log data of the target system within the target time interval is collected.

[0007] Based on the time windows corresponding to each time period in the target time interval, the target time interval is divided into multiple sub-intervals, and the log data in each sub-interval is determined based on the time information of each log data.

[0008] For each sub-interval, based on the upper quartile, lower quartile, kurtosis, and skewness of the key parameter values ​​in the log data within the sub-interval, the interquartile range of the key parameter values ​​in the log data within the sub-interval, as well as the upper and lower limits of the anomaly corresponding to the interquartile range, are determined. Based on the interquartile range, the upper limit of the anomaly, and the lower limit of the anomaly, it is determined whether there are any anomalies in the parameter values ​​of the key parameters in the log data within the sub-interval, and anomaly detection results for the sub-interval are generated.

[0009] For each sub-interval where the anomaly detection result is abnormal, a feature vector of the sub-interval is constructed based on the interquartile range of the key parameters and related parameters in the log data within the sub-interval. Based on the feature vector, the anomaly pattern label of the sub-interval is determined.

[0010] The anomaly detection page provides the user with the anomaly detection results and related information for each sub-range.

[0011] According to another aspect of the present invention, a log data anomaly detection device is provided, comprising:

[0012] The data acquisition module is used to determine the target time interval of the log anomaly detection request after receiving a log anomaly detection request sent by a user corresponding to the target system, and to collect various log data of the target system within the target time interval.

[0013] The interval division module is used to divide the target time interval into multiple sub-intervals according to the time windows corresponding to each time period in the target time interval, and to determine the log data in each sub-interval according to the time information of each log data.

[0014] The anomaly detection module is used to determine the interquartile range of the key parameter values ​​in the log data of the sub-interval based on the upper quartile, lower quartile, kurtosis, and skewness of the parameter values ​​of the key parameters in the log data of the sub-interval, as well as the upper and lower limits of the anomaly corresponding to the interquartile range. Based on the interquartile range, the upper limit of the anomaly, and the lower limit of the anomaly, it determines whether there are anomalies in the parameter values ​​of the key parameters in the log data of the sub-interval, and generates the anomaly detection result of the sub-interval.

[0015] The label determination module is used to construct a feature vector for each sub-interval where the anomaly detection result is abnormal, based on the interquartile range of the parameter values ​​of key parameters and related parameters in the log data within the sub-interval, and to determine the anomaly pattern label of the sub-interval based on the feature vector.

[0016] The results provision module is used to provide the user with the anomaly detection results and detection association information for each sub-interval through the anomaly detection page.

[0017] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0018] At least one processor;

[0019] and a memory communicatively connected to the at least one processor;

[0020] The memory stores a computer program that is executed by the at least one processor, which enables the at least one processor to perform the log data anomaly detection method according to any embodiment of the present invention.

[0021] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the log data anomaly detection method according to any embodiment of the present invention.

[0022] According to another aspect of the present invention, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, implements the log data anomaly detection method according to any embodiment of the present invention.

[0023] The technical solution of this invention involves, after receiving a log anomaly detection request sent by a user corresponding to the target system, collecting various log data of the target system within a target time interval; then, dividing the target time interval into multiple sub-intervals according to the time windows corresponding to various time periods within the target time interval; determining the log data within each sub-interval based on the time information of each log data; and for each sub-interval, determining the interquartile range of the key parameter values ​​in the log data within the sub-interval based on the upper quartile, lower quartile, kurtosis, and skewness of the parameter values. The system identifies the upper and lower limits of anomalies corresponding to the interquartile range (ICM). Based on these limits, it determines whether the values ​​of key parameters in the log data within a sub-interval are abnormal, generating an anomaly detection result for that sub-interval. For each sub-interval with an anomaly detection result, a feature vector is constructed based on the ICM of the key parameters and related parameters in the log data within that sub-interval. The anomaly pattern label for the sub-interval is then determined based on this feature vector. Finally, the anomaly detection results and associated information for each sub-interval are provided to the user through an anomaly detection page, thus addressing the log... Traditional data anomaly detection solutions rely on manual operation, resulting in high time and labor costs and difficulty in guaranteeing accuracy. This solution addresses these issues by automatically collecting log data within a specified time interval after receiving a log anomaly detection request. Then, based on the time windows corresponding to different time periods within the interval, the system divides the time interval into multiple sub-intervals. For each sub-interval, the system automatically determines whether the key parameter values ​​are abnormal based on the interquartile range (ICM) of the log data, along with the corresponding upper and lower limits for anomalies. This generates anomaly detection results for each sub-interval. Based on feature vectors, the system identifies anomaly pattern labels for sub-intervals with abnormal detection results. The results and correlation information for each sub-interval are then provided to the user through a visual interface. This approach automates the collection of log data within a specified time interval, rationally divides the time interval into multiple sub-intervals, and quickly and accurately determines whether the key parameter values ​​are abnormal within each sub-interval. This reduces the time and labor costs of log data anomaly detection and improves its accuracy.

[0024] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart of a log data anomaly detection method provided in Embodiment 1 of the present invention.

[0027] Figure 2 This is a flowchart of a log data anomaly detection method provided in Embodiment 2 of the present invention.

[0028] Figure 3 This is a schematic diagram of the structure of a log data anomaly detection device provided in Embodiment 3 of the present invention.

[0029] Figure 4 A schematic diagram of the structure of an electronic device for implementing the log data anomaly detection method of this invention. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "target," "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising," "including," and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] Example 1

[0033] Figure 1This is a flowchart illustrating a log data anomaly detection method according to Embodiment 1 of the present invention. This embodiment is applicable to detecting whether the parameter values ​​of key parameters in the log data of an enterprise system are abnormal within a specified time interval. This method can be executed by a log data anomaly detection device, which can be implemented in hardware and / or software and can be configured in an electronic device installed within the enterprise. The electronic device can be an electronic device installed within the enterprise for detecting whether the parameter values ​​of key parameters in the log data of the enterprise system are abnormal within a specified time interval. For example... Figure 1 As shown, the method includes:

[0034] Step 101: After receiving the log anomaly detection request sent by the user corresponding to the target system, determine the target time interval of the log anomaly detection request, and collect the log data of the target system within the target time interval.

[0035] Optionally, the systems set up within an enterprise may include computer systems, network systems, and industrial control systems for providing specified functions. The system's performance parameters may be multiple parameters used to evaluate system performance. These performance parameters include, but are not limited to, system response time, system utilization, number of system threads, and number of bytes received by the system. System response time can refer to the time it takes for the system to respond to an external input request. System utilization can refer to the utilization rate of the central processing unit (CPU) in the system. The number of system threads can refer to the total number of running threads in the system. The number of bytes received by the system can refer to the number of bytes of data the system has received. The parameter name for each performance parameter can be a pre-defined text used to uniquely identify the performance parameter. The parameter value for each performance parameter can be a specific numerical value of the detected performance parameter. A key parameter among the performance parameters can refer to one of the parameters used to evaluate the stability of the system. For example, the key parameter among the performance parameters is system response time.

[0036] Optionally, each system can periodically detect various performance parameters at preset time intervals to obtain the parameter values ​​of each performance parameter. The preset time interval can be a pre-set time interval. For example, the preset time interval can be 1 second, 10 seconds, or 15 seconds. After each detection of the parameter value of each performance parameter, the system can arrange the parameter name and parameter value of each performance parameter in a preset order to obtain a sequence. Separators are inserted between the parameter names and parameter values ​​of each performance parameter in the sequence, and time information is added to the beginning of the sequence. The sequence with separators and time information is defined as log data and stored in the system log data file. The time information can be the time recorded by the system when the detection of each performance parameter began during this detection process. The system log data file can be a file set in the system for storing system log data. Each log data contains time information and parameter information. The parameter information contains the parameter name and parameter value of each performance parameter.

[0037] Optionally, the user can be a technical staff member responsible for managing the enterprise's systems. For each system, the corresponding log anomaly detection request can be information indicating whether there are any anomalies in the parameter values ​​of key parameters in the log data of the enterprise's system within a user-specified time interval. The user can send the system-specific log anomaly detection request via a terminal device to an electronic device used to detect whether there are any anomalies in the parameter values ​​of key parameters in the log data of the enterprise's system within a specified time interval, instructing the electronic device to detect whether there are any anomalies in the parameter values ​​of key parameters in the system's log data within the user-specified time interval. The target time interval for the log anomaly detection request is the user-specified time interval. The target time interval is typically from 0:00 to 24:00 on a specified date. Each hour within the target time interval is a time period. The target time interval contains 24 time periods. The log anomaly detection request includes the start and end times of the target time interval. The target time interval can be determined based on the start and end times.

[0038] Optionally, it can detect whether the electronic device has received a log anomaly detection request corresponding to each system. Upon detecting that the electronic device has received a log anomaly detection request corresponding to the target system, it acquires the log anomaly detection request received by the electronic device, thereby obtaining the log anomaly detection request sent by the user corresponding to the target system. Then, it determines the target time interval for the log anomaly detection request, collects various log data from the target system within the target time interval, and stores each log data, its time information, and parameter information in a preset storage component. The target system can be any system within the enterprise that needs to have its stability evaluated. For example, the target system can be a computer system, network system, or industrial control system set up within the enterprise to provide specified functions.

[0039] Optionally, after receiving the log anomaly detection request sent by the user corresponding to the target system, the start and end times of the target time interval of the log anomaly detection request can be extracted from the log anomaly detection request, and the target time interval of the log anomaly detection request can be determined based on the start and end times of the target time interval of the log anomaly detection request.

[0040] Optionally, the log data of the target system within the target time interval can refer to the log data whose time information stored in the system log data file of the target system is included within the target time interval. It is possible to detect whether the time information in each log data stored in the system log data file of the target system is included within the target time interval, and to obtain the log data whose time information is included within the target time interval, thereby collecting the log data of the target system within the target time interval.

[0041] Optionally, for each log data point from the target system within a target time interval, the time information in the log data can be identified and extracted to obtain the log data's time information, and the format of the log data's time information can be converted to a target time format. The target time format can be a pre-set time format. For example, the time information of a log data point in the target time format is "2011-09-12T13:00:42.000Z". The separators between parameter names and values ​​of each performance parameter in the log data, as well as the parameter names and values ​​of each performance parameter, can be identified and extracted to obtain the parameter information of the log data. The log data, its time information, and parameter information can be stored in a pre-defined storage component.

[0042] Optionally, the preset storage component can be a software or hardware module within the electronic device used to store log data that needs to be detected for anomalies. The data in the preset storage component can be cleared after each log data detection process, facilitating the storage of log data that needs to be detected for anomalies in the next log data detection process.

[0043] Step 102: Divide the target time interval into multiple sub-intervals according to the time windows corresponding to each time period in the target time interval, and determine the log data in each sub-interval according to the time information of each log data.

[0044] Optionally, the target time interval is divided into multiple sub-intervals based on time windows corresponding to each time period within the target time interval, including: determining the time period type of each time period within the target time interval; wherein the time period type is a peak business period, a normal business period, or a low business period; for a time period of peak business period type, a first time window is determined as the time window corresponding to the time period, and the time period is divided into multiple sub-intervals based on the time window corresponding to the time period; for a time period of normal business period type, a second time window is determined as the time window corresponding to the time period, and the time period is divided into multiple sub-intervals based on the time window corresponding to the time period; for a time period of low business period type, a third time window is determined as the time window corresponding to the time period, and the time period is divided into multiple sub-intervals based on the time window corresponding to the time period; wherein the first time window is smaller than the second time window, and the second time window is smaller than the third time window.

[0045] Optionally, for each time period within the target time interval, the corresponding time window is a preset duration suitable for dividing the time period into multiple sub-intervals. The time period can be divided into multiple sub-intervals based on the corresponding time window. The duration of each sub-interval obtained from the time period division is the same. The time period type can be information used to characterize whether the time period is a period of high system load. A time period type of "peak business period" indicates that the time period is a period of high system load. A time period type of "normal business period" indicates that the time period is a period of normal system load. A time period type of "low business period" indicates that the time period is a period of low system load. The first time window, second time window, and third time window are three preset durations. The first time window is a suitable preset duration for dividing a time period of "peak business period" into multiple sub-intervals. The second time window is a suitable preset duration for dividing a time period of "normal business period" into multiple sub-intervals. The third time window is a suitable preset duration for dividing a time period of "low business period" into multiple sub-intervals. The first time window is smaller than the second time window, and the second time window is smaller than the third time window.

[0046] Optionally, the peak business hours within the target time interval are periods with high system load. Peak business hours can be 9:00 AM to 6:00 PM or 12:00 PM to 6:00 PM. Periods falling within the peak business hours are those within the target time interval. The off-peak business hours within the target time interval are periods with low system load. Off-peak business hours can be 12:00 AM to 5:00 AM. Periods falling within the off-peak business hours are those within the peak business hours. Periods within the target time interval that are neither peak nor off-peak are those with normal system load. Periods not falling within either peak or off-peak business hours are considered periods that are neither peak nor off-peak.

[0047] Optionally, determining the time period type of each time period within the target time interval includes: detecting whether each time period within the target time interval belongs to a peak business period or a low business period; for time periods belonging to a peak business period, determining the time period type as a peak business period; for time periods belonging to a low business period, determining the time period type as a low business period; and for time periods that do not belong to either a peak business period or a low business period, determining the time period type as a normal business period.

[0048] Optionally, for time periods classified as peak business hours, the first time window can be defined as the time window corresponding to the time period, and the time period can be divided into multiple sub-intervals based on the time window corresponding to the time period. For example, the first time window is 5 minutes. Starting from the start time of the time period, every 5 minutes can be defined as a sub-interval, dividing the time period into 12 sub-intervals with a duration of 5 minutes.

[0049] Optionally, for time periods classified as normal business periods, the second time window can be defined as the time window corresponding to the time period, and the time period can be divided into multiple sub-intervals based on the time window corresponding to the time period. For example, the second time window is 10 minutes. Starting from the start time of the time period, each 10 minutes can be defined as a sub-interval, dividing the time period into 6 sub-intervals with a duration of 10 minutes.

[0050] Optionally, for time periods classified as low-traffic periods, a third time window can be defined as the time window corresponding to the time period, and the time period can be divided into multiple sub-intervals based on the time window corresponding to the time period. For example, the third time window is 30 minutes. Starting from the start time of the time period, a sub-interval can be defined every 30 minutes, thus dividing the time period into two sub-intervals of 30 minutes each.

[0051] Therefore, based on a suitable time window, each period in the target time interval is divided into multiple sub-intervals, thus dividing the target time interval into multiple sub-intervals.

[0052] Optionally, for each sub-interval, the log data within the sub-interval can refer to log data whose time information falls within the sub-interval. Determining the log data within each sub-interval based on the time information of each log data includes: for each sub-interval, identifying the log data whose time information falls within the sub-interval as the log data within that sub-interval. For each sub-interval, it is possible to detect whether the time information of each log data in the target system within the target time interval falls within the sub-interval, and identify the log data whose time information falls within the sub-interval as the log data within that sub-interval.

[0053] Step 103: For each sub-interval, based on the upper quartile, lower quartile, kurtosis, and skewness of the key parameter values ​​in the log data within the sub-interval, determine the interquartile range of the key parameter values ​​in the log data within the sub-interval, as well as the upper and lower limits of the anomaly corresponding to the interquartile range. Based on the interquartile range, the upper limit of the anomaly, and the lower limit of the anomaly, determine whether there are any anomalies in the parameter values ​​of the key parameters in the log data within the sub-interval, and generate the anomaly detection result for the sub-interval.

[0054] Optionally, for each sub-interval, the parameter values ​​of key parameters can be extracted from the parameter information of each log data within the identified sub-interval. The upper quartile of the key parameter values ​​within the sub-interval can be defined as the quartile of the extracted key parameter values ​​after sorting them from smallest to largest. The location parameter values, and the individual parameter values ​​of the extracted key parameters. The parameter value is less than or equal to the parameter value. The lower quartile of the key parameter values ​​in the log data within a sub-interval can be defined as the quartile of the extracted key parameter values ​​after being arranged in ascending order. The location parameter values, and the individual parameter values ​​of the extracted key parameters. The parameter value is less than or equal to the parameter value. The kurtosis of the key parameter values ​​in the log data within a sub-interval can refer to the kurtosis of the dataset composed of the individual parameter values ​​of the extracted key parameters. The skewness of the key parameter values ​​in the log data within a sub-interval can refer to the skewness of the dataset composed of the individual parameter values ​​of the extracted key parameters. The interquartile range of the key parameter values ​​in the log data within a sub-interval can refer to the difference between the upper and lower quartiles of the key parameter values ​​in the log data within the sub-interval. The interquartile range of the key parameter values ​​in the log data within a sub-interval can be used to measure the data dispersion of the dataset composed of the individual parameter values ​​of the extracted key parameters. The anomaly detection result of the sub-interval can be used to characterize whether there are anomalies in the parameter values ​​of the key parameters in the log data within the sub-interval. The anomaly detection result of the sub-interval is normal or abnormal. A normal anomaly detection result indicates that there are no anomalies in the parameter values ​​of the key parameters in the log data within the sub-interval. An abnormal anomaly detection result indicates that there are anomalies in the parameter values ​​of the key parameters in the log data within the sub-interval.

[0055] Optionally, based on the upper quartile, lower quartile, kurtosis, and skewness of the key parameter values ​​in the log data within the sub-interval, the interquartile range (IQM) and the corresponding upper and lower limits of anomalies are determined. This includes: determining the upper quartile, lower quartile, kurtosis, and skewness of the key parameter values ​​in the log data within the sub-interval; determining the IQM based on the upper and lower quartiles; determining the anomaly limit calculation coefficient corresponding to the IQM based on the kurtosis and skewness; substituting the upper quartile, the anomaly limit calculation coefficient, and the IQM into a preset formula for calculating the upper limit of anomalies to obtain the upper limit of anomalies corresponding to the IQM; and substituting the lower quartile, the anomaly limit calculation coefficient, and the IQM into a preset formula for calculating the lower limit of anomalies to obtain the lower limit of anomalies corresponding to the IQM.

[0056] Optionally, determining the upper quartile, lower quartile, kurtosis, and skewness of the key parameter values ​​in the log data within the sub-interval includes: extracting the parameter values ​​of the key parameters from each log data within the determined sub-interval, and then inputting the extracted parameter values ​​of the key parameters into the calculation component to obtain the upper quartile, lower quartile, kurtosis, and skewness of the key parameter values ​​in the log data within the sub-interval output by the calculation component.

[0057] Optionally, the electronic device includes a computing component. This computing component can be a software or hardware module used to calculate the upper quartile, lower quartile, kurtosis, and skewness of the key parameters in the log data within the sub-interval, based on the extracted values ​​of key parameters from each log data point identified as a sub-interval. For each sub-interval, the values ​​of the key parameters in each log data point within the sub-interval can be extracted. These extracted values ​​are then input into the computing component, which calculates the upper quartile, lower quartile, kurtosis, and skewness of the key parameters in the log data within the sub-interval based on these values. Finally, the computing component outputs these values. It can obtain the upper quartile, lower quartile, kurtosis, and skewness of the parameter values ​​of key parameters in the log data within the sub-interval output by the calculation component, thereby determining the upper quartile, lower quartile, kurtosis, and skewness of the parameter values ​​of key parameters in the log data within the sub-interval.

[0058] Optionally, determining the interquartile range (IQR) of the key parameter values ​​in the log data within the sub-interval based on the upper and lower quartiles includes: calculating the difference between the upper and lower quartiles of the key parameter values ​​in the log data within the sub-interval to obtain the IQR of the key parameter values ​​in the log data within the sub-interval. The IQR of the key parameter values ​​in the log data within the sub-interval can be expressed as: IQR = Q3 - Q1. Where IQR is the IQR of the key parameter values ​​in the log data within the sub-interval, Q3 is the upper quartile of the key parameter values ​​in the log data within the sub-interval, and Q1 is the lower quartile of the key parameter values ​​in the log data within the sub-interval.

[0059] Optionally, the upper limit of the anomaly corresponding to the interquartile range of the key parameter values ​​in the log data within the sub-interval is a value that the interquartile range of the key parameter values ​​in the log data within the sub-interval must be less than when there are no anomalies in the key parameter values. The lower limit of the anomaly corresponding to the interquartile range of the key parameter values ​​in the log data within the sub-interval is a value that the interquartile range of the key parameter values ​​in the log data within the sub-interval must be greater than when there are no anomalies in the key parameter values. Generally, if the interquartile range of the key parameter values ​​in the log data within the sub-interval is greater than the upper limit of the anomaly corresponding to the interquartile range of the key parameter values ​​in the log data within the sub-interval, or if the interquartile range of the key parameter values ​​in the log data within the sub-interval is less than the lower limit of the anomaly corresponding to the interquartile range of the key parameter values ​​in the log data within the sub-interval, it can be determined that there is an anomaly in the key parameter values ​​in the log data within the sub-interval. If the interquartile range of the key parameter values ​​in the log data within the sub-interval is less than or equal to the upper limit of the anomaly corresponding to the interquartile range of the key parameter values ​​in the log data within the sub-interval, and the interquartile range of the key parameter values ​​in the log data within the sub-interval is greater than or equal to the lower limit of the anomaly corresponding to the interquartile range of the key parameter values ​​in the log data within the sub-interval, it can be determined that the key parameter values ​​in the log data within the sub-interval are not abnormal.

[0060] Optionally, the electronic device is equipped with preset formulas for calculating upper and lower limits of anomalies. The preset upper limit formula is used to calculate the upper limit of anomalies corresponding to the interquartile range of the parameter values ​​of key parameters in the log data within the sub-interval. The preset lower limit formula is used to calculate the lower limit of anomalies corresponding to the interquartile range of the parameter values ​​of key parameters in the log data within the sub-interval. The anomaly limit calculation coefficient corresponding to the interquartile range of the parameter values ​​of key parameters in the log data within the sub-interval is a multiplier coefficient used when calculating the upper and lower limits of anomalies corresponding to the interquartile range of the parameter values ​​of key parameters in the log data within the sub-interval based on the preset upper and lower limit formulas.

[0061] Optionally, the preset formula for calculating the upper limit of abnormal values ​​can be: The preset formula for calculating the lower limit of anomalies can be: Where S1 is the upper limit of the anomaly corresponding to the interquartile range of the key parameter values ​​in the log data within the sub-interval, S2 is the lower limit of the anomaly corresponding to the interquartile range of the key parameter values ​​in the log data within the sub-interval, IQR is the interquartile range of the key parameter values ​​in the log data within the sub-interval, Q3 is the upper quartile of the key parameter values ​​in the log data within the sub-interval, Q1 is the lower quartile of the key parameter values ​​in the log data within the sub-interval, and k is the anomaly limit calculation coefficient corresponding to the interquartile range of the key parameter values ​​in the log data within the sub-interval.

[0062] Optionally, the kurtosis and skewness of the key parameter values ​​in the log data within the sub-interval can be used to determine the appropriate outlier limit calculation coefficient corresponding to the interquartile range of the key parameter values ​​in the log data within the sub-interval. Typically, when the kurtosis of the key parameter values ​​in the log data within the sub-interval is within a preset kurtosis threshold range and the skewness of the key parameter values ​​in the log data within the sub-interval is within a preset skewness threshold range, the appropriate outlier limit calculation coefficient corresponding to the interquartile range of the key parameter values ​​in the log data within the sub-interval is 1.5. The preset kurtosis threshold range can be a pre-set numerical range. Kurtosis within the preset kurtosis threshold range means that the kurtosis is less than the upper limit of the preset kurtosis threshold range and greater than the lower limit of the preset kurtosis threshold range. The preset skewness threshold range can also be a pre-set numerical range. Skewness within the preset skewness threshold range means that the skewness is less than the upper limit of the preset skewness threshold range and greater than the lower limit of the preset skewness threshold range. When the kurtosis of the key parameter values ​​in the log data within a sub-interval exceeds the upper limit of the preset kurtosis threshold range, it indicates that there are many extreme values ​​among the key parameter values ​​in the log data within the sub-interval. It is necessary to appropriately increase the anomaly limit calculation coefficient corresponding to the interquartile range of the key parameter values ​​in the log data within the sub-interval to ensure the accuracy of the calculated upper and lower anomaly limits. This ensures that the anomaly of the key parameter values ​​in the log data within the sub-interval can be accurately detected based on the calculated upper and lower anomaly limits. The appropriate anomaly limit calculation coefficient corresponding to the interquartile range of the key parameter values ​​in the log data within the sub-interval is greater than 1.5. When the kurtosis of the key parameter values ​​in the log data within a sub-interval is less than the lower limit of the preset kurtosis threshold range, and the skewness of the key parameter values ​​in the log data within the sub-interval is less than the lower limit of the preset skewness threshold range, it indicates that the data distribution of the key parameter values ​​in the log data within the sub-interval is very concentrated and symmetrical. It is necessary to appropriately reduce the anomaly limit calculation coefficient corresponding to the interquartile range of the key parameter values ​​in the log data within the sub-interval to ensure the accuracy of the calculated upper and lower anomaly limits. This ensures that the anomaly of the key parameter values ​​in the log data within the sub-interval can be accurately detected based on the calculated upper and lower anomaly limits. The appropriate anomaly limit calculation coefficient corresponding to the interquartile range of the key parameter values ​​in the log data within the sub-interval is less than 1.5.

[0063] Optionally, the electronic device includes a pre-trained coefficient detection model. This model can be a machine learning model used to analyze and detect the kurtosis and skewness of key parameter values ​​in log data within a sub-interval, determining suitable outlier limit calculation coefficients corresponding to the interquartile range of the key parameter values ​​within the sub-interval. The input to the coefficient detection model is the kurtosis and skewness of the key parameter values ​​in the log data within the sub-interval. The output of the coefficient detection model is the outlier limit calculation coefficient corresponding to the interquartile range of the key parameter values ​​within the sub-interval. The outlier limit calculation coefficient output by the coefficient detection model, corresponding to the interquartile range of the key parameter values ​​within the sub-interval, is the suitable outlier limit calculation coefficient determined by the coefficient detection model based on the kurtosis and skewness of the key parameter values ​​within the sub-interval. After inputting the kurtosis and skewness of key parameter values ​​from the log data within a sub-interval into the coefficient detection model, the model analyzes and detects these values ​​to determine suitable outlier limit calculation coefficients corresponding to the interquartile range of the key parameter values ​​within the sub-interval. The model then outputs these outlier limit calculation coefficients. The outlier limit calculation coefficients corresponding to the interquartile range of the key parameter values ​​within the sub-interval can be obtained from the coefficient detection model. The coefficient detection model can be pre-trained and configured in electronic devices by the company's technical personnel.

[0064] Optionally, determining the anomaly limit calculation coefficient corresponding to the interquartile range based on the kurtosis and skewness includes: inputting the kurtosis and skewness into a pre-trained coefficient detection model to obtain the anomaly limit calculation coefficient output by the coefficient detection model; wherein the input of the coefficient detection model is the kurtosis and skewness of the parameter values ​​of key parameters in the log data within the sub-interval, and the output of the coefficient detection model is the anomaly limit calculation coefficient corresponding to the interquartile range of the parameter values ​​of key parameters in the log data within the sub-interval. The kurtosis and skewness of the parameter values ​​of key parameters in the log data within the sub-interval can be input into the coefficient detection model. The coefficient detection model analyzes and detects the kurtosis and skewness of the parameter values ​​of key parameters in the log data within the sub-interval, determines a suitable anomaly limit calculation coefficient corresponding to the interquartile range of the parameter values ​​of key parameters in the log data within the sub-interval, and then outputs the suitable anomaly limit calculation coefficient corresponding to the interquartile range of the parameter values ​​of key parameters in the log data within the sub-interval. The outlier calculation coefficients corresponding to the interquartile range of the key parameter values ​​in the log data within the sub-interval can be obtained from the coefficient detection model output. This allows us to determine the outlier calculation coefficients corresponding to the interquartile range of the key parameter values ​​in the log data within the sub-interval.

[0065] Optionally, for each sub-interval, the upper quartile of the key parameter value in the log data within the sub-interval, the anomaly limit calculation coefficient corresponding to the interquartile distance of the key parameter value in the log data within the sub-interval, and the interquartile distance of the key parameter value in the log data within the sub-interval can be substituted into a preset anomaly upper limit calculation formula to calculate the anomaly upper limit value corresponding to the interquartile distance of the key parameter value in the log data within the sub-interval, thereby determining the anomaly upper limit value corresponding to the interquartile distance of the key parameter value in the log data within the sub-interval.

[0066] Optionally, for each sub-interval, the lower quartile of the key parameter value in the log data within the sub-interval, the anomaly limit calculation coefficient corresponding to the interquartile distance of the key parameter value in the log data within the sub-interval, and the interquartile distance of the key parameter value in the log data within the sub-interval can be substituted into a preset anomaly lower limit calculation formula to calculate the anomaly lower limit value corresponding to the interquartile distance of the key parameter value in the log data within the sub-interval, thereby determining the anomaly lower limit value corresponding to the interquartile distance of the key parameter value in the log data within the sub-interval.

[0067] Therefore, based on kurtosis and skewness, the outlier limit calculation coefficient corresponding to the interquartile range can be dynamically calculated. Then, based on the interquartile range and the outlier limit calculation coefficient corresponding to the interquartile range, the dynamic upper and lower outlier limits corresponding to the interquartile range can be calculated.

[0068] Optionally, based on the interquartile range, the upper limit of the anomaly index, and the lower limit of the anomaly index, it is determined whether the parameter values ​​of key parameters in the log data within the sub-interval are abnormal, and an anomaly detection result for the sub-interval is generated. This includes: detecting whether the interquartile range is greater than the upper limit of the anomaly index and whether it is less than the lower limit of the anomaly index; if the interquartile range is greater than the upper limit of the anomaly index or less than the lower limit of the anomaly index, it is determined that the parameter values ​​of key parameters in the log data within the sub-interval are abnormal, and the anomaly detection result for the sub-interval is determined to be abnormal; if the interquartile range is less than or equal to the upper limit of the anomaly index and the interquartile range is greater than or equal to the lower limit of the anomaly index, it is determined that the parameter values ​​of key parameters in the log data within the sub-interval are not abnormal, and the anomaly detection result for the sub-interval is determined to be normal.

[0069] Step 104: For each sub-interval where the anomaly detection result is abnormal, construct a feature vector for the sub-interval based on the interquartile range of the key parameters and related parameters in the log data within the sub-interval, and determine the anomaly pattern label of the sub-interval based on the feature vector.

[0070] Optionally, for each sub-interval where the anomaly detection result is abnormal, the anomaly pattern label for the sub-interval can be text describing the direct cause of the abnormal parameter value in the log data within the sub-interval. The electronic device stores multiple preset anomaly pattern labels. Each preset anomaly pattern label is text describing various causes of the abnormal parameter value in the log data within the sub-interval. The causes described by each preset anomaly pattern label are different. Each preset anomaly pattern label is specifically used to describe one cause that leads to the abnormal parameter value in the log data within the sub-interval. For example, the first anomaly pattern label, the second anomaly pattern label, and the third anomaly pattern label are three preset anomaly pattern labels. The first anomaly pattern label is resource bottleneck. The second anomaly pattern label is network jitter. The third anomaly pattern label is slow query or code error.

[0071] Optionally, the relevant parameters can refer to multiple performance parameters that are related to the key parameters. For each sub-interval where the anomaly detection result is abnormal, the feature vector of the sub-interval can be a vector that can be used to characterize the abnormality of the parameter values ​​of the key parameters in the log data within the sub-interval.

[0072] Optionally, the key parameter is the system response time, and the related parameters include system utilization, number of system threads, and number of system input bytes. Based on the interquartile range of the key and related parameter values ​​in the log data within the sub-interval, a feature vector for the sub-interval is constructed, including: determining the interquartile range of the parameter values ​​for system utilization, number of system threads, and number of system input bytes in the log data within the sub-interval; and constructing the feature vector for the sub-interval based on the interquartile range of the parameter values ​​for system response time, system utilization, number of system threads, and number of system input bytes in the log data within the sub-interval.

[0073] Optionally, the key performance parameter is system response time. Related parameters that are interrelated with this key parameter include system utilization, number of system threads, and number of bytes passed into the system.

[0074] Optionally, determining the interquartile range of the system utilization parameter values ​​in the log data within the sub-interval includes: extracting the system utilization parameter values ​​from each log data within the determined sub-interval, then inputting the extracted system utilization parameter values ​​into the calculation component to obtain the upper quartile, lower quartile, kurtosis, and skewness of the system utilization parameter values ​​in the log data within the sub-interval output by the calculation component; calculating the difference between the upper quartile and lower quartile of the system utilization parameter values ​​in the log data within the sub-interval to obtain the interquartile range of the system utilization parameter values ​​in the log data within the sub-interval.

[0075] Optionally, determining the interquartile range of the system thread count parameter value in the log data within the sub-interval includes: extracting the system thread count parameter value from each log data within the determined sub-interval, then inputting the extracted system thread count parameter values ​​into the calculation component to obtain the upper quartile, lower quartile, kurtosis, and skewness of the system thread count parameter value in the log data within the sub-interval output by the calculation component; calculating the difference between the upper quartile and lower quartile of the system thread count parameter value in the log data within the sub-interval to obtain the interquartile range of the system thread count parameter value in the log data within the sub-interval.

[0076] Optionally, determining the interquartile range of the system-input byte count parameter value in the log data within the sub-interval includes: extracting the system-input byte count parameter value from each log data identified as the sub-interval, then inputting the extracted system-input byte count parameter values ​​into the calculation component to obtain the upper quartile, lower quartile, kurtosis, and skewness of the system-input byte count parameter value in the log data within the sub-interval output by the calculation component; calculating the difference between the upper quartile and lower quartile of the system-input byte count parameter value in the log data within the sub-interval to obtain the interquartile range of the system-input byte count parameter value in the log data within the sub-interval.

[0077] Optionally, for each sub-interval where the anomaly detection result is abnormal, a 4-dimensional row vector can be constructed based on the interquartile range of the parameter values ​​for system response time, system utilization, number of system threads, and number of system-input bytes in the log data within the sub-interval. The first dimension of the row vector is the interquartile range of the system response time parameter value in the log data within the sub-interval; the second dimension is the interquartile range of the system utilization parameter value; the third dimension is the interquartile range of the number of system threads parameter value; and the fourth dimension is the interquartile range of the number of system-input bytes parameter value. This constructed 4-dimensional row vector can be used to characterize the abnormal parameter values ​​of key parameters in the log data within the sub-interval. This 4-dimensional row vector can be determined as the feature vector of the sub-interval, thus constructing the feature vector of the sub-interval.

[0078] Optionally, the electronic device is equipped with a pre-trained label detection model. The label detection model can be a machine learning model used to analyze and detect the feature vectors of sub-intervals where the anomaly detection result is abnormal, and select a preset anomaly pattern label from various preset anomaly pattern labels that can describe the direct cause of the abnormal parameter values ​​in the log data within the sub-interval as the anomaly pattern label for the sub-interval. The input to the label detection model is the feature vector of the sub-interval where the anomaly detection result is abnormal. The output of the label detection model is the anomaly pattern label of the sub-interval where the anomaly detection result is abnormal. After the feature vector of the sub-interval where the anomaly detection result is abnormal is input into the label detection model, the model analyzes and detects the feature vector of the sub-interval, selects a preset anomaly pattern label from various preset anomaly pattern labels that can describe the direct cause of the abnormal parameter values ​​in the log data within the sub-interval as the anomaly pattern label for the sub-interval, and then outputs the anomaly pattern label of the sub-interval. The anomaly pattern label of the sub-interval output by the label detection model can be obtained. The label detection model can be pre-trained and installed in the electronic device by the enterprise's technical personnel.

[0079] Optionally, determining the anomaly pattern label of a sub-interval based on the feature vector includes: inputting the feature vector into a pre-trained label detection model to obtain the anomaly pattern label output by the label detection model; wherein the input of the label detection model is the feature vector of the sub-interval with anomaly detection result, and the output of the label detection model is the anomaly pattern label of the sub-interval with anomaly detection result. For each sub-interval with anomaly detection result, the feature vector of the sub-interval can be input into the label detection model. The label detection model analyzes and detects the feature vector of the sub-interval, selects a preset anomaly pattern label from various preset anomaly pattern labels that can be used to describe the direct cause of the abnormal parameter value of the key parameter in the log data within the sub-interval as the anomaly pattern label of the sub-interval, and then outputs the anomaly pattern label of the sub-interval. The anomaly pattern label of the sub-interval output by the label detection model can be obtained, thereby determining the anomaly pattern label of the sub-interval.

[0080] Optionally, after determining the abnormal pattern label of the sub-interval based on the feature vector, the method further includes: outputting alarm information corresponding to the abnormal pattern label.

[0081] Optionally, the electronic device stores alarm information, anomaly type information, inferred root cause information, impact scope information, and suggested measures information corresponding to each preset anomaly pattern label. For each preset anomaly pattern label, the alarm information corresponding to the preset anomaly pattern label can be text indicating that the parameter value of a key parameter is abnormal due to the cause described by the preset anomaly pattern label. The alarm information corresponding to the preset anomaly pattern label can include inferred root cause information corresponding to the preset anomaly pattern label. The anomaly type information corresponding to the preset anomaly pattern label can be text describing the type of anomaly in the parameter value of a key parameter in the log data within the sub-interval where the anomaly pattern label is the preset anomaly pattern label. For example, the anomaly type could be a resource bottleneck. The inferred root cause information corresponding to the preset anomaly pattern label can be text describing the root cause of the abnormal parameter value in the log data within the sub-interval where the anomaly pattern label is the preset anomaly pattern label. For example, the root cause of the anomaly could be a CPU overload. The impact scope information corresponding to the preset anomaly pattern label can be text describing the impact scope of the abnormal parameter value in the log data within the sub-interval where the anomaly pattern label is the preset anomaly pattern label. For example, the scope of an anomaly's impact can be... The request is for external input. The suggested measures information corresponding to the preset exception mode label can be text describing the resolution measures for exceptions where key parameter values ​​exist in the log data within the sub-interval of the preset exception mode label. For example, the exception resolution measure could be a suggestion to expand capacity. The exception resolution measure could also be checking for high-consumption processes.

[0082] Optionally, for each sub-interval where the anomaly detection result is abnormal, the anomaly pattern label for the sub-interval is a preset anomaly pattern label selected from various preset anomaly pattern labels. This preset label describes the direct cause of the abnormal parameter value in the log data within the sub-interval. After determining the anomaly pattern label for the sub-interval based on its feature vector, the alarm information corresponding to the anomaly pattern label can be sent to the user's terminal device, thereby outputting the alarm information corresponding to the anomaly pattern label for the sub-interval.

[0083] Step 105: Provide the user with the anomaly detection results and detection association information for each sub-interval through the anomaly detection page.

[0084] Optionally, the anomaly detection page can be a page set up in an electronic device for user interaction, providing users with information related to the anomaly detection process. The anomaly detection results and associated information for each sub-range can be displayed on the anomaly detection page, thus providing the user with the anomaly detection results and associated information for each sub-range.

[0085] Optionally, for sub-intervals where the anomaly detection results are normal, the detection correlation information for each sub-interval may include the interquartile range of the key parameter values ​​in the log data within the sub-interval, and the anomaly limit calculation coefficient corresponding to the interquartile range of the key parameter values ​​in the log data within the sub-interval. For sub-intervals where the anomaly detection results are abnormal, the detection correlation information for each sub-interval may include the interquartile range of the key parameter values ​​in the log data within the sub-interval, the anomaly limit calculation coefficient corresponding to the interquartile range of the key parameter values ​​in the log data within the sub-interval, the anomaly pattern label of the sub-interval, the anomaly type information corresponding to the anomaly pattern label of the sub-interval, the inferred root cause information corresponding to the anomaly pattern label of the sub-interval, the impact range information corresponding to the anomaly pattern label of the sub-interval, and the suggested measures information corresponding to the anomaly pattern label of the sub-interval. The anomaly detection results and detection correlation information of each sub-interval constitute the detection report of this anomaly detection process, which can provide decision support for users, reveal potential risks and performance bottlenecks of the target system, and provide data-driven decision-making basis for capacity planning, architecture optimization, and performance tuning of the target system.

[0086] The technical solution of this invention involves, after receiving a log anomaly detection request sent by a user corresponding to the target system, collecting various log data of the target system within a target time interval; then, dividing the target time interval into multiple sub-intervals according to the time windows corresponding to various time periods within the target time interval; determining the log data within each sub-interval based on the time information of each log data; and for each sub-interval, determining the interquartile range of the key parameter values ​​in the log data within the sub-interval based on the upper quartile, lower quartile, kurtosis, and skewness of the parameter values. The system identifies the upper and lower limits of anomalies corresponding to the interquartile range (ICM). Based on these limits, it determines whether the values ​​of key parameters in the log data within a sub-interval are abnormal, generating an anomaly detection result for that sub-interval. For each sub-interval with an anomaly detection result, a feature vector is constructed based on the ICM of the key parameters and related parameters in the log data within that sub-interval. The anomaly pattern label for the sub-interval is then determined based on this feature vector. Finally, the anomaly detection results and associated information for each sub-interval are provided to the user through an anomaly detection page, thus addressing the log... Traditional data anomaly detection solutions rely on manual operation, resulting in high time and labor costs and difficulty in guaranteeing accuracy. This solution addresses these issues by automatically collecting log data within a specified time interval after receiving a log anomaly detection request. Then, based on the time windows corresponding to different time periods within the interval, the system divides the time interval into multiple sub-intervals. For each sub-interval, the system automatically determines whether the key parameter values ​​are abnormal based on the interquartile range (ICM) of the log data, along with the corresponding upper and lower limits for anomalies. This generates anomaly detection results for each sub-interval. Based on feature vectors, the system identifies anomaly pattern labels for sub-intervals with abnormal detection results. The results and correlation information for each sub-interval are then provided to the user through a visual interface. This approach automates the collection of log data within a specified time interval, rationally divides the time interval into multiple sub-intervals, and quickly and accurately determines whether the key parameter values ​​are abnormal within each sub-interval. This reduces the time and labor costs of log data anomaly detection and improves its accuracy.

[0087] The technical solution of this invention can reasonably divide a time interval into multiple sub-intervals according to the time windows corresponding to each time period in the time interval, thereby avoiding false alarms and missed alarms caused by dividing a time interval into multiple sub-intervals according to a fixed time window.

[0088] The technical solution of this invention can dynamically calculate the anomaly limit calculation coefficient corresponding to the interquartile range based on kurtosis and skewness. Then, based on the interquartile range and the anomaly limit calculation coefficient corresponding to the interquartile range, it can calculate the dynamic upper and lower anomaly limits corresponding to the interquartile range. By using the interquartile range of the parameter values ​​of key parameters in the log data within the sub-interval, and the dynamically calculated upper and lower anomaly limits corresponding to the interquartile range, it can quickly and accurately determine whether there are anomalies in the parameter values ​​of key parameters in the log data within the sub-interval, thereby improving the detection accuracy and enabling the system to provide just the right sensitivity for both volatile and stable business operations.

[0089] Example 2

[0090] Figure 2 This is a flowchart illustrating a log data anomaly detection method provided in Embodiment 2 of the present invention. This embodiment of the present invention can be combined with various optional solutions from one or more of the above embodiments. For example... Figure 2 As shown, the method includes:

[0091] Step 201: After receiving the log anomaly detection request sent by the user corresponding to the target system, determine the target time interval of the log anomaly detection request, and collect the log data of the target system within the target time interval.

[0092] Step 202: Divide the target time interval into multiple sub-intervals according to the time windows corresponding to each time period in the target time interval, and determine the log data in each sub-interval according to the time information of each log data.

[0093] Step 203: For each sub-interval, based on the upper quartile, lower quartile, kurtosis, and skewness of the key parameter values ​​in the log data within the sub-interval, determine the interquartile range of the key parameter values ​​in the log data within the sub-interval, as well as the upper and lower limits of the anomaly corresponding to the interquartile range. Based on the interquartile range, the upper limit of the anomaly, and the lower limit of the anomaly, determine whether there are any anomalies in the parameter values ​​of the key parameters in the log data within the sub-interval, and generate the anomaly detection result for the sub-interval.

[0094] Step 204: For each sub-interval where the anomaly detection result is abnormal, construct a feature vector for the sub-interval based on the interquartile range of the key parameters and related parameters in the log data within the sub-interval. Input the feature vector into a pre-trained label detection model to obtain the anomaly pattern label output by the label detection model, and output alarm information corresponding to the anomaly pattern label.

[0095] Step 205: Provide the user with the anomaly detection results and detection association information for each sub-interval through the anomaly detection page.

[0096] The technical solution of this invention can determine the abnormal pattern label of the sub-interval where the abnormal detection result is abnormal based on the label detection model and feature vector, and output the alarm information corresponding to the abnormal pattern label.

[0097] Example 3

[0098] Figure 3 This is a schematic diagram of a log data anomaly detection device provided in Embodiment 3 of the present invention. The device can be configured in an electronic device. Figure 3 As shown, the device includes: a data acquisition module 301, an interval division module 302, an anomaly detection module 303, a label determination module 304, and a result provision module 305.

[0099] The data acquisition module 301 is used to, after receiving a log anomaly detection request sent by a user corresponding to the target system, determine the target time interval of the log anomaly detection request and collect various log data of the target system within the target time interval; the interval division module 302 is used to divide the target time interval into multiple sub-intervals according to the time windows corresponding to various time periods in the target time interval, and determine the log data in each sub-interval according to the time information of each log data; the anomaly detection module 303 is used to, for each sub-interval, determine the key parameters in the log data within the sub-interval based on the upper quartile, lower quartile, kurtosis, and skewness of the parameter values. The system uses the interquartile range of key parameter values, along with corresponding upper and lower anomaly limits, to determine whether the key parameter values ​​in the log data within a sub-interval are abnormal, generating an anomaly detection result for that sub-interval. A label determination module 304, for each sub-interval with an anomaly detection result, constructs a feature vector for that sub-interval based on the interquartile range of key and related parameter values ​​in the log data within that sub-interval, and determines the anomaly pattern label for that sub-interval based on the feature vector. A result providing module 305 provides the anomaly detection results and associated information for each sub-interval to the user through an anomaly detection page.

[0100] The technical solution of this invention involves, after receiving a log anomaly detection request sent by a user corresponding to the target system, collecting various log data of the target system within a target time interval; then, dividing the target time interval into multiple sub-intervals according to the time windows corresponding to various time periods within the target time interval; determining the log data within each sub-interval based on the time information of each log data; and for each sub-interval, determining the interquartile range of the key parameter values ​​in the log data within the sub-interval based on the upper quartile, lower quartile, kurtosis, and skewness of the parameter values. The system identifies the upper and lower limits of anomalies corresponding to the interquartile range (ICM). Based on these limits, it determines whether the values ​​of key parameters in the log data within a sub-interval are abnormal, generating an anomaly detection result for that sub-interval. For each sub-interval with an anomaly detection result, a feature vector is constructed based on the ICM of the key parameters and related parameters in the log data within that sub-interval. The anomaly pattern label for the sub-interval is then determined based on this feature vector. Finally, the anomaly detection results and associated information for each sub-interval are provided to the user through an anomaly detection page, thus addressing the log... Traditional data anomaly detection solutions rely on manual operation, resulting in high time and labor costs and difficulty in guaranteeing accuracy. This solution addresses these issues by automatically collecting log data within a specified time interval after receiving a log anomaly detection request. Then, based on the time windows corresponding to different time periods within the interval, the system divides the time interval into multiple sub-intervals. For each sub-interval, the system automatically determines whether the key parameter values ​​are abnormal based on the interquartile range (ICM) of the log data, along with the corresponding upper and lower limits for anomalies. This generates anomaly detection results for each sub-interval. Based on feature vectors, the system identifies anomaly pattern labels for sub-intervals with abnormal detection results. The results and correlation information for each sub-interval are then provided to the user through a visual interface. This approach automates the collection of log data within a specified time interval, rationally divides the time interval into multiple sub-intervals, and quickly and accurately determines whether the key parameter values ​​are abnormal within each sub-interval. This reduces the time and labor costs of log data anomaly detection and improves its accuracy.

[0101] In an optional embodiment of the present invention, the interval division module 302, when performing the operation of dividing the target time interval into multiple sub-intervals according to the time windows corresponding to each time period in the target time interval, is specifically configured to: determine the time period type of each time period in the target time interval; wherein the time period type is a peak business period, a normal business period, or a low business period; for a time period whose time period type is a peak business period, a first time window is determined as the time window corresponding to the time period, and the time period is divided into multiple sub-intervals according to the time window corresponding to the time period; for a time period whose time period type is a normal business period, a second time window is determined as the time window corresponding to the time period, and the time period is divided into multiple sub-intervals according to the time window corresponding to the time period; for a time period whose time period type is a low business period, a third time window is determined as the time window corresponding to the time period, and the time period is divided into multiple sub-intervals according to the time window corresponding to the time period; wherein the first time window is smaller than the second time window, and the second time window is smaller than the third time window.

[0102] In an optional embodiment of the present invention, the anomaly detection module 303, when performing the operation of determining the interquartile range of the key parameter values ​​in the log data within the sub-interval based on the upper quartile, lower quartile, kurtosis, and skewness of the parameter values ​​of the key parameters in the log data within the sub-interval, and the corresponding upper and lower limit values ​​of anomalies, is specifically configured to: determine the upper quartile, lower quartile, kurtosis, and skewness of the parameter values ​​of the key parameters in the log data within the sub-interval; and determine the upper quartile and lower quartile values ​​of anomalies based on ... and lower quartile values ​​of the key parameters in the log data within the sub-interval. The lower quartile is used to determine the interquartile range of key parameter values ​​in the log data within the sub-interval; based on the kurtosis and skewness, an anomaly limit calculation coefficient corresponding to the interquartile range is determined; the upper quartile, the anomaly limit calculation coefficient, and the interquartile range are substituted into a preset anomaly upper limit calculation formula to obtain the anomaly upper limit value corresponding to the interquartile range; the lower quartile, the anomaly limit calculation coefficient, and the interquartile range are substituted into a preset anomaly lower limit calculation formula to obtain the anomaly lower limit value corresponding to the interquartile range.

[0103] In an optional embodiment of the present invention, the anomaly detection module 303, when performing the operation of determining the anomaly limit calculation coefficient corresponding to the interquartile range based on the kurtosis and the skewness, is specifically configured to: input the kurtosis and the skewness into a pre-trained coefficient detection model to obtain the anomaly limit calculation coefficient output by the coefficient detection model; wherein, the input of the coefficient detection model is the kurtosis and skewness of the parameter values ​​of key parameters in the log data within the sub-interval, and the output of the coefficient detection model is the anomaly limit calculation coefficient corresponding to the interquartile range of the parameter values ​​of key parameters in the log data within the sub-interval.

[0104] In an optional embodiment of the present invention, the key parameter may be the system response time, and the related parameters may include system utilization, the number of system threads, and the number of system input bytes. When the tag determination module 304 performs the operation of constructing a feature vector of a sub-interval based on the interquartile range of the parameter values ​​of the key parameters and related parameters in the log data within the sub-interval, it is specifically used to: determine the interquartile range of the parameter values ​​of the system utilization, the number of system threads, and the number of system input bytes in the log data within the sub-interval; and construct a feature vector of the sub-interval based on the interquartile range of the parameter values ​​of the system response time, system utilization, the number of system threads, and the number of system input bytes in the log data within the sub-interval.

[0105] In an optional embodiment of the present invention, the label determination module 304, when performing the operation of determining the abnormal pattern label of the sub-interval based on the feature vector, is specifically configured to: input the feature vector into a pre-trained label detection model to obtain the abnormal pattern label output by the label detection model; wherein, the input of the label detection model is the feature vector of the sub-interval where the abnormal detection result is abnormal, and the output of the label detection model is the abnormal pattern label of the sub-interval where the abnormal detection result is abnormal.

[0106] In an optional embodiment of the present invention, the tag determination module 304 may also be used to: output alarm information corresponding to the abnormal mode tag.

[0107] The log data anomaly detection device provided in this embodiment of the invention can execute the log data anomaly detection method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0108] Example 4

[0109] Figure 4A schematic diagram of an electronic device 10, which can be used to implement the log data anomaly detection method of embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, electronic devices, blade electronic devices, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0110] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory 12 or a random access memory 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the read-only memory 12 or loaded from storage unit 18 into the random access memory 13. The random access memory 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, read-only memory 12, and random access memory 13 are interconnected via a bus 14. An input / output interface 15 is also connected to the bus 14.

[0111] Multiple components in electronic device 10 are connected to input / output interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of monitors, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0112] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, central processing units, graphics processing units, various special-purpose artificial intelligence computing chips, various processors running machine learning model algorithms, digital signal processors, and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as log data anomaly detection methods.

[0113] In some embodiments, the log data anomaly detection method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on a heterogeneous hardware accelerator via read-only memory and / or a communication unit. When the computer program is loaded into random access memory and executed by a processor, one or more steps of the log data anomaly detection method described above may be performed. Alternatively, in other embodiments, the processor may be configured to perform the log data anomaly detection method by any other suitable means (e.g., by means of firmware).

[0114] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays, application-specific integrated circuits (ASICs), application-specific standard products (ASICs), systems-on-a-chip (SoCs), payload programmable logic devices (PLCs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0115] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or electronic device.

[0116] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, optical fibers, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0117] To provide user interaction, the systems and techniques described herein can be implemented on a heterogeneous hardware accelerator, which includes: a display device (e.g., a cathode ray tube or liquid crystal display monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the heterogeneous hardware accelerator. Other types of devices can also be used to provide user interaction; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or haptic feedback); and input from the user can be received in any form (including sound input, voice input, or haptic input).

[0118] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data electronic devices), or computing systems that include middleware components (e.g., application electronic devices), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0119] A computing system can include clients and electronic devices. Clients and electronic devices are generally geographically separated and typically interact via communication networks. The client-electronic device relationship is created by computer programs running on the respective computers and establishing a client-electronic device relationship between them. Electronic devices can be cloud electronic devices, also known as cloud computing electronic devices or cloud servers, which are host products within the cloud computing service system. These address the shortcomings of traditional physical hosts and virtual private server services, such as high management difficulty and weak business scalability.

[0120] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0121] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for detecting log data anomalies, characterized in that, include: After receiving the log anomaly detection request sent by the user corresponding to the target system, the target time interval of the log anomaly detection request is determined, and the log data of the target system within the target time interval is collected. Based on the time windows corresponding to each time period in the target time interval, the target time interval is divided into multiple sub-intervals, and the log data in each sub-interval is determined based on the time information of each log data. For each sub-interval, based on the upper quartile, lower quartile, kurtosis, and skewness of the key parameter values ​​in the log data within the sub-interval, the interquartile range of the key parameter values ​​in the log data within the sub-interval, as well as the upper and lower limits of the anomaly corresponding to the interquartile range, are determined. Based on the interquartile range, the upper limit of the anomaly, and the lower limit of the anomaly, it is determined whether there are any anomalies in the parameter values ​​of the key parameters in the log data within the sub-interval, and anomaly detection results for the sub-interval are generated. For each sub-interval where the anomaly detection result is abnormal, a feature vector for the sub-interval is constructed based on the interquartile range of the key parameters and related parameters in the log data within the sub-interval. Based on the feature vector, the anomaly pattern label for the sub-interval is determined. The key parameter is system response time, and the related parameters include system utilization, number of system threads, and number of system-inbound bytes. Specifically, the interquartile range of the system utilization, number of system threads, and number of system-inbound bytes in the log data within the sub-interval is determined. The feature vector for the sub-interval is constructed based on the interquartile range of the system response time, system utilization, number of system threads, and number of system-inbound bytes in the log data within the sub-interval. The anomaly detection page provides the user with the anomaly detection results and related information for each sub-range.

2. The log data anomaly detection method according to claim 1, characterized in that, Based on the time windows corresponding to each time period within the target time interval, the target time interval is divided into multiple sub-intervals, including: Determine the time period type for each time period within the target time interval; wherein, the time period type is a peak business period, a normal business period, or a low business period; For time periods classified as peak business periods, the first time window is determined as the time window corresponding to the time period, and the time period is divided into multiple sub-intervals based on the time window corresponding to the time period; For time periods whose time type is normal business time period, the second time window is determined as the time window corresponding to the time period, and the time period is divided into multiple sub-intervals according to the time window corresponding to the time period; For time periods that are considered to be business off-peak periods, the third time window is determined as the time window corresponding to the time period, and the time period is divided into multiple sub-intervals based on the time window corresponding to the time period. Wherein, the first time window is smaller than the second time window, and the second time window is smaller than the third time window.

3. The log data anomaly detection method according to claim 1, characterized in that, Based on the upper quartile, lower quartile, kurtosis, and skewness of the key parameter values ​​in the log data within the sub-interval, determine the interquartile range of the key parameter values ​​in the log data within the sub-interval, as well as the upper and lower limits of anomalies corresponding to the interquartile ranges, including: Determine the upper quartile, lower quartile, kurtosis, and skewness of the key parameter values ​​in the log data within the sub-interval; Based on the upper quartile and the lower quartile, determine the interquartile range of the parameter values ​​of key parameters in the log data within the sub-interval; Based on the kurtosis and the skewness, determine the outlier limit calculation coefficient corresponding to the interquartile range; Substitute the upper quartile, the anomaly limit calculation coefficient, and the interquartile range into the preset anomaly upper limit calculation formula to obtain the anomaly upper limit value corresponding to the interquartile range; Substituting the lower quartile, the anomaly limit calculation coefficient, and the interquartile range into the preset anomaly lower limit calculation formula, an anomaly lower limit value corresponding to the interquartile range is obtained.

4. The log data anomaly detection method according to claim 3, characterized in that, Based on the kurtosis and skewness, determine the outlier limit calculation coefficient corresponding to the interquartile range, including: The kurtosis and skewness are input into a pre-trained coefficient detection model to obtain the anomaly limit calculation coefficients output by the coefficient detection model. The input to the coefficient detection model is the kurtosis and skewness of the parameter values ​​of key parameters in the log data within the sub-interval, and the output of the coefficient detection model is the anomaly limit calculation coefficient corresponding to the interquartile range of the parameter values ​​of key parameters in the log data within the sub-interval.

5. The log data anomaly detection method according to claim 1, characterized in that, Based on the feature vector, the abnormal pattern label of the sub-interval is determined, including: The feature vector is input into a pre-trained label detection model to obtain the abnormal pattern label output by the label detection model; The input of the label detection model is the feature vector of the sub-interval where the anomaly detection result is abnormal, and the output of the label detection model is the anomaly pattern label of the sub-interval where the anomaly detection result is abnormal.

6. The log data anomaly detection method according to claim 1, characterized in that, After determining the anomaly pattern label of the sub-interval based on the feature vector, the method further includes: Output the alarm information corresponding to the abnormal mode label.

7. A log data anomaly detection device, characterized in that, include: The data acquisition module is used to determine the target time interval of the log anomaly detection request after receiving a log anomaly detection request sent by a user corresponding to the target system, and to collect various log data of the target system within the target time interval. The interval division module is used to divide the target time interval into multiple sub-intervals according to the time windows corresponding to each time period in the target time interval, and to determine the log data in each sub-interval according to the time information of each log data. The anomaly detection module is used to determine the interquartile range of the key parameter values ​​in the log data of the sub-interval based on the upper quartile, lower quartile, kurtosis, and skewness of the parameter values ​​of the key parameters in the log data of the sub-interval, as well as the upper and lower limits of the anomaly corresponding to the interquartile range. Based on the interquartile range, the upper limit of the anomaly, and the lower limit of the anomaly, it determines whether there are anomalies in the parameter values ​​of the key parameters in the log data of the sub-interval, and generates the anomaly detection result of the sub-interval. The label determination module is used to construct a feature vector for each sub-interval where the anomaly detection result is abnormal, based on the interquartile range of the parameter values ​​of key parameters and related parameters in the log data within the sub-interval. Based on the feature vector, the anomaly pattern label of the sub-interval is determined. The key parameter is system response time, and the related parameters include system utilization, number of system threads, and number of system input bytes. Specifically, the interquartile range of the parameter values ​​of system utilization, number of system threads, and number of system input bytes in the log data within the sub-interval is determined; the feature vector of the sub-interval is constructed based on the interquartile range of the parameter values ​​of system response time, system utilization, number of system threads, and number of system input bytes in the log data within the sub-interval. The results provision module is used to provide the user with the anomaly detection results and detection association information for each sub-interval through the anomaly detection page.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores a computer program that is executed by the at least one processor, which enables the at least one processor to perform the log data anomaly detection method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the log data anomaly detection method according to any one of claims 1-6.

Images