Log data processing method and device and nonvolatile storage medium

Abstract

The invention discloses a log data processing method and device and a nonvolatile storage medium. The method comprises the following steps: acquiring a log data set corresponding to a data source; counting the quantity of the abnormal log data according to a preset time window; obtaining a target quantity of abnormal log data in the target time window, and obtaining a first quantity of abnormal log data in a first time window within a preset historical duration; determining a plurality of statistical results between the target number and the first number by using a plurality of different preset statistical algorithms; according to the use sequence of each preset statistical algorithm in the time dimension, determining a weight corresponding to each preset statistical algorithm; and performing weighted summation calculation on a statistical result of the preset statistical algorithm, and determining whether alarm information for the abnormal log data in the target time window is generated or not according to a calculation result. The technical problem that the abnormal log generated by the data source cannot be warned in time is solved.

Description

Technical Field

[0001] The present application relates to the technical field of big data processing, and in particular, to a method and device for processing log data, and a non-volatile storage medium. Background Art

[0002] In large-scale distributed systems, the immediate collection and real-time alarm of system abnormality logs are the key to ensuring system stability and reliability. However, the traditional log processing process has significant time delays, usually involving users discovering abnormalities, reporting them, and then R&D personnel analyzing the logs to confirm the abnormalities and causes. This process is complex and the timeliness and accuracy of processing are poor.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a method and device for processing log data, and a non-volatile storage medium, so as to at least solve the technical problem of being unable to promptly issue an alarm for abnormal logs generated by a data source.

[0005] According to one aspect of an embodiment of the present application, a method for processing log data is provided, including: obtaining a log data set corresponding to a data source; periodically obtaining abnormal log data recorded according to preset rules in the log data set according to a preset time period, and counting the number of abnormal log data according to a preset time window, wherein the duration corresponding to the preset time window is less than the duration corresponding to the preset time period; obtaining a target number of abnormal log data in a target time window, and obtaining a first number of abnormal log data in a first time window within a preset historical duration, wherein the target time window is a time window in which the abnormal log data to be processed is located, and the first time window is a time window adjacent to the target time window; using multiple different preset statistical algorithms, determining multiple statistical results between the target number and the first number; determining the weight corresponding to each preset statistical algorithm according to the order in which each preset statistical algorithm is used in the time dimension; performing weighted sum calculation on the statistical results of the preset statistical algorithm according to the weight corresponding to each preset statistical algorithm, and determining whether to generate an alarm message for the abnormal log data in the target time window according to the calculation result.

[0006] Optionally, multiple statistical results between the target quantity and the first quantity are determined using multiple different preset statistical algorithms, including: using a K-sigma algorithm to determine the standard deviation between the target quantity and the target average value, wherein the target average value is the average value of the first quantity corresponding to the first time window within a preset historical time length; when the standard deviation is greater than a first preset threshold, the first statistical result output by the K-sigma algorithm is determined to be 1, and when the standard deviation is not greater than the first preset threshold, the first statistical result output by the K-sigma algorithm is determined to be 0; using a kernel density estimation algorithm to determine a probability density estimate corresponding to the target quantity; when the probability density estimate is greater than a second preset threshold, the second statistical result output by the kernel density estimation algorithm is determined to be 1, and when the probability density estimate is not greater than the second preset threshold, the second statistical result output by the kernel density estimation algorithm is determined to be 0; using an isolation forest algorithm to determine an abnormality score corresponding to the target quantity; when the abnormality score is greater than a third preset threshold, the third statistical result output by the isolation forest algorithm is determined to be 1, and when the abnormality score is not greater than the third preset threshold, the third statistical result output by the isolation forest algorithm is determined to be 0.

[0007] Optionally, after determining whether to generate an alarm message for the abnormal log data in the target time window, the method further includes: obtaining a weight list, wherein the weight list includes a weight corresponding to each preset statistical algorithm; if it is determined to generate an alarm message for the abnormal log data in the target time window based on the calculation result, the initial weight corresponding to the first target algorithm is increased to obtain a first weight, and the initial weight corresponding to the first target algorithm is updated in the weight list by the first weight to obtain an updated weight list, wherein the first target algorithm is a preset statistical algorithm with a statistical result of 1; if it is determined not to generate an alarm message for the abnormal log data in the target time window based on the calculation result, the initial weight corresponding to the second target algorithm is increased to obtain a second weight, and the initial weight corresponding to the second target algorithm is updated in the weight list by the second weight to obtain an updated weight list, wherein the second target algorithm is a preset statistical algorithm with a statistical result of 0; using multiple different preset statistical algorithms, determine multiple first statistical results between the second number and the first number of abnormal log data in the first target time window; according to the updated weight list, perform weighted sum calculation on the first statistical results, and according to the calculation result, determine whether to generate an alarm message for the abnormal log data in the first target time window.

[0008] Optionally, according to the order in which each preset statistical algorithm is used in the time dimension, the weight corresponding to each preset statistical algorithm is determined, including: for the i-th preset statistical algorithm used in the time dimension, its corresponding identification information is determined as i, where i is a positive integer not greater than n, and n is the number of preset statistical algorithms; the weight w corresponding to the i-th preset statistical algorithm is determined by the following formula i :

[0009] Optionally, based on the calculation result, determine whether to generate alarm information for abnormal log data in the target time window, including: judging whether the calculation result is greater than the inverse of the number of a preset statistical algorithm, if the calculation result is greater than the inverse of the number of a preset statistical algorithm, generate alarm information for abnormal log data in the target time window.

[0010] Optionally, when determining to generate alarm information for abnormal log data in the target time window, the method also includes: querying the abnormal log content of the abnormal log data in the target time window; combining the abnormal log content with a preset statement to obtain a target statement, and calling a pre-trained deep learning model through the target statement; obtaining the output result generated by the pre-trained deep learning model in response to the target statement.

[0011] Optionally, after obtaining the output result generated by the pre-trained deep learning model in response to the target statement, the method also includes: sending the output result to a terminal device corresponding to a pre-stored terminal device identification set, wherein the output result includes at least one of the following: abnormal cause information and processing solution information corresponding to the log content.

[0012] According to another aspect of the embodiment of the present application, a log data processing device is also provided, including: a first acquisition module, used to acquire a log data set corresponding to a data source; a statistical module, used to periodically acquire abnormal log data recorded according to a preset rule in the log data set according to a preset time period, and count the number of abnormal log data according to a preset time window, wherein the duration corresponding to the preset time window is less than the duration corresponding to the preset time period; a second acquisition module, used to acquire a target number of abnormal log data in a target time window, and acquire a first number of abnormal log data in a first time window within a preset historical duration, wherein the target time window is a time window where the abnormal log data to be processed is located, and the first time window is a time window adjacent to the target time window; a first determination module, used to determine a plurality of statistical results between the target number and the first number using a plurality of different preset statistical algorithms; a second determination module, used to determine a weight corresponding to each preset statistical algorithm according to the order in which each preset statistical algorithm is used in the time dimension; a third determination module, used to perform weighted sum calculation on the statistical results of the preset statistical algorithm according to the weight corresponding to each preset statistical algorithm, and determine whether to generate an alarm information for the abnormal log data in the target time window according to the calculation result.

[0013] According to another aspect of the embodiments of the present application, a non-volatile storage medium is provided, the storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the above log data processing method.

[0014] According to another aspect of the embodiments of the present application, there is further provided an electronic device, including: a memory and a processor, the processor being configured to run a program stored in the memory, wherein the above log data processing method is executed when the program is run.

[0015] According to yet another aspect of the embodiments of the present application, a computer program is further provided, wherein when the computer program is executed by a processor, the above log data processing method is implemented.

[0016] According to another aspect of the embodiments of the present application, a computer program product is also provided, the computer program product comprising a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above log data processing method is implemented.

[0017] In an embodiment of the present application, a log data set corresponding to a data source is obtained; according to a preset time period, abnormal log data recorded according to preset rules in the log data set is periodically obtained, and the number of abnormal log data is counted according to a preset time window, wherein the duration corresponding to the preset time window is less than the duration corresponding to the preset time period; a target number of abnormal log data in a target time window is obtained, and a first number of abnormal log data in a first time window within a preset historical duration is obtained, wherein the target time window is a time window in which the abnormal log data to be processed is located, and the first time window is a time window adjacent to the target time window; a plurality of statistical results between the target number and the first number are determined using a plurality of different preset statistical algorithms; a weight corresponding to each preset statistical algorithm is determined according to the order in which each preset statistical algorithm is used in the time dimension; a weighted sum calculation is performed on the statistical results of the preset statistical algorithm according to the weight corresponding to each preset statistical algorithm, and according to the calculation result, it is determined whether to generate an alarm information for the abnormal log data in the target time window, thereby achieving a technical effect of timely alarming the abnormal logs generated by the data source, thereby solving the technical problem that the abnormal logs generated by the data source cannot be timely alarmed. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 is a flow chart of a method for processing log data according to an embodiment of the present application;

[0020] Figure 2 is a flowchart of another method for processing log data according to an embodiment of the present application;

[0021] Figure 3 is a flow chart of a method for processing a time window according to an embodiment of the present application;

[0022] Figure 4 It is a flow chart of a method for log anomaly judgment and alarm triggering according to an embodiment of the present application;

[0023] Figure 5 This is a schematic diagram of an alarm judgment according to an embodiment of the present application;

[0024] Figure 6 is a structural diagram of a log data processing device according to an embodiment of the present application;

[0025] Figure 7 It is a hardware structure block diagram of a computer terminal according to a method for processing log data in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] According to an embodiment of the present application, a method embodiment of a method for processing log data is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0029] Figure 1 is a flow chart of a method for processing log data according to an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0030] Step S101, obtaining a log data set corresponding to a data source.

[0031] Among them, the data source is, for example, a distributed system. For complex environments such as distributed systems, the data source includes log files on multiple server nodes, which need to be uniformly collected and processed for subsequent anomaly detection and analysis. The following are the specific implementation steps of step S101: Analyze the distributed system architecture, determine all server nodes that may generate logs, and confirm the log types that need to be monitored, such as ERROR-level logs or specific exception logs. Deploy log collection tools such as Filebeat on all server nodes, and configure the log collection tools to monitor specified log files or directories. Set the frequency of log collection to ensure that log data can be collected in real time or near real time, and configure log filtering rules, such as using regular expressions to filter ERROR-level logs or specific exception logs. Configure Filebeat to send the collected log data to middleware such as Kafka, and create a corresponding data source topic in Kafka for receiving and storing log data. In the downstream processing flow of Kafka, data preprocessing steps can be set, such as parsing log formats, extracting key fields, etc., and storing the preprocessed log data in a persistent storage system, such as a clickhouse database, for subsequent analysis and retrieval.

[0032] Step S102, periodically obtain abnormal log data recorded according to preset rules in the log data set according to a preset time period, and count the number of abnormal log data according to a preset time window, wherein the duration corresponding to the preset time window is less than the duration corresponding to the preset time period.

[0033] For example, the preset time period is 12 hours, which means that the abnormal log data of the log data set is captured and analyzed once every 12 hours. The preset rules are the identification criteria for abnormal logs. For example, the lines in the log containing specific error codes or abnormal messages will be regarded as abnormal log data. The preset time window is, for example, 5 minutes, which means that the system will use every 5 minutes as a statistical unit to calculate the number of abnormal log data.

[0034] Specifically, you can read the abnormal log data in the past 12 hours from KAFKA. Group the log data by occurrence time and count each 5-minute window. Store the statistical results in the error_log_detail table in the clickhouse database to ensure fast data query and analysis capabilities.

[0035] Step S103, obtain the target number of abnormal log data in the target time window, and obtain the first number of abnormal log data in the first time window within the preset historical time length, wherein the target time window is the time window where the abnormal log data to be processed is located, and the first time window is the time window adjacent to the target time window.

[0036] The length of the target time window, such as 5 minutes, and the preset historical length, such as the most recent year. Within the target time window (such as the current 5-minute time period), monitor and collect abnormal log data through Filebeat. Filebeat sends this data to Kafka, and then Kafka persists the data to the backend repository, such as the error_log_detail table of the ClickHouse database. Query all abnormal log records within the target time window from the error_log_detail table, and count the total number of abnormal logs as the target number. The first time window refers to the time window adjacent to the target time window. For example, if the target time window is the current 5 minutes, the first time window can be the previous 5-minute and / or next 5-minute time window. Query all abnormal log records within the first time window from the error_log_detail table, and count the total number of abnormal logs as the first number.

[0037] Step S104: using a plurality of different preset statistical algorithms to determine a plurality of statistical results between the target quantity and the first quantity.

[0038] The preset statistical algorithms include, but are not limited to, K-sigma algorithm, kernel density estimation algorithm, and isolation forest algorithm. K-sigma algorithm is a statistical anomaly detection method that uses the properties of normal distribution to identify data points that deviate from the average value (mean) by more than a certain standard deviation (sigma). In log analysis, K-sigma algorithm can identify abnormal values ​​that suddenly appear within the normal operating range. For example, if the value of a certain indicator suddenly exceeds its historical average plus several standard deviations, this may mean that an abnormality has occurred in the system. Kernel Density Estimation Algorithm (KDE): KDE is a non-parametric probability density function estimation method that estimates the distribution of data by placing a kernel function around each data point and calculating its weighted average. In log analysis, KDE can be used to identify sparse areas in the data distribution that may indicate the occurrence of abnormal behavior. Unlike K-sigma, KDE does not rely on the assumption that the data follows a specific distribution, so it can still effectively detect anomalies when the log data may not satisfy the normal distribution. Isolation Forest: This is an anomaly detection algorithm based on ensemble learning. It isolates outliers in a data set by building multiple random binary trees (part of a random forest). The core idea of ​​Isolation Forest is that outliers are often different from other normal values ​​in the feature space, so they are more easily isolated by the randomly divided tree structure than normal values. In log analysis, Isolation Forest can effectively identify log records that behave abnormally in a multidimensional feature space, especially when the anomaly is very obvious in some dimensions.

[0039] Step S105 , determining the weight corresponding to each preset statistical algorithm according to the order in which each preset statistical algorithm is used in the time dimension.

[0040] For example, if the K-sigma algorithm performs better during business peak hours, it can be given a higher weight during peak hours. Therefore, when the number of abnormal log data is at peak hours, the Isolation Forest algorithm or KDE algorithm can be used first, and the K-sigma algorithm can be used last. The Isolation Forest algorithm may be more effective when the data is unevenly distributed, so it is given a higher weight during non-peak hours or when the data fluctuates abnormally. Therefore, when the number of abnormal log data is at non-peak hours, the K-sigma algorithm or KDE algorithm can be used first, and the IsolationForest algorithm can be used last.

[0041] Step S106, performing weighted sum calculation on the statistical results of the preset statistical algorithms according to the weights corresponding to each preset statistical algorithm, and determining whether to generate alarm information for abnormal log data in the target time window according to the calculation results.

[0042] For example, the statistical results are weighted and summed using the following formula:

[0043]

[0044] Among them, TS stands for Total Score, R i represents the output result of the ith algorithm, W i represents the weight calculated by the i-th algorithm through the adaptive weighting method provided in step S105.

[0045] That is, for each of the n algorithms (from algorithm 1 to algorithm i), its output result (Ri) is multiplied by the weight (Wi) calculated by the adaptive weighting method. The multiplication results of all algorithms are added together to obtain the total score (TS).

[0046] Finally, the calculated total score (TS) is compared with 1 / n. If the total score is greater than this threshold, an alarm is triggered. This means that after the algorithm is divided by rights, there are two or more algorithms with alarms. If the total score is less than or greater than this threshold, no alarm is triggered.

[0047] Through the above steps, a log data set corresponding to a data source is obtained; according to a preset time period, abnormal log data recorded according to preset rules in the log data set is periodically obtained, and the number of abnormal log data is counted according to a preset time window, wherein the duration corresponding to the preset time window is less than the duration corresponding to the preset time period; a target number of abnormal log data in a target time window is obtained, and a first number of abnormal log data in a first time window within a preset historical duration is obtained, wherein the target time window is a time window in which the abnormal log data to be processed is located, and the first time window is a time window adjacent to the target time window; a plurality of statistical results between the target number and the first number are determined using a plurality of different preset statistical algorithms; a weight corresponding to each preset statistical algorithm is determined according to the order in which each preset statistical algorithm is used in the time dimension; a weighted sum calculation is performed on the statistical results of the preset statistical algorithm according to the weight corresponding to each preset statistical algorithm, and according to the calculation result, it is determined whether to generate an alarm information for the abnormal log data in the target time window, thereby achieving a technical effect of timely alarming the abnormal logs generated by the data source.

[0048] The following Figure 1 The steps shown are exemplary and explanatory.

[0049] According to some optional embodiments of the present application, using multiple different preset statistical algorithms to determine multiple statistical results between the target quantity and the first quantity can be achieved by the following method: using a K-sigma algorithm to determine the standard deviation between the target quantity and the target average value, wherein the target average value is the average value of the first quantity corresponding to the first time window within a preset historical time length; when the standard deviation is greater than a first preset threshold, the first statistical result output by the K-sigma algorithm is determined to be 1, and when the standard deviation is not greater than the first preset threshold, the first statistical result output by the K-sigma algorithm is determined to be 0; using a kernel density estimation algorithm to determine a probability density estimate corresponding to the target quantity; when the probability density estimate is greater than a second preset threshold, the second statistical result output by the kernel density estimation algorithm is determined to be 1, and when the probability density estimate is not greater than the second preset threshold, the second statistical result output by the kernel density estimation algorithm is determined to be 0; using an isolation forest algorithm to determine an abnormality score corresponding to the target quantity; when the abnormality score is greater than a third preset threshold, the third statistical result output by the isolation forest algorithm is determined to be 1, and when the abnormality score is not greater than the third preset threshold, the third statistical result output by the isolation forest algorithm is determined to be 0.

[0050] For example, the current time window (target time window) is 2023-03-21 10:00:00-2023-03-21 10:04:59, and the number of abnormal log data in the current time window is the target number. Get the average value (target average value) of the number of abnormal log data in the first time window in the past 30 days (preset historical duration). Set the first preset threshold to 3 (i.e. 3σ), and use the K-sigma algorithm to determine the standard deviation between the target number and the target average. If the standard deviation between the target number and the target average is greater than 3σ), then determine that the first statistical result output by the K-sigma algorithm is 1 (abnormal), otherwise it is 0 (normal).

[0051] Further, after determining whether to generate an alarm message for the abnormal log data in the target time window, the following steps may be performed: obtaining a weight list, wherein the weight list includes a weight corresponding to each preset statistical algorithm; if, according to the calculation result, it is determined to generate an alarm message for the abnormal log data in the target time window, the initial weight corresponding to the first target algorithm is increased to obtain a first weight, and the initial weight corresponding to the first target algorithm is updated in the weight list by the first weight to obtain an updated weight list, wherein the first target algorithm is a preset statistical algorithm with a statistical result of 1; if, according to the calculation result, it is determined not to generate an alarm message for the abnormal log data in the target time window, the initial weight corresponding to the second target algorithm is increased to obtain a second weight, and the initial weight corresponding to the second target algorithm is updated in the weight list by the second weight to obtain an updated weight list, wherein the second target algorithm is a preset statistical algorithm with a statistical result of 0; using multiple different preset statistical algorithms, determine multiple first statistical results between the second number and the first number of abnormal log data in the first target time window; according to the updated weight list, perform weighted sum calculation on the first statistical results, and according to the calculation result, determine whether to generate an alarm message for the abnormal log data in the first target time window.

[0052] It is understandable that the first target time window is the next time window to be processed after the target time window is processed. For example, first, a weight list can be read from a configuration file or a database, which contains the initial weights corresponding to preset statistical algorithms such as K-sigma, KDE, and Isolation Forest. The specific format of the weight list is as follows: weights = [w1, w2, w3], where w1, w2, and w3 are the initial weights of the K-sigma, KDE, and Isolation Forest algorithms, respectively.

[0053] When the system determines that it needs to generate an alarm for abnormal log data in the target time window based on the calculation results of the preset statistical algorithm, that is, when the statistical result is 1, the following steps are performed: Determine the first target algorithm, that is, the algorithm with a statistical result of 1, assuming it is the K-sigma algorithm. Increase the initial weight w1 corresponding to the first target algorithm (K-sigma), and update w1 in the weight list to new w1 , forming the updated weight list new weights =[new w1 ,w2,w3].

[0054] When the system determines that it is not necessary to generate an alarm for abnormal log data in the target time window based on the calculation results of the preset statistical algorithm, that is, when the statistical result is 0, the following steps are performed: Determine the second target algorithm, that is, the algorithm with a statistical result of 0, assuming it is the KDE algorithm. Increase the initial weight w2 corresponding to the second target algorithm (KDE) to obtain the second weight new w2 , update w2 in the weight list to new w2 , forming the updated weight list new weights =[w1,new w2 ,w3].

[0055] Use the updated weight list new weights A weighted sum calculation is performed on multiple first statistical results of abnormal log data in the first target time window. The calculation formula is as follows: sum =new w1 *R1+new w2 *R2+new w3 *R3, where R1, R2 and R3 are the statistical results of K-sigma, KDE and Isolation Forest algorithms on abnormal log data respectively. Finally, according to the calculation results, it is determined whether to generate alarm information for abnormal log data in the first target time window.

[0056] According to some other optional embodiments of the present application, the weight corresponding to each preset statistical algorithm is determined according to the order in which each preset statistical algorithm is used in the time dimension, which can be achieved by the following method: for the preset statistical algorithm used for the i-th time dimension, its corresponding identification information is determined as i, where i is a positive integer not greater than n, and n is the number of preset statistical algorithms; the weight w corresponding to the i-th preset statistical algorithm is determined by the following formula: i :

[0057]

[0058] From the above formula, we can see that the design of this formula ensures the increasing trend of the result. Specifically, as the input i gradually increases, the calculated f(x) also increases accordingly. This feature ensures that the weight value can truly reflect the growth of the input parameter and avoids information distortion caused by unchanged or decreasing weights.

[0059] In addition, the formula uses a convergence algorithm, and this design ingeniously solves the problem of weight imbalance that may be caused by the rapid growth of data. In actual application scenarios such as alarm systems, if the frequency of an alarm is abnormally high, the use of a linear growth method to calculate the weight may cause its weight value to expand rapidly, thereby suppressing the performance of other alarm algorithms, forming a "one-man show" phenomenon, and affecting the accuracy and comprehensiveness of the overall alarm. By introducing a convergence algorithm, this embodiment can effectively limit the weight of the alarm to a reasonable and acceptable range, so that other alarm algorithms designed for specific scenarios can also get sufficient display opportunities, thereby working together to improve the accuracy and efficiency of the overall alarm. This convergence not only ensures the stability and fairness of the system, but also significantly enhances the adaptability and flexibility of the system.

[0060] In some optional embodiments of the present application, determining whether to generate alarm information for abnormal log data in a target time window based on a calculation result can be achieved by the following method: judging whether the calculation result is greater than the inverse of the number of a preset statistical algorithm; if the calculation result is greater than the inverse of the number of a preset statistical algorithm, generating alarm information for abnormal log data in the target time window.

[0061] As some optional embodiments of the present application, when it is determined to generate alarm information for abnormal log data in a target time window, the following steps can also be performed: query the abnormal log content of the abnormal log data in the target time window; combine the abnormal log content with a preset statement to obtain a target statement, and call a pre-trained deep learning model through the target statement; obtain the output result generated by the pre-trained deep learning model in response to the target statement.

[0062] Furthermore, after obtaining the output result generated by the pre-trained deep learning model in response to the target statement, the following steps can also be performed: sending the output result to the terminal device corresponding to the pre-stored terminal device identification set, wherein the output result includes at least one of the following: abnormal cause information and processing solution information corresponding to the log content.

[0063] Specifically, after determining that the current situation meets the conditions for initiating an alarm, the system will perform a series of steps to further analyze and handle the abnormal situation. First, the system will retrieve all abnormal log entries recorded in the current time window. These logs describe in detail the abnormal events encountered during the operation of the system. Subsequently, the system will combine these abnormal log information with the pre-set query template to form a special large model call statement. For example, assuming that the abnormal information "CommunicationsException: Communications link failure" frequently appears in the system log, the system will construct the following call statement: "A large number of abnormal logs suddenly appear in the system. The types of abnormal logs include but are not limited to the following. Please analyze the root cause in detail and provide feasible solutions for reference: CommunicationsException: Communications link failure". This call statement is intended to guide the large model to conduct in-depth analysis of the abnormal logs in order to obtain specific abnormal causes and response strategies.

[0064] After receiving the above call statement, the big model will use its powerful data processing and pattern recognition capabilities to conduct a detailed analysis of the "CommunicationsException: Communications link failure" in the exception log. An example of the analysis result may be as follows: The exception log indicates that there is a database connection communication failure. To solve this problem, it is recommended to take the following steps: First, check whether the network connection is normal to ensure that data transmission is unimpeded; second, confirm the database configuration information, including whether the login credentials and connection parameters are correct; third, monitor the current status of the database service to confirm whether it is running normally; in addition, adjust the database connection timeout setting to adapt to the current network environment; check the compatibility of the database driver with the current system environment; review the configuration of the database connection pool to ensure that resources are allocated reasonably; monitor the resource utilization of the database server to evaluate whether the exception is caused by resource shortage; finally, continue to monitor and analyze the exception log to prevent similar problems from happening again.

[0065] Once the analysis results of the big model are returned, including in-depth interpretation of the abnormal log and recommended solutions, this information will be integrated into the alarm notification. The alarm notification will not only point out the existence of the problem, but also provide the possible cause of the problem and targeted solution suggestions, so as to help the team members receiving the alarm quickly understand the background of the problem, take effective countermeasures, shorten the fault response time, and improve the overall maintenance efficiency of the system.

[0066] In summary, the advantages of this application are as follows: 1. Rapid response to system anomalies: The relevant abnormal log problem discovery mode mainly relies on system users to discover problems and report them during use, and then the R&D team queries the system log to confirm whether there is a problem. This process is often time-consuming, especially in large-scale distributed systems. This operation is not only inefficient, but may also greatly reduce the response speed of the problem. The application introduces an abnormal log analysis and alarm method based on adaptive weighted calculation, which can significantly shorten the time of abnormal discovery, thereby quickly reducing the impact of abnormalities on the business and greatly improving the stability of the system. The core of this method is to achieve accurate parsing and abnormal identification of system logs by comprehensively using multiple algorithms for weighted calculation and combining large model analysis technology. More importantly, according to the specific needs of different system scenarios, the sub-algorithms of the multi-algorithm weighted calculation method can be weighted to achieve higher alarm accuracy. This flexibility and adaptability enable our alarm system to better meet the needs of various complex scenarios, thereby providing users with more reliable and efficient services. 2. Real-time calculation of adaptive thresholds: Traditional alarms usually require manual setting of fixed thresholds, which lack flexibility. In some cases, this fixed threshold cannot accurately trigger an alarm, such as when the business is low in the early morning and the log volume is small. If the threshold is set according to the business volume during the day, potential problems at night may not be detected; conversely, if the threshold is set according to the business volume at night, it may cause false alarms during the day. Therefore, real-time calculation of adaptive thresholds is crucial to improving the accuracy of alarms. By calculating the thresholds by algorithmic weights, real-time adaptive adjustments can be made to improve the accuracy and practicality of alarms. 3. Large models analyze abnormal causes and generate solutions: Related processing alarms require R&D personnel to check system logs, query the code step by step, or analyze the cause of the error through search engines, and then propose solutions. This process takes a long time. By introducing a large model, this application can delegate some of the analysis and solution tasks to this model. When R&D personnel receive an alarm, they can immediately obtain the approximate cause of the problem and solution, thereby reducing the work pressure of R&D personnel.

[0067] Figure 2 is a flowchart of another method for processing log data according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0068] Step S201, use Filebeat to monitor logs.

[0069] Step S2011, configure the log file or directory in the monitoring system.

[0070] Step S2012, filter and merge ERROR level logs and exception logs.

[0071] Step S2013, sending the collected ERROR level or exception log data to the Kafka middleware.

[0072] For example, you can deploy the Filebeat service as follows: 1.1 Configure Filebeat's filebeat.inputs.path to: / opt / ZSMART_HOME / logs / prod_coor_service. 1.2 Configure Filebeat's multiline.pattern to: ^[[:space:]]+(at|.{3})\b|^Caused by. 1.3 Configure Filebeat's multiline.match to after. 1.4 Configure Filebeat's multiline.negate to false. 1.5 Configure Filebeat's logging.level to error. 1.6 Configure Filebeat's name to 100.80.211.244. 1.7 Configure Filebeat's filebeat.inputs.fields.log_topic to coordService. 1.8 Configure Filebeat's filebeat.inputs.enabled to true. 1.9 Configure Filebeat's output.kafka.enabled to true. 1.10 Configure Filebeat's output.kafka.hosts to ["10.152.8.244:8422","10.152.8.220:8422","10.152.8.219:8422"]. 1.11 Configure Filebeat's output.kafka.topic to coordErrorLogTopic. 1.12 Configure Filebeat's output.kafka.compression to log.

[0073] Step S202: Store logs. Kafka, as a middleware, receives and stores a large amount of log data to ensure data reliability and scalability.

[0074] Specifically, the logs can be stored in the error_log_detail table of the clickhouse database (the design is shown in the table below). The fields include source_ip (log generating machine), happened_date (log generating time), and log_data (log details).

[0075] Field encoding Field Name type illustrate id ID string uuid souce_ip The IP address where the abnormal log occurs string happened_date Abnormal log occurrence time string log_data Log details string app_name Application Name string create_date Data writing time string

[0076] Then, query the error_log_detail table to calculate the number of abnormal logs within 5 minutes. See the following table for an example:

[0077]

[0078] Figure 3 is a flow chart of a method for processing a time window according to an embodiment of the present application, such as Figure 3 As shown, the indicator values ​​of the current window, the previous window, and the next window within 365 days in the t_point_value table are queried. The data in the t_point_value table is executed by a scheduled task every 12 hours. Each time the query is executed, the number of abnormal logs in the error_log_detail table every five minutes within 12 hours is used as an indicator, and the multi-algorithm weighted analysis method is used to calculate its abnormality, and the data is inserted into the t_point_value table.

[0079] Step S203, analyzing the log.

[0080] Step S2031, timing task statistical indicators.

[0081] Step S20311: The scheduled task is executed once every 12 hours, with each 5-minute window. In each 5-minute window, the number of abnormal logs is used as a statistical indicator, and a multi-algorithm weighted analysis is used to calculate whether it is an abnormal indicator. If it is not an abnormal indicator, the indicator is stored in the indicator table.

[0082] Step S20312, query the abnormal log data within the last 5 minutes, and query the indicator table to obtain the indicator data of the three windows of the time point, before the time point, and after the time point in the last year.

[0083] Step S2032: log anomaly determination and alarm triggering. Figure 4 is a flow chart of a method for log anomaly judgment and alarm triggering according to an embodiment of the present application, such as Figure 4 As shown, the method includes the following steps.

[0084] Step S20321, obtaining historical indicator data and data to be tested.

[0085] Step S20322, use the K-sigma algorithm to calculate whether the log is abnormal, and return 1 if abnormal, and 0 if normal.

[0086] Step S20323, use the KDE algorithm to calculate whether the log is abnormal, and return 1 if abnormal, and 0 if normal.

[0087] Step S20324, use the Isolation Forest algorithm to calculate whether the log is abnormal, and return 1 if abnormal, and 0 if normal.

[0088] Step S20325, obtain the weight calculation value of each algorithm, and use adaptive weighting to calculate the real weight of each algorithm. The adaptive weighting calculation formula is as follows: Among them, n is the number of the algorithm, and f(x) is the weight calculation result.

[0089] Step S20326, after obtaining the true weights of all algorithms, calculate the percentage of all algorithms in the weight list, and use the percentage value as the new weight. Use the weighted analysis method to calculate whether the current log is abnormal. If the weighted result is greater than 1 / (total number of algorithms), return abnormal (that is, the weighted result has two or more algorithm alarms), otherwise return normal. If normal, the weight calculation value of the algorithm that needs to be alarmed will be reduced by 1 (or a natural number not less than 1, because only the algorithm determines that an alarm is needed but no alarm is issued, which is understood as a false alarm). If abnormal, the weight calculation value returned that needs to be alarmed will be increased by 1 (indicating that the alarm judgment is successful) to strengthen the influence of these algorithms in subsequent analysis.

[0090] Step S20327, using an adaptive weighted calculation method to perform weighted calculation.

[0091] The algorithm output results are weighted and calculated using the following formula:

[0092]

[0093] Among them, TS stands for Total Score, R i represents the output result of the ith algorithm, W i Represents the weight calculated by the i-th algorithm through the adaptive weighting method.

[0094] In the above steps, the adaptive weighting method is creatively used to dynamically calculate the weights of various algorithms. This method makes the weights learnable, and has the characteristics of multi-angle detection, high robustness, comprehensive judgment, diversity of abnormal patterns, adaptability to different data distributions and high efficiency. After obtaining the results of multiple anomaly detection algorithms, the multi-algorithm weighted analysis method uses a weighted voting algorithm to calculate whether there is an anomaly. The main advantage of the adaptive weighting method is that it combines the results of multiple independent algorithms and makes the weights of each algorithm learnable, thereby greatly reducing the false alarm rate and weakening the impact of the shortcomings of each independent algorithm on the results. By comprehensively judging the outputs of multiple algorithms, the multi-algorithm weighted analysis method can improve the accuracy of anomaly detection. In addition, the multi-algorithm weighted analysis method is suitable for various scenarios and can adapt to different data distributions and data characteristics. Its multi-angle detection and diversity of abnormal patterns enable it to capture anomalies in the data more comprehensively. Finally, the multi-algorithm weighted analysis method is also efficient and can complete the anomaly detection task in a shorter time. By combining the results of multiple algorithms and using a weighted voting algorithm for comprehensive judgment, the multi-algorithm weighted analysis method can obtain the results of anomaly detection more quickly.

[0095] Step S204: large model analysis.

[0096] After determining that the current situation meets the conditions for initiating an alarm, the system will perform a series of steps to further analyze and handle the abnormal situation. First, the system will retrieve all abnormal log entries recorded in the current time window. These logs describe in detail the abnormal events encountered during the operation of the system. Then, the system will combine these abnormal log information with the pre-set query template to form a special large model call statement. For example, assuming that the abnormal information "CommunicationsException: Communications link failure" frequently appears in the system log, the system will construct the following call statement: "A large number of abnormal logs suddenly appear in the system. The types of abnormal logs include but are not limited to the following. Please analyze the root cause in detail and provide feasible solutions for reference: CommunicationsException: Communications link failure". This call statement is intended to guide the large model to conduct in-depth analysis of the abnormal logs in order to obtain specific abnormal causes and response strategies.

[0097] After receiving the above call statement, the big model will use its powerful data processing and pattern recognition capabilities to conduct a detailed analysis of the "CommunicationsException: Communications link failure" in the exception log. An example of the analysis result may be as follows: The exception log indicates that there is a database connection communication failure. To solve this problem, it is recommended to take the following steps: First, check whether the network connection is normal to ensure that data transmission is unimpeded; second, confirm the database configuration information, including whether the login credentials and connection parameters are correct; third, monitor the current status of the database service to confirm whether it is running normally; in addition, adjust the database connection timeout setting to adapt to the current network environment; check the compatibility of the database driver with the current system environment; review the configuration of the database connection pool to ensure that resources are allocated reasonably; monitor the resource utilization of the database server to evaluate whether the exception is caused by resource shortage; finally, continue to monitor and analyze the exception log to prevent similar problems from happening again.

[0098] Once the analysis results of the big model are returned, including in-depth interpretation of the abnormal log and recommended solutions, this information will be integrated into the alarm notification. The alarm notification will not only point out the existence of the problem, but also provide the possible cause of the problem and targeted solution suggestions, so as to help the team members receiving the alarm quickly understand the background of the problem, take effective countermeasures, shorten the fault response time, and improve the overall maintenance efficiency of the system.

[0099] Step S205: Based on the results of the demerits calculated by the adaptive weighting method and the large model abnormal analysis, the system will intelligently call the corresponding alarm method. If an alarm is required, the alarm template is taken out to fill in the real data (fill in the alarm content, occurrence time, detailed cause of the problem returned by the large model, solution, etc.), as shown in the following table:

[0100]

[0101] Furthermore, the alarm system starts an asynchronous thread to connect to each alarm channel for alarm. The alarm record and alarm content, alarm channel, alarm time, etc. are written into t_warn_history. See the following table for an example:

[0102]

[0103]

[0104] Finally, if Figure 5 As shown, the exception log records are displayed graphically for easy viewing.

[0105] In the above steps, firstly, the combination of Filebeat and Kafka is used to collect the abnormal logs under the distributed system at high speed, and then the multi-algorithm adaptive weighted voting analysis method is used to judge the alarm. The multi-algorithm adaptive weighted calculation method and its sub-algorithms adopted in this patent are all statistical anomaly detection algorithms, which are extremely fast. Through fast log collection and fast anomaly detection, the time from the occurrence of anomalies to the discovery of anomalies can be greatly reduced. At the same time, different weights can be set for the sub-algorithms of the multi-algorithm weighted voting analysis method for different system scenarios to improve the accuracy of alarms in various scenarios. Therefore, the present invention can improve the timeliness and accuracy of alarms. After the anomaly is found, the preset content is combined with the abnormal log, and the relevant capabilities of the large model are called to allow the large model to analyze the approximate cause of the anomaly and the relevant solutions, and the approximate cause and solution are encapsulated in the final alarm SMS to remind relevant personnel. The use of large model analysis can save a lot of time for maintenance personnel to find logs, analyze logs, and troubleshoot causes, so that relevant personnel can analyze and solve problems faster.

[0106] Figure 6 is a structural diagram of a log data processing device according to an embodiment of the present application, such as Figure 6 As shown, the device comprises:

[0107] The first acquisition module 61 is used to acquire a log data set corresponding to a data source.

[0108] The statistical module 62 is used to periodically obtain abnormal log data recorded according to preset rules in the log data set according to a preset time period, and count the number of abnormal log data according to a preset time window, wherein the duration corresponding to the preset time window is less than the duration corresponding to the preset time period.

[0109] The second acquisition module 63 is used to obtain the target number of abnormal log data in the target time window, and to obtain the first number of abnormal log data in the first time window within a preset historical time length, wherein the target time window is the time window where the abnormal log data to be processed is located, and the first time window is the time window adjacent to the target time window.

[0110] The first determination module 64 is used to determine multiple statistical results between the target quantity and the first quantity by using multiple different preset statistical algorithms.

[0111] The second determination module 65 is used to determine the weight corresponding to each preset statistical algorithm according to the usage order of each preset statistical algorithm in the time dimension.

[0112] The third determination module 66 is used to perform weighted sum calculation on the statistical results of the preset statistical algorithms according to the weights corresponding to each preset statistical algorithm, and determine whether to generate alarm information for abnormal log data in the target time window according to the calculation results.

[0113] Optionally, the first determination module 64 is also used to perform the following steps: using a K-sigma algorithm to determine the standard deviation between the target quantity and the target average value, wherein the target average value is the average value of the first quantity corresponding to the first time window within a preset historical time length; when the standard deviation is greater than a first preset threshold, determining the first statistical result output by the K-sigma algorithm to be 1, and when the standard deviation is not greater than the first preset threshold, determining the first statistical result output by the K-sigma algorithm to be 0; using a kernel density estimation algorithm to determine a probability density estimate corresponding to the target quantity; when the probability density estimate is greater than a second preset threshold, determining the second statistical result output by the kernel density estimation algorithm to be 1, and when the probability density estimate is not greater than the second preset threshold, determining the second statistical result output by the kernel density estimation algorithm to be 0; using an isolation forest algorithm to determine an abnormality score corresponding to the target quantity; when the abnormality score is greater than a third preset threshold, determining the third statistical result output by the isolation forest algorithm to be 1, and when the abnormality score is not greater than the third preset threshold, determining the third statistical result output by the isolation forest algorithm to be 0.

[0114] Optionally, the log data processing device also includes: a fourth determination module, which is used to perform the following steps after determining whether to generate an alarm message for abnormal log data in the target time window: obtaining a weight list, wherein the weight list includes weights corresponding to each preset statistical algorithm; if, according to the calculation result, it is determined to generate an alarm message for abnormal log data in the target time window, the initial weight corresponding to the first target algorithm is increased to obtain a first weight, and the initial weight corresponding to the first target algorithm is updated in the weight list by the first weight to obtain an updated weight list, wherein the first target algorithm is a preset statistical algorithm whose statistical result is 1; if, according to the calculation result, it is determined to generate an alarm message for abnormal log data in the target time window, the initial weight corresponding to the first target algorithm is increased to obtain a first weight, and the initial weight corresponding to the first target algorithm is updated in the weight list by the first weight to obtain an updated weight list, wherein the first target algorithm is a preset statistical algorithm whose statistical result is 1; Determine whether to generate alarm information for abnormal log data in the target time window, increase the initial weight corresponding to the second target algorithm to obtain a second weight, and update the initial weight corresponding to the second target algorithm by the second weight in the weight list to obtain an updated weight list, wherein the second target algorithm is a preset statistical algorithm with a statistical result of 0; use multiple different preset statistical algorithms to determine multiple first statistical results between the second number and the first number of abnormal log data in the first target time window; perform weighted sum calculation on the first statistical results according to the updated weight list, and determine whether to generate alarm information for the abnormal log data in the first target time window according to the calculation result.

[0115] Optionally, the second determination module 65 is further used to perform the following steps: for the i-th preset statistical algorithm used in the time dimension, determine its corresponding identification information as i, where i is a positive integer not greater than n, and n is the number of preset statistical algorithms; determine the weight w corresponding to the i-th preset statistical algorithm by the following formula: i :

[0116]

[0117] Optionally, the third determination module 66 is further used to perform the following steps: determine whether the calculation result is greater than the inverse of the number of preset statistical algorithms, and if the calculation result is greater than the inverse of the number of preset statistical algorithms, generate an alarm message for abnormal log data in the target time window.

[0118] Optionally, the log data processing device is also used to execute the following steps when it is determined to generate alarm information for abnormal log data in the target time window: query the abnormal log content of the abnormal log data in the target time window; combine the abnormal log content with a preset statement to obtain a target statement, and call a pre-trained deep learning model through the target statement; obtain the output result generated by the pre-trained deep learning model in response to the target statement.

[0119] Optionally, the log data processing device is also used to perform the following steps after obtaining the output result generated by the pre-trained deep learning model in response to the target statement: sending the output result to the terminal device corresponding to the pre-stored terminal device identification set, wherein the output result includes at least one of the following: abnormal cause information and processing plan information corresponding to the log content.

[0120] It should be noted that the above Figure 6 The modules in the embodiment may be program modules (e.g., a set of program instructions for implementing a specific function) or hardware modules. For the latter, they may be expressed in the following forms, but are not limited thereto: the expression form of the above modules is a processor, or the functions of the above modules are implemented by a processor.

[0121] It should be noted that Figure 6 The preferred implementation of the illustrated embodiment can be found in Figure 1 The relevant description of the illustrated embodiment will not be repeated here.

[0122] Figure 7 FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing a method for processing log data. Figure 7As shown, the computer terminal 70 may include one or more (shown as 702a, 702b, ..., 702n in the figure) processors 702 (the processor 702 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 704 for storing data, and a transmission module 706 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 7 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 7 More or fewer components as shown, or with Figure 7 Different configurations are shown.

[0123] It should be noted that the one or more processors 702 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 70. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0124] The memory 704 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the method for processing log data in the embodiment of the present application. The processor 702 executes various functional applications and data processing by running the software programs and modules stored in the memory 704, that is, the method for processing log data described above is realized. The memory 704 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 704 may further include a memory remotely arranged relative to the processor 702, and these remote memories may be connected to the computer terminal 70 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0125] The transmission module 706 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 70. In one example, the transmission module 706 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 706 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0126] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 70 .

[0127] It should be noted that, in some optional embodiments, the above Figure 7 The computer terminal shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. It should be noted that Figure 7 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computer terminal described above.

[0128] It should be noted that Figure 7 The computer terminal shown is used to execute Figure 1 The method for processing log data shown in the figure, therefore the relevant explanations in the execution method of the above command are also applicable to the electronic device and will not be repeated here.

[0129] An embodiment of the present application further provides a non-volatile storage medium, which includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the above log data processing method.

[0130] A program for a non-volatile storage medium to perform the following functions: obtaining a log data set corresponding to a data source; periodically obtaining abnormal log data recorded according to preset rules in the log data set according to a preset time period, and counting the number of abnormal log data according to a preset time window, wherein the duration corresponding to the preset time window is less than the duration corresponding to the preset time period; obtaining a target number of abnormal log data in a target time window, and obtaining a first number of abnormal log data in a first time window within a preset historical duration, wherein the target time window is a time window where the abnormal log data to be processed is located, and the first time window is a time window adjacent to the target time window; using multiple different preset statistical algorithms, determining multiple statistical results between the target number and the first number; determining the weight corresponding to each preset statistical algorithm according to the order in which each preset statistical algorithm is used in the time dimension; performing weighted sum calculation on the statistical results of the preset statistical algorithm according to the weight corresponding to each preset statistical algorithm, and determining whether to generate an alarm message for the abnormal log data in the target time window according to the calculation result.

[0131] An embodiment of the present application further provides an electronic device, including: a memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the above log data processing method is executed when the program is running.

[0132] The processor is used to run a program that performs the following functions: obtain a log data set corresponding to a data source; periodically obtain abnormal log data recorded according to preset rules in the log data set according to a preset time period, and count the number of abnormal log data according to a preset time window, wherein the duration corresponding to the preset time window is less than the duration corresponding to the preset time period; obtain a target number of abnormal log data in a target time window, and obtain a first number of abnormal log data in a first time window within a preset historical duration, wherein the target time window is a time window where the abnormal log data to be processed is located, and the first time window is a time window adjacent to the target time window; use multiple different preset statistical algorithms to determine multiple statistical results between the target number and the first number; determine the weight corresponding to each preset statistical algorithm according to the order in which each preset statistical algorithm is used in the time dimension; perform weighted sum calculation on the statistical results of the preset statistical algorithm according to the weight corresponding to each preset statistical algorithm, and determine whether to generate an alarm message for the abnormal log data in the target time window according to the calculation result.

[0133] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0134] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0135] In the above-mentioned embodiments of the present application, the collected information is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary protection measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0136] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0137] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0138] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0139] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0140] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for processing log data, characterized in that: include: Get the log data set corresponding to the data source; According to a preset time period, periodically obtain the abnormal log data recorded according to the preset rules in the log data set, and count the number of the abnormal log data according to a preset time window, wherein the duration corresponding to the preset time window is less than the duration corresponding to the preset time period; Obtaining a target number of abnormal log data in a target time window, and obtaining a first number of abnormal log data in a first time window within a preset historical duration, wherein the target time window is a time window in which the abnormal log data to be processed is located, and the first time window is a time window adjacent to the target time window; Determine a plurality of statistical results between the target quantity and the first quantity using a plurality of different preset statistical algorithms; Determine the weight corresponding to each of the preset statistical algorithms according to the order in which each of the preset statistical algorithms is used in the time dimension; According to the weight corresponding to each of the preset statistical algorithms, a weighted sum calculation is performed on the statistical results of the preset statistical algorithms, and according to the calculation results, it is determined whether to generate alarm information for abnormal log data in the target time window.

2. The method according to claim 1, characterized in that: Using a plurality of different preset statistical algorithms, a plurality of statistical results between the target quantity and the first quantity are determined, including: Determine the standard deviation between the target quantity and the target average value by using a K-sigma algorithm, wherein the target average value is the average value of the first quantity corresponding to the first time window within the preset historical duration; when the standard deviation is greater than a first preset threshold, determine that a first statistical result output by the K-sigma algorithm is 1; and when the standard deviation is not greater than the first preset threshold, determine that the first statistical result output by the K-sigma algorithm is 0; Determine a probability density estimation value corresponding to the target quantity by using a kernel density estimation algorithm; when the probability density estimation value is greater than a second preset threshold, determine that a second statistical result output by the kernel density estimation algorithm is 1; and when the probability density estimation value is not greater than the second preset threshold, determine that the second statistical result output by the kernel density estimation algorithm is 0; Using the isolation forest algorithm, determine the abnormality score corresponding to the target quantity; when the abnormality score is greater than a third preset threshold, determine that the third statistical result output by the isolation forest algorithm is 1; when the abnormality score is not greater than the third preset threshold, determine that the third statistical result output by the isolation forest algorithm is 0.

3. The method according to claim 2, characterized in that After determining whether to generate alarm information for abnormal log data in the target time window, the method further includes: Obtain a weight list, wherein the weight list includes the weight corresponding to each of the preset statistical algorithms; If it is determined to generate alarm information for abnormal log data in the target time window according to the calculation result, the initial weight corresponding to the first target algorithm is increased to obtain a first weight, and the initial weight corresponding to the first target algorithm is updated by the first weight in the weight list to obtain an updated weight list, wherein the first target algorithm is a preset statistical algorithm with a statistical result of 1; If it is determined according to the calculation result that no alarm information is generated for the abnormal log data in the target time window, the initial weight corresponding to the second target algorithm is increased to obtain a second weight, and the initial weight corresponding to the second target algorithm is updated in the weight list by the second weight to obtain an updated weight list, wherein the second target algorithm is a preset statistical algorithm with a statistical result of 0; Determine a plurality of first statistical results between a second amount of abnormal log data in a first target time window and the first amount using a plurality of different preset statistical algorithms; According to the updated weight list, a weighted sum calculation is performed on the first statistical result, and according to the calculation result, it is determined whether to generate alarm information for abnormal log data in the first target time window.

4. The method according to claim 1, characterized in that Determining a weight corresponding to each of the preset statistical algorithms according to the order in which each of the preset statistical algorithms is used in the time dimension includes: For the preset statistical algorithm used in the time dimension, the corresponding identification information is determined as i, where i is a positive integer not greater than n, and n is the number of the preset statistical algorithms; the weight w corresponding to the i-th preset statistical algorithm is determined by the following formula: i :

5. The method according to claim 1, characterized in that Determining whether to generate warning information for abnormal log data in the target time window according to the calculation result includes: It is determined whether the calculation result is greater than the inverse of the number of the preset statistical algorithm. If the calculation result is greater than the inverse of the number of the preset statistical algorithm, an alarm message for abnormal log data in the target time window is generated.

6. The method according to claim 1, characterized in that In the case of determining to generate alarm information for abnormal log data in the target time window, the method further includes: Query the abnormal log content of the abnormal log data in the target time window; Combining the abnormal log content with a preset sentence to obtain a target sentence, and calling a pre-trained deep learning model through the target sentence; Obtain an output result generated by the pre-trained deep learning model in response to the target sentence.

7. The method according to claim 6, characterized in that After obtaining the output result generated by the pre-trained deep learning model in response to the target sentence, the method further includes: The output result is sent to the terminal device corresponding to the pre-stored terminal device identification set, wherein the output result includes at least one of the following: abnormal cause information and processing solution information corresponding to the log content.

8. A log data processing device, characterized in that: include: The first acquisition module is used to acquire the log data set corresponding to the data source; A statistics module, used to periodically obtain abnormal log data recorded according to preset rules in the log data set according to a preset time period, and count the number of abnormal log data according to a preset time window, wherein the duration corresponding to the preset time window is less than the duration corresponding to the preset time period; A second acquisition module is used to acquire a target number of abnormal log data in a target time window, and acquire a first number of abnormal log data in a first time window within a preset historical duration, wherein the target time window is a time window in which the abnormal log data to be processed is located, and the first time window is a time window adjacent to the target time window; A first determination module, configured to determine a plurality of statistical results between the target quantity and the first quantity using a plurality of different preset statistical algorithms; A second determination module is used to determine the weight corresponding to each of the preset statistical algorithms according to the order in which each of the preset statistical algorithms is used in the time dimension; The third determination module is used to perform weighted sum calculation on the statistical results of the preset statistical algorithms according to the weights corresponding to each of the preset statistical algorithms, and determine whether to generate alarm information for abnormal log data in the target time window according to the calculation results.

9. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein when the program is executed, the device where the non-volatile storage medium is located is controlled to execute the log data processing method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: A memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the program executes the log data processing method described in any one of claims 1 to 7 when running.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for processing log data according to any one of claims 1 to 7 is implemented.

Images