Log anomaly detection method and device, equipment and medium

Abstract

The invention discloses a log anomaly detection method and device, equipment and a medium. The log anomaly detection method comprises the following steps: based on log scene matching sample data, training a linear regression model and a lasso algorithm model, and determining a log scene automatic matching model; according to the to-be-analyzed log and the log scene automatic matching model, determining a log request response prediction duration, and obtaining a request actual response duration of the to-be-analyzed log; and when the residual between the actual request response duration of the to-be-analyzed log and the log request response prediction duration is greater than a target residual threshold, generating abnormal log alarm data. According to the technical scheme provided by the embodiment of the invention, the abnormal log in a complex service scene can be identified, and the identification accuracy is high.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a log anomaly detection method, device, equipment and medium. Background Art

[0002] With the scale and complexity of modern information systems, log data has become a core data source for system performance monitoring, security threat detection, and business operations. In addition, the rapid growth of log data has put higher demands on system processing capabilities. How to process large amounts of log data automatically and in multiple scenarios and discover potential problems in real time has become one of the core needs of enterprise operations.

[0003] Traditional log anomaly detection often uses a unified analysis method, which is difficult to adapt to the dynamic changes in the system operating environment and cannot perform optimized analysis for different business needs. It can only rely on operation and maintenance personnel to manually set features and anomaly detection conditions based on specific logs. It cannot automatically adapt to changes in log data, which increases the workload and is prone to misjudgment. It is even more powerless in complex business scenarios. Summary of the invention

[0004] The present invention provides a log anomaly detection method, device, equipment and medium to solve the problem that the existing log anomaly detection cannot adapt to complex and changeable business scenarios and has a high misjudgment rate.

[0005] According to one aspect of the present invention, a log anomaly detection method is provided, comprising:

[0006] Based on the log scene matching sample data, the linear regression model and the lasso algorithm model are trained to determine the log scene automatic matching model;

[0007] Automatically match the model based on the log to be analyzed and the log scenario to determine the predicted response time of the log request, and obtain the actual response time of the request of the log to be analyzed;

[0008] When the residual between the actual response time of the request of the log to be analyzed and the predicted response time of the log request is greater than the target residual threshold, abnormal log alarm data is generated.

[0009] According to another aspect of the present invention, a log anomaly detection device is provided, comprising:

[0010] Model training module, used to train linear regression model and lasso algorithm model based on log scene matching sample data, and determine the log scene automatic matching model;

[0011] The response time determination module is used to automatically match the model according to the log to be analyzed and the log scenario, determine the predicted response time of the log request, and obtain the actual response time of the request of the log to be analyzed;

[0012] The residual calculation module is used to generate abnormal log alarm data when the residual between the actual response time of the request of the log to be analyzed and the predicted response time of the log request is greater than the target residual threshold.

[0013] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the log anomaly detection method described in any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the log anomaly detection method described in any embodiment of the present invention when executed.

[0018] The technical solution of the embodiment of the present invention is to determine the log scene automatic matching model by training the linear regression model and the lasso algorithm model based on the log scene matching sample data, thereby determining the predicted duration of the log request response according to the log to be analyzed and the log scene automatic matching model, and obtaining the actual response duration of the request of the log to be analyzed, and then generating abnormal log alarm data when the residual of the actual response duration of the request of the log to be analyzed and the predicted duration of the log request response is greater than the target residual threshold. In this solution, the log scene automatic matching model can automatically adapt to the business scenario of the log to be analyzed, more flexibly identify the abnormalities of the log to be analyzed, reduce the recognition complexity of abnormal logs in diversified business scenarios, solve the problem that the existing log anomaly detection cannot adapt to complex and changeable business scenarios, and has a high misjudgment rate, and can identify abnormal logs in complex business scenarios with high recognition accuracy.

[0019] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 A flow chart of a log anomaly detection method provided in Example 1 of the present invention;

[0022] Figure 2 A flow chart of a log anomaly detection method provided in Embodiment 2 of the present invention;

[0023] Figure 3 A schematic diagram of the structure of a log anomaly detection device provided in Embodiment 3 of the present invention;

[0024] Figure 4 A schematic diagram of the structure of an electronic device that can be used to implement an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "original", "target", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] Embodiment 1

[0028] Figure 1This is a flow chart of a log anomaly detection method provided in the first embodiment of the present invention. This embodiment can be applied to accurately identify abnormal logs in complex scenarios. The method can be performed by a log anomaly detection device, which can be implemented in the form of hardware and / or software. The log anomaly detection device can be configured in an electronic device. The electronic device includes a computer or a server. Figure 1 As shown, the method includes:

[0029] Step 110: Based on the log scene matching sample data, a linear regression model and a lasso algorithm model are trained to determine a log scene automatic matching model.

[0030] The log scene matching sample data may be log sample data in different business scenarios, that is, the log data in different business scenarios is processed according to the model recognition requirements and the sample data is obtained. The lasso algorithm model is a LASSO regression model. The log scene automatic matching model may be a model for adapting and analyzing log data in different business scenarios.

[0031] In an embodiment of the present invention, log scene matching sample data can be obtained, and then the linear regression model and the lasso algorithm model can be trained respectively using the log scene matching sample data. Based on the analysis advantages of the trained linear regression model and the lasso algorithm model for different scenarios, the log scene automatic matching model is determined.

[0032] In an optional embodiment of the present invention, before determining the log scene automatic matching model based on the log scene matching sample data, training the linear regression model and the lasso algorithm model, it can also include: obtaining the original log data; performing data preprocessing on the original log data to obtain preprocessed log data, and extracting features of the preprocessed log data to obtain log scene matching sample data.

[0033] The original log data may be log data in different business scenarios, and the pre-processed log data may be the result of data processing on the original log data based on preset data processing rules.

[0034] In an embodiment of the present invention, log data in a selected time window for different business scenarios can be collected, and the collected log data can be used as original log data. Then, the original log data can be preprocessed using preset data processing rules to obtain preprocessed log data. Then, the features of the preprocessed log data can be extracted for the model input data form of the linear regression model and the lasso algorithm model to obtain log scenario matching sample data.

[0035] Optionally, multiple key features can be extracted from different fields of the preprocessed log data, and the key features can be standardized to use the standardized features as log scene matching sample data. For example, timestamps can be converted into time periods, IP (Internet Protocol) addresses can be encoded, and numerical features (such as system load, response time) can be standardized.

[0036] Exemplarily, the selected time window may be the past week or month, which can be set by the user. The preset data processing rules may include, but are not limited to, log cleaning processing. Log cleaning processing includes removing invalid or duplicate log data and correcting erroneous log data based on automated tools. Removing invalid logs means removing blank or erroneous format logs. Correcting erroneous format log data can be understood as performing data normalization, such as converting timestamps of different formats in log data into a unified format, such as YYYY-MM-DD HH:MM:SS.

[0037] In a specific example, assuming that the original log data is: 2024-10-17 12:30:45, UID12345, GET, 192.168.1.100, 0.25 seconds, 75%. After data preprocessing and feature extraction, the obtained log scene matching sample data is: 12:30-13:00, user ID: UID12345, request type: GET, IP address code: 1001, response time: 0.25, system load: 0.75.

[0038] Step 120: determine the predicted response time of the log request according to the log to be analyzed and the log scenario automatic matching model, and obtain the actual response time of the request of the log to be analyzed.

[0039] The log to be analyzed may be a log currently being identified for anomalies, for example, real-time log data. The predicted response time of a log request may be a predicted response time of a request corresponding to the log to be analyzed. The actual response time of a request may be an actual response time of a request corresponding to the log to be analyzed.

[0040] In an embodiment of the present invention, features of the log to be analyzed can be extracted, and the extracted features can be input into the log scenario automatic matching model, so that a model suitable for the log scenario (equivalent to the business scenario) can be automatically adapted through the log scenario automatic matching model to determine the predicted response time of the log request of the log to be analyzed, and obtain the actual response time of the request of the log to be analyzed.

[0041] Step 130: When the residual between the actual response time of the request of the log to be analyzed and the predicted response time of the log request is greater than the target residual threshold, abnormal log alarm data is generated.

[0042] The target residual threshold may be a preset residual threshold. The abnormal log alarm data may be alarm data generated when the log to be analyzed is an abnormal log.

[0043] In an embodiment of the present invention, the residual between the actual response time of the request of the log to be analyzed and the predicted response time of the log request can be compared with the target residual threshold. If it is less than the target residual threshold, the log to be analyzed is characterized as a normal log. If it is greater than the target residual threshold, the log to be analyzed is characterized as an abnormal log, and then relevant information of the log to be analyzed is collected to generate abnormal log alarm data.

[0044] The technical solution of the embodiment of the present invention is to determine the log scene automatic matching model by training the linear regression model and the lasso algorithm model based on the log scene matching sample data, thereby determining the predicted duration of the log request response according to the log to be analyzed and the log scene automatic matching model, and obtaining the actual response duration of the request of the log to be analyzed, and then generating abnormal log alarm data when the residual of the actual response duration of the request of the log to be analyzed and the predicted duration of the log request response is greater than the target residual threshold. In this solution, the log scene automatic matching model can automatically adapt to the business scenario of the log to be analyzed, more flexibly identify the abnormalities of the log to be analyzed, reduce the recognition complexity of abnormal logs in diversified business scenarios, solve the problem that the existing log anomaly detection cannot adapt to complex and changeable business scenarios, and has a high misjudgment rate, and can identify abnormal logs in complex business scenarios with high recognition accuracy.

[0045] Embodiment 2

[0046] Figure 2 This is a flow chart of a log anomaly detection method provided in the second embodiment of the present invention. This embodiment is specific based on the above embodiment and provides a specific optional implementation method for determining the log scene automatic matching model based on log scene matching sample data, training a linear regression model and a lasso algorithm model. Figure 2 As shown, the method includes:

[0047] Step 210: After matching sample data based on log scenarios and synchronously training the linear regression model and the lasso algorithm model, the model performance of the linear regression model and the lasso algorithm model in different log scenarios is evaluated based on model goodness of fit and / or residuals.

[0048] Among them, the model goodness of fit can be an indicator to measure the degree of fit of the model to the data.

[0049] In an embodiment of the present invention, the log scene matching sample data can be input into the linear regression model and the lasso algorithm model respectively, and the linear regression model and the lasso algorithm model can be trained synchronously. Based on the model fitting goodness of each sample data in the log scene matching sample data determined by the two models, and / or the residual, the model performance of the linear regression model and the lasso algorithm model under different log scenarios is evaluated.

[0050] In an optional embodiment of the present invention, the log scenario matching sample data may include: log feature data of a website monitoring scenario, log feature data of a server performance monitoring scenario, and log feature data of a security threat detection scenario.

[0051] Among them, log feature data can be used to distinguish log scenarios. Log feature data of website monitoring scenarios can be used to describe the characteristics of logs in website monitoring scenarios. Log feature data of website monitoring scenarios may include the presence of a large amount of request information and user access behavior data, etc. Log feature data of server performance monitoring scenarios are used to describe the characteristics of logs in server performance monitoring scenarios. Log feature data of website monitoring scenarios may include system performance indicator data such as system load, CPU (Central Processing Unit) usage, memory usage, etc. Log feature data of security threat detection scenarios are used to describe the characteristics of logs in security threat detection scenarios. Log feature data of security threat detection scenarios may include the presence of a large number of user IDs (identification identities), geographic locations, ports, data packet sizes, etc.

[0052] For example, the log scene matching sample data a is: time period: 12:30-13:00, user ID: UID12345, IP address: 192.168.1.100, request type: GET, response time: 0.25 seconds. The log scene automatic matching model can identify the log scene of the log scene matching sample data a as a website monitoring scene, because it involves a large amount of request information and access behavior data. The log scene matching sample data b is: 90%, CPU usage: 85%, memory usage: 80%, response time: 1.5 seconds, error rate: 0.01. The log scene automatic matching model can identify the log scene of the log scene matching sample data b as a server performance monitoring scene, because the logs are concentrated on system performance indicators.

[0053] In an embodiment of the present invention, a log scenario automatic matching model is constructed according to the log scenario to be identified to predict the target variable (such as response time, system load, etc.) and perform anomaly detection. The independent variables selected for each scenario are different. Exemplarily, the independent variables in the website monitoring scenario include time period, request frequency, user ID, IP address, request type, and the target variable includes response time. The independent variables in the server performance monitoring scenario include system load, CPU usage, memory usage, response time, error rate, and the target variable includes response time. The independent variables in the security threat detection scenario include user ID, IP address, request type, geographic location, port, data packet size, and the target variable includes data packet size.

[0054] Optionally, when there is a linear relationship between the dependent variable and the target variable, the log scene automatic matching model is a linear regression model. For example, the linear relationship between request frequency and response time in a website monitoring scenario. When there are many features corresponding to the independent variable and automatic feature selection is required, the log scene automatic matching model is a lasso algorithm model. The lasso algorithm model reduces the regression coefficient of unimportant features to 0 by introducing an L1 regularization term to simplify the model.

[0055] For the website monitoring scenario, assuming that the request frequency and time period are key features, the regression equation corresponding to the linear regression model can be Y=β0+β1X1+β2X2+∈. The regression coefficients β1 and β2 are calculated specifically by the gradient descent method or other optimization methods. Among them, Y represents the response time, β0 represents the intercept term, and ∈ represents an unobserved random variable. Exemplarily, it is calculated that β0=0.5, β1=0.3 (indicating that for every increase in request frequency of 1 time / minute, the response time increases by 0.03 seconds), and β2=-0.1 (indicating that the response time during non-peak hours decreases by 0.1 seconds).

[0056] For example, for the website monitoring scenario, the predicted response time of the log request predicted by the log scenario automatic matching model is 0.3 seconds, and the actual response time of the request is 0.35 seconds. When evaluating the model performance through the model fit goodness, the model fit goodness is calculated to be 0.85, indicating that the model fits the data well.

[0057] Step 220: Determine a log scene automatic matching model based on the model performance of the linear regression model and the lasso algorithm model in different log scenes.

[0058] In an embodiment of the present invention, the model performance of the linear regression model and the lasso algorithm model in different log scenarios can be compared to determine the model with better model performance in the same log scenario, so as to establish a mapping between different log scenarios and the better performing model and obtain a log scenario automatic matching model.

[0059] Step 230: determine the predicted response time of the log request according to the log to be analyzed and the log scenario automatic matching model, and obtain the actual response time of the request of the log to be analyzed.

[0060] In an optional embodiment of the present invention, determining the predicted duration of a log request response based on the log to be analyzed and the log scenario automatic matching model may include: extracting log feature data of the log to be analyzed; inputting the log feature data of the log to be analyzed into the log scenario automatic matching model to obtain the predicted duration of a log request response.

[0061] In an embodiment of the present invention, the log to be analyzed can be parsed to obtain log feature data of the log to be analyzed, and then the log feature data of the log to be analyzed can be input into the log scenario automatic matching model to analyze the log feature data of the log to be analyzed through the log scenario automatic matching model, and output the predicted response time of the log request.

[0062] In an optional embodiment of the present invention, after obtaining the actual response time of the request of the log to be analyzed, it can also include: obtaining a residual threshold adjustment request; parsing the residual threshold adjustment request to obtain a current residual threshold, and using the current residual threshold as a target residual threshold.

[0063] The residual threshold adjustment request may be a request issued by a user to update a target residual threshold. The current residual threshold may be a residual threshold adjusted based on the residual threshold adjustment request.

[0064] In an embodiment of the present invention, after obtaining the residual threshold adjustment request sent by the user, the residual threshold adjustment request is parsed, and then based on the adjustment parameters configured by the user in the residual threshold adjustment request, the current residual threshold is determined, so that the current residual threshold is used as the target residual threshold.

[0065] Optionally, the target residual threshold = μ+k×σ; where μ represents the mean of historical residuals, σ represents the residual standard deviation, and k is an adjustable parameter, which is equivalent to the adjustment parameter configured in the residual threshold adjustment request.

[0066] Step 240: When the residual between the actual response time of the request of the log to be analyzed and the predicted response time of the log request is greater than the target residual threshold, abnormal log alarm data is generated.

[0067] Exemplarily, after receiving the log to be analyzed, the independent variable of the corresponding scenario is extracted and input into the pre-trained log scenario automatic matching model to calculate the predicted duration of the log request response. For example, in the server performance monitoring scenario, the log to be analyzed received by the system in real time is: system load: 90%, CPU usage: 85%, memory usage: 80%, response time: 1.5 seconds, error rate: 0.01. The predicted duration of the log request response determined by the log scenario automatic matching model is 1.35 seconds. Assuming that the actual response duration of the request is 1.5 seconds, the residual is 0.15 seconds (1.5-1.35). Assuming the target residual threshold is 0.1, the log to be analyzed is determined to be an abnormal log.

[0068] In an optional embodiment of the present invention, after generating abnormal log alarm data, it can also include: sending the abnormal log alarm data to the target operation and maintenance personnel via email or text message, and recording the abnormal log alarm data; obtaining model update sample data according to the model update cycle, and training the log scenario automatic matching model through the model update sample data.

[0069] The target operation and maintenance personnel may be personnel who receive abnormal log alarm data. The model update cycle may be a pre-set model optimization cycle. The model update sample data may be sample data for updating and training the log scenario automatic matching model.

[0070] In an embodiment of the present invention, the generated abnormal log alarm data can be sent to the target operation and maintenance personnel via email or text message to promptly notify the target operation and maintenance personnel to perform operation and maintenance, and the abnormal log alarm data can be recorded and stored for technical personnel to analyze. It is also possible to collect model update sample data based on a preset model update cycle, and automatically match the model through the log scenario training model update sample data to ensure that the model can reflect the latest status of the system.

[0071] This solution can automatically identify and distinguish different types of log scenarios (such as website monitoring scenarios, server performance monitoring scenarios, and security threat detection scenarios), automatically select appropriate features according to different scenarios, and build a log scenario automatic matching model. The log scenario automatic matching model is highly flexible and can be optimized for different business needs. It is suitable for different scenarios, avoiding the tedious process of manually setting rules in traditional solutions, and can also regularly update and optimize the model according to changes in real-time data.

[0072] Through feature selection and regularization (such as the lasso algorithm model), unimportant features can be automatically removed, model efficiency and accuracy can be improved, the model can be kept simple and efficient, and the risk of overfitting can be reduced. Dynamic thresholds are calculated based on historical residual data to adjust the sensitivity of anomaly detection and reduce false positives and negatives. This dynamic adjustment mechanism can adapt to changes in the system operating environment and improve the accuracy of anomaly detection. After receiving real-time log data, the model is automatically matched based on the trained log scenario for real-time prediction, and abnormal behavior is identified through residual analysis. When the residual exceeds the target residual threshold, the alarm mechanism is automatically triggered to ensure that the operation and maintenance team can deal with potential problems in a timely manner, greatly improving the efficiency of system monitoring.

[0073] The technical solution of the embodiment of the present invention is to evaluate the model performance of the linear regression model and the lasso algorithm model in different log scenarios based on the model goodness of fit and / or residual after synchronously training the linear regression model and the lasso algorithm model based on the log scene matching sample data, and then determine the log scene automatic matching model according to the model performance of the linear regression model and the lasso algorithm model in different log scenarios, thereby determining the predicted duration of the log request response according to the log to be analyzed and the log scene automatic matching model, and obtaining the actual response duration of the request of the log to be analyzed, and further generating abnormal log alarm data when the residual of the actual response duration of the request of the log to be analyzed and the predicted duration of the log request response is greater than the target residual threshold. In this solution, the log scene automatic matching model can automatically adapt to the business scenario of the log to be analyzed, more flexibly identify the abnormality of the log to be analyzed, reduce the recognition complexity of abnormal logs in diversified business scenarios, solve the problem that the existing log abnormality detection cannot adapt to complex and changeable business scenarios, and the error rate is high, and can identify abnormal logs in complex business scenarios, and the recognition accuracy is high.

[0074] Embodiment 3

[0075] Figure 3 A schematic diagram of the structure of a log anomaly detection device provided in Embodiment 3 of the present invention.

[0076] like Figure 3 As shown, the device comprises:

[0077] A model training module 310 is used to train a linear regression model and a lasso algorithm model based on log scene matching sample data to determine a log scene automatic matching model;

[0078] The response time determination module 320 is used to automatically match the model according to the log to be analyzed and the log scenario, determine the predicted response time of the log request, and obtain the actual response time of the request of the log to be analyzed;

[0079] The residual calculation module 330 is used to generate abnormal log alarm data when the residual between the actual response time of the request of the log to be analyzed and the predicted response time of the log request is greater than the target residual threshold.

[0080] The technical solution of the embodiment of the present invention is to determine the log scene automatic matching model by training the linear regression model and the lasso algorithm model based on the log scene matching sample data, thereby determining the predicted duration of the log request response according to the log to be analyzed and the log scene automatic matching model, and obtaining the actual response duration of the request of the log to be analyzed, and then generating abnormal log alarm data when the residual of the actual response duration of the request of the log to be analyzed and the predicted duration of the log request response is greater than the target residual threshold. In this solution, the log scene automatic matching model can automatically adapt to the business scenario of the log to be analyzed, more flexibly identify the abnormalities of the log to be analyzed, reduce the recognition complexity of abnormal logs in diversified business scenarios, solve the problem that the existing log anomaly detection cannot adapt to complex and changeable business scenarios, and has a high misjudgment rate, and can identify abnormal logs in complex business scenarios with high recognition accuracy.

[0081] Optionally, the log anomaly detection device also includes a feature extraction model for obtaining original log data; performing data preprocessing on the original log data to obtain preprocessed log data, and extracting features of the preprocessed log data to obtain the log scene matching sample data.

[0082] Optionally, the log scenario matching sample data may include: log feature data of a website monitoring scenario, log feature data of a server performance monitoring scenario, and log feature data of a security threat detection scenario.

[0083] Optionally, the model training module 310 is specifically used to evaluate the model performance of the linear regression model and the lasso algorithm model in different log scenarios based on the model goodness of fit and / or residuals after synchronously training the linear regression model and the lasso algorithm model based on the log scenario matching sample data; determine the log scenario automatic matching model according to the model performance of the linear regression model and the lasso algorithm model in different log scenarios.

[0084] Optionally, the response duration determination module 320 is specifically used to extract log feature data of the log to be analyzed; input the log feature data of the log to be analyzed into the log scenario automatic matching model to obtain the predicted duration of the log request response.

[0085] Optionally, the log anomaly detection device further includes a target residual threshold determination module, which is used to obtain a residual threshold adjustment request; parse the residual threshold adjustment request to obtain a current residual threshold, and use the current residual threshold as the target residual threshold.

[0086] Optionally, the log anomaly detection device also includes a model update module, which is used to send the abnormal log alarm data to the target operation and maintenance personnel via email or text message, and record the abnormal log alarm data; according to the model update cycle, obtain model update sample data, and train the log scenario automatic matching model through the model update sample data.

[0087] The log anomaly detection device provided in the embodiment of the present invention can execute the log anomaly detection method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0088] Embodiment 4

[0089] Figure 4 A schematic diagram of an electronic device that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0090] like Figure 4 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0091] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0092] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a log anomaly detection method.

[0093] In some embodiments, the log anomaly detection method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the log anomaly detection method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the log anomaly detection method in any other appropriate manner (e.g., by means of firmware).

[0094] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0095] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0096] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0097] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0098] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0099] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of traditional physical hosts and VPS servers, which are difficult to manage and have weak business scalability.

[0100] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0101] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A log anomaly detection method, characterized in that: include: Based on the log scene matching sample data, the linear regression model and the lasso algorithm model are trained to determine the log scene automatic matching model; According to the log to be analyzed and the log scenario automatic matching model, the predicted response time of the log request is determined, and the actual response time of the request of the log to be analyzed is obtained; When the residual between the actual response time of the request of the log to be analyzed and the predicted response time of the log request is greater than the target residual threshold, abnormal log alarm data is generated.

2. The method according to claim 1, characterized in that: Before the step of matching sample data based on log scenarios, training a linear regression model and a lasso algorithm model, and determining an automatic log scenario matching model, the step further includes: Get the original log data; The original log data is preprocessed to obtain preprocessed log data, and features of the preprocessed log data are extracted to obtain the log scene matching sample data.

3. The method according to claim 2, characterized in that The log scenario matching sample data includes: log feature data of a website monitoring scenario, log feature data of a server performance monitoring scenario, and log feature data of a security threat detection scenario.

4. The method according to claim 3, characterized in that The method of matching sample data based on log scenes, training a linear regression model and a lasso algorithm model, and determining a log scene automatic matching model includes: After matching the sample data based on the log scenario and synchronously training the linear regression model and the lasso algorithm model, the model performance of the linear regression model and the lasso algorithm model in different log scenarios is evaluated based on the model goodness of fit and / or residuals; The log scene automatic matching model is determined according to the model performance of the linear regression model and the lasso algorithm model in different log scenes.

5. The method according to claim 1, characterized in that: The automatic matching model based on the log to be analyzed and the log scenario determines the predicted duration of the log request response, including: Extracting log feature data of the log to be analyzed; The log feature data of the log to be analyzed is input into the log scene automatic matching model to obtain the predicted response time of the log request.

6. The method according to claim 1, characterized in that After obtaining the actual response time of the request for the log to be analyzed, the method further includes: Get residual threshold adjustment request; The residual threshold adjustment request is parsed to obtain a current residual threshold, and the current residual threshold is used as the target residual threshold.

7. The method according to claim 1, characterized in that After generating the abnormal log warning data, the method further includes: Send the abnormal log alarm data to the target operation and maintenance personnel via email or SMS, and record the abnormal log alarm data; According to the model update cycle, the model update sample data is obtained, and the log scene automatic matching model is trained by the model update sample data.

8. A log anomaly detection device, characterized in that: include: Model training module, used to train linear regression model and lasso algorithm model based on log scene matching sample data, and determine the log scene automatic matching model; The response time determination module is used to automatically match the model according to the log to be analyzed and the log scenario, determine the predicted response time of the log request, and obtain the actual response time of the request of the log to be analyzed; The residual calculation module is used to generate abnormal log alarm data when the residual of the actual response time of the request of the log to be analyzed and the predicted response time of the log request is greater than the target residual threshold.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the log anomaly detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the log anomaly detection method according to any one of claims 1 to 7 when executed.

Images